From 4b230cbd0f7e5c5ac4358b3c011e0861b1186458 Mon Sep 17 00:00:00 2001 From: Kurt Himebauch <136133082+xXJSONDeruloXx@users.noreply.github.com> Date: Tue, 27 Oct 2026 13:56:18 -0400 Subject: [PATCH] perf(proot): add opt-in glibc syscall patching --- docs/development/proroot-analysis.md | 311 +++++++++++++++++++++++++++ tools/proot/bench/svc-check.c | 121 +++++++++++ tools/proot/bench/svc-probe.c | 93 ++++++++ tools/proot/bench/svc-storage.c | 74 +++++++ tools/proot/bench/svc-workload.c | 91 ++++++++ tools/proot/fastpath/fastpath.c | 2 + tools/proot/fastpath/svc-patch.h | 250 +++++++++++++++++++++ tools/tests/test_fastpath_svc.py | 62 ++++++ 8 files changed, 1004 insertions(+) create mode 100644 docs/development/proroot-analysis.md create mode 100644 tools/proot/bench/svc-check.c create mode 100644 tools/proot/bench/svc-probe.c create mode 100644 tools/proot/bench/svc-storage.c create mode 100644 tools/proot/bench/svc-workload.c create mode 100644 tools/proot/fastpath/svc-patch.h create mode 100644 tools/tests/test_fastpath_svc.py diff --git a/docs/development/proroot-analysis.md b/docs/development/proroot-analysis.md new file mode 100644 index 0000000000..81be73a112 --- /dev/null +++ b/docs/development/proroot-analysis.md @@ -0,0 +1,311 @@ +# proroot: what it does and what DroidDeck can take from it + +Static analysis of [coderredlab/proroot](https://github.com/coderredlab/proroot) v1.2.8 +(`c01039ab`, 2026-06-17). The repository holds only five arm64 binaries and a README. The licence is +proprietary, and redistributing modified binaries is not allowed, so only the ideas below are +usable, not the code. The binaries were read with the NDK's `llvm-readelf`, `llvm-nm`, +`llvm-objdump` and `llvm-strings`. `libproroot-stub-loader.so` and `libproroot-bridge.so` still carry +`.symtab`, and `libproroot-runtime.so` exports about 260 names, so the function names below are +proroot's own. + +## Architecture + +proroot does not use ptrace. Everything runs inside the guest process. + +| file | kind | role | +|---|---|---| +| `libproroot.so` | bionic PIE executable | Launcher. Parses `-r/-b/-w/-0/--link2symlink`, writes a config file (`.proroot-config-`, passed down as `PROROOT_CFG_FD`), and execs the guest through `libproroot-linker.so` or the stub loader. | +| `libproroot-linker.so` | freestanding PIE, run as an executable | A from-scratch glibc dynamic linker ("cleanroom linker") that replaces `ld-linux-aarch64.so.1`. It maps the program and its `DT_NEEDED` libraries, relocates them, sets up static TLS, and stubs `_dl_*` and `__tunable_get_val`. It finds `_rtld_global_ro` through a table of offsets per glibc version ("no offset table for glibc X, patches skipped"). It also installs a SIGSYS handler with an allow-list. | +| `libproroot-stub-loader.so` | freestanding static PIE | Userland `exec`. It maps the target and its `PT_INTERP` itself and builds the stack and auxv. It then **rewrites the `svc #0` sites** in the loaded images (`proroot_svc_scan`, `proroot_svc_patch_apply`, `select_trampoline_for_site`) to branch into per-syscall trampolines (`proroot_tramp_openat`, `_statx`, `_faccessat2`, `_clone3`, `_rt_sigaction`, `_shmget`, ...). Sites it cannot patch fall back to an in-process seccomp `RET_TRAP` filter with a SIGSYS handler ("trampoline fallback, activating trap-mode fallback"). | +| `libproroot-runtime.so` | glibc `LD_PRELOAD` | Hooks about 200 libc entry points: path translation, fake root (`fake_id0_*`), link2symlink, exec routing, SysV shm, netlink, `/proc` and `/sys` views, `brk`, a fake DRM/KMS and GBM device plus libseat for compositors, X11 and Unix-socket fixups. After load it **patches the svc sites in glibc itself** (`patch_libc_syscalls`). | +| `libproroot-bridge.so` | static `ET_EXEC` at `0x7000000000` | The "trampoline" for static guest binaries. It installs a path-trap seccomp filter plus a SIGSYS handler, maps the target, and serves a virtual `/proc` (`status`, `stat`, `cmdline`, `environ`, `mounts`, `mountinfo`, `fdinfo`, `auxv`). | + +The executable files all live in `nativeLibraryDir`, which is the only place an app may exec from. +That is why the dynamic linker and the static loader each have their own userland loader. proot +handles the same constraint with `libproot-loader.so`. + +## How the in-process syscall interception works + +1. **Patching svc sites in glibc** (`patch_phdr_callback`, called through `dl_iterate_phdr`): + - It scans only an allow-list of objects: `libc.so.6`/`libc-`, `ld-linux`, `libpthread`, + `librt`, `libdl`. Main executables get patched early only for listed programs (apt, dpkg, + curl, openssl, node, chromium/electron/playwright, passwd/useradd/...). Other programs are + patched later (`log_deferred_libc_patch`, `PROROOT_FORCE_EARLY_PATCH`). + - In each executable `PT_LOAD` it finds `svc #0` (`0xd4000001`). It checks the instructions + before it for a `mov x8, #nr` it recognises. One fixed glibc `openat` call sequence gets + special treatment: its `bl` is redirected to a relay (`emit_openat_call_relay_entry`). + - The site is replaced with a `b`/`bl` to a relay stub on a page allocated within ±128 MiB + (`alloc_relay_page`). Sites out of range are logged as `bl-out-of-range`; `large-image + multi-cover tramp` places extra relay pages for big images. + - **Write path:** it first tries `mprotect(PROT_READ|PROT_WRITE|PROT_EXEC)` on the segment. + If that is refused, it `pwrite`s the 4-byte instruction through `/proc/self/mem`. Either way + it then calls `__clear_cache`. The log line counts sites by kind: `patched %d seccomp + %d + path (%d relay) + %d brk`. +2. **In-process SIGSYS fallback**: + - A BPF filter allows syscalls whose instruction pointer is inside proroot's own code and + returns `SECCOMP_RET_TRAP` for the path, exec, clone and wait syscalls from anywhere else. + The IP check is the same idea as our patch 0014, with TRAP instead of TRACE. + - The handler emulates the call and writes the result into the signal frame. Guest SIGSYS + handlers are kept and chained (`proroot_record_user_sigsys`, `invoke_user_sigsys`). +3. **Keeping SIGSYS deliverable**, the objection in option 4 of + [proot-performance](proot-performance.md): + - The `sigprocmask` and `pthread_sigmask` hooks call `sigtrap_unblocked_mask`, which + `sigdelset`s `SIGSYS` (and `SIGTRAP` for some guests) from any `SIG_BLOCK`/`SIG_SETMASK` set. + - `rt_sigaction` is guarded so the guest cannot replace the handler (`proroot_tramp_rt_sigaction`, + the `sigaction_guard` trace points). + - The raw `rt_sigprocmask` that glibc makes internally, for example `pthread_create` blocking + every signal, goes through a patched svc site. The stub loader and the runtime both compare + against syscall 135. +4. **Android-blocked syscalls are answered in-process.** The stub loader dispatches + `set_robust_list`, `rseq`, `faccessat2`, `clone3`, `futex_waitv` and `seccomp`. The runtime + emulates SysV shm (`emu_shm*`), `set*id`/`get*id` under `-0`, and `brk` + (`emulate_guest_brk`, over a reserved mapping). + +## Compared with DroidDeck + +| | DroidDeck (proot + 0001–0018 + `libblfastpath.so`) | proroot | +|---|---|---| +| exec, `PT_INTERP`, `/proc/self/exe` | proot's loader under ptrace, 0.4–0.8 ms per exec | Its own loaders, with no tracer; also needs its own ld.so with a glibc offset table | +| path calls through libc wrappers | fast path, about 0.6–1.9 µs | runtime hooks | +| path calls from glibc internals (ld.so, fts, realpath, nss, `__open_nocancel`) and raw `svc` | **still go to proot (15–60 µs)** | **patched svc sites** | +| Android-blocked syscalls (`set_robust_list`, `faccessat2`, `clone3`, ...) | SIGSYS stop answered by proot (0016 etc.), one ptrace round trip and one signal round trip each | in-process | +| `brk` | proot emulation, a trap per call (mitigated by `top_pad`) | in-process emulation | +| correctness fallback | proot is always there | none: anything missed runs unconfined or is emulated wrongly | +| long Unix socket paths | fast path `/proc/self/fd/N/name` aliases | short socket aliases (`proroot_create_short_unix_socket_alias`) | +| rseq | `GLIBC_TUNABLES=glibc.pthread.rseq=0` | trampoline | + +proroot is not a model for replacing proot. It has no tracer to fall back on, so a missed call +site or an unexpected glibc build silently breaks isolation or correctness. Its own linker depends +on glibc internals for each version. Proton, Wine, FEX and Steam's self-updating ELFs are a much +wider surface than the Node, Python and Chromium workloads proroot was tested on. Its value for +DroidDeck is as a fast path *in front of* proot. + +## What is worth taking + +1. **Patch glibc's svc sites from `libblfastpath.so`'s constructor.** This is the largest gain + and the cheapest way to do option 6 in [proot-performance](proot-performance.md) ("a proot-aware + glibc") without rebuilding glibc. + - For `libc.so.6` (and `ld-linux-aarch64.so.1` for runtime `dlopen`), rewrite each `svc #0` + that follows a recognised `mov x8, #nr` for the path syscalls the fast path already handles. + The rewrite is a `b` to a relay that calls the existing translator. + - The relay issues the syscall from the `0xffff00000` trampoline page, which 0014 already lets + through. Anything the translator declines is re-issued from an svc in the relay page, outside + the trampoline, so proot traps it exactly as it does today. Correctness still rests on proot. + - This covers fts (`find` over 2.7k files, unchanged today), realpath, nss, `__open_nocancel`, + and `dlopen` after start-up. + - Library loading at start-up, before the constructor runs, still traps. +2. **Answer the Android-blocked syscalls through those relays.** + - `set_robust_list` and `get_robust_list`: keep the head in TLS, as 0016 already does in + proot. + - `faccessat2`: answer `ENOSYS`, or translate it when flags are 0, as the wrapper does. + - `clone3`: answer `ENOSYS` before Android 15, so glibc falls back to `clone`. + - `futex_waitv`: droiddeck-fsync already handles this. + - Each one removes a SIGSYS stop plus a ptrace round trip, about 30–50 µs. `set_robust_list` + alone is one per thread, which matters for Steam/CEF and Wine thread churn. +3. **The `/proc/self/mem` write as a fallback for patching.** If the app domain refuses + `mprotect(PROT_WRITE|PROT_EXEC)` on rootfs (`app_data_file`) text, i.e. an `execmod` denial, + a write through `/proc/self/mem` makes the page copy-on-write without an `mprotect`. That page + is never writable and executable at once. Check this on the Thor before relying on it. The + same question is still open for the trampoline page (`execmem`). +4. **Keep SIGSYS unblocked if in-process `SECCOMP_RET_TRAP` is ever used.** Strip `SIGSYS` in the + `sigprocmask`/`pthread_sigmask`/`rt_sigprocmask` paths, and keep the handler installed while + chaining the guest's. This only becomes safe once item 1 covers glibc's internal + `rt_sigprocmask`. It is a prerequisite, not a recommendation to switch. +5. **Cache patch sites by build-id.** proroot rescans each image on every exec and defers patching + for most programs to keep exec cheap. A scan of libc's roughly 1.5 MiB of text is cheap, but a + per-build-id site table, written once in `/tmp` by the fast path, would make it free. + +Not worth taking: the cleanroom ld.so and its glibc offset table; the userland exec loader (proot +already does this); the fake DRM/GBM/libseat device (DroidDeck has its own compositor path); +netlink and SysV shm (`netif.c` and `sysv.c` already cover these); and the long-socket aliasing +(the fast path covers it). + +## Open questions before prototyping item 1 + +- Does the app domain allow `mprotect(RWX)` on a private mapping of an `app_data_file`? Does a + `/proc/self/mem` write work under the app's SELinux context and proot's own `ptrace` attach? + `/proc/self/mem` writes need `PTRACE_MODE_ATTACH` on self, which is always allowed. +- How many `libc.so.6` path syscall sites fall outside a recognisable `mov x8, #nr` window? For + example, `syscall()` takes the number in a register, which needs the existing `syscall()` hook. +- Patched pages become private copies, about 4 KiB per patched page per process. Measure RSS + across the Steam/CEF tree. +- Equivalence: run `bench/equiv.py` plus a `find`/`realpath`/nss workload with `PROOT_FP_STATS`. + Compare the trap counts from patch 0018 before and after. + +## First implementation and Thor validation (2026-10-27) + +`tools/proot/fastpath/svc-patch.h`, included by `fastpath.c`, implements an opt-in first +slice of item 1. Set `PROOT_FP_PATCH_SVC=1` in a fresh process using the replacement +`libblfastpath.so`. The ordinary fast-path configuration and tracer-key check must also +succeed. It is disabled by default, skips processes that already have multiple threads, +and declines SVE/SME hardware because the relay preserves the base ARM64 register state. + +The scanner visits executable segments of the loaded `libc.so.6` and +`ld-linux-aarch64.so.1`. It only patches `openat` and `newfstatat` sites with an +unshifted `movz x8/w8, #nr` within the preceding eight instructions. Intervening +instructions must belong to a small allow-list of argument moves/sign-extension, +immediate moves, stack loads/stores and NOPs that leave x8 unchanged. Branches, +calls, x8 writes and unknown instructions stop recognition. This includes libc's +non-adjacent O_CREAT sequences. Register-number dispatchers and other syscall +numbers remain under PRoot. No glibc version offsets, proroot code, or modified +proroot binaries are used. + +Each replacement is a four-byte `b` into a nearby relay. Relay pages are filled while +writable, then changed to RX before publishing any branch. Text writes use +`/proc/self/mem`; failure leaves the site unpatched. The relay preserves x1–x30, all +32 SIMD registers, NZCV, FPCR/FPSR and the caller's `errno`, returning a raw kernel-style +result in x0. It calls the existing path translator, using the existing allowed +trampoline for translated calls. Declined and recursive calls execute an ordinary +`svc` in this library, where PRoot still intercepts them. No SIGSYS handler or new +seccomp filter is installed. `PROOT_FP_STATS=1` adds `fastpath-svc` exit counters. + +The connected AYN Thor was running runtime r9, glibc 2.43, with a 4 KiB page size in +`u:r:untrusted_app_27:s0`. `bench/svc-probe.c`, run through the app's guest bridge, +confirmed all of the following while PRoot was attached: + +- An RX private mapping of the rootfs libc accepted `mprotect(RWX)`. +- `/proc/self/mem` accepted a changed byte in that RX mapping; readback and restoration + succeeded, and the backing file stayed unchanged. +- An anonymous RW mapping could be changed to RX. +- libc contained 415 `svc #0` sites: 327 with an adjacent number load, 76 with a + number load only within the preceding eight instructions, and 12 with neither. + The linker contained 56 sites, 45 adjacent and 11 nearby-only. Nearby counts are + scan statistics, not proof that the number remains live at the syscall. + +The initial adjacent-only scanner patched nine sites; the final scanner patched +14, including file-creation paths, across libc and the linker. It used two relay +pages in standalone validation processes. `bench/svc-check.c` exercises the +actual emitted relay and checks raw success/error results, errno, x1–x30, q0–q31, +NZCV, FPCR/FPSR, four simultaneous threads, the multi-thread patching guard, and fork. +Null arguments and unsupported flags fall back to PRoot. The instruction decoder +and exact ±128 MiB branch limits also have a host regression test. + +Five alternating pairs of `bench/svc-workload.c` runs compared the same replacement +library with patching off/on, with fast-path stats enabled in both. The workload +gets `fstatat` and `__open64_nocancel` directly from libc, bypassing the preload's +public wrappers, and makes 5,000 of each call against a prepared regular file: + +| median | wrappers only | svc patching | +|---|---:|---:| +| internal `fstatat` | 38.6 µs | 3.0 µs | +| internal open + close | 39.3 µs | 5.8 µs | +| voluntary context switches over both loops | 10,000 | 0 | + +### SD allocation and writes + +`bench/svc-storage.c` separates the operations relevant to Steam downloads. Three +alternating pairs ran on internal `/tmp` and the selected `/mnt/droiddeck-sd` library, +with Steam idle and no simultaneous local Steam download. Each fresh directory held 64 long-named files +extended to 256 KiB, followed by 1,000 libc-internal stats and a 64 MiB sequential +write followed by `fdatasync`. Both modes used the same replacement library; only +`PROOT_FP_PATCH_SVC` changed. Medians on SD were: + +| phase | wrappers only | svc patching | +|---|---:|---:| +| create/extend/close 64 files | 1.500 s | 1.486 s | +| 1,000 internal stats | 61.2 ms | 5.4 ms | +| 64 MiB write calls | 43.9 ms | 40.9 ms | +| final `fdatasync` | 2.514 s | 2.539 s | + +Internal allocation fell from 7.1 ms to 1.9 ms; internal stats fell from 39.5 ms +to 2.5 ms. Internal `fdatasync` varied from 36 to 164 ms across individual runs. +The patched SD workload recorded 1,065 internal calls handled in-process, covering +64 file creations, 1,000 stats and the streaming-file creation. Per-phase medians +are independent and should not be added as a median whole-run duration. + +The SD metadata gain does not establish faster Steam downloads. File creation +improved by only a few percent, and write/flush speed did not improve in this probe. +The remaining SD cost is much larger than the saved syscall-interception time. +The earlier [Thor SD investigation](thor-sd-storage-investigation.md) likewise +identified native exFAT metadata waits; the current benchmark does not independently +attribute those waits to a particular mount flag. A real Steam install comparison +must separate preallocation, transfer/staging and final commit before claiming an +end-to-end improvement. + +### Real Steam downloads to SD + +Four fresh installs of owned, initially uninstalled **7 Billion Humans** (792100) +used library folder 1, `/mnt/droiddeck-sd`, and were uninstalled between trials. +Each downloaded 228,935,584 bytes and staged 237,717,350 bytes in 19 files, ending +at BuildID 8016557, depots 792101/5052600457178655633 and +792104/8245917369753676198. Every trial logged zero reused/stored/delta bytes at +the start and completed with `No Error`. Existing installed games were retained. + +The repeat off/on pair used the exact same replacement library, SHA-256 +`ab605c021aee271cd49fa1cb337d7ff0c762dbac198b711e9818639c0cd4b17d`. +The first off run used the earlier adjacent-only build, with patching disabled. +Only the opt-in environment flag changed for the final pair. Times below come +from Steam's one-second-resolution `content_log.txt`; total starts at Reconfiguring +and ends at scheduler completion, excluding session startup: + +| trial | patching | preallocate | download/stage | commit to completion | total | +|---|---|---:|---:|---:|---:| +| first baseline | off | 1 s | 123 s | 1 s | 135 s | +| first patched | on | <1 s | 47 s | 1 s | 50 s | +| repeat baseline | off | <1 s | 30 s | 1 s | 33 s | +| repeat patched | on | <1 s | 29 s | 2 s | 32 s | + +Reconfiguration accounts for the remaining total time. The first baseline +overlapped another Steam client's network download; that client's telemetry must +not be mistaken for the Thor's. Later samples were filtered to local client `0`. +The repeat pair reached similar network rates, roughly 7–8 MB/s while transferring. +All four attempts logged the same unsupported large-file preallocation retry. + +**This does not demonstrate a useful SD-download speedup.** The apparent initial +135→50 second gain disappeared when baseline was repeated. The final 33 versus +32 seconds is within this small experiment's timing/network variability. This +19-file title also does not settle performance for games with thousands of loose +files. Metadata calls clearly improve, while SD creation/writeback remain the +stronger next targets. No network settings, SD mount flags or root privileges +were changed for these tests. + +The internal-call equivalence workload, including fts, realpath, reentrant NSS, +runtime dlopen, missing paths and fallback errors, matched `PROOT_FP_OFF=1` at both +the default parent-cache TTL and TTL 0. The existing `equiv.py` also matched after +normalizing random/thermal read payloads to their lengths, volatile PID values in +the `/proc` directory listing, and tolerating Android's +denied `/dev` directory read. Its root fixtures were relocated into a temporary tree. + +`equiv2.py` used temporary root/SD fixtures, normalized `/proc/` output, +and tolerated the SD filesystem's unsupported symlink creation. Its patched output +matched the wrapper-only output at both TTLs. **It did not fully match PRoot:** +binding a Unix socket in the selected SD bind returned EPERM with the existing +wrappers, whereas PRoot succeeded. This mismatch reproduced with svc patching off; +it is a separate existing fast-path socket issue. The suite is not a clean full +equivalence pass. `equiv3.py` was not run because it rewrites live NSS configuration. + +A fresh Steam session using the override and `PROOT_FP_PATCH_SVC=1` rendered its +library UI. Sonic Mania (584400) gained focus and rendered its title screen. This +is startup smoke coverage, not gameplay or a Steam startup-speed measurement. +The installed PRoot pack emitted no profiling start/counter records despite the +queued profiling setting; the profiler report had `hasCounters: false`. That is a +measurement gap, not evidence of zero remaining traps. + +Before enabling this by default: validate signal/cancellation/unwind behavior, +additional glibc builds and workloads, startup cost and Steam/CEF tree RSS; obtain +matching patch-0018 tracer measurements; and resolve the existing SD socket mismatch. +Additional syscall families, Android-blocked syscall emulation and build-id caching +remain follow-ups. In particular this implementation does not remove startup loader +traps before the preload constructor, nor patch raw calls in arbitrary guest ELFs. + +Build the tools with the same ARM64 glibc toolchain as the fast path: + +```sh +aarch64-linux-gnu-gcc -O2 -Wall -Wextra -pthread \ + -o svc-check tools/proot/bench/svc-check.c -ldl +aarch64-linux-gnu-gcc -O2 -Wall -Wextra \ + -o svc-probe tools/proot/bench/svc-probe.c -ldl +aarch64-linux-gnu-gcc -O2 -Wall -Wextra \ + -o svc-workload tools/proot/bench/svc-workload.c -ldl +aarch64-linux-gnu-gcc -O2 -Wall -Wextra \ + -o svc-storage tools/proot/bench/svc-storage.c -ldl +``` + +Run `svc-probe` inside the app's guest environment, not from `adb shell`. +`svc-check` embeds the fast path and requires its normal enabled tracer environment. +For `svc-workload`, pass a prepared directory containing `file` and a `link` symlink +to it; add any second argument to enable timing. Use the documented library override +workflow in [proot-performance](proot-performance.md) for a fresh session, adding +`PROOT_FP_PATCH_SVC=1` through `droiddeckctl env set` before starting it. +`svc-storage` takes a new, unused directory name; it creates and removes its own test +files and directory. It writes 64 MiB plus 16 MiB of file extensions per run. diff --git a/tools/proot/bench/svc-check.c b/tools/proot/bench/svc-check.c new file mode 100644 index 0000000000..fc76a68526 --- /dev/null +++ b/tools/proot/bench/svc-check.c @@ -0,0 +1,121 @@ +#include "../fastpath/fastpath.c" +#include +#include + +#if defined(__aarch64__) +struct snapshot { long x[32]; unsigned char q[32][16]; unsigned long nzcv, fpcr, fpsr; }; +extern uint32_t svc_check_site; +extern void svc_register_probe(struct snapshot *, const long *); + +__asm__( + ".text\n.p2align 4\n.type svc_register_probe,%function\nsvc_register_probe:\n" + "sub sp, sp, #1072\n" + "stp x19, x20, [sp]\nstp x21, x22, [sp, #16]\nstp x23, x24, [sp, #32]\n" + "stp x25, x26, [sp, #48]\nstp x27, x28, [sp, #64]\nstp x29, x30, [sp, #80]\n" + "stp q8, q9, [sp, #96]\nstp q10, q11, [sp, #128]\n" + "stp q12, q13, [sp, #160]\nstp q14, q15, [sp, #192]\n" + "stp x0, x1, [sp, #224]\n" + ".irp n,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n" + "mov x\\n, #\\n\n.endr\n" + ".irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31\n" + "movi v\\n\\().16b, #0x5a\n.endr\n" + "mov x0, #0xa0000000\nmsr nzcv, x0\n" + "ldr x0, [sp, #232]\nldr x8, [x0, #48]\n" + "ldp x4, x5, [x0, #32]\nldp x2, x3, [x0, #16]\nldp x0, x1, [x0]\n" + ".global svc_check_site\nsvc_check_site:\nsvc #0\n" + ".irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n" + "str x\\n, [sp, #(256 + \\n * 8)]\n.endr\n" + ".irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31\n" + "str q\\n, [sp, #(512 + \\n * 16)]\n.endr\n" + "mrs x0, nzcv\nstr x0, [sp, #1024]\nmrs x0, fpcr\nstr x0, [sp, #1032]\n" + "mrs x0, fpsr\nstr x0, [sp, #1040]\n" + "ldr x0, [sp, #224]\nadd x1, sp, #256\nmov x2, #792\nbl memcpy\n" + "ldp q8, q9, [sp, #96]\nldp q10, q11, [sp, #128]\n" + "ldp q12, q13, [sp, #160]\nldp q14, q15, [sp, #192]\n" + "ldp x19, x20, [sp]\nldp x21, x22, [sp, #16]\nldp x23, x24, [sp, #32]\n" + "ldp x25, x26, [sp, #48]\nldp x27, x28, [sp, #64]\nldp x29, x30, [sp, #80]\n" + "add sp, sp, #1072\nret\n.size svc_register_probe, .-svc_register_probe\n" +); + +static void check(const long *args, long want) { + struct snapshot out = {0}; + unsigned long fpcr, fpsr; + __asm__ volatile("mrs %0, fpcr\nmrs %1, fpsr" : "=r"(fpcr), "=r"(fpsr)); + errno = 123; + svc_register_probe(&out, args); + assert(errno == 123); + assert(out.x[0] == want); + for (int i = 1; i <= 30; i++) { + long expected = i <= 5 ? args[i] : i == 8 ? args[6] : i; + if (out.x[i] != expected) { fprintf(stderr, "x%d: %lx != %lx\n", i, out.x[i], expected); abort(); } + } + for (int i = 0; i < 32; i++) + for (int j = 0; j < 16; j++) assert(out.q[i][j] == 0x5a); + assert(out.nzcv == 0xa0000000UL); + assert(out.fpcr == fpcr && out.fpsr == fpsr); +} + +static void *thread_check(void *unused) { + (void)unused; + struct stat st; + const long args[] = {AT_FDCWD, (long)"/usr/lib/libc.so.6", (long)&st, 0, 0, 0, SYS_newfstatat}; + for (int i = 0; i < 100; i++) check(args, 0); + return NULL; +} + +static pthread_barrier_t barrier; +static void *thread_probe(void *unused) { + pthread_barrier_wait(&barrier); + thread_check(unused); + pthread_barrier_wait(&barrier); + return NULL; +} + +int main(void) { + assert(fp_ready()); + assert(svc_single_thread()); + size_t page = sysconf(_SC_PAGESIZE); + uint32_t *relay = svc_relay_page((uintptr_t)&svc_check_site, page), branch; + assert(relay && svc_emit(relay, &svc_check_site, &branch)); + __builtin___clear_cache((char *)relay, (char *)relay + 32); + assert(mprotect(relay, page, PROT_READ | PROT_EXEC) == 0); + int mem = open("/proc/self/mem", O_RDWR | O_CLOEXEC); + assert(mem >= 0 && pwrite(mem, &branch, 4, (off_t)(uintptr_t)&svc_check_site) == 4); + close(mem); + __builtin___clear_cache((char *)&svc_check_site, (char *)(&svc_check_site + 1)); + long args[] = {0, 1, 2, 3, 4, 5, SYS_getpid}; + check(args, getpid()); + struct stat st; + args[0] = AT_FDCWD; args[1] = (long)"/usr/lib/libc.so.6"; args[2] = (long)&st; + args[3] = 0; args[6] = SYS_newfstatat; + check(args, 0); + assert(st.st_size > 0); + args[1] = (long)"/usr/lib/proot-svc-check-does-not-exist"; + check(args, -ENOENT); + args[1] = 0; + check(args, -EFAULT); + args[1] = (long)"/usr/lib/libc.so.6"; args[2] = 0; + check(args, -EFAULT); + args[1] = (long)"/proc/self/exe"; args[2] = (long)&st; + check(args, 0); + args[3] = 0x40000000; + check(args, -EINVAL); + pthread_t threads[4]; + assert(pthread_barrier_init(&barrier, NULL, 5) == 0); + for (int i = 0; i < 4; i++) assert(pthread_create(&threads[i], NULL, thread_probe, NULL) == 0); + pthread_barrier_wait(&barrier); + assert(!svc_single_thread()); + pthread_barrier_wait(&barrier); + for (int i = 0; i < 4; i++) assert(pthread_join(threads[i], NULL) == 0); + assert(pthread_barrier_destroy(&barrier) == 0); + pid_t pid = fork(); + assert(pid >= 0); + if (!pid) { thread_check(NULL); _exit(0); } + int status; + assert(waitpid(pid, &status, 0) == pid && WIFEXITED(status) && WEXITSTATUS(status) == 0); + puts("PASS: raw results, errno, x1-x30, q0-q31, NZCV, FPCR/FPSR, 4 threads, fork"); + return 0; +} +#else +int main(void) { return 77; } +#endif diff --git a/tools/proot/bench/svc-probe.c b/tools/proot/bench/svc-probe.c new file mode 100644 index 0000000000..43e123b42e --- /dev/null +++ b/tools/proot/bench/svc-probe.c @@ -0,0 +1,93 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include + +static int mov_sysnum(uint32_t insn) { + if ((insn & 0xffe0001fU) == 0xd2800008U || + (insn & 0xffe0001fU) == 0x52800008U) + return (insn >> 5) & 0xffff; + return -1; +} + +static int scan(struct dl_phdr_info *info, size_t size, void *unused) { + (void)size; + (void)unused; + const char *name = strrchr(info->dlpi_name, '/'); + name = name ? name + 1 : info->dlpi_name; + if (strcmp(name, "libc.so.6") && strcmp(name, "ld-linux-aarch64.so.1")) return 0; + unsigned total = 0, adjacent = 0, nearby = 0, counts[512] = {0}; + for (int i = 0; i < info->dlpi_phnum; i++) { + const ElfW(Phdr) *ph = &info->dlpi_phdr[i]; + if (ph->p_type != PT_LOAD || !(ph->p_flags & PF_X)) continue; + const uint32_t *text = (const uint32_t *)(info->dlpi_addr + ph->p_vaddr); + for (size_t j = 0; j < ph->p_memsz / 4; j++) { + if (text[j] != 0xd4000001U) continue; + total++; + int nr = j ? mov_sysnum(text[j - 1]) : -1; + if (nr >= 0) { + adjacent++; + if (nr < 512) counts[nr]++; + } else { + for (size_t k = 2; k <= 8 && k <= j; k++) + if (mov_sysnum(text[j - k]) >= 0) { nearby++; break; } + } + } + } + printf("{\"object\":\"%s\",\"svc\":%u,\"adjacent_mov_x8\":%u,\"nearby_only\":%u,\"adjacent_syscalls\":{", + name, total, adjacent, nearby); + int comma = 0; + for (unsigned nr = 0; nr < 512; nr++) + if (counts[nr]) { printf("%s\"%u\":%u", comma ? "," : "", nr, counts[nr]); comma = 1; } + puts("}}"); + return 0; +} + +int main(void) { + long page = sysconf(_SC_PAGESIZE); + int fd = open("/usr/lib/libc.so.6", O_RDONLY | O_CLOEXEC); + if (fd < 0) { perror("libc"); return 1; } + unsigned char *p = mmap(NULL, page, PROT_READ | PROT_EXEC, MAP_PRIVATE, fd, 0); + if (p == MAP_FAILED) { perror("file mmap RX"); close(fd); return 1; } + unsigned char before = p[0], changed = before ^ 1; + errno = 0; + int rwx = mprotect(p, page, PROT_READ | PROT_WRITE | PROT_EXEC), rwx_errno = errno; + if (rwx == 0) { + p[0] = changed; + p[0] = before; + if (mprotect(p, page, PROT_READ | PROT_EXEC)) { perror("restore RX"); return 1; } + } + int mem = open("/proc/self/mem", O_RDWR | O_CLOEXEC), mem_errno = mem < 0 ? errno : 0; + ssize_t written = -1; + int write_errno = 0, verified = 0, restored = 0; + if (mem >= 0) { + errno = 0; + written = pwrite(mem, &changed, 1, (off_t)(uintptr_t)p); + write_errno = errno; + verified = written == 1 && *(volatile unsigned char *)p == changed; + restored = pwrite(mem, &before, 1, (off_t)(uintptr_t)p) == 1 && + *(volatile unsigned char *)p == before; + close(mem); + } + unsigned char disk = 0; + int file_unchanged = pread(fd, &disk, 1, 0) == 1 && disk == before; + munmap(p, page); + close(fd); + void *anon = mmap(NULL, page, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + errno = 0; + int execmem = anon == MAP_FAILED ? -1 : mprotect(anon, page, PROT_READ | PROT_EXEC); + int execmem_errno = errno; + if (anon != MAP_FAILED) munmap(anon, page); + printf("{\"page_size\":%ld,\"rwx\":%d,\"rwx_errno\":%d,\"mem_open_errno\":%d," + "\"mem_written\":%ld,\"mem_write_errno\":%d,\"mem_verified\":%d,\"mem_restored\":%d," + "\"file_unchanged\":%d,\"execmem\":%d,\"execmem_errno\":%d}\n", + page, rwx, rwx_errno, mem_errno, (long)written, write_errno, verified, restored, + file_unchanged, execmem, execmem_errno); + dl_iterate_phdr(scan, NULL); + return 0; +} diff --git a/tools/proot/bench/svc-storage.c b/tools/proot/bench/svc-storage.c new file mode 100644 index 0000000000..2b9dd40ca6 --- /dev/null +++ b/tools/proot/bench/svc-storage.c @@ -0,0 +1,74 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static double now(void) { + struct timespec t; + assert(clock_gettime(CLOCK_MONOTONIC, &t) == 0); + return t.tv_sec + t.tv_nsec / 1e9; +} + +int main(int argc, char **argv) { + assert(argc == 2 && mkdir(argv[1], 0700) == 0); + void *libc = dlopen("libc.so.6", RTLD_NOW); + assert(libc); + int (*internal_open)(const char *, int, ...) = dlsym(libc, "__open64_nocancel"); + int (*internal_stat)(int, const char *, struct stat *, int) = dlsym(libc, "fstatat"); + assert(internal_open && internal_stat); + struct rusage before, after; + assert(getrusage(RUSAGE_SELF, &before) == 0); + char paths[64][4096], stream[4096]; + double creation = 0, extension = 0, closing = 0, total = now(); + for (int i = 0; i < 64; i++) { + snprintf(paths[i], sizeof paths[i], "%s/asset-with-a-longish-steam-depot-filename-%03d.dat", argv[1], i); + double start = now(); + int fd = internal_open(paths[i], O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC, 0600); + assert(fd >= 0); + creation += now() - start; + start = now(); + assert(ftruncate(fd, 262144) == 0); + extension += now() - start; + start = now(); + assert(close(fd) == 0); + closing += now() - start; + } + double allocation = now() - total; + struct stat st; + double start = now(); + for (int i = 0; i < 1000; i++) assert(internal_stat(AT_FDCWD, paths[i % 64], &st, 0) == 0); + double metadata = now() - start; + snprintf(stream, sizeof stream, "%s/stream.bin", argv[1]); + start = now(); + int fd = internal_open(stream, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC, 0600); + assert(fd >= 0); + double stream_create = now() - start; + static unsigned char block[1024 * 1024]; + memset(block, 0x5a, sizeof block); + start = now(); + for (int i = 0; i < 64; i++) assert(write(fd, block, sizeof block) == sizeof block); + double writing = now() - start; + start = now(); + assert(fdatasync(fd) == 0); + double syncing = now() - start; + assert(close(fd) == 0); + assert(getrusage(RUSAGE_SELF, &after) == 0); + printf("{\"files\":64,\"bytes_per_file\":262144,\"create_s\":%.6f,\"extend_s\":%.6f," + "\"close_s\":%.6f,\"allocation_s\":%.6f,\"stat1000_s\":%.6f," + "\"stream_bytes\":67108864,\"stream_create_s\":%.6f,\"write_s\":%.6f," + "\"fdatasync_s\":%.6f,\"voluntary_switches\":%ld}\n", + creation, extension, closing, allocation, metadata, stream_create, writing, + syncing, after.ru_nvcsw - before.ru_nvcsw); + fflush(stdout); + for (int i = 0; i < 64; i++) assert(unlink(paths[i]) == 0); + assert(unlink(stream) == 0 && rmdir(argv[1]) == 0); + assert(dlclose(libc) == 0); + return 0; +} diff --git a/tools/proot/bench/svc-workload.c b/tools/proot/bench/svc-workload.c new file mode 100644 index 0000000000..350a7081ca --- /dev/null +++ b/tools/proot/bench/svc-workload.c @@ -0,0 +1,91 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static double now(void) { + struct timespec t; + assert(clock_gettime(CLOCK_MONOTONIC, &t) == 0); + return t.tv_sec * 1e9 + t.tv_nsec; +} + +int main(int argc, char **argv) { + assert(argc >= 2); + void *libc = dlopen("libc.so.6", RTLD_NOW); + assert(libc); + int (*internal_stat)(int, const char *, struct stat *, int) = dlsym(libc, "fstatat"); + int (*internal_open)(const char *, int, ...) = dlsym(libc, "__open64_nocancel"); + assert(internal_stat && internal_open); + char path[4096], link[4096], missing[4096]; + snprintf(path, sizeof path, "%s/file", argv[1]); + snprintf(link, sizeof link, "%s/link", argv[1]); + snprintf(missing, sizeof missing, "%s/missing", argv[1]); + const char *paths[] = {path, link, missing, "/etc/os-release", "/lib/libc.so.6", "/proc/self/exe", "/dev/null", "", NULL}; + struct stat st; + for (size_t i = 0; i < sizeof paths / sizeof *paths; i++) { + errno = 0; + int r = internal_stat(AT_FDCWD, paths[i], &st, 0); + printf("stat %zu: %d %d", i, r, r ? errno : 0); + if (!r) printf(" %o %ld", st.st_mode & S_IFMT, S_ISREG(st.st_mode) && i != 5 ? st.st_size : 0L); + putchar('\n'); + errno = 0; + int fd = internal_open(paths[i], O_RDONLY | O_CLOEXEC); + int err = fd < 0 ? errno : 0; + printf("open %zu: %d %d\n", i, fd < 0 ? -1 : 0, err); + if (fd >= 0) close(fd); + } + errno = 0; + int r = internal_stat(AT_FDCWD, path, &st, 0x40000000); + printf("flags: %d %d\n", r, r ? errno : 0); + char *tree[] = {argv[1], NULL}; + FTS *fts = fts_open(tree, FTS_PHYSICAL | FTS_NOCHDIR, NULL); + assert(fts); + unsigned files = 0, dirs = 0, links = 0; + FTSENT *ent; + while ((ent = fts_read(fts))) { + if (ent->fts_info == FTS_F) files++; + if (ent->fts_info == FTS_D) dirs++; + if (ent->fts_info == FTS_SL) links++; + } + assert(fts_close(fts) == 0); + printf("fts: %u files %u dirs %u links\n", files, dirs, links); + char resolved[4096]; + assert(realpath(link, resolved) && !strcmp(resolved, path)); + struct passwd pw, *found; + char buf[16384]; + r = getpwnam_r("root", &pw, buf, sizeof buf, &found); + printf("nss: %d %d\n", r, found != NULL); + void *libm = dlopen("libm.so.6", RTLD_NOW | RTLD_LOCAL); + assert(libm && dlsym(libm, "cos")); + assert(dlclose(libm) == 0); + puts("realpath/dlopen: PASS"); + if (argc > 2) { + const int loops = 5000; + struct rusage before, after; + assert(getrusage(RUSAGE_SELF, &before) == 0); + double start = now(); + for (int i = 0; i < loops; i++) assert(internal_stat(AT_FDCWD, path, &st, 0) == 0); + double stat_ns = (now() - start) / loops; + start = now(); + for (int i = 0; i < loops; i++) { + int fd = internal_open(path, O_RDONLY | O_CLOEXEC); + assert(fd >= 0 && close(fd) == 0); + } + double open_ns = (now() - start) / loops; + assert(getrusage(RUSAGE_SELF, &after) == 0); + printf("bench: stat %.0f ns, open+close %.0f ns, voluntary switches %ld\n", + stat_ns, open_ns, after.ru_nvcsw - before.ru_nvcsw); + } + assert(dlclose(libc) == 0); + return 0; +} diff --git a/tools/proot/fastpath/fastpath.c b/tools/proot/fastpath/fastpath.c index bd3fc0d210..e036808657 100644 --- a/tools/proot/fastpath/fastpath.c +++ b/tools/proot/fastpath/fastpath.c @@ -1828,3 +1828,5 @@ struct passwd *getpwuid(uid_t uid) { missed(__func__, AT_FDCWD, "(passwd)"); return real_getpwuid(uid); } + +#include "svc-patch.h" diff --git a/tools/proot/fastpath/svc-patch.h b/tools/proot/fastpath/svc-patch.h new file mode 100644 index 0000000000..f247f18d22 --- /dev/null +++ b/tools/proot/fastpath/svc-patch.h @@ -0,0 +1,250 @@ +static inline int svc_number(uint32_t insn) { + if ((insn & 0xffe0001fU) == 0xd2800008U || + (insn & 0xffe0001fU) == 0x52800008U) + return (insn >> 5) & 0xffff; + return -1; +} + +static inline int svc_argument_instruction(uint32_t insn) { + if (insn == 0xd503201fU) return 1; + if ((insn & 0xffc003e0U) == 0x290003e0U || + (insn & 0xffc003e0U) == 0xa90003e0U) return 1; + if ((insn & 31) == 8) return 0; + return (insn & 0xffe0ffe0U) == 0xaa0003e0U || + (insn & 0xffe0ffe0U) == 0x2a0003e0U || + (insn & 0xfffffc00U) == 0x93407c00U || + (insn & 0x7f800000U) == 0x52800000U || + (insn & 0x7f800000U) == 0x12800000U || + (insn & 0xffc003e0U) == 0xb94003e0U || + (insn & 0xffc003e0U) == 0xf94003e0U || + (insn & 0xffc003e0U) == 0xb98003e0U; +} + +static inline int svc_site_number(const uint32_t *text, size_t at) { + for (size_t back = 1; back <= 8 && back <= at; back++) { + uint32_t insn = text[at - back]; + int nr = svc_number(insn); + if (nr >= 0) return nr; + if (!svc_argument_instruction(insn)) break; + } + return -1; +} + +static inline int svc_branch(uintptr_t from, uintptr_t to, uint32_t *out) { + int64_t delta = (int64_t)to - (int64_t)from; + if ((delta & 3) || delta < -0x8000000LL || delta >= 0x8000000LL) return 0; + *out = 0x14000000U | ((uint32_t)(delta / 4) & 0x03ffffffU); + return 1; +} + +#if defined(__aarch64__) && !defined(FP_SVC_ENCODINGS_ONLY) +#include +#include + +static __thread int svc_busy __attribute__((tls_model("initial-exec"))); +static unsigned svc_sites, svc_pages; +static unsigned long svc_fast, svc_slow; + +static long svc_original(const long *r) { + register long x0 __asm__("x0") = r[0]; + register long x1 __asm__("x1") = r[1]; + register long x2 __asm__("x2") = r[2]; + register long x3 __asm__("x3") = r[3]; + register long x4 __asm__("x4") = r[4]; + register long x5 __asm__("x5") = r[5]; + register long x8 __asm__("x8") = r[8]; + __asm__ volatile("svc #0" : "+r"(x0) : "r"(x1), "r"(x2), "r"(x3), "r"(x4), "r"(x5), "r"(x8) : "memory", "cc"); + return x0; +} + +__attribute__((used, visibility("hidden"))) +long fp_svc_dispatch(const long *r) { + int saved_errno = errno; + long result = FP_SLOW; + if (!svc_busy) { + svc_busy = 1; + if (r[8] == SYS_openat) + result = fp_open((int)r[0], (const char *)r[1], (int)r[2], (mode_t)r[3]); + else if (r[8] == SYS_newfstatat && r[2]) + result = fp_stat((int)r[0], (const char *)r[1], (int)r[3], (struct stat *)r[2]); + if (result != FP_SLOW) __atomic_fetch_add(&svc_fast, 1, __ATOMIC_RELAXED); + svc_busy = 0; + } + if (result == FP_SLOW) { + __atomic_fetch_add(&svc_slow, 1, __ATOMIC_RELAXED); + result = svc_original(r); + } + errno = saved_errno; + return result; +} + +void fp_svc_entry(void) __attribute__((visibility("hidden"))); +/* A replaced svc may have live values in every register, including NZCV and SIMD. */ +__asm__( + ".text\n.p2align 4\n.hidden fp_svc_entry\n.type fp_svc_entry,%function\nfp_svc_entry:\n" + "bti j\nsub sp, sp, #800\n" + "stp x0, x1, [sp, #0]\nstp x2, x3, [sp, #16]\n" + "stp x4, x5, [sp, #32]\nstp x6, x7, [sp, #48]\n" + "stp x8, x9, [sp, #64]\nstp x10, x11, [sp, #80]\n" + "stp x12, x13, [sp, #96]\nstp x14, x15, [sp, #112]\n" + "stp x18, x19, [sp, #128]\nstp x20, x21, [sp, #144]\n" + "stp x22, x23, [sp, #160]\nstp x24, x25, [sp, #176]\n" + "stp x26, x27, [sp, #192]\nstp x28, x29, [sp, #208]\n" + "stp x30, x17, [sp, #224]\nmrs x0, nzcv\nstr x0, [sp, #240]\n" + "mrs x0, fpcr\nstr x0, [sp, #768]\nmrs x0, fpsr\nstr x0, [sp, #776]\n" + "stp q0, q1, [sp, #256]\nstp q2, q3, [sp, #288]\n" + "stp q4, q5, [sp, #320]\nstp q6, q7, [sp, #352]\n" + "stp q8, q9, [sp, #384]\nstp q10, q11, [sp, #416]\n" + "stp q12, q13, [sp, #448]\nstp q14, q15, [sp, #480]\n" + "stp q16, q17, [sp, #512]\nstp q18, q19, [sp, #544]\n" + "stp q20, q21, [sp, #576]\nstp q22, q23, [sp, #608]\n" + "stp q24, q25, [sp, #640]\nstp q26, q27, [sp, #672]\n" + "stp q28, q29, [sp, #704]\nstp q30, q31, [sp, #736]\n" + "mov x0, sp\nbl fp_svc_dispatch\nstr x0, [sp]\n" + "ldp q0, q1, [sp, #256]\nldp q2, q3, [sp, #288]\n" + "ldp q4, q5, [sp, #320]\nldp q6, q7, [sp, #352]\n" + "ldp q8, q9, [sp, #384]\nldp q10, q11, [sp, #416]\n" + "ldp q12, q13, [sp, #448]\nldp q14, q15, [sp, #480]\n" + "ldp q16, q17, [sp, #512]\nldp q18, q19, [sp, #544]\n" + "ldp q20, q21, [sp, #576]\nldp q22, q23, [sp, #608]\n" + "ldp q24, q25, [sp, #640]\nldp q26, q27, [sp, #672]\n" + "ldp q28, q29, [sp, #704]\nldp q30, q31, [sp, #736]\n" + "ldr x0, [sp, #768]\nmsr fpcr, x0\nldr x0, [sp, #776]\nmsr fpsr, x0\n" + "ldr x0, [sp, #240]\nmsr nzcv, x0\n" + "ldp x0, x1, [sp, #0]\nldp x2, x3, [sp, #16]\n" + "ldp x4, x5, [sp, #32]\nldp x6, x7, [sp, #48]\n" + "ldp x8, x9, [sp, #64]\nldp x10, x11, [sp, #80]\n" + "ldp x12, x13, [sp, #96]\nldp x14, x15, [sp, #112]\n" + "ldp x18, x19, [sp, #128]\nldp x20, x21, [sp, #144]\n" + "ldp x22, x23, [sp, #160]\nldp x24, x25, [sp, #176]\n" + "ldp x26, x27, [sp, #192]\nldp x28, x29, [sp, #208]\n" + "ldr x30, [sp, #224]\nldr x16, [sp, #232]\n" + "add sp, sp, #800\nbr x16\n.size fp_svc_entry, .-fp_svc_entry\n" +); + +static int svc_emit(uint32_t *code, uint32_t *site, uint32_t *branch) { + uint32_t back; + if (!svc_branch((uintptr_t)site, (uintptr_t)code, branch) || + !svc_branch((uintptr_t)(code + 5), (uintptr_t)(site + 1), &back)) return 0; + code[0] = 0xa9bf47f0U; /* stp x16, x17, [sp, #-16]! */ + code[1] = 0x580000b0U; /* ldr x16, literal at +24 */ + code[2] = 0x10000051U; /* adr x17, +8 */ + code[3] = 0xd61f0200U; /* br x16 */ + code[4] = 0xa8c147f0U; /* ldp x16, x17, [sp], #16 */ + code[5] = back; + uintptr_t entry = (uintptr_t)fp_svc_entry; + memcpy(code + 6, &entry, sizeof entry); + return 1; +} + +struct svc_site { uint32_t *site; uint32_t branch; }; +static struct svc_site svc_plan[256]; +static size_t svc_planned; + +static int svc_object(struct dl_phdr_info *info, size_t size, void *unused) { + (void)size; + (void)unused; + const char *name = strrchr(info->dlpi_name, '/'); + name = name ? name + 1 : info->dlpi_name; + if (strcmp(name, "libc.so.6") && strcmp(name, "ld-linux-aarch64.so.1")) return 0; + for (int i = 0; i < info->dlpi_phnum; i++) { + const ElfW(Phdr) *ph = &info->dlpi_phdr[i]; + if (ph->p_type != PT_LOAD || !(ph->p_flags & PF_X)) continue; + uint32_t *text = (uint32_t *)(info->dlpi_addr + ph->p_vaddr); + for (size_t j = 1; j < ph->p_memsz / 4 && svc_planned < 256; j++) { + if (text[j] != 0xd4000001U) continue; + int nr = svc_site_number(text, j); + if (nr != SYS_openat && nr != SYS_newfstatat) continue; + svc_plan[svc_planned++].site = &text[j]; + } + } + return 0; +} + +static uint32_t *svc_relay_page(uintptr_t site, size_t page) { + uintptr_t base = site & ~(uintptr_t)(page - 1); + for (uintptr_t delta = page; delta < 0x7ff0000UL; delta += 0x10000UL) { + for (int direction = -1; direction <= 1; direction += 2) { + uintptr_t at = direction < 0 ? base - delta : base + delta; + if (at < page) continue; + void *p = mmap((void *)at, page, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0); + if (p == (void *)at) return p; + if (p != MAP_FAILED) munmap(p, page); + } + } + return NULL; +} + +static int svc_single_thread(void) { + struct entry { uint64_t ino; int64_t off; unsigned short len; unsigned char type; char name[]; }; + char buf[4096] __attribute__((aligned(8))); + long fd = sc(SYS_openat, AT_FDCWD, (long)"/proc/self/task", O_RDONLY | O_DIRECTORY | O_CLOEXEC, 0, 0); + if (fd < 0) return 0; + int count = 0; + long n; + while ((n = sc(SYS_getdents64, fd, (long)buf, sizeof buf, 0, 0)) > 0) { + for (long at = 0; at < n;) { + const struct entry *e = (const struct entry *)(buf + at); + if (n - at < (long)offsetof(struct entry, name) + 1 || + e->len < offsetof(struct entry, name) + 1 || e->len > n - at) { count = -1; goto done; } + if (e->name[0] >= '0' && e->name[0] <= '9' && ++count > 1) goto done; + at += e->len; + } + } + if (n < 0) count = -1; +done: + sc(SYS_close, fd, 0, 0, 0, 0); + return count == 1; +} + +static void svc_stats(void) { + char line[160]; + int n = snprintf(line, sizeof line, "fastpath-svc[%d]: %u sites, %u relay pages, %lu fast, %lu to proot\n", + getpid(), svc_sites, svc_pages, svc_fast, svc_slow); + if (write(2, line, n) < 0) {} +} + +__attribute__((constructor)) static void svc_patch_init(void) { + const char *enabled = getenv("PROOT_FP_PATCH_SVC"); + if (!enabled || strcmp(enabled, "1") || !fp_ready()) return; + if (getauxval(AT_HWCAP) & HWCAP_SVE) return; +#ifdef HWCAP2_SME + if (getauxval(AT_HWCAP2) & HWCAP2_SME) return; +#endif + if (!svc_single_thread()) return; + size_t page = (size_t)sysconf(_SC_PAGESIZE); + if (page < 4096 || (page & (page - 1))) return; + int mem = (int)sc(SYS_openat, AT_FDCWD, (long)"/proc/self/mem", O_RDWR | O_CLOEXEC, 0, 0); + if (mem < 0) return; + dl_iterate_phdr(svc_object, NULL); + for (size_t first = 0; first < svc_planned;) { + uint32_t *relay = svc_relay_page((uintptr_t)svc_plan[first].site, page); + if (!relay) { first++; continue; } + size_t end = first; + for (; end < svc_planned && (end - first + 1) * 32 <= page; end++) { + uint32_t *code = relay + (end - first) * 8; + if (!svc_emit(code, svc_plan[end].site, &svc_plan[end].branch)) break; + } + __builtin___clear_cache((char *)relay, (char *)relay + page); + if (end == first || mprotect(relay, page, PROT_READ | PROT_EXEC)) { + munmap(relay, page); + first++; + continue; + } + unsigned patched = 0; + for (size_t i = first; i < end; i++) { + struct svc_site *s = &svc_plan[i]; + long written = sc(SYS_pwrite64, mem, (long)&s->branch, sizeof s->branch, (long)s->site, 0); + if (written != sizeof s->branch) continue; + __builtin___clear_cache((char *)s->site, (char *)(s->site + 1)); + patched++; + } + if (patched) { svc_sites += patched; svc_pages++; } + else munmap(relay, page); + first = end; + } + sc(SYS_close, mem, 0, 0, 0, 0); + if (getenv("PROOT_FP_STATS")) atexit(svc_stats); +} +#endif diff --git a/tools/tests/test_fastpath_svc.py b/tools/tests/test_fastpath_svc.py new file mode 100644 index 0000000000..1f76a34656 --- /dev/null +++ b/tools/tests/test_fastpath_svc.py @@ -0,0 +1,62 @@ +from pathlib import Path +import shutil +import subprocess +import tempfile +import unittest + + +HEADER = Path(__file__).resolve().parents[1] / "proot/fastpath/svc-patch.h" + + +class FastPathSvcEncodingTest(unittest.TestCase): + def test_instruction_decoding_and_branch_limits(self): + if shutil.which("cc") is None: + self.skipTest("cc is not available") + source = r''' +#include +#include +#include +#define FP_SVC_ENCODINGS_ONLY +#include "HEADER" +int main(void) { + assert(svc_number(0xd2800708) == 56); + assert(svc_number(0x528009e8) == 79); + assert(svc_number(0xd2800709) == -1); + assert(svc_number(0xd2a00708) == -1); + assert(svc_number(0xf2800708) == -1); + assert(svc_number(0xaa0003e8) == -1); + const uint32_t create[] = {0xd2800708, 0x29047fe0, 0x92800c60, 0xd4000001}; + const uint32_t create_at[] = {0xd2800708, 0x29047fe3, 0xb9404be3, 0xd4000001}; + const uint32_t xstat[] = {0xd28009e8, 0xaa0203e1, 0xaa0303e2, 0x93407c83, 0xd4000001}; + const uint32_t overwritten[] = {0xd2800708, 0xaa0003e8, 0xd4000001}; + const uint32_t loaded[] = {0xd2800708, 0xf94003e8, 0xd4000001}; + const uint32_t branch[] = {0xd2800708, 0x14000001, 0xd4000001}; + const uint32_t call[] = {0xd2800708, 0x94000001, 0xd4000001}; + const uint32_t shift[] = {0xd2800708, 0xd2a00708, 0xd4000001}; + assert(svc_site_number(create, 3) == 56); + assert(svc_site_number(create_at, 3) == 56); + assert(svc_site_number(xstat, 4) == 79); + assert(svc_site_number(overwritten, 2) == -1); + assert(svc_site_number(loaded, 2) == -1); + assert(svc_site_number(branch, 2) == -1); + assert(svc_site_number(call, 2) == -1); + assert(svc_site_number(shift, 2) == -1); + uint32_t out; + uintptr_t pc = 0x10000000; + assert(svc_branch(pc, pc + 4, &out) && out == 0x14000001); + assert(svc_branch(pc, pc - 4, &out) && out == 0x17ffffff); + assert(svc_branch(pc, pc + 0x7fffffc, &out) && out == 0x15ffffff); + assert(svc_branch(pc, pc - 0x8000000, &out) && out == 0x16000000); + assert(!svc_branch(pc, pc + 0x8000000, &out)); + assert(!svc_branch(pc, pc - 0x8000004, &out)); + assert(!svc_branch(pc, pc + 2, &out)); + return 0; +} +''' + with tempfile.TemporaryDirectory() as directory: + work = Path(directory) + program = work / "check.c" + program.write_text(source.replace("HEADER", HEADER.as_posix())) + output = work / "check" + subprocess.run(["cc", "-O2", "-Wall", "-Wextra", "-Werror", str(program), "-o", str(output)], check=True) + subprocess.run([str(output)], check=True, timeout=10)