perf(proot): add opt-in glibc syscall patching

This commit is contained in:
Kurt Himebauch committed 2026-10-27 14:06:52 -04:00
1 parent ef9866e423
commit 4b230cbd0f
8 files changed
+1004

No files matched your search

+311
View File
@@ -0,0 +1,311 @@
# proroot: what it does and what DroidDeck can take from it
Static analysis of [coderredlab/proroot](https://github.com/coderredlab/proroot) v1.2.8
(`c01039ab`, 2026-06-17). The repository holds only five arm64 binaries and a README. The licence is
proprietary, and redistributing modified binaries is not allowed, so only the ideas below are
usable, not the code. The binaries were read with the NDK's `llvm-readelf`, `llvm-nm`,
`llvm-objdump` and `llvm-strings`. `libproroot-stub-loader.so` and `libproroot-bridge.so` still carry
`.symtab`, and `libproroot-runtime.so` exports about 260 names, so the function names below are
proroot's own.
## Architecture
proroot does not use ptrace. Everything runs inside the guest process.
| file | kind | role |
|---|---|---|
| `libproroot.so` | bionic PIE executable | Launcher. Parses `-r/-b/-w/-0/--link2symlink`, writes a config file (`.proroot-config-<pid>`, passed down as `PROROOT_CFG_FD`), and execs the guest through `libproroot-linker.so` or the stub loader. |
| `libproroot-linker.so` | freestanding PIE, run as an executable | A from-scratch glibc dynamic linker ("cleanroom linker") that replaces `ld-linux-aarch64.so.1`. It maps the program and its `DT_NEEDED` libraries, relocates them, sets up static TLS, and stubs `_dl_*` and `__tunable_get_val`. It finds `_rtld_global_ro` through a table of offsets per glibc version ("no offset table for glibc X, patches skipped"). It also installs a SIGSYS handler with an allow-list. |
| `libproroot-stub-loader.so` | freestanding static PIE | Userland `exec`. It maps the target and its `PT_INTERP` itself and builds the stack and auxv. It then **rewrites the `svc #0` sites** in the loaded images (`proroot_svc_scan`, `proroot_svc_patch_apply`, `select_trampoline_for_site`) to branch into per-syscall trampolines (`proroot_tramp_openat`, `_statx`, `_faccessat2`, `_clone3`, `_rt_sigaction`, `_shmget`, ...). Sites it cannot patch fall back to an in-process seccomp `RET_TRAP` filter with a SIGSYS handler ("trampoline fallback, activating trap-mode fallback"). |
| `libproroot-runtime.so` | glibc `LD_PRELOAD` | Hooks about 200 libc entry points: path translation, fake root (`fake_id0_*`), link2symlink, exec routing, SysV shm, netlink, `/proc` and `/sys` views, `brk`, a fake DRM/KMS and GBM device plus libseat for compositors, X11 and Unix-socket fixups. After load it **patches the svc sites in glibc itself** (`patch_libc_syscalls`). |
| `libproroot-bridge.so` | static `ET_EXEC` at `0x7000000000` | The "trampoline" for static guest binaries. It installs a path-trap seccomp filter plus a SIGSYS handler, maps the target, and serves a virtual `/proc` (`status`, `stat`, `cmdline`, `environ`, `mounts`, `mountinfo`, `fdinfo`, `auxv`). |
The executable files all live in `nativeLibraryDir`, which is the only place an app may exec from.
That is why the dynamic linker and the static loader each have their own userland loader. proot
handles the same constraint with `libproot-loader.so`.
## How the in-process syscall interception works
1. **Patching svc sites in glibc** (`patch_phdr_callback`, called through `dl_iterate_phdr`):
- It scans only an allow-list of objects: `libc.so.6`/`libc-`, `ld-linux`, `libpthread`,
`librt`, `libdl`. Main executables get patched early only for listed programs (apt, dpkg,
curl, openssl, node, chromium/electron/playwright, passwd/useradd/...). Other programs are
patched later (`log_deferred_libc_patch`, `PROROOT_FORCE_EARLY_PATCH`).
- In each executable `PT_LOAD` it finds `svc #0` (`0xd4000001`). It checks the instructions
before it for a `mov x8, #nr` it recognises. One fixed glibc `openat` call sequence gets
special treatment: its `bl` is redirected to a relay (`emit_openat_call_relay_entry`).
- The site is replaced with a `b`/`bl` to a relay stub on a page allocated within ±128 MiB
(`alloc_relay_page`). Sites out of range are logged as `bl-out-of-range`; `large-image
multi-cover tramp` places extra relay pages for big images.
- **Write path:** it first tries `mprotect(PROT_READ|PROT_WRITE|PROT_EXEC)` on the segment.
If that is refused, it `pwrite`s the 4-byte instruction through `/proc/self/mem`. Either way
it then calls `__clear_cache`. The log line counts sites by kind: `patched %d seccomp + %d
path (%d relay) + %d brk`.
2. **In-process SIGSYS fallback**:
- A BPF filter allows syscalls whose instruction pointer is inside proroot's own code and
returns `SECCOMP_RET_TRAP` for the path, exec, clone and wait syscalls from anywhere else.
The IP check is the same idea as our patch 0014, with TRAP instead of TRACE.
- The handler emulates the call and writes the result into the signal frame. Guest SIGSYS
handlers are kept and chained (`proroot_record_user_sigsys`, `invoke_user_sigsys`).
3. **Keeping SIGSYS deliverable**, the objection in option 4 of
[proot-performance](proot-performance.md):
- The `sigprocmask` and `pthread_sigmask` hooks call `sigtrap_unblocked_mask`, which
`sigdelset`s `SIGSYS` (and `SIGTRAP` for some guests) from any `SIG_BLOCK`/`SIG_SETMASK` set.
- `rt_sigaction` is guarded so the guest cannot replace the handler (`proroot_tramp_rt_sigaction`,
the `sigaction_guard` trace points).
- The raw `rt_sigprocmask` that glibc makes internally, for example `pthread_create` blocking
every signal, goes through a patched svc site. The stub loader and the runtime both compare
against syscall 135.
4. **Android-blocked syscalls are answered in-process.** The stub loader dispatches
`set_robust_list`, `rseq`, `faccessat2`, `clone3`, `futex_waitv` and `seccomp`. The runtime
emulates SysV shm (`emu_shm*`), `set*id`/`get*id` under `-0`, and `brk`
(`emulate_guest_brk`, over a reserved mapping).
## Compared with DroidDeck
| | DroidDeck (proot + 0001–0018 + `libblfastpath.so`) | proroot |
|---|---|---|
| exec, `PT_INTERP`, `/proc/self/exe` | proot's loader under ptrace, 0.4–0.8 ms per exec | Its own loaders, with no tracer; also needs its own ld.so with a glibc offset table |
| path calls through libc wrappers | fast path, about 0.6–1.9 µs | runtime hooks |
| path calls from glibc internals (ld.so, fts, realpath, nss, `__open_nocancel`) and raw `svc` | **still go to proot (15–60 µs)** | **patched svc sites** |
| Android-blocked syscalls (`set_robust_list`, `faccessat2`, `clone3`, ...) | SIGSYS stop answered by proot (0016 etc.), one ptrace round trip and one signal round trip each | in-process |
| `brk` | proot emulation, a trap per call (mitigated by `top_pad`) | in-process emulation |
| correctness fallback | proot is always there | none: anything missed runs unconfined or is emulated wrongly |
| long Unix socket paths | fast path `/proc/self/fd/N/name` aliases | short socket aliases (`proroot_create_short_unix_socket_alias`) |
| rseq | `GLIBC_TUNABLES=glibc.pthread.rseq=0` | trampoline |
proroot is not a model for replacing proot. It has no tracer to fall back on, so a missed call
site or an unexpected glibc build silently breaks isolation or correctness. Its own linker depends
on glibc internals for each version. Proton, Wine, FEX and Steam's self-updating ELFs are a much
wider surface than the Node, Python and Chromium workloads proroot was tested on. Its value for
DroidDeck is as a fast path *in front of* proot.
## What is worth taking
1. **Patch glibc's svc sites from `libblfastpath.so`'s constructor.** This is the largest gain
and the cheapest way to do option 6 in [proot-performance](proot-performance.md) ("a proot-aware
glibc") without rebuilding glibc.
- For `libc.so.6` (and `ld-linux-aarch64.so.1` for runtime `dlopen`), rewrite each `svc #0`
that follows a recognised `mov x8, #nr` for the path syscalls the fast path already handles.
The rewrite is a `b` to a relay that calls the existing translator.
- The relay issues the syscall from the `0xffff00000` trampoline page, which 0014 already lets
through. Anything the translator declines is re-issued from an svc in the relay page, outside
the trampoline, so proot traps it exactly as it does today. Correctness still rests on proot.
- This covers fts (`find` over 2.7k files, unchanged today), realpath, nss, `__open_nocancel`,
and `dlopen` after start-up.
- Library loading at start-up, before the constructor runs, still traps.
2. **Answer the Android-blocked syscalls through those relays.**
- `set_robust_list` and `get_robust_list`: keep the head in TLS, as 0016 already does in
proot.
- `faccessat2`: answer `ENOSYS`, or translate it when flags are 0, as the wrapper does.
- `clone3`: answer `ENOSYS` before Android 15, so glibc falls back to `clone`.
- `futex_waitv`: droiddeck-fsync already handles this.
- Each one removes a SIGSYS stop plus a ptrace round trip, about 30–50 µs. `set_robust_list`
alone is one per thread, which matters for Steam/CEF and Wine thread churn.
3. **The `/proc/self/mem` write as a fallback for patching.** If the app domain refuses
`mprotect(PROT_WRITE|PROT_EXEC)` on rootfs (`app_data_file`) text, i.e. an `execmod` denial,
a write through `/proc/self/mem` makes the page copy-on-write without an `mprotect`. That page
is never writable and executable at once. Check this on the Thor before relying on it. The
same question is still open for the trampoline page (`execmem`).
4. **Keep SIGSYS unblocked if in-process `SECCOMP_RET_TRAP` is ever used.** Strip `SIGSYS` in the
`sigprocmask`/`pthread_sigmask`/`rt_sigprocmask` paths, and keep the handler installed while
chaining the guest's. This only becomes safe once item 1 covers glibc's internal
`rt_sigprocmask`. It is a prerequisite, not a recommendation to switch.
5. **Cache patch sites by build-id.** proroot rescans each image on every exec and defers patching
for most programs to keep exec cheap. A scan of libc's roughly 1.5 MiB of text is cheap, but a
per-build-id site table, written once in `/tmp` by the fast path, would make it free.
Not worth taking: the cleanroom ld.so and its glibc offset table; the userland exec loader (proot
already does this); the fake DRM/GBM/libseat device (DroidDeck has its own compositor path);
netlink and SysV shm (`netif.c` and `sysv.c` already cover these); and the long-socket aliasing
(the fast path covers it).
## Open questions before prototyping item 1
- Does the app domain allow `mprotect(RWX)` on a private mapping of an `app_data_file`? Does a
`/proc/self/mem` write work under the app's SELinux context and proot's own `ptrace` attach?
`/proc/self/mem` writes need `PTRACE_MODE_ATTACH` on self, which is always allowed.
- How many `libc.so.6` path syscall sites fall outside a recognisable `mov x8, #nr` window? For
example, `syscall()` takes the number in a register, which needs the existing `syscall()` hook.
- Patched pages become private copies, about 4 KiB per patched page per process. Measure RSS
across the Steam/CEF tree.
- Equivalence: run `bench/equiv.py` plus a `find`/`realpath`/nss workload with `PROOT_FP_STATS`.
Compare the trap counts from patch 0018 before and after.
## First implementation and Thor validation (2026-10-27)
`tools/proot/fastpath/svc-patch.h`, included by `fastpath.c`, implements an opt-in first
slice of item 1. Set `PROOT_FP_PATCH_SVC=1` in a fresh process using the replacement
`libblfastpath.so`. The ordinary fast-path configuration and tracer-key check must also
succeed. It is disabled by default, skips processes that already have multiple threads,
and declines SVE/SME hardware because the relay preserves the base ARM64 register state.
The scanner visits executable segments of the loaded `libc.so.6` and
`ld-linux-aarch64.so.1`. It only patches `openat` and `newfstatat` sites with an
unshifted `movz x8/w8, #nr` within the preceding eight instructions. Intervening
instructions must belong to a small allow-list of argument moves/sign-extension,
immediate moves, stack loads/stores and NOPs that leave x8 unchanged. Branches,
calls, x8 writes and unknown instructions stop recognition. This includes libc's
non-adjacent O_CREAT sequences. Register-number dispatchers and other syscall
numbers remain under PRoot. No glibc version offsets, proroot code, or modified
proroot binaries are used.
Each replacement is a four-byte `b` into a nearby relay. Relay pages are filled while
writable, then changed to RX before publishing any branch. Text writes use
`/proc/self/mem`; failure leaves the site unpatched. The relay preserves x1–x30, all
32 SIMD registers, NZCV, FPCR/FPSR and the caller's `errno`, returning a raw kernel-style
result in x0. It calls the existing path translator, using the existing allowed
trampoline for translated calls. Declined and recursive calls execute an ordinary
`svc` in this library, where PRoot still intercepts them. No SIGSYS handler or new
seccomp filter is installed. `PROOT_FP_STATS=1` adds `fastpath-svc` exit counters.
The connected AYN Thor was running runtime r9, glibc 2.43, with a 4 KiB page size in
`u:r:untrusted_app_27:s0`. `bench/svc-probe.c`, run through the app's guest bridge,
confirmed all of the following while PRoot was attached:
- An RX private mapping of the rootfs libc accepted `mprotect(RWX)`.
- `/proc/self/mem` accepted a changed byte in that RX mapping; readback and restoration
succeeded, and the backing file stayed unchanged.
- An anonymous RW mapping could be changed to RX.
- libc contained 415 `svc #0` sites: 327 with an adjacent number load, 76 with a
number load only within the preceding eight instructions, and 12 with neither.
The linker contained 56 sites, 45 adjacent and 11 nearby-only. Nearby counts are
scan statistics, not proof that the number remains live at the syscall.
The initial adjacent-only scanner patched nine sites; the final scanner patched
14, including file-creation paths, across libc and the linker. It used two relay
pages in standalone validation processes. `bench/svc-check.c` exercises the
actual emitted relay and checks raw success/error results, errno, x1–x30, q0–q31,
NZCV, FPCR/FPSR, four simultaneous threads, the multi-thread patching guard, and fork.
Null arguments and unsupported flags fall back to PRoot. The instruction decoder
and exact ±128 MiB branch limits also have a host regression test.
Five alternating pairs of `bench/svc-workload.c` runs compared the same replacement
library with patching off/on, with fast-path stats enabled in both. The workload
gets `fstatat` and `__open64_nocancel` directly from libc, bypassing the preload's
public wrappers, and makes 5,000 of each call against a prepared regular file:
| median | wrappers only | svc patching |
|---|---:|---:|
| internal `fstatat` | 38.6 µs | 3.0 µs |
| internal open + close | 39.3 µs | 5.8 µs |
| voluntary context switches over both loops | 10,000 | 0 |
### SD allocation and writes
`bench/svc-storage.c` separates the operations relevant to Steam downloads. Three
alternating pairs ran on internal `/tmp` and the selected `/mnt/droiddeck-sd` library,
with Steam idle and no simultaneous local Steam download. Each fresh directory held 64 long-named files
extended to 256 KiB, followed by 1,000 libc-internal stats and a 64 MiB sequential
write followed by `fdatasync`. Both modes used the same replacement library; only
`PROOT_FP_PATCH_SVC` changed. Medians on SD were:
| phase | wrappers only | svc patching |
|---|---:|---:|
| create/extend/close 64 files | 1.500 s | 1.486 s |
| 1,000 internal stats | 61.2 ms | 5.4 ms |
| 64 MiB write calls | 43.9 ms | 40.9 ms |
| final `fdatasync` | 2.514 s | 2.539 s |
Internal allocation fell from 7.1 ms to 1.9 ms; internal stats fell from 39.5 ms
to 2.5 ms. Internal `fdatasync` varied from 36 to 164 ms across individual runs.
The patched SD workload recorded 1,065 internal calls handled in-process, covering
64 file creations, 1,000 stats and the streaming-file creation. Per-phase medians
are independent and should not be added as a median whole-run duration.
The SD metadata gain does not establish faster Steam downloads. File creation
improved by only a few percent, and write/flush speed did not improve in this probe.
The remaining SD cost is much larger than the saved syscall-interception time.
The earlier [Thor SD investigation](thor-sd-storage-investigation.md) likewise
identified native exFAT metadata waits; the current benchmark does not independently
attribute those waits to a particular mount flag. A real Steam install comparison
must separate preallocation, transfer/staging and final commit before claiming an
end-to-end improvement.
### Real Steam downloads to SD
Four fresh installs of owned, initially uninstalled **7 Billion Humans** (792100)
used library folder 1, `/mnt/droiddeck-sd`, and were uninstalled between trials.
Each downloaded 228,935,584 bytes and staged 237,717,350 bytes in 19 files, ending
at BuildID 8016557, depots 792101/5052600457178655633 and
792104/8245917369753676198. Every trial logged zero reused/stored/delta bytes at
the start and completed with `No Error`. Existing installed games were retained.
The repeat off/on pair used the exact same replacement library, SHA-256
`ab605c021aee271cd49fa1cb337d7ff0c762dbac198b711e9818639c0cd4b17d`.
The first off run used the earlier adjacent-only build, with patching disabled.
Only the opt-in environment flag changed for the final pair. Times below come
from Steam's one-second-resolution `content_log.txt`; total starts at Reconfiguring
and ends at scheduler completion, excluding session startup:
| trial | patching | preallocate | download/stage | commit to completion | total |
|---|---|---:|---:|---:|---:|
| first baseline | off | 1 s | 123 s | 1 s | 135 s |
| first patched | on | <1 s | 47 s | 1 s | 50 s |
| repeat baseline | off | <1 s | 30 s | 1 s | 33 s |
| repeat patched | on | <1 s | 29 s | 2 s | 32 s |
Reconfiguration accounts for the remaining total time. The first baseline
overlapped another Steam client's network download; that client's telemetry must
not be mistaken for the Thor's. Later samples were filtered to local client `0`.
The repeat pair reached similar network rates, roughly 7–8 MB/s while transferring.
All four attempts logged the same unsupported large-file preallocation retry.
**This does not demonstrate a useful SD-download speedup.** The apparent initial
135→50 second gain disappeared when baseline was repeated. The final 33 versus
32 seconds is within this small experiment's timing/network variability. This
19-file title also does not settle performance for games with thousands of loose
files. Metadata calls clearly improve, while SD creation/writeback remain the
stronger next targets. No network settings, SD mount flags or root privileges
were changed for these tests.
The internal-call equivalence workload, including fts, realpath, reentrant NSS,
runtime dlopen, missing paths and fallback errors, matched `PROOT_FP_OFF=1` at both
the default parent-cache TTL and TTL 0. The existing `equiv.py` also matched after
normalizing random/thermal read payloads to their lengths, volatile PID values in
the `/proc` directory listing, and tolerating Android's
denied `/dev` directory read. Its root fixtures were relocated into a temporary tree.
`equiv2.py` used temporary root/SD fixtures, normalized `/proc/<self-pid>` output,
and tolerated the SD filesystem's unsupported symlink creation. Its patched output
matched the wrapper-only output at both TTLs. **It did not fully match PRoot:**
binding a Unix socket in the selected SD bind returned EPERM with the existing
wrappers, whereas PRoot succeeded. This mismatch reproduced with svc patching off;
it is a separate existing fast-path socket issue. The suite is not a clean full
equivalence pass. `equiv3.py` was not run because it rewrites live NSS configuration.
A fresh Steam session using the override and `PROOT_FP_PATCH_SVC=1` rendered its
library UI. Sonic Mania (584400) gained focus and rendered its title screen. This
is startup smoke coverage, not gameplay or a Steam startup-speed measurement.
The installed PRoot pack emitted no profiling start/counter records despite the
queued profiling setting; the profiler report had `hasCounters: false`. That is a
measurement gap, not evidence of zero remaining traps.
Before enabling this by default: validate signal/cancellation/unwind behavior,
additional glibc builds and workloads, startup cost and Steam/CEF tree RSS; obtain
matching patch-0018 tracer measurements; and resolve the existing SD socket mismatch.
Additional syscall families, Android-blocked syscall emulation and build-id caching
remain follow-ups. In particular this implementation does not remove startup loader
traps before the preload constructor, nor patch raw calls in arbitrary guest ELFs.
Build the tools with the same ARM64 glibc toolchain as the fast path:
```sh
aarch64-linux-gnu-gcc -O2 -Wall -Wextra -pthread \
-o svc-check tools/proot/bench/svc-check.c -ldl
aarch64-linux-gnu-gcc -O2 -Wall -Wextra \
-o svc-probe tools/proot/bench/svc-probe.c -ldl
aarch64-linux-gnu-gcc -O2 -Wall -Wextra \
-o svc-workload tools/proot/bench/svc-workload.c -ldl
aarch64-linux-gnu-gcc -O2 -Wall -Wextra \
-o svc-storage tools/proot/bench/svc-storage.c -ldl
```
Run `svc-probe` inside the app's guest environment, not from `adb shell`.
`svc-check` embeds the fast path and requires its normal enabled tracer environment.
For `svc-workload`, pass a prepared directory containing `file` and a `link` symlink
to it; add any second argument to enable timing. Use the documented library override
workflow in [proot-performance](proot-performance.md) for a fresh session, adding
`PROOT_FP_PATCH_SVC=1` through `droiddeckctl env set` before starting it.
`svc-storage` takes a new, unused directory name; it creates and removes its own test
files and directory. It writes 64 MiB plus 16 MiB of file extensions per run.
+121
View File
@@ -0,0 +1,121 @@
#include "../fastpath/fastpath.c"
#include <assert.h>
#include <sys/wait.h>
#if defined(__aarch64__)
struct snapshot { long x[32]; unsigned char q[32][16]; unsigned long nzcv, fpcr, fpsr; };
extern uint32_t svc_check_site;
extern void svc_register_probe(struct snapshot *, const long *);
__asm__(
".text\n.p2align 4\n.type svc_register_probe,%function\nsvc_register_probe:\n"
"sub sp, sp, #1072\n"
"stp x19, x20, [sp]\nstp x21, x22, [sp, #16]\nstp x23, x24, [sp, #32]\n"
"stp x25, x26, [sp, #48]\nstp x27, x28, [sp, #64]\nstp x29, x30, [sp, #80]\n"
"stp q8, q9, [sp, #96]\nstp q10, q11, [sp, #128]\n"
"stp q12, q13, [sp, #160]\nstp q14, q15, [sp, #192]\n"
"stp x0, x1, [sp, #224]\n"
".irp n,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n"
"mov x\\n, #\\n\n.endr\n"
".irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31\n"
"movi v\\n\\().16b, #0x5a\n.endr\n"
"mov x0, #0xa0000000\nmsr nzcv, x0\n"
"ldr x0, [sp, #232]\nldr x8, [x0, #48]\n"
"ldp x4, x5, [x0, #32]\nldp x2, x3, [x0, #16]\nldp x0, x1, [x0]\n"
".global svc_check_site\nsvc_check_site:\nsvc #0\n"
".irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n"
"str x\\n, [sp, #(256 + \\n * 8)]\n.endr\n"
".irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31\n"
"str q\\n, [sp, #(512 + \\n * 16)]\n.endr\n"
"mrs x0, nzcv\nstr x0, [sp, #1024]\nmrs x0, fpcr\nstr x0, [sp, #1032]\n"
"mrs x0, fpsr\nstr x0, [sp, #1040]\n"
"ldr x0, [sp, #224]\nadd x1, sp, #256\nmov x2, #792\nbl memcpy\n"
"ldp q8, q9, [sp, #96]\nldp q10, q11, [sp, #128]\n"
"ldp q12, q13, [sp, #160]\nldp q14, q15, [sp, #192]\n"
"ldp x19, x20, [sp]\nldp x21, x22, [sp, #16]\nldp x23, x24, [sp, #32]\n"
"ldp x25, x26, [sp, #48]\nldp x27, x28, [sp, #64]\nldp x29, x30, [sp, #80]\n"
"add sp, sp, #1072\nret\n.size svc_register_probe, .-svc_register_probe\n"
);
static void check(const long *args, long want) {
struct snapshot out = {0};
unsigned long fpcr, fpsr;
__asm__ volatile("mrs %0, fpcr\nmrs %1, fpsr" : "=r"(fpcr), "=r"(fpsr));
errno = 123;
svc_register_probe(&out, args);
assert(errno == 123);
assert(out.x[0] == want);
for (int i = 1; i <= 30; i++) {
long expected = i <= 5 ? args[i] : i == 8 ? args[6] : i;
if (out.x[i] != expected) { fprintf(stderr, "x%d: %lx != %lx\n", i, out.x[i], expected); abort(); }
}
for (int i = 0; i < 32; i++)
for (int j = 0; j < 16; j++) assert(out.q[i][j] == 0x5a);
assert(out.nzcv == 0xa0000000UL);
assert(out.fpcr == fpcr && out.fpsr == fpsr);
}
static void *thread_check(void *unused) {
(void)unused;
struct stat st;
const long args[] = {AT_FDCWD, (long)"/usr/lib/libc.so.6", (long)&st, 0, 0, 0, SYS_newfstatat};
for (int i = 0; i < 100; i++) check(args, 0);
return NULL;
}
static pthread_barrier_t barrier;
static void *thread_probe(void *unused) {
pthread_barrier_wait(&barrier);
thread_check(unused);
pthread_barrier_wait(&barrier);
return NULL;
}
int main(void) {
assert(fp_ready());
assert(svc_single_thread());
size_t page = sysconf(_SC_PAGESIZE);
uint32_t *relay = svc_relay_page((uintptr_t)&svc_check_site, page), branch;
assert(relay && svc_emit(relay, &svc_check_site, &branch));
__builtin___clear_cache((char *)relay, (char *)relay + 32);
assert(mprotect(relay, page, PROT_READ | PROT_EXEC) == 0);
int mem = open("/proc/self/mem", O_RDWR | O_CLOEXEC);
assert(mem >= 0 && pwrite(mem, &branch, 4, (off_t)(uintptr_t)&svc_check_site) == 4);
close(mem);
__builtin___clear_cache((char *)&svc_check_site, (char *)(&svc_check_site + 1));
long args[] = {0, 1, 2, 3, 4, 5, SYS_getpid};
check(args, getpid());
struct stat st;
args[0] = AT_FDCWD; args[1] = (long)"/usr/lib/libc.so.6"; args[2] = (long)&st;
args[3] = 0; args[6] = SYS_newfstatat;
check(args, 0);
assert(st.st_size > 0);
args[1] = (long)"/usr/lib/proot-svc-check-does-not-exist";
check(args, -ENOENT);
args[1] = 0;
check(args, -EFAULT);
args[1] = (long)"/usr/lib/libc.so.6"; args[2] = 0;
check(args, -EFAULT);
args[1] = (long)"/proc/self/exe"; args[2] = (long)&st;
check(args, 0);
args[3] = 0x40000000;
check(args, -EINVAL);
pthread_t threads[4];
assert(pthread_barrier_init(&barrier, NULL, 5) == 0);
for (int i = 0; i < 4; i++) assert(pthread_create(&threads[i], NULL, thread_probe, NULL) == 0);
pthread_barrier_wait(&barrier);
assert(!svc_single_thread());
pthread_barrier_wait(&barrier);
for (int i = 0; i < 4; i++) assert(pthread_join(threads[i], NULL) == 0);
assert(pthread_barrier_destroy(&barrier) == 0);
pid_t pid = fork();
assert(pid >= 0);
if (!pid) { thread_check(NULL); _exit(0); }
int status;
assert(waitpid(pid, &status, 0) == pid && WIFEXITED(status) && WEXITSTATUS(status) == 0);
puts("PASS: raw results, errno, x1-x30, q0-q31, NZCV, FPCR/FPSR, 4 threads, fork");
return 0;
}
#else
int main(void) { return 77; }
#endif
+93
View File
@@ -0,0 +1,93 @@
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <link.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
static int mov_sysnum(uint32_t insn) {
if ((insn & 0xffe0001fU) == 0xd2800008U ||
(insn & 0xffe0001fU) == 0x52800008U)
return (insn >> 5) & 0xffff;
return -1;
}
static int scan(struct dl_phdr_info *info, size_t size, void *unused) {
(void)size;
(void)unused;
const char *name = strrchr(info->dlpi_name, '/');
name = name ? name + 1 : info->dlpi_name;
if (strcmp(name, "libc.so.6") && strcmp(name, "ld-linux-aarch64.so.1")) return 0;
unsigned total = 0, adjacent = 0, nearby = 0, counts[512] = {0};
for (int i = 0; i < info->dlpi_phnum; i++) {
const ElfW(Phdr) *ph = &info->dlpi_phdr[i];
if (ph->p_type != PT_LOAD || !(ph->p_flags & PF_X)) continue;
const uint32_t *text = (const uint32_t *)(info->dlpi_addr + ph->p_vaddr);
for (size_t j = 0; j < ph->p_memsz / 4; j++) {
if (text[j] != 0xd4000001U) continue;
total++;
int nr = j ? mov_sysnum(text[j - 1]) : -1;
if (nr >= 0) {
adjacent++;
if (nr < 512) counts[nr]++;
} else {
for (size_t k = 2; k <= 8 && k <= j; k++)
if (mov_sysnum(text[j - k]) >= 0) { nearby++; break; }
}
}
}
printf("{\"object\":\"%s\",\"svc\":%u,\"adjacent_mov_x8\":%u,\"nearby_only\":%u,\"adjacent_syscalls\":{",
name, total, adjacent, nearby);
int comma = 0;
for (unsigned nr = 0; nr < 512; nr++)
if (counts[nr]) { printf("%s\"%u\":%u", comma ? "," : "", nr, counts[nr]); comma = 1; }
puts("}}");
return 0;
}
int main(void) {
long page = sysconf(_SC_PAGESIZE);
int fd = open("/usr/lib/libc.so.6", O_RDONLY | O_CLOEXEC);
if (fd < 0) { perror("libc"); return 1; }
unsigned char *p = mmap(NULL, page, PROT_READ | PROT_EXEC, MAP_PRIVATE, fd, 0);
if (p == MAP_FAILED) { perror("file mmap RX"); close(fd); return 1; }
unsigned char before = p[0], changed = before ^ 1;
errno = 0;
int rwx = mprotect(p, page, PROT_READ | PROT_WRITE | PROT_EXEC), rwx_errno = errno;
if (rwx == 0) {
p[0] = changed;
p[0] = before;
if (mprotect(p, page, PROT_READ | PROT_EXEC)) { perror("restore RX"); return 1; }
}
int mem = open("/proc/self/mem", O_RDWR | O_CLOEXEC), mem_errno = mem < 0 ? errno : 0;
ssize_t written = -1;
int write_errno = 0, verified = 0, restored = 0;
if (mem >= 0) {
errno = 0;
written = pwrite(mem, &changed, 1, (off_t)(uintptr_t)p);
write_errno = errno;
verified = written == 1 && *(volatile unsigned char *)p == changed;
restored = pwrite(mem, &before, 1, (off_t)(uintptr_t)p) == 1 &&
*(volatile unsigned char *)p == before;
close(mem);
}
unsigned char disk = 0;
int file_unchanged = pread(fd, &disk, 1, 0) == 1 && disk == before;
munmap(p, page);
close(fd);
void *anon = mmap(NULL, page, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
errno = 0;
int execmem = anon == MAP_FAILED ? -1 : mprotect(anon, page, PROT_READ | PROT_EXEC);
int execmem_errno = errno;
if (anon != MAP_FAILED) munmap(anon, page);
printf("{\"page_size\":%ld,\"rwx\":%d,\"rwx_errno\":%d,\"mem_open_errno\":%d,"
"\"mem_written\":%ld,\"mem_write_errno\":%d,\"mem_verified\":%d,\"mem_restored\":%d,"
"\"file_unchanged\":%d,\"execmem\":%d,\"execmem_errno\":%d}\n",
page, rwx, rwx_errno, mem_errno, (long)written, write_errno, verified, restored,
file_unchanged, execmem, execmem_errno);
dl_iterate_phdr(scan, NULL);
return 0;
}
+74
View File
@@ -0,0 +1,74 @@
#define _GNU_SOURCE
#include <assert.h>
#include <dlfcn.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
static double now(void) {
struct timespec t;
assert(clock_gettime(CLOCK_MONOTONIC, &t) == 0);
return t.tv_sec + t.tv_nsec / 1e9;
}
int main(int argc, char **argv) {
assert(argc == 2 && mkdir(argv[1], 0700) == 0);
void *libc = dlopen("libc.so.6", RTLD_NOW);
assert(libc);
int (*internal_open)(const char *, int, ...) = dlsym(libc, "__open64_nocancel");
int (*internal_stat)(int, const char *, struct stat *, int) = dlsym(libc, "fstatat");
assert(internal_open && internal_stat);
struct rusage before, after;
assert(getrusage(RUSAGE_SELF, &before) == 0);
char paths[64][4096], stream[4096];
double creation = 0, extension = 0, closing = 0, total = now();
for (int i = 0; i < 64; i++) {
snprintf(paths[i], sizeof paths[i], "%s/asset-with-a-longish-steam-depot-filename-%03d.dat", argv[1], i);
double start = now();
int fd = internal_open(paths[i], O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC, 0600);
assert(fd >= 0);
creation += now() - start;
start = now();
assert(ftruncate(fd, 262144) == 0);
extension += now() - start;
start = now();
assert(close(fd) == 0);
closing += now() - start;
}
double allocation = now() - total;
struct stat st;
double start = now();
for (int i = 0; i < 1000; i++) assert(internal_stat(AT_FDCWD, paths[i % 64], &st, 0) == 0);
double metadata = now() - start;
snprintf(stream, sizeof stream, "%s/stream.bin", argv[1]);
start = now();
int fd = internal_open(stream, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC, 0600);
assert(fd >= 0);
double stream_create = now() - start;
static unsigned char block[1024 * 1024];
memset(block, 0x5a, sizeof block);
start = now();
for (int i = 0; i < 64; i++) assert(write(fd, block, sizeof block) == sizeof block);
double writing = now() - start;
start = now();
assert(fdatasync(fd) == 0);
double syncing = now() - start;
assert(close(fd) == 0);
assert(getrusage(RUSAGE_SELF, &after) == 0);
printf("{\"files\":64,\"bytes_per_file\":262144,\"create_s\":%.6f,\"extend_s\":%.6f,"
"\"close_s\":%.6f,\"allocation_s\":%.6f,\"stat1000_s\":%.6f,"
"\"stream_bytes\":67108864,\"stream_create_s\":%.6f,\"write_s\":%.6f,"
"\"fdatasync_s\":%.6f,\"voluntary_switches\":%ld}\n",
creation, extension, closing, allocation, metadata, stream_create, writing,
syncing, after.ru_nvcsw - before.ru_nvcsw);
fflush(stdout);
for (int i = 0; i < 64; i++) assert(unlink(paths[i]) == 0);
assert(unlink(stream) == 0 && rmdir(argv[1]) == 0);
assert(dlclose(libc) == 0);
return 0;
}
+91
View File
@@ -0,0 +1,91 @@
#define _GNU_SOURCE
#include <assert.h>
#include <dlfcn.h>
#include <errno.h>
#include <fcntl.h>
#include <fts.h>
#include <pwd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
static double now(void) {
struct timespec t;
assert(clock_gettime(CLOCK_MONOTONIC, &t) == 0);
return t.tv_sec * 1e9 + t.tv_nsec;
}
int main(int argc, char **argv) {
assert(argc >= 2);
void *libc = dlopen("libc.so.6", RTLD_NOW);
assert(libc);
int (*internal_stat)(int, const char *, struct stat *, int) = dlsym(libc, "fstatat");
int (*internal_open)(const char *, int, ...) = dlsym(libc, "__open64_nocancel");
assert(internal_stat && internal_open);
char path[4096], link[4096], missing[4096];
snprintf(path, sizeof path, "%s/file", argv[1]);
snprintf(link, sizeof link, "%s/link", argv[1]);
snprintf(missing, sizeof missing, "%s/missing", argv[1]);
const char *paths[] = {path, link, missing, "/etc/os-release", "/lib/libc.so.6", "/proc/self/exe", "/dev/null", "", NULL};
struct stat st;
for (size_t i = 0; i < sizeof paths / sizeof *paths; i++) {
errno = 0;
int r = internal_stat(AT_FDCWD, paths[i], &st, 0);
printf("stat %zu: %d %d", i, r, r ? errno : 0);
if (!r) printf(" %o %ld", st.st_mode & S_IFMT, S_ISREG(st.st_mode) && i != 5 ? st.st_size : 0L);
putchar('\n');
errno = 0;
int fd = internal_open(paths[i], O_RDONLY | O_CLOEXEC);
int err = fd < 0 ? errno : 0;
printf("open %zu: %d %d\n", i, fd < 0 ? -1 : 0, err);
if (fd >= 0) close(fd);
}
errno = 0;
int r = internal_stat(AT_FDCWD, path, &st, 0x40000000);
printf("flags: %d %d\n", r, r ? errno : 0);
char *tree[] = {argv[1], NULL};
FTS *fts = fts_open(tree, FTS_PHYSICAL | FTS_NOCHDIR, NULL);
assert(fts);
unsigned files = 0, dirs = 0, links = 0;
FTSENT *ent;
while ((ent = fts_read(fts))) {
if (ent->fts_info == FTS_F) files++;
if (ent->fts_info == FTS_D) dirs++;
if (ent->fts_info == FTS_SL) links++;
}
assert(fts_close(fts) == 0);
printf("fts: %u files %u dirs %u links\n", files, dirs, links);
char resolved[4096];
assert(realpath(link, resolved) && !strcmp(resolved, path));
struct passwd pw, *found;
char buf[16384];
r = getpwnam_r("root", &pw, buf, sizeof buf, &found);
printf("nss: %d %d\n", r, found != NULL);
void *libm = dlopen("libm.so.6", RTLD_NOW | RTLD_LOCAL);
assert(libm && dlsym(libm, "cos"));
assert(dlclose(libm) == 0);
puts("realpath/dlopen: PASS");
if (argc > 2) {
const int loops = 5000;
struct rusage before, after;
assert(getrusage(RUSAGE_SELF, &before) == 0);
double start = now();
for (int i = 0; i < loops; i++) assert(internal_stat(AT_FDCWD, path, &st, 0) == 0);
double stat_ns = (now() - start) / loops;
start = now();
for (int i = 0; i < loops; i++) {
int fd = internal_open(path, O_RDONLY | O_CLOEXEC);
assert(fd >= 0 && close(fd) == 0);
}
double open_ns = (now() - start) / loops;
assert(getrusage(RUSAGE_SELF, &after) == 0);
printf("bench: stat %.0f ns, open+close %.0f ns, voluntary switches %ld\n",
stat_ns, open_ns, after.ru_nvcsw - before.ru_nvcsw);
}
assert(dlclose(libc) == 0);
return 0;
}
+2
View File
@@ -1828,3 +1828,5 @@ struct passwd *getpwuid(uid_t uid) {
missed(__func__, AT_FDCWD, "(passwd)");
return real_getpwuid(uid);
}
#include "svc-patch.h"
+250
View File
@@ -0,0 +1,250 @@
static inline int svc_number(uint32_t insn) {
if ((insn & 0xffe0001fU) == 0xd2800008U ||
(insn & 0xffe0001fU) == 0x52800008U)
return (insn >> 5) & 0xffff;
return -1;
}
static inline int svc_argument_instruction(uint32_t insn) {
if (insn == 0xd503201fU) return 1;
if ((insn & 0xffc003e0U) == 0x290003e0U ||
(insn & 0xffc003e0U) == 0xa90003e0U) return 1;
if ((insn & 31) == 8) return 0;
return (insn & 0xffe0ffe0U) == 0xaa0003e0U ||
(insn & 0xffe0ffe0U) == 0x2a0003e0U ||
(insn & 0xfffffc00U) == 0x93407c00U ||
(insn & 0x7f800000U) == 0x52800000U ||
(insn & 0x7f800000U) == 0x12800000U ||
(insn & 0xffc003e0U) == 0xb94003e0U ||
(insn & 0xffc003e0U) == 0xf94003e0U ||
(insn & 0xffc003e0U) == 0xb98003e0U;
}
static inline int svc_site_number(const uint32_t *text, size_t at) {
for (size_t back = 1; back <= 8 && back <= at; back++) {
uint32_t insn = text[at - back];
int nr = svc_number(insn);
if (nr >= 0) return nr;
if (!svc_argument_instruction(insn)) break;
}
return -1;
}
static inline int svc_branch(uintptr_t from, uintptr_t to, uint32_t *out) {
int64_t delta = (int64_t)to - (int64_t)from;
if ((delta & 3) || delta < -0x8000000LL || delta >= 0x8000000LL) return 0;
*out = 0x14000000U | ((uint32_t)(delta / 4) & 0x03ffffffU);
return 1;
}
#if defined(__aarch64__) && !defined(FP_SVC_ENCODINGS_ONLY)
#include <asm/hwcap.h>
#include <link.h>
static __thread int svc_busy __attribute__((tls_model("initial-exec")));
static unsigned svc_sites, svc_pages;
static unsigned long svc_fast, svc_slow;
static long svc_original(const long *r) {
register long x0 __asm__("x0") = r[0];
register long x1 __asm__("x1") = r[1];
register long x2 __asm__("x2") = r[2];
register long x3 __asm__("x3") = r[3];
register long x4 __asm__("x4") = r[4];
register long x5 __asm__("x5") = r[5];
register long x8 __asm__("x8") = r[8];
__asm__ volatile("svc #0" : "+r"(x0) : "r"(x1), "r"(x2), "r"(x3), "r"(x4), "r"(x5), "r"(x8) : "memory", "cc");
return x0;
}
__attribute__((used, visibility("hidden")))
long fp_svc_dispatch(const long *r) {
int saved_errno = errno;
long result = FP_SLOW;
if (!svc_busy) {
svc_busy = 1;
if (r[8] == SYS_openat)
result = fp_open((int)r[0], (const char *)r[1], (int)r[2], (mode_t)r[3]);
else if (r[8] == SYS_newfstatat && r[2])
result = fp_stat((int)r[0], (const char *)r[1], (int)r[3], (struct stat *)r[2]);
if (result != FP_SLOW) __atomic_fetch_add(&svc_fast, 1, __ATOMIC_RELAXED);
svc_busy = 0;
}
if (result == FP_SLOW) {
__atomic_fetch_add(&svc_slow, 1, __ATOMIC_RELAXED);
result = svc_original(r);
}
errno = saved_errno;
return result;
}
void fp_svc_entry(void) __attribute__((visibility("hidden")));
/* A replaced svc may have live values in every register, including NZCV and SIMD. */
__asm__(
".text\n.p2align 4\n.hidden fp_svc_entry\n.type fp_svc_entry,%function\nfp_svc_entry:\n"
"bti j\nsub sp, sp, #800\n"
"stp x0, x1, [sp, #0]\nstp x2, x3, [sp, #16]\n"
"stp x4, x5, [sp, #32]\nstp x6, x7, [sp, #48]\n"
"stp x8, x9, [sp, #64]\nstp x10, x11, [sp, #80]\n"
"stp x12, x13, [sp, #96]\nstp x14, x15, [sp, #112]\n"
"stp x18, x19, [sp, #128]\nstp x20, x21, [sp, #144]\n"
"stp x22, x23, [sp, #160]\nstp x24, x25, [sp, #176]\n"
"stp x26, x27, [sp, #192]\nstp x28, x29, [sp, #208]\n"
"stp x30, x17, [sp, #224]\nmrs x0, nzcv\nstr x0, [sp, #240]\n"
"mrs x0, fpcr\nstr x0, [sp, #768]\nmrs x0, fpsr\nstr x0, [sp, #776]\n"
"stp q0, q1, [sp, #256]\nstp q2, q3, [sp, #288]\n"
"stp q4, q5, [sp, #320]\nstp q6, q7, [sp, #352]\n"
"stp q8, q9, [sp, #384]\nstp q10, q11, [sp, #416]\n"
"stp q12, q13, [sp, #448]\nstp q14, q15, [sp, #480]\n"
"stp q16, q17, [sp, #512]\nstp q18, q19, [sp, #544]\n"
"stp q20, q21, [sp, #576]\nstp q22, q23, [sp, #608]\n"
"stp q24, q25, [sp, #640]\nstp q26, q27, [sp, #672]\n"
"stp q28, q29, [sp, #704]\nstp q30, q31, [sp, #736]\n"
"mov x0, sp\nbl fp_svc_dispatch\nstr x0, [sp]\n"
"ldp q0, q1, [sp, #256]\nldp q2, q3, [sp, #288]\n"
"ldp q4, q5, [sp, #320]\nldp q6, q7, [sp, #352]\n"
"ldp q8, q9, [sp, #384]\nldp q10, q11, [sp, #416]\n"
"ldp q12, q13, [sp, #448]\nldp q14, q15, [sp, #480]\n"
"ldp q16, q17, [sp, #512]\nldp q18, q19, [sp, #544]\n"
"ldp q20, q21, [sp, #576]\nldp q22, q23, [sp, #608]\n"
"ldp q24, q25, [sp, #640]\nldp q26, q27, [sp, #672]\n"
"ldp q28, q29, [sp, #704]\nldp q30, q31, [sp, #736]\n"
"ldr x0, [sp, #768]\nmsr fpcr, x0\nldr x0, [sp, #776]\nmsr fpsr, x0\n"
"ldr x0, [sp, #240]\nmsr nzcv, x0\n"
"ldp x0, x1, [sp, #0]\nldp x2, x3, [sp, #16]\n"
"ldp x4, x5, [sp, #32]\nldp x6, x7, [sp, #48]\n"
"ldp x8, x9, [sp, #64]\nldp x10, x11, [sp, #80]\n"
"ldp x12, x13, [sp, #96]\nldp x14, x15, [sp, #112]\n"
"ldp x18, x19, [sp, #128]\nldp x20, x21, [sp, #144]\n"
"ldp x22, x23, [sp, #160]\nldp x24, x25, [sp, #176]\n"
"ldp x26, x27, [sp, #192]\nldp x28, x29, [sp, #208]\n"
"ldr x30, [sp, #224]\nldr x16, [sp, #232]\n"
"add sp, sp, #800\nbr x16\n.size fp_svc_entry, .-fp_svc_entry\n"
);
static int svc_emit(uint32_t *code, uint32_t *site, uint32_t *branch) {
uint32_t back;
if (!svc_branch((uintptr_t)site, (uintptr_t)code, branch) ||
!svc_branch((uintptr_t)(code + 5), (uintptr_t)(site + 1), &back)) return 0;
code[0] = 0xa9bf47f0U; /* stp x16, x17, [sp, #-16]! */
code[1] = 0x580000b0U; /* ldr x16, literal at +24 */
code[2] = 0x10000051U; /* adr x17, +8 */
code[3] = 0xd61f0200U; /* br x16 */
code[4] = 0xa8c147f0U; /* ldp x16, x17, [sp], #16 */
code[5] = back;
uintptr_t entry = (uintptr_t)fp_svc_entry;
memcpy(code + 6, &entry, sizeof entry);
return 1;
}
struct svc_site { uint32_t *site; uint32_t branch; };
static struct svc_site svc_plan[256];
static size_t svc_planned;
static int svc_object(struct dl_phdr_info *info, size_t size, void *unused) {
(void)size;
(void)unused;
const char *name = strrchr(info->dlpi_name, '/');
name = name ? name + 1 : info->dlpi_name;
if (strcmp(name, "libc.so.6") && strcmp(name, "ld-linux-aarch64.so.1")) return 0;
for (int i = 0; i < info->dlpi_phnum; i++) {
const ElfW(Phdr) *ph = &info->dlpi_phdr[i];
if (ph->p_type != PT_LOAD || !(ph->p_flags & PF_X)) continue;
uint32_t *text = (uint32_t *)(info->dlpi_addr + ph->p_vaddr);
for (size_t j = 1; j < ph->p_memsz / 4 && svc_planned < 256; j++) {
if (text[j] != 0xd4000001U) continue;
int nr = svc_site_number(text, j);
if (nr != SYS_openat && nr != SYS_newfstatat) continue;
svc_plan[svc_planned++].site = &text[j];
}
}
return 0;
}
static uint32_t *svc_relay_page(uintptr_t site, size_t page) {
uintptr_t base = site & ~(uintptr_t)(page - 1);
for (uintptr_t delta = page; delta < 0x7ff0000UL; delta += 0x10000UL) {
for (int direction = -1; direction <= 1; direction += 2) {
uintptr_t at = direction < 0 ? base - delta : base + delta;
if (at < page) continue;
void *p = mmap((void *)at, page, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0);
if (p == (void *)at) return p;
if (p != MAP_FAILED) munmap(p, page);
}
}
return NULL;
}
static int svc_single_thread(void) {
struct entry { uint64_t ino; int64_t off; unsigned short len; unsigned char type; char name[]; };
char buf[4096] __attribute__((aligned(8)));
long fd = sc(SYS_openat, AT_FDCWD, (long)"/proc/self/task", O_RDONLY | O_DIRECTORY | O_CLOEXEC, 0, 0);
if (fd < 0) return 0;
int count = 0;
long n;
while ((n = sc(SYS_getdents64, fd, (long)buf, sizeof buf, 0, 0)) > 0) {
for (long at = 0; at < n;) {
const struct entry *e = (const struct entry *)(buf + at);
if (n - at < (long)offsetof(struct entry, name) + 1 ||
e->len < offsetof(struct entry, name) + 1 || e->len > n - at) { count = -1; goto done; }
if (e->name[0] >= '0' && e->name[0] <= '9' && ++count > 1) goto done;
at += e->len;
}
}
if (n < 0) count = -1;
done:
sc(SYS_close, fd, 0, 0, 0, 0);
return count == 1;
}
static void svc_stats(void) {
char line[160];
int n = snprintf(line, sizeof line, "fastpath-svc[%d]: %u sites, %u relay pages, %lu fast, %lu to proot\n",
getpid(), svc_sites, svc_pages, svc_fast, svc_slow);
if (write(2, line, n) < 0) {}
}
__attribute__((constructor)) static void svc_patch_init(void) {
const char *enabled = getenv("PROOT_FP_PATCH_SVC");
if (!enabled || strcmp(enabled, "1") || !fp_ready()) return;
if (getauxval(AT_HWCAP) & HWCAP_SVE) return;
#ifdef HWCAP2_SME
if (getauxval(AT_HWCAP2) & HWCAP2_SME) return;
#endif
if (!svc_single_thread()) return;
size_t page = (size_t)sysconf(_SC_PAGESIZE);
if (page < 4096 || (page & (page - 1))) return;
int mem = (int)sc(SYS_openat, AT_FDCWD, (long)"/proc/self/mem", O_RDWR | O_CLOEXEC, 0, 0);
if (mem < 0) return;
dl_iterate_phdr(svc_object, NULL);
for (size_t first = 0; first < svc_planned;) {
uint32_t *relay = svc_relay_page((uintptr_t)svc_plan[first].site, page);
if (!relay) { first++; continue; }
size_t end = first;
for (; end < svc_planned && (end - first + 1) * 32 <= page; end++) {
uint32_t *code = relay + (end - first) * 8;
if (!svc_emit(code, svc_plan[end].site, &svc_plan[end].branch)) break;
}
__builtin___clear_cache((char *)relay, (char *)relay + page);
if (end == first || mprotect(relay, page, PROT_READ | PROT_EXEC)) {
munmap(relay, page);
first++;
continue;
}
unsigned patched = 0;
for (size_t i = first; i < end; i++) {
struct svc_site *s = &svc_plan[i];
long written = sc(SYS_pwrite64, mem, (long)&s->branch, sizeof s->branch, (long)s->site, 0);
if (written != sizeof s->branch) continue;
__builtin___clear_cache((char *)s->site, (char *)(s->site + 1));
patched++;
}
if (patched) { svc_sites += patched; svc_pages++; }
else munmap(relay, page);
first = end;
}
sc(SYS_close, mem, 0, 0, 0, 0);
if (getenv("PROOT_FP_STATS")) atexit(svc_stats);
}
#endif
+62
View File
@@ -0,0 +1,62 @@
from pathlib import Path
import shutil
import subprocess
import tempfile
import unittest
HEADER = Path(__file__).resolve().parents[1] / "proot/fastpath/svc-patch.h"
class FastPathSvcEncodingTest(unittest.TestCase):
def test_instruction_decoding_and_branch_limits(self):
if shutil.which("cc") is None:
self.skipTest("cc is not available")
source = r'''
#include <assert.h>
#include <stdint.h>
#include <stddef.h>
#define FP_SVC_ENCODINGS_ONLY
#include "HEADER"
int main(void) {
assert(svc_number(0xd2800708) == 56);
assert(svc_number(0x528009e8) == 79);
assert(svc_number(0xd2800709) == -1);
assert(svc_number(0xd2a00708) == -1);
assert(svc_number(0xf2800708) == -1);
assert(svc_number(0xaa0003e8) == -1);
const uint32_t create[] = {0xd2800708, 0x29047fe0, 0x92800c60, 0xd4000001};
const uint32_t create_at[] = {0xd2800708, 0x29047fe3, 0xb9404be3, 0xd4000001};
const uint32_t xstat[] = {0xd28009e8, 0xaa0203e1, 0xaa0303e2, 0x93407c83, 0xd4000001};
const uint32_t overwritten[] = {0xd2800708, 0xaa0003e8, 0xd4000001};
const uint32_t loaded[] = {0xd2800708, 0xf94003e8, 0xd4000001};
const uint32_t branch[] = {0xd2800708, 0x14000001, 0xd4000001};
const uint32_t call[] = {0xd2800708, 0x94000001, 0xd4000001};
const uint32_t shift[] = {0xd2800708, 0xd2a00708, 0xd4000001};
assert(svc_site_number(create, 3) == 56);
assert(svc_site_number(create_at, 3) == 56);
assert(svc_site_number(xstat, 4) == 79);
assert(svc_site_number(overwritten, 2) == -1);
assert(svc_site_number(loaded, 2) == -1);
assert(svc_site_number(branch, 2) == -1);
assert(svc_site_number(call, 2) == -1);
assert(svc_site_number(shift, 2) == -1);
uint32_t out;
uintptr_t pc = 0x10000000;
assert(svc_branch(pc, pc + 4, &out) && out == 0x14000001);
assert(svc_branch(pc, pc - 4, &out) && out == 0x17ffffff);
assert(svc_branch(pc, pc + 0x7fffffc, &out) && out == 0x15ffffff);
assert(svc_branch(pc, pc - 0x8000000, &out) && out == 0x16000000);
assert(!svc_branch(pc, pc + 0x8000000, &out));
assert(!svc_branch(pc, pc - 0x8000004, &out));
assert(!svc_branch(pc, pc + 2, &out));
return 0;
}
'''
with tempfile.TemporaryDirectory() as directory:
work = Path(directory)
program = work / "check.c"
program.write_text(source.replace("HEADER", HEADER.as_posix()))
output = work / "check"
subprocess.run(["cc", "-O2", "-Wall", "-Wextra", "-Werror", str(program), "-o", str(output)], check=True)
subprocess.run([str(output)], check=True, timeout=10)