Add proot fast path for in-process path lookups

Preload library answers stat/open/access/readlink in-process via a
trampoline page proot 0014 lets through seccomp. On by default with a
Performance toggle; also sets GLIBC_TUNABLES rseq=0 and malloc top_pad.
This commit is contained in:
maxjivi05 committed 2026-09-30 21:05:10 -04:00
1 parent 941a6c3ef9
commit 8cfd54a739
19 files changed
+1626 -2

No files matched your search

+310
View File
@@ -0,0 +1,310 @@
# proot, glibc and where the time goes
Research notes for `feat/performance-fixes-2` (2026-09-30). The question was whether Steam and the
games should keep running under proot, run directly on glibc, or a mix of the two, and what the
fastest correct setup is. Everything below was measured on a Snapdragon 8 Elite Gen 5 (SM8850, kernel
6.12.38-android16, Android 16) over adb, unless it says otherwise.
## Short answer
- **Don't drop proot.** On Android it does two jobs. It translates paths. It also answers every
syscall that Android's app seccomp policy blocks (`rseq`, `set_robust_list`, `faccessat2`,
`fchmodat2`, `openat2`, the `set*id` family, `futex_waitv`, ...).
- Stock Arch glibc without a tracer is killed by `SIGSYS` on its first `pthread_create`. That
stays true even with an in-process `SIGSYS` handler, because glibc makes those calls with every
signal blocked.
- Going "pure glibc" means building Termux-style glibc (relocated prefix, blocked syscalls
rewritten), an exec shim for every ELF Steam self-updates, a `/proc/self/exe` emulator for
Chromium, path redirects for `/tmp`, `/dev/shm`, `/mnt/*` and the fake `/proc` and `/sys`
files, and Wine's hard-coded `/tmp/.wine-<uid>`.
- That is a project of its own, and its failure mode is a crash instead of a slow call.
- **During gameplay, proot already costs almost nothing**, thanks to patches 0004–0013. GPU submits
(`ioctl`), `futex`, `read`/`write`, `mmap`, `ppoll`/`epoll`, `sendmsg`/`recvmsg` and
`clock_gettime` (via the vDSO) all run at native speed under the filter.
- **What proot still costs** is every *path* syscall, `execve`, `brk`, thread and process creation,
and every Android-blocked syscall: **15–60 µs each**, against ~0.5 µs natively. All of it goes
through **one tracer thread**, which tops out around 75–95k traps/s. Steam start-up, game loading
and asset streaming, Proton's Python, and any burst on several threads at once all hit this.
- **Most of that cost is not proot's code but the two cross-core wake-ups per trap.** Any
out-of-process mechanism pays them: ptrace, and seccomp user-notification too (measured the same).
The only real fix is to not leave the process.
- **Recommendation: proot plus an in-process fast path.** A preloaded library answers the common
path calls inside the process. It sends them through a trampoline page that proot's filter lets
through, and it falls back to proot for anything it can't prove proot would answer the same way.
- The prototype is in this branch: `tools/proot/fastpath/` and `patches/0014`.
- Speed: `stat` 25 µs → 0.6 µs, `open` 25 µs → 0.9 µs, missing-file lookups 23 µs → 1.6 µs.
Eight threads doing `stat` went from 89k/s to 3.6M/s.
- Correctness: an equivalence suite of awkward paths produced byte-identical results.
- It only uses syscalls the app policy allows.
## How a session runs today
```
SessionService ─HostProcess─▶ libproot.so (ONE tracer process, nice -6 after 2 s)
└─ /usr/bin/env -i … bannerlator-session steam
└─ gamescope ─▶ bannerlator-session (BL_INSIDE) ─▶ steamrtarm64/steam (+ steamwebhelper/CEF tree)
└─ reaper ─▶ bannerlator-game-env ─▶ bannerlator-proton ─▶ Valve ARM64 Proton (python)
└─ wine (arm64ec, FEX loaded in-process as a DLL) ─▶ wineserver, game.exe, …
```
- Every process in the tree, from the session script to the game, is a tracee of the same proot.
- The options are `--kill-on-exit`, `--kernel-release=…` (kompat, for the hostname), `-i uid:uid`
(fake_id0, for Xwayland's `setgid`/`setuid`), and about 25 binds: `/dev`, `/proc`, `/sys`, fake
`/proc` and `/sys` files, the GPU node, `/dev/shm`, storage and the game libraries.
- `/etc/ld.so.preload` loads `libblsession.so` (the `tools/linuxfs/preload/*.c` shims) into every
guest process.
- The Flatpak sandboxes (`BwrapSpawner`) already run in **separate** proot instances, started from
the Android side.
### What proot does per traced syscall
The tracee runs into proot's seccomp filter, which returns `SECCOMP_RET_TRACE`. Then:
1. The kernel stops the tracee and wakes the tracer: a cross-core IPI, and possibly an exit from a
deep idle state.
2. The tracer reads the registers (`PTRACE_GETREGSET`) and reads the path (`process_vm_readv`).
3. It canonicalises the path against the rootfs and bindings: an `lstat` per component, or one
`O_PATH` open plus a `/proc/self/fd` readlink with patch 0002.
4. It writes the host path into a scratch area in the tracee (`process_vm_writev`) and updates the
registers.
5. It resumes the tracee, which is another cross-core wake-up.
Calls flagged `FILTER_SYSEXIT` stop a second time on the way out, for example to translate the
result of `readlink` or `getcwd`. Android-blocked syscalls arrive as a `SIGSYS` signal stop instead,
and proot emulates them (`src/tracee/seccomp.c`). Fork, clone, exec and thread exit each add
`PTRACE_EVENT_*` stops.
## Measurements
The tools are in `tools/proot/bench/`; `run-device.sh` explains the device layout.
**Caveat:** `adb shell` is *not* under the app seccomp filter. Mechanism costs are the same, but
syscalls that Android blocks only for apps (see below) can't be reproduced there.
**Caveat:** the phone in Doze (screen off) runs everything about 7× slower. All numbers below were
taken with the screen awake.
### Per call, as a session runs proot today
`sysbench` in the patched (`patched`) and unpatched (`vanilla`) columns, using the session's own
proot options and a similar set of binds.
| call | direct glibc | termux proot (vanilla) | this repo's proot (0001–0013) | + fast path (0014) |
|---|---|---|---|---|
| getppid / futex / ioctl / pread (untraced) | 0.1–0.4 µs | futex **16.6**, fstat **32.8**, ioctl **30.5** µs | 0.2–0.4 µs | 0.2–0.4 µs |
| stat, deep absolute path | 0.8 µs | 41.7 | 25.3 | **0.6** |
| open + close | 1.7 µs | 40.4 | 25.3 | **0.9** |
| access | | 27.2 | 26.2 | **1.0** |
| fstatat relative to a dirfd | | 45.5 | 26.8 | **1.9** |
| stat of a missing file | | 38.9 | 22.9 | **1.6** |
| readlink /proc/self/exe | | 42.8 | 38–43 | (proot) |
| getcwd / uname / brk | | 31–35 | 30–33 | (proot) |
| memfd_create | | 20.4 | 18.6 | (proot) |
| pthread create+join | | 58 | 48 | 48 |
| fork + exec + wait | ~1.1 ms | 1.16 ms | 0.70–0.78 ms | +0.2–0.4 ms (see open items) |
The patches already in the repo pay off: kompat (0011) and fake_id0 (0012) took `futex`, `fstat` and
`sendmsg` off the trap list, and 0013 did the same for `ioctl`.
### What one trap costs, by mechanism (`mechbench`; the handler does nothing)
| mechanism | cost |
|---|---|
| native syscall | 0.1 µs |
| seccomp filter of 0 / 50 / 150 / 300 compares, allowed syscall | **no difference** (kernel ≥5.11 caches constant decisions per syscall number) |
| ptrace + seccomp, tracer and tracee on the **same core** | 6.2 µs (6.7 µs with register get/set) |
| ptrace + seccomp, **unpinned** (as the app runs it) | 15.5 µs |
| ptrace + seccomp, different cores (same cluster / across clusters) | 34–36 / 42–61 µs |
| ptrace with a spinning tracer, different cores | 20 µs (the tracee's side still has to wake) |
| seccomp user-notification (the "modern proot" design) | 5.7 µs same core, 34–58 µs across, 43 µs unpinned; `SYNC_WAKE_UP` doesn't help |
| **in-process SIGSYS** (`SECCOMP_RET_TRAP`) | **0.83 µs** |
| openat2 `RESOLVE_IN_ROOT` (kernel-side chroot lookup) | 1.0 µs, but **blocked in apps** |
Takeaways:
- The tracer's own work is about 8 µs of the ~14 µs same-core trap. Everything above that is
scheduling.
- Moving to seccomp user-notification would be a rewrite for no gain.
- The filter's length doesn't matter. What matters is not breaking the kernel's cache:
- **Any check on an argument or the instruction pointer must come after the syscall-number
dispatch.** The first version of 0014 checked the address first. That made every syscall in
the session run the whole filter, and cost +0.5 ms per exec.
- Patches 0004 and 0013 already follow this rule.
### One tracer for everything (`parstat`: N threads doing `stat`)
| | 1 thread | 4 threads | 8 threads |
|---|---|---|---|
| one proot | 37k/s, 27 µs | 74–95k/s, 42–54 µs | 68–89k/s, **90–118 µs** each |
| two proots, 2 threads each | | 187k/s total, 21 µs | |
| proot + fast path | 1.64M/s, 0.6 µs | 2.76M/s | **3.60M/s, 2.2 µs** |
A burst of path calls on several threads therefore queues behind the single tracer. CEF loading
assets, Proton's Python, a game streaming files on worker threads and the shader-cache threads all
produce such bursts.
### CPU placement
The workload ran with `taskset`; no fast path. Times are per traced call, and the last column is one
fork + exec.
| affinity | stat | exec |
|---|---|---|
| default | 25.8 µs | 750 µs |
| one core | 14.0 µs | 370 µs |
| the two prime cores | 14.6 µs | |
| the six-core cluster | 25.9 µs | |
The scheduler's wake-affine placement helps when it can put the tracer and the tracee together, and
spreading them apart hurts. **Never pin the tracer away from its tracees.** Today `raiseTracer`
only renices it, which is correct.
### Whole programs
Times are in ms, each with proot's ~40 ms start-up already subtracted.
| | direct glibc | vanilla proot | this repo's proot | + fast path |
|---|---|---|---|---|
| python3: 10 stdlib imports | 75–89 | ~130 | ~110 | ~90 (878 of 879 path calls answered in-process) |
| tar of 2.7k files | 10–40 | ~180 | ~80 | ~20 |
| 200 × fork+exec of `true` | 230 | ~600 | ~500 | ~500 |
| find over 2.7k files | 16–23 | noisy | noisy | unchanged: glibc's fts calls internal entry points |
## Android's app seccomp policy decides more than proot does
The app filter is generated from bionic's `SYSCALLS.TXT`, minus the blocklists, plus the
allowlists. It is the same for every app whatever its `targetSdk`, and anything not listed gets
`SECCOMP_RET_TRAP`, a `SIGSYS`. Filters stack, and the strictest one wins: a blocked call never
reaches proot's `RET_TRACE`. It arrives as a `SIGSYS` stop, and proot answers it (usually `ENOSYS`).
Checked against bionic `android16-release`:
| syscall | app policy | who calls it |
|---|---|---|
| `rseq`, `set_robust_list` | **blocked** | glibc, at the start of every thread, while all signals are blocked |
| `faccessat2` | **blocked** | glibc's `faccessat()`, which tries it on **every** call before falling back, so 2 round trips (`access()` calls the plain `faccessat` syscall directly) |
| `fchmodat2`, `openat2` | **blocked** | newer glibc, Flatpak/libglnx |
| `setuid`, `setgid` and the rest of the family | **blocked** | Xwayland (hence `-i`) |
| `futex_waitv`, `io_uring_*`, `landlock_*` | **blocked** | Proton's fsync, some engines |
| `clone3` | blocked before Android 15 | glibc `pthread_create` |
| `close_range` | blocked on Android 12 | |
| `statx`, `process_vm_readv`/`writev`, `memfd_create`, `pidfd_*`, `seccomp` | allowed | |
On top of the ptrace round trips, every one of these costs a signal round trip through proot.
## Every traced syscall: what it needs and whether it can bypass proot
This is the filter of termux proot at `source.env` plus patches 0001–0013. "Fast path" means the
call can be answered in-process with the 0014 design.
| syscalls | why proot traps them | can it bypass proot? |
|---|---|---|
| `openat`/`open`/`creat`, `newfstatat`/`stat`/`lstat`, `statx`, `faccessat`/`access`, `readlinkat` (exit), `mkdirat`, `unlinkat`, `renameat(2)`, `symlinkat`, `linkat`, `fchmodat`, `fchownat`, `utimensat`, `truncate`, `*xattr`, `inotify_add_watch`, `name_to_handle_at` | path translation (guest → host), plus exit fixups for `readlink` and `rename` | **Yes, through libc wrappers.** The prototype covers open, stat, statx, access, readlink, opendir and fopen; the write family is next. Raw `svc` and glibc's internal callers (ld.so, fts, realpath, nss) still go to proot. |
| `chdir`/`fchdir`, `getcwd` (exit) | proot emulates the cwd; until 0014 the kernel's cwd never moved | With 0014 the kernel's cwd follows the guest's. `getcwd` stays in proot (cheap, rare). |
| `execve`/`execveat` | runs the loader, maps `PT_INTERP` inside the rootfs, shebangs, tracks `/proc/self/exe` | **No.** This is the core of what proot is for. Costs 0.4–0.8 ms per exec; the session scripts already avoid exec-heavy loops (`nap`). |
| `brk` (entry and exit) | heap emulation: programs are mapped by proot's loader, so the kernel's brk area is the loader's | No (`PR_SET_MM` needs `CAP_SYS_RESOURCE`). **Fewer calls:** `GLIBC_TUNABLES=glibc.malloc.top_pad=…` grows the heap in larger steps. |
| `bind`, `connect`, `accept(4)`, `getsockname`, `getpeername` | `sun_path` of Unix sockets | Same-path binds (runtime dir, files dir) need no translation, but the filter can't see the address. Possible fast-path candidate. |
| `wait4`/`waitpid`, `ptrace` | ptrace emulation inside the guest (breakpad, gdb) | Could become opt-in; then `wait4` is free (one stop per wait today). |
| `prctl(PR_SET_DUMPABLE)`, `setrlimit`/`prlimit64(RLIMIT_STACK)` | loader and stack fixups | Already narrowed to these arguments (0004). |
| `ioctl(TCSETSF, termios2, FICLONE)` | Android pty policy, FICLONE `EACCES` | Already narrowed to these requests; GPU ioctls are free (0013). |
| `uname`, `sethostname`, `setdomainname` | the `DroidDeck` hostname (kompat) | Rare. Keep. |
| `set*id`, `*setxattr` | fake_id0 identity mode (0012) | Rare. Keep. |
| `memfd_create` | Qt JIT and php workarounds (string argument) | Every Wayland `wl_shm` buffer and Chromium shared memory pays one stop. Could be dropped if the DroidDeck runtime doesn't need those workarounds. |
| `statfs` (exit) | fakes tmpfs for `/dev/shm` | Rare. |
| `open_tree`, `move_mount`, `fspick`, `mount_setattr`, `openat2` | answered `ENOSYS` (0008, 0010) | Fine. |
| Android-blocked calls (previous table) | `SIGSYS` emulation | **Avoidable at the source:** `GLIBC_TUNABLES=glibc.pthread.rseq=0` removes one stop per thread. The rest need a patched glibc. |
| clone/fork/vfork/exec/exit events | tracking tracees | Needed while proot traces. |
## Options weighed
1. **Pure glibc, no proot.** Rejected as the main path, for the reasons in the short answer.
- Closest prior art: huntergdavis/steamclienttermux, which runs `steamrtarm64` on a patched
Termux glibc 2.44 with an exec shim and a `/tmp` and `/dev/shm` redirect preload.
- They report Steam's first window 7× faster and about 5% more FPS, with proot having used 60–65%
of a core.
- **They still run Proton and the games under proot.**
- DroidDeck also has four package names (`tools/release/variants.txt`), so a relocated prefix
would need four builds or Winlator-style padded path rewriting.
2. **Hybrid by process (Steam in proot, game on glibc, or the reverse).** The game side is the
*harder* half. Wine re-execs itself and `wineserver`, which hits `PT_INTERP`. Its server
directory is hard-coded under `/tmp`, and Proton's esync and fsync use `shm_open` (`/dev/shm`)
and `futex_waitv`, which is blocked. A game's steady state is already mostly untraced, so the
gain is in loading. Not worth it before the fast path.
3. **Seccomp user-notification instead of ptrace.** It measures the same as ptrace, and it can't
rewrite arguments, so every call would have to be emulated. No.
4. **In-process `SIGSYS` for everything** (0.8 µs). The kernel resets a blocked `SIGSYS` to the
default action and kills the process (`force_sig_seccomp`, `HANDLER_CURRENT`). glibc and Wine
block all signals around exactly these calls. Unsafe without also trapping `rt_sigprocmask`. No.
5. **proot plus an in-process fast path (this branch).** Recommended. Details below.
6. **A proot-aware glibc**, the follow-up to 5. Rebuild the rootfs's own glibc (same version, same
Arch package) with two changes:
- its internal path entry points (`__open_nocancel`, `__fstatat64`, ld.so's opens) go through the
fast path, which covers ld.so, fts, realpath and nss;
- it never makes the Android-blocked calls, as Termux's `fakesyscall.json` does.
This removes the remaining library-loading traps and all the `SIGSYS` round trips. The rootfs is
versioned (`linuxfs-rN`), so glibc rebuilds stay under control.
7. **On the tracer side, with no in-process changes:**
- **A second proot for the game tree.** Measured 2.5× trap throughput, and Steam/CEF bursts no
longer stall the game. `BwrapSpawner` already shows the pattern: a stand-in process in the
session relays stdio, the exit status and signals to a proot started from the app.
- Leave the tracer unpinned (above).
- Set `GLIBC_TUNABLES=glibc.pthread.rseq=0:glibc.malloc.top_pad=16777216` in the session
environment.
## The fast path (prototype)
The code is `tools/proot/fastpath/fastpath.c`, preloaded into every guest process. On the proot side
it needs `patches/0014`, enabled with `PROOT_FASTPATH=1`.
- **Trampoline.** The library maps a 4 KiB executable page at the fixed address `0xffff00000`. It
holds `mov x8,x0 … svc #0; ret`. With `PROOT_FASTPATH`, proot's filter checks the caller's address
just before each `RET_TRACE` and allows syscalls coming from that page.
- **Mapping a path.** The library takes the literal guest path and finds the longest binding whose
guest path is a prefix of it; otherwise the path is under the rootfs.
- It opens the *parent* host directory `O_PATH` and requires `/proc/self/fd` to name exactly that
path. That proves no component on the way was a symlink, so proot would have walked the same
directories.
- The last component is never followed: a symlink there goes to proot.
- **Missing files.** For a missing file, the deepest existing ancestor is verified the same way, and
the next component must not exist even as a symlink. Then `ENOENT` is exact.
- **What always goes to proot:**
- `..`, a trailing `/`, `/proc`, `/dev/fd` and `/dev/std*`;
- `O_TMPFILE`, `AT_EACCESS`, and any flag not handled;
- any doubt at all. "Go to proot" means calling the real libc function, so proot decides.
- **Relative paths.** Patch 0014 keeps the kernel's cwd at the guest cwd's host directory. Relative
and dirfd lookups are mapped through `/proc/self/{cwd,fd/N}`.
- **Verified-parent cache.** Verified parents are cached for 2 s (`PROOT_FP_TTL_MS`). That is the
same trade-off `preload/pathcache.c` already makes for the Steam client. A parent directory turned
into a symlink inside that window would be missed; `PROOT_FP_TTL_MS=0` verifies every call, at
2.0 µs instead of 0.6 µs.
- **Allowed syscalls only.** It uses only `openat`, `newfstatat`, `statx`, `faccessat`, `readlinkat`
and `close`, all allowed for apps. A first version used `openat2 RESOLVE_IN_ROOT`, which the app
policy blocks.
- **Equivalence check.** `bench/equiv.py` runs stat, lstat, readlink, read, listdir and access over
awkward paths, from several working directories and through a dirfd. Its output is byte-identical
with and without the fast path, at TTL 0 and at TTL 2000. The paths include:
- absolute and relative symlinks into binds, `..` symlinks, dangling links and loops;
- files inside binds, nested binds (`/dev/shm` inside `/dev`) and `/proc`;
- missing paths behind symlinks.
### Open items before shipping it
- **Validate inside the app**, not just from adb. Three things to check:
- The app domain allows an anonymous `PROT_EXEC` page (`execmem`). If not, map a page of the
library file itself at the fixed address instead, as proot's loader does for every program.
- Nothing claims `0xffff00000` first.
- The stub's syscalls pass Android's filter. They should: all are allowed.
- **Integrate into `libblsession.so`** rather than a second preload. `opens.c` and `pathcache.c`
already define `open` and `access`, so their "real" call should become the fast path. That also
removes the second library's load cost, about 0.2–0.4 ms per exec that is still unexplained (see
the exec row above).
- **Wire the launcher.** The app already knows the binds (`LinuxRuntime.binds`):
- set `PROOT_FASTPATH=1` in proot's environment;
- pass `PROOT_FP_ROOT` and `PROOT_FP_BINDS` to the guest;
- add a Performance toggle next to "Run proot without seccomp", with the fast path off whenever
seccomp is off.
- **Cover the write family** (mkdir, unlink, rename, symlink, chmod, utimens) and Unix-socket
`connect`/`bind` for same-path binds. Then measure a real Steam start in the app, a game load,
and `PROOT_FP_STATS` hit rates for steam, steamwebhelper, wine and wineserver.
- **FEX**, inside Proton as a DLL, translates x86 path syscalls through glibc's `syscall()`, not the
wrappers, so they still trap. A `syscall()` hook could route `openat` and `newfstatat` there too.