diff --git a/AGENTS.md b/AGENTS.md index 9dcd88f0..e8e4d628 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ Setup and verify commands: [docs/toolchain-windows.md](docs/toolchain-windows.md - `tests\diff\atomic.py`: exclusive load/store sequences and ARMv8.1 atomics; needs no game files. - `tests\diff\func.py`: whole leaf functions as the generator emits them, loops and jump tables included. - `tests\excl_mt.py`: the exclusive monitor helpers of `cpu.h` under contention and ABA, on several threads; needs no game files. -- Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after); relinking a build that has `shader-prepared.bin` re-warms the GPU driver's cache, which is keyed to the exe (`totk.exe --warm-shaders`, about 3.5 min; `--no-warm` skips it). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle (on 1.4.3 too: `--exefs build\extracted\v143\exefs`). +- Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after). The AMD driver cache that makes prepared shaders fast is keyed to the exe path and holds a limited amount: preparing another build dir can evict this one's libraries (40-220 ms per new effect in play); `totk.exe --warm-shaders` rebuilds them from the shader cache (seconds when they are still cached). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle (on 1.4.3 too: `--exefs build\extracted\v143\exefs`). - Game version 1.4.3: the same commands with `v143` (`--patches recomp\patches-v143.txt`, `build_game.py --version v143`, `make_romfs_patches.py --version v143`). Its builds default to `build\profile-v143`; `run_game.py` runs it on copies of the 1.2.1 save backups. - Shader census (`.venv-py314`): `tools\shader_census.py --version v121 [--translate]` extracts every Maxwell shader to `build/v121/shaders/` and runs it through `build\xlate\totk_xlate.exe` and `spirv-val`; failure groups go to `failures.txt`. `tools\xlate_smoke.py` checks the translator alone, without game files. - Shader preparation (`.venv-py314`, once per build dir and translator): `tools\prepare_shaders.py build\v121` translates every game shader's default variant and the variants of the MacNativeGames shared shader list (read from `refs/` at run time, with the rows its history dropped: `tools/shader_list.py`) into the build's shader cache and builds their stage libraries into the driver cache (`totk.exe --prepare-shaders`; 1.4.3: 29,587 variants, 7 min on 24 threads, 1.7 GB); play then translates and compiles only variants neither covers. diff --git a/docs/backlog.md b/docs/backlog.md index 32ea98b3..c7f42677 100644 --- a/docs/backlog.md +++ b/docs/backlog.md @@ -71,7 +71,10 @@ machine A runs scripted routes; B and C send logs from the user's play. 12. **Driver cache re-warm.** The AMD driver keys its pipeline library cache by exe path, so a moved folder stutters until `totk.exe --prepare-shaders` runs again. Branch `task/rewarm` (not merged, never run in game) stamps `shader-prepared.bin` with device, driver and exe path and re-warms at start on a mismatch. Check whether NVIDIA - and older AMD drivers cache the same way. Needed before machines B and C get an installer build. + and older AMD drivers cache the same way. Needed before machines B and C get an installer build. 2026-10-27: a + relink keeps the cache (`totk.exe --warm-shaders` on a relinked exe: 29,587 variants in 1.8 s), but its size is + limited: preparing a new build dir (28,408 cold libraries) left `build 143` cold, and a library a draw needed took + 40-220 ms in user play instead of 0.2 ms. `--warm-shaders` re-warms without translating. 13. **Shared shader list.** Newer MacNativeGames lists collapse variants that differ only in texture kind into `"*"` rows, which our runtime cannot match (it translates for the exact kind); `tools/prepare_shaders.py` reads `"*"` as 2D. Option: match `"*"` rows at run time, as MacNativeGames' `prepare_shader_seed.py` does. diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index d56fcbad..39aaf2a5 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -2926,9 +2926,10 @@ int vk_pipeline_prepare_shaders(void) { return rc; } -// totk.exe --warm-shaders (tools/build_game.py after it relinked a prepared build): the driver's disk cache is keyed to -// the exe, so a rebuilt totk.exe finds it cold and a new library costs 40-220 ms in play instead of about 2 ms. This -// builds every variant of shader-prepared.bin into it again from the shader cache, without translating the seed. +// totk.exe --warm-shaders: every variant of shader-prepared.bin built into the driver's disk cache again from the shader +// cache, without translating the seed. The AMD cache is keyed to the exe path (a relink keeps it: 29,587 variants in +// 1.8 s) and holds a limited amount: preparing another build dir evicted this one's libraries, and a library a draw +// needed then took 40-220 ms in play instead of 0.2 ms. Seconds when the libraries are still there. int vk_pipeline_warm_shaders(void) { if (!ensure_init()) { fprintf(stderr, "warm-shaders: no Vulkan device\n"); diff --git a/tools/build_game.py b/tools/build_game.py index 026149a9..77fc7d15 100644 --- a/tools/build_game.py +++ b/tools/build_game.py @@ -567,27 +567,6 @@ def mib(n): return f'{n / (1 << 20):,.1f} MiB' -def warm_driver_cache(exe): - """Rebuild the GPU driver's disk cache of a prepared build after its totk.exe changed. - - tools/prepare_shaders.py builds every prepared variant's stage library once so the driver's own disk cache holds it - (vk_pipeline.c "stage library prebuild"). That cache is keyed to the exe: after a relink, a library a draw needs - took 40-220 ms in play instead of about 2 ms. totk.exe --warm-shaders builds them again from the shader cache - (about 3.5 min on 24 threads), without translating. Builds without shader-prepared.bin are left alone. - """ - if not (exe.parent / 'shader-prepared.bin').is_file(): - return - print('build_game: totk.exe changed in a prepared build: warming the driver cache (--no-warm skips it)', flush=True) - t0 = time.monotonic() - r = subprocess.run([str(exe), '--warm-shaders'], cwd=str(exe.parent), stdout=subprocess.PIPE, text=True, - errors='replace') - last = (r.stdout.strip().splitlines() or [''])[-1] - print(f'warm: rc={r.returncode}, {time.monotonic() - t0:.0f} s; {last}') - if r.returncode: - print(f'build_game: warning: the driver cache stays cold; run python tools/prepare_shaders.py ' - f'{below_root(exe.parent)}', file=sys.stderr) - - def main(): ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument('--version', default='v121', help='game version (default v121)') @@ -621,8 +600,6 @@ def main(): f'({", ".join(sorted(INSTALL_SIDE))} and modules.c still compile)') ap.add_argument('--expected', type=Path, help='expected hashes (default tools/data/expected-.json)') - ap.add_argument('--no-warm', action='store_true', - help='leave the driver cache cold after relinking a prepared build (see warm_driver_cache)') args = ap.parse_args() gen = (args.gen or ROOT / 'generated' / args.version).resolve() @@ -729,7 +706,6 @@ def main(): return if args.make_runtime_lib: exe.unlink(missing_ok=True) # ar rcs would keep members of deleted sources - before = exe.stat().st_mtime_ns if exe.exists() else 0 code, dt, peak_one, _ = run_measured(base + [flag_path(exe)], env) print(f'link: {dt:.1f} s, peak {mib(peak_one)}') if code: @@ -746,8 +722,6 @@ def main(): f'image {mib(image)}; ' + ', '.join(f'{k} {mib(v)}' for k, v in sections.items() if k.startswith('.') and v)) if image > 3 * GiB // 2: print('build_game: warning: image over 1.5 GiB; RIP-relative code reaches only 2 GiB', file=sys.stderr) - if exe.stat().st_mtime_ns != before and not args.no_warm: - warm_driver_cache(exe) if __name__ == '__main__':