mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-22 13:05:57 +00:00
Lift again only the chunks whose inputs changed
gen.py lifted every chunk on each run and only skipped rewriting the ones whose text came out the same, so a rebuild after one changed code cheat still spent the whole lift (102 s on 12 cores, 178 s with 4 workers) to recompile two chunks. A chunk's text depends only on the generator's sources and options, the module and the function starts discovered from it, the chunk's number and functions, and the patch sites and spin waits inside its address range, so the SHA-256 of these is now its key. <mod>/lift_cache.json records each chunk's key with the SHA-256, size and mtime of its file and the statistics and folded ranges the manifest and folded.txt need, under a digest of the entry. A chunk whose entry is intact, whose key matches and whose file is unchanged is reused without a worker; no pool starts when nothing is left to lift. The cache is written after the manifest, and a failed write only warns. --no-cache, or TOTK_LIFT_CACHE=0 for the installer's lift, lifts everything. On 1.4.3 with the installed mods' cheats, a run with unchanged inputs takes 4 s and one with a changed cheat word lifts one chunk in 7 s. Its output equals a --no-cache lift into an empty folder and the previous gen.py's output byte for byte. tests/lift_cache_test.py checks reuse and invalidation on a module it makes itself.
This commit is contained in:
1 parent
7a37fc3c34
commit
5f6fc13712
6 files changed
+430
-26
No files matched your search
+7
-2
@@ -100,7 +100,7 @@ of them.
|
||||
objects before it links: any AVX, BMI or other instruction above x86-64-v2 outside a function listed there as
|
||||
dispatched by a runtime CPU check stops the build.
|
||||
|
||||
`gen.py` sizes its emit pool from the CPU count and the free memory; `--jobs N` sets it, and so does the environment variable `TOTK_LIFT_JOBS=N`, which the installer's lift step honours too (it has no `--jobs`). When a worker runs out of memory, the pool restarts with half the workers (`emit: out of memory with N workers, retrying with M`), down to one; only an out-of-memory error with one worker fails the step.
|
||||
`gen.py` sizes its emit pool from the CPU count and the free memory; `--jobs N` sets it, and so does the environment variable `TOTK_LIFT_JOBS=N`, which the installer's lift step honours too (it has no `--jobs`). When a worker runs out of memory, the pool restarts with half the workers (`emit: out of memory with N workers, retrying with M`), down to one; only an out-of-memory error with one worker fails the step. A later lift into the same `--out` lifts again only the chunks whose inputs changed (`<mod>/lift_cache.json`, see the top of `recomp/gen.py`); `--no-cache`, or `TOTK_LIFT_CACHE=0` (which the installer's lift step honours too), lifts every chunk.
|
||||
|
||||
Lifting takes about 15 s, a clean compile about 200 s on 24 threads; later builds recompile only what changed.
|
||||
A compiler cache in front of clang shortens it further: with `SCCACHE` set to an sccache executable, `build_game.py`
|
||||
@@ -222,7 +222,12 @@ when a setting is described neither in `totk.ini.example` nor in [settings.md](s
|
||||
`.venv\Scripts\python tests\exefs_patches_test.py`, which makes its own modules and runs `gen.py` on one with a jump
|
||||
table (no game files), and `totk.exe --probe-mods`. A change to the cheat VM or the cheat opcode tables
|
||||
(`runtime/cheats.c`, `tools/cheats.py`) passes `.venv\Scripts\python tests\cheats_vm_check.py`, which runs both on
|
||||
cheats of its own with clang from PATH (`--cc` names another).
|
||||
cheats of its own with clang from PATH (`--cc` names another). A change to what a generated chunk depends on, or to
|
||||
the lift cache of `recomp/gen.py` (`<mod>/lift_cache.json`), passes `.venv\Scripts\python tests\lift_cache_test.py`,
|
||||
which lifts a module it makes itself with and without the cache and compares the output (no game files); a new input
|
||||
of the chunk text goes into `chunk_key` with a case there. On game files, a lift with the cache into a used output
|
||||
folder and a `--no-cache` lift into an empty one give the same files apart from `manifest.json` and
|
||||
`lift_cache.json`.
|
||||
|
||||
The lifter (`recomp/`) and the CPU headers (`runtime/cpu.h`, `runtime/simd.h`) are checked against Unicorn. Every lifter
|
||||
change passes these, all run with `.venv`:
|
||||
|
||||
+7
-3
@@ -95,9 +95,13 @@ A cancelled or failed setup resumes from the last finished step (`totk-install.j
|
||||
|
||||
## Rebuilding for the mods
|
||||
|
||||
The generated C and the objects stay in the temp folder, so the next lift rewrites only the chunks whose C changed and
|
||||
Ninja recompiles only those: a new or edited code cheat costs the lift (under a minute on 24 cores, a few minutes on 4)
|
||||
and a few dozen chunks, not a full compile, and the shaders stay prepared.
|
||||
The generated C and the objects stay in the temp folder. The next lift lifts again only the chunks whose inputs
|
||||
changed (the lift cache of `recomp/gen.py`: the patch sites inside a chunk, with their set indices) and rewrites only
|
||||
those whose C changed, and Ninja recompiles only those. A new or edited code cheat costs a few seconds of lift
|
||||
(discovery and a hash of the kept chunks; 7 s for one changed cheat word on 12 cores, against 102 s for the whole lift)
|
||||
and the compile of the chunks that hold its sites, not a full compile, and the shaders stay prepared. A cheat set added
|
||||
or removed before others in the patch file moves their set indices, so the chunks of their sites are lifted too. A
|
||||
change to `recomp/*.py` lifts everything.
|
||||
|
||||
`build-cache/rebuild.txt` holds the command that opens the setup window in its rebuild mode on the install folder
|
||||
(`installer/gui.py --rebuild <install folder>`, started through `python -c` so the release folder is on `sys.path`
|
||||
|
||||
Reference in new issue
Block a user