Merge task/buildkit: the setup installs a build kit into the game folder, so code mods rebuild without the setup folder and after a move (#845)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-11-17 12:04:35 +03:00
commit 4957dc78e3
18 files changed
+493 -93

No files matched your search

+63 -39
View File
@@ -1,8 +1,9 @@
# Installer
A public release that turns the player's own game files into a playable TotK folder. The release carries no game
data and no code derived from the game: the installer builds the game on the player's machine. The installed folder
holds only the game; the installer and its tools stay outside it.
data and no code derived from the game: the installer builds the game on the player's machine. On Windows the
installed folder also holds the build kit, the installer's tools the game rebuilds itself with for code mods, so the
setup folder can be deleted after the install and the install folder moved.
## What the player does
@@ -16,7 +17,8 @@ holds only the game; the installer and its tools stay outside it.
needed for Chinese, Korean and Traditional Chinese text;
- the game language and the install folder.
3. Waits for the setup, one time. The window shows each step and its progress.
4. Plays `totk.exe` from the install folder. The installer and the source NSP or XCI (or NSZ, XCZ) can be deleted.
4. Plays `totk.exe` from the install folder. The setup folder and the source NSP or XCI (or NSZ, XCZ) can be deleted,
and the install folder can be moved: code mods rebuild the game with the build kit inside it.
The install folder (`main.c` `portable_paths()` reads this layout; it stays portable):
@@ -32,16 +34,25 @@ The install folder (`main.c` `portable_paths()` reads this layout; it stays port
logs/ totk.log of the last run, totk.1.log to totk.4.log of the runs before it
shader and pipeline caches prepared by the setup; shader-cache/shader-cache.pack holds the translations
totk-install.json what built this folder: release version, input hashes, finished steps
build-kit/ Windows: the release folder without TotK-Setup.exe (about 0.65 GB); the game's rebuild
for code mods runs the setup window from it
build-cache/ what the rebuild built in (cheats.status, exefs.status, plugins.status), rebuild.txt,
and work/: the generated C, the lift cache and the objects (about 1.6 GB, Windows)
```
## Updating the port
The player runs a newer installer on the same folder. It reads `totk-install.json`, keeps `romfs/`, `exefs/`,
`shared-fonts/`, `profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the
shaders. A newer release that changes nothing game-derived skips the rebuild. The window starts with the last install folder
and the base, update, keys and firmware paths it ran with (paths only, never the keys file's contents), kept in
`%LOCALAPPDATA%\TotK Recomp\setup.json` (`$XDG_STATE_HOME/totk-recomp/setup.json` elsewhere). Entries that no longer exist
stay empty. A reinstall or a switch to another game version starts filled in.
`shared-fonts/`, `profile/` and `totk.ini` after checking their hashes, replaces `build-kit/` with its own files,
rebuilds `totk.exe` and re-prepares the shaders. The setup is an installer and updater: nothing after the install needs
its folder. An install made by a setup before the build kit kept its temp folder in that setup's folder
(`<setup>/temp/<install name>-<hash>`, the setup `build-cache/rebuild.txt` named); the first newer setup moves it into
`build-cache/work` when both are on one disk (on another it starts empty) and writes the new `rebuild.txt`. Until then
the old `rebuild.txt` keeps using the old setup folder. A newer release that changes nothing game-derived skips the
rebuild. The window starts with the last install folder and the base, update, keys and firmware paths it ran with (paths
only, never the keys file's contents), kept in `%LOCALAPPDATA%\TotK Recomp\setup.json`
(`$XDG_STATE_HOME/totk-recomp/setup.json` elsewhere). Entries that no longer exist stay empty. A reinstall or a switch
to another game version starts filled in.
Given an update file other than the recorded one (1.1.0, then 1.2.1, then 1.4.3), the installer extracts and builds again from it
and keeps `profile/`. Saves carry over both ways between 1.2.1 and 1.4.3: 1.4.3 converts a 1.2.1 save on load, and 1.2.1
@@ -53,21 +64,28 @@ loads the saves 1.4.3 wrote.
`tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a
first install, 20 GiB after it), the CPU is at least x86-64-v2 (SSE4.2; with AVX2/FMA the faster `v3` build is used,
otherwise the `v2` one), a Vulkan 1.3 GPU exists. Each failure is one clear message.
2. **Fonts** with Chinese, Korean or Traditional Chinese text and a firmware folder: extract the Switch system fonts
2. **Kit** (Windows): copy the release folder without `TotK-Setup.exe` into `build-kit/` in the install folder
(`installer/steps.py` `Kit`, `installer/layout.py` `KIT_ITEMS`); every step after it runs its tools from there, so
the commands the build records (the compiler's path in the Ninja file) are the ones the game's rebuild runs later.
A newer setup replaces the kit; the kit's own rebuild keeps it. The steps' keys fingerprint file contents, not
paths (`runtime_key` leaves out the lines of `clang --version` that name its folder), so a rebuild from the kit
or from a moved install folder keeps every step the setup recorded.
3. **Fonts** with Chinese, Korean or Traditional Chinese text and a firmware folder: extract the Switch system fonts
from the firmware into `shared-fonts/` (MacNativeGames' `extract_shared_fonts.py`, which verifies every NCA layer
and BFTTF header). Without a firmware folder the setup warns and goes on; the game then lacks those glyphs.
3. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
4. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt` or `patches-v143.txt`) into a temp folder next
to the installer. The code cheats of the mods' cheat files and the mods' ExeFS patch files become patch sets first
(`tools/cheats.py`, `tools/exefs_patches.py`, status in `build-cache/cheats.status` and `exefs.status`); a file the
tools refuse costs those cheats or patches, not the install. The patch files' writes into main's data go to
`build-cache/exefs.data`, which the game stores into main at start; after the lift they are checked against the
bytes the lifted code read as constants (`<gen>/main/folded.txt`, its jump tables).
5. **Compile** the generated C, `modules.c`, `romfs.c`, `patches.c` and `ov_zstd.c` (they bake the per-install layout, RomFS
4. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
5. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt` or `patches-v143.txt`) into the temp folder,
`build-cache/work` in the install folder (`.setup-temp` on Linux, `--temp DIR` elsewhere). The code cheats of the
mods' cheat files and the mods' ExeFS patch files become patch sets first (`tools/cheats.py`,
`tools/exefs_patches.py`, status in `build-cache/cheats.status` and `exefs.status`); a file the tools refuse costs
those cheats or patches, not the install. The patch files' writes into main's data go to `build-cache/exefs.data`,
which the game stores into main at start; after the lift they are checked against the bytes the lifted code read as
constants (`<gen>/main/folded.txt`, its jump tables).
6. **Compile** the generated C, `modules.c`, `romfs.c`, `patches.c` and `ov_zstd.c` (they bake the per-install layout, RomFS
hashes and the game version's patch page) and link
them with the prebuilt runtime library for the host's level (`libtotkrt.a` or `libtotkrt-v2.a`) into `totk.exe`
(`build_game.py --runtime-lib`; about 3 min on 24 cores).
6. **Prepare** (only when the runtime, the translator or the game version changed, not for a new lift) the RomFS
7. **Prepare** (only when the runtime, the translator or the game version changed, not for a new lift) the RomFS
patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`,
`totk.exe --prepare-shaders`) in the install folder, then fold the shader cache's loose files (four per shader, about
117,000) into `shader-cache/shader-cache.pack` (`tools/pack_shader_cache.py`). totk.exe reads the pack first and
@@ -82,7 +100,7 @@ loads the saves 1.4.3 wrote.
When the graphics driver crashes while building the shaders, `prepare_shaders.py` runs that part again without the
variants it was building ([building.md](building.md), step 6), and the setup finishes with a warning that names how
many were left out.
7. **Shapes**: the UI laid out for each screen shape this player needs, `romfs-patches-<w>x<h>`
8. **Shapes**: the UI laid out for each screen shape this player needs, `romfs-patches-<w>x<h>`
(`--aspect`), and with Xbox or PC prompts in `totk.ini` that layout with those prompts too,
`romfs-patches-<style>-<w>x<h>` (`--prompts <style> --aspect --reuse romfs-patches-<style>`, which hard-links every
file but the UI archive; for `pc` with the same `--binds` as the PC prompts step, whose key the step follows). The shapes are those of every connected display (Windows: `EnumDisplayDevices`; Linux:
@@ -93,39 +111,44 @@ loads the saves 1.4.3 wrote.
again when the displays, the prompts or the patches change; at a game resolution whose shape has no folder the
game draws 16:9 with bars, and the settings menu marks such sizes and can write the shape's folders in the
background for the next start.
8. **DLSS** (the window's tick box, on by default, or `--dlss`; Windows with an NVIDIA RTX GPU): download
9. **DLSS** (the window's tick box, on by default, or `--dlss`; Windows with an NVIDIA RTX GPU): download
`nvngx_dlss.dll` from NVIDIA's own DLSS SDK repository (github.com/NVIDIA/DLSS, a pinned tag, NVIDIA RTX SDK
license) into the install folder and check its size and SHA-256 (`steps.py`, `Dlss`). The release does not ship
the library; NGX loads it from the folder of totk.exe for `dlaa` and the DLSS modes. A DLSS library already in the
folder (a newer one the player put there) is kept. Without an RTX GPU the step only logs and does nothing; a failed
download warns, the game runs without DLSS, and the next run of the setup tries again.
9. **Clean up** the temp folder, except the generated C and the objects (about 1.3 GB), and write
10. **Clean up** the temp folder, except the generated C and the objects (about 1.3 GB), and write
`build-cache/rebuild.txt` in the install folder.
A cancelled or failed setup resumes from the last finished step (`totk-install.json`).
## Rebuilding for the mods
The generated C and the objects stay in the temp folder. The next lift lifts again only the chunks whose inputs
changed (the lift cache of `recomp/gen.py`: the patch sites inside a chunk, with their set indices) and rewrites only
those whose C changed, and Ninja recompiles only those. A new or edited code cheat costs a few seconds of lift
(discovery and a hash of the kept chunks; 7 s for one changed cheat word on 12 cores, against 102 s for the whole lift)
and the compile of the chunks that hold its sites, not a full compile, and the shaders stay prepared. A cheat set added
or removed before others in the patch file moves their set indices, so the chunks of their sites are lifted too. A
change to `recomp/*.py` lifts everything.
The generated C and the objects stay in the temp folder, inside the install folder. The next lift lifts again only the
chunks whose inputs changed (the lift cache of `recomp/gen.py`: the patch sites inside a chunk, with their set indices)
and rewrites only those whose C changed, and Ninja recompiles only those. A new or edited code cheat costs a few seconds
of lift (discovery and a hash of the kept chunks; 7 s for one changed cheat word on 12 cores, against 102 s for the
whole lift) and the compile of the chunks that hold its sites, not a full compile, and the shaders stay prepared. A
cheat set added or removed before others in the patch file moves their set indices, so the chunks of their sites are
lifted too. A change to `recomp/*.py` lifts everything.
`build-cache/rebuild.txt` holds the command that opens the setup window in its rebuild mode on the install folder
(`installer/gui.py --rebuild <install folder>`, started through `python -c` so the release folder is on `sys.path`
whatever the working folder), one argument per line. When an enabled mod has code cheats or ExeFS patch files the build
lacks, `totk.exe` runs it at start and exits (`runtime/cheats.c` `rebuild_if_needed`); the
window runs the setup on the folder at once and starts the game again when it succeeds. A failed or cancelled rebuild
offers to start the game with the code it has; `build-cache/rebuild.skip` then keeps the same files from starting it
again, until "Rebuild the game now" on the Cheats or Mods tab. A command that does not start at all leaves no
`rebuild.skip`, so the next start tries it again. `[mods] mods_rebuild = off` turns it off. The rebuild needs the
release folder the setup ran from: a moved or deleted release folder means running a setup again. On Linux the command
starts the AppImage file itself (`$APPIMAGE --rebuild <install folder>`), because the AppImage's mounted folder is gone
once the setup exits; a moved or deleted AppImage means running a setup again. Installs set up by an AppImage before
this change hold a command into that mount: running the new setup once on the install folder writes the new one.
whatever the working folder), one argument per line. On Windows it names the build kit relative to the install folder
(`build-kit\python\pythonw.exe`, `build-kit`, `--rebuild .`): `totk.exe` takes the program from its own folder and
starts the command there, so no path of the setup or of the install folder is stored. When an enabled mod has code
cheats or ExeFS patch files the build lacks, `totk.exe` runs it at start and exits (`runtime/cheats.c`
`rebuild_if_needed`); the window runs the setup on the folder at once and starts the game again when it succeeds. A
failed or cancelled rebuild offers to start the game with the code it has; `build-cache/rebuild.skip` then keeps the
same files from starting it again, until "Rebuild the game now" on the Cheats or Mods tab. A command that does not start
at all leaves no `rebuild.skip`, so the next start tries it again. `[mods] mods_rebuild = off` turns it off. A moved
install folder keeps its incremental rebuild: the kit's builds run `tools/build_game.py --portable`, whose Ninja file
and deps log name every path relative to the kit (3.5 s to rebuild after a move instead of a full compile). A build with
a mod plugin names the plugin's file by its absolute path (`recomp/layout.py`), so after a move the game finds it gone
and runs the rebuild once. On Linux the command starts the AppImage file itself (`$APPIMAGE --rebuild <install
folder>`), because the AppImage's mounted folder is gone once the setup exits; a moved or deleted AppImage means running
a setup again. Installs set up by an AppImage before this change hold a command into that mount: running the new setup
once on the install folder writes the new one.
## Command line
@@ -170,7 +193,8 @@ python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR
The unpacked release; `tools/make_release.py` assembles it and `--zip` packs it:
```
TotK-Setup.exe starts python/pythonw.exe -m installer.gui (installer/launcher/launcher.c, no console)
TotK-Setup.exe starts python/pythonw.exe -m installer.gui (installer/launcher/launcher.c, no console); the
rest is what the Kit step copies into the install folder's build-kit/
installer/ the installer: steps, CLI, window, en/ru/zh-Hans text catalog
python/ Python 3.14.9 embeddable + numpy, lz4, cryptography, cffi, pycparser (+ tkinter for the window)
recomp/, tools/ the lifter and the build, extraction and shader scripts the steps run (copied from the repo)
+5 -3
View File
@@ -80,9 +80,11 @@ ahead of time, so these patches are built into `totk.exe`:
2. Until the rebuild succeeds, the whole mod stays off, its other files included.
3. After the rebuild the mod switches on and off from the Mods tab like any other, without another rebuild.
The rebuild needs the setup folder the game was installed from. When that folder was moved or deleted, run
`TotK-Setup.exe` again on the same install folder: it does the same job. `mods_rebuild = off` under `[mods]` in
`totk.ini` turns the automatic rebuild off.
The rebuild runs from `build-kit` in the game folder, which the setup put there: the setup folder can be deleted
and the game folder moved. After a move, a game with a code plugin mod rebuilds once at its next start. A game
installed by an older setup (no `build-kit` in its folder) still rebuilds from that setup folder until a newer
setup runs once on it.
`mods_rebuild = off` under `[mods]` in `totk.ini` turns the automatic rebuild off.
A code patch over one of the port's own patches (the 60 FPS patch, the resolution patch and the other entries under
**Built into the port**) turns that built-in patch off while the mod is on. The mod's row names it.
+7 -17
View File
@@ -12,7 +12,6 @@ line, 3 cancelled (Ctrl+C or Ctrl+Break).
Windows with an NVIDIA RTX GPU (steps.py, Dlss).
"""
import argparse
import hashlib
import signal
import sys
import time
@@ -22,7 +21,7 @@ from . import layout as layouts
from . import state as statefile
from .events import Reporter
from .runner import Context, run_steps
from .steps import all_steps, migrate_state
from .steps import adopt_old_temp, all_steps, migrate_state
EXIT = {'ok': 0, 'failed': 1, 'cancelled': 3}
@@ -33,17 +32,6 @@ class Parser(argparse.ArgumentParser):
sys.exit(2)
# The temp folder inside the install folder when the layout names none (the Linux release); the check step does not
# count it when it requires an empty install folder
TEMP_IN_DEST = '.setup-temp'
def temp_folder(base, dest):
"""One temp folder per install folder, so two installs never share generated files."""
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
return base / f'{dest.name or "root"}-{tag}'
def cancel(signum, frame):
raise KeyboardInterrupt
@@ -58,8 +46,8 @@ def main(argv=None):
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
ap.add_argument('--dest', type=Path, required=True, help='install folder')
ap.add_argument('--root', type=Path, help='release folder or repository (default: the folder above installer/)')
ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release on Windows, .setup-temp in '
'the install folder on Linux, build/install-temp in a repository)')
ap.add_argument('--temp', type=Path, help='temp folder (default: build-cache/work in the install folder on Windows, '
'.setup-temp there on Linux, build/install-temp in a repository)')
ap.add_argument('--verify', action='store_true', help='hash every kept file, also those this release made')
ap.add_argument('--dlss', action='store_true', help="download NVIDIA's nvngx_dlss.dll (NVIDIA RTX SDK license) next "
'to the game, for dlaa and the DLSS modes')
@@ -72,10 +60,12 @@ def main(argv=None):
signal.signal(getattr(signal, name), cancel)
layout = layouts.find(args.root)
dest = args.dest.resolve()
base = args.temp or layout.temp
temp = temp_folder(base.resolve(), dest) if base else dest / TEMP_IN_DEST
temp = layouts.temp_for(layout, dest, args.temp)
adopted = adopt_old_temp(layout, dest, temp)
log = temp / 'logs' / time.strftime('install-%Y%m%d-%H%M%S.log')
rep = Reporter(log_path=log)
if adopted:
rep.log(adopted)
t0 = time.monotonic()
try:
state = statefile.load(dest)
+3 -1
View File
@@ -746,7 +746,9 @@ def main(argv=None):
return selftest(layout, args.shots)
fix_tcl_paths()
root = tk.Tk()
App(root, args.lang or windows_ui_language(), layout, rebuild=args.rebuild)
# resolved now: the game starts the build kit's rebuild with the install folder as '.' and the working folder
rebuild = args.rebuild.resolve() if args.rebuild else None
App(root, args.lang or windows_ui_language(), layout, rebuild=rebuild)
root.mainloop()
return 0
+2 -1
View File
@@ -27,7 +27,8 @@ TEXT = Path(__file__).resolve().parent / 'text'
UI_LANGUAGES = ('en', 'ru', 'pt-BR', 'zh-Hans')
# what the language switch in the window's corner shows for each
UI_LANGUAGE_LABELS = {'en': 'English', 'ru': 'Русский', 'pt-BR': 'Português (Brasil)', 'zh-Hans': '简体中文'}
STEP_ORDER = ('check', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss', 'cleanup')
STEP_ORDER = ('check', 'kit', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss',
'cleanup')
# message parameter -> catalog id prefix of the names that replace its raw values
PARAM_NAMES = {'lang': 'lang.', 'known': 'lang.', 'step': 'step.', 'input': 'param.input.'}
LANG_RU = 0x19 # primary language id of Russian in a Windows LANGID
+46 -6
View File
@@ -16,7 +16,13 @@ Release (docs/installer.md "Release layout"), root = the unpacked folder:
without one the game is built at -O1
totk.ini.example
VERSION the release version, one line
temp/ the default temp folder (created)
Build kit (Windows release): the setup copies the release folder without TotK-Setup.exe into <install>/build-kit
(KIT_ITEMS, steps.py Kit) and runs every later step from it, so the game's rebuild for its mods
(build-cache/rebuild.txt, relative to the install folder) needs neither the setup folder nor its path: the setup can be
deleted and the install folder moved. The kit is a release folder itself (release_layout). The temp folder with the
rebuild's working state (gen/, build/, the lift cache) is <install>/build-cache/work (TEMP_IN_DEST), so it moves
with the install and survives a newer setup's kit.
Linux release (the AppImage's folder, docs/installer.md "Linux release"): python/bin/python3 (python-build-standalone,
with tkinter), llvm/ with clang, lld and llvm-ar plus llvm/sysroot (the glibc headers and libraries the game links
@@ -26,9 +32,11 @@ Dev mode, root = a repository checkout (no python/python.exe): build/py314-embed
Python the release ships, with the same packages), else this Python; toolchain/llvm-mingw-*/ and toolchain/deps-mingw;
runtime/ as the sources with build/install-rtlib/libtotkrt.a and libtotkrt-v2.a (tools/build_game.py --make-runtime-lib
--out build/install-rtlib, once per --cpu); MSYS2 UCRT64 ninja.exe; toolchain/hactool; build/xlate/totk_xlate.exe; the shader list from
refs/MacNativeGames; build/pgo/pgo-<v>.profdata when there is one; build/install-temp. The version is VERSION with -dev and the commit: 0.1.0-dev+973c626. On Linux,
dev mode takes clang, lld and ninja from PATH and the system's libraries (no deps folder, no SDL3 copy).
refs/MacNativeGames; build/pgo/pgo-<v>.profdata when there is one; build/install-temp/<install name>-<hash>. The
version is VERSION with -dev and the commit: 0.1.0-dev+973c626. Dev mode and the Linux release install no build kit:
the repository, or the AppImage file, is what the rebuild runs. On Linux, dev mode takes clang, lld and ninja from PATH and the system's libraries (no deps folder, no SDL3 copy).
"""
import hashlib
import os
import shutil
import subprocess
@@ -40,6 +48,14 @@ WIN = os.name == 'nt'
EXE = '.exe' if WIN else ''
UCRT = Path(r'C:\msys64\ucrt64\bin')
SUPPORTED = ('1.2.1', '1.4.3', '1.1.0') # updates the installer builds; the check identifies the player's by hash
KIT = 'build-kit' # the build kit's folder in the install folder
# What the build kit holds of a release folder (tools/make_release.py writes them): all but TotK-Setup.exe. licenses/
# and THIRD_PARTY_NOTICES.md go along with the binaries they cover.
KIT_ITEMS = ('python', 'llvm', 'runtime', 'bin', 'recomp', 'tools', 'data', 'installer', 'licenses',
'THIRD_PARTY_NOTICES.md', 'totk.ini.example', 'VERSION')
# The temp folder inside the install folder, for a layout that names none (every release); the check step does not
# count it when it requires an empty install folder. Linux keeps the name its installs already use.
TEMP_IN_DEST = Path('build-cache', 'work') if WIN else Path('.setup-temp')
def version_tag(version):
@@ -139,9 +155,10 @@ def release_layout(root):
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll') if WIN else (root / 'bin' / 'libSDL3.so.0',),
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example',
pgo_dir=root / 'data' if WIN else None,
# Linux: None, the install folder's .setup-temp (installer/cli.py): the AppImage is mounted read-only, and on
# a Steam Deck /home is often too full for the generated C while the install folder is on the SD card
temp=root / 'temp' if WIN else None,
# None: the install folder's TEMP_IN_DEST (temp_for). Windows: the rebuild's working state stays with the install
# and its build kit. Linux: the AppImage is mounted read-only, and on a Steam Deck /home is often too full for
# the generated C while the install folder is on the SD card
temp=None,
sysroot=None if WIN else root / 'llvm' / 'sysroot')
@@ -191,3 +208,26 @@ def find(root=None):
"""The layout of root (default: the folder above this package)."""
root = Path(root or Path(__file__).resolve().parent.parent).resolve()
return release_layout(root) if is_release(root) else dev_layout(root)
def uses_kit(layout):
"""True when the setup installs a build kit (Windows release); dev mode and the Linux AppImage run from where they
are."""
return WIN and not layout.dev
def kit_folder(dest):
return Path(dest) / KIT
def temp_folder(base, dest):
"""One temp folder per install folder below a shared base (dev mode, --temp), so two installs never share
generated files."""
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
return base / f'{dest.name or "root"}-{tag}'
def temp_for(layout, dest, override=None):
"""The temp folder of a run on dest: below --temp or the layout's, else TEMP_IN_DEST in the install folder."""
base = override or layout.temp
return temp_folder(Path(base).resolve(), dest) if base else dest / TEMP_IN_DEST
+6
View File
@@ -49,6 +49,10 @@ class Step:
"""Do the work. Returns fields to record with the step."""
return {}
def settle(self, ctx):
"""After the step ran or was skipped: a step whose output the later steps run from (the build kit) points the
context at it."""
class Context:
def __init__(self, layout, dest, temp, reporter, options, state=None, verify=False):
@@ -121,6 +125,7 @@ def run_steps(steps, ctx):
for s in steps:
if not ctx.runs[s.id]:
rep.step(s.id, 'skipped', reason=ctx.reasons[s.id])
s.settle(ctx)
continue
current = s.id
rep.step(s.id, 'started')
@@ -133,6 +138,7 @@ def run_steps(steps, ctx):
ctx.save() # the first save, after the check, creates the install folder
rep.step(s.id, 'done', seconds=seconds)
current = None
s.settle(ctx)
ctx.state['release'] = ctx.layout.release
ctx.save()
return 'ok'
+141 -15
View File
@@ -1,5 +1,5 @@
"""The ten setup steps of docs/installer.md: check, fonts, extract, lift, compile, prepare, pc_prompts, shapes, dlss,
cleanup.
"""The eleven setup steps of docs/installer.md: check, kit, fonts, extract, lift, compile, prepare, pc_prompts, shapes,
dlss, cleanup.
Extraction runs in this process through tools/extract.py (the package's sibling tools/, which ships with it); fonts
runs the vendored extract_shared_fonts.py on the firmware folder; lift, compile and prepare run the bundled Python on
@@ -11,9 +11,11 @@ Install folder: exefs/, romfs/ (romfs.bin, index.json), profile/, shared-fonts/
and its DLLs, nvngx_dlss.dll (with --dlss), romfs-index.bin, romfs-patches/, romfs-patches-xbox/,
romfs-patches-xbox-label/, romfs-patches-pc/, romfs-patches-<w>x<h>/ and romfs-patches-<style>-<w>x<h>/ (for the
screen shapes and the prompts style this player needs, installer/screens.py), totk_xlate.exe, the
shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json.
Temp folder (<temp>/<dest name>-<hash>): check/ (ExeFS of both inputs, the host check), unpacked/ (the NSP of an NSZ
or XCZ input until the extraction is done), gen/ (C), build/ (objects), logs/. The cleanup keeps gen/, build/ and
shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json, and on Windows build-kit/ (Kit): the release
folder the steps after the kit step run from, and the game's rebuild with them.
Temp folder (build-cache/work in the install folder on Windows, .setup-temp there on Linux, <temp>/<dest name>-<hash>
with --temp or in a repository; installer/layout.py temp_for): check/ (ExeFS of both inputs, the host check),
unpacked/ (the NSP of an NSZ or XCZ input until the extraction is done), gen/ (C), build/ (objects), logs/. The cleanup keeps gen/, build/ and
logs/: the next run lifts into the same gen/, where recomp/gen.py lifts again only the chunks whose inputs changed (its
lift cache, gen/<mod>/lift_cache.json) and rewrites only those whose C changed, and Ninja in build/ recompiles only
those. A change of the mods' code cheats then costs a few seconds of lift and the chunks holding its sites, not a full
@@ -22,8 +24,8 @@ lift or compile.
Install folder build-cache/: cheats.status (tools/cheats.py), exefs.status and exefs.data (tools/exefs_patches.py),
plugins.status (tools/exefs_plugins.py; plugins.stale is the game's list of plugins whose start differed) and
rebuild.txt, the command that starts this setup in its rebuild mode on the install folder (installer/gui.py --rebuild),
one argument per line; totk.exe runs it at start when an enabled mod has code cheats the build lacks
(runtime/cheats.c).
one argument per line, relative to the install folder for the build kit; totk.exe runs it at start when an enabled mod
has code cheats the build lacks (runtime/cheats.c).
"""
import hashlib
import http.client
@@ -40,6 +42,7 @@ import urllib.request
from pathlib import Path
from . import ini, screens
from . import layout as layouts
from .layout import EXE, SUPPORTED, version_tag
from .runner import InstallError, Step
@@ -53,6 +56,7 @@ from extract_exefs import (common_tickets, container_entries, load_keys, nca_ctr
GiB = 1 << 30
DEST_EXTRA = 4 * GiB # beside the RomFS: shader and pipeline caches (about 2 GB), exefs, totk.exe, seed files
TEMP_NEED = 3 * GiB # generated C (about 0.9 GB) and objects
KIT_NEED = 1 * GiB # the build kit (Kit), about 0.65 GB
HOSTCHECK = Path(__file__).resolve().parent / 'hostcheck.c'
STAMP = '.install-key'
@@ -598,6 +602,8 @@ class Check(Step):
dest_need -= romfs.stat().st_size
if ctx.will_run('fonts'):
dest_need += FONTS_NEED
if ctx.will_run('kit') and layouts.uses_kit(ctx.layout):
dest_need += KIT_NEED
if ctx.will_run('prepare') or ctx.will_run('extract'):
dest_need += DEST_EXTRA
if ctx.will_run('lift') or ctx.will_run('compile'):
@@ -650,6 +656,114 @@ class Check(Step):
ctx.state['version'] = found
# ---- 1b. build kit
class Kit(Step):
"""The build kit (installer/layout.py, Windows release): the release folder without TotK-Setup.exe, copied into
<install>/build-kit. Every step after this one runs its tools from there (settle), and the game's rebuild for its
mods starts the setup window from there, so the setup folder can go once the install is done and the install
folder can move. A newer setup replaces the kit; the kit's own rebuild keeps it. The steps' keys fingerprint file
contents, not paths, so the kit, a copy of the setup's files, keeps every step the setup recorded."""
id = 'kit'
def key(self, ctx):
return fingerprint('kit-1', ctx.layout.release if layouts.uses_kit(ctx.layout) else 'none')
def current(self, ctx, record):
if not layouts.uses_kit(ctx.layout):
return True
kit = layouts.kit_folder(ctx.dest)
if not layouts.is_release(kit):
return False
found = layouts.release_layout(kit)
return found.release == ctx.layout.release and not found.missing(getattr(ctx, 'cpu', 'v3'))
def run(self, ctx):
layout, rep = ctx.layout, ctx.reporter
if not layouts.uses_kit(layout):
return {}
kit = layouts.kit_folder(ctx.dest)
if layout.root.resolve() == kit.resolve(): # the kit's own rebuild: it is in place
return {}
new, old = kit.with_name(layouts.KIT + '.new'), kit.with_name(layouts.KIT + '.old')
try:
remove(new) # what a cancelled or failed run left
remove(old)
except OSError as e:
raise InstallError('kit.in_use', path=str(e.filename or old)) from None
files = []
for name in layouts.KIT_ITEMS:
src = layout.root / name
if src.is_dir():
files += [f for f in sorted(src.rglob('*')) if f.is_file() and '__pycache__' not in f.parts]
elif src.is_file():
files.append(src)
total, done = size_of(files), 0
rep.progress('kit', 'copy', 0, total, 'bytes', force=True)
try:
for f in files:
dst = new / f.relative_to(layout.root)
dst.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(f, dst)
done += f.stat().st_size
rep.progress('kit', 'copy', done, total, 'bytes')
except OSError as e:
raise InstallError('kit.copy_failed', path=str(e.filename or new), detail=str(e)) from None
# An open rebuild window, or a layout generator the game's settings menu started, holds files of the old kit:
# then nothing is replaced, and the message asks to close the game and run the setup again
moved = False
try:
if kit.exists():
os.replace(kit, old)
moved = True
os.replace(new, kit)
except OSError:
if moved and not kit.exists():
os.replace(old, kit) # the game keeps the kit it had
raise InstallError('kit.in_use', path=str(kit)) from None
try:
remove(old)
except OSError as e:
rep.log(f'kit: the old kit stays until the next run: {e}')
rep.log(f'kit: {len(files)} files, {total / GiB:.2f} GiB in {kit}')
return {'bytes': total}
def settle(self, ctx):
if not layouts.uses_kit(ctx.layout):
return
kit = layouts.kit_folder(ctx.dest).resolve()
if ctx.layout.root.resolve() != kit:
ctx.layout = layouts.release_layout(kit)
ctx.reporter.log(f'kit: the next steps run from {kit}')
def adopt_old_temp(layout, dest, temp):
"""An install by a setup before the build kit has its temp folder (generated C, lift cache, objects) in that
setup's folder: <setup>/temp/<install name>-<hash>, the setup build-cache/rebuild.txt names. The first run of a
setup with the build kit moves it into the install folder when both are on one disk, and the steps' keys decide
what of it still holds. On another disk the new temp folder starts empty: an update recompiles everything anyway
(the compiler's path is in every compile command), so a copy would save only part of the lift.
Runs before the run's log opens in temp; returns the line for the log, None when there was nothing to move."""
if not layouts.uses_kit(layout) or temp != dest / layouts.TEMP_IN_DEST or temp.exists():
return None
try:
lines = (dest / 'build-cache' / 'rebuild.txt').read_text(encoding='utf-8').splitlines()
except OSError:
return None
if len(lines) < 4 or lines[1] != '-c' or not os.path.isabs(lines[3]):
return None
old = layouts.temp_folder(Path(lines[3]) / 'temp', dest)
if not (old / 'gen').is_dir():
return None
try:
temp.parent.mkdir(parents=True, exist_ok=True)
os.replace(old, temp)
return f'temp: moved the temp folder of the setup in {lines[3]} to {temp}'
except OSError as e:
return f'temp: the temp folder {old} stays where it is ({e}); {temp} starts empty'
# ---- 2. fonts
class Fonts(Step):
@@ -1056,7 +1170,10 @@ def runtime_key(ctx):
"""Fingerprint of what totk.exe is built from besides the lifted code: the runtime, the toolchain and the build
tools. Shader preparation depends on these, not on the code cheats a lift adds."""
layout, cpu = ctx.layout, getattr(ctx, 'cpu', 'v3')
clang = subprocess.run([str(layout.clang), '--version'], capture_output=True, text=True).stdout
# without the lines that name clang's folder: the build kit of a moved install folder is the same toolchain
clang = '\n'.join(line for line in subprocess.run([str(layout.clang), '--version'], capture_output=True,
text=True).stdout.splitlines()
if not line.startswith(('InstalledDir:', 'Configuration file:')))
tools = [layout.tools / n for n in ('build_game.py', 'romfs_index.py', 'check_content.py', 'check_isa.py')]
return fingerprint('runtime-1', cpu, layout.runtime_lib_for(cpu), layout.runtime, layout.deps or 'system',
layout.sysroot or 'host', *tools, clang, *[d.name for d in layout.dlls])
@@ -1074,7 +1191,9 @@ def build_args(ctx, gen):
'--ninja', layout.ninja,
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
'--profile', dest / 'profile']
'--profile', dest / 'profile',
# the build kit and the temp folder move with the install folder: Ninja's paths relative to the kit
*(['--portable'] if layouts.uses_kit(layout) else [])]
class Compile(Step):
@@ -1450,10 +1569,11 @@ class Cleanup(Step):
return {}
# Puts the release folder (the argument after -c) on sys.path and runs installer.gui with the rest. `-m installer.gui`
# finds the package only through the release Python's ._pth: the Linux release's Python and the dev-mode embeddable one
# lack that entry, and totk.exe starts the command in the install folder.
REBUILD_BOOT = ('import runpy, sys; sys.path.insert(0, sys.argv.pop(1)); '
# Puts the release folder or the build kit (the argument after -c, made absolute: the kit's is relative to the install
# folder) on sys.path and runs installer.gui with the rest. `-m installer.gui` finds the package only through the
# release Python's ._pth: the Linux release's Python and the dev-mode embeddable one lack that entry, and totk.exe
# starts the command in the install folder.
REBUILD_BOOT = ('import os, runpy, sys; sys.path.insert(0, os.path.abspath(sys.argv.pop(1))); '
"runpy.run_module('installer.gui', run_name='__main__', alter_sys=True)")
@@ -1472,7 +1592,13 @@ def rebuild_command(ctx):
python = Path(sys.executable) if layout.dev else layout.python
if os.name == 'nt' and python.with_name('pythonw.exe').is_file():
python = python.with_name('pythonw.exe')
return [str(python), '-c', REBUILD_BOOT, str(layout.root), '--root', str(layout.root), '--rebuild', str(ctx.dest)]
root, dest = str(layout.root), str(ctx.dest)
if layouts.uses_kit(layout) and layout.root.resolve() == layouts.kit_folder(ctx.dest).resolve():
# the build kit: paths relative to the install folder, which totk.exe resolves against its own folder and
# starts the command in, so a moved install folder keeps its rebuild
python = Path(layouts.KIT) / python.relative_to(layout.root)
root, dest = layouts.KIT, '.'
return [str(python), '-c', REBUILD_BOOT, root, '--root', root, '--rebuild', dest]
def write_rebuild_command(ctx):
@@ -1500,4 +1626,4 @@ def romfs_problem(romfs, expected):
def all_steps():
return [Check(), Fonts(), Extract(), Lift(), Compile(), Prepare(), PcPrompts(), Shapes(), Dlss(), Cleanup()]
return [Check(), Kit(), Fonts(), Extract(), Lift(), Compile(), Prepare(), PcPrompts(), Shapes(), Dlss(), Cleanup()]
+4
View File
@@ -1,5 +1,6 @@
{
"step.check": "Checking the computer and the game files",
"step.kit": "Copying the build kit into the game folder",
"step.fonts": "Extracting the system fonts",
"step.extract": "Extracting the game",
"step.lift": "Translating the game code to C",
@@ -17,6 +18,7 @@
"phase.check.unpack_base": "Unpacking the compressed base game",
"phase.check.unpack_update": "Unpacking the compressed update",
"phase.check.versions": "Checking the game versions",
"phase.kit.copy": "Copying the tools the game rebuilds itself with for code mods",
"phase.fonts.extract": "Extracting the shared fonts from the firmware",
"phase.fonts.copy": "Copying the shared fonts into the install folder",
"phase.extract.hash_base": "Computing the checksum of the base game",
@@ -95,6 +97,8 @@
"lift.failed": "Translating the game code failed (exit code {code}). Details: {log}",
"compile.failed": "Compiling {game_exe} failed (exit code {code}). Details: {log}",
"compile.exe_in_use": "{path} is in use. Close the game and run the installer again.",
"kit.in_use": "{path} is in use. Close the game and any open setup or rebuild window, then run the installer again.",
"kit.copy_failed": "The build kit cannot be copied to {path}: {detail}",
"compile.exe_locked": "{path} cannot be written: another program holds it, a game started from that folder or an antivirus that flagged the file. Close the game, add the setup's folder and the install folder to the antivirus exclusions, restart Windows, then run the installer again. Details: {log}",
"prepare.xlate_in_use": "{path} is in use. Close the game, end any totk_xlate process left from an earlier run, then run the installer again.",
"prepare.shader_list_missing": "The shader list is missing.",
+4
View File
@@ -1,5 +1,6 @@
{
"step.check": "Verificando o computador e os arquivos do jogo",
"step.kit": "Copiando o kit de compilação para a pasta do jogo",
"step.fonts": "Extraindo as fontes do sistema",
"step.extract": "Extraindo o jogo",
"step.lift": "Traduzindo o código do jogo para C",
@@ -17,6 +18,7 @@
"phase.check.unpack_base": "Descompactando o jogo base compactado",
"phase.check.unpack_update": "Descompactando a atualização compactada",
"phase.check.versions": "Verificando as versões do jogo",
"phase.kit.copy": "Copiando as ferramentas com que o jogo se recompila para mods de código",
"phase.fonts.extract": "Extraindo as fontes compartilhadas do firmware",
"phase.fonts.copy": "Copiando as fontes compartilhadas para a pasta de instalação",
"phase.extract.hash_base": "Calculando o checksum do jogo base",
@@ -95,6 +97,8 @@
"lift.failed": "Falha ao traduzir o código do jogo (código de saída {code}). Detalhes: {log}",
"compile.failed": "Falha ao compilar o {game_exe} (código de saída {code}). Detalhes: {log}",
"compile.exe_in_use": "{path} está em uso. Feche o jogo e execute o instalador de novo.",
"kit.in_use": "{path} está em uso. Feche o jogo e qualquer janela de instalação ou recompilação aberta, depois execute o instalador de novo.",
"kit.copy_failed": "Não foi possível copiar o kit de compilação para {path}: {detail}",
"compile.exe_locked": "Não é possível gravar {path}: outro programa o está usando, um jogo aberto dessa pasta ou um antivírus que marcou o arquivo. Feche o jogo, adicione a pasta do instalador e a pasta de instalação às exceções do antivírus, reinicie o Windows e execute o instalador de novo. Detalhes: {log}",
"prepare.xlate_in_use": "{path} está em uso. Feche o jogo, encerre qualquer processo totk_xlate que tenha sobrado de uma execução anterior e execute o instalador de novo.",
"prepare.shader_list_missing": "A lista de shaders está faltando.",
+4
View File
@@ -1,5 +1,6 @@
{
"step.check": "Проверка компьютера и игровых файлов",
"step.kit": "Копирование набора сборки в папку игры",
"step.fonts": "Извлечение системных шрифтов",
"step.extract": "Распаковка игры",
"step.lift": "Перевод кода игры в C",
@@ -17,6 +18,7 @@
"phase.check.unpack_base": "Распаковка сжатой базовой игры",
"phase.check.unpack_update": "Распаковка сжатого обновления",
"phase.check.versions": "Проверка версий игры",
"phase.kit.copy": "Копирование инструментов, которыми игра пересобирает себя для модов с кодом",
"phase.fonts.extract": "Извлечение общих шрифтов из прошивки",
"phase.fonts.copy": "Копирование общих шрифтов в папку установки",
"phase.extract.hash_base": "Подсчёт контрольной суммы базовой игры",
@@ -95,6 +97,8 @@
"lift.failed": "Не удалось перевести код игры (код выхода {code}). Подробности: {log}",
"compile.failed": "Не удалось собрать {game_exe} (код выхода {code}). Подробности: {log}",
"compile.exe_in_use": "Файл {path} занят. Закройте игру и запустите установку снова.",
"kit.in_use": "{path} занят другой программой. Закройте игру и все окна установки или пересборки, затем запустите установку снова.",
"kit.copy_failed": "Не удалось скопировать набор сборки в {path}: {detail}",
"compile.exe_locked": "Файл {path} не удаётся записать: его держит другая программа — игра, запущенная из этой папки, или антивирус, который пометил файл. Закройте игру, добавьте папку установщика и папку установки в исключения антивируса, перезагрузите Windows и запустите установку снова. Подробности: {log}",
"prepare.xlate_in_use": "Файл {path} занят. Закройте игру, завершите процесс totk_xlate, оставшийся от прошлого запуска, и запустите установку снова.",
"prepare.shader_list_missing": "Нет списка шейдеров.",
+4
View File
@@ -1,5 +1,6 @@
{
"step.check": "检查电脑和游戏文件",
"step.kit": "正在把构建工具包复制到游戏文件夹",
"step.fonts": "正在提取系统字体",
"step.extract": "解包游戏",
"step.lift": "将游戏代码转换为 C 代码",
@@ -17,6 +18,7 @@
"phase.check.unpack_base": "正在解压压缩的游戏本体",
"phase.check.unpack_update": "正在解压压缩的更新",
"phase.check.versions": "检查游戏版本",
"phase.kit.copy": "正在复制游戏为代码模组重新构建自身所用的工具",
"phase.fonts.extract": "正在从固件中提取共享字体",
"phase.fonts.copy": "正在将共享字体复制到安装文件夹",
"phase.extract.hash_base": "计算游戏本体的校验和",
@@ -95,6 +97,8 @@
"lift.failed": "游戏代码转换失败(退出代码 {code})。详情:{log}",
"compile.failed": "编译 {game_exe} 失败(退出代码 {code})。详情:{log}",
"compile.exe_in_use": "{path} 正在使用中。请关闭游戏后重新运行安装程序。",
"kit.in_use": "{path} 正在被占用。请关闭游戏以及所有打开的安装或重新构建窗口,再重新运行安装程序。",
"kit.copy_failed": "无法把构建工具包复制到 {path}:{detail}",
"compile.exe_locked": "无法写入 {path}:另一个程序正在占用它,可能是从该文件夹启动的游戏,或标记了该文件的杀毒软件。请关闭游戏,将安装程序文件夹和安装文件夹加入杀毒软件的排除项,重启 Windows,然后重新运行安装程序。详情:{log}",
"prepare.xlate_in_use": "{path} 正在被占用。请关闭游戏,结束上次残留的 totk_xlate 进程,再重新运行安装程序。",
"prepare.shader_list_missing": "缺少着色器列表。",
+9 -1
View File
@@ -516,7 +516,10 @@ static void plain_name(char *s) {
// ---- rebuild: code cheats and ExeFS patches the build lacks
// The setup writes build-cache\rebuild.txt (installer/steps.py write_rebuild_command), the command that starts its
// window in rebuild mode on this install folder, one argument per line. When an enabled mod has code cheats this build
// window in rebuild mode on this install folder, one argument per line. The build kit's command (installer/layout.py)
// is relative to the install folder: its program is taken from this exe's folder, and the command runs with that
// folder as its working folder, against which the setup resolves its other arguments. When an enabled mod has code
// cheats this build
// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), or the build has a mod plugin whose file is
// gone (ldr_plugins_gone: its mod was removed), the game runs it and exits; the setup lifts and compiles them in (or the
// plugin out) and starts the game again.
@@ -555,6 +558,7 @@ static void rebuild_if_needed(const char *dir) {
return;
}
const char *argv[32];
char program[PLAT_PATH_MAX + 64];
int argc = 0;
for (char *line = cmd; line && *line && argc < 31;) {
char *end = strchr(line, '\n'), *next = end ? end + 1 : NULL;
@@ -565,6 +569,10 @@ static void rebuild_if_needed(const char *dir) {
line = next;
}
argv[argc] = NULL;
// CreateProcess resolves a relative program against this process's working folder, which a shortcut may set
if (argc && !plat_path_absolute(argv[0]) &&
snprintf(program, sizeof program, "%s%s", dir, argv[0]) < (int)sizeof program)
argv[0] = program;
if (argc < 2 || !plat_file_write(skip_path, want, strlen(want))) {
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; cannot start the "
"rebuild (%s)", need, patches, cmd_path);
+11
View File
@@ -379,6 +379,17 @@ static inline char *plat_last_sep(const char *path) {
if (*p == '/' || *p == '\\') last = (char *)p;
return last;
}
// 1 when path does not depend on the working directory: it starts with a separator (a root or UNC path) or, on
// Windows, with a drive letter and a separator.
static inline int plat_path_absolute(const char *path) {
if (path[0] == '/' || path[0] == '\\') return 1;
#ifdef _WIN32
char c = path[0];
return ((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z')) && path[1] == ':' && (path[2] == '/' || path[2] == '\\');
#else
return 0;
#endif
}
// Environment variable as UTF-8 into out; its length, or 0 when unset, empty or too long for cap.
size_t plat_getenv(const char *name, char *out, size_t cap);
// The user's UI language as a BCP-47 tag ("ru-RU", "en-US"); "" when the system gives none. Valid for the process.
+8 -3
View File
@@ -1146,7 +1146,8 @@ int romfs_shaped_prompts_ready(const char *folder, unsigned w, unsigned h) { ret
// The setup writes the shaped folders for the displays connected when it ran (installer/steps.py Shapes). After a
// monitor change the menu can start the same generator for the shape of the game resolution: the setup's Python and
// tools/make_romfs_patches.py, which build-cache/rebuild.txt names (installer/steps.py write_rebuild_command: line 1 the
// Python, pythonw on Windows, so no console opens; line 4 the setup's folder), on the install folder's romfs/. Child
// Python, pythonw on Windows, so no console opens; line 4 the setup's folder; both relative to the install folder for
// the build kit, installer/layout.py), on the install folder's romfs/. Child
// processes the game does not wait for: the generator writes manifest.tsv last, so a folder it has not finished is not
// taken, and the next start loads the finished one.
static int prepare_tools(char *python, size_t pcap, char *script, size_t scap, char *dir, size_t dcap) {
@@ -1166,8 +1167,12 @@ static int prepare_tools(char *python, size_t pcap, char *script, size_t scap, c
lines[k++] = p;
p = e + 1;
}
int ok = k >= 4 && !strcmp(lines[1], "-c") && snprintf(python, pcap, "%s", lines[0]) < (int)pcap &&
snprintf(script, scap, "%s/tools/make_romfs_patches.py", lines[3]) < (int)scap;
int ok = k >= 4 && !strcmp(lines[1], "-c");
if (ok) { // the build kit's lines are relative to the install folder
const char *at0 = plat_path_absolute(lines[0]) ? "" : dir, *at3 = plat_path_absolute(lines[3]) ? "" : dir;
ok = snprintf(python, pcap, "%s%s%s", at0, *at0 ? "/" : "", lines[0]) < (int)pcap &&
snprintf(script, scap, "%s%s%s/tools/make_romfs_patches.py", at3, *at3 ? "/" : "", lines[3]) < (int)scap;
}
free(text);
snprintf(path, sizeof path, "%s/romfs/romfs.bin", dir);
return ok && plat_file_exists(python) && plat_file_exists(script) && plat_file_exists(path);
+136 -3
View File
@@ -18,6 +18,7 @@ ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
from installer import ini, runner, screens, state, steps # noqa: E402
from installer.events import Reporter # noqa: E402
from installer import layout as layouts # noqa: E402
from installer.layout import Layout # noqa: E402
@@ -326,7 +327,7 @@ class LiftRunTest(unittest.TestCase):
self.dest, temp = base / 'install', base / 'temp'
self.dest.mkdir()
temp.mkdir()
layout = SimpleNamespace(recomp=ROOT / 'recomp', tools=ROOT / 'tools',
layout = SimpleNamespace(dev=True, recomp=ROOT / 'recomp', tools=ROOT / 'tools',
patches=lambda version: base / 'patches-v143.txt')
self.ctx = SimpleNamespace(dest=self.dest, temp=temp, layout=layout, keys={'lift': 'k'},
state={'version': '1.4.3'}, reporter=Reporter(out=io.StringIO()))
@@ -477,7 +478,7 @@ class CompileLockedTest(unittest.TestCase):
dest, temp = base / 'install', base / 'temp'
dest.mkdir()
temp.mkdir()
layout = SimpleNamespace(tools=ROOT / 'tools', llvm=base / 'llvm', deps=None, sysroot=None, runtime=base,
layout = SimpleNamespace(dev=True, tools=ROOT / 'tools', llvm=base / 'llvm', deps=None, sysroot=None, runtime=base,
ninja=base / 'ninja', dlls=[], runtime_lib_for=lambda cpu: None,
pgo=lambda version: None, expected=lambda version: base / 'expected.json')
self.ctx = SimpleNamespace(dest=dest, temp=temp, layout=layout, state={'version': '1.4.3'},
@@ -634,6 +635,137 @@ class CleanupTest(unittest.TestCase):
self.assertEqual(self.run_cleanup(), 'cleanup.romfs_unreadable')
def fake_release(root, version='1.0+abc'):
"""A folder release_layout reads as a release: VERSION, the Python file and one file per other kit item."""
for name in layouts.KIT_ITEMS:
if '.' in name or name == 'VERSION':
(root / name).parent.mkdir(parents=True, exist_ok=True)
(root / name).write_text(version + '\n' if name == 'VERSION' else name)
else:
(root / name).mkdir(parents=True, exist_ok=True)
(root / name / 'f.txt').write_text(name)
layouts.release_python(root).parent.mkdir(parents=True, exist_ok=True)
layouts.release_python(root).write_bytes(b'')
(root / 'TotK-Setup.exe').write_bytes(b'launcher')
(root / 'installer' / '__pycache__').mkdir()
(root / 'installer' / '__pycache__' / 'x.pyc').write_bytes(b'')
return layouts.release_layout(root)
@unittest.skipUnless(steps.os.name == 'nt', 'the build kit is Windows-only')
class KitTest(unittest.TestCase):
"""The build kit: the setup's release folder copied into the install folder, the later steps and the rebuild
command pointed at it, and an older setup's temp folder taken over."""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
base = Path(self.tmp.name)
self.setup, self.dest = base / 'TotK-Setup', base / 'game'
self.dest.mkdir()
self.layout = fake_release(self.setup)
self.missing = mock.patch.object(Layout, 'missing', lambda self, cpu='v3': [])
self.missing.start()
def tearDown(self):
self.missing.stop()
self.tmp.cleanup()
def ctx(self, layout):
return SimpleNamespace(layout=layout, dest=self.dest, reporter=Reporter(out=io.StringIO()))
def install(self):
ctx = self.ctx(self.layout)
kit = steps.Kit()
self.assertFalse(kit.current(ctx, {}))
kit.run(ctx)
kit.settle(ctx)
return ctx, kit
def test_copies_the_release_and_the_next_steps_run_from_it(self):
ctx, kit = self.install()
folder = self.dest / 'build-kit'
self.assertEqual(ctx.layout.root, folder.resolve())
self.assertEqual(ctx.layout.release, '1.0+abc')
self.assertEqual(sorted(p.name for p in folder.iterdir()), sorted(layouts.KIT_ITEMS))
self.assertFalse((folder / 'installer' / '__pycache__').exists())
self.assertFalse((self.dest / 'build-kit.new').exists() or (self.dest / 'build-kit.old').exists())
self.assertTrue(kit.current(self.ctx(self.layout), {}))
def test_the_kits_own_rebuild_keeps_it(self):
ctx, kit = self.install()
(self.dest / 'build-kit' / 'tools' / 'mark').write_text('kept')
self.assertTrue(kit.current(ctx, {}))
kit.run(ctx)
kit.settle(ctx)
self.assertEqual((self.dest / 'build-kit' / 'tools' / 'mark').read_text(), 'kept')
def test_a_newer_setup_replaces_the_kit(self):
self.install()
(self.dest / 'build-kit' / 'tools' / 'old-only').write_text('old')
self.layout = fake_release(Path(self.tmp.name) / 'TotK-Setup-2', '1.1+def')
ctx, _ = self.install()
self.assertEqual(ctx.layout.release, '1.1+def')
self.assertFalse((self.dest / 'build-kit' / 'tools' / 'old-only').exists())
def test_rebuild_command_is_relative_to_the_install_folder(self):
ctx, _ = self.install()
cmd = steps.rebuild_command(ctx)
self.assertEqual(cmd, [str(Path('build-kit', 'python', 'pythonw.exe' if (
ctx.layout.python.with_name('pythonw.exe')).is_file() else 'python.exe')), '-c', steps.REBUILD_BOOT,
'build-kit', '--root', 'build-kit', '--rebuild', '.'])
self.assertFalse(any(str(self.dest) in a or str(self.setup) in a for a in cmd))
def test_the_kits_builds_are_portable(self):
ctx, _ = self.install()
ctx.temp, ctx.state = self.dest / 'build-cache' / 'work', {'version': '1.4.3'}
self.assertIn('--portable', steps.build_args(ctx, ctx.temp / 'gen'))
ctx.layout = Layout(root=ROOT, dev=True, release='x', python=Path('python'), tools=Path('tools'),
recomp=Path('recomp'), llvm=Path('llvm'), deps=None, runtime=Path('rt'),
runtime_lib=Path('lib'), ninja=Path('ninja'), hactool=Path('hactool'), xlate=Path('xlate'))
self.assertNotIn('--portable', steps.build_args(ctx, ctx.temp / 'gen'))
def test_dev_mode_has_no_kit(self):
ctx = self.ctx(SimpleNamespace(dev=True, release='x', root=ROOT))
kit = steps.Kit()
self.assertTrue(kit.current(ctx, {}))
self.assertEqual(kit.run(ctx), {})
kit.settle(ctx)
self.assertFalse((self.dest / 'build-kit').exists())
def test_temp_folder_of_an_older_setup_moves_in(self):
old = layouts.temp_folder(self.setup / 'temp', self.dest)
(old / 'gen').mkdir(parents=True)
(old / 'gen' / '.install-key').write_text('k')
(self.dest / 'build-cache').mkdir()
(self.dest / 'build-cache' / 'rebuild.txt').write_text(
'\n'.join(['py', '-c', 'boot', str(self.setup), '--root', str(self.setup), '--rebuild', str(self.dest)]))
temp = layouts.temp_for(self.layout, self.dest)
self.assertEqual(temp, self.dest / 'build-cache' / 'work')
self.assertIn('moved', steps.adopt_old_temp(self.layout, self.dest, temp))
self.assertEqual((temp / 'gen' / '.install-key').read_text(), 'k')
self.assertFalse(old.exists())
self.assertIsNone(steps.adopt_old_temp(self.layout, self.dest, temp)) # once: the new one exists now
def test_no_temp_folder_to_take_over(self):
temp = layouts.temp_for(self.layout, self.dest)
self.assertIsNone(steps.adopt_old_temp(self.layout, self.dest, temp))
(self.dest / 'build-cache').mkdir()
(self.dest / 'build-cache' / 'rebuild.txt').write_text('build-kit\\python\\pythonw.exe\n-c\nboot\nbuild-kit\n')
self.assertIsNone(steps.adopt_old_temp(self.layout, self.dest, temp))
class RuntimeKeyTest(unittest.TestCase):
def test_clangs_folder_is_not_in_the_key(self):
layout = SimpleNamespace(clang=Path('clang'), tools=Path('tools'), runtime=Path('rt'), deps=None, sysroot=None,
dlls=(), runtime_lib_for=lambda cpu: None)
keys = []
for where in ('C:/setup/llvm/bin', 'D:/game/build-kit/llvm/bin'):
out = f'clang version 23\nTarget: x86_64-w64-windows-gnu\nInstalledDir: {where}\nConfiguration file: {where}/x.cfg\n'
with mock.patch.object(steps.subprocess, 'run', return_value=SimpleNamespace(stdout=out)):
keys.append(steps.runtime_key(SimpleNamespace(layout=layout)))
self.assertEqual(keys[0], keys[1])
class FreeSpaceTest(unittest.TestCase):
def test_missing_folders_use_the_nearest_existing_parent(self):
with tempfile.TemporaryDirectory() as d:
@@ -806,7 +938,8 @@ class FontsTest(unittest.TestCase):
def test_all_steps_order(self):
self.assertEqual([s.id for s in steps.all_steps()],
['check', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss', 'cleanup'])
['check', 'kit', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss',
'cleanup'])
def test_extracts_into_the_install_folder(self):
self.fake_tool()
+39 -4
View File
@@ -44,6 +44,9 @@ big machine does not run out of memory on the largest chunks. Each phase reports
of its processes (Windows job object accounting: committed private bytes).
--ninja PATH names the Ninja to run; without MSYS2 the bin/ninja.exe of the release folder (tools/make_release.py) is used.
--portable (the setup's builds, installer/steps.py build_args) writes every path on the tools' drive relative to the
folder above tools/, through '..' where needed, so the install folder that holds the build kit, the generated C and the
build folder can move and the next build stays incremental (PORTABLE).
--toolchain DIR builds with an llvm-mingw tree (DIR/bin/clang.exe) instead of MSYS2 UCRT64 clang. Ninja still comes
from MSYS2. The libraries and headers MSYS2 supplies (Vulkan, SDL3, Opus, libvpx, VMA) then come from --deps DIR
(default toolchain/deps-mingw: include/, lib/ with the import libraries and libvpx.a, bin/ with the DLLs copied next to
@@ -117,6 +120,13 @@ def set_cpu(cpu):
# keeps the names equal between the training build and a build from another tree (a worktree, an install).
PGO_NAMES = ['-mllvm', '-static-func-full-module-prefix=false']
PGO_INPUT = None # the .profdata of --pgo FILE: every compile depends on it, so a new profile rebuilds them
# --portable (the setup's builds in an install folder, installer/steps.py build_args): build.ninja names every path on
# ROOT's drive relative to ROOT, the compiler too. clang names its own headers by its absolute folder whatever argv[0]
# says, so with --toolchain the compiles also get -resource-dir and --sysroot relative to ROOT (the folders clang would
# take itself); the Ninja deps log then holds no absolute path. Ninja runs in ROOT, so a folder that holds ROOT, the
# generated C and the build folder (the install folder with build-kit/ and build-cache/work/) can move without a full
# recompile.
PORTABLE = False
def set_pgo(pgo, out):
@@ -210,14 +220,27 @@ MEM_BUDGET = 50 * GiB # total for all compile jobs
def below_root(p):
"""Resolved p relative to ROOT, else to MAIN_ROOT (the same relative path through the worktree's junctions), else
absolute."""
absolute; with --portable relative to ROOT through '..' when it is on ROOT's drive."""
p = Path(p).resolve()
for root in (ROOT, MAIN_ROOT):
if p.is_relative_to(root):
return p.relative_to(root)
if PORTABLE:
try:
return Path(os.path.relpath(p, ROOT))
except ValueError: # another drive
pass
return p
def tool_arg(p):
"""A program for a Ninja command line: relative like the other paths with --portable (with backslashes on Windows:
CreateProcess does not find a relative program named with forward slashes), else absolute."""
if not PORTABLE:
return cmd_arg(p.as_posix())
return cmd_arg(str(below_root(p)) if WIN else below_root(p).as_posix())
def ninja_path(p):
"""A path for a build line: relative to the repo root where possible, forward slashes, Ninja-escaped."""
return below_root(p).as_posix().replace('$', '$$').replace(' ', '$ ').replace(':', '$:')
@@ -434,7 +457,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
'# Written by tools/build_game.py; do not edit.',
'ninja_required_version = 1.10',
f'builddir = {ninja_path(out)}',
f'cc = {cache}{cmd_arg(CLANG.as_posix())}',
f'cc = {cache}{tool_arg(CLANG)}',
f'genflags = {" ".join(GEN_FLAGS)} {inc}',
f'rtflags = {" ".join(RT_FLAGS)} {rt_inc}',
f'ldflags = {" ".join(LINK_FLAGS)}',
@@ -456,7 +479,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
' description = CC $in',
'',
'rule rc',
f' command = {cmd_arg(WINDRES.as_posix())} -I {cmd_path(RUNTIME)} -O coff $in -o $out',
f' command = {tool_arg(WINDRES)} -I {cmd_path(RUNTIME)} -O coff $in -o $out',
' description = RC $in',
'',
'rule link',
@@ -498,7 +521,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
exe = out / EXE
if make_lib:
exe = out / LIB_NAME
lines += ['', 'rule ar', f' command = {cmd_arg(AR.as_posix())} rcs $out $in', ' description = AR $out', '', f'build {ninja_path(exe)}: ar ' + ' '.join(ninja_path(o) for o in every),
lines += ['', 'rule ar', f' command = {tool_arg(AR)} rcs $out $in', ' description = AR $out', '', f'build {ninja_path(exe)}: ar ' + ' '.join(ninja_path(o) for o in every),
'build objects: phony ' + ' '.join(ninja_path(o) for o in every), f'default {ninja_path(exe)}', '']
return '\n'.join(lines), objs, exe
rule = 'link'
@@ -862,8 +885,14 @@ def main():
help='compile the generated code at -O2 without a profile (the --pgo level), for A/B builds')
ap.add_argument('--expected', type=Path,
help='expected hashes (default tools/data/expected-<display version>.json)')
ap.add_argument('--portable', action='store_true',
help='paths in build.ninja relative to the tools\' folder, also through .. (an install folder that '
'moves with its build kit and build folder)')
args = ap.parse_args()
set_cpu(args.cpu)
if args.portable:
global PORTABLE
PORTABLE = True
gen = (args.gen or ROOT / 'generated' / args.version).resolve()
out = (args.out or ROOT / 'build' / args.version).resolve()
@@ -902,6 +931,12 @@ def main():
CLANG = TOOLCHAIN_BIN / f'clang{suffix}'
AR = TOOLCHAIN_BIN / f'llvm-ar{suffix}'
WINDRES = TOOLCHAIN_BIN / 'llvm-windres.exe'
if PORTABLE and WIN:
resource = sorted((TOOLCHAIN_BIN.parent / 'lib' / 'clang').glob('*'))
if len(resource) != 1:
sys.exit(f'--portable: {TOOLCHAIN_BIN.parent / "lib" / "clang"} holds {len(resource)} version folders')
for flags in (GEN_FLAGS, RT_FLAGS):
flags += ['-resource-dir', cmd_path(resource[0]), f'--sysroot={cmd_path(TOOLCHAIN_BIN.parent)}']
if args.toolchain and WIN or args.deps and (args.toolchain or not WIN): # Linux: --deps only, never deps-mingw
DEPS = (args.deps or DEFAULT_DEPS).resolve()
dlls = sorted((DEPS / 'bin').glob('*.dll')) if WIN else []
+1
View File
@@ -19,6 +19,7 @@ cached in toolchain/release-dl/ and checked against the sha256 in PINS; a mismat
install compiles that update's code and the runtime libraries with it
VERSION the repository's VERSION plus the commit, 0.1.0+973c626; the installer's release string
TotK-Setup.exe installer/launcher/launcher.c built with llvm/ (-mwindows): starts python/pythonw.exe -m installer.gui
Everything else is the build kit the setup copies into the install folder (installer/layout.py KIT_ITEMS)
--prove compiles the generated C and links build/release-check/totk.exe with nothing but the folder's python/, llvm/,
runtime/ and bin/ (PATH holds the Windows system directory and those), then runs the four probes there.