Merge the installer: setup steps, window, launcher and release bundle

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-10-24 10:04:19 +03:00
commit 3505258a06
26 files changed
+4178 -35

No files matched your search

+22 -2
View File
@@ -19,7 +19,7 @@ Code copied or adapted from another project is listed here with its source, the
| deko3d block-linear image layout, as ported in MacNativeGames' `nvn_tex.c` (Apple parts dropped, layout math unchanged; licence text in `games/totk/third_party/d2r_gfx/LICENSE.deko3d`, reproduced in the file header) | https://github.com/devkitPro/deko3d (`source/dk_image.cpp`), via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/nvn_tex.c`, `nvn_tex.h`), Copyright (C) 2018-2020 fincs | zlib | M4 task T3b: `runtime/nvn_tex.c`, `runtime/nvn_tex.h` |
| MacNativeGames texture auto-layout goldens (40 original-SDK size and block-height cases) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/tests/texture_auto_layout/probe.c`), Copyright (c) 2026 jhaizhou | MIT | M4 task T3b: `runtime/nvn_tex_probe.c`; T3f: `runtime/nvn_probe.c` (four of the cases, through the procs) |
| MacNativeGames block-linear deswizzler (address function; copy loop split out as `tex_bl_copy`) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/tex/nvn_deswizzle.c`), Copyright (c) 2026 jhaizhou | MIT | M4 task T5a: `runtime/tex_deswizzle.c`, `runtime/tex.h` |
| SDL 3.4.18 (MSYS2 `mingw-w64-ucrt-x86_64-SDL3`), dynamic link, `SDL3.dll` and its dependency `libiconv-2.dll` (LGPL-2.1, unmodified, not linked into the exe) copied next to `totk.exe` | https://libsdl.org, Copyright (c) 1997-2026 Sam Lantinga | zlib | `runtime/input_win.c` (gamepad input, rumble, gyro and accelerometer) |
| SDL 3.4.18, dynamic link, `SDL3.dll` copied next to `totk.exe`. Developer builds use the MSYS2 build (`mingw-w64-ucrt-x86_64-sdl3`, which imports `libiconv-2.dll`, LGPL-2.1, unmodified, copied beside it); the installer release ships the upstream release `SDL3-3.4.18-win32-x64.zip`, which imports no libiconv | https://libsdl.org, Copyright (c) 1997-2026 Sam Lantinga | zlib | `runtime/input_win.c` (gamepad input, rumble, gyro and accelerometer) |
| libopus (MSYS2 `mingw-w64-ucrt-x86_64-opus`), dynamic link, `libopus-0.dll` copied next to `totk.exe` | https://opus-codec.org, Copyright (c) 2001-2023 Xiph.Org, Skype Limited, Octasic, Jean-Marc Valin, Timothy B. Terriberry, CSIRO, Gregory Maxwell, Mark Borgerding, Erik de Castro Lopo, Mozilla, Amazon | BSD-3-Clause | `runtime/ov_media.c` (nn::codec hardware Opus decoder) |
| Vulkan Memory Allocator 3.4.0 (MSYS2 `mingw-w64-ucrt-x86_64-vulkan-memory-allocator`), header-only, compiled into `totk.exe` | https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator, Copyright (c) 2017-2026 Advanced Micro Devices, Inc. | MIT | `runtime/vk_vma.cpp` (the allocator behind `runtime/vk_memory.c`) |
| MacNativeGames host movie decoder (`movie::Decoder` for VP9 over libvpx: entry list, buffer and event semantics, NV12 output format, PTS pacing) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/runtime/movie_host.c`, `movie_host.h`), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/ov_movie.c`, `runtime/movie.h` |
@@ -31,6 +31,26 @@ These tools run on the developer's or player's machine during the build. They ar
| Tool | Source | License | Purpose |
|---|---|---|---|
| hactool | https://github.com/SciresM/hactool @ `1d64a83450e025622f3468c28fc4164dad2c5ef6`, with patches from MacNativeGames | ISC | NCA and update container extraction |
| hactool (the installer release ships it, see below) | https://github.com/SciresM/hactool @ `1d64a83450e025622f3468c28fc4164dad2c5ef6`, with patches from MacNativeGames | ISC | NCA and update container extraction |
| Capstone | https://www.capstone-engine.org | BSD-3-Clause | ARM64 disassembly for analysis |
| Unicorn | https://www.unicorn-engine.org | GPL-2.0 (Python bindings BSD) | Differential testing of the lifter; never linked into the game |
## Distributed in the installer release
`tools/make_release.py` assembles these into the release folder of `docs/installer.md`. Versions and sha256 of every download are pinned in that script; licence texts are in the folder's `licenses/` and in each component's own folder.
| Component | Source | License | Where in the release |
|---|---|---|---|
| CPython 3.14.9 embeddable package, with `tkinter` and Tcl/Tk 9.0.4 taken from the python.org 3.14.9 installer's component MSI (unpacked, never run); includes `vcruntime140*.dll` (Microsoft, redistributable with Python), OpenSSL, libffi, SQLite, zlib, libtommath | https://www.python.org/downloads/release/python-3149/ | PSF-2.0 (Tcl/Tk: Tcl/Tk licence, BSD-style) | `python/` |
| numpy 2.5.4 (with its bundled OpenBLAS), lz4 4.4.5, cryptography 50.0.2 (with its bundled OpenSSL), cffi 2.1.1, pycparser 3.11 | PyPI binary wheels, pins in `tools/requirements-installer.txt` | BSD-3-Clause; lz4 BSD-3-Clause; cryptography Apache-2.0 OR BSD-3-Clause; cffi MIT; pycparser BSD-3-Clause | `python/Lib/site-packages/` |
| llvm-mingw 20261020 (LLVM/Clang/LLD 23, mingw-w64, libc++, libunwind, compiler-rt), x86_64 subset | https://github.com/mstorsjo/llvm-mingw/releases/tag/20261020 | Apache-2.0 with LLVM exception; mingw-w64 ZPL-2.1/BSD/public domain; libc++ and libunwind Apache-2.0 with LLVM exception and MIT/UIUC; winpthreads MIT | `llvm/` |
| SDL 3.4.18 upstream Windows release (`SDL3.dll`) and mingw development package (headers, `libSDL3.dll.a`) | https://github.com/libsdl-org/SDL/releases/tag/release-3.4.18 | zlib | `bin/SDL3.dll`, `runtime/deps/` |
| libopus 1.6.1 (MSYS2 `mingw-w64-ucrt-x86_64-opus` 1.6.1-2; `libopus-0.dll` imports only KERNEL32 and the UCRT api-ms-win-crt DLLs), headers and import library | https://repo.msys2.org/mingw/ucrt64/ | BSD-3-Clause | `bin/libopus-0.dll`, `runtime/deps/` |
| libvpx 1.17.0 (MSYS2 `mingw-w64-ucrt-x86_64-libvpx` 1.17.0-1), headers and static library compiled into `totk.exe`; `libvpx-LICENSE.txt` and `libvpx-PATENTS.txt` come from the v1.17.0 source tag | https://github.com/webmproject/libvpx | BSD-3-Clause with the WebM patent grant | `runtime/deps/`, `licenses/` |
| Vulkan-Headers and Vulkan-Loader 1.4.363.0 (MSYS2 packages; headers and `libvulkan-1.dll.a` only, no loader DLL) | https://github.com/KhronosGroup | Apache-2.0 OR MIT | `runtime/deps/` |
| Vulkan Memory Allocator 3.4.0 (MSYS2 package), header | https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator | MIT | `runtime/deps/` |
| Ninja 1.13.2 (`ninja-win.zip`) | https://github.com/ninja-build/ninja/releases/tag/v1.13.2 | Apache-2.0 | `bin/ninja.exe` |
| hactool (SciresM @ `1d64a83450e025622f3468c28fc4164dad2c5ef6` plus MacNativeGames' `portable-patches.diff`: `MNG_TITLE_KEYS` and a relaxed BKTR size check), built with MSYS2 UCRT64 clang; imports only KERNEL32 and the UCRT | https://github.com/SciresM/hactool | ISC | `bin/hactool.exe` |
| `totk_xlate.exe` (Ryujinx shader translator, NativeAOT build of `tools/xlate/`, see the Ryujinx row above) | built by `tools/build_xlate.py` | MIT | `bin/totk_xlate.exe` |
| MacNativeGames shader variant list | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/assets/shader_list.tsv`), Copyright (c) 2026 MacNativeGames contributors | MIT | `data/shader_list.tsv` |
+113
View File
@@ -0,0 +1,113 @@
# Installer
A public release that turns the player's own game files into a playable TotK folder. The release carries no game
data and no code derived from the game: the installer builds the game on the player's machine. The installed folder
holds only the game; the installer and its tools stay outside it.
## What the player does
1. Downloads `TotK-Setup-<version>-windows.zip` from GitHub Releases, unpacks it anywhere and starts `TotK-Setup.exe`.
2. Picks:
- the base game (XCI or NSP) and the 1.2.1 update (NSP);
- their `prod.keys`;
- the game language and the install folder.
3. Waits for the setup, one time. The window shows each step and its progress.
4. Plays `totk.exe` from the install folder. The installer and the source NSP or XCI can be deleted.
The install folder (`main.c` `portable_paths()` reads this layout; it stays portable):
```
<install folder>/
totk.exe, SDL3.dll, libopus-0.dll, totk_xlate.exe
totk.ini language; the other keys as commented defaults (totk.ini.example)
exefs/, romfs/ extracted from the player's files
profile/ saves
shader and pipeline caches prepared by the setup
totk-install.json what built this folder: release version, input hashes, finished steps
```
## Updating the port
The player runs a newer installer on the same folder. It reads `totk-install.json`, keeps `romfs/`, `exefs/`,
`profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the shaders. A newer
release that changes nothing game-derived skips the rebuild.
## Setup steps
1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 (by hash, as
`tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a
first install, 20 GiB after it), the CPU has
AVX2 (`-march=x86-64-v3`), a Vulkan 1.3 GPU exists. Each failure is one clear message.
2. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
3. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt`) into a temp folder next to the installer.
4. **Compile** the generated C, `modules.c` and `romfs.c` (they bake per-install layout and RomFS hashes) and link
them with the prebuilt runtime library into `totk.exe` (`build_game.py --runtime-lib`, proven on task/llvmmingw:
3 min on 24 cores).
5. **Prepare** the RomFS patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`,
`totk.exe --prepare-shaders`) in the install folder.
6. **Clean up** the temp folder (generated C and objects, about 1.3 GB).
A cancelled or failed setup resumes from the last finished step (`totk-install.json`).
## Command line
`installer/` (stdlib only; `installer/layout.py` maps the release folder and, in a repository, the dev toolchain):
```
python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR [--root DIR] [--temp DIR] [--verify]
```
- On a folder with `totk-install.json` only `--dest` is needed; the game files only when the extraction must run again.
- stdout: one JSON event per line, types `start`, `plan`, `step` (started, done, skipped), `progress` (step, phase,
done/total/unit), `space` (bytes needed and free on the install disk and the temp disk), `warning`, `error`, `end` (`installer/events.py`). Messages are ids with parameters;
`installer/text/en.json` holds their English text. The human-readable log goes to `<temp>/logs/`.
- Exit codes: 0 done, 1 failed, 2 bad command line, 3 cancelled. The window cancels with Ctrl+Break to the
installer's process group.
- Each step records a key over its inputs (runner.py). A step with the recorded key and intact output is skipped, so
resume and update are one rule; output kept from another release is re-hashed first.
- Peak space on the install disk is about 36 GiB, not 22: the extraction copies both Program NCAs (16.3 GiB) next to
the merged RomFS (15.9 GiB) before it deletes them. The check computes the exact figure.
- Window (`python -m installer.gui`, `installer/gui.py` draws, `installer/guilogic.py` decides): form, step list with
progress, space line, collapsible log, Cancel (Ctrl+Break to the CLI's process group, then it waits for the exit),
finish screen with Open folder. English or Russian, chosen from the Windows display language, switchable in the
window; every sentence is an id in `installer/text/<en|ru>.json`. A folder with `totk-install.json` needs only the
folder. `--selftest [--shots DIR]` builds every screen in both languages and closes itself.
- `tests/installer_gui_test.py`: event model, form, catalogs, cancel, the self-closing window and one end-to-end run
of the CLI in `--verify` mode (needs `build/install-test` and `build/extracted/v121`; see its docstring).
- `tests/installer_test.py`: bookkeeping with fake steps. `tests/installer_e2e.py`: a dev-mode run with the files of
`config.local.toml`.
## Release layout
The unpacked release; `tools/make_release.py` assembles it, CI zips it:
```
TotK-Setup.exe starts python/pythonw.exe -m installer.gui (installer/launcher/launcher.c, no console)
installer/ the installer: steps, CLI, window, en/ru text catalog
python/ Python 3.14.9 embeddable + numpy, lz4, cryptography, cffi, pycparser (+ tkinter for the window)
recomp/, tools/ the lifter and the build, extraction and shader scripts the steps run (copied from the repo)
llvm/ llvm-mingw, x86_64-only subset (Apache-2.0 with LLVM exception; mingw-w64 parts permissive)
runtime/ libtotkrt.a and the headers the generated code and modules.c/romfs.c include
bin/ totk_xlate.exe, hactool.exe, SDL3.dll, libopus-0.dll, ninja.exe
data/ shader_list.tsv (MacNativeGames, MIT)
THIRD_PARTY_NOTICES.md
totk.ini.example, VERSION the default ini; the release string (git describe of the build)
```
- SDL3 comes from the upstream release, not MSYS2: MSYS2's DLL imports the LGPL libiconv.
- Every bundled third-party piece gets a row in `THIRD_PARTY_NOTICES.md`.
- The release contains nothing game-derived; `tools/check_content.py` runs over it in CI.
## Constraints
- The installer steps contain no Win32 calls: Linux follows with its own compiler and the same steps.
- The keys file path is passed to the tools; key values never appear in the UI, logs or files.
- The window is a front end over the CLI (`python -m installer --base ... --update ... --keys ... --lang ... --dest
...`), which prints one JSON progress event per line. Tests and CI use the CLI.
- Unsigned exe: SmartScreen warns on first start.
## Later
- Re-warm the driver cache on start after a move, GPU or driver change (WIP on task/rewarm: stamp in
`shader-prepared.bin`).
- NSZ and XCZ input. Linux release.
+10
View File
@@ -0,0 +1,10 @@
"""The TotK installer: builds a playable folder from the player's own game files (docs/installer.md).
python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR [--root DIR] [--temp DIR]
Modules: layout (where the bundled tools are, release or repository), events (JSON progress on stdout and the log
file), state (totk-install.json), runner (which steps run: resume and update), steps (the six steps), ini (totk.ini),
cli (the command line). text/en.json holds the English text of every message id the events carry.
Standard library only, apart from what the extraction tools import (cryptography). Runs on Python 3.14.
"""
+5
View File
@@ -0,0 +1,5 @@
import sys
from installer.cli import main
sys.exit(main())
+78
View File
@@ -0,0 +1,78 @@
"""Command line: python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR [--root DIR]
[--temp DIR] [--verify]
A first install needs every option. On a folder with totk-install.json (resume after a cancel or failure, or update
with a newer release) only --dest is needed: --lang defaults to the recorded language, and the game files are needed
only when the extraction has to run again. Prints JSON events on stdout (events.py). Exit code: 0 done, 1 failed,
2 bad command line, 3 cancelled (Ctrl+C or Ctrl+Break).
"""
import argparse
import hashlib
import signal
import sys
import time
from pathlib import Path
from . import layout as layouts
from . import state as statefile
from .events import Reporter
from .runner import Context, run_steps
from .steps import all_steps
EXIT = {'ok': 0, 'failed': 1, 'cancelled': 3}
class Parser(argparse.ArgumentParser):
def error(self, message):
Reporter().error('cli.usage', {'detail': message})
sys.exit(2)
def temp_folder(base, dest):
"""One temp folder per install folder, so two installs never share generated files."""
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
return base / f'{dest.name or "root"}-{tag}'
def cancel(signum, frame):
raise KeyboardInterrupt
def main(argv=None):
ap = Parser(prog='python -m installer', description=__doc__.splitlines()[0])
ap.add_argument('--base', type=Path, help='base game, XCI or NSP')
ap.add_argument('--update', type=Path, help='1.2.1 update NSP')
ap.add_argument('--keys', type=Path, help='prod.keys')
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
ap.add_argument('--dest', type=Path, required=True, help='install folder')
ap.add_argument('--root', type=Path, help='release folder or repository (default: the folder above installer/)')
ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release, build/install-temp in a '
'repository)')
ap.add_argument('--verify', action='store_true', help='hash every kept file, also those this release made')
args = ap.parse_args(argv)
# The window cancels with Ctrl+Break to the installer's process group (CREATE_NEW_PROCESS_GROUP), which also reaches
# the tools it runs; Python's default for it ends the process without recording anything.
for name in ('SIGBREAK', 'SIGTERM'):
if hasattr(signal, name):
signal.signal(getattr(signal, name), cancel)
layout = layouts.find(args.root)
dest = args.dest.resolve()
temp = temp_folder((args.temp or layout.temp).resolve(), dest)
log = temp / 'logs' / time.strftime('install-%Y%m%d-%H%M%S.log')
rep = Reporter(log_path=log)
t0 = time.monotonic()
try:
state = statefile.load(dest)
except statefile.StateError as e:
rep.error(e.id, {'path': str(e.path)})
rep.emit('end', result='failed', seconds=0)
return 1
ctx = Context(layout, dest, temp, rep, args, state=state, verify=args.verify)
steps = all_steps()
rep.emit('start', release=layout.release, mode=ctx.mode, dev=layout.dev, dest=str(dest), log=str(log),
steps=[s.id for s in steps])
result = run_steps(steps, ctx)
rep.emit('end', result=result, seconds=round(time.monotonic() - t0, 1))
rep.close()
return EXIT[result]
+119
View File
@@ -0,0 +1,119 @@
"""Progress events: one JSON object per line on stdout for the window and tests, the same in English to a log file.
Every event has "type" and "t" (seconds since the start). Types:
start release, mode (install, resume, update), dest, log, steps (the step ids in order)
step step, state: started | done (seconds) | skipped (reason: current, not_needed)
progress step, phase, and done/total/unit (bytes, chunks, objects, shaders) when the amount is known
space dest, need, free (bytes on the install disk; with the temp folder on the same disk need includes it),
temp, temp_need, temp_free, same_disk
warning id, params
error id, params, step (absent outside a step)
end result: ok | failed | cancelled, seconds
Text is never the payload: ids name a step ("step.<id>"), a phase ("phase.<step>.<phase>") or a message ("<id>"),
and installer/text/en.json holds their English text with {param} placeholders. The log renders events with it.
"""
import json
import sys
import threading
import time
from pathlib import Path
CATALOG = Path(__file__).resolve().parent / 'text' / 'en.json'
PROGRESS_EVERY = 0.5 # seconds between two progress events of one phase
def load_catalog(path=CATALOG):
return json.loads(path.read_text(encoding='utf-8'))
def render(catalog, id, params=None):
"""The English text of id, or the id itself when the catalog lacks it."""
text = catalog.get(id, id)
try:
return text.format(**(params or {}))
except (KeyError, IndexError, ValueError):
return f'{text} {params}'
class Reporter:
def __init__(self, out=None, log_path=None):
self.out = out or sys.stdout
self.t0 = time.monotonic()
self.catalog = load_catalog()
self.lock = threading.Lock()
self.log_file = None
self.last_progress = {}
if log_path:
Path(log_path).parent.mkdir(parents=True, exist_ok=True)
self.log_file = open(log_path, 'a', encoding='utf-8', buffering=1)
self.log_path = log_path
def elapsed(self):
return time.monotonic() - self.t0
def log(self, text):
"""A line for the log file only."""
if self.log_file:
with self.lock:
self.log_file.write(f'[{self.elapsed():8.1f}] {text}\n')
def emit(self, type, **fields):
event = {'type': type, 't': round(self.elapsed(), 2), **fields}
with self.lock:
self.out.write(json.dumps(event, ensure_ascii=False) + '\n')
self.out.flush()
self.log(self.describe(event))
return event
def describe(self, e):
c = self.catalog
kind = e['type']
if kind == 'step':
name = render(c, 'step.' + e['step'])
extra = f' ({e["seconds"]:.1f} s)' if 'seconds' in e else f' ({e["reason"]})' if 'reason' in e else ''
return f'{name}: {e["state"]}{extra}'
if kind == 'progress':
text = render(c, f'phase.{e["step"]}.{e["phase"]}')
if e.get('total'):
text += f' {e["done"]}/{e["total"]} {e.get("unit", "")}'.rstrip()
return text
if kind in ('warning', 'error'):
return f'{kind}: {render(c, e["id"], e.get("params"))} [{e["id"]}]'
return json.dumps(e, ensure_ascii=False)
def step(self, step, state, **fields):
return self.emit('step', step=step, state=state, **fields)
def progress(self, step, phase, done=None, total=None, unit=None, force=False):
"""A progress event, at most one per PROGRESS_EVERY per phase unless it is the first or the last."""
now = time.monotonic()
key = (step, phase)
last = self.last_progress.get(key)
final = total is not None and done is not None and done >= total
if last is not None and not final and not force and now - last < PROGRESS_EVERY:
return None
self.last_progress[key] = now
fields = {'step': step, 'phase': phase}
if done is not None:
fields['done'] = done
if total is not None:
fields['total'] = total
if unit:
fields['unit'] = unit
return self.emit('progress', **fields)
def warning(self, id, **params):
return self.emit('warning', id=id, params=params)
def error(self, id, params, step=None):
fields = {'id': id, 'params': params}
if step:
fields['step'] = step
return self.emit('error', **fields)
def close(self):
if self.log_file:
self.log_file.close()
self.log_file = None
+587
View File
@@ -0,0 +1,587 @@
"""The installer window: python -m installer.gui
A front end over the CLI (docs/installer.md): it collects the game files, the keys file path, the game language and the
install folder, runs `python -m installer ...` as a subprocess and shows its JSON events. Decisions live in guilogic.py;
this file only draws. The window text is in installer/text/<en|ru>.json under "ui.*" and "launcher.*" ids.
--selftest [--shots DIR] build every screen in both languages and close by itself (PNG per screen with --shots)
--lang en|ru window language (default: the Windows display language)
"""
import argparse
import json
import os
import sys
import tkinter as tk
from pathlib import Path
from tkinter import filedialog, messagebox, ttk
try:
from PIL import ImageGrab # only the screenshots of --selftest --shots need it; the release has no PIL
except ImportError:
ImageGrab = None
from . import ini
from . import layout as layouts
from .guilogic import (DONE, FAILED, PENDING, RUNNING, SKIPPED, UI_LANGUAGES, Catalog, Form, Model, Process,
amount, default_game_language, describe_event, parse_line, windows_ui_language)
POLL_MS = 100
GLYPH = {PENDING: '\u25cb', RUNNING: '\u25b6', DONE: '\u2714', SKIPPED: '\u2013', FAILED: '\u2716'}
COLOR = {PENDING: '#808080', RUNNING: '#1a5fb4', DONE: '#26a269', SKIPPED: '#808080', FAILED: '#c01c28'}
PAD = 14
def fix_tcl_paths():
"""The embeddable Python keeps Tcl/Tk next to python.exe (python/tcl); point Tcl at it when it is not found."""
tcl = Path(sys.executable).resolve().parent / 'tcl'
if tcl.is_dir() and 'TCL_LIBRARY' not in os.environ:
for lib, var in (('tcl9.0', 'TCL_LIBRARY'), ('tk9.0', 'TK_LIBRARY')):
if (tcl / lib).is_dir():
os.environ[var] = str(tcl / lib)
class App:
def __init__(self, root, ui_lang, layout, run_args=None):
self.root, self.layout = root, layout
self.cat = Catalog(ui_lang)
self.model = Model()
self.process = None
self.closing = False
self.run_args = run_args or {} # extra CLI options for tests: temp, verify
self.log_entries = [] # events and raw lines of the run, rendered by render_log
self.bound = [] # (widget, option, id, params) retranslated on a language switch
self.vars = {n: tk.StringVar() for n in ('base', 'update', 'keys', 'dest')}
self.vars['lang'] = tk.StringVar()
self.lang_code = default_game_language(ui_lang)
self.screen = None
root.title(self.cat('ui.title'))
root.minsize(680, 560)
root.protocol('WM_DELETE_WINDOW', self.on_close)
style = ttk.Style(root)
style.configure('Title.TLabel', font=('Segoe UI', 15, 'bold'))
style.configure('Notice.TLabel', background='#fff3cd', foreground='#664d03', padding=8)
style.configure('Err.TLabel', foreground='#c01c28')
top = ttk.Frame(root, padding=(PAD, PAD, PAD, 0))
top.pack(fill='x')
self.title = self.label(top, 'ui.title', style='Title.TLabel')
self.title.pack(side='left')
sw = ttk.Frame(top)
sw.pack(side='right')
self.ui_var = tk.StringVar(value=self.cat.lang)
for code in UI_LANGUAGES:
ttk.Radiobutton(sw, text=code.upper(), value=code, variable=self.ui_var, style='Toolbutton',
command=self.switch_language).pack(side='left', padx=1)
self.body = ttk.Frame(root, padding=PAD)
self.body.pack(fill='both', expand=True)
self.frames = {'form': self.build_form(), 'run': self.build_run(), 'finish': self.build_finish()}
self.vars['lang'].set(self.lang_label(self.lang_code))
self.vars['dest'].trace_add('write', lambda *_: self.refresh_form())
self.show('form')
# ---- text -----------------------------------------------------------------------------------------------------
def bind_text(self, widget, option, id, **params):
self.bound.append((widget, option, id, params))
widget.configure(**{option: self.cat(id, **params)})
def label(self, parent, id, **kw):
w = ttk.Label(parent, **kw)
self.bind_text(w, 'text', id)
return w
def button(self, parent, id, command, **kw):
w = ttk.Button(parent, command=command, **kw)
self.bind_text(w, 'text', id)
return w
def retext(self):
self.root.title(self.cat('ui.title'))
for widget, option, id, params in self.bound:
widget.configure(**{option: self.cat(id, **params)})
self.fill_languages()
self.refresh_form()
self.refresh_run()
self.refresh_finish()
self.render_log()
def switch_language(self):
self.cat.set_language(self.ui_var.get())
code = self.selected_lang()
self.retext()
self.vars['lang'].set(self.lang_label(code))
def lang_label(self, code):
return self.cat('lang.' + code)
def fill_languages(self):
code = self.selected_lang()
self.lang_box.configure(values=[self.lang_label(c) for c in ini.LANGUAGES])
self.vars['lang'].set(self.lang_label(code))
def selected_lang(self):
"""The game language code behind the combobox text (kept when it is not one of the labels)."""
shown = self.vars['lang'].get()
for code in ini.LANGUAGES:
for lang in UI_LANGUAGES:
if shown == Catalog(lang)('lang.' + code):
return code
return self.lang_code
# ---- screens --------------------------------------------------------------------------------------------------
def show(self, name):
for f in self.frames.values():
f.pack_forget()
self.frames[name].pack(fill='both', expand=True)
self.screen = name
def notice(self, parent):
w = ttk.Label(parent, style='Notice.TLabel', wraplength=620, justify='left')
self.bind_text(w, 'text', 'ui.notice')
return w
def build_form(self):
f = ttk.Frame(self.body)
self.form_intro = self.label(f, 'ui.intro', wraplength=620, justify='left')
self.form_intro.pack(anchor='w', pady=(0, 8))
self.notice(f).pack(fill='x', pady=(0, 10))
grid = ttk.Frame(f)
grid.pack(fill='x')
grid.columnconfigure(1, weight=1)
rows = (('base', 'ui.base', 'ui.pick_game'), ('update', 'ui.update', 'ui.pick_update'),
('keys', 'ui.keys', 'ui.pick_keys'))
self.form_rows = {}
for i, (name, label_id, pick_id) in enumerate(rows):
lab = self.label(grid, label_id)
lab.grid(row=i, column=0, sticky='w', pady=4, padx=(0, 10))
ttk.Entry(grid, textvariable=self.vars[name]).grid(row=i, column=1, sticky='ew', pady=4)
b = self.button(grid, 'ui.browse', lambda n=name, p=pick_id: self.pick_file(n, p))
b.grid(row=i, column=2, padx=(8, 0))
self.form_rows[name] = lab
i = len(rows)
self.label(grid, 'ui.game_language').grid(row=i, column=0, sticky='w', pady=4)
self.lang_box = ttk.Combobox(grid, textvariable=self.vars['lang'], state='readonly', width=30)
self.lang_box.grid(row=i, column=1, sticky='w', pady=4)
self.lang_touched = False
self.lang_box.bind('<<ComboboxSelected>>', lambda _e: setattr(self, 'lang_touched', True))
self.fill_languages()
self.label(grid, 'ui.dest').grid(row=i + 1, column=0, sticky='w', pady=4)
ttk.Entry(grid, textvariable=self.vars['dest']).grid(row=i + 1, column=1, sticky='ew', pady=4)
self.button(grid, 'ui.browse', self.pick_dest).grid(row=i + 1, column=2, padx=(8, 0))
self.update_hint = ttk.Label(f, wraplength=620, justify='left')
self.update_hint.pack(anchor='w', pady=(8, 0))
self.form_error = ttk.Label(f, style='Err.TLabel', wraplength=620, justify='left')
self.form_error.pack(anchor='w', pady=(8, 0))
self.label(f, 'ui.keys_private', wraplength=620, justify='left').pack(anchor='w', pady=(8, 0))
bar = ttk.Frame(f)
bar.pack(side='bottom', fill='x', pady=(12, 0))
self.start_button = self.button(bar, 'ui.start', self.start)
self.start_button.pack(side='right')
return f
def build_run(self):
f = ttk.Frame(self.body)
self.run_header = ttk.Label(f, font=('Segoe UI', 11, 'bold'))
self.run_header.pack(anchor='w')
self.step_rows = {}
self.steps_box = ttk.Frame(f)
self.steps_box.pack(fill='x', pady=(8, 8))
self.phase_label = ttk.Label(f, wraplength=620, justify='left')
self.phase_label.pack(anchor='w')
self.bar = ttk.Progressbar(f, mode='determinate', maximum=1000)
self.bar.pack(fill='x', pady=(4, 6))
self.space_label = ttk.Label(f, wraplength=620, justify='left')
self.space_label.pack(anchor='w')
self.warn_label = ttk.Label(f, wraplength=620, justify='left', foreground='#9a6700')
self.warn_label.pack(anchor='w')
bar = ttk.Frame(f)
bar.pack(side='bottom', fill='x', pady=(8, 0))
self.cancel_button = self.button(bar, 'ui.cancel', self.cancel)
self.cancel_button.pack(side='right')
self.log_toggle = self.button(bar, 'ui.show_log', self.toggle_log)
self.log_toggle.pack(side='left')
self.log_frame = ttk.Frame(f)
self.log = tk.Text(self.log_frame, height=10, wrap='word', state='disabled', font=('Consolas', 9))
scroll = ttk.Scrollbar(self.log_frame, command=self.log.yview)
self.log.configure(yscrollcommand=scroll.set)
scroll.pack(side='right', fill='y')
self.log.pack(side='left', fill='both', expand=True)
self.log_open = False
return f
def build_finish(self):
f = ttk.Frame(self.body)
self.finish_title = ttk.Label(f, style='Title.TLabel', wraplength=620, justify='left')
self.finish_title.pack(anchor='w')
self.finish_text = ttk.Label(f, wraplength=620, justify='left')
self.finish_text.pack(anchor='w', pady=(8, 8))
self.finish_steps = ttk.Frame(f)
self.finish_steps.pack(fill='x', pady=(0, 8))
self.finish_notice = self.notice(f)
self.finish_notice.pack(fill='x', pady=(8, 0))
bar = ttk.Frame(f)
bar.pack(side='bottom', fill='x', pady=(12, 0))
self.button(bar, 'ui.close', self.on_close).pack(side='right')
self.open_button = self.button(bar, 'ui.open_folder', self.open_folder)
self.open_button.pack(side='right', padx=(0, 8))
self.back_button = self.button(bar, 'ui.back', lambda: self.show('form'))
self.back_button.pack(side='left')
return f
# ---- form -----------------------------------------------------------------------------------------------------
def form(self):
return Form(base=self.vars['base'].get(), update=self.vars['update'].get(), keys=self.vars['keys'].get(),
lang=self.selected_lang(), dest=self.vars['dest'].get())
def refresh_form(self):
form = self.form()
state = form.installed_state()
if state is not None:
self.update_hint.configure(text=self.cat('ui.update_mode'))
if state.get('lang') in ini.LANGUAGES and not self.lang_touched:
self.vars['lang'].set(self.lang_label(state['lang']))
else:
self.update_hint.configure(text='')
def pick_file(self, name, pick_id):
types = [(self.cat('ui.type_game'), '*.nsp *.xci'), (self.cat('ui.type_all'), '*.*')]
if name == 'update':
types = [(self.cat('ui.type_nsp'), '*.nsp'), (self.cat('ui.type_all'), '*.*')]
if name == 'keys':
types = [(self.cat('ui.type_keys'), '*.keys'), (self.cat('ui.type_all'), '*.*')]
path = filedialog.askopenfilename(title=self.cat(pick_id), filetypes=types)
if path:
self.vars[name].set(os.path.normpath(path))
def pick_dest(self):
path = filedialog.askdirectory(title=self.cat('ui.pick_dest'), mustexist=False)
if path:
self.vars['dest'].set(os.path.normpath(path))
# ---- run ------------------------------------------------------------------------------------------------------
def start(self):
form = self.form()
problems = form.problems()
if problems:
self.form_error.configure(text=' '.join(self.cat(p) for p in problems))
return
self.form_error.configure(text='')
command = form.command(self.layout.python, root=self.layout.root if self.layout.dev else None,
temp=self.run_args.get('temp'), verify=self.run_args.get('verify', False))
self.begin(command, self.layout.root)
def begin(self, command, cwd):
self.model = Model()
self.model.dest = self.vars['dest'].get().strip()
self.clear_log()
self.build_step_rows(self.model.steps)
self.cancel_button.configure(state='normal')
self.bound = [b for b in self.bound if b[0] is not self.cancel_button]
self.bind_text(self.cancel_button, 'text', 'ui.cancel')
self.show('run')
self.refresh_run()
self.process = Process(command, cwd=cwd, helper_python=self.layout.python)
self.root.after(POLL_MS, self.poll)
def build_step_rows(self, steps):
for w in self.steps_box.winfo_children():
w.destroy()
self.step_rows = {}
for s in steps:
row = ttk.Frame(self.steps_box)
row.pack(fill='x', pady=1)
glyph = ttk.Label(row, width=3)
glyph.pack(side='left')
name = ttk.Label(row)
name.pack(side='left')
self.step_rows[s] = (glyph, name)
def poll(self):
if self.process is None:
return
try:
while True:
line = self.process.lines.get_nowait()
if line is not None:
self.feed(line)
except Exception:
pass
code = self.process.poll()
self.refresh_run()
if code is None:
self.root.after(POLL_MS, self.poll)
return
self.model.finish(code)
self.process = None
self.refresh_run()
self.refresh_finish()
self.show('finish')
if self.closing:
self.root.destroy()
def feed(self, line):
"""One stdout line of the CLI: an event updates the model and the log, anything else goes to the log as is."""
event = parse_line(line)
if event is None:
if line.strip():
self.log_entries.append(line.rstrip())
self.append_log(line.rstrip())
return
before = list(self.model.steps)
self.model.apply(event)
if self.model.steps != before:
self.build_step_rows(self.model.steps)
self.log_entries.append(event)
self.append_log(describe_event(self.cat, event))
def refresh_run(self):
m = self.model
for s, (glyph, name) in self.step_rows.items():
state = m.states.get(s, PENDING)
glyph.configure(text=GLYPH[state], foreground=COLOR[state])
name.configure(text=self.cat('step.' + s) + (f' ({self.cat("ui.state_" + state)})'
if state in (SKIPPED, FAILED) else ''),
font=('Segoe UI', 10, 'bold') if state == RUNNING else ('Segoe UI', 10))
mode = m.mode or 'install'
self.run_header.configure(text=self.cat('ui.running_' + mode if mode in ('install', 'resume', 'update')
else 'ui.running_install'))
if m.current and m.phase:
text = self.cat(f'phase.{m.current}.{m.phase}')
if m.total:
text += ' ' + amount(self.cat, m.done, m.total, m.unit)
elif m.current:
text = self.cat('step.' + m.current)
else:
text = ''
self.phase_label.configure(text=text)
frac = m.fraction()
if frac is None:
if str(self.bar.cget('mode')) != 'indeterminate':
self.bar.configure(mode='indeterminate')
self.bar.start(15)
else:
if str(self.bar.cget('mode')) == 'indeterminate':
self.bar.stop()
self.bar.configure(mode='determinate')
self.bar.configure(value=frac * 1000)
self.space_label.configure(text=m.need_text(self.cat))
self.warn_label.configure(text='\n'.join(self.cat(i, **p) for i, p in m.warnings))
def toggle_log(self):
self.log_open = not self.log_open
if self.log_open:
self.log_frame.pack(fill='both', expand=True, pady=(8, 0))
else:
self.log_frame.pack_forget()
self.bound = [b for b in self.bound if b[0] is not self.log_toggle]
self.bind_text(self.log_toggle, 'text', 'ui.hide_log' if self.log_open else 'ui.show_log')
def clear_log(self):
self.log_entries = []
self.render_log()
def render_log(self):
"""The whole log from the kept events in the current language (raw CLI lines stay as they are)."""
self.log.configure(state='normal')
self.log.delete('1.0', 'end')
for entry in self.log_entries:
text = entry if isinstance(entry, str) else describe_event(self.cat, entry)
if text:
self.log.insert('end', text + '\n')
self.log.see('end')
self.log.configure(state='disabled')
def append_log(self, text):
if text:
self.log.configure(state='normal')
self.log.insert('end', text + '\n')
self.log.see('end')
self.log.configure(state='disabled')
def cancel(self):
if self.process:
self.cancel_button.configure(state='disabled')
self.bound = [b for b in self.bound if b[0] is not self.cancel_button]
self.bind_text(self.cancel_button, 'text', 'ui.cancelling')
self.process.cancel()
def on_close(self):
if self.process:
if not messagebox.askyesno(self.cat('ui.title'), self.cat('ui.confirm_cancel')):
return
self.closing = True
self.cancel()
return
self.root.destroy()
# ---- finish ---------------------------------------------------------------------------------------------------
def refresh_finish(self):
m = self.model
result = m.result or 'ok'
self.finish_title.configure(text=self.cat('ui.result_' + result))
if result == 'ok':
text = self.cat('ui.done_text', dest=m.dest)
elif result == 'cancelled':
text = self.cat('ui.cancelled_text')
else:
err = m.error or ('ui.process_died', {'code': m.exit_code}, None)
text = self.cat(err[0], **err[1])
if m.log_path:
text += '\n\n' + self.cat('ui.log_file', path=m.log_path)
self.finish_text.configure(text=text)
for w in self.finish_steps.winfo_children():
w.destroy()
for s in m.steps:
state = m.states.get(s, PENDING)
ttk.Label(self.finish_steps, text=f'{GLYPH[state]} {self.cat("step." + s)}',
foreground=COLOR[state]).pack(anchor='w')
if result == 'ok':
self.open_button.state(['!disabled'])
else:
self.open_button.state(['disabled' if not Path(m.dest or '.').is_dir() else '!disabled'])
def open_folder(self):
dest = self.model.dest
if dest and Path(dest).is_dir():
os.startfile(dest)
# ---- self test ---------------------------------------------------------------------------------------------------
SAMPLE_START = {'type': 'start', 'release': 'selftest', 'mode': 'install', 'dest': 'Z:/selftest/TotK', 'log': 'D:\\log.txt',
'steps': ['check', 'extract', 'lift', 'compile', 'prepare', 'cleanup']}
SAMPLE_RUN = [
SAMPLE_START,
{'type': 'plan', 'run': ['check', 'extract'], 'skip': {}},
{'type': 'step', 'step': 'check', 'state': 'started'},
{'type': 'progress', 'step': 'check', 'phase': 'inputs'},
{'type': 'space', 'dest': 'Z:/selftest/TotK', 'need': 36_400_000_000, 'free': 210_000_000_000, 'temp': 'C:\\t',
'temp_need': 0, 'temp_free': 0, 'same_disk': True},
{'type': 'warning', 'id': 'warn.lang_needs_fonts', 'params': {'lang': 'ko'}},
{'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 12.0},
{'type': 'step', 'step': 'extract', 'state': 'started'},
{'type': 'progress', 'step': 'extract', 'phase': 'copy', 'done': 6_000_000_000, 'total': 16_000_000_000,
'unit': 'bytes'},
]
SAMPLE_OK = [SAMPLE_START, {'type': 'plan', 'run': ['check', 'extract'], 'skip': {'lift': 'current', 'compile': 'current',
'prepare': 'current', 'cleanup': 'not_needed'}},
{'type': 'step', 'step': 'check', 'state': 'started'},
{'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 1.0},
{'type': 'step', 'step': 'extract', 'state': 'started'},
{'type': 'step', 'step': 'extract', 'state': 'done', 'seconds': 1.0},
{'type': 'step', 'step': 'lift', 'state': 'skipped', 'reason': 'current'},
{'type': 'step', 'step': 'compile', 'state': 'skipped', 'reason': 'current'},
{'type': 'step', 'step': 'prepare', 'state': 'skipped', 'reason': 'current'},
{'type': 'step', 'step': 'cleanup', 'state': 'done', 'seconds': 0.1},
{'type': 'end', 'result': 'ok', 'seconds': 3}]
def grab(root, path):
if ImageGrab is None:
raise SystemExit('--shots needs Pillow')
root.attributes('-topmost', True)
root.update()
x, y, w, h = root.winfo_rootx(), root.winfo_rooty(), root.winfo_width(), root.winfo_height()
ImageGrab.grab(bbox=(x, y, x + w, y + h), all_screens=True).save(path)
def selftest(layout, shots=None):
"""Every screen in both languages; the window closes itself. Returns 0 when nothing raised."""
fix_tcl_paths()
root = tk.Tk()
root.geometry('720x640+60+40')
app = App(root, 'en', layout)
steps, mixed = [], []
def shot(name):
root.update()
if shots:
Path(shots).mkdir(parents=True, exist_ok=True)
grab(root, Path(shots) / f'{name}.png')
def run_all():
for lang in UI_LANGUAGES:
app.ui_var.set(lang)
app.switch_language()
tag = lang
app.vars['dest'].set('Z:/selftest/TotK')
app.vars['base'].set('Z:/selftest/totk.xci')
app.vars['update'].set('Z:/selftest/totk-1.2.1.nsp')
app.vars['keys'].set('Z:/selftest/prod.keys')
app.show('form')
app.start() # missing files: shows the problems
steps.append(app.form_error.cget('text'))
shot(f'form-{tag}')
app.model = Model()
app.clear_log()
app.build_step_rows(app.model.steps)
app.show('run')
for e in SAMPLE_RUN:
app.feed(json.dumps(e))
app.log_open = False
app.toggle_log()
app.refresh_run()
shot(f'run-{tag}')
if lang == 'en': # the log follows the window language: no English line may remain after the switch
app.ui_var.set('ru')
app.switch_language()
text = app.log.get('1.0', 'end')
mixed.append('Checking' in text or 'Extracting' not in text and 'Распаковка' not in text)
app.ui_var.set('en')
app.switch_language()
app.model = Model()
for e in SAMPLE_OK:
app.model.apply(e)
app.model.dest = 'Z:/selftest/TotK'
app.model.finish(0)
app.refresh_finish()
app.show('finish')
shot(f'finish-ok-{tag}')
app.model = Model()
for e in SAMPLE_RUN[:3] + [{'type': 'error', 'id': 'check.disk_space', 'step': 'check',
'params': {'path': 'Z:/selftest/TotK', 'need_gb': 36.4, 'free_gb': 20.1}},
{'type': 'end', 'result': 'failed', 'seconds': 3}]:
app.model.apply(e)
app.model.finish(1)
app.refresh_finish()
shot(f'finish-failed-{tag}')
app.model = Model()
app.model.finish(3)
app.refresh_finish()
shot(f'finish-cancelled-{tag}')
app.toggle_log()
root.destroy()
root.after(200, run_all)
root.after(60000, root.destroy) # never leaves a window open
root.mainloop()
return 0 if len(steps) == len(UI_LANGUAGES) and all(steps) and mixed == [False] else 1
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--selftest', action='store_true')
ap.add_argument('--shots', type=Path)
ap.add_argument('--lang', choices=UI_LANGUAGES)
ap.add_argument('--root', type=Path)
args = ap.parse_args(argv)
layout = layouts.find(args.root)
if args.selftest:
return selftest(layout, args.shots)
fix_tcl_paths()
root = tk.Tk()
root.geometry('720x640')
App(root, args.lang or windows_ui_language(), layout)
root.mainloop()
return 0
if __name__ == '__main__':
sys.exit(main())
+361
View File
@@ -0,0 +1,361 @@
"""The installer window without Tk: text catalogs, the form, the event model and the CLI subprocess.
gui.py draws; everything it decides lives here so tests/installer_gui_test.py runs it headless.
Catalog text of an id in the window language (installer/text/<lang>.json), English when that lacks it
Form what the player entered; command() and problems() build and check the CLI call
Model folds the CLI's JSON events (events.py) into step states, progress, space, errors and the result
Process runs `python -m installer ...`, cancels it with Ctrl+Break and feeds lines to a queue
"""
import ctypes
import json
import os
import queue
import subprocess
import sys
import threading
import time
from dataclasses import dataclass
from pathlib import Path
from . import ini
from . import state as statefile
from .events import render
TEXT = Path(__file__).resolve().parent / 'text'
UI_LANGUAGES = ('en', 'ru')
STEP_ORDER = ('check', 'extract', 'lift', 'compile', 'prepare', 'cleanup')
# message parameter -> catalog id prefix of the names that replace its raw values
PARAM_NAMES = {'lang': 'lang.', 'known': 'lang.', 'step': 'step.', 'input': 'param.input.'}
LANG_RU = 0x19 # primary language id of Russian in a Windows LANGID
def load_texts(lang):
path = TEXT / f'{lang}.json'
return json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
class Catalog:
"""Window text: ids from text/<lang>.json, then text/en.json, then the id itself."""
def __init__(self, lang='en'):
self.english = load_texts('en')
self.set_language(lang)
def set_language(self, lang):
self.lang = lang if lang in UI_LANGUAGES else 'en'
self.texts = load_texts(self.lang)
def has(self, id):
return id in self.texts or id in self.english
def __call__(self, id, **params):
text = self.texts.get(id) or self.english.get(id)
if text is None:
return id
return render({id: text}, id, {k: self.param(k, v) for k, v in params.items()})
def param(self, name, value):
"""A message parameter as the player should read it: language codes, step ids and input names become the
catalog's names (the ones the pickers and the step list show); the rest stays as it is."""
prefix = PARAM_NAMES.get(name)
if prefix:
if isinstance(value, (list, tuple)):
return ', '.join(self.param(name, v) for v in value)
ident = prefix + str(value)
if self.has(ident):
return self(ident)
if name == 'known':
return ', '.join(self(prefix + c) if self.has(prefix + c) else c for c in str(value).split(', '))
return flat(value)
def flat(value):
return ', '.join(str(v) for v in value) if isinstance(value, (list, tuple)) else value
def windows_ui_language():
"""'ru' when the Windows display language is Russian, else 'en'."""
try:
langid = ctypes.windll.kernel32.GetUserDefaultUILanguage()
except (AttributeError, OSError):
return 'en'
return 'ru' if langid & 0x3ff == LANG_RU else 'en'
def default_game_language(ui):
return 'ru' if ui == 'ru' else 'en-US'
def gib(n):
"""Bytes as decimal gigabytes with one digit, the unit the check.disk_space message uses."""
return f'{n / 1e9:.1f}'
def amount(cat, done, total, unit):
"""'6.0 / 16.0 GB', '120 / 4000 files'..., the progress of a phase whose size is known."""
if unit == 'bytes':
return cat('ui.amount_bytes', done=gib(done or 0), total=gib(total))
id = f'ui.amount_{unit}'
return cat(id if cat.has(id) else 'ui.amount_other', done=done or 0, total=total, unit=unit or '')
# ---- the form ----------------------------------------------------------------------------------------------------
@dataclass
class Form:
base: str = ''
update: str = ''
keys: str = ''
lang: str = ''
dest: str = ''
def installed_state(self):
"""The recorded state of the install folder (totk-install.json), or None. Never raises."""
if not self.dest.strip():
return None
try:
return statefile.load(Path(self.dest.strip()))
except statefile.StateError:
return None
def update_mode(self):
return self.installed_state() is not None
def problems(self):
"""Ids of the entries that are missing or wrong, in screen order. A folder with totk-install.json needs only
itself; the CLI asks for the game files when it must extract again."""
found = []
update = self.update_mode()
if not self.dest.strip():
found.append('ui.need_dest')
if not update:
for name in ('base', 'update', 'keys'):
value = getattr(self, name).strip()
if not value:
found.append(f'ui.need_{name}')
elif not Path(value).is_file():
found.append(f'ui.missing_{name}')
if self.lang not in ini.LANGUAGES:
found.append('ui.need_lang')
else:
for name in ('base', 'update', 'keys'):
value = getattr(self, name).strip()
if value and not Path(value).is_file():
found.append(f'ui.missing_{name}')
return found
def command(self, python, root=None, temp=None, verify=False):
cmd = [str(python), '-m', 'installer']
for flag, value in (('--base', self.base), ('--update', self.update), ('--keys', self.keys),
('--lang', self.lang), ('--dest', self.dest)):
if value.strip():
cmd += [flag, value.strip()]
if root:
cmd += ['--root', str(root)]
if temp:
cmd += ['--temp', str(temp)]
if verify:
cmd.append('--verify')
return cmd
# ---- events -> state ---------------------------------------------------------------------------------------------
PENDING, RUNNING, DONE, SKIPPED, FAILED = 'pending', 'running', 'done', 'skipped', 'failed'
def parse_line(line):
"""The event dict of a stdout line, or None when it is not a JSON event (a traceback line, noise)."""
line = line.strip()
if not line.startswith('{'):
return None
try:
event = json.loads(line)
except ValueError:
return None
return event if isinstance(event, dict) and isinstance(event.get('type'), str) else None
class Model:
"""What the window shows, folded from events. apply() takes one event; nothing else mutates it."""
def __init__(self):
self.steps = list(STEP_ORDER)
self.states = {s: PENDING for s in self.steps}
self.mode = ''
self.release = ''
self.log_path = ''
self.dest = ''
self.current = None # step id running
self.phase = None
self.done = self.total = None
self.unit = None
self.space = None # the space event
self.warnings = [] # (id, params)
self.error = None # (id, params, step)
self.result = None # ok | failed | cancelled once the end event arrived
self.seconds = None
self.exit_code = None
def apply(self, e):
kind = e['type']
if kind == 'start':
self.__init__() # a new run: nothing of an earlier one stays
self.steps = list(e.get('steps') or STEP_ORDER)
self.states = {s: PENDING for s in self.steps}
self.mode, self.release = e.get('mode', ''), e.get('release', '')
self.log_path, self.dest = e.get('log', ''), e.get('dest', '')
elif kind == 'plan':
for s, reason in (e.get('skip') or {}).items():
if s in self.states and self.states[s] == PENDING:
self.states[s] = SKIPPED
elif kind == 'step':
s, state = e.get('step'), e.get('state')
if s not in self.states:
self.states[s] = PENDING
self.steps.append(s)
if state == 'started':
self.states[s], self.current = RUNNING, s
self.phase = self.done = self.total = self.unit = None
elif state == 'done':
self.states[s] = DONE
self.current = None if self.current == s else self.current
elif state == 'skipped':
self.states[s] = SKIPPED
elif kind == 'progress':
if e.get('step') in self.states and self.states[e['step']] == PENDING:
self.states[e['step']] = RUNNING
self.current = e.get('step', self.current)
self.phase = e.get('phase')
self.done, self.total, self.unit = e.get('done'), e.get('total'), e.get('unit')
elif kind == 'space':
self.space = e
elif kind == 'warning':
self.warnings.append((e.get('id', ''), e.get('params') or {}))
elif kind == 'error':
self.error = (e.get('id', ''), e.get('params') or {}, e.get('step'))
if e.get('step') in self.states:
self.states[e['step']] = FAILED
elif kind == 'end':
self.result = e.get('result')
self.seconds = e.get('seconds')
self.current = None
def fraction(self):
"""0..1 of the current phase, or None when its size is unknown."""
if self.total:
return max(0.0, min(1.0, (self.done or 0) / self.total))
return None
def finish(self, exit_code):
"""The process ended. A missing end event (the process died) counts as failed, or cancelled on exit code 3."""
self.exit_code = exit_code
if self.result is None:
self.result = 'cancelled' if exit_code == 3 else 'failed'
if self.result == 'failed' and self.error is None:
self.error = ('ui.process_died', {'code': exit_code}, self.current)
for s, state in self.states.items():
if state == RUNNING:
self.states[s] = FAILED if self.result == 'failed' else PENDING
def need_text(self, cat):
"""The space line, or '' before the check step reported."""
sp = self.space
if not sp:
return ''
key = 'ui.space_one' if sp.get('same_disk') or not sp.get('temp_need') else 'ui.space_two'
return cat(key, path=sp.get('dest', ''), need=gib(sp.get('need', 0)), free=gib(sp.get('free', 0)),
temp=sp.get('temp', ''), temp_need=gib(sp.get('temp_need', 0)),
temp_free=gib(sp.get('temp_free', 0)))
def describe_event(cat, e):
"""One log line for an event, in the window language."""
kind = e['type']
if kind == 'step':
text = f'{cat("step." + e["step"])}: {cat("ui.state_" + e["state"])}'
if 'seconds' in e:
text += f' ({cat("ui.seconds", n=format(e["seconds"], ".1f"))})'
elif e.get('reason'):
text += f' ({e["reason"]})'
return text
if kind == 'progress':
text = cat(f'phase.{e["step"]}.{e["phase"]}')
if e.get('total'):
text += ' ' + amount(cat, e.get('done'), e['total'], e.get('unit'))
return text
if kind in ('warning', 'error'):
return f'{cat("ui.log_" + kind)}: {cat(e["id"], **(e.get("params") or {}))}'
if kind == 'space':
return cat('ui.log_space', need=gib(e.get('need', 0)), free=gib(e.get('free', 0)))
if kind == 'end':
return f'{cat("ui.log_end")}: {cat("ui.result_" + str(e.get("result")))}'
return ''
# ---- the CLI subprocess ------------------------------------------------------------------------------------------
CREATE_NEW_PROCESS_GROUP = 0x200
CREATE_NO_WINDOW = 0x08000000
_BREAK = """
import ctypes, sys, time
k = ctypes.windll.kernel32
pid = int(sys.argv[1])
k.FreeConsole()
if k.AttachConsole(pid):
k.SetConsoleCtrlHandler(None, True)
k.GenerateConsoleCtrlEvent(1, pid)
time.sleep(1)
k.FreeConsole()
else:
sys.exit(1)
"""
def send_break(proc, python=None):
"""Ctrl+Break to the process group of proc (started with CREATE_NEW_PROCESS_GROUP). The window itself may have no
console (pythonw), so a short helper process attaches to the child's console and sends it."""
helper = [str(python or sys.executable), '-c', _BREAK, str(proc.pid)]
flags = CREATE_NO_WINDOW if os.name == 'nt' else 0
r = subprocess.run(helper, creationflags=flags, capture_output=True)
return r.returncode == 0
class Process:
"""`python -m installer ...` with its stdout lines going to self.lines (a queue).
poll() returns the exit code once the process ended and its output is drained, else None."""
def __init__(self, command, cwd=None, helper_python=None, creationflags=None):
self.command = command
self.helper_python = helper_python
self.lines = queue.Queue()
flags = (CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW) if creationflags is None else creationflags
env = dict(os.environ, PYTHONIOENCODING='utf-8', PYTHONUTF8='1', PYTHONUNBUFFERED='1')
self.proc = subprocess.Popen(command, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, creationflags=flags)
self.cancelled_at = None
self.reader = threading.Thread(target=self.read, daemon=True)
self.reader.start()
def read(self):
for raw in iter(self.proc.stdout.readline, b''):
self.lines.put(raw.decode('utf-8', errors='replace'))
self.proc.stdout.close()
def cancel(self):
if self.cancelled_at is None and self.proc.poll() is None:
self.cancelled_at = time.monotonic()
send_break(self.proc, self.helper_python)
def poll(self):
code = self.proc.poll()
if code is not None and not self.reader.is_alive() and self.lines.empty():
return code
return None
def kill(self):
if self.proc.poll() is None:
self.proc.kill()
+65
View File
@@ -0,0 +1,65 @@
// Host check of the installer's first step, compiled with the bundled clang at install time: does the CPU run
// -march=x86-64-v3 code, and does a Vulkan 1.3 device exist. Prints one JSON object on stdout.
// The Vulkan loader is opened at run time, so a machine without one gets "loader": false instead of a failed start.
#include <stdio.h>
#include <string.h>
#include <vulkan/vulkan.h>
#ifdef _WIN32
#include <windows.h>
#define open_lib() LoadLibraryA("vulkan-1.dll")
#define lib_sym(l, n) ((void *)GetProcAddress((l), (n)))
#else
#include <dlfcn.h>
#define open_lib() dlopen("libvulkan.so.1", RTLD_NOW)
#define lib_sym(l, n) dlsym((l), (n))
#endif
static void json_string(const char *s) {
putchar('"');
for (; *s; s++) {
if (*s == '"' || *s == '\\') printf("\\%c", *s);
else if ((unsigned char)*s < 0x20) printf("\\u%04x", *s);
else putchar(*s);
}
putchar('"');
}
int main(void) {
__builtin_cpu_init();
printf("{\"x86_64_v3\": %s, ", __builtin_cpu_supports("x86-64-v3") ? "true" : "false");
void *lib = open_lib();
PFN_vkGetInstanceProcAddr gipa = lib ? (PFN_vkGetInstanceProcAddr)lib_sym(lib, "vkGetInstanceProcAddr") : NULL;
if (!gipa) {
printf("\"loader\": false, \"devices\": []}\n");
return 0;
}
printf("\"loader\": true, ");
PFN_vkCreateInstance create = (PFN_vkCreateInstance)gipa(NULL, "vkCreateInstance");
VkApplicationInfo app = {.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO, .pApplicationName = "totk-install",
.apiVersion = VK_API_VERSION_1_3};
VkInstanceCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, .pApplicationInfo = &app};
VkInstance inst = VK_NULL_HANDLE;
VkResult r = create ? create(&ci, NULL, &inst) : VK_ERROR_INITIALIZATION_FAILED;
if (r != VK_SUCCESS) {
printf("\"instance\": %d, \"devices\": []}\n", (int)r);
return 0;
}
PFN_vkEnumeratePhysicalDevices enumerate = (PFN_vkEnumeratePhysicalDevices)gipa(inst, "vkEnumeratePhysicalDevices");
PFN_vkGetPhysicalDeviceProperties props = (PFN_vkGetPhysicalDeviceProperties)gipa(inst, "vkGetPhysicalDeviceProperties");
PFN_vkDestroyInstance destroy = (PFN_vkDestroyInstance)gipa(inst, "vkDestroyInstance");
VkPhysicalDevice devs[16];
uint32_t n = 16;
if (enumerate(inst, &n, devs) < 0) n = 0;
printf("\"instance\": 0, \"devices\": [");
for (uint32_t i = 0; i < n; i++) {
VkPhysicalDeviceProperties p;
props(devs[i], &p);
printf("%s{\"name\": ", i ? ", " : "");
json_string(p.deviceName);
printf(", \"api\": [%u, %u, %u], \"type\": %d}", VK_API_VERSION_MAJOR(p.apiVersion),
VK_API_VERSION_MINOR(p.apiVersion), VK_API_VERSION_PATCH(p.apiVersion), (int)p.deviceType);
}
printf("]}\n");
destroy(inst, NULL);
return 0;
}
+69
View File
@@ -0,0 +1,69 @@
"""totk.ini: set the game language and keep everything else (keys and syntax as runtime/cfg.c reads them).
A new file starts from totk.ini.example, whose lines are all commented defaults. The language goes into `lang`
(a Switch language code, runtime/ov_system.c): an active `lang = ...` line is rewritten in place, else the commented
`# lang = ...` line of the example is activated, else the line is added to [language] (created at the end if
missing). Other lines, comments and line order stay as they are.
"""
import os
import re
from pathlib import Path
# runtime/ov_system.c: the codes TOTK_LANG accepts.
LANGUAGES = ('ja', 'en-US', 'en-GB', 'fr', 'fr-CA', 'de', 'it', 'es', 'es-419', 'nl', 'pt', 'pt-BR', 'ru', 'ko',
'zh-CN', 'zh-TW', 'zh-Hans', 'zh-Hant')
# Their text falls back to the Switch shared fonts (runtime/ov_system.c), which the installer does not extract.
NEED_SHARED_FONTS = ('ko', 'zh-CN', 'zh-TW', 'zh-Hans', 'zh-Hant')
ACTIVE = re.compile(r'^(\s*)(?:totk_)?lang(\s*=\s*)(.*?)(\s*)$', re.IGNORECASE)
COMMENTED = re.compile(r'^\s*[#;]\s*(?:totk_)?lang\s*=.*$', re.IGNORECASE)
SECTION = re.compile(r'^\s*\[(.*)\]\s*$')
def set_language(text, lang):
"""text with `lang = <lang>`; see the module docstring."""
lines = text.splitlines()
for i, line in enumerate(lines):
m = ACTIVE.match(line)
if m:
lines[i] = f'{m.group(1)}lang{m.group(2)}{lang}'
return '\n'.join(lines) + '\n'
for i, line in enumerate(lines):
if COMMENTED.match(line):
lines[i] = f'lang = {lang}'
return '\n'.join(lines) + '\n'
for i, line in enumerate(lines):
m = SECTION.match(line)
if m and m.group(1).strip().lower() == 'language':
lines.insert(i + 1, f'lang = {lang}')
return '\n'.join(lines) + '\n'
if lines and lines[-1].strip():
lines.append('')
lines += ['[language]', f'lang = {lang}']
return '\n'.join(lines) + '\n'
def get_language(text):
for line in text.splitlines():
m = ACTIVE.match(line)
if m:
return m.group(3)
return None
def write(dest, lang, example=None):
"""Write dest/totk.ini with lang; an existing file keeps its other lines. Returns the path."""
path = Path(dest) / 'totk.ini'
if path.is_file():
text = path.read_text(encoding='utf-8-sig')
elif example and Path(example).is_file():
text = Path(example).read_text(encoding='utf-8')
else:
text = ''
new = set_language(text, lang)
if path.is_file() and new == text:
return path
tmp = path.with_name('totk.ini.tmp')
tmp.write_text(new, encoding='utf-8', newline='\n')
os.replace(tmp, path)
return path
+38
View File
@@ -0,0 +1,38 @@
/* TotK-Setup.exe: starts python\pythonw.exe -m installer.gui with this exe's folder as the working directory.
No console window. The two message strings come from the catalog (installer/text/<lang>.json, ids launcher.title and
launcher.python_missing); tools/make_release.py writes them into launcher_text.h before it compiles this file.
Build: llvm/bin/clang -O2 -mwindows launcher.c -o TotK-Setup.exe */
#include <windows.h>
#include <wchar.h>
#include "launcher_text.h"
#define LANGID_RUSSIAN 0x19
#define PATH_CHARS (MAX_PATH * 2)
static void fail(void)
{
int russian = (GetUserDefaultUILanguage() & 0x3ff) == LANGID_RUSSIAN;
MessageBoxW(NULL, russian ? LAUNCHER_PYTHON_MISSING_RU : LAUNCHER_PYTHON_MISSING_EN,
russian ? LAUNCHER_TITLE_RU : LAUNCHER_TITLE_EN, MB_OK | MB_ICONERROR);
}
int WINAPI WinMain(HINSTANCE instance, HINSTANCE previous, LPSTR command_line, int show)
{
wchar_t folder[PATH_CHARS], python[PATH_CHARS], command[PATH_CHARS + 64];
DWORD n = GetModuleFileNameW(NULL, folder, PATH_CHARS);
STARTUPINFOW si = { sizeof si };
PROCESS_INFORMATION pi;
(void)instance; (void)previous; (void)command_line; (void)show;
if (n == 0 || n >= PATH_CHARS) { fail(); return 1; }
while (n > 0 && folder[n - 1] != L'\\') n--;
folder[n > 0 ? n - 1 : 0] = 0;
if (swprintf(python, PATH_CHARS, L"%ls\\python\\pythonw.exe", folder) < 0 ||
GetFileAttributesW(python) == INVALID_FILE_ATTRIBUTES) { fail(); return 1; }
if (swprintf(command, PATH_CHARS + 64, L"\"%ls\" -m installer.gui", python) < 0) { fail(); return 1; }
if (!CreateProcessW(python, command, NULL, NULL, FALSE, 0, NULL, folder, &si, &pi)) { fail(); return 1; }
CloseHandle(pi.hThread);
CloseHandle(pi.hProcess);
return 0;
}
+113
View File
@@ -0,0 +1,113 @@
"""Where the installer finds what it runs: the one place that maps the release folder and a repository checkout.
Release (docs/installer.md "Release layout"), root = the unpacked folder:
python/python.exe runs every tool (numpy, lz4, cryptography, compression.zstd); its ._pth puts .., ../tools
and ../recomp on sys.path, so `python -m installer` imports from the release folder
recomp/, tools/ the lifter and the scripts; recomp/patches-v121.txt, tools/data/expected-v1.2.1.json
llvm/ llvm-mingw (bin/clang.exe)
runtime/ libtotkrt.a, the headers and install-side sources (romfs.c); runtime/deps/ holds
include/ + lib/ of Vulkan, SDL3, Opus, libvpx and VMA (build_game.py --deps)
bin/ totk_xlate.exe (+ totk_xlate.id), hactool.exe, ninja.exe, SDL3.dll, libopus-0.dll
data/shader_list.tsv the MacNativeGames shader list
totk.ini.example
VERSION the release version, one line
temp/ the default temp folder (created)
Dev mode, root = a repository checkout (no python/python.exe): build/py314-embed/py/python.exe (the embeddable
Python the release ships, with the same packages), else this Python; toolchain/llvm-mingw-*/ and toolchain/deps-mingw;
runtime/ as the sources with build/install-rtlib/libtotkrt.a (tools/build_game.py --make-runtime-lib --out
build/install-rtlib); MSYS2 UCRT64 ninja.exe; toolchain/hactool; build/xlate/totk_xlate.exe; the shader list from
refs/MacNativeGames; build/install-temp. The version is dev-<commit>.
"""
import os
import subprocess
import sys
from dataclasses import dataclass, field
from pathlib import Path
EXE = '.exe' if os.name == 'nt' else ''
UCRT = Path(r'C:\msys64\ucrt64\bin')
SHADER_LIST_REF = 'origin/HEAD:games/totk/assets/shader_list.tsv' # dev: tools/prepare_shaders.py LIST_REF
@dataclass(frozen=True)
class Layout:
root: Path
dev: bool
release: str
python: Path # runs the tools
tools: Path
recomp: Path
patches: Path # code patch table for recomp/gen.py
expected: Path # expected hashes of the supported update
llvm: Path # llvm-mingw tree
deps: Path # include/ and lib/ for the link
runtime: Path # runtime headers and install-side sources
runtime_lib: Path # libtotkrt.a
ninja: Path
hactool: Path
xlate: Path # totk_xlate.exe; a totk_xlate.id beside it is copied too
dlls: tuple = field(default_factory=tuple)
shader_list: Path | None = None # None: dev, read from refs/MacNativeGames through git
ini_example: Path | None = None
temp: Path | None = None
@property
def clang(self):
return self.llvm / 'bin' / ('clang' + EXE)
def missing(self):
"""Paths the steps need that do not exist."""
need = [self.python, self.tools / 'extract.py', self.tools / 'build_game.py', self.tools / 'prepare_shaders.py',
self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.patches, self.expected, self.clang,
self.deps / 'include', self.deps / 'lib', self.runtime, self.runtime_lib, self.ninja, self.hactool,
self.xlate, *self.dlls]
if self.shader_list:
need.append(self.shader_list)
return [str(p) for p in need if not p.exists()]
def is_release(root):
return (root / 'python' / ('python' + EXE)).is_file()
def release_layout(root):
version = root / 'VERSION'
return Layout(
root=root, dev=False,
release=version.read_text(encoding='utf-8').strip() if version.is_file() else 'unknown',
python=root / 'python' / ('python' + EXE),
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
llvm=root / 'llvm', deps=root / 'runtime' / 'deps', runtime=root / 'runtime',
runtime_lib=root / 'runtime' / 'libtotkrt.a',
ninja=root / 'bin' / ('ninja' + EXE), hactool=root / 'bin' / ('hactool' + EXE),
xlate=root / 'bin' / ('totk_xlate' + EXE),
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll'),
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example', temp=root / 'temp')
def dev_layout(root):
commit = subprocess.run(['git', '-C', str(root), 'rev-parse', '--short', 'HEAD'], capture_output=True,
text=True).stdout.strip()
embed = root / 'build' / 'py314-embed' / 'py' / ('python' + EXE)
trees = sorted(p.parent.parent for p in (root / 'toolchain').glob('llvm-mingw-*/llvm-mingw-*/bin/clang' + EXE))
deps = root / 'toolchain' / 'deps-mingw'
return Layout(
root=root, dev=True, release=f'dev-{commit or "unknown"}',
python=embed if embed.is_file() else Path(sys.executable),
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
llvm=trees[-1] if trees else root / 'toolchain' / 'llvm-mingw-missing', deps=deps,
runtime=root / 'runtime', runtime_lib=root / 'build' / 'install-rtlib' / 'libtotkrt.a',
ninja=UCRT / ('ninja' + EXE), hactool=root / 'toolchain' / 'hactool' / ('hactool' + EXE),
xlate=root / 'build' / 'xlate' / ('totk_xlate' + EXE),
dlls=tuple(sorted((deps / 'bin').glob('*.dll'))),
shader_list=None, ini_example=root / 'totk.ini.example', temp=root / 'build' / 'install-temp')
def find(root=None):
"""The layout of root (default: the folder above this package)."""
root = Path(root or Path(__file__).resolve().parent.parent).resolve()
return release_layout(root) if is_release(root) else dev_layout(root)
+151
View File
@@ -0,0 +1,151 @@
"""Which steps run, in what order, and the bookkeeping around them: fresh install, resume and update are one rule.
Each step has a key: a fingerprint of everything its output depends on, computed before anything runs (a step's key
includes the keys of the steps it reads from, so a change upstream reaches every step below it). A step is skipped
when totk-install.json records it with the same key and its output still matches the record (Step.current, cheap).
Otherwise it runs. A transient step (its output lives in the temp folder and only the next step reads it) runs only
when the next step runs. `always` steps (check, cleanup) run every time.
Update mode is the same rule with a newer release: the extraction's key depends only on the expected hashes, so a
release that changes the lifter or the runtime keeps romfs/ and exefs/ and rebuilds from lift on. Before keeping the
output of a step recorded by another release (or with --verify), the check step calls Step.verify (expensive: the full
RomFS hash); a mismatch makes the step run again.
A failed or cancelled run leaves the finished steps recorded; the next run starts at the first step that is not.
"""
import datetime
import traceback
from . import state as statefile
class InstallError(Exception):
"""A failure with a message id (installer/text/en.json) and its parameters."""
def __init__(self, id, **params):
super().__init__(id)
self.id, self.params = id, params
class Step:
id = ''
transient = False
always = False
def key(self, ctx):
"""Fingerprint of the step's inputs; may read ctx.keys of earlier steps."""
return ''
def current(self, ctx, record):
"""True when the recorded output is still in place (cheap checks only)."""
return True
def verify(self, ctx, record):
"""True when the recorded output passes the full check (hashes); called before keeping another release's
output."""
return True
def run(self, ctx):
"""Do the work. Returns fields to record with the step."""
return {}
class Context:
def __init__(self, layout, dest, temp, reporter, options, state=None, verify=False):
self.layout, self.dest, self.temp, self.reporter, self.options = layout, dest, temp, reporter, options
self.state = state or statefile.empty(layout.release)
self.verify = verify
self.mode = 'install' if state is None else 'resume' if state.get('release') == layout.release else 'update'
self.keys, self.runs, self.reasons, self.steps, self.forced = {}, {}, {}, [], set()
def record(self, step_id):
return self.state['steps'].get(step_id)
def will_run(self, step_id):
return self.runs.get(step_id, False)
def save(self):
self.dest.mkdir(parents=True, exist_ok=True)
statefile.save(self.dest, self.state)
def plan(self, forced=()):
"""Fill keys (once), runs and reasons. forced: step ids that must run."""
if not self.keys:
for s in self.steps:
self.keys[s.id] = s.key(self)
for s in self.steps:
rec = self.record(s.id)
if s.always or s.id in forced:
self.runs[s.id] = True
else:
self.runs[s.id] = not (rec and rec.get('key') == self.keys[s.id] and s.current(self, rec))
self.reasons[s.id] = '' if self.runs[s.id] else 'current'
for i in range(len(self.steps) - 2, -1, -1):
s, after = self.steps[i], self.steps[i + 1]
if s.transient and self.runs[s.id] and not self.runs[after.id]:
self.runs[s.id], self.reasons[s.id] = False, 'not_needed'
self.forced = set(forced)
def verify_kept(self, progress_step):
"""Full check of every kept step output that another release recorded (or of all with --verify). A step that
fails it is planned to run. Returns the ids that failed."""
failed = []
for s in self.steps:
rec = self.record(s.id)
if self.runs[s.id] or self.reasons[s.id] != 'current' or s.always:
continue
if not (self.verify or rec.get('release') != self.layout.release):
continue
if not s.verify(self, rec):
failed.append(s.id)
if failed:
for f in failed:
self.reporter.warning('state.output_changed', step=f)
self.plan(self.forced | set(failed))
return failed
def now_utc():
return datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
def run_steps(steps, ctx):
"""Run the plan. Returns 'ok', 'failed' or 'cancelled'; events and the log say why."""
rep = ctx.reporter
ctx.steps = steps
current = None
try:
ctx.plan()
rep.emit('plan', run=[s.id for s in steps if ctx.runs[s.id]],
skip={s.id: ctx.reasons[s.id] for s in steps if not ctx.runs[s.id]})
for s in steps:
if not ctx.runs[s.id]:
rep.step(s.id, 'skipped', reason=ctx.reasons[s.id])
continue
current = s.id
rep.step(s.id, 'started')
t0 = rep.elapsed()
fields = s.run(ctx) or {}
seconds = round(rep.elapsed() - t0, 1)
if not s.always:
ctx.state['steps'][s.id] = {'key': ctx.keys[s.id], 'release': ctx.layout.release,
'finished': now_utc(), 'seconds': seconds, **fields}
ctx.save() # the first save, after the check, creates the install folder
rep.step(s.id, 'done', seconds=seconds)
current = None
ctx.state['release'] = ctx.layout.release
ctx.save()
return 'ok'
except InstallError as e:
rep.error(e.id, e.params, step=current)
return 'failed'
except statefile.StateError as e:
rep.error(e.id, {'path': str(e.path)}, step=current)
return 'failed'
except KeyboardInterrupt:
rep.log(f'cancelled during {current}')
return 'cancelled'
except Exception as e:
rep.log(traceback.format_exc())
rep.error('internal', {'exception': type(e).__name__, 'detail': str(e)}, step=current)
return 'failed'
+54
View File
@@ -0,0 +1,54 @@
"""totk-install.json in the install folder: what built it and which steps finished.
{"format": 1, "release": "...", "lang": "en-US",
"inputs": {"base": {"name", "size", "sha256"}, "update": {...}},
"steps": {"<step>": {"key": "<input fingerprint>", "release": "...", "finished": "<UTC time>", "seconds": 12.3,
...the step's own fields}}}
A step's key fingerprints everything its output depends on (runner.py), so a step whose recorded key equals the
current one and whose output still checks out is not run again. The file is replaced atomically after each step.
No key value or keys file content is ever stored.
"""
import json
import os
from pathlib import Path
NAME = 'totk-install.json'
FORMAT = 1
class StateError(Exception):
def __init__(self, id, path):
super().__init__(id)
self.id, self.path = id, path
def empty(release):
return {'format': FORMAT, 'release': release, 'inputs': {}, 'steps': {}}
def load(dest):
"""The state of dest, or None when the folder has none. StateError for an unreadable or newer file."""
path = Path(dest) / NAME
if not path.is_file():
return None
try:
data = json.loads(path.read_text(encoding='utf-8'))
except (OSError, ValueError):
raise StateError('state.unreadable', path) from None
if not isinstance(data, dict) or not isinstance(data.get('steps'), dict):
raise StateError('state.unreadable', path)
if data.get('format', 0) > FORMAT:
raise StateError('state.newer_format', path)
return data
def save(dest, data):
path = Path(dest) / NAME
tmp = path.with_name(NAME + '.tmp')
with open(tmp, 'w', encoding='utf-8', newline='\n') as f:
json.dump(data, f, indent=1, ensure_ascii=False)
f.write('\n')
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
+629
View File
@@ -0,0 +1,629 @@
"""The six setup steps of docs/installer.md: check, extract, lift, compile, prepare, cleanup.
Extraction runs in this process through tools/extract.py (the package's sibling tools/, which ships with it); lift,
compile and prepare run the bundled Python on recomp/gen.py, tools/build_game.py, tools/make_romfs_patches.py and
tools/prepare_shaders.py, whose output goes to <temp>/logs/<step>-<tool>.log and is read for progress. No Win32 call
here: the host check is a small C program built with the bundled clang (hostcheck.c).
Install folder: exefs/, romfs/ (romfs.bin, index.json), profile/, totk.exe and its DLLs, romfs-index.bin,
romfs-patches/, totk_xlate.exe, the shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json.
Temp folder (<temp>/<dest name>-<hash>): check/ (ExeFS of both inputs, the host check), gen/ (C), build/ (objects),
logs/ (kept by the cleanup).
"""
import hashlib
import json
import os
import re
import shutil
import subprocess
import sys
import threading
import time
from pathlib import Path
from . import ini
from .layout import EXE
from .runner import InstallError, Step
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / 'tools'))
import extract # noqa: E402
from extract_exefs import (common_tickets, container_entries, load_keys, nca_ctr_key, nca_inventory, # noqa: E402
program_nca)
GiB = 1 << 30
DEST_EXTRA = 4 * GiB # beside the RomFS: shader and pipeline caches (about 2 GB), exefs, totk.exe, seed files
TEMP_NEED = 3 * GiB # generated C (about 0.9 GB) and objects
UPDATE_VERSION = '1.2.1'
HOSTCHECK = Path(__file__).resolve().parent / 'hostcheck.c'
STAMP = '.install-key'
# ---- helpers
def file_sha256(path, progress=None):
"""SHA-256 hex of a file; progress(done, total) after each block."""
h = hashlib.sha256()
total = path.stat().st_size
done = 0
with open(path, 'rb', buffering=0) as f:
buf = bytearray(8 << 20)
view = memoryview(buf)
while n := f.readinto(buf):
h.update(view[:n])
done += n
if progress:
progress(done, total)
return h.hexdigest()
def fingerprint(*parts):
"""SHA-256 over strings and the contents of files and directory trees (a missing path counts as missing)."""
h = hashlib.sha256()
for p in parts:
if isinstance(p, Path):
if p.is_dir():
for f in sorted(p.rglob('*')):
if f.is_file() and '__pycache__' not in f.parts:
h.update(f.relative_to(p).as_posix().encode() + b'\0' + bytes.fromhex(file_sha256(f)))
elif p.is_file():
h.update(p.name.encode() + b'\0' + bytes.fromhex(file_sha256(p)))
else:
h.update(b'missing\0' + p.name.encode())
else:
h.update(str(p).encode())
h.update(b'\1')
return h.hexdigest()
def remove(path):
if path.is_dir() and not path.is_symlink():
shutil.rmtree(path)
elif path.exists() or path.is_symlink():
path.unlink()
def replace_file(src, dst):
"""Copy src over dst through a temporary name, so a cancelled copy never leaves half a file under dst's name."""
tmp = dst.with_name(dst.name + '.tmp')
shutil.copy2(src, tmp)
os.replace(tmp, dst)
def kill_tree(proc):
if proc.poll() is not None:
return
if os.name == 'nt':
subprocess.run(['taskkill', '/T', '/F', '/PID', str(proc.pid)], capture_output=True)
else:
proc.kill()
def run_tool(ctx, step, script, args, parse=None, cwd=None):
"""Run a tool script with the bundled Python. Its output goes to <temp>/logs/<step>-<script>.log and, line by line,
to parse(line). Returns (exit code, log path)."""
logs = ctx.temp / 'logs'
logs.mkdir(parents=True, exist_ok=True)
log = logs / f'{step}-{Path(script).stem}.log'
command = [str(ctx.layout.python), '-c', extract.RUN_SCRIPT, str(script), *map(str, args)]
ctx.reporter.log('run ' + ' '.join(command[3:]))
env = dict(os.environ, PYTHONUNBUFFERED='1', PYTHONIOENCODING='utf-8')
with open(log, 'w', encoding='utf-8', newline='\n') as out:
proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=cwd or ctx.layout.root,
env=env, text=True, encoding='utf-8', errors='replace', bufsize=1)
try:
for line in proc.stdout:
out.write(line)
if parse:
parse(line.rstrip())
code = proc.wait()
except BaseException:
kill_tree(proc)
raise
ctx.reporter.log(f'{Path(script).name}: exit code {code}; output in {log}')
return code, log
def free_bytes(path):
"""Free bytes on the disk of path (its nearest existing parent) and that parent's device."""
p = Path(path)
while not p.exists():
p = p.parent
return shutil.disk_usage(p).free, os.stat(p).st_dev
def expected_update():
return extract.load_expected()[UPDATE_VERSION]
def shader_list_text(layout):
if layout.shader_list:
return layout.shader_list.read_text(encoding='utf-8')
refs = layout.root / 'refs' / 'MacNativeGames'
r = subprocess.run(['git', '-C', str(refs), 'show', 'origin/HEAD:games/totk/assets/shader_list.tsv'],
capture_output=True, encoding='utf-8')
if r.returncode:
raise InstallError('prepare.shader_list_missing')
return r.stdout
class Watch(threading.Thread):
"""Polls a measure() of bytes written while a long call runs and reports it as progress."""
def __init__(self, reporter, step):
super().__init__(daemon=True)
self.reporter, self.step = reporter, step
self.phase, self.measure, self.total = None, None, None
self.stopped = threading.Event()
def set(self, phase, measure=None, total=None):
self.phase, self.measure, self.total = phase, measure, total
self.reporter.progress(self.step, phase, 0 if total else None, total, 'bytes' if total else None, force=True)
def run(self):
while not self.stopped.wait(1.0):
phase, measure, total = self.phase, self.measure, self.total
if phase and measure:
try:
done = min(measure(), total)
except OSError:
continue
self.reporter.progress(self.step, phase, done, total, 'bytes')
def stop(self):
self.stopped.set()
def size_of(paths):
return sum(p.stat().st_size for p in paths if p.is_file())
# ---- 1. check
class Check(Step):
id = 'check'
always = True
def run(self, ctx):
layout, rep, dest = ctx.layout, ctx.reporter, ctx.dest
missing = layout.missing()
if missing:
raise InstallError('check.release_incomplete', paths=missing[:5])
if dest.exists() and not dest.is_dir():
raise InstallError('check.dest_not_folder', path=str(dest))
if ctx.mode == 'install' and dest.is_dir() and any(dest.iterdir()):
raise InstallError('check.dest_not_empty', path=str(dest))
lang = ctx.options.lang or ctx.state.get('lang')
if not lang:
raise InstallError('check.lang_missing')
if lang not in ini.LANGUAGES:
raise InstallError('check.lang_unknown', lang=lang, known=', '.join(ini.LANGUAGES))
if lang in ini.NEED_SHARED_FONTS:
rep.warning('warn.lang_needs_fonts', lang=lang)
ctx.lang = lang
ctx.temp.mkdir(parents=True, exist_ok=True)
rep.progress('check', 'host', force=True)
self.host(ctx)
rep.progress('check', 'verify', force=True)
ctx.verify_kept('check')
ncas = 0
if ctx.will_run('extract'):
rep.progress('check', 'inputs', force=True)
ncas = self.inputs(ctx)
self.space(ctx, ncas)
if ctx.will_run('extract'):
rep.progress('check', 'versions', force=True)
self.versions(ctx)
return {}
def host(self, ctx):
layout, work = ctx.layout, ctx.temp / 'check'
work.mkdir(parents=True, exist_ok=True)
exe = work / ('hostcheck' + EXE)
command = [str(layout.clang), '-O1', str(HOSTCHECK), '-I', str(layout.deps / 'include'), '-o', str(exe)]
if os.name != 'nt':
command.append('-ldl')
r = subprocess.run(command, capture_output=True, text=True, errors='replace')
if r.returncode:
ctx.reporter.log(r.stdout + r.stderr)
raise InstallError('check.toolchain_broken', detail=(r.stderr or r.stdout).strip()[-400:])
r = subprocess.run([str(exe)], capture_output=True, text=True, errors='replace')
try:
host = json.loads(r.stdout)
except ValueError:
raise InstallError('check.host_probe_failed', code=r.returncode) from None
ctx.reporter.log(f'host: {r.stdout.strip()}')
if not host['x86_64_v3']:
raise InstallError('check.cpu_no_avx2')
if not host['loader']:
raise InstallError('check.vulkan_missing')
devices = host['devices']
if not devices:
raise InstallError('check.vulkan_no_device')
if not any(tuple(d['api'][:2]) >= (1, 3) for d in devices):
raise InstallError('check.vulkan_too_old',
devices=', '.join(f'{d["name"]} ({d["api"][0]}.{d["api"][1]})' for d in devices))
ctx.host = host
def inputs(self, ctx):
"""Open both files with the keys as the extraction will. Returns the bytes of both Program NCAs."""
o = ctx.options
for name in ('base', 'update', 'keys'):
path = getattr(o, name)
if not path:
raise InstallError('check.inputs_needed')
if not Path(path).is_file():
raise InstallError('check.input_missing', input=name, path=str(path))
try:
keys = load_keys(Path(o.keys))
except (ValueError, OSError, UnicodeDecodeError):
raise InstallError('check.keys_unreadable') from None
if 'header_key' not in keys:
raise InstallError('check.keys_no_header_key')
total = 0
for name, is_update in (('base', False), ('update', True)):
path = Path(getattr(o, name))
with path.open('rb') as f:
total += self.open_input(f, path.stat().st_size, keys, name, is_update)
del keys
return total
@staticmethod
def open_input(f, size, keys, name, is_update):
try:
kind, entries = container_entries(f, size)
except ValueError:
raise InstallError('check.not_game_file', input=name) from None
if is_update and kind != 'nsp':
raise InstallError('check.update_not_nsp')
try:
tickets = common_tickets(f, entries)
except ValueError as e:
if 'Personalized' in str(e):
raise InstallError('check.ticket_personalized', input=name) from None
raise InstallError('check.file_unreadable', input=name, detail=str(e)) from None
try:
records = nca_inventory(f, entries, keys)
except ValueError as e:
if 'NCA3' in str(e):
raise InstallError('check.keys_wrong', input=name) from None
raise InstallError('check.file_unreadable', input=name, detail=str(e)) from None
try:
record, header = program_nca(records, update=is_update)
except ValueError as e:
if is_update and 'BKTR' in str(e):
raise InstallError('check.update_is_base') from None
raise InstallError('check.update_not_totk' if is_update else 'check.base_not_totk', detail=str(e)) from None
try:
nca_ctr_key(header, keys, tickets)
except ValueError as e:
if 'ticket' in str(e):
raise InstallError('check.ticket_missing', input=name) from None
raise InstallError('check.keys_too_old', input=name,
generation=max(header[0x206], header[0x220])) from None
return record['size']
def space(self, ctx, ncas):
dest_need = temp_need = 0
if ctx.will_run('extract'):
dest_need += expected_update()['romfs']['size'] + ncas
romfs = ctx.dest / 'romfs' / 'romfs.bin'
if romfs.is_file(): # a resumed extraction replaces it
dest_need -= romfs.stat().st_size
if ctx.will_run('prepare') or ctx.will_run('extract'):
dest_need += DEST_EXTRA
if ctx.will_run('lift') or ctx.will_run('compile'):
temp_need += TEMP_NEED
dest_free, dest_dev = free_bytes(ctx.dest)
temp_free, temp_dev = free_bytes(ctx.temp)
ctx.reporter.log(f'space: install folder needs {dest_need / GiB:.1f} GiB of {dest_free / GiB:.1f} free, '
f'temp folder {temp_need / GiB:.1f} GiB of {temp_free / GiB:.1f}')
if dest_dev == temp_dev:
dest_need += temp_need
temp_need = 0
ctx.reporter.emit('space', dest=str(ctx.dest), need=dest_need, free=dest_free, temp=str(ctx.temp),
temp_need=temp_need, temp_free=temp_free, same_disk=dest_dev == temp_dev)
for path, need, free in ((ctx.dest, dest_need, dest_free), (ctx.temp, temp_need, temp_free)):
if need > free:
raise InstallError('check.disk_space', path=str(path), need_gb=round(need / 1e9, 1),
free_gb=round(free / 1e9, 1))
def versions(self, ctx):
"""Extract both ExeFS into the temp folder (verifying every block) and identify the versions by hash."""
o, work = ctx.options, ctx.temp / 'check'
logs = ctx.temp / 'logs'
logs.mkdir(parents=True, exist_ok=True)
versions = extract.load_expected()
for name, is_update in (('base', False), ('update', True)):
out = work / name
remove(out)
try:
extract.extract_exefs(Path(getattr(o, name)), Path(o.keys), out, logs, update=is_update)
except ValueError:
raise InstallError('check.exefs_failed', input=name, log=str(logs / f'extract_exefs-{name}.log')) \
from None
problems = extract.exefs_problems(work / 'base' / 'exefs', versions['1.0.0'])
if problems:
raise InstallError('check.base_wrong_version', modules=', '.join(problems))
try:
found = extract.identify_update(work / 'update' / 'exefs', versions)['display_version']
except ValueError:
found = 'unknown'
if found != UPDATE_VERSION:
raise InstallError('check.update_wrong_version', found=found, wanted=UPDATE_VERSION)
# ---- 2. extract
class Extract(Step):
id = 'extract'
def key(self, ctx):
# The output is whatever matches the expected hashes, whichever files and tools produced it.
return fingerprint('extract-1', ctx.layout.expected)
def current(self, ctx, record):
expected = expected_update()
romfs = ctx.dest / 'romfs'
if extract.exefs_problems(ctx.dest / 'exefs', expected) or not extract.romfs_verified(romfs, expected):
return False
st = (romfs / 'romfs.bin').stat()
return st.st_size == record.get('romfs_size') and st.st_mtime_ns == record.get('romfs_mtime_ns')
def verify(self, ctx, record):
image = ctx.dest / 'romfs' / 'romfs.bin'
rep = ctx.reporter
digest = file_sha256(image, lambda d, t: rep.progress('check', 'verify_romfs', d, t, 'bytes'))
return digest == expected_update()['romfs']['sha256']
def run(self, ctx):
o, dest, rep = ctx.options, ctx.dest, ctx.reporter
expected = expected_update()
base, update, keys = Path(o.base), Path(o.update), Path(o.keys)
inputs = {}
for name, path in (('base', base), ('update', update)):
known = ctx.state['inputs'].get(name, {})
st = path.stat()
if known.get('size') == st.st_size and known.get('mtime_ns') == st.st_mtime_ns and known.get('sha256'):
inputs[name] = known
continue
digest = file_sha256(path, lambda d, t, n=name: rep.progress('extract', 'hash_' + n, d, t, 'bytes'))
inputs[name] = {'name': path.name, 'size': st.st_size, 'mtime_ns': st.st_mtime_ns, 'sha256': digest}
ctx.state['inputs'] = inputs
ctx.save()
dest.mkdir(parents=True, exist_ok=True)
for stale in dest.glob('update-merge-*'): # a killed merge leaves its NCA copies (and key copies) behind
remove(stale)
rep.progress('extract', 'exefs', force=True)
staged = ctx.temp / 'check' / 'update' / 'exefs'
if not staged.is_dir():
remove(ctx.temp / 'check' / 'update')
try:
extract.extract_exefs(update, keys, ctx.temp / 'check' / 'update', ctx.temp / 'logs', update=True)
except ValueError:
raise InstallError('check.exefs_failed', input='update',
log=str(ctx.temp / 'logs' / 'extract_exefs-update.log')) from None
remove(dest / 'exefs')
shutil.move(staged, dest / 'exefs')
problems = extract.exefs_problems(dest / 'exefs', expected)
if problems:
raise InstallError('extract.exefs_mismatch', modules=', '.join(problems))
if extract.sha256(dest / 'exefs' / 'main.npdm') != expected['exefs_sha256']['main.npdm']:
rep.warning('warn.npdm_differs')
romfs = dest / 'romfs'
if self.id in ctx.forced:
remove(romfs) # the full hash failed: index.json still claims the extraction's hash
if not extract.romfs_verified(romfs, expected):
self.merge(ctx, base, update, keys, romfs, expected)
report = json.loads((romfs / 'index.json').read_text())
if report['romfs_size'] != expected['romfs']['size'] or report['romfs_sha256'] != expected['romfs']['sha256']:
raise InstallError('extract.romfs_mismatch', size=report['romfs_size'], sha256=report['romfs_sha256'][:16])
(dest / 'profile').mkdir(exist_ok=True)
st = (romfs / 'romfs.bin').stat()
return {'romfs_size': st.st_size, 'romfs_mtime_ns': st.st_mtime_ns}
def merge(self, ctx, base, update, keys, romfs, expected):
dest, logs = ctx.dest, ctx.temp / 'logs'
ncas = extract.merge_space(base, update, keys)
watch = Watch(ctx.reporter, 'extract')
def phase(name):
if name == 'copy':
watch.set('copy', lambda: size_of(dest.glob('update-merge-*/*.nca')), ncas)
elif name == 'merge':
watch.set('merge', lambda: size_of(dest.glob('update-merge-*/result/romfs.bin')),
expected['romfs']['size'])
else:
watch.set(name)
watch.start()
try:
extract.merge_update_romfs(base, update, keys, ctx.layout.hactool, romfs, logs, phase=phase)
except ValueError as e:
raise InstallError('extract.romfs_merge_failed', detail=str(e), log=str(logs / 'merge_romfs-'
f'{dest.name}.log')) from None
finally:
watch.stop()
# ---- 3. lift
EMITTED = re.compile(r'^emitted (\d+)/(\d+) chunks')
DISCOVERED = re.compile(r'^(\w+): \d+ functions \(.*discovery')
class Lift(Step):
id = 'lift'
transient = True
def key(self, ctx):
return fingerprint('lift-1', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches)
def current(self, ctx, record):
gen = ctx.temp / 'gen'
stamp = gen / STAMP
return (gen / 'layout.json').is_file() and stamp.is_file() and stamp.read_text().strip() == ctx.keys['lift']
def run(self, ctx):
layout, rep, gen = ctx.layout, ctx.reporter, ctx.temp / 'gen'
remove(gen)
rep.progress('lift', 'discover', force=True)
def parse(line):
m = EMITTED.match(line)
if m:
rep.progress('lift', 'emit', int(m.group(1)), int(m.group(2)), 'chunks')
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py',
['--exefs', ctx.dest / 'exefs', '--out', gen, '--patches', layout.patches], parse)
if code:
raise InstallError('lift.failed', code=code, log=str(log))
(gen / STAMP).write_text(ctx.keys['lift'] + '\n')
return {}
# ---- 4. compile
NINJA_LINE = re.compile(r'^\[(\d+)/(\d+)\] ')
class Compile(Step):
id = 'compile'
def key(self, ctx):
layout = ctx.layout
clang = subprocess.run([str(layout.clang), '--version'], capture_output=True, text=True).stdout
tools = [layout.tools / n for n in ('build_game.py', 'romfs_index.py', 'check_content.py')]
return fingerprint('compile-1', ctx.keys['lift'], layout.runtime, layout.runtime_lib, layout.deps, *tools,
clang, *[d.name for d in layout.dlls])
def current(self, ctx, record):
exe = ctx.dest / ('totk' + EXE)
if not exe.is_file() or not (ctx.dest / 'romfs-index.bin').is_file():
return False
if not all((ctx.dest / d.name).is_file() for d in ctx.layout.dlls):
return False
return file_sha256(exe) == record.get('exe_sha256')
def run(self, ctx):
layout, rep, dest = ctx.layout, ctx.reporter, ctx.dest
build = ctx.temp / 'build'
phase = ['compile']
def parse(line):
m = NINJA_LINE.match(line)
if m:
rep.progress('compile', phase[0], int(m.group(1)), int(m.group(2)), 'objects')
elif line.startswith('compile: '):
phase[0] = 'link'
rep.progress('compile', 'link', force=True)
rep.progress('compile', 'compile', force=True)
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--deps', layout.deps,
'--runtime', layout.runtime, '--runtime-lib', layout.runtime_lib, '--ninja', layout.ninja,
'--expected', layout.expected, '--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
'--profile', dest / 'profile'], parse)
exe = build / ('totk' + EXE)
if code or not exe.is_file():
raise InstallError('compile.failed', code=code, log=str(log))
try:
replace_file(exe, dest / exe.name)
except PermissionError:
raise InstallError('compile.exe_in_use', path=str(dest / exe.name)) from None
replace_file(build / 'romfs-index.bin', dest / 'romfs-index.bin')
for dll in layout.dlls:
replace_file(dll, dest / dll.name)
return {'exe_sha256': file_sha256(dest / exe.name)}
# ---- 5. prepare
TRANSLATED = re.compile(r'gfx_shader: prepare: (\d+) of (\d+) pairs')
class Prepare(Step):
id = 'prepare'
def key(self, ctx):
layout = ctx.layout
xlate_id = layout.xlate.with_suffix('.id')
tools = [layout.tools / n for n in ('prepare_shaders.py', 'shader_census.py', 'make_romfs_patches.py')]
return fingerprint('prepare-1', ctx.keys['compile'], layout.xlate, xlate_id, shader_list_text(layout), *tools)
def current(self, ctx, record):
dest = ctx.dest
xlate = dest / ctx.layout.xlate.name
return ((dest / 'shader-prepared.bin').is_file() and (dest / 'romfs-patches' / 'manifest.tsv').is_file()
and xlate.is_file() and file_sha256(xlate) == record.get('xlate_sha256'))
def run(self, ctx):
layout, rep, dest = ctx.layout, ctx.reporter, ctx.dest
xlate = dest / layout.xlate.name
if xlate.is_file() and file_sha256(xlate) != file_sha256(layout.xlate):
# Another translator: its cache keys differ, so the old translations would only take space.
for name in ('shader-cache', 'shader-prepared.bin'):
remove(dest / name)
replace_file(layout.xlate, xlate)
xlate_id = layout.xlate.with_suffix('.id')
if xlate_id.is_file():
replace_file(xlate_id, xlate.with_suffix('.id'))
else:
remove(xlate.with_suffix('.id'))
rep.progress('prepare', 'romfs_patches', force=True)
remove(dest / 'romfs-patches')
code, log = run_tool(ctx, 'prepare', layout.tools / 'make_romfs_patches.py',
['--romfs', dest / 'romfs', '--out', dest / 'romfs-patches'])
if code:
raise InstallError('prepare.romfs_patches_failed', code=code, log=str(log))
shader_list = layout.shader_list
if not shader_list:
shader_list = ctx.temp / 'shader_list.tsv'
shader_list.write_text(shader_list_text(layout), encoding='utf-8', newline='\n')
def parse(line):
m = TRANSLATED.search(line)
if m:
rep.progress('prepare', 'translate', int(m.group(1)), int(m.group(2)), 'shaders')
elif 'listed variants' in line:
rep.progress('prepare', 'translate', force=True)
elif 'gfx_shader: prepare:' in line and 'translated' in line and 'programs' in line:
rep.progress('prepare', 'libraries', force=True)
rep.progress('prepare', 'seed', force=True)
code, log = run_tool(ctx, 'prepare', layout.tools / 'prepare_shaders.py',
[dest, '--romfs', dest / 'romfs', '--exefs', dest / 'exefs', '--list', shader_list], parse)
if code or not (dest / 'shader-prepared.bin').is_file():
raise InstallError('prepare.shaders_failed', code=code, log=str(log))
return {'xlate_sha256': file_sha256(xlate)}
# ---- 6. clean up
class Cleanup(Step):
id = 'cleanup'
always = True
def run(self, ctx):
rep, dest = ctx.reporter, ctx.dest
rep.progress('cleanup', 'ini', force=True)
try:
ini.write(dest, ctx.lang, ctx.layout.ini_example)
except OSError as e:
raise InstallError('cleanup.ini_failed', detail=str(e)) from None
ctx.state['lang'] = ctx.lang
(dest / 'profile').mkdir(exist_ok=True)
rep.progress('cleanup', 'temp', force=True)
for stale in dest.glob('update-merge-*'):
remove(stale)
for name in ('gen', 'build', 'check', 'shader_list.tsv'):
remove(ctx.temp / name)
return {}
def all_steps():
return [Check(), Extract(), Lift(), Compile(), Prepare(), Cleanup()]
+170
View File
@@ -0,0 +1,170 @@
{
"step.check": "Checking the computer and the game files",
"step.extract": "Extracting the game",
"step.lift": "Translating the game code to C",
"step.compile": "Compiling totk.exe",
"step.prepare": "Preparing the shaders",
"step.cleanup": "Cleaning up",
"phase.check.host": "Checking the processor and the graphics card",
"phase.check.verify": "Checking the files already installed",
"phase.check.verify_romfs": "Checking the installed game data",
"phase.check.inputs": "Opening the game files with the keys",
"phase.check.versions": "Checking the game versions",
"phase.extract.hash_base": "Computing the checksum of the base game",
"phase.extract.hash_update": "Computing the checksum of the update",
"phase.extract.exefs": "Extracting the game code",
"phase.extract.copy": "Copying the game data",
"phase.extract.merge": "Applying the update to the game data",
"phase.extract.index": "Verifying the game data",
"phase.lift.discover": "Finding the functions",
"phase.lift.emit": "Writing C code",
"phase.compile.compile": "Compiling",
"phase.compile.link": "Linking totk.exe",
"phase.prepare.romfs_patches": "Writing the 60 fps game data patches",
"phase.prepare.seed": "Reading the shaders from the game data",
"phase.prepare.translate": "Translating the shaders",
"phase.prepare.libraries": "Building the shaders for the graphics driver",
"phase.cleanup.ini": "Writing totk.ini",
"phase.cleanup.temp": "Removing temporary files",
"cli.usage": "Invalid command line: {detail}",
"internal": "Unexpected error ({exception}): {detail}",
"state.unreadable": "{path} cannot be read. Choose another folder or delete that file.",
"state.newer_format": "{path} was written by a newer installer. Use that installer or another folder.",
"state.output_changed": "Installed files of step '{step}' changed since they were made; that step runs again.",
"check.release_incomplete": "The installer folder is incomplete; missing: {paths}. Unpack the release again.",
"check.dest_not_folder": "The install location {path} is a file, not a folder.",
"check.dest_not_empty": "The install folder {path} is not empty and holds no earlier installation. Choose an empty folder.",
"check.lang_missing": "Choose the game language.",
"check.lang_unknown": "Unknown language '{lang}'. Known: {known}.",
"check.toolchain_broken": "The bundled compiler does not work: {detail}",
"check.host_probe_failed": "The computer check did not run (exit code {code}).",
"check.cpu_no_avx2": "This processor lacks AVX2 (x86-64-v3), which the game needs. Intel Haswell (2013) or AMD Excavator / Zen and newer have it.",
"check.vulkan_missing": "No Vulkan driver is installed. Install the latest driver for your graphics card.",
"check.vulkan_no_device": "Vulkan finds no graphics card. Install the latest driver for your graphics card.",
"check.vulkan_too_old": "The game needs a graphics card with Vulkan 1.3; found: {devices}. Update the graphics driver.",
"check.inputs_needed": "The game files are needed: choose the base game, the 1.2.1 update and prod.keys.",
"check.input_missing": "File not found ({input}): {path}",
"check.keys_unreadable": "The keys file cannot be read. Choose a prod.keys file.",
"check.keys_no_header_key": "The keys file has no header_key. Dump prod.keys from your console again.",
"check.keys_wrong": "The keys cannot open the {input} file. Use the prod.keys of your console.",
"check.keys_too_old": "The keys file is too old for the {input} file (needs key generation {generation}). Dump prod.keys from a console with newer firmware.",
"check.not_game_file": "The {input} file is not an XCI or NSP game file.",
"check.file_unreadable": "The {input} file cannot be read: {detail}",
"check.ticket_personalized": "The {input} file carries a personalized ticket; a common ticket is needed.",
"check.ticket_missing": "The {input} file lacks the ticket its content needs.",
"check.base_not_totk": "The base game file is not The Legend of Zelda: Tears of the Kingdom ({detail}).",
"check.update_not_nsp": "The update must be an NSP file.",
"check.update_is_base": "The update file holds a base game. Choose the 1.2.1 update NSP.",
"check.update_not_totk": "The update file is not a Tears of the Kingdom update ({detail}).",
"check.exefs_failed": "The game code of the {input} file failed its checks; see {log}.",
"check.base_wrong_version": "The base game is not version 1.0.0 ({modules} differ).",
"check.update_wrong_version": "The update is version {found}; this release needs {wanted}.",
"check.disk_space": "Not enough free space for {path}: {need_gb} GB needed, {free_gb} GB free.",
"extract.exefs_mismatch": "The extracted game code does not match version 1.2.1 ({modules}).",
"extract.romfs_merge_failed": "Applying the update to the game data failed: {detail}. Details: {log}",
"extract.romfs_mismatch": "The extracted game data does not match version 1.2.1 (size {size}, checksum {sha256}...).",
"lift.failed": "Translating the game code failed (exit code {code}). Details: {log}",
"compile.failed": "Compiling totk.exe failed (exit code {code}). Details: {log}",
"compile.exe_in_use": "{path} is in use. Close the game and run the installer again.",
"prepare.shader_list_missing": "The shader list is missing.",
"prepare.romfs_patches_failed": "Writing the game data patches failed (exit code {code}). Details: {log}",
"prepare.shaders_failed": "Preparing the shaders failed (exit code {code}). Details: {log}",
"cleanup.ini_failed": "totk.ini cannot be written: {detail}",
"warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.",
"warn.npdm_differs": "main.npdm differs from the reference 1.2.1 dump; only the code modules are used.",
"ui.title": "TotK Setup",
"ui.intro": "Builds a playable Tears of the Kingdom folder from your own game files. This takes a while and runs once.",
"ui.notice": "The folder this setup builds contains the game. It is for your own use only; do not share or upload it.",
"ui.keys_private": "The keys file is read in place. Its contents are never shown, copied or stored.",
"ui.base": "Base game (NSP or XCI)",
"ui.update": "Update 1.2.1 (NSP)",
"ui.keys": "prod.keys",
"ui.game_language": "Game language",
"ui.dest": "Install folder",
"ui.browse": "Browse...",
"ui.pick_game": "Choose the base game file",
"ui.pick_update": "Choose the 1.2.1 update file",
"ui.pick_keys": "Choose prod.keys",
"ui.pick_dest": "Choose the install folder",
"ui.type_game": "Game files (NSP, XCI)",
"ui.type_nsp": "NSP files",
"ui.type_keys": "Key files (*.keys)",
"ui.type_all": "All files",
"ui.start": "Start",
"ui.cancel": "Cancel",
"ui.cancelling": "Cancelling...",
"ui.show_log": "Show log",
"ui.hide_log": "Hide log",
"ui.close": "Close",
"ui.back": "Back",
"ui.open_folder": "Open folder",
"ui.update_mode": "This folder holds an earlier installation. The setup continues or updates it; the game files are asked for only if they are needed again.",
"ui.need_dest": "Choose the install folder.",
"ui.need_base": "Choose the base game file.",
"ui.need_update": "Choose the 1.2.1 update file.",
"ui.need_keys": "Choose the prod.keys file.",
"ui.need_lang": "Choose the game language.",
"ui.missing_base": "The base game file does not exist.",
"ui.missing_update": "The update file does not exist.",
"ui.missing_keys": "The keys file does not exist.",
"ui.running_install": "Installing",
"ui.running_resume": "Continuing the earlier setup",
"ui.running_update": "Updating the installation",
"ui.state_started": "started",
"ui.state_done": "done",
"ui.state_skipped": "skipped",
"ui.state_pending": "waiting",
"ui.state_running": "running",
"ui.state_failed": "failed",
"ui.log_warning": "Warning",
"ui.log_error": "Error",
"ui.log_space": "Space needed: {need} GB, free: {free} GB",
"ui.log_end": "Finished",
"ui.result_ok": "The game is ready",
"ui.result_failed": "The setup failed",
"ui.result_cancelled": "The setup was cancelled",
"ui.done_text": "Start totk.exe from {dest}. The setup program and the game files you chose are no longer needed.",
"ui.cancelled_text": "The finished steps are kept. Start the setup again with the same folder to continue.",
"ui.log_file": "Log file: {path}",
"ui.process_died": "The setup stopped unexpectedly (exit code {code}).",
"ui.confirm_cancel": "The setup is running. Cancel it and close the window?",
"ui.space_one": "Free space needed on {path}: {need} GB (free: {free} GB).",
"ui.space_two": "Free space needed on {path}: {need} GB (free: {free} GB); on {temp}: {temp_need} GB (free: {temp_free} GB).",
"ui.amount_bytes": "{done} / {total} GB",
"ui.amount_chunks": "{done} / {total} parts",
"ui.amount_objects": "{done} / {total} files",
"ui.amount_shaders": "{done} / {total} shaders",
"ui.amount_other": "{done} / {total} {unit}",
"ui.seconds": "{n} s",
"param.input.base": "base game",
"param.input.update": "update",
"param.input.keys": "keys",
"launcher.title": "TotK Setup",
"launcher.python_missing": "The setup files are incomplete: the python folder next to TotK-Setup.exe is missing. Unpack the whole archive again and start TotK-Setup.exe from the unpacked folder.",
"lang.ja": "Japanese",
"lang.en-US": "English (US)",
"lang.en-GB": "English (UK)",
"lang.fr": "French",
"lang.fr-CA": "French (Canada)",
"lang.de": "German",
"lang.it": "Italian",
"lang.es": "Spanish",
"lang.es-419": "Spanish (Latin America)",
"lang.nl": "Dutch",
"lang.pt": "Portuguese",
"lang.pt-BR": "Portuguese (Brazil)",
"lang.ru": "Russian",
"lang.ko": "Korean",
"lang.zh-CN": "Chinese (Simplified, China)",
"lang.zh-TW": "Chinese (Traditional, Taiwan)",
"lang.zh-Hans": "Chinese (Simplified)",
"lang.zh-Hant": "Chinese (Traditional)"
}
+170
View File
@@ -0,0 +1,170 @@
{
"step.check": "Проверка компьютера и игровых файлов",
"step.extract": "Распаковка игры",
"step.lift": "Перевод кода игры в C",
"step.compile": "Сборка totk.exe",
"step.prepare": "Подготовка шейдеров",
"step.cleanup": "Уборка",
"phase.check.host": "Проверка процессора и видеокарты",
"phase.check.verify": "Проверка уже установленных файлов",
"phase.check.verify_romfs": "Проверка установленных данных игры",
"phase.check.inputs": "Открытие игровых файлов с помощью ключей",
"phase.check.versions": "Проверка версий игры",
"phase.extract.hash_base": "Подсчёт контрольной суммы базовой игры",
"phase.extract.hash_update": "Подсчёт контрольной суммы обновления",
"phase.extract.exefs": "Распаковка кода игры",
"phase.extract.copy": "Копирование данных игры",
"phase.extract.merge": "Применение обновления к данным игры",
"phase.extract.index": "Проверка данных игры",
"phase.lift.discover": "Поиск функций",
"phase.lift.emit": "Запись кода на C",
"phase.compile.compile": "Компиляция",
"phase.compile.link": "Компоновка totk.exe",
"phase.prepare.romfs_patches": "Запись патчей данных игры для 60 кадров/с",
"phase.prepare.seed": "Чтение шейдеров из данных игры",
"phase.prepare.translate": "Трансляция шейдеров",
"phase.prepare.libraries": "Сборка шейдеров для драйвера видеокарты",
"phase.cleanup.ini": "Запись totk.ini",
"phase.cleanup.temp": "Удаление временных файлов",
"cli.usage": "Неверные параметры командной строки: {detail}",
"internal": "Непредвиденная ошибка ({exception}): {detail}",
"state.unreadable": "Не удаётся прочитать {path}. Выберите другую папку или удалите этот файл.",
"state.newer_format": "Файл {path} создан более новой версией установки. Используйте её или выберите другую папку.",
"state.output_changed": "Установленные файлы шага «{step}» изменились после создания; этот шаг выполнится заново.",
"check.release_incomplete": "Папка установки неполная, не хватает: {paths}. Распакуйте архив заново.",
"check.dest_not_folder": "Место установки {path} — это файл, а не папка.",
"check.dest_not_empty": "Папка установки {path} не пуста и не содержит прежней установки. Выберите пустую папку.",
"check.lang_missing": "Выберите язык игры.",
"check.lang_unknown": "Неизвестный язык «{lang}». Доступные: {known}.",
"check.toolchain_broken": "Встроенный компилятор не работает: {detail}",
"check.host_probe_failed": "Проверка компьютера не выполнилась (код выхода {code}).",
"check.cpu_no_avx2": "Этому процессору не хватает AVX2 (x86-64-v3), который нужен игре. AVX2 есть у Intel Haswell (2013) и новее, у AMD Excavator и Zen и новее.",
"check.vulkan_missing": "Драйвер Vulkan не установлен. Установите свежий драйвер видеокарты.",
"check.vulkan_no_device": "Vulkan не находит видеокарту. Установите свежий драйвер видеокарты.",
"check.vulkan_too_old": "Игре нужна видеокарта с Vulkan 1.3; найдено: {devices}. Обновите драйвер видеокарты.",
"check.inputs_needed": "Нужны игровые файлы: выберите базовую игру, обновление 1.2.1 и prod.keys.",
"check.input_missing": "Файл не найден ({input}): {path}",
"check.keys_unreadable": "Не удаётся прочитать файл ключей. Выберите файл prod.keys.",
"check.keys_no_header_key": "В файле ключей нет header_key. Снова получите prod.keys со своей консоли.",
"check.keys_wrong": "Эти ключи не открывают файл «{input}». Используйте prod.keys со своей консоли.",
"check.keys_too_old": "Файл ключей слишком старый для файла «{input}» (нужно поколение ключей {generation}). Получите prod.keys с консоли с более новой прошивкой.",
"check.not_game_file": "Файл «{input}» не является игровым файлом XCI или NSP.",
"check.file_unreadable": "Не удаётся прочитать файл «{input}»: {detail}",
"check.ticket_personalized": "В файле «{input}» личный тикет; нужен общий тикет.",
"check.ticket_missing": "В файле «{input}» нет тикета, который нужен его содержимому.",
"check.base_not_totk": "Файл базовой игры — не The Legend of Zelda: Tears of the Kingdom ({detail}).",
"check.update_not_nsp": "Обновление должно быть файлом NSP.",
"check.update_is_base": "Файл обновления содержит базовую игру. Выберите NSP с обновлением 1.2.1.",
"check.update_not_totk": "Файл обновления — не обновление Tears of the Kingdom ({detail}).",
"check.exefs_failed": "Код игры из файла «{input}» не прошёл проверку; подробности в {log}.",
"check.base_wrong_version": "Базовая игра не версии 1.0.0 (отличаются модули: {modules}).",
"check.update_wrong_version": "Версия обновления {found}; этому выпуску нужна {wanted}.",
"check.disk_space": "Не хватает свободного места для {path}: нужно {need_gb} ГБ, свободно {free_gb} ГБ.",
"extract.exefs_mismatch": "Распакованный код игры не соответствует версии 1.2.1 ({modules}).",
"extract.romfs_merge_failed": "Не удалось применить обновление к данным игры: {detail}. Подробности: {log}",
"extract.romfs_mismatch": "Распакованные данные игры не соответствуют версии 1.2.1 (размер {size}, контрольная сумма {sha256}...).",
"lift.failed": "Не удалось перевести код игры (код выхода {code}). Подробности: {log}",
"compile.failed": "Не удалось собрать totk.exe (код выхода {code}). Подробности: {log}",
"compile.exe_in_use": "Файл {path} занят. Закройте игру и запустите установку снова.",
"prepare.shader_list_missing": "Нет списка шейдеров.",
"prepare.romfs_patches_failed": "Не удалось записать патчи данных игры (код выхода {code}). Подробности: {log}",
"prepare.shaders_failed": "Не удалось подготовить шейдеры (код выхода {code}). Подробности: {log}",
"cleanup.ini_failed": "Не удаётся записать totk.ini: {detail}",
"warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.",
"warn.npdm_differs": "main.npdm отличается от эталонного дампа 1.2.1; используются только модули с кодом.",
"ui.title": "Установка TotK",
"ui.intro": "Собирает готовую к запуску папку с Tears of the Kingdom из ваших игровых файлов. Занимает много времени и выполняется один раз.",
"ui.notice": "Папка, которую создаёт установка, содержит игру. Она предназначена только для личного использования: не передавайте и не выкладывайте её.",
"ui.keys_private": "Файл ключей читается на месте. Его содержимое не показывается, не копируется и не сохраняется.",
"ui.base": "Базовая игра (NSP или XCI)",
"ui.update": "Обновление 1.2.1 (NSP)",
"ui.keys": "prod.keys",
"ui.game_language": "Язык игры",
"ui.dest": "Папка установки",
"ui.browse": "Обзор...",
"ui.pick_game": "Выберите файл базовой игры",
"ui.pick_update": "Выберите файл обновления 1.2.1",
"ui.pick_keys": "Выберите prod.keys",
"ui.pick_dest": "Выберите папку установки",
"ui.type_game": "Файлы игры (NSP, XCI)",
"ui.type_nsp": "Файлы NSP",
"ui.type_keys": "Файлы ключей (*.keys)",
"ui.type_all": "Все файлы",
"ui.start": "Начать",
"ui.cancel": "Отмена",
"ui.cancelling": "Отмена...",
"ui.show_log": "Показать журнал",
"ui.hide_log": "Скрыть журнал",
"ui.close": "Закрыть",
"ui.back": "Назад",
"ui.open_folder": "Открыть папку",
"ui.update_mode": "В этой папке уже есть установка. Программа продолжит или обновит её; файлы игры понадобятся, только если их снова придётся читать.",
"ui.need_dest": "Выберите папку установки.",
"ui.need_base": "Выберите файл базовой игры.",
"ui.need_update": "Выберите файл обновления 1.2.1.",
"ui.need_keys": "Выберите файл prod.keys.",
"ui.need_lang": "Выберите язык игры.",
"ui.missing_base": "Файл базовой игры не найден.",
"ui.missing_update": "Файл обновления не найден.",
"ui.missing_keys": "Файл ключей не найден.",
"ui.running_install": "Установка",
"ui.running_resume": "Продолжение прерванной установки",
"ui.running_update": "Обновление установки",
"ui.state_started": "начат",
"ui.state_done": "готово",
"ui.state_skipped": "пропущен",
"ui.state_pending": "ожидает",
"ui.state_running": "выполняется",
"ui.state_failed": "ошибка",
"ui.log_warning": "Предупреждение",
"ui.log_error": "Ошибка",
"ui.log_space": "Нужно места: {need} ГБ, свободно: {free} ГБ",
"ui.log_end": "Завершено",
"ui.result_ok": "Игра готова",
"ui.result_failed": "Установка не удалась",
"ui.result_cancelled": "Установка отменена",
"ui.done_text": "Запустите totk.exe из папки {dest}. Программа установки и выбранные игровые файлы больше не нужны.",
"ui.cancelled_text": "Завершённые шаги сохранены. Чтобы продолжить, снова запустите установку с той же папкой.",
"ui.log_file": "Файл журнала: {path}",
"ui.process_died": "Установка неожиданно остановилась (код выхода {code}).",
"ui.confirm_cancel": "Установка выполняется. Отменить её и закрыть окно?",
"ui.space_one": "Нужно свободного места на {path}: {need} ГБ (свободно: {free} ГБ).",
"ui.space_two": "Нужно свободного места на {path}: {need} ГБ (свободно: {free} ГБ); на {temp}: {temp_need} ГБ (свободно: {temp_free} ГБ).",
"ui.amount_bytes": "{done} / {total} ГБ",
"ui.amount_chunks": "{done} / {total} частей",
"ui.amount_objects": "{done} / {total} файлов",
"ui.amount_shaders": "{done} / {total} шейдеров",
"ui.amount_other": "{done} / {total} {unit}",
"ui.seconds": "{n} с",
"param.input.base": "базовая игра",
"param.input.update": "обновление",
"param.input.keys": "ключи",
"launcher.title": "Установка TotK",
"launcher.python_missing": "Файлы установки неполные: рядом с TotK-Setup.exe нет папки python. Распакуйте архив целиком заново и запустите TotK-Setup.exe из распакованной папки.",
"lang.ja": "Японский",
"lang.en-US": "Английский (США)",
"lang.en-GB": "Английский (Великобритания)",
"lang.fr": "Французский",
"lang.fr-CA": "Французский (Канада)",
"lang.de": "Немецкий",
"lang.it": "Итальянский",
"lang.es": "Испанский",
"lang.es-419": "Испанский (Латинская Америка)",
"lang.nl": "Нидерландский",
"lang.pt": "Португальский",
"lang.pt-BR": "Португальский (Бразилия)",
"lang.ru": "Русский",
"lang.ko": "Корейский",
"lang.zh-CN": "Китайский (упрощённый, КНР)",
"lang.zh-TW": "Китайский (традиционный, Тайвань)",
"lang.zh-Hans": "Китайский (упрощённый)",
"lang.zh-Hant": "Китайский (традиционный)"
}
+79
View File
@@ -0,0 +1,79 @@
"""End-to-end run of the installer in dev mode with the game files named in config.local.toml.
Reads input.base, the 1.2.1 entry of input.updates (the one build/extracted/v121 was extracted from, else the only
one) and input.keys, and passes them to `python -m installer` started the way the release launcher starts it: the
embeddable Python (build/py314-embed) with this checkout on sys.path. The paths are never printed. Events go to
<events> (JSON lines); stdout gets each step's state and time, warnings and errors.
python tests/installer_e2e.py --dest build/install-test --lang en-US [--events FILE] [--no-inputs] [-- extra args]
--no-inputs leaves the game files out (update mode on an existing folder).
"""
import argparse
import json
import subprocess
import sys
import tomllib
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
EMBED = ROOT / 'build' / 'py314-embed' / 'py' / 'python.exe'
BOOT = ('import runpy, sys; sys.path.insert(0, sys.argv.pop(1)); '
"runpy.run_module('installer', run_name='__main__', alter_sys=True)")
def game_inputs():
config = tomllib.loads((ROOT / 'config.local.toml').read_text(encoding='utf-8'))['input']
updates = [Path(u) for u in config.get('updates', [])]
manifest = ROOT / 'build' / 'extracted' / 'v121' / 'extraction-manifest.json'
update = None
if manifest.is_file():
source = json.loads(manifest.read_text())['source']
update = next((u for u in updates if str(u.resolve()) == source), None)
if update is None and len(updates) == 1:
update = updates[0]
if update is None:
sys.exit('installer_e2e: cannot tell which input.updates entry is 1.2.1')
return ['--base', config['base'], '--update', str(update), '--keys', config['keys']]
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--dest', type=Path, required=True)
ap.add_argument('--lang')
ap.add_argument('--events', type=Path)
ap.add_argument('--no-inputs', action='store_true')
ap.add_argument('extra', nargs='*')
a = ap.parse_args()
python = EMBED if EMBED.is_file() else Path(sys.executable)
command = [str(python), '-c', BOOT, str(ROOT), '--dest', str(a.dest.resolve())]
if a.lang:
command += ['--lang', a.lang]
if not a.no_inputs:
command += game_inputs()
command += a.extra
events = a.events or a.dest.resolve().parent / f'{a.dest.name}-events.jsonl'
with open(events, 'w', encoding='utf-8') as log:
proc = subprocess.Popen(command, stdout=subprocess.PIPE, text=True, encoding='utf-8', cwd=ROOT)
for line in proc.stdout:
log.write(line)
log.flush()
try:
e = json.loads(line)
except ValueError:
print('non-JSON line on stdout:', line.rstrip())
continue
if e['type'] == 'step' and e['state'] != 'started':
extra = f'{e["seconds"]:.1f} s' if 'seconds' in e else e.get('reason', '')
print(f'[{e["t"]:8.1f}] {e["step"]:8} {e["state"]} {extra}', flush=True)
elif e['type'] == 'step':
print(f'[{e["t"]:8.1f}] {e["step"]:8} started', flush=True)
elif e['type'] in ('warning', 'error', 'plan', 'end'):
print(f'[{e["t"]:8.1f}] {json.dumps({k: v for k, v in e.items() if k != "t"})}', flush=True)
elif e['type'] == 'start':
print(f'[{e["t"]:8.1f}] start mode={e["mode"]} release={e["release"]}', flush=True)
return proc.wait()
if __name__ == '__main__':
sys.exit(main())
+394
View File
@@ -0,0 +1,394 @@
"""The installer window: event model, form, catalogs, cancel, a self-closing Tk run and one end-to-end run.
.venv-py314\\Scripts\\python tests\\installer_gui_test.py
The end-to-end test runs the real CLI in --verify mode on a finished install (every step skips) and folds its events
through the same code the window uses. The install is build/inst-gui/e2e-install: hard links to build/install-test, so
nothing there is written, except the state file and totk.ini, which a run rewrites and which are copied. The run hashes
the 17 GB RomFS; it is skipped when the hash differs from the expected one in all three tries. On the developer machine
(RAM with a bad bit) a buffered read of a file sometimes returns flipped bits from the page cache while the file on
disk is intact (an unbuffered read of install-test's romfs.bin gives the expected hash), so such a skip is the machine,
not the installer. It is also skipped when build/install-test is absent.
"""
import json
import os
import queue
import re
import shutil
import subprocess
import sys
import tempfile
import time
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
from installer import state as statefile # noqa: E402
from installer.guilogic import (DONE, FAILED, PENDING, RUNNING, SKIPPED, Catalog, Form, Model, Process, # noqa: E402
amount, describe_event, parse_line, windows_ui_language)
INSTALLED = ROOT / 'build' / 'install-test'
FIXTURE = ROOT / 'build' / 'inst-gui' / 'e2e-install'
COPIED = ('totk.ini', statefile.NAME) # written by a run, so never linked
START = {'type': 'start', 'release': 'r1', 'mode': 'install', 'dest': 'D:\\T', 'log': 'D:\\T.log',
'steps': ['check', 'extract', 'lift', 'compile', 'prepare', 'cleanup']}
def fold(*events):
m = Model()
for e in events:
m.apply(e)
return m
class ParseTest(unittest.TestCase):
def test_event_line(self):
self.assertEqual(parse_line('{"type": "step", "step": "check", "state": "started"}\n')['step'], 'check')
def test_noise_is_not_an_event(self):
for line in ('', 'Traceback (most recent call last):', '{broken', '[1, 2]', '{"no_type": 1}', '{"type": 3}'):
self.assertIsNone(parse_line(line), line)
def test_unicode_survives(self):
e = parse_line(json.dumps({'type': 'warning', 'id': 'x', 'params': {'p': 'Привет'}}, ensure_ascii=False))
self.assertEqual(e['params']['p'], 'Привет')
class ModelTest(unittest.TestCase):
def test_steps_follow_the_events(self):
m = fold(START, {'type': 'plan', 'run': ['check', 'extract'], 'skip': {'lift': 'current', 'compile': 'current'}},
{'type': 'step', 'step': 'check', 'state': 'started'})
self.assertEqual([m.states[s] for s in m.steps],
[RUNNING, PENDING, SKIPPED, SKIPPED, PENDING, PENDING])
self.assertEqual(m.current, 'check')
m.apply({'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 1.0})
self.assertEqual(m.states['check'], DONE)
self.assertIsNone(m.current)
m.apply({'type': 'step', 'step': 'extract', 'state': 'skipped', 'reason': 'current'})
self.assertEqual(m.states['extract'], SKIPPED)
def test_progress_fraction(self):
m = fold(START, {'type': 'step', 'step': 'extract', 'state': 'started'},
{'type': 'progress', 'step': 'extract', 'phase': 'copy', 'done': 5, 'total': 20, 'unit': 'bytes'})
self.assertEqual((m.current, m.phase, m.fraction()), ('extract', 'copy', 0.25))
m.apply({'type': 'progress', 'step': 'extract', 'phase': 'merge'})
self.assertIsNone(m.fraction()) # a phase without size is indeterminate
m.apply({'type': 'progress', 'step': 'extract', 'phase': 'copy', 'done': 30, 'total': 20})
self.assertEqual(m.fraction(), 1.0)
def test_a_new_step_resets_progress(self):
m = fold(START, {'type': 'progress', 'step': 'check', 'phase': 'inputs', 'done': 1, 'total': 2},
{'type': 'step', 'step': 'extract', 'state': 'started'})
self.assertIsNone(m.phase)
self.assertIsNone(m.fraction())
def test_error_marks_the_step(self):
m = fold(START, {'type': 'step', 'step': 'check', 'state': 'started'},
{'type': 'error', 'id': 'check.disk_space', 'step': 'check', 'params': {'path': 'D:\\T'}},
{'type': 'end', 'result': 'failed', 'seconds': 2})
self.assertEqual(m.states['check'], FAILED)
self.assertEqual(m.error[0], 'check.disk_space')
self.assertEqual(m.result, 'failed')
def test_space_text(self):
m = fold({'type': 'space', 'dest': 'D:\\T', 'need': 36_400_000_000, 'free': 100_000_000_000, 'temp': 'C:\\t',
'temp_need': 0, 'temp_free': 5, 'same_disk': True})
text = m.need_text(Catalog('en'))
self.assertIn('36.4', text)
self.assertIn('100.0', text)
self.assertEqual(Model().need_text(Catalog('en')), '')
m = fold({'type': 'space', 'dest': 'D:\\T', 'need': 20_000_000_000, 'free': 1, 'temp': 'C:\\t',
'temp_need': 2_000_000_000, 'temp_free': 3, 'same_disk': False})
self.assertIn('C:\\t', m.need_text(Catalog('en')))
def test_warnings_collect(self):
m = fold(START, {'type': 'warning', 'id': 'warn.lang_needs_fonts', 'params': {'lang': 'ko'}})
self.assertEqual(m.warnings, [('warn.lang_needs_fonts', {'lang': 'ko'})])
def test_finish_without_end_event(self):
m = fold(START, {'type': 'step', 'step': 'lift', 'state': 'started'})
m.finish(-1073741510)
self.assertEqual(m.result, 'failed')
self.assertEqual(m.states['lift'], FAILED)
self.assertEqual(m.error[0], 'ui.process_died')
def test_finish_after_cancel(self):
m = fold(START, {'type': 'step', 'step': 'lift', 'state': 'started'}, {'type': 'end', 'result': 'cancelled'})
m.finish(3)
self.assertEqual(m.result, 'cancelled')
self.assertIsNone(m.error)
self.assertEqual(m.states['lift'], PENDING) # not finished; the next run resumes here
def test_exit_code_3_alone_is_cancelled(self):
m = Model()
m.finish(3)
self.assertEqual(m.result, 'cancelled')
def test_unknown_event_is_ignored(self):
m = fold(START, {'type': 'future_thing', 'x': 1})
self.assertEqual(m.states['check'], PENDING)
class RunStatesTest(unittest.TestCase):
OK = [START, {'type': 'plan', 'run': ['check', 'extract'], 'skip': {'lift': 'current', 'compile': 'current',
'prepare': 'current', 'cleanup': 'not_needed'}},
{'type': 'step', 'step': 'check', 'state': 'started'}, {'type': 'step', 'step': 'check', 'state': 'done'},
{'type': 'step', 'step': 'extract', 'state': 'started'}, {'type': 'step', 'step': 'extract', 'state': 'done'},
{'type': 'step', 'step': 'lift', 'state': 'skipped'}, {'type': 'step', 'step': 'compile', 'state': 'skipped'},
{'type': 'step', 'step': 'prepare', 'state': 'skipped'}, {'type': 'step', 'step': 'cleanup', 'state': 'done'},
{'type': 'end', 'result': 'ok', 'seconds': 1}]
FAILED = [START, {'type': 'step', 'step': 'check', 'state': 'started'}, {'type': 'step', 'step': 'check', 'state': 'done'},
{'type': 'step', 'step': 'extract', 'state': 'started'},
{'type': 'error', 'id': 'extract.romfs_mismatch', 'step': 'extract', 'params': {}},
{'type': 'warning', 'id': 'warn.lang_needs_fonts', 'params': {'lang': 'ko'}},
{'type': 'end', 'result': 'failed', 'seconds': 1}]
def test_ok_run_has_only_done_and_skipped_steps(self):
m = fold(*self.OK)
m.finish(0)
self.assertEqual(m.result, 'ok')
self.assertIsNone(m.error)
self.assertEqual({m.states[s] for s in m.steps}, {DONE, SKIPPED})
def test_a_new_run_keeps_nothing_of_the_failed_one(self):
m = fold(*self.FAILED)
m.finish(1)
self.assertEqual(m.states['extract'], FAILED)
for e in self.OK: # the same model object fed a second run (the window makes a new one, but even then)
m.apply(e)
m.finish(0)
self.assertEqual({m.states[s] for s in m.steps}, {DONE, SKIPPED})
self.assertIsNone(m.error)
self.assertEqual(m.warnings, [])
self.assertEqual(m.result, 'ok')
class FormTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.dir = Path(self.tmp.name)
for name in ('base.nsp', 'upd.nsp', 'prod.keys'):
(self.dir / name).write_bytes(b'x')
def tearDown(self):
self.tmp.cleanup()
def form(self, **kw):
values = dict(base=str(self.dir / 'base.nsp'), update=str(self.dir / 'upd.nsp'), keys=str(self.dir / 'prod.keys'),
lang='en-US', dest=str(self.dir / 'out'))
values.update(kw)
return Form(**values)
def test_complete_form(self):
self.assertEqual(self.form().problems(), [])
def test_empty_form(self):
self.assertEqual(Form().problems(), ['ui.need_dest', 'ui.need_base', 'ui.need_update', 'ui.need_keys',
'ui.need_lang'])
def test_missing_files(self):
self.assertEqual(self.form(base=str(self.dir / 'no.nsp'), keys=str(self.dir / 'no.keys')).problems(),
['ui.missing_base', 'ui.missing_keys'])
def test_unknown_language(self):
self.assertEqual(self.form(lang='xx').problems(), ['ui.need_lang'])
def test_command(self):
cmd = self.form().command('py.exe', root=ROOT, temp=self.dir / 't', verify=True)
self.assertEqual(cmd[:3], ['py.exe', '-m', 'installer'])
for flag in ('--base', '--update', '--keys', '--lang', '--dest', '--root', '--temp', '--verify'):
self.assertIn(flag, cmd)
self.assertEqual(cmd[cmd.index('--lang') + 1], 'en-US')
def test_installed_folder_needs_only_itself(self):
out = self.dir / 'out'
statefile.save(self._mkdir(out), statefile.empty('r0') | {'lang': 'ru'})
form = Form(dest=str(out))
self.assertTrue(form.update_mode())
self.assertEqual(form.problems(), [])
cmd = form.command('py.exe')
self.assertEqual(cmd, ['py.exe', '-m', 'installer', '--dest', str(out)])
def test_installed_folder_still_checks_given_files(self):
out = self.dir / 'out'
statefile.save(self._mkdir(out), statefile.empty('r0'))
self.assertEqual(Form(dest=str(out), base=str(self.dir / 'no.nsp')).problems(), ['ui.missing_base'])
def test_broken_state_is_a_new_install(self):
out = self._mkdir(self.dir / 'out')
(out / statefile.NAME).write_text('{nope', encoding='utf-8')
self.assertFalse(Form(dest=str(out)).update_mode())
@staticmethod
def _mkdir(path):
path.mkdir(parents=True, exist_ok=True)
return path
class CatalogTest(unittest.TestCase):
def test_russian_covers_english_with_the_same_placeholders(self):
en, ru = Catalog('en').english, Catalog('ru').texts
self.assertEqual(sorted(set(en) - set(ru)), [])
self.assertEqual(sorted(set(ru) - set(en)), [])
holes = [k for k in en if sorted(re.findall(r'\{\w+\}', en[k])) != sorted(re.findall(r'\{\w+\}', ru[k]))]
self.assertEqual(holes, [])
def test_every_id_the_window_uses_exists(self):
cat = Catalog('en')
ids = ['ui.' + n for n in ('title', 'start', 'cancel', 'open_folder', 'notice', 'space_one', 'space_two',
'process_died', 'result_ok', 'result_failed', 'result_cancelled')]
ids += [f'step.{s}' for s in ('check', 'extract', 'lift', 'compile', 'prepare', 'cleanup')]
ids += ['ui.state_' + n for n in ('started', 'done', 'skipped', 'pending', 'running', 'failed')]
for lang in ('en', 'ru'):
cat.set_language(lang)
self.assertEqual([i for i in ids if not cat.has(i)], [])
def test_codes_become_names(self):
for lang, korean in (('en', 'Korean'), ('ru', 'Корейский')):
cat = Catalog(lang)
self.assertIn(korean, cat('warn.lang_needs_fonts', lang='ko'))
self.assertNotIn(' ko ', cat('warn.lang_needs_fonts', lang='ko'))
self.assertNotIn('extract', cat('state.output_changed', step='extract'))
for word in ('base', 'update', 'keys'):
self.assertIn(cat('param.input.' + word), cat('check.keys_wrong', input=word))
self.assertNotIn('en-US', cat('check.lang_unknown', lang='xx', known='ja, en-US'))
self.assertEqual(Catalog('en')('check.lang_unknown', lang='xx', known='ja, en-US'),
"Unknown language 'xx'. Known: Japanese, English (US).")
def test_fallback_and_params(self):
cat = Catalog('ru')
self.assertIn('36.4', cat('ui.log_space', need='36.4', free='1.0'))
self.assertEqual(cat('no.such.id'), 'no.such.id')
self.assertIn('a, b', cat('check.release_incomplete', paths=['a', 'b']))
cat.set_language('de') # unknown language falls back to English
self.assertEqual(cat.lang, 'en')
def test_describe_and_amount(self):
cat = Catalog('en')
self.assertEqual(amount(cat, 6e9, 16e9, 'bytes'), '6.0 / 16.0 GB')
self.assertEqual(amount(cat, 1, 9, 'shaders'), '1 / 9 shaders')
self.assertEqual(amount(cat, 1, 9, 'widgets'), '1 / 9 widgets')
line = describe_event(cat, {'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 1.5})
self.assertIn('1.5 s', line)
self.assertEqual(describe_event(cat, {'type': 'start'}), '')
def test_windows_language_is_a_known_value(self):
self.assertIn(windows_ui_language(), ('en', 'ru'))
BREAK_CHILD = 'import time\nprint("ready", flush=True)\ntime.sleep(60)\n'
class CancelTest(unittest.TestCase):
def test_ctrl_break_ends_the_process_group(self):
p = Process([sys.executable, '-c', BREAK_CHILD])
try:
self.assertEqual(p.lines.get(timeout=20).strip(), 'ready')
t0 = time.monotonic()
p.cancel()
code = None
while code is None and time.monotonic() - t0 < 20:
code = p.poll()
time.sleep(0.05)
self.assertIsNotNone(code, 'the child ignored Ctrl+Break')
self.assertNotEqual(code, 0)
finally:
p.kill()
class SmokeTest(unittest.TestCase):
def test_every_screen_in_both_languages(self):
try:
import tkinter # noqa: F401
except ImportError:
self.skipTest('no tkinter')
r = subprocess.run([sys.executable, '-m', 'installer.gui', '--selftest'], cwd=ROOT, timeout=120,
capture_output=True, text=True)
self.assertEqual(r.returncode, 0, r.stderr[-2000:])
def make_fixture():
"""build/inst-gui/e2e-install: links to install-test (see the module docstring); rebuilt when absent."""
if (FIXTURE / statefile.NAME).is_file():
return FIXTURE
shutil.rmtree(FIXTURE, ignore_errors=True)
for folder, _dirs, files in os.walk(INSTALLED):
target = FIXTURE / Path(folder).relative_to(INSTALLED)
target.mkdir(parents=True, exist_ok=True)
for name in files:
src, dst = Path(folder) / name, target / name
if name in COPIED and target == FIXTURE:
shutil.copy2(src, dst)
else:
os.link(src, dst)
return FIXTURE
@unittest.skipUnless((INSTALLED / statefile.NAME).is_file(), 'build/install-test is absent')
class EndToEndTest(unittest.TestCase):
@staticmethod
def run_cli(form, temp, tries=1):
"""Runs the CLI in --verify mode through Process, up to `tries` times while the RomFS hash reads wrong."""
for attempt in range(tries):
p = Process(form.command(sys.executable, root=ROOT, temp=temp, verify=True), cwd=ROOT)
model, stray = Model(), []
t0, code = time.monotonic(), None
while code is None and time.monotonic() - t0 < 600:
try:
line = p.lines.get(timeout=0.2)
except queue.Empty:
line = ''
if line:
event = parse_line(line)
if event:
model.apply(event)
elif line.strip():
stray.append(line)
code = p.poll()
p.kill()
model.finish(code)
if not any(w[0] == 'state.output_changed' for w in model.warnings):
break
return code, model, stray, time.monotonic() - t0
def test_verify_run_skips_every_step(self):
form = Form(dest=str(make_fixture()))
self.assertTrue(form.update_mode())
code, model, stray, elapsed = self.run_cli(form, ROOT / 'build' / 'inst-gui' / 'e2e-temp', tries=3)
print(f'\nend-to-end: exit {code}, result {model.result}, {elapsed:.1f} s, states {model.states}')
if any(w[0] == 'state.output_changed' for w in model.warnings):
self.skipTest('the RomFS hash read wrong three times (page cache of the faulty RAM, see the module docstring)')
self.assertEqual(code, 0, stray[-5:] or model.error)
self.assertEqual(model.result, 'ok')
self.assertIn(model.mode, ('resume', 'update'))
self.assertEqual({s: model.states[s] for s in model.steps if s not in ('check', 'cleanup')},
{s: SKIPPED for s in model.steps if s not in ('check', 'cleanup')})
self.assertIsNotNone(model.space)
self.assertEqual(stray, [])
def test_cancel_during_the_check(self):
cmd = Form(dest=str(make_fixture())).command(sys.executable, root=ROOT, temp=ROOT / 'build' / 'inst-gui' / 'e2e-temp',
verify=True)
p = Process(cmd, cwd=ROOT)
model, t0, code, cancelled = Model(), time.monotonic(), None, False
while code is None and time.monotonic() - t0 < 120:
try:
event = parse_line(p.lines.get(timeout=0.2))
except queue.Empty:
event = None
if event:
model.apply(event)
if event.get('phase') == 'verify_romfs' and not cancelled:
cancelled = True
p.cancel()
code = p.poll()
p.kill()
model.finish(code)
self.assertTrue(cancelled)
self.assertEqual((code, model.result), (3, 'cancelled'))
self.assertIsNone(model.error)
if __name__ == '__main__':
unittest.main(verbosity=2)
+264
View File
@@ -0,0 +1,264 @@
"""installer/ bookkeeping with fake steps: fresh install, resume after a failure or cancel, transient steps, update
with a newer release, totk.ini and the event stream. No game files, no tools; stdlib only.
Run: python tests/installer_test.py
"""
import io
import json
import re
import sys
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
from installer import ini, runner, state # noqa: E402
from installer.events import Reporter # noqa: E402
from installer.layout import Layout # noqa: E402
class Fake(runner.Step):
"""A step whose output is <dir>/<id>.out holding its key; dir is the temp folder for a transient step."""
def __init__(self, id, transient=False, always=False, previous=None):
self.id, self.transient, self.always, self.previous = id, transient, always, previous
def out(self, ctx):
return (ctx.temp if self.transient else ctx.dest) / f'{self.id}.out'
def key(self, ctx):
upstream = ctx.keys[self.previous] if self.previous else ''
return f'{self.id}:{ctx.options.inputs.get(self.id, "v1")}:{upstream}'
def current(self, ctx, record):
p = self.out(ctx)
return p.is_file() and p.read_text() == ctx.keys[self.id]
def verify(self, ctx, record):
ctx.options.verified.append(self.id)
return ctx.options.verify_ok.get(self.id, True)
def run(self, ctx):
o = ctx.options
o.ran.append(self.id)
if self.id in ctx.forced:
o.forced.append(self.id)
if self.id == 'check':
ctx.verify_kept(self.id)
failure = o.fail.pop(self.id, None)
if failure == 'error':
raise runner.InstallError('fake.failed', step=self.id)
if failure == 'cancel':
raise KeyboardInterrupt
if self.id == 'cleanup':
for p in ctx.temp.glob('*.out'):
p.unlink()
return {}
p = self.out(ctx)
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(ctx.keys[self.id])
return {'note': self.id}
def fake_steps():
return [Fake('check', always=True), Fake('extract'), Fake('lift', transient=True, previous='extract'),
Fake('compile', previous='lift'), Fake('prepare', previous='compile'), Fake('cleanup', always=True)]
class Bed:
"""A dest and temp folder plus one run of the fake steps per call."""
def __init__(self, base):
self.dest, self.temp = base / 'dest', base / 'temp'
self.temp.mkdir()
def run(self, release='1.0', inputs=None, fail=None, verify_ok=None, verify=False):
layout = Layout(root=Path('.'), dev=True, release=release, python=Path('python'), tools=Path('tools'),
recomp=Path('recomp'), patches=Path('p'), expected=Path('e'), llvm=Path('llvm'),
deps=Path('deps'), runtime=Path('rt'), runtime_lib=Path('lib'), ninja=Path('ninja'),
hactool=Path('hactool'), xlate=Path('xlate'))
options = SimpleNamespace(inputs=inputs or {}, fail=dict(fail or {}), verify_ok=verify_ok or {}, ran=[],
verified=[], forced=[])
out = io.StringIO()
ctx = runner.Context(layout, self.dest, self.temp, Reporter(out=out), options, state=state.load(self.dest),
verify=verify)
mode = ctx.mode
result = runner.run_steps(fake_steps(), ctx)
events = [json.loads(line) for line in out.getvalue().splitlines()]
return SimpleNamespace(result=result, ran=options.ran, verified=options.verified, forced=options.forced,
events=events, mode=mode,
state=state.load(self.dest))
@staticmethod
def skipped(r):
return {e['step']: e['reason'] for e in r.events if e['type'] == 'step' and e['state'] == 'skipped'}
ALL = ['check', 'extract', 'lift', 'compile', 'prepare', 'cleanup']
class RunnerTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.bed = Bed(Path(self.tmp.name))
def tearDown(self):
self.tmp.cleanup()
def test_fresh_install_runs_everything_and_records_it(self):
r = self.bed.run()
self.assertEqual((r.result, r.mode, r.ran), ('ok', 'install', ALL))
self.assertEqual(set(r.state['steps']), {'extract', 'lift', 'compile', 'prepare'})
self.assertEqual(r.state['steps']['compile']['note'], 'compile')
self.assertEqual(r.state['release'], '1.0')
started = [e['step'] for e in r.events if e['type'] == 'step' and e['state'] == 'started']
done = [e['step'] for e in r.events if e['type'] == 'step' and e['state'] == 'done']
self.assertEqual((started, done), (ALL, ALL))
def test_rerun_skips_finished_steps_and_unneeded_transient(self):
self.bed.run()
r = self.bed.run()
self.assertEqual((r.result, r.mode, r.ran), ('ok', 'resume', ['check', 'cleanup']))
self.assertEqual(self.bed.skipped(r), {'extract': 'current', 'lift': 'not_needed', 'compile': 'current',
'prepare': 'current'})
self.assertEqual(r.verified, []) # same release: no full checks
def test_resume_after_failure(self):
r = self.bed.run(fail={'compile': 'error'})
self.assertEqual((r.result, r.ran), ('failed', ['check', 'extract', 'lift', 'compile']))
error = [e for e in r.events if e['type'] == 'error'][0]
self.assertEqual((error['id'], error['step'], error['params']), ('fake.failed', 'compile', {'step': 'compile'}))
self.assertEqual(set(r.state['steps']), {'extract', 'lift'})
r = self.bed.run()
# lift's output is still in the temp folder (no cleanup ran), so only compile and later run.
self.assertEqual((r.result, r.mode, r.ran), ('ok', 'resume', ['check', 'compile', 'prepare', 'cleanup']))
def test_resume_after_cancel_with_transient_output_gone(self):
r = self.bed.run(fail={'prepare': 'cancel'})
self.assertEqual(r.result, 'cancelled')
self.assertNotIn('prepare', r.state['steps'])
for p in self.bed.temp.glob('*.out'):
p.unlink()
r = self.bed.run()
self.assertEqual(r.ran, ['check', 'prepare', 'cleanup'])
self.assertEqual(self.bed.skipped(r)['lift'], 'not_needed')
def test_failed_check_writes_nothing(self):
r = self.bed.run(fail={'check': 'error'})
self.assertEqual(r.result, 'failed')
self.assertFalse(self.bed.dest.exists())
def test_removed_output_runs_again(self):
self.bed.run()
(self.bed.dest / 'compile.out').unlink()
r = self.bed.run()
self.assertEqual(r.ran, ['check', 'lift', 'compile', 'cleanup'])
def test_update_with_new_lifter_keeps_extraction(self):
self.bed.run(release='1.0')
r = self.bed.run(release='1.1', inputs={'lift': 'v2'})
self.assertEqual((r.result, r.mode), ('ok', 'update'))
self.assertEqual(r.ran, ['check', 'lift', 'compile', 'prepare', 'cleanup'])
self.assertEqual(r.verified, ['extract']) # kept from another release: full check
self.assertEqual(r.state['steps']['extract']['release'], '1.0')
self.assertEqual(r.state['steps']['compile']['release'], '1.1')
self.assertEqual(r.state['release'], '1.1')
def test_update_that_changes_nothing_skips_the_rebuild(self):
self.bed.run(release='1.0')
r = self.bed.run(release='1.1')
self.assertEqual(r.ran, ['check', 'cleanup'])
self.assertEqual(r.verified, ['extract', 'compile', 'prepare'])
def test_update_with_changed_kept_files_extracts_again(self):
self.bed.run(release='1.0')
r = self.bed.run(release='1.1', verify_ok={'extract': False})
self.assertEqual(r.ran, ['check', 'extract', 'cleanup'])
self.assertEqual(r.forced, ['extract']) # the step knows not to trust what it left behind
warnings = [e for e in r.events if e['type'] == 'warning']
self.assertEqual([(w['id'], w['params']) for w in warnings], [('state.output_changed', {'step': 'extract'})])
def test_verify_option_checks_same_release(self):
self.bed.run()
r = self.bed.run(verify=True)
self.assertEqual(r.verified, ['extract', 'compile', 'prepare'])
def test_state_errors(self):
self.bed.dest.mkdir()
(self.bed.dest / state.NAME).write_text('{not json')
with self.assertRaises(state.StateError) as e:
state.load(self.bed.dest)
self.assertEqual(e.exception.id, 'state.unreadable')
(self.bed.dest / state.NAME).write_text(json.dumps({'format': 99, 'steps': {}}))
with self.assertRaises(state.StateError) as e:
state.load(self.bed.dest)
self.assertEqual(e.exception.id, 'state.newer_format')
class IniTest(unittest.TestCase):
EXAMPLE = '[display]\n# vk_present = mailbox\n\n[language]\n# locale = en-US\n# lang = en-US\n# x = 1\n'
def test_example_line_is_activated(self):
out = ini.set_language(self.EXAMPLE, 'ru')
self.assertIn('\nlang = ru\n', out)
self.assertNotIn('# lang', out)
self.assertIn('# locale = en-US', out)
def test_active_line_is_rewritten_and_the_rest_kept(self):
text = '[display]\nvk_present = fifo\n[language]\n LANG = ja \nfps60 = 0\n'
out = ini.set_language(text, 'de')
self.assertEqual(out, '[display]\nvk_present = fifo\n[language]\n lang = de\nfps60 = 0\n')
self.assertEqual(ini.get_language(out), 'de')
def test_missing_section_is_added(self):
self.assertEqual(ini.set_language('fps60 = 0\n', 'fr'), 'fps60 = 0\n\n[language]\nlang = fr\n')
self.assertEqual(ini.set_language('[language]\nlocale = de-DE\n', 'fr'),
'[language]\nlang = fr\nlocale = de-DE\n')
def test_write_keeps_user_lines(self):
with tempfile.TemporaryDirectory() as d:
d = Path(d)
example = d / 'totk.ini.example'
example.write_text(self.EXAMPLE)
path = ini.write(d / '', 'en-GB', example)
self.assertEqual(ini.get_language(path.read_text()), 'en-GB')
path.write_text(path.read_text().replace('# vk_present = mailbox', 'vk_present = fifo'))
ini.write(d, 'ja', example)
text = path.read_text()
self.assertIn('vk_present = fifo', text)
self.assertEqual(ini.get_language(text), 'ja')
class EventsTest(unittest.TestCase):
def test_lines_are_json_and_the_log_is_english(self):
with tempfile.TemporaryDirectory() as d:
out = io.StringIO()
rep = Reporter(out=out, log_path=Path(d) / 'x.log')
rep.step('check', 'started')
rep.progress('extract', 'copy', 0, 10, 'bytes')
rep.progress('extract', 'copy', 5, 10, 'bytes') # throttled
rep.progress('extract', 'copy', 10, 10, 'bytes') # last one always goes out
rep.error('check.disk_space', {'path': 'D:\\x', 'need_gb': 40, 'free_gb': 3}, step='check')
rep.close()
events = [json.loads(line) for line in out.getvalue().splitlines()]
self.assertEqual([e['type'] for e in events], ['step', 'progress', 'progress', 'error'])
self.assertEqual([e.get('done') for e in events[1:3]], [0, 10])
self.assertTrue(all('t' in e for e in events))
log = (Path(d) / 'x.log').read_text(encoding='utf-8')
self.assertIn('40 GB needed, 3 GB free', log)
self.assertIn('[check.disk_space]', log)
def test_catalog_has_every_id_the_code_uses(self):
catalog = json.loads((ROOT / 'installer' / 'text' / 'en.json').read_text(encoding='utf-8'))
used = set()
for path in (ROOT / 'installer').glob('*.py'):
used |= set(re.findall(r"InstallError\('([a-z_.]+)'", path.read_text()))
used |= set(re.findall(r"warning\('([a-z_.]+)'", path.read_text()))
used |= {f'phase.{s}.{p}' for s, p in re.findall(r"progress\('(\w+)', '(\w+)'[,)]", path.read_text())}
self.assertEqual(sorted(i for i in used if i not in catalog), [])
if __name__ == '__main__':
unittest.main()
+90 -10
View File
@@ -7,6 +7,8 @@ recompiled only when it, a header it includes or its flags change.
.venv\\Scripts\\python tools\\build_game.py --modules subsdk0 # one module (links if it has no calls into others)
.venv\\Scripts\\python tools\\build_game.py --jobs 8
.venv\\Scripts\\python tools\\build_game.py --out build/v121-t2 # a separate build directory
.venv\\Scripts\\python tools\\build_game.py --toolchain toolchain/llvm-mingw-<version>/llvm-mingw-<version>-ucrt-x86_64 \\
--out build/v121-llvmmingw # llvm-mingw instead of MSYS2 UCRT64
Every runtime/*.c is compiled. build/<version>/modules.c carries the layout, the linked function tables, the
expected SHA-256 of each ExeFS file (tools/data/expected-<display version>.json) and the default game paths
@@ -34,6 +36,16 @@ Compile jobs share a memory budget (MEM_BUDGET, default the smaller of 50 GiB an
big machine does not run out of memory on the largest chunks. Each phase reports its wall time and the peak memory
of its processes (Windows job object accounting: committed private bytes).
--ninja PATH names the Ninja to run; without MSYS2 the bin/ninja.exe of the release folder (tools/make_release.py) is used.
--toolchain DIR builds with an llvm-mingw tree (DIR/bin/clang.exe) instead of MSYS2 UCRT64 clang. Ninja still comes
from MSYS2. The libraries and headers MSYS2 supplies (Vulkan, SDL3, Opus, libvpx, VMA) then come from --deps DIR
(default toolchain/deps-mingw: include/, lib/ with the import libraries and libvpx.a, bin/ with the DLLs), and the C++
runtime is libc++ from llvm-mingw. Such a build never reuses another build's generated objects.
Split build. --make-runtime-lib compiles only runtime/ into <out>/libtotkrt.a (what CI prebuilds). --runtime-lib PATH
then compiles only the generated C and the install-time files (modules.c and INSTALL_SIDE, which bake the player's
game hashes in) and links them with that library. The library holds no game-derived data.
Standard library only.
"""
import argparse
@@ -67,6 +79,15 @@ RT_FLAGS = ['-O2', '-g', '-Wall', '-fno-strict-aliasing', '-march=x86-64-v3']
LINK_FLAGS = ['-fuse-ld=lld', '-Wl,--stack,16777216', '-Wl,--Map=$builddir/totk.map'] # the map feeds tools/prof_report.py
LINK_LIBS = ['-lpsapi', '-lbcrypt', '-lvulkan-1', '-luser32', '-lgdi32', '-ldwmapi', '-lwinmm', '-lSDL3', '-lopus', '-l:libvpx.a', '-l:libstdc++.a', '-l:libwinpthread.a'] # Vulkan back end of M4; libvpx static (movies)
VMA_INCLUDE = UCRT.parent / 'include' / 'vma' # vk_mem_alloc.h; MSYS2 package vulkan-memory-allocator
# --toolchain: an llvm-mingw tree. The libraries, headers and DLLs MSYS2 supplies come from the deps directory instead.
LLVM_LINK_LIBS = ['-lpsapi', '-lbcrypt', '-luser32', '-lgdi32', '-ldwmapi', '-lwinmm', '-lvulkan-1', '-lSDL3', '-lopus',
'-l:libvpx.a', '-l:libc++.a', '-l:libc++abi.a', '-l:libunwind.a', '-l:libwinpthread.a']
DEFAULT_DEPS = ROOT / 'toolchain' / 'deps-mingw'
INSTALL_SIDE = {'romfs.c'} # runtime sources that include romfs_config.h: compiled per install, not in the library
LIB_NAME = 'libtotkrt.a'
AR = UCRT / 'llvm-ar.exe'
TOOLCHAIN_BIN = None # llvm-mingw bin directory, set by main() for --toolchain
DEPS = None # the deps directory of an llvm-mingw build
# tools/data/expected-<display version>.json for each build version name.
DISPLAY_VERSIONS = {'v0': '1.0.0', 'v112': '1.1.2', 'v121': '1.2.1', 'v143': '1.4.3'}
@@ -182,11 +203,14 @@ def module_glue(layout, linked, hashes, paths):
return '\n'.join(out)
def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False):
def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=None, make_lib=False):
"""build.ninja text, and the object lists per module. With prebuilt the generated objects already exist in
out/obj and are link inputs without a compile edge."""
out/obj and are link inputs without a compile edge. runtime_lib: a prebuilt library replaces the runtime objects
except INSTALL_SIDE. make_lib: build only that library, from the runtime objects except INSTALL_SIDE."""
inc = f'-I {flag_path(RUNTIME)} -I {flag_path(gen)}'
rt_inc = f'{inc} -I {flag_path(out)} -I {flag_path(VMA_INCLUDE)}' # romfs_config.h, vk_mem_alloc.h
vma = f'{flag_path(DEPS / "include")} -I {flag_path(DEPS / "include" / "vma")}' if DEPS else flag_path(VMA_INCLUDE)
rt_inc = f'{inc} -I {flag_path(out)} -I {vma}' # romfs_config.h, vk_mem_alloc.h (and the deps headers)
libs = [f'-L{flag_path(DEPS / "lib")}'] + LLVM_LINK_LIBS if DEPS else LINK_LIBS
lines = [
'# Written by tools/build_game.py; do not edit.',
'ninja_required_version = 1.10',
@@ -195,7 +219,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False):
f'genflags = {" ".join(GEN_FLAGS)} {inc}',
f'rtflags = {" ".join(RT_FLAGS)} {rt_inc}',
f'ldflags = {" ".join(LINK_FLAGS)}',
f'libs = {" ".join(LINK_LIBS)}',
f'libs = {" ".join(libs)}',
'',
'pool gen',
f' depth = {pool_depth}',
@@ -229,14 +253,35 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False):
lines.append(' pool = gen')
objs[mod].append(o)
rt = []
for src in runtime_sources(skip) + [out / 'modules.c']:
sources = runtime_sources(skip)
if make_lib:
sources = [p for p in sources if p.name not in INSTALL_SIDE]
elif runtime_lib:
sources = [p for p in sources if p.name in INSTALL_SIDE] + [out / 'modules.c']
else:
sources += [out / 'modules.c']
for src in sources:
o = runtime_object(out, src)
lines.append(f'build {ninja_path(o)}: cc_rt {ninja_path(src)}')
rt.append(o)
objs['runtime'] = rt
every = [o for v in objs.values() for o in v]
exe = out / 'totk.exe'
lines += ['', f'build {ninja_path(exe)}: link ' + ' '.join(ninja_path(o) for o in every),
if make_lib:
exe = out / LIB_NAME
lines += ['', 'rule ar', f' command = {AR.as_posix()} rcs $out $in', ' description = AR $out', '', f'build {ninja_path(exe)}: ar ' + ' '.join(ninja_path(o) for o in every),
'build objects: phony ' + ' '.join(ninja_path(o) for o in every), f'default {ninja_path(exe)}', '']
return '\n'.join(lines), objs, exe
rule = 'link'
if runtime_lib:
# Whole archive: the weak probe stubs in gfx_stubs.c would otherwise satisfy references to the probes before the
# linker looks into the library, and the real probes would stay out of totk.exe.
rule = 'link_lib'
lines += ['', 'rule link_lib',
f' command = $cc $ldflags @$out.rsp -Wl,--whole-archive {flag_path(runtime_lib)} -Wl,--no-whole-archive $libs -o $out',
' rspfile = $out.rsp', ' rspfile_content = $in_newline', ' description = LINK $out']
lines += ['', f'build {ninja_path(exe)}: {rule} ' + ' '.join(ninja_path(o) for o in every) +
(f' | {ninja_path(runtime_lib)}' if runtime_lib else ''),
'build objects: phony ' + ' '.join(ninja_path(o) for o in every), f'default {ninja_path(exe)}', '']
return '\n'.join(lines), objs, exe
@@ -517,6 +562,16 @@ def main():
ap.add_argument('--romfs', type=Path, help="totk.exe's default RomFS directory with romfs.bin and index.json "
'(default build/extracted/<version>/romfs)')
ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile)")
ap.add_argument('--toolchain', type=Path, help='llvm-mingw directory (with bin/clang.exe) to build with instead of '
'MSYS2 UCRT64 clang')
ap.add_argument('--ninja', type=Path, help='ninja.exe to build with (default: MSYS2 UCRT64, else bin/ninja.exe of the '
'release folder)')
ap.add_argument('--deps', type=Path, help=f'with --toolchain: include/, lib/ and bin/ of Vulkan, SDL3, Opus, libvpx '
f'and VMA (default {DEFAULT_DEPS.relative_to(ROOT).as_posix()})')
ap.add_argument('--make-runtime-lib', action='store_true',
help=f'build only <out>/{LIB_NAME} from runtime/ (no generated C, no totk.exe)')
ap.add_argument('--runtime-lib', type=Path, help=f'link against this {LIB_NAME} instead of compiling runtime/ '
f'({", ".join(sorted(INSTALL_SIDE))} and modules.c still compile)')
ap.add_argument('--expected', type=Path,
help='expected hashes (default tools/data/expected-<display version>.json)')
args = ap.parse_args()
@@ -533,6 +588,18 @@ def main():
sys.exit(f'no expected hashes known for {args.version}; pass --expected')
expected = ROOT / 'tools' / 'data' / f'expected-v{DISPLAY_VERSIONS[args.version]}.json'
hashes = json.loads(Path(expected).read_text())['exefs_sha256']
global CLANG, AR, NINJA, TOOLCHAIN_BIN, DEPS
if args.ninja:
NINJA = args.ninja.resolve()
elif not NINJA.is_file() and (ROOT / 'bin' / 'ninja.exe').is_file():
NINJA = ROOT / 'bin' / 'ninja.exe' # the release folder (tools/make_release.py)
dlls = SDL3_DLLS
if args.toolchain:
TOOLCHAIN_BIN = (args.toolchain / 'bin').resolve()
CLANG = TOOLCHAIN_BIN / 'clang.exe'
AR = TOOLCHAIN_BIN / 'llvm-ar.exe'
DEPS = (args.deps or DEFAULT_DEPS).resolve()
dlls = sorted((DEPS / 'bin').glob('*.dll'))
for tool in (CLANG, NINJA):
if not tool.is_file():
sys.exit(f'{tool} not found: install MSYS2 UCRT64 clang, lld and ninja (docs/toolchain-msys2-python.md)')
@@ -566,8 +633,12 @@ def main():
skip = {s for s in args.skip_runtime.split(',') if s}
main_out = (ROOT / 'build' / args.version).resolve()
pairs = None
if args.full_build:
if args.make_runtime_lib:
mods, linked, why = {}, [], ''
elif args.full_build:
why = '--full-build'
elif args.toolchain:
why = f'--toolchain {flag_path(args.toolchain)}'
elif out == main_out:
why = ''
else:
@@ -580,10 +651,14 @@ def main():
unlink_shared(out, mods)
if why:
print(f'build_game: compiling all generated C: {why}')
text, objs, exe = ninja_file(gen, out, mods, jobs, skip, pairs is not None)
runtime_lib = args.runtime_lib.resolve() if args.runtime_lib else None
if runtime_lib and not runtime_lib.is_file():
sys.exit(f'{runtime_lib} not found: build it with --make-runtime-lib')
text, objs, exe = ninja_file(gen, out, mods, jobs, skip, pairs is not None, runtime_lib,
args.make_runtime_lib)
write_if_changed(out / 'build.ninja', text)
env = dict(os.environ, PATH=str(UCRT) + os.pathsep + os.environ.get('PATH', ''))
env = dict(os.environ, PATH=str(TOOLCHAIN_BIN or UCRT) + os.pathsep + os.environ.get('PATH', ''))
base = [str(NINJA), '-f', str(out / 'build.ninja'), '-j', str(jobs)]
nfiles = sum(len(v) for v in mods.values())
print(f'build_game: {nfiles} generated files in {len(linked)} modules ({", ".join(linked)}), {jobs} jobs', flush=True)
@@ -598,13 +673,18 @@ def main():
verify_objects(out, objs, pairs is not None, main_out, pairs, base, env)
if args.compile_only:
return
if args.make_runtime_lib:
exe.unlink(missing_ok=True) # ar rcs would keep members of deleted sources
code, dt, peak_one, _ = run_measured(base + [flag_path(exe)], env)
print(f'link: {dt:.1f} s, peak {mib(peak_one)}')
if code:
print(f'build_game: link failed, {flag_path(exe)} is the old one (a running totk.exe from this dir locks it)',
file=sys.stderr)
sys.exit(code)
for dll in SDL3_DLLS: # dynamic SDL3 (zlib) and its libiconv (LGPL, unmodified DLL), found next to totk.exe
if args.make_runtime_lib:
print(f'{below_root(exe)}: {mib(exe.stat().st_size)}')
return
for dll in dlls: # dynamic SDL3 (zlib) and its libiconv (LGPL, unmodified DLL), found next to totk.exe
shutil.copy2(dll, exe.parent / dll.name)
sections, image = pe_sections(exe)
print(f'{below_root(exe)}: file {mib(exe.stat().st_size)}, '
+7 -1
View File
@@ -47,7 +47,9 @@ def list_files(staged: bool) -> list[Path]:
return [Path(p) for p in out.decode("utf-8").split("\0") if p]
def problems_for(path: Path) -> list[str]:
def problems_for(path: Path, allow_large: bool = False) -> list[str]:
"""allow_large: files over MAX_BYTES (toolchain binaries in a release folder) are checked for the generated marker
only, not refused."""
found = []
if path.suffix.lower() in BLOCKED_SUFFIXES:
found.append(f"blocked file type {path.suffix}")
@@ -56,6 +58,10 @@ def problems_for(path: Path) -> list[str]:
if not path.is_file():
return found
size = path.stat().st_size
if size > MAX_BYTES and allow_large:
if GENERATED_MARKER.encode() in path.read_bytes():
found.append("generated from game code")
return found
if size > MAX_BYTES:
found.append(f"{size // 1024} KiB, over the {MAX_BYTES // 1024} KiB limit")
return found
+59 -18
View File
@@ -9,7 +9,9 @@ Steps, each skipped when its verified output already exists:
Steps 1 and 3 run for every file in `input.updates`. Each update's version is identified by the SHA-256 of its `main`,
not by its file name. Output: build/extracted/v0/{exefs,romfs} and build/extracted/v<version>/{exefs,romfs}.
Runs under .venv-py314. Key values are read by the tools from the keys file and never printed.
--base, --update, --keys and --work name the files and the output directory instead of the config (the installer
imports the functions instead). Runs under .venv-py314. Key values are read by the tools from the keys file and never
printed.
"""
import argparse
import hashlib
@@ -45,6 +47,11 @@ def say(message):
print(message, flush=True)
def shown(path):
"""path relative to the repository when it is below it (log lines), else as given."""
return path.relative_to(ROOT) if path.is_relative_to(ROOT) else path
def sha256(path):
with path.open('rb') as f:
return hashlib.file_digest(f, 'sha256').hexdigest()
@@ -62,10 +69,17 @@ def version_dir(expected):
return 'v0' if expected['version'] == 0 else 'v' + expected['display_version'].replace('.', '')
# The embeddable Python (a ._pth file) does not put a script's directory on sys.path, so a vendored script that imports
# its siblings fails when started by path. This stub adds the directory, then runs the script as __main__.
RUN_SCRIPT = ('import os, runpy, sys; p = sys.argv[1]; sys.path.insert(0, os.path.dirname(p)); sys.argv = sys.argv[1:]; '
"runpy.run_path(p, run_name='__main__')")
def run_logged(command, log):
args = [str(c) for c in command]
with log.open('w') as out:
code = subprocess.run([str(c) for c in command], stdout=out, stderr=subprocess.STDOUT).returncode
require(code == 0, f'{Path(command[1]).name} failed; see {log.relative_to(ROOT)}')
code = subprocess.run([args[0], '-c', RUN_SCRIPT, *args[1:]], stdout=out, stderr=subprocess.STDOUT).returncode
require(code == 0, f'{Path(command[1]).name} failed; see {shown(log)}')
def exefs_problems(exefs, expected):
@@ -125,13 +139,9 @@ def extract_base_romfs(out, keys, logs):
'--manifest', out / 'extraction-manifest.json', '--output', romfs], logs / 'extract_romfs-v0.log')
def merge_update_romfs(base, update, keys_path, hactool, output, logs):
"""extract_update_romfs.merge() without its Darwin-only bounded view of the base: the base Program NCA is copied out.
Needs room for both Program NCAs (about 17 GB together) and the merged image (about 17 GB).
"""
if output.exists():
shutil.rmtree(output)
def program_ncas(base, update, keys_path):
"""(tickets, [(source, record, header)] of the base and the update Program NCA). Raises ValueError for a file
that is no TotK base game or update, or keys that cannot open it."""
keys = load_keys(keys_path)
tickets = {}
inputs = []
@@ -146,6 +156,28 @@ def merge_update_romfs(base, update, keys_path, hactool, output, logs):
rights = header[0x230:0x240]
require(rights == bytes(16) or rights in tickets, 'Missing common ticket')
inputs.append((source, record, header))
return tickets, inputs
def merge_space(base, update, keys_path):
"""Bytes merge_update_romfs needs free next to its output, beyond the merged image itself: the two Program NCAs."""
_, inputs = program_ncas(base, update, keys_path)
return inputs[0][1]['size'] + inputs[1][1]['size']
MERGE_PHASES = {'copy': ' copying the base and update Program NCAs', 'merge': ' merging with hactool',
'index': ' hashing and indexing the merged image'}
def merge_update_romfs(base, update, keys_path, hactool, output, logs, phase=lambda name: say(MERGE_PHASES[name])):
"""extract_update_romfs.merge() without its Darwin-only bounded view of the base: the base Program NCA is copied out.
Needs room for both Program NCAs (about 17 GB together) and the merged image (about 17 GB). phase(name) is called
as each part of MERGE_PHASES starts.
"""
if output.exists():
shutil.rmtree(output)
tickets, inputs = program_ncas(base, update, keys_path)
sections = [i for i in range(4) if inputs[1][2][0x400 + i * 512 + 4] == 4]
require(len(sections) == 1, 'Expected one BKTR section')
need = inputs[0][1]['size'] + inputs[1][1]['size'] + 18 * (1 << 30)
@@ -160,13 +192,13 @@ def merge_update_romfs(base, update, keys_path, hactool, output, logs):
f.write(f'{rights.hex()} = {key.hex()}\n')
hactool_keys = tmp / 'hactool.keys'
hactool_prod_keys(keys_path, hactool_keys)
say(' copying the base and update Program NCAs')
phase('copy')
base_nca, update_nca = tmp / 'base.nca', tmp / 'update.nca'
for (source, record, _), target in zip(inputs, (base_nca, update_nca)):
copy_range(source, record_offset(record), record['size'], target)
result = tmp / 'result'
result.mkdir()
say(' merging with hactool')
phase('merge')
command = [str(hactool), '-x', '-y', '--disablekeywarns', '--suppresskeys', '-k', str(hactool_keys),
f'--basenca={base_nca}', f'--section{sections[0]}={result / "romfs.bin"}', str(update_nca)]
environment = dict(os.environ, MNG_TITLE_KEYS=str(title_keys))
@@ -175,10 +207,10 @@ def merge_update_romfs(base, update, keys_path, hactool, output, logs):
log_path.write_text(log)
checked = '\n'.join(line for line in log.splitlines() if not UNSIGNED.match(line))
require(completed.returncode == 0 and not HACTOOL_ERROR.search(checked),
f'hactool merge failed; see {log_path.relative_to(ROOT)}')
f'hactool merge failed; see {shown(log_path)}')
require(all(re.search(rf'Level {i} \(GOOD\)', log) for i in range(6)), 'hactool did not verify all six IVFC levels')
image = result / 'romfs.bin'
say(' hashing and indexing the merged image')
phase('index')
files = index_romfs(image)
report = dict(status='pass', romfs_size=image.stat().st_size, romfs_sha256=sha256(image), files=files,
integrity='BKTR merge; all six IVFC levels verified')
@@ -190,12 +222,21 @@ def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--config', type=Path, default=ROOT / 'config.local.toml')
parser.add_argument('--hactool', type=Path, default=ROOT / 'toolchain' / 'hactool' / 'hactool.exe')
parser.add_argument('--base', type=Path, help='base game (default: input.base of the config)')
parser.add_argument('--update', type=Path, action='append',
help='an update NSP, repeatable (default: input.updates of the config)')
parser.add_argument('--keys', type=Path, help='prod.keys (default: input.keys of the config)')
parser.add_argument('--work', type=Path, help='output directory with extracted/ and logs/ (default: work.dir of '
'the config, else build)')
args = parser.parse_args()
config = tomllib.loads(args.config.read_text(encoding='utf-8'))
base, keys = Path(config['input']['base']), Path(config['input']['keys'])
updates = config['input'].get('updates', [])
work = ROOT / config.get('work', {}).get('dir', 'build')
config = {}
if not (args.base and args.keys and args.update and args.work):
config = tomllib.loads(args.config.read_text(encoding='utf-8'))
given = config.get('input', {})
base, keys = args.base or Path(given['base']), args.keys or Path(given['keys'])
updates = args.update or given.get('updates', [])
work = args.work or ROOT / config.get('work', {}).get('dir', 'build')
extracted, logs = work / 'extracted', work / 'logs'
logs.mkdir(parents=True, exist_ok=True)
versions = load_expected()
+509
View File
@@ -0,0 +1,509 @@
"""Assemble the release folder of docs/installer.md ("Release layout") from pinned downloads.
.venv-py314\\Scripts\\python tools\\make_release.py --out build/release/TotK-Setup [--zip build/release/TotK-Setup.zip]
.venv-py314\\Scripts\\python tools\\make_release.py --out build/release/TotK-Setup --prove
Needs Python 3.14 (tarfile reads the MSYS2 .zst packages), uv on PATH and network access on the first run. Downloads are
cached in toolchain/release-dl/ and checked against the sha256 in PINS; a mismatch stops the build. The folder holds:
python/ python.org embeddable 3.14.9 + the packages of tools/requirements-installer.txt + tkinter (the Tcl/Tk 9.0
files come out of the python.org installer's own component MSI, unpacked with msiexec /a and expand.exe;
the installer is never run)
llvm/ llvm-mingw, x86_64 only, the files clang and lld need to compile and link totk.exe
runtime/ libtotkrt.a (build_game.py --make-runtime-lib with llvm/), the runtime headers, romfs.c (compiled per
install), deps/ (headers and import libraries of SDL3, Opus, libvpx, Vulkan and VMA)
bin/ totk_xlate.exe (+ .id), hactool.exe, SDL3.dll (upstream), libopus-0.dll, ninja.exe
recomp/, tools/, data/, installer/, licenses/, THIRD_PARTY_NOTICES.md, totk.ini.example
VERSION git describe of the checkout (else the short hash); the installer's release string
TotK-Setup.exe installer/launcher/launcher.c built with llvm/ (-mwindows): starts python/pythonw.exe -m installer.gui
--prove compiles generated/v121 and links build/release-check/totk.exe with nothing but the folder's python/, llvm/,
runtime/ and bin/ (PATH holds the Windows system directory and those), then runs the four probes there. It needs a
generated/v121 and build/extracted/v121 from the usual game build (they are read, nothing is written to them).
The folder contains nothing derived from the game: check_content runs over it before the script finishes. Then the
folder's own python runs a check-only installer pass (no game files: it must get through the release, toolchain and host
checks and stop at the missing game files) and the window's --selftest.
"""
import argparse
import hashlib
import json
import os
import shutil
import struct
import subprocess
import sys
import tarfile
import time
import urllib.request
import zipfile
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import check_content # noqa: E402
ROOT = Path(__file__).resolve().parent.parent
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/, generated/, toolchain/ and refs/ are junctions in a worktree
DL = MAIN_ROOT / 'toolchain' / 'release-dl'
WORK = MAIN_ROOT / 'toolchain' / 'release-work' # unpacked downloads
MSYS2 = 'https://repo.msys2.org/mingw/ucrt64/'
SDL_VERSION = '3.4.18'
LLVM_VERSION = '20261020'
MNG_COMMIT = 'b75be8cfa704df3701e86130c8ced268e79d6b11' # MacNativeGames, the shader list (refs/MacNativeGames)
# name: (url, sha256)
PINS = {
'python-embed': ('https://www.python.org/ftp/python/3.14.9/python-3.14.9-embed-amd64.zip',
'3c01d7cfc666f746a6f064f60fb2156ab0ec464d3d738ab796371a46a9bb24a2'),
'python-installer': ('https://www.python.org/ftp/python/3.14.9/python-3.14.9-amd64.exe',
'a247a0b612244a803f507b39fb7121e3602e4160d59c3b99d98e735289c53394'),
'llvm-mingw': (f'https://github.com/mstorsjo/llvm-mingw/releases/download/{LLVM_VERSION}/'
f'llvm-mingw-{LLVM_VERSION}-ucrt-x86_64.zip',
'fd5462b7f5d93b2ff981fcd3f9c7e427e47f3203c64ea91c27bcc444eebd74cd'),
'sdl3-dll': (f'https://github.com/libsdl-org/SDL/releases/download/release-{SDL_VERSION}/SDL3-{SDL_VERSION}-win32-x64.zip',
'75c2c0fc74e7d1206aaedc22893a35887332f4f85f5c8b74fddc10d177985f2c'),
'sdl3-devel': (f'https://github.com/libsdl-org/SDL/releases/download/release-{SDL_VERSION}/SDL3-devel-{SDL_VERSION}-mingw.tar.gz',
'049b9dd711e3fb3f8d84feb998dfec54a75e17cf539c4a80ff4d058d839771cf'),
'ninja': ('https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-win.zip',
'07fc8261b42b20e71d1720b39068c2e14ffcee6396b76fb7a795fb460b78dc65'),
'opus': (MSYS2 + 'mingw-w64-ucrt-x86_64-opus-1.6.1-2-any.pkg.tar.zst',
'bb114bcf26225f50f874497ebfd6b8224fff2a20bda3f8ee45cd07ce5f02657b'),
'libvpx': (MSYS2 + 'mingw-w64-ucrt-x86_64-libvpx-1.17.0-1-any.pkg.tar.zst',
'8ff6919dbb88ea26ce28d22fd81d6f0d6e200cf56a4dd81ffc2d33124fe98024'),
'libvpx-src': ('https://github.com/webmproject/libvpx/archive/refs/tags/v1.17.0.tar.gz', # LICENSE and PATENTS only
'1020f184046187baa2985dbde38e0691f49c44088bca7a1842b0236c6081dc0a'),
'vulkan-headers': (MSYS2 + 'mingw-w64-ucrt-x86_64-vulkan-headers-1~1.4.363.0-1-any.pkg.tar.zst',
'be602b2657cace531316e78b1eda4087e8f7f84ebb877f08fba198a2db06406a'),
'vulkan-loader': (MSYS2 + 'mingw-w64-ucrt-x86_64-vulkan-loader-1~1.4.363.0-1-any.pkg.tar.zst', # import library only
'065076fe13d39ab97eea4740596426d5ae741b5044e48e1c9495d04835c92dbe'),
'vma': (MSYS2 + 'mingw-w64-ucrt-x86_64-vulkan-memory-allocator-3.4.0-1-any.pkg.tar.zst',
'44c81162337915afecda3185f66fd98268a54454580b2717080c9dfef57fe781'),
}
# The component MSI inside the python.org installer that carries Tcl/Tk and tkinter (Burn names its payloads a0..a27).
TCLTK_PAYLOAD = ('a12', 'ac87609db3f715404dd41a989c2f1706ea1512f3d86217f938d11c63af8eb505')
SDL3_DLL_SHA256 = '2b4b128d8d510cc2d442693cf848c050d700d43dc667afe9e5fd82b0570d667e' # SDL3.dll inside the SDL zip
# llvm-mingw: what clang and lld need to compile and link totk.exe for x86_64-w64-mingw32 (dynamic LLVM, libc++, compiler-rt).
LLVM_BIN = {'clang.exe', 'clang++.exe', 'clang-23.exe', 'ld.lld.exe', 'llvm-ar.exe', 'llvm-ranlib.exe', 'llvm-nm.exe',
'llvm-objdump.exe', 'llvm-objcopy.exe', 'llvm-strip.exe', 'llvm-readobj.exe', 'llvm-dlltool.exe',
'llvm-rc.exe', 'llvm-cvtres.exe', 'llvm-windres.exe', 'llvm-wrapper.exe', 'mingw32-common.cfg',
'x86_64-w64-windows-gnu.cfg', 'x86_64-pc-windows-gnu.cfg', 'x86_64-w64-mingw32-clang.exe',
'x86_64-w64-mingw32-clang++.exe', 'libLLVM-23.dll', 'libclang-cpp.dll', 'libc++.dll', 'libunwind.dll',
'libwinpthread-1.dll', 'libffi-8.dll'}
LLVM_SKIP_DIRS = {'aarch64-w64-mingw32', 'armv7-w64-mingw32', 'i686-w64-mingw32', 'busybox', 'python', 'share'}
LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'}
# tools/ files the installer steps run (and what they import), copied from the repo.
TOOLS_FILES = ['build_game.py', 'check_content.py', 'extract.py', 'make_romfs_patches.py', 'prepare_shaders.py',
'romfs_index.py', 'shader_census.py']
TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes
RUNTIME_SUFFIXES = {'.h', '.inc'}
PROBES = ['--probe-kernel', '--probe-vkexec', '--probe-nvn-tex', '--probe-shader']
def sha256_file(path):
h = hashlib.sha256()
with open(path, 'rb') as f:
for block in iter(lambda: f.read(1 << 20), b''):
h.update(block)
return h.hexdigest()
def fetch(name):
"""The pinned download, from the cache or the network, checked against its sha256."""
url, want = PINS[name]
DL.mkdir(parents=True, exist_ok=True)
path = DL / Path(url.split('?')[0]).name.replace('%7E', '~')
if name == 'libvpx-src':
path = DL / 'libvpx-1.17.0.tar.gz'
if not path.is_file():
print(f'download {url}')
with urllib.request.urlopen(url) as r, open(path, 'wb') as f:
shutil.copyfileobj(r, f)
got = sha256_file(path)
if got != want:
sys.exit(f'{path}: sha256 {got}, pinned {want}')
return path
def unpack(name, member_filter=None):
"""Extract a pinned archive into toolchain/release-work/<name> (once) and return that directory."""
dest = WORK / name
stamp = dest / '.unpacked'
if stamp.is_file():
return dest
shutil.rmtree(dest, ignore_errors=True)
dest.mkdir(parents=True)
path = fetch(name)
if path.suffix == '.zip':
with zipfile.ZipFile(path) as z:
z.extractall(dest)
else:
with tarfile.open(path) as t:
t.extractall(dest, filter='data')
stamp.write_text('')
return dest
def copy_tree(src, dst, skip=lambda rel: False):
"""Copy src into dst file by file (sorted), leaving out what skip(relative posix path) names."""
for p in sorted(src.rglob('*')):
rel = p.relative_to(src)
if p.is_dir() or skip(rel.as_posix()):
continue
(dst / rel).parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(p, dst / rel)
def clean_env(*path_dirs):
"""A process environment without MSYS2 and the user's tools: the Windows system directory and path_dirs only."""
system = os.environ.get('SystemRoot', r'C:\Windows')
keep = ('SYSTEMROOT', 'SYSTEMDRIVE', 'TEMP', 'TMP', 'USERPROFILE', 'LOCALAPPDATA', 'APPDATA', 'COMSPEC', 'PATHEXT',
'NUMBER_OF_PROCESSORS') # os.environ keys are upper case on Windows
env = {k: v for k, v in os.environ.items() if k.upper() in keep}
env['PATH'] = os.pathsep.join([*(str(Path(d)) for d in path_dirs), rf'{system}\System32', system])
return env
def run(cmd, **kw):
print('+', ' '.join(str(c) for c in cmd), flush=True)
return subprocess.run([str(c) for c in cmd], check=True, **kw)
def mib(n):
return f'{n / (1 << 20):.1f} MiB'
def dir_size(p):
return sum(f.stat().st_size for f in Path(p).rglob('*') if f.is_file())
# ---- python/ -------------------------------------------------------------------------------------------------------
def tcltk_files():
"""Directory holding the python.org installer's tcltk payload, unpacked (not installed) into toolchain/release-work."""
dest = WORK / 'tcltk'
if (dest / 'DLLs' / '_tkinter.pyd').is_file():
return dest
exe = fetch('python-installer')
data = exe.read_bytes()
cabs = []
pos = data.find(b'MSCF\0\0\0\0')
while pos >= 0:
cabs.append((struct.unpack_from('<I', data, pos + 8)[0], pos))
pos = data.find(b'MSCF\0\0\0\0', pos + 1)
size, pos = max(cabs) # the attached container holds all component MSIs
shutil.rmtree(WORK / 'tcltk-src', ignore_errors=True)
src = WORK / 'tcltk-src'
src.mkdir(parents=True)
(src / 'container.cab').write_bytes(data[pos:pos + size])
name, want = TCLTK_PAYLOAD
expand = Path(os.environ.get('SystemRoot', r'C:\Windows')) / 'System32' / 'expand.exe'
run([expand, '-F:' + name, src / 'container.cab', src], stdout=subprocess.DEVNULL)
got = sha256_file(src / name)
if got != want:
sys.exit(f'{src / name}: sha256 {got}, pinned {want} (not the tcltk component of python 3.14.9)')
msi = src / (name + '.msi')
(src / name).rename(msi)
shutil.rmtree(dest, ignore_errors=True)
# Administrative image: copies the MSI's files to TARGETDIR, installs nothing and shows no window.
run(['msiexec', '/a', msi, '/qn', f'TARGETDIR={dest}'])
if not (dest / 'DLLs' / '_tkinter.pyd').is_file():
sys.exit(f'{dest}: no _tkinter.pyd after msiexec /a')
return dest
def build_python(out):
py = out / 'python'
with zipfile.ZipFile(fetch('python-embed')) as z:
z.extractall(py)
(py / 'python314._pth').write_text('python314.zip\n.\nLib/site-packages\n..\n../tools\n../recomp\n', encoding='utf-8')
# The embeddable Python ignores the script directory and PYTHONPATH once a ._pth file exists, so the folder's own code
# (installer/, and tools/*.py and recomp/*.py importing their siblings) has to be on the list.
env = dict(os.environ, UV_CACHE_DIR=str(MAIN_ROOT / 'toolchain' / 'uv-cache'))
run(['uv', 'pip', 'install', '--target', py / 'Lib' / 'site-packages', '--python', sys.executable,
'--python-platform', 'windows', '--python-version', '3.14', '--only-binary', ':all:',
'-r', ROOT / 'tools' / 'requirements-installer.txt'], env=env)
shutil.rmtree(py / 'Lib' / 'site-packages' / 'bin', ignore_errors=True) # numpy/cffi console scripts
tk = tcltk_files()
for dll in ('_tkinter.pyd', 'tcl90.dll', 'tcl9tk90.dll', 'zlib1.dll'):
shutil.copy2(tk / 'DLLs' / dll, py / dll)
copy_tree(tk / 'Lib' / 'tkinter', py / 'tkinter', lambda r: r.startswith('__pycache__'))
copy_tree(tk / 'tcl', py / 'tcl', lambda r: r.endswith(('.lib', '.sh')) or r.startswith('nmake'))
check = subprocess.run([str(py / 'python.exe'), '-I', '-c',
'import tkinter, numpy, lz4.block, cryptography, cffi, pycparser; '
'print("python ok: Tcl", tkinter.Tcl().eval("info patchlevel"), "numpy", numpy.__version__)'],
env=clean_env(py), capture_output=True, text=True)
if check.returncode:
sys.exit('bundled python check failed:\n' + check.stderr)
print(check.stdout.strip())
# ---- llvm/ ---------------------------------------------------------------------------------------------------------
def build_llvm(out):
src = unpack('llvm-mingw') / f'llvm-mingw-{LLVM_VERSION}-ucrt-x86_64'
dst = out / 'llvm'
def skip(rel):
top = rel.split('/')[0]
if top in LLVM_SKIP_DIRS:
return True
if top == 'bin':
return rel not in {f'bin/{n}' for n in LLVM_BIN}
if top == 'lib': # lib/clang/<major>/include and the x86_64 Windows builtins; no sanitizers, no other targets
if rel.startswith('lib/clang/') and '/include/' in rel:
return False
return not rel.endswith('/lib/windows/libclang_rt.builtins-x86_64.a')
return Path(rel).suffix.lower() in LLVM_SKIP_SUFFIXES or rel.startswith('include/ddk/')
copy_tree(src, dst, skip)
return dst
# ---- bin/ and deps ---------------------------------------------------------------------------------------------------
def build_bin(out):
b = out / 'bin'
b.mkdir(parents=True, exist_ok=True)
for f in ('totk_xlate.exe', 'totk_xlate.id'):
shutil.copy2(MAIN_ROOT / 'build' / 'xlate' / f, b / f)
shutil.copy2(MAIN_ROOT / 'toolchain' / 'hactool' / 'hactool.exe', b / 'hactool.exe')
dll = unpack('sdl3-dll') / 'SDL3.dll'
if sha256_file(dll) != SDL3_DLL_SHA256:
sys.exit(f'{dll}: not the pinned SDL3.dll')
shutil.copy2(dll, b / 'SDL3.dll')
shutil.copy2(unpack('opus') / 'ucrt64' / 'bin' / 'libopus-0.dll', b / 'libopus-0.dll')
shutil.copy2(unpack('ninja') / 'ninja.exe', b / 'ninja.exe')
def build_deps(out):
"""runtime/deps: the headers and import libraries build_game.py --deps expects (include/, lib/)."""
d = out / 'runtime' / 'deps'
(d / 'include').mkdir(parents=True, exist_ok=True)
(d / 'lib').mkdir(parents=True, exist_ok=True)
sdl = unpack('sdl3-devel') / f'SDL3-{SDL_VERSION}' / 'x86_64-w64-mingw32'
copy_tree(sdl / 'include', d / 'include')
shutil.copy2(sdl / 'lib' / 'libSDL3.dll.a', d / 'lib')
for pkg, dirs, libs in (('opus', ['opus'], ['libopus.dll.a']), ('libvpx', ['vpx'], ['libvpx.a']),
('vulkan-headers', ['vulkan', 'vk_video'], []), ('vulkan-loader', [], ['libvulkan-1.dll.a']),
('vma', ['vma'], [])):
root = unpack(pkg) / 'ucrt64'
for inc in dirs:
copy_tree(root / 'include' / inc, d / 'include' / inc)
for lib in libs:
shutil.copy2(root / 'lib' / lib, d / 'lib')
def collect_licenses(out):
lic = out / 'licenses'
lic.mkdir(parents=True, exist_ok=True)
pairs = [
(unpack('opus') / 'ucrt64' / 'share' / 'licenses' / 'opus' / 'COPYING', 'opus-COPYING.txt'),
(unpack('vulkan-headers') / 'ucrt64' / 'share' / 'licenses' / 'vulkan-headers' / 'LICENSE', 'vulkan-headers-LICENSE.txt'),
(unpack('vulkan-loader') / 'ucrt64' / 'share' / 'licenses' / 'vulkan-loader' / 'LICENSE', 'vulkan-loader-LICENSE.txt'),
(unpack('vma') / 'ucrt64' / 'share' / 'licenses' / 'VulkanMemoryAllocator' / 'LICENSE', 'vulkan-memory-allocator-LICENSE.txt'),
(unpack('sdl3-dll') / 'LICENSE.txt', 'SDL3-LICENSE.txt'),
(MAIN_ROOT / 'toolchain' / 'hactool' / 'LICENSE', 'hactool-LICENSE.txt'),
(MAIN_ROOT / 'refs' / 'MacNativeGames' / 'LICENSE', 'MacNativeGames-LICENSE.txt'),
]
for src, name in pairs:
shutil.copy2(src, lic / name)
with tarfile.open(fetch('libvpx-src')) as t: # the MSYS2 package carries no licence text; the notice needs both
for member, name in (('libvpx-1.17.0/LICENSE', 'libvpx-LICENSE.txt'), ('libvpx-1.17.0/PATENTS', 'libvpx-PATENTS.txt')):
(lic / name).write_bytes(t.extractfile(member).read())
for name in ('LICENSE.TXT',):
shutil.copy2(out / 'llvm' / name, lic / 'llvm-mingw-LICENSE.txt')
# ---- runtime/, recomp/, tools/, data/ ---------------------------------------------------------------------------------
def runtime_lib(out, llvm):
"""Build libtotkrt.a with the folder's llvm/ and copy it, the headers and romfs.c into runtime/."""
build = MAIN_ROOT / 'build' / 'release-check' / 'rtlib'
gen = MAIN_ROOT / 'generated' / 'v121'
run([sys.executable, ROOT / 'tools' / 'build_game.py', '--make-runtime-lib', '--toolchain', llvm,
'--deps', out / 'runtime' / 'deps', '--ninja', out / 'bin' / 'ninja.exe', '--gen', gen, '--out', build],
env=clean_env(llvm / 'bin', out / 'bin'))
rt = out / 'runtime'
shutil.copy2(build / 'libtotkrt.a', rt / 'libtotkrt.a')
for p in sorted((ROOT / 'runtime').rglob('*')):
if p.suffix in RUNTIME_SUFFIXES and p.is_file():
(rt / p.relative_to(ROOT / 'runtime')).parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(p, rt / p.relative_to(ROOT / 'runtime'))
shutil.copy2(ROOT / 'runtime' / 'romfs.c', rt / 'romfs.c') # build_game.py INSTALL_SIDE
def build_sources(out):
copy_tree(ROOT / 'recomp', out / 'recomp', lambda r: not r.endswith(('.py', '.txt')) or '__pycache__' in r)
for f in TOOLS_FILES:
(out / 'tools').mkdir(exist_ok=True)
shutil.copy2(ROOT / 'tools' / f, out / 'tools' / f)
for d in TOOLS_DIRS:
copy_tree(ROOT / 'tools' / d, out / 'tools' / d, lambda r: '__pycache__' in r or r.endswith('.md'))
(out / 'data').mkdir(exist_ok=True)
text = subprocess.run(['git', '-C', MAIN_ROOT / 'refs' / 'MacNativeGames', 'show',
f'{MNG_COMMIT}:games/totk/assets/shader_list.tsv'], check=True, capture_output=True).stdout
(out / 'data' / 'shader_list.tsv').write_bytes(text)
copy_tree(ROOT / 'installer', out / 'installer', lambda r: '__pycache__' in r or r.startswith('launcher/'))
shutil.copy2(ROOT / 'totk.ini.example', out / 'totk.ini.example')
shutil.copy2(ROOT / 'THIRD_PARTY_NOTICES.md', out / 'THIRD_PARTY_NOTICES.md')
(out / 'VERSION').write_text(version() + '\n', encoding='utf-8')
def version():
"""git describe of the checkout, else its short hash."""
for args in (['describe', '--tags', '--always'], ['rev-parse', '--short', 'HEAD']):
r = subprocess.run(['git', '-C', ROOT, *args], capture_output=True, text=True)
if r.returncode == 0 and r.stdout.strip():
return r.stdout.strip()
sys.exit('no git version: not a git checkout')
# ---- TotK-Setup.exe --------------------------------------------------------------------------------------------------
def c_wide(text):
"""A C wide string literal of text: ASCII as is, everything else as a universal character name."""
bs = chr(92)
out = []
for ch in text:
if ch in (bs, '"'):
out.append(bs + ch)
elif 32 <= ord(ch) < 127:
out.append(ch)
elif ord(ch) < 0x10000:
out.append(f'{bs}u{ord(ch):04x}')
else:
sys.exit(f'launcher text: {ch!r} is outside the BMP')
return 'L"' + ''.join(out) + '"'
def launcher_header(path):
"""launcher_text.h: the launcher's two messages in English and Russian, taken from the installer's catalog."""
lines = ['/* generated by tools/make_release.py from installer/text/{en,ru}.json */']
for lang in ('en', 'ru'):
texts = json.loads((ROOT / 'installer' / 'text' / f'{lang}.json').read_text(encoding='utf-8'))
for key in ('title', 'python_missing'):
lines.append(f'#define LAUNCHER_{key.upper()}_{lang.upper()} {c_wide(texts["launcher." + key])}')
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text('\n'.join(lines) + '\n', encoding='ascii')
def build_launcher(out, llvm):
work = MAIN_ROOT / 'build' / 'release-check' / 'launcher'
launcher_header(work / 'launcher_text.h')
run([llvm / 'bin' / 'clang.exe', '-O2', '-mwindows', '-Wall', '-Wl,-s', '-I', work,
ROOT / 'installer' / 'launcher' / 'launcher.c', '-o', out / 'TotK-Setup.exe'], env=clean_env(llvm / 'bin'))
# ---- checks ----------------------------------------------------------------------------------------------------------
def check_folder(out):
"""check_content over every file of the release folder; a hit stops the build."""
bad = []
for p in sorted(out.rglob('*')):
if p.is_file():
bad += [f'{p.relative_to(out)}: {problem}' for problem in check_content.problems_for(p, allow_large=True)]
if bad:
sys.exit('game-derived content in the release folder:\n' + '\n'.join(bad))
print(f'check_content: {sum(1 for p in out.rglob("*") if p.is_file())} files, no hit')
def check_bundle(out):
"""The folder's own python runs the installer without game files and the window's selftest."""
chk = MAIN_ROOT / 'build' / 'release-check' / 'bundle'
shutil.rmtree(chk, ignore_errors=True)
env = clean_env(out / 'python')
py = out / 'python' / 'python.exe'
r = subprocess.run([py, '-m', 'installer', '--lang', 'en-US', '--dest', chk / 'install', '--temp', chk / 'temp'],
cwd=out, env=env, capture_output=True, text=True, timeout=600)
events = [json.loads(line) for line in r.stdout.splitlines() if line.startswith('{')]
errors = [e['id'] for e in events if e['type'] == 'error']
started = any(e['type'] == 'start' and e.get('release') == (out / 'VERSION').read_text().strip() for e in events)
steps = [e['phase'] for e in events if e['type'] == 'progress']
if errors != ['check.inputs_needed'] or not started or 'inputs' not in steps:
sys.exit(f'bundle check: exit {r.returncode}, errors {errors}, release read {started}\n{r.stdout[-2000:]}\n{r.stderr[-2000:]}')
print(f'bundle check: the installer ran the host check from the folder and stopped at the game files (exit {r.returncode})')
r = subprocess.run([py, '-m', 'installer.gui', '--selftest'], cwd=out, env=env, capture_output=True, text=True, timeout=300)
if r.returncode:
sys.exit(f'bundle check: window selftest exit {r.returncode}\n{r.stdout[-2000:]}\n{r.stderr[-2000:]}')
print('bundle check: the window built every screen in both languages and closed')
def sizes(out):
rows = [(p.name if p.is_dir() else p.name, dir_size(p) if p.is_dir() else p.stat().st_size) for p in sorted(out.iterdir())]
for name, n in rows:
print(f' {name:26} {mib(n):>12}')
print(f' {"total":26} {mib(sum(n for _, n in rows)):>12}')
def write_zip(out, path):
path.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(path, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as z:
for p in sorted(out.rglob('*')):
if p.is_file():
info = zipfile.ZipInfo((Path(out.name) / p.relative_to(out)).as_posix(), (1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = 0o644 << 16
z.writestr(info, p.read_bytes(), compresslevel=6)
print(f'{path}: {mib(path.stat().st_size)}')
def prove(out):
"""Compile generated/v121 and link totk.exe with the folder's tools only, then run the probes next to it."""
out = out.resolve()
gen = MAIN_ROOT / 'generated' / 'v121'
ext = MAIN_ROOT / 'build' / 'extracted' / 'v121'
chk = MAIN_ROOT / 'build' / 'release-check'
env = clean_env(out / 'python', out / 'llvm' / 'bin', out / 'bin')
t0 = time.time()
run([out / 'python' / 'python.exe', out / 'tools' / 'build_game.py', '--toolchain', out / 'llvm',
'--deps', out / 'runtime' / 'deps', '--runtime', out / 'runtime', '--runtime-lib', out / 'runtime' / 'libtotkrt.a',
'--ninja', out / 'bin' / 'ninja.exe', '--gen', gen, '--out', chk, '--exefs', ext / 'exefs', '--romfs', ext / 'romfs',
'--profile', chk / 'profile'], env=env)
print(f'prove: build {time.time() - t0:.0f} s')
for dll in ('SDL3.dll', 'libopus-0.dll'):
shutil.copy2(out / 'bin' / dll, chk / dll)
failed = []
for probe in PROBES:
r = subprocess.run([str(chk / 'totk.exe'), probe], cwd=chk, env=clean_env(chk), capture_output=True, text=True,
timeout=900)
tail = (r.stdout.strip().splitlines() or [''])[-1]
print(f'{probe}: exit {r.returncode}: {tail}')
if r.returncode:
failed.append(probe)
if failed:
sys.exit(f'probes failed: {failed}')
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--out', type=Path, required=True, help='release folder (replaced)')
ap.add_argument('--zip', type=Path, help='also write the folder as this zip')
ap.add_argument('--prove', action='store_true', help='only run the compile-and-probe check on an existing folder')
args = ap.parse_args()
out = args.out.resolve()
if args.prove:
prove(out)
return
if out.exists():
shutil.rmtree(out)
out.mkdir(parents=True)
build_python(out)
llvm = build_llvm(out)
build_bin(out)
build_deps(out)
runtime_lib(out, llvm)
build_sources(out)
build_launcher(out, llvm)
collect_licenses(out)
check_folder(out)
check_bundle(out)
sizes(out)
if args.zip:
write_zip(out, args.zip.resolve())
if __name__ == '__main__':
main()
+15 -4
View File
@@ -16,10 +16,12 @@ pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "sta
a 12-core machine (inventory 1, translation 3, library builds 0.6) and 1.3 GB of shader cache; the seed files can be
deleted afterwards.
Game-derived output stays in the build directory (ignored). The list is read from refs/ at preparation time; nothing
of it is copied into the repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314:
Game-derived output stays in the build directory (ignored). The list is read at preparation time from tools/data/shader_list.tsv when that file exists (the installer archive
bundles it; it stays out of the repository), else from refs/ through git; nothing of it is copied into the
repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14:
.venv-py314\\Scripts\\python tools\\prepare_shaders.py build\\v121 [--version v121] [--list <tsv>] [--no-run]
[--romfs DIR --exefs DIR] # instead of build\\extracted\\<version>
"""
import argparse
import hashlib
@@ -50,9 +52,16 @@ CONTROL = 0x880
ANS_REC = 4 + 32 + 32 + 147 + 1
BUNDLED_LIST = ROOT / 'tools' / 'data' / 'shader_list.tsv' # the installer archive carries a copy (no git, no refs/)
if not BUNDLED_LIST.is_file():
BUNDLED_LIST = ROOT / 'data' / 'shader_list.tsv' # release folder layout (tools/make_release.py)
def read_list(path):
if path:
return Path(path).read_text(encoding='utf-8')
if BUNDLED_LIST.is_file():
return BUNDLED_LIST.read_text(encoding='utf-8')
return subprocess.run(['git', '-C', str(ROOT / 'refs' / 'MacNativeGames'), 'show', ':'.join(LIST_REF)],
check=True, capture_output=True, encoding='utf-8').stdout
@@ -111,13 +120,15 @@ def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('build', type=Path, help='build directory holding totk.exe (build\\v121)')
ap.add_argument('--version', default='v121', help='build/extracted/<version>')
ap.add_argument('--list', help='a shader_list.tsv instead of the one in refs/MacNativeGames')
ap.add_argument('--romfs', type=Path, help='RomFS directory (default build/extracted/<version>/romfs)')
ap.add_argument('--exefs', type=Path, help='ExeFS directory (default build/extracted/<version>/exefs)')
ap.add_argument('--list', help='a shader_list.tsv instead of tools/data/shader_list.tsv or refs/MacNativeGames')
ap.add_argument('--no-run', action='store_true', help='write the seed files only')
a = ap.parse_args()
t0 = time.monotonic()
variants, prio = parse_list(read_list(a.list))
extracted = ROOT / 'build' / 'extracted' / a.version
shaders, _ = inventory(extracted / 'romfs', extracted / 'exefs')
shaders, _ = inventory(a.romfs or extracted / 'romfs', a.exefs or extracted / 'exefs')
t1 = time.monotonic()
progs, recs, skipped = {}, [], {'not in this version': 0, 'stage': 0}
for key, deps in variants:
+7
View File
@@ -0,0 +1,7 @@
# Packages of the release's bundled Python 3.14.9 (tools/make_release.py): what the installer steps import.
# numpy: recomp/gen.py. lz4: recomp/nso.py, tools/shader_census.py. cryptography, cffi, pycparser: tools/extract.py and the vendored extract scripts.
numpy==2.5.4
lz4==4.4.5
cryptography==50.0.2
cffi==2.1.1
pycparser==3.11