mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-19 12:25:56 +00:00
Merge the installer: setup steps, window, launcher and release bundle
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
3505258a06
26 files changed
+4178
-35
No files matched your search
+22
-2
@@ -19,7 +19,7 @@ Code copied or adapted from another project is listed here with its source, the
|
||||
| deko3d block-linear image layout, as ported in MacNativeGames' `nvn_tex.c` (Apple parts dropped, layout math unchanged; licence text in `games/totk/third_party/d2r_gfx/LICENSE.deko3d`, reproduced in the file header) | https://github.com/devkitPro/deko3d (`source/dk_image.cpp`), via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/nvn_tex.c`, `nvn_tex.h`), Copyright (C) 2018-2020 fincs | zlib | M4 task T3b: `runtime/nvn_tex.c`, `runtime/nvn_tex.h` |
|
||||
| MacNativeGames texture auto-layout goldens (40 original-SDK size and block-height cases) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/tests/texture_auto_layout/probe.c`), Copyright (c) 2026 jhaizhou | MIT | M4 task T3b: `runtime/nvn_tex_probe.c`; T3f: `runtime/nvn_probe.c` (four of the cases, through the procs) |
|
||||
| MacNativeGames block-linear deswizzler (address function; copy loop split out as `tex_bl_copy`) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/tex/nvn_deswizzle.c`), Copyright (c) 2026 jhaizhou | MIT | M4 task T5a: `runtime/tex_deswizzle.c`, `runtime/tex.h` |
|
||||
| SDL 3.4.18 (MSYS2 `mingw-w64-ucrt-x86_64-SDL3`), dynamic link, `SDL3.dll` and its dependency `libiconv-2.dll` (LGPL-2.1, unmodified, not linked into the exe) copied next to `totk.exe` | https://libsdl.org, Copyright (c) 1997-2026 Sam Lantinga | zlib | `runtime/input_win.c` (gamepad input, rumble, gyro and accelerometer) |
|
||||
| SDL 3.4.18, dynamic link, `SDL3.dll` copied next to `totk.exe`. Developer builds use the MSYS2 build (`mingw-w64-ucrt-x86_64-sdl3`, which imports `libiconv-2.dll`, LGPL-2.1, unmodified, copied beside it); the installer release ships the upstream release `SDL3-3.4.18-win32-x64.zip`, which imports no libiconv | https://libsdl.org, Copyright (c) 1997-2026 Sam Lantinga | zlib | `runtime/input_win.c` (gamepad input, rumble, gyro and accelerometer) |
|
||||
| libopus (MSYS2 `mingw-w64-ucrt-x86_64-opus`), dynamic link, `libopus-0.dll` copied next to `totk.exe` | https://opus-codec.org, Copyright (c) 2001-2023 Xiph.Org, Skype Limited, Octasic, Jean-Marc Valin, Timothy B. Terriberry, CSIRO, Gregory Maxwell, Mark Borgerding, Erik de Castro Lopo, Mozilla, Amazon | BSD-3-Clause | `runtime/ov_media.c` (nn::codec hardware Opus decoder) |
|
||||
| Vulkan Memory Allocator 3.4.0 (MSYS2 `mingw-w64-ucrt-x86_64-vulkan-memory-allocator`), header-only, compiled into `totk.exe` | https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator, Copyright (c) 2017-2026 Advanced Micro Devices, Inc. | MIT | `runtime/vk_vma.cpp` (the allocator behind `runtime/vk_memory.c`) |
|
||||
| MacNativeGames host movie decoder (`movie::Decoder` for VP9 over libvpx: entry list, buffer and event semantics, NV12 output format, PTS pacing) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/runtime/movie_host.c`, `movie_host.h`), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/ov_movie.c`, `runtime/movie.h` |
|
||||
@@ -31,6 +31,26 @@ These tools run on the developer's or player's machine during the build. They ar
|
||||
|
||||
| Tool | Source | License | Purpose |
|
||||
|---|---|---|---|
|
||||
| hactool | https://github.com/SciresM/hactool @ `1d64a83450e025622f3468c28fc4164dad2c5ef6`, with patches from MacNativeGames | ISC | NCA and update container extraction |
|
||||
| hactool (the installer release ships it, see below) | https://github.com/SciresM/hactool @ `1d64a83450e025622f3468c28fc4164dad2c5ef6`, with patches from MacNativeGames | ISC | NCA and update container extraction |
|
||||
| Capstone | https://www.capstone-engine.org | BSD-3-Clause | ARM64 disassembly for analysis |
|
||||
| Unicorn | https://www.unicorn-engine.org | GPL-2.0 (Python bindings BSD) | Differential testing of the lifter; never linked into the game |
|
||||
|
||||
## Distributed in the installer release
|
||||
|
||||
`tools/make_release.py` assembles these into the release folder of `docs/installer.md`. Versions and sha256 of every download are pinned in that script; licence texts are in the folder's `licenses/` and in each component's own folder.
|
||||
|
||||
| Component | Source | License | Where in the release |
|
||||
|---|---|---|---|
|
||||
| CPython 3.14.9 embeddable package, with `tkinter` and Tcl/Tk 9.0.4 taken from the python.org 3.14.9 installer's component MSI (unpacked, never run); includes `vcruntime140*.dll` (Microsoft, redistributable with Python), OpenSSL, libffi, SQLite, zlib, libtommath | https://www.python.org/downloads/release/python-3149/ | PSF-2.0 (Tcl/Tk: Tcl/Tk licence, BSD-style) | `python/` |
|
||||
| numpy 2.5.4 (with its bundled OpenBLAS), lz4 4.4.5, cryptography 50.0.2 (with its bundled OpenSSL), cffi 2.1.1, pycparser 3.11 | PyPI binary wheels, pins in `tools/requirements-installer.txt` | BSD-3-Clause; lz4 BSD-3-Clause; cryptography Apache-2.0 OR BSD-3-Clause; cffi MIT; pycparser BSD-3-Clause | `python/Lib/site-packages/` |
|
||||
| llvm-mingw 20261020 (LLVM/Clang/LLD 23, mingw-w64, libc++, libunwind, compiler-rt), x86_64 subset | https://github.com/mstorsjo/llvm-mingw/releases/tag/20261020 | Apache-2.0 with LLVM exception; mingw-w64 ZPL-2.1/BSD/public domain; libc++ and libunwind Apache-2.0 with LLVM exception and MIT/UIUC; winpthreads MIT | `llvm/` |
|
||||
| SDL 3.4.18 upstream Windows release (`SDL3.dll`) and mingw development package (headers, `libSDL3.dll.a`) | https://github.com/libsdl-org/SDL/releases/tag/release-3.4.18 | zlib | `bin/SDL3.dll`, `runtime/deps/` |
|
||||
| libopus 1.6.1 (MSYS2 `mingw-w64-ucrt-x86_64-opus` 1.6.1-2; `libopus-0.dll` imports only KERNEL32 and the UCRT api-ms-win-crt DLLs), headers and import library | https://repo.msys2.org/mingw/ucrt64/ | BSD-3-Clause | `bin/libopus-0.dll`, `runtime/deps/` |
|
||||
| libvpx 1.17.0 (MSYS2 `mingw-w64-ucrt-x86_64-libvpx` 1.17.0-1), headers and static library compiled into `totk.exe`; `libvpx-LICENSE.txt` and `libvpx-PATENTS.txt` come from the v1.17.0 source tag | https://github.com/webmproject/libvpx | BSD-3-Clause with the WebM patent grant | `runtime/deps/`, `licenses/` |
|
||||
| Vulkan-Headers and Vulkan-Loader 1.4.363.0 (MSYS2 packages; headers and `libvulkan-1.dll.a` only, no loader DLL) | https://github.com/KhronosGroup | Apache-2.0 OR MIT | `runtime/deps/` |
|
||||
| Vulkan Memory Allocator 3.4.0 (MSYS2 package), header | https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator | MIT | `runtime/deps/` |
|
||||
| Ninja 1.13.2 (`ninja-win.zip`) | https://github.com/ninja-build/ninja/releases/tag/v1.13.2 | Apache-2.0 | `bin/ninja.exe` |
|
||||
| hactool (SciresM @ `1d64a83450e025622f3468c28fc4164dad2c5ef6` plus MacNativeGames' `portable-patches.diff`: `MNG_TITLE_KEYS` and a relaxed BKTR size check), built with MSYS2 UCRT64 clang; imports only KERNEL32 and the UCRT | https://github.com/SciresM/hactool | ISC | `bin/hactool.exe` |
|
||||
| `totk_xlate.exe` (Ryujinx shader translator, NativeAOT build of `tools/xlate/`, see the Ryujinx row above) | built by `tools/build_xlate.py` | MIT | `bin/totk_xlate.exe` |
|
||||
| MacNativeGames shader variant list | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/assets/shader_list.tsv`), Copyright (c) 2026 MacNativeGames contributors | MIT | `data/shader_list.tsv` |
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Installer
|
||||
|
||||
A public release that turns the player's own game files into a playable TotK folder. The release carries no game
|
||||
data and no code derived from the game: the installer builds the game on the player's machine. The installed folder
|
||||
holds only the game; the installer and its tools stay outside it.
|
||||
|
||||
## What the player does
|
||||
|
||||
1. Downloads `TotK-Setup-<version>-windows.zip` from GitHub Releases, unpacks it anywhere and starts `TotK-Setup.exe`.
|
||||
2. Picks:
|
||||
- the base game (XCI or NSP) and the 1.2.1 update (NSP);
|
||||
- their `prod.keys`;
|
||||
- the game language and the install folder.
|
||||
3. Waits for the setup, one time. The window shows each step and its progress.
|
||||
4. Plays `totk.exe` from the install folder. The installer and the source NSP or XCI can be deleted.
|
||||
|
||||
The install folder (`main.c` `portable_paths()` reads this layout; it stays portable):
|
||||
|
||||
```
|
||||
<install folder>/
|
||||
totk.exe, SDL3.dll, libopus-0.dll, totk_xlate.exe
|
||||
totk.ini language; the other keys as commented defaults (totk.ini.example)
|
||||
exefs/, romfs/ extracted from the player's files
|
||||
profile/ saves
|
||||
shader and pipeline caches prepared by the setup
|
||||
totk-install.json what built this folder: release version, input hashes, finished steps
|
||||
```
|
||||
|
||||
## Updating the port
|
||||
|
||||
The player runs a newer installer on the same folder. It reads `totk-install.json`, keeps `romfs/`, `exefs/`,
|
||||
`profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the shaders. A newer
|
||||
release that changes nothing game-derived skips the rebuild.
|
||||
|
||||
## Setup steps
|
||||
|
||||
1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 (by hash, as
|
||||
`tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a
|
||||
first install, 20 GiB after it), the CPU has
|
||||
AVX2 (`-march=x86-64-v3`), a Vulkan 1.3 GPU exists. Each failure is one clear message.
|
||||
2. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
|
||||
3. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt`) into a temp folder next to the installer.
|
||||
4. **Compile** the generated C, `modules.c` and `romfs.c` (they bake per-install layout and RomFS hashes) and link
|
||||
them with the prebuilt runtime library into `totk.exe` (`build_game.py --runtime-lib`, proven on task/llvmmingw:
|
||||
3 min on 24 cores).
|
||||
5. **Prepare** the RomFS patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`,
|
||||
`totk.exe --prepare-shaders`) in the install folder.
|
||||
6. **Clean up** the temp folder (generated C and objects, about 1.3 GB).
|
||||
|
||||
A cancelled or failed setup resumes from the last finished step (`totk-install.json`).
|
||||
|
||||
## Command line
|
||||
|
||||
`installer/` (stdlib only; `installer/layout.py` maps the release folder and, in a repository, the dev toolchain):
|
||||
|
||||
```
|
||||
python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR [--root DIR] [--temp DIR] [--verify]
|
||||
```
|
||||
|
||||
- On a folder with `totk-install.json` only `--dest` is needed; the game files only when the extraction must run again.
|
||||
- stdout: one JSON event per line, types `start`, `plan`, `step` (started, done, skipped), `progress` (step, phase,
|
||||
done/total/unit), `space` (bytes needed and free on the install disk and the temp disk), `warning`, `error`, `end` (`installer/events.py`). Messages are ids with parameters;
|
||||
`installer/text/en.json` holds their English text. The human-readable log goes to `<temp>/logs/`.
|
||||
- Exit codes: 0 done, 1 failed, 2 bad command line, 3 cancelled. The window cancels with Ctrl+Break to the
|
||||
installer's process group.
|
||||
- Each step records a key over its inputs (runner.py). A step with the recorded key and intact output is skipped, so
|
||||
resume and update are one rule; output kept from another release is re-hashed first.
|
||||
- Peak space on the install disk is about 36 GiB, not 22: the extraction copies both Program NCAs (16.3 GiB) next to
|
||||
the merged RomFS (15.9 GiB) before it deletes them. The check computes the exact figure.
|
||||
- Window (`python -m installer.gui`, `installer/gui.py` draws, `installer/guilogic.py` decides): form, step list with
|
||||
progress, space line, collapsible log, Cancel (Ctrl+Break to the CLI's process group, then it waits for the exit),
|
||||
finish screen with Open folder. English or Russian, chosen from the Windows display language, switchable in the
|
||||
window; every sentence is an id in `installer/text/<en|ru>.json`. A folder with `totk-install.json` needs only the
|
||||
folder. `--selftest [--shots DIR]` builds every screen in both languages and closes itself.
|
||||
- `tests/installer_gui_test.py`: event model, form, catalogs, cancel, the self-closing window and one end-to-end run
|
||||
of the CLI in `--verify` mode (needs `build/install-test` and `build/extracted/v121`; see its docstring).
|
||||
- `tests/installer_test.py`: bookkeeping with fake steps. `tests/installer_e2e.py`: a dev-mode run with the files of
|
||||
`config.local.toml`.
|
||||
|
||||
## Release layout
|
||||
|
||||
The unpacked release; `tools/make_release.py` assembles it, CI zips it:
|
||||
|
||||
```
|
||||
TotK-Setup.exe starts python/pythonw.exe -m installer.gui (installer/launcher/launcher.c, no console)
|
||||
installer/ the installer: steps, CLI, window, en/ru text catalog
|
||||
python/ Python 3.14.9 embeddable + numpy, lz4, cryptography, cffi, pycparser (+ tkinter for the window)
|
||||
recomp/, tools/ the lifter and the build, extraction and shader scripts the steps run (copied from the repo)
|
||||
llvm/ llvm-mingw, x86_64-only subset (Apache-2.0 with LLVM exception; mingw-w64 parts permissive)
|
||||
runtime/ libtotkrt.a and the headers the generated code and modules.c/romfs.c include
|
||||
bin/ totk_xlate.exe, hactool.exe, SDL3.dll, libopus-0.dll, ninja.exe
|
||||
data/ shader_list.tsv (MacNativeGames, MIT)
|
||||
THIRD_PARTY_NOTICES.md
|
||||
totk.ini.example, VERSION the default ini; the release string (git describe of the build)
|
||||
```
|
||||
|
||||
- SDL3 comes from the upstream release, not MSYS2: MSYS2's DLL imports the LGPL libiconv.
|
||||
- Every bundled third-party piece gets a row in `THIRD_PARTY_NOTICES.md`.
|
||||
- The release contains nothing game-derived; `tools/check_content.py` runs over it in CI.
|
||||
|
||||
## Constraints
|
||||
|
||||
- The installer steps contain no Win32 calls: Linux follows with its own compiler and the same steps.
|
||||
- The keys file path is passed to the tools; key values never appear in the UI, logs or files.
|
||||
- The window is a front end over the CLI (`python -m installer --base ... --update ... --keys ... --lang ... --dest
|
||||
...`), which prints one JSON progress event per line. Tests and CI use the CLI.
|
||||
- Unsigned exe: SmartScreen warns on first start.
|
||||
|
||||
## Later
|
||||
|
||||
- Re-warm the driver cache on start after a move, GPU or driver change (WIP on task/rewarm: stamp in
|
||||
`shader-prepared.bin`).
|
||||
- NSZ and XCZ input. Linux release.
|
||||
@@ -0,0 +1,10 @@
|
||||
"""The TotK installer: builds a playable folder from the player's own game files (docs/installer.md).
|
||||
|
||||
python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR [--root DIR] [--temp DIR]
|
||||
|
||||
Modules: layout (where the bundled tools are, release or repository), events (JSON progress on stdout and the log
|
||||
file), state (totk-install.json), runner (which steps run: resume and update), steps (the six steps), ini (totk.ini),
|
||||
cli (the command line). text/en.json holds the English text of every message id the events carry.
|
||||
|
||||
Standard library only, apart from what the extraction tools import (cryptography). Runs on Python 3.14.
|
||||
"""
|
||||
@@ -0,0 +1,5 @@
|
||||
import sys
|
||||
|
||||
from installer.cli import main
|
||||
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Command line: python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR [--root DIR]
|
||||
[--temp DIR] [--verify]
|
||||
|
||||
A first install needs every option. On a folder with totk-install.json (resume after a cancel or failure, or update
|
||||
with a newer release) only --dest is needed: --lang defaults to the recorded language, and the game files are needed
|
||||
only when the extraction has to run again. Prints JSON events on stdout (events.py). Exit code: 0 done, 1 failed,
|
||||
2 bad command line, 3 cancelled (Ctrl+C or Ctrl+Break).
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from . import layout as layouts
|
||||
from . import state as statefile
|
||||
from .events import Reporter
|
||||
from .runner import Context, run_steps
|
||||
from .steps import all_steps
|
||||
|
||||
EXIT = {'ok': 0, 'failed': 1, 'cancelled': 3}
|
||||
|
||||
|
||||
class Parser(argparse.ArgumentParser):
|
||||
def error(self, message):
|
||||
Reporter().error('cli.usage', {'detail': message})
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def temp_folder(base, dest):
|
||||
"""One temp folder per install folder, so two installs never share generated files."""
|
||||
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
|
||||
return base / f'{dest.name or "root"}-{tag}'
|
||||
|
||||
|
||||
def cancel(signum, frame):
|
||||
raise KeyboardInterrupt
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = Parser(prog='python -m installer', description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--base', type=Path, help='base game, XCI or NSP')
|
||||
ap.add_argument('--update', type=Path, help='1.2.1 update NSP')
|
||||
ap.add_argument('--keys', type=Path, help='prod.keys')
|
||||
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
|
||||
ap.add_argument('--dest', type=Path, required=True, help='install folder')
|
||||
ap.add_argument('--root', type=Path, help='release folder or repository (default: the folder above installer/)')
|
||||
ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release, build/install-temp in a '
|
||||
'repository)')
|
||||
ap.add_argument('--verify', action='store_true', help='hash every kept file, also those this release made')
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
# The window cancels with Ctrl+Break to the installer's process group (CREATE_NEW_PROCESS_GROUP), which also reaches
|
||||
# the tools it runs; Python's default for it ends the process without recording anything.
|
||||
for name in ('SIGBREAK', 'SIGTERM'):
|
||||
if hasattr(signal, name):
|
||||
signal.signal(getattr(signal, name), cancel)
|
||||
layout = layouts.find(args.root)
|
||||
dest = args.dest.resolve()
|
||||
temp = temp_folder((args.temp or layout.temp).resolve(), dest)
|
||||
log = temp / 'logs' / time.strftime('install-%Y%m%d-%H%M%S.log')
|
||||
rep = Reporter(log_path=log)
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
state = statefile.load(dest)
|
||||
except statefile.StateError as e:
|
||||
rep.error(e.id, {'path': str(e.path)})
|
||||
rep.emit('end', result='failed', seconds=0)
|
||||
return 1
|
||||
ctx = Context(layout, dest, temp, rep, args, state=state, verify=args.verify)
|
||||
steps = all_steps()
|
||||
rep.emit('start', release=layout.release, mode=ctx.mode, dev=layout.dev, dest=str(dest), log=str(log),
|
||||
steps=[s.id for s in steps])
|
||||
result = run_steps(steps, ctx)
|
||||
rep.emit('end', result=result, seconds=round(time.monotonic() - t0, 1))
|
||||
rep.close()
|
||||
return EXIT[result]
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Progress events: one JSON object per line on stdout for the window and tests, the same in English to a log file.
|
||||
|
||||
Every event has "type" and "t" (seconds since the start). Types:
|
||||
|
||||
start release, mode (install, resume, update), dest, log, steps (the step ids in order)
|
||||
step step, state: started | done (seconds) | skipped (reason: current, not_needed)
|
||||
progress step, phase, and done/total/unit (bytes, chunks, objects, shaders) when the amount is known
|
||||
space dest, need, free (bytes on the install disk; with the temp folder on the same disk need includes it),
|
||||
temp, temp_need, temp_free, same_disk
|
||||
warning id, params
|
||||
error id, params, step (absent outside a step)
|
||||
end result: ok | failed | cancelled, seconds
|
||||
|
||||
Text is never the payload: ids name a step ("step.<id>"), a phase ("phase.<step>.<phase>") or a message ("<id>"),
|
||||
and installer/text/en.json holds their English text with {param} placeholders. The log renders events with it.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
CATALOG = Path(__file__).resolve().parent / 'text' / 'en.json'
|
||||
PROGRESS_EVERY = 0.5 # seconds between two progress events of one phase
|
||||
|
||||
|
||||
def load_catalog(path=CATALOG):
|
||||
return json.loads(path.read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def render(catalog, id, params=None):
|
||||
"""The English text of id, or the id itself when the catalog lacks it."""
|
||||
text = catalog.get(id, id)
|
||||
try:
|
||||
return text.format(**(params or {}))
|
||||
except (KeyError, IndexError, ValueError):
|
||||
return f'{text} {params}'
|
||||
|
||||
|
||||
class Reporter:
|
||||
def __init__(self, out=None, log_path=None):
|
||||
self.out = out or sys.stdout
|
||||
self.t0 = time.monotonic()
|
||||
self.catalog = load_catalog()
|
||||
self.lock = threading.Lock()
|
||||
self.log_file = None
|
||||
self.last_progress = {}
|
||||
if log_path:
|
||||
Path(log_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
self.log_file = open(log_path, 'a', encoding='utf-8', buffering=1)
|
||||
self.log_path = log_path
|
||||
|
||||
def elapsed(self):
|
||||
return time.monotonic() - self.t0
|
||||
|
||||
def log(self, text):
|
||||
"""A line for the log file only."""
|
||||
if self.log_file:
|
||||
with self.lock:
|
||||
self.log_file.write(f'[{self.elapsed():8.1f}] {text}\n')
|
||||
|
||||
def emit(self, type, **fields):
|
||||
event = {'type': type, 't': round(self.elapsed(), 2), **fields}
|
||||
with self.lock:
|
||||
self.out.write(json.dumps(event, ensure_ascii=False) + '\n')
|
||||
self.out.flush()
|
||||
self.log(self.describe(event))
|
||||
return event
|
||||
|
||||
def describe(self, e):
|
||||
c = self.catalog
|
||||
kind = e['type']
|
||||
if kind == 'step':
|
||||
name = render(c, 'step.' + e['step'])
|
||||
extra = f' ({e["seconds"]:.1f} s)' if 'seconds' in e else f' ({e["reason"]})' if 'reason' in e else ''
|
||||
return f'{name}: {e["state"]}{extra}'
|
||||
if kind == 'progress':
|
||||
text = render(c, f'phase.{e["step"]}.{e["phase"]}')
|
||||
if e.get('total'):
|
||||
text += f' {e["done"]}/{e["total"]} {e.get("unit", "")}'.rstrip()
|
||||
return text
|
||||
if kind in ('warning', 'error'):
|
||||
return f'{kind}: {render(c, e["id"], e.get("params"))} [{e["id"]}]'
|
||||
return json.dumps(e, ensure_ascii=False)
|
||||
|
||||
def step(self, step, state, **fields):
|
||||
return self.emit('step', step=step, state=state, **fields)
|
||||
|
||||
def progress(self, step, phase, done=None, total=None, unit=None, force=False):
|
||||
"""A progress event, at most one per PROGRESS_EVERY per phase unless it is the first or the last."""
|
||||
now = time.monotonic()
|
||||
key = (step, phase)
|
||||
last = self.last_progress.get(key)
|
||||
final = total is not None and done is not None and done >= total
|
||||
if last is not None and not final and not force and now - last < PROGRESS_EVERY:
|
||||
return None
|
||||
self.last_progress[key] = now
|
||||
fields = {'step': step, 'phase': phase}
|
||||
if done is not None:
|
||||
fields['done'] = done
|
||||
if total is not None:
|
||||
fields['total'] = total
|
||||
if unit:
|
||||
fields['unit'] = unit
|
||||
return self.emit('progress', **fields)
|
||||
|
||||
def warning(self, id, **params):
|
||||
return self.emit('warning', id=id, params=params)
|
||||
|
||||
def error(self, id, params, step=None):
|
||||
fields = {'id': id, 'params': params}
|
||||
if step:
|
||||
fields['step'] = step
|
||||
return self.emit('error', **fields)
|
||||
|
||||
def close(self):
|
||||
if self.log_file:
|
||||
self.log_file.close()
|
||||
self.log_file = None
|
||||
@@ -0,0 +1,587 @@
|
||||
"""The installer window: python -m installer.gui
|
||||
|
||||
A front end over the CLI (docs/installer.md): it collects the game files, the keys file path, the game language and the
|
||||
install folder, runs `python -m installer ...` as a subprocess and shows its JSON events. Decisions live in guilogic.py;
|
||||
this file only draws. The window text is in installer/text/<en|ru>.json under "ui.*" and "launcher.*" ids.
|
||||
|
||||
--selftest [--shots DIR] build every screen in both languages and close by itself (PNG per screen with --shots)
|
||||
--lang en|ru window language (default: the Windows display language)
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tkinter as tk
|
||||
from pathlib import Path
|
||||
from tkinter import filedialog, messagebox, ttk
|
||||
|
||||
try:
|
||||
from PIL import ImageGrab # only the screenshots of --selftest --shots need it; the release has no PIL
|
||||
except ImportError:
|
||||
ImageGrab = None
|
||||
|
||||
from . import ini
|
||||
from . import layout as layouts
|
||||
from .guilogic import (DONE, FAILED, PENDING, RUNNING, SKIPPED, UI_LANGUAGES, Catalog, Form, Model, Process,
|
||||
amount, default_game_language, describe_event, parse_line, windows_ui_language)
|
||||
|
||||
POLL_MS = 100
|
||||
GLYPH = {PENDING: '\u25cb', RUNNING: '\u25b6', DONE: '\u2714', SKIPPED: '\u2013', FAILED: '\u2716'}
|
||||
COLOR = {PENDING: '#808080', RUNNING: '#1a5fb4', DONE: '#26a269', SKIPPED: '#808080', FAILED: '#c01c28'}
|
||||
PAD = 14
|
||||
|
||||
|
||||
def fix_tcl_paths():
|
||||
"""The embeddable Python keeps Tcl/Tk next to python.exe (python/tcl); point Tcl at it when it is not found."""
|
||||
tcl = Path(sys.executable).resolve().parent / 'tcl'
|
||||
if tcl.is_dir() and 'TCL_LIBRARY' not in os.environ:
|
||||
for lib, var in (('tcl9.0', 'TCL_LIBRARY'), ('tk9.0', 'TK_LIBRARY')):
|
||||
if (tcl / lib).is_dir():
|
||||
os.environ[var] = str(tcl / lib)
|
||||
|
||||
|
||||
class App:
|
||||
def __init__(self, root, ui_lang, layout, run_args=None):
|
||||
self.root, self.layout = root, layout
|
||||
self.cat = Catalog(ui_lang)
|
||||
self.model = Model()
|
||||
self.process = None
|
||||
self.closing = False
|
||||
self.run_args = run_args or {} # extra CLI options for tests: temp, verify
|
||||
self.log_entries = [] # events and raw lines of the run, rendered by render_log
|
||||
self.bound = [] # (widget, option, id, params) retranslated on a language switch
|
||||
self.vars = {n: tk.StringVar() for n in ('base', 'update', 'keys', 'dest')}
|
||||
self.vars['lang'] = tk.StringVar()
|
||||
self.lang_code = default_game_language(ui_lang)
|
||||
self.screen = None
|
||||
|
||||
root.title(self.cat('ui.title'))
|
||||
root.minsize(680, 560)
|
||||
root.protocol('WM_DELETE_WINDOW', self.on_close)
|
||||
style = ttk.Style(root)
|
||||
style.configure('Title.TLabel', font=('Segoe UI', 15, 'bold'))
|
||||
style.configure('Notice.TLabel', background='#fff3cd', foreground='#664d03', padding=8)
|
||||
style.configure('Err.TLabel', foreground='#c01c28')
|
||||
|
||||
top = ttk.Frame(root, padding=(PAD, PAD, PAD, 0))
|
||||
top.pack(fill='x')
|
||||
self.title = self.label(top, 'ui.title', style='Title.TLabel')
|
||||
self.title.pack(side='left')
|
||||
sw = ttk.Frame(top)
|
||||
sw.pack(side='right')
|
||||
self.ui_var = tk.StringVar(value=self.cat.lang)
|
||||
for code in UI_LANGUAGES:
|
||||
ttk.Radiobutton(sw, text=code.upper(), value=code, variable=self.ui_var, style='Toolbutton',
|
||||
command=self.switch_language).pack(side='left', padx=1)
|
||||
self.body = ttk.Frame(root, padding=PAD)
|
||||
self.body.pack(fill='both', expand=True)
|
||||
self.frames = {'form': self.build_form(), 'run': self.build_run(), 'finish': self.build_finish()}
|
||||
self.vars['lang'].set(self.lang_label(self.lang_code))
|
||||
self.vars['dest'].trace_add('write', lambda *_: self.refresh_form())
|
||||
self.show('form')
|
||||
|
||||
# ---- text -----------------------------------------------------------------------------------------------------
|
||||
|
||||
def bind_text(self, widget, option, id, **params):
|
||||
self.bound.append((widget, option, id, params))
|
||||
widget.configure(**{option: self.cat(id, **params)})
|
||||
|
||||
def label(self, parent, id, **kw):
|
||||
w = ttk.Label(parent, **kw)
|
||||
self.bind_text(w, 'text', id)
|
||||
return w
|
||||
|
||||
def button(self, parent, id, command, **kw):
|
||||
w = ttk.Button(parent, command=command, **kw)
|
||||
self.bind_text(w, 'text', id)
|
||||
return w
|
||||
|
||||
def retext(self):
|
||||
self.root.title(self.cat('ui.title'))
|
||||
for widget, option, id, params in self.bound:
|
||||
widget.configure(**{option: self.cat(id, **params)})
|
||||
self.fill_languages()
|
||||
self.refresh_form()
|
||||
self.refresh_run()
|
||||
self.refresh_finish()
|
||||
self.render_log()
|
||||
|
||||
def switch_language(self):
|
||||
self.cat.set_language(self.ui_var.get())
|
||||
code = self.selected_lang()
|
||||
self.retext()
|
||||
self.vars['lang'].set(self.lang_label(code))
|
||||
|
||||
def lang_label(self, code):
|
||||
return self.cat('lang.' + code)
|
||||
|
||||
def fill_languages(self):
|
||||
code = self.selected_lang()
|
||||
self.lang_box.configure(values=[self.lang_label(c) for c in ini.LANGUAGES])
|
||||
self.vars['lang'].set(self.lang_label(code))
|
||||
|
||||
def selected_lang(self):
|
||||
"""The game language code behind the combobox text (kept when it is not one of the labels)."""
|
||||
shown = self.vars['lang'].get()
|
||||
for code in ini.LANGUAGES:
|
||||
for lang in UI_LANGUAGES:
|
||||
if shown == Catalog(lang)('lang.' + code):
|
||||
return code
|
||||
return self.lang_code
|
||||
|
||||
# ---- screens --------------------------------------------------------------------------------------------------
|
||||
|
||||
def show(self, name):
|
||||
for f in self.frames.values():
|
||||
f.pack_forget()
|
||||
self.frames[name].pack(fill='both', expand=True)
|
||||
self.screen = name
|
||||
|
||||
def notice(self, parent):
|
||||
w = ttk.Label(parent, style='Notice.TLabel', wraplength=620, justify='left')
|
||||
self.bind_text(w, 'text', 'ui.notice')
|
||||
return w
|
||||
|
||||
def build_form(self):
|
||||
f = ttk.Frame(self.body)
|
||||
self.form_intro = self.label(f, 'ui.intro', wraplength=620, justify='left')
|
||||
self.form_intro.pack(anchor='w', pady=(0, 8))
|
||||
self.notice(f).pack(fill='x', pady=(0, 10))
|
||||
grid = ttk.Frame(f)
|
||||
grid.pack(fill='x')
|
||||
grid.columnconfigure(1, weight=1)
|
||||
rows = (('base', 'ui.base', 'ui.pick_game'), ('update', 'ui.update', 'ui.pick_update'),
|
||||
('keys', 'ui.keys', 'ui.pick_keys'))
|
||||
self.form_rows = {}
|
||||
for i, (name, label_id, pick_id) in enumerate(rows):
|
||||
lab = self.label(grid, label_id)
|
||||
lab.grid(row=i, column=0, sticky='w', pady=4, padx=(0, 10))
|
||||
ttk.Entry(grid, textvariable=self.vars[name]).grid(row=i, column=1, sticky='ew', pady=4)
|
||||
b = self.button(grid, 'ui.browse', lambda n=name, p=pick_id: self.pick_file(n, p))
|
||||
b.grid(row=i, column=2, padx=(8, 0))
|
||||
self.form_rows[name] = lab
|
||||
i = len(rows)
|
||||
self.label(grid, 'ui.game_language').grid(row=i, column=0, sticky='w', pady=4)
|
||||
self.lang_box = ttk.Combobox(grid, textvariable=self.vars['lang'], state='readonly', width=30)
|
||||
self.lang_box.grid(row=i, column=1, sticky='w', pady=4)
|
||||
self.lang_touched = False
|
||||
self.lang_box.bind('<<ComboboxSelected>>', lambda _e: setattr(self, 'lang_touched', True))
|
||||
self.fill_languages()
|
||||
self.label(grid, 'ui.dest').grid(row=i + 1, column=0, sticky='w', pady=4)
|
||||
ttk.Entry(grid, textvariable=self.vars['dest']).grid(row=i + 1, column=1, sticky='ew', pady=4)
|
||||
self.button(grid, 'ui.browse', self.pick_dest).grid(row=i + 1, column=2, padx=(8, 0))
|
||||
self.update_hint = ttk.Label(f, wraplength=620, justify='left')
|
||||
self.update_hint.pack(anchor='w', pady=(8, 0))
|
||||
self.form_error = ttk.Label(f, style='Err.TLabel', wraplength=620, justify='left')
|
||||
self.form_error.pack(anchor='w', pady=(8, 0))
|
||||
self.label(f, 'ui.keys_private', wraplength=620, justify='left').pack(anchor='w', pady=(8, 0))
|
||||
bar = ttk.Frame(f)
|
||||
bar.pack(side='bottom', fill='x', pady=(12, 0))
|
||||
self.start_button = self.button(bar, 'ui.start', self.start)
|
||||
self.start_button.pack(side='right')
|
||||
return f
|
||||
|
||||
def build_run(self):
|
||||
f = ttk.Frame(self.body)
|
||||
self.run_header = ttk.Label(f, font=('Segoe UI', 11, 'bold'))
|
||||
self.run_header.pack(anchor='w')
|
||||
self.step_rows = {}
|
||||
self.steps_box = ttk.Frame(f)
|
||||
self.steps_box.pack(fill='x', pady=(8, 8))
|
||||
self.phase_label = ttk.Label(f, wraplength=620, justify='left')
|
||||
self.phase_label.pack(anchor='w')
|
||||
self.bar = ttk.Progressbar(f, mode='determinate', maximum=1000)
|
||||
self.bar.pack(fill='x', pady=(4, 6))
|
||||
self.space_label = ttk.Label(f, wraplength=620, justify='left')
|
||||
self.space_label.pack(anchor='w')
|
||||
self.warn_label = ttk.Label(f, wraplength=620, justify='left', foreground='#9a6700')
|
||||
self.warn_label.pack(anchor='w')
|
||||
bar = ttk.Frame(f)
|
||||
bar.pack(side='bottom', fill='x', pady=(8, 0))
|
||||
self.cancel_button = self.button(bar, 'ui.cancel', self.cancel)
|
||||
self.cancel_button.pack(side='right')
|
||||
self.log_toggle = self.button(bar, 'ui.show_log', self.toggle_log)
|
||||
self.log_toggle.pack(side='left')
|
||||
self.log_frame = ttk.Frame(f)
|
||||
self.log = tk.Text(self.log_frame, height=10, wrap='word', state='disabled', font=('Consolas', 9))
|
||||
scroll = ttk.Scrollbar(self.log_frame, command=self.log.yview)
|
||||
self.log.configure(yscrollcommand=scroll.set)
|
||||
scroll.pack(side='right', fill='y')
|
||||
self.log.pack(side='left', fill='both', expand=True)
|
||||
self.log_open = False
|
||||
return f
|
||||
|
||||
def build_finish(self):
|
||||
f = ttk.Frame(self.body)
|
||||
self.finish_title = ttk.Label(f, style='Title.TLabel', wraplength=620, justify='left')
|
||||
self.finish_title.pack(anchor='w')
|
||||
self.finish_text = ttk.Label(f, wraplength=620, justify='left')
|
||||
self.finish_text.pack(anchor='w', pady=(8, 8))
|
||||
self.finish_steps = ttk.Frame(f)
|
||||
self.finish_steps.pack(fill='x', pady=(0, 8))
|
||||
self.finish_notice = self.notice(f)
|
||||
self.finish_notice.pack(fill='x', pady=(8, 0))
|
||||
bar = ttk.Frame(f)
|
||||
bar.pack(side='bottom', fill='x', pady=(12, 0))
|
||||
self.button(bar, 'ui.close', self.on_close).pack(side='right')
|
||||
self.open_button = self.button(bar, 'ui.open_folder', self.open_folder)
|
||||
self.open_button.pack(side='right', padx=(0, 8))
|
||||
self.back_button = self.button(bar, 'ui.back', lambda: self.show('form'))
|
||||
self.back_button.pack(side='left')
|
||||
return f
|
||||
|
||||
# ---- form -----------------------------------------------------------------------------------------------------
|
||||
|
||||
def form(self):
|
||||
return Form(base=self.vars['base'].get(), update=self.vars['update'].get(), keys=self.vars['keys'].get(),
|
||||
lang=self.selected_lang(), dest=self.vars['dest'].get())
|
||||
|
||||
def refresh_form(self):
|
||||
form = self.form()
|
||||
state = form.installed_state()
|
||||
if state is not None:
|
||||
self.update_hint.configure(text=self.cat('ui.update_mode'))
|
||||
if state.get('lang') in ini.LANGUAGES and not self.lang_touched:
|
||||
self.vars['lang'].set(self.lang_label(state['lang']))
|
||||
else:
|
||||
self.update_hint.configure(text='')
|
||||
|
||||
def pick_file(self, name, pick_id):
|
||||
types = [(self.cat('ui.type_game'), '*.nsp *.xci'), (self.cat('ui.type_all'), '*.*')]
|
||||
if name == 'update':
|
||||
types = [(self.cat('ui.type_nsp'), '*.nsp'), (self.cat('ui.type_all'), '*.*')]
|
||||
if name == 'keys':
|
||||
types = [(self.cat('ui.type_keys'), '*.keys'), (self.cat('ui.type_all'), '*.*')]
|
||||
path = filedialog.askopenfilename(title=self.cat(pick_id), filetypes=types)
|
||||
if path:
|
||||
self.vars[name].set(os.path.normpath(path))
|
||||
|
||||
def pick_dest(self):
|
||||
path = filedialog.askdirectory(title=self.cat('ui.pick_dest'), mustexist=False)
|
||||
if path:
|
||||
self.vars['dest'].set(os.path.normpath(path))
|
||||
|
||||
# ---- run ------------------------------------------------------------------------------------------------------
|
||||
|
||||
def start(self):
|
||||
form = self.form()
|
||||
problems = form.problems()
|
||||
if problems:
|
||||
self.form_error.configure(text=' '.join(self.cat(p) for p in problems))
|
||||
return
|
||||
self.form_error.configure(text='')
|
||||
command = form.command(self.layout.python, root=self.layout.root if self.layout.dev else None,
|
||||
temp=self.run_args.get('temp'), verify=self.run_args.get('verify', False))
|
||||
self.begin(command, self.layout.root)
|
||||
|
||||
def begin(self, command, cwd):
|
||||
self.model = Model()
|
||||
self.model.dest = self.vars['dest'].get().strip()
|
||||
self.clear_log()
|
||||
self.build_step_rows(self.model.steps)
|
||||
self.cancel_button.configure(state='normal')
|
||||
self.bound = [b for b in self.bound if b[0] is not self.cancel_button]
|
||||
self.bind_text(self.cancel_button, 'text', 'ui.cancel')
|
||||
self.show('run')
|
||||
self.refresh_run()
|
||||
self.process = Process(command, cwd=cwd, helper_python=self.layout.python)
|
||||
self.root.after(POLL_MS, self.poll)
|
||||
|
||||
def build_step_rows(self, steps):
|
||||
for w in self.steps_box.winfo_children():
|
||||
w.destroy()
|
||||
self.step_rows = {}
|
||||
for s in steps:
|
||||
row = ttk.Frame(self.steps_box)
|
||||
row.pack(fill='x', pady=1)
|
||||
glyph = ttk.Label(row, width=3)
|
||||
glyph.pack(side='left')
|
||||
name = ttk.Label(row)
|
||||
name.pack(side='left')
|
||||
self.step_rows[s] = (glyph, name)
|
||||
|
||||
def poll(self):
|
||||
if self.process is None:
|
||||
return
|
||||
try:
|
||||
while True:
|
||||
line = self.process.lines.get_nowait()
|
||||
if line is not None:
|
||||
self.feed(line)
|
||||
except Exception:
|
||||
pass
|
||||
code = self.process.poll()
|
||||
self.refresh_run()
|
||||
if code is None:
|
||||
self.root.after(POLL_MS, self.poll)
|
||||
return
|
||||
self.model.finish(code)
|
||||
self.process = None
|
||||
self.refresh_run()
|
||||
self.refresh_finish()
|
||||
self.show('finish')
|
||||
if self.closing:
|
||||
self.root.destroy()
|
||||
|
||||
def feed(self, line):
|
||||
"""One stdout line of the CLI: an event updates the model and the log, anything else goes to the log as is."""
|
||||
event = parse_line(line)
|
||||
if event is None:
|
||||
if line.strip():
|
||||
self.log_entries.append(line.rstrip())
|
||||
self.append_log(line.rstrip())
|
||||
return
|
||||
before = list(self.model.steps)
|
||||
self.model.apply(event)
|
||||
if self.model.steps != before:
|
||||
self.build_step_rows(self.model.steps)
|
||||
self.log_entries.append(event)
|
||||
self.append_log(describe_event(self.cat, event))
|
||||
|
||||
def refresh_run(self):
|
||||
m = self.model
|
||||
for s, (glyph, name) in self.step_rows.items():
|
||||
state = m.states.get(s, PENDING)
|
||||
glyph.configure(text=GLYPH[state], foreground=COLOR[state])
|
||||
name.configure(text=self.cat('step.' + s) + (f' ({self.cat("ui.state_" + state)})'
|
||||
if state in (SKIPPED, FAILED) else ''),
|
||||
font=('Segoe UI', 10, 'bold') if state == RUNNING else ('Segoe UI', 10))
|
||||
mode = m.mode or 'install'
|
||||
self.run_header.configure(text=self.cat('ui.running_' + mode if mode in ('install', 'resume', 'update')
|
||||
else 'ui.running_install'))
|
||||
if m.current and m.phase:
|
||||
text = self.cat(f'phase.{m.current}.{m.phase}')
|
||||
if m.total:
|
||||
text += ' ' + amount(self.cat, m.done, m.total, m.unit)
|
||||
elif m.current:
|
||||
text = self.cat('step.' + m.current)
|
||||
else:
|
||||
text = ''
|
||||
self.phase_label.configure(text=text)
|
||||
frac = m.fraction()
|
||||
if frac is None:
|
||||
if str(self.bar.cget('mode')) != 'indeterminate':
|
||||
self.bar.configure(mode='indeterminate')
|
||||
self.bar.start(15)
|
||||
else:
|
||||
if str(self.bar.cget('mode')) == 'indeterminate':
|
||||
self.bar.stop()
|
||||
self.bar.configure(mode='determinate')
|
||||
self.bar.configure(value=frac * 1000)
|
||||
self.space_label.configure(text=m.need_text(self.cat))
|
||||
self.warn_label.configure(text='\n'.join(self.cat(i, **p) for i, p in m.warnings))
|
||||
|
||||
def toggle_log(self):
|
||||
self.log_open = not self.log_open
|
||||
if self.log_open:
|
||||
self.log_frame.pack(fill='both', expand=True, pady=(8, 0))
|
||||
else:
|
||||
self.log_frame.pack_forget()
|
||||
self.bound = [b for b in self.bound if b[0] is not self.log_toggle]
|
||||
self.bind_text(self.log_toggle, 'text', 'ui.hide_log' if self.log_open else 'ui.show_log')
|
||||
|
||||
def clear_log(self):
|
||||
self.log_entries = []
|
||||
self.render_log()
|
||||
|
||||
def render_log(self):
|
||||
"""The whole log from the kept events in the current language (raw CLI lines stay as they are)."""
|
||||
self.log.configure(state='normal')
|
||||
self.log.delete('1.0', 'end')
|
||||
for entry in self.log_entries:
|
||||
text = entry if isinstance(entry, str) else describe_event(self.cat, entry)
|
||||
if text:
|
||||
self.log.insert('end', text + '\n')
|
||||
self.log.see('end')
|
||||
self.log.configure(state='disabled')
|
||||
|
||||
def append_log(self, text):
|
||||
if text:
|
||||
self.log.configure(state='normal')
|
||||
self.log.insert('end', text + '\n')
|
||||
self.log.see('end')
|
||||
self.log.configure(state='disabled')
|
||||
|
||||
def cancel(self):
|
||||
if self.process:
|
||||
self.cancel_button.configure(state='disabled')
|
||||
self.bound = [b for b in self.bound if b[0] is not self.cancel_button]
|
||||
self.bind_text(self.cancel_button, 'text', 'ui.cancelling')
|
||||
self.process.cancel()
|
||||
|
||||
def on_close(self):
|
||||
if self.process:
|
||||
if not messagebox.askyesno(self.cat('ui.title'), self.cat('ui.confirm_cancel')):
|
||||
return
|
||||
self.closing = True
|
||||
self.cancel()
|
||||
return
|
||||
self.root.destroy()
|
||||
|
||||
# ---- finish ---------------------------------------------------------------------------------------------------
|
||||
|
||||
def refresh_finish(self):
|
||||
m = self.model
|
||||
result = m.result or 'ok'
|
||||
self.finish_title.configure(text=self.cat('ui.result_' + result))
|
||||
if result == 'ok':
|
||||
text = self.cat('ui.done_text', dest=m.dest)
|
||||
elif result == 'cancelled':
|
||||
text = self.cat('ui.cancelled_text')
|
||||
else:
|
||||
err = m.error or ('ui.process_died', {'code': m.exit_code}, None)
|
||||
text = self.cat(err[0], **err[1])
|
||||
if m.log_path:
|
||||
text += '\n\n' + self.cat('ui.log_file', path=m.log_path)
|
||||
self.finish_text.configure(text=text)
|
||||
for w in self.finish_steps.winfo_children():
|
||||
w.destroy()
|
||||
for s in m.steps:
|
||||
state = m.states.get(s, PENDING)
|
||||
ttk.Label(self.finish_steps, text=f'{GLYPH[state]} {self.cat("step." + s)}',
|
||||
foreground=COLOR[state]).pack(anchor='w')
|
||||
if result == 'ok':
|
||||
self.open_button.state(['!disabled'])
|
||||
else:
|
||||
self.open_button.state(['disabled' if not Path(m.dest or '.').is_dir() else '!disabled'])
|
||||
|
||||
def open_folder(self):
|
||||
dest = self.model.dest
|
||||
if dest and Path(dest).is_dir():
|
||||
os.startfile(dest)
|
||||
|
||||
|
||||
# ---- self test ---------------------------------------------------------------------------------------------------
|
||||
|
||||
SAMPLE_START = {'type': 'start', 'release': 'selftest', 'mode': 'install', 'dest': 'Z:/selftest/TotK', 'log': 'D:\\log.txt',
|
||||
'steps': ['check', 'extract', 'lift', 'compile', 'prepare', 'cleanup']}
|
||||
SAMPLE_RUN = [
|
||||
SAMPLE_START,
|
||||
{'type': 'plan', 'run': ['check', 'extract'], 'skip': {}},
|
||||
{'type': 'step', 'step': 'check', 'state': 'started'},
|
||||
{'type': 'progress', 'step': 'check', 'phase': 'inputs'},
|
||||
{'type': 'space', 'dest': 'Z:/selftest/TotK', 'need': 36_400_000_000, 'free': 210_000_000_000, 'temp': 'C:\\t',
|
||||
'temp_need': 0, 'temp_free': 0, 'same_disk': True},
|
||||
{'type': 'warning', 'id': 'warn.lang_needs_fonts', 'params': {'lang': 'ko'}},
|
||||
{'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 12.0},
|
||||
{'type': 'step', 'step': 'extract', 'state': 'started'},
|
||||
{'type': 'progress', 'step': 'extract', 'phase': 'copy', 'done': 6_000_000_000, 'total': 16_000_000_000,
|
||||
'unit': 'bytes'},
|
||||
]
|
||||
|
||||
|
||||
SAMPLE_OK = [SAMPLE_START, {'type': 'plan', 'run': ['check', 'extract'], 'skip': {'lift': 'current', 'compile': 'current',
|
||||
'prepare': 'current', 'cleanup': 'not_needed'}},
|
||||
{'type': 'step', 'step': 'check', 'state': 'started'},
|
||||
{'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 1.0},
|
||||
{'type': 'step', 'step': 'extract', 'state': 'started'},
|
||||
{'type': 'step', 'step': 'extract', 'state': 'done', 'seconds': 1.0},
|
||||
{'type': 'step', 'step': 'lift', 'state': 'skipped', 'reason': 'current'},
|
||||
{'type': 'step', 'step': 'compile', 'state': 'skipped', 'reason': 'current'},
|
||||
{'type': 'step', 'step': 'prepare', 'state': 'skipped', 'reason': 'current'},
|
||||
{'type': 'step', 'step': 'cleanup', 'state': 'done', 'seconds': 0.1},
|
||||
{'type': 'end', 'result': 'ok', 'seconds': 3}]
|
||||
|
||||
|
||||
def grab(root, path):
|
||||
if ImageGrab is None:
|
||||
raise SystemExit('--shots needs Pillow')
|
||||
root.attributes('-topmost', True)
|
||||
root.update()
|
||||
x, y, w, h = root.winfo_rootx(), root.winfo_rooty(), root.winfo_width(), root.winfo_height()
|
||||
ImageGrab.grab(bbox=(x, y, x + w, y + h), all_screens=True).save(path)
|
||||
|
||||
|
||||
def selftest(layout, shots=None):
|
||||
"""Every screen in both languages; the window closes itself. Returns 0 when nothing raised."""
|
||||
fix_tcl_paths()
|
||||
root = tk.Tk()
|
||||
root.geometry('720x640+60+40')
|
||||
app = App(root, 'en', layout)
|
||||
steps, mixed = [], []
|
||||
|
||||
def shot(name):
|
||||
root.update()
|
||||
if shots:
|
||||
Path(shots).mkdir(parents=True, exist_ok=True)
|
||||
grab(root, Path(shots) / f'{name}.png')
|
||||
|
||||
def run_all():
|
||||
for lang in UI_LANGUAGES:
|
||||
app.ui_var.set(lang)
|
||||
app.switch_language()
|
||||
tag = lang
|
||||
app.vars['dest'].set('Z:/selftest/TotK')
|
||||
app.vars['base'].set('Z:/selftest/totk.xci')
|
||||
app.vars['update'].set('Z:/selftest/totk-1.2.1.nsp')
|
||||
app.vars['keys'].set('Z:/selftest/prod.keys')
|
||||
app.show('form')
|
||||
app.start() # missing files: shows the problems
|
||||
steps.append(app.form_error.cget('text'))
|
||||
shot(f'form-{tag}')
|
||||
app.model = Model()
|
||||
app.clear_log()
|
||||
app.build_step_rows(app.model.steps)
|
||||
app.show('run')
|
||||
for e in SAMPLE_RUN:
|
||||
app.feed(json.dumps(e))
|
||||
app.log_open = False
|
||||
app.toggle_log()
|
||||
app.refresh_run()
|
||||
shot(f'run-{tag}')
|
||||
if lang == 'en': # the log follows the window language: no English line may remain after the switch
|
||||
app.ui_var.set('ru')
|
||||
app.switch_language()
|
||||
text = app.log.get('1.0', 'end')
|
||||
mixed.append('Checking' in text or 'Extracting' not in text and 'Распаковка' not in text)
|
||||
app.ui_var.set('en')
|
||||
app.switch_language()
|
||||
app.model = Model()
|
||||
for e in SAMPLE_OK:
|
||||
app.model.apply(e)
|
||||
app.model.dest = 'Z:/selftest/TotK'
|
||||
app.model.finish(0)
|
||||
app.refresh_finish()
|
||||
app.show('finish')
|
||||
shot(f'finish-ok-{tag}')
|
||||
app.model = Model()
|
||||
for e in SAMPLE_RUN[:3] + [{'type': 'error', 'id': 'check.disk_space', 'step': 'check',
|
||||
'params': {'path': 'Z:/selftest/TotK', 'need_gb': 36.4, 'free_gb': 20.1}},
|
||||
{'type': 'end', 'result': 'failed', 'seconds': 3}]:
|
||||
app.model.apply(e)
|
||||
app.model.finish(1)
|
||||
app.refresh_finish()
|
||||
shot(f'finish-failed-{tag}')
|
||||
app.model = Model()
|
||||
app.model.finish(3)
|
||||
app.refresh_finish()
|
||||
shot(f'finish-cancelled-{tag}')
|
||||
app.toggle_log()
|
||||
root.destroy()
|
||||
|
||||
root.after(200, run_all)
|
||||
root.after(60000, root.destroy) # never leaves a window open
|
||||
root.mainloop()
|
||||
return 0 if len(steps) == len(UI_LANGUAGES) and all(steps) and mixed == [False] else 1
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--selftest', action='store_true')
|
||||
ap.add_argument('--shots', type=Path)
|
||||
ap.add_argument('--lang', choices=UI_LANGUAGES)
|
||||
ap.add_argument('--root', type=Path)
|
||||
args = ap.parse_args(argv)
|
||||
layout = layouts.find(args.root)
|
||||
if args.selftest:
|
||||
return selftest(layout, args.shots)
|
||||
fix_tcl_paths()
|
||||
root = tk.Tk()
|
||||
root.geometry('720x640')
|
||||
App(root, args.lang or windows_ui_language(), layout)
|
||||
root.mainloop()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,361 @@
|
||||
"""The installer window without Tk: text catalogs, the form, the event model and the CLI subprocess.
|
||||
|
||||
gui.py draws; everything it decides lives here so tests/installer_gui_test.py runs it headless.
|
||||
|
||||
Catalog text of an id in the window language (installer/text/<lang>.json), English when that lacks it
|
||||
Form what the player entered; command() and problems() build and check the CLI call
|
||||
Model folds the CLI's JSON events (events.py) into step states, progress, space, errors and the result
|
||||
Process runs `python -m installer ...`, cancels it with Ctrl+Break and feeds lines to a queue
|
||||
"""
|
||||
import ctypes
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from . import ini
|
||||
from . import state as statefile
|
||||
from .events import render
|
||||
|
||||
TEXT = Path(__file__).resolve().parent / 'text'
|
||||
UI_LANGUAGES = ('en', 'ru')
|
||||
STEP_ORDER = ('check', 'extract', 'lift', 'compile', 'prepare', 'cleanup')
|
||||
# message parameter -> catalog id prefix of the names that replace its raw values
|
||||
PARAM_NAMES = {'lang': 'lang.', 'known': 'lang.', 'step': 'step.', 'input': 'param.input.'}
|
||||
LANG_RU = 0x19 # primary language id of Russian in a Windows LANGID
|
||||
|
||||
|
||||
def load_texts(lang):
|
||||
path = TEXT / f'{lang}.json'
|
||||
return json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
|
||||
|
||||
|
||||
class Catalog:
|
||||
"""Window text: ids from text/<lang>.json, then text/en.json, then the id itself."""
|
||||
|
||||
def __init__(self, lang='en'):
|
||||
self.english = load_texts('en')
|
||||
self.set_language(lang)
|
||||
|
||||
def set_language(self, lang):
|
||||
self.lang = lang if lang in UI_LANGUAGES else 'en'
|
||||
self.texts = load_texts(self.lang)
|
||||
|
||||
def has(self, id):
|
||||
return id in self.texts or id in self.english
|
||||
|
||||
def __call__(self, id, **params):
|
||||
text = self.texts.get(id) or self.english.get(id)
|
||||
if text is None:
|
||||
return id
|
||||
return render({id: text}, id, {k: self.param(k, v) for k, v in params.items()})
|
||||
|
||||
def param(self, name, value):
|
||||
"""A message parameter as the player should read it: language codes, step ids and input names become the
|
||||
catalog's names (the ones the pickers and the step list show); the rest stays as it is."""
|
||||
prefix = PARAM_NAMES.get(name)
|
||||
if prefix:
|
||||
if isinstance(value, (list, tuple)):
|
||||
return ', '.join(self.param(name, v) for v in value)
|
||||
ident = prefix + str(value)
|
||||
if self.has(ident):
|
||||
return self(ident)
|
||||
if name == 'known':
|
||||
return ', '.join(self(prefix + c) if self.has(prefix + c) else c for c in str(value).split(', '))
|
||||
return flat(value)
|
||||
|
||||
|
||||
def flat(value):
|
||||
return ', '.join(str(v) for v in value) if isinstance(value, (list, tuple)) else value
|
||||
|
||||
|
||||
def windows_ui_language():
|
||||
"""'ru' when the Windows display language is Russian, else 'en'."""
|
||||
try:
|
||||
langid = ctypes.windll.kernel32.GetUserDefaultUILanguage()
|
||||
except (AttributeError, OSError):
|
||||
return 'en'
|
||||
return 'ru' if langid & 0x3ff == LANG_RU else 'en'
|
||||
|
||||
|
||||
def default_game_language(ui):
|
||||
return 'ru' if ui == 'ru' else 'en-US'
|
||||
|
||||
|
||||
def gib(n):
|
||||
"""Bytes as decimal gigabytes with one digit, the unit the check.disk_space message uses."""
|
||||
return f'{n / 1e9:.1f}'
|
||||
|
||||
|
||||
def amount(cat, done, total, unit):
|
||||
"""'6.0 / 16.0 GB', '120 / 4000 files'..., the progress of a phase whose size is known."""
|
||||
if unit == 'bytes':
|
||||
return cat('ui.amount_bytes', done=gib(done or 0), total=gib(total))
|
||||
id = f'ui.amount_{unit}'
|
||||
return cat(id if cat.has(id) else 'ui.amount_other', done=done or 0, total=total, unit=unit or '')
|
||||
|
||||
|
||||
# ---- the form ----------------------------------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class Form:
|
||||
base: str = ''
|
||||
update: str = ''
|
||||
keys: str = ''
|
||||
lang: str = ''
|
||||
dest: str = ''
|
||||
|
||||
def installed_state(self):
|
||||
"""The recorded state of the install folder (totk-install.json), or None. Never raises."""
|
||||
if not self.dest.strip():
|
||||
return None
|
||||
try:
|
||||
return statefile.load(Path(self.dest.strip()))
|
||||
except statefile.StateError:
|
||||
return None
|
||||
|
||||
def update_mode(self):
|
||||
return self.installed_state() is not None
|
||||
|
||||
def problems(self):
|
||||
"""Ids of the entries that are missing or wrong, in screen order. A folder with totk-install.json needs only
|
||||
itself; the CLI asks for the game files when it must extract again."""
|
||||
found = []
|
||||
update = self.update_mode()
|
||||
if not self.dest.strip():
|
||||
found.append('ui.need_dest')
|
||||
if not update:
|
||||
for name in ('base', 'update', 'keys'):
|
||||
value = getattr(self, name).strip()
|
||||
if not value:
|
||||
found.append(f'ui.need_{name}')
|
||||
elif not Path(value).is_file():
|
||||
found.append(f'ui.missing_{name}')
|
||||
if self.lang not in ini.LANGUAGES:
|
||||
found.append('ui.need_lang')
|
||||
else:
|
||||
for name in ('base', 'update', 'keys'):
|
||||
value = getattr(self, name).strip()
|
||||
if value and not Path(value).is_file():
|
||||
found.append(f'ui.missing_{name}')
|
||||
return found
|
||||
|
||||
def command(self, python, root=None, temp=None, verify=False):
|
||||
cmd = [str(python), '-m', 'installer']
|
||||
for flag, value in (('--base', self.base), ('--update', self.update), ('--keys', self.keys),
|
||||
('--lang', self.lang), ('--dest', self.dest)):
|
||||
if value.strip():
|
||||
cmd += [flag, value.strip()]
|
||||
if root:
|
||||
cmd += ['--root', str(root)]
|
||||
if temp:
|
||||
cmd += ['--temp', str(temp)]
|
||||
if verify:
|
||||
cmd.append('--verify')
|
||||
return cmd
|
||||
|
||||
|
||||
# ---- events -> state ---------------------------------------------------------------------------------------------
|
||||
|
||||
PENDING, RUNNING, DONE, SKIPPED, FAILED = 'pending', 'running', 'done', 'skipped', 'failed'
|
||||
|
||||
|
||||
def parse_line(line):
|
||||
"""The event dict of a stdout line, or None when it is not a JSON event (a traceback line, noise)."""
|
||||
line = line.strip()
|
||||
if not line.startswith('{'):
|
||||
return None
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except ValueError:
|
||||
return None
|
||||
return event if isinstance(event, dict) and isinstance(event.get('type'), str) else None
|
||||
|
||||
|
||||
class Model:
|
||||
"""What the window shows, folded from events. apply() takes one event; nothing else mutates it."""
|
||||
|
||||
def __init__(self):
|
||||
self.steps = list(STEP_ORDER)
|
||||
self.states = {s: PENDING for s in self.steps}
|
||||
self.mode = ''
|
||||
self.release = ''
|
||||
self.log_path = ''
|
||||
self.dest = ''
|
||||
self.current = None # step id running
|
||||
self.phase = None
|
||||
self.done = self.total = None
|
||||
self.unit = None
|
||||
self.space = None # the space event
|
||||
self.warnings = [] # (id, params)
|
||||
self.error = None # (id, params, step)
|
||||
self.result = None # ok | failed | cancelled once the end event arrived
|
||||
self.seconds = None
|
||||
self.exit_code = None
|
||||
|
||||
def apply(self, e):
|
||||
kind = e['type']
|
||||
if kind == 'start':
|
||||
self.__init__() # a new run: nothing of an earlier one stays
|
||||
self.steps = list(e.get('steps') or STEP_ORDER)
|
||||
self.states = {s: PENDING for s in self.steps}
|
||||
self.mode, self.release = e.get('mode', ''), e.get('release', '')
|
||||
self.log_path, self.dest = e.get('log', ''), e.get('dest', '')
|
||||
elif kind == 'plan':
|
||||
for s, reason in (e.get('skip') or {}).items():
|
||||
if s in self.states and self.states[s] == PENDING:
|
||||
self.states[s] = SKIPPED
|
||||
elif kind == 'step':
|
||||
s, state = e.get('step'), e.get('state')
|
||||
if s not in self.states:
|
||||
self.states[s] = PENDING
|
||||
self.steps.append(s)
|
||||
if state == 'started':
|
||||
self.states[s], self.current = RUNNING, s
|
||||
self.phase = self.done = self.total = self.unit = None
|
||||
elif state == 'done':
|
||||
self.states[s] = DONE
|
||||
self.current = None if self.current == s else self.current
|
||||
elif state == 'skipped':
|
||||
self.states[s] = SKIPPED
|
||||
elif kind == 'progress':
|
||||
if e.get('step') in self.states and self.states[e['step']] == PENDING:
|
||||
self.states[e['step']] = RUNNING
|
||||
self.current = e.get('step', self.current)
|
||||
self.phase = e.get('phase')
|
||||
self.done, self.total, self.unit = e.get('done'), e.get('total'), e.get('unit')
|
||||
elif kind == 'space':
|
||||
self.space = e
|
||||
elif kind == 'warning':
|
||||
self.warnings.append((e.get('id', ''), e.get('params') or {}))
|
||||
elif kind == 'error':
|
||||
self.error = (e.get('id', ''), e.get('params') or {}, e.get('step'))
|
||||
if e.get('step') in self.states:
|
||||
self.states[e['step']] = FAILED
|
||||
elif kind == 'end':
|
||||
self.result = e.get('result')
|
||||
self.seconds = e.get('seconds')
|
||||
self.current = None
|
||||
|
||||
def fraction(self):
|
||||
"""0..1 of the current phase, or None when its size is unknown."""
|
||||
if self.total:
|
||||
return max(0.0, min(1.0, (self.done or 0) / self.total))
|
||||
return None
|
||||
|
||||
def finish(self, exit_code):
|
||||
"""The process ended. A missing end event (the process died) counts as failed, or cancelled on exit code 3."""
|
||||
self.exit_code = exit_code
|
||||
if self.result is None:
|
||||
self.result = 'cancelled' if exit_code == 3 else 'failed'
|
||||
if self.result == 'failed' and self.error is None:
|
||||
self.error = ('ui.process_died', {'code': exit_code}, self.current)
|
||||
for s, state in self.states.items():
|
||||
if state == RUNNING:
|
||||
self.states[s] = FAILED if self.result == 'failed' else PENDING
|
||||
|
||||
def need_text(self, cat):
|
||||
"""The space line, or '' before the check step reported."""
|
||||
sp = self.space
|
||||
if not sp:
|
||||
return ''
|
||||
key = 'ui.space_one' if sp.get('same_disk') or not sp.get('temp_need') else 'ui.space_two'
|
||||
return cat(key, path=sp.get('dest', ''), need=gib(sp.get('need', 0)), free=gib(sp.get('free', 0)),
|
||||
temp=sp.get('temp', ''), temp_need=gib(sp.get('temp_need', 0)),
|
||||
temp_free=gib(sp.get('temp_free', 0)))
|
||||
|
||||
|
||||
def describe_event(cat, e):
|
||||
"""One log line for an event, in the window language."""
|
||||
kind = e['type']
|
||||
if kind == 'step':
|
||||
text = f'{cat("step." + e["step"])}: {cat("ui.state_" + e["state"])}'
|
||||
if 'seconds' in e:
|
||||
text += f' ({cat("ui.seconds", n=format(e["seconds"], ".1f"))})'
|
||||
elif e.get('reason'):
|
||||
text += f' ({e["reason"]})'
|
||||
return text
|
||||
if kind == 'progress':
|
||||
text = cat(f'phase.{e["step"]}.{e["phase"]}')
|
||||
if e.get('total'):
|
||||
text += ' ' + amount(cat, e.get('done'), e['total'], e.get('unit'))
|
||||
return text
|
||||
if kind in ('warning', 'error'):
|
||||
return f'{cat("ui.log_" + kind)}: {cat(e["id"], **(e.get("params") or {}))}'
|
||||
if kind == 'space':
|
||||
return cat('ui.log_space', need=gib(e.get('need', 0)), free=gib(e.get('free', 0)))
|
||||
if kind == 'end':
|
||||
return f'{cat("ui.log_end")}: {cat("ui.result_" + str(e.get("result")))}'
|
||||
return ''
|
||||
|
||||
|
||||
# ---- the CLI subprocess ------------------------------------------------------------------------------------------
|
||||
|
||||
CREATE_NEW_PROCESS_GROUP = 0x200
|
||||
CREATE_NO_WINDOW = 0x08000000
|
||||
|
||||
_BREAK = """
|
||||
import ctypes, sys, time
|
||||
k = ctypes.windll.kernel32
|
||||
pid = int(sys.argv[1])
|
||||
k.FreeConsole()
|
||||
if k.AttachConsole(pid):
|
||||
k.SetConsoleCtrlHandler(None, True)
|
||||
k.GenerateConsoleCtrlEvent(1, pid)
|
||||
time.sleep(1)
|
||||
k.FreeConsole()
|
||||
else:
|
||||
sys.exit(1)
|
||||
"""
|
||||
|
||||
|
||||
def send_break(proc, python=None):
|
||||
"""Ctrl+Break to the process group of proc (started with CREATE_NEW_PROCESS_GROUP). The window itself may have no
|
||||
console (pythonw), so a short helper process attaches to the child's console and sends it."""
|
||||
helper = [str(python or sys.executable), '-c', _BREAK, str(proc.pid)]
|
||||
flags = CREATE_NO_WINDOW if os.name == 'nt' else 0
|
||||
r = subprocess.run(helper, creationflags=flags, capture_output=True)
|
||||
return r.returncode == 0
|
||||
|
||||
|
||||
class Process:
|
||||
"""`python -m installer ...` with its stdout lines going to self.lines (a queue).
|
||||
|
||||
poll() returns the exit code once the process ended and its output is drained, else None."""
|
||||
|
||||
def __init__(self, command, cwd=None, helper_python=None, creationflags=None):
|
||||
self.command = command
|
||||
self.helper_python = helper_python
|
||||
self.lines = queue.Queue()
|
||||
flags = (CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW) if creationflags is None else creationflags
|
||||
env = dict(os.environ, PYTHONIOENCODING='utf-8', PYTHONUTF8='1', PYTHONUNBUFFERED='1')
|
||||
self.proc = subprocess.Popen(command, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, creationflags=flags)
|
||||
self.cancelled_at = None
|
||||
self.reader = threading.Thread(target=self.read, daemon=True)
|
||||
self.reader.start()
|
||||
|
||||
def read(self):
|
||||
for raw in iter(self.proc.stdout.readline, b''):
|
||||
self.lines.put(raw.decode('utf-8', errors='replace'))
|
||||
self.proc.stdout.close()
|
||||
|
||||
def cancel(self):
|
||||
if self.cancelled_at is None and self.proc.poll() is None:
|
||||
self.cancelled_at = time.monotonic()
|
||||
send_break(self.proc, self.helper_python)
|
||||
|
||||
def poll(self):
|
||||
code = self.proc.poll()
|
||||
if code is not None and not self.reader.is_alive() and self.lines.empty():
|
||||
return code
|
||||
return None
|
||||
|
||||
def kill(self):
|
||||
if self.proc.poll() is None:
|
||||
self.proc.kill()
|
||||
@@ -0,0 +1,65 @@
|
||||
// Host check of the installer's first step, compiled with the bundled clang at install time: does the CPU run
|
||||
// -march=x86-64-v3 code, and does a Vulkan 1.3 device exist. Prints one JSON object on stdout.
|
||||
// The Vulkan loader is opened at run time, so a machine without one gets "loader": false instead of a failed start.
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
#include <vulkan/vulkan.h>
|
||||
#ifdef _WIN32
|
||||
#include <windows.h>
|
||||
#define open_lib() LoadLibraryA("vulkan-1.dll")
|
||||
#define lib_sym(l, n) ((void *)GetProcAddress((l), (n)))
|
||||
#else
|
||||
#include <dlfcn.h>
|
||||
#define open_lib() dlopen("libvulkan.so.1", RTLD_NOW)
|
||||
#define lib_sym(l, n) dlsym((l), (n))
|
||||
#endif
|
||||
|
||||
static void json_string(const char *s) {
|
||||
putchar('"');
|
||||
for (; *s; s++) {
|
||||
if (*s == '"' || *s == '\\') printf("\\%c", *s);
|
||||
else if ((unsigned char)*s < 0x20) printf("\\u%04x", *s);
|
||||
else putchar(*s);
|
||||
}
|
||||
putchar('"');
|
||||
}
|
||||
|
||||
int main(void) {
|
||||
__builtin_cpu_init();
|
||||
printf("{\"x86_64_v3\": %s, ", __builtin_cpu_supports("x86-64-v3") ? "true" : "false");
|
||||
void *lib = open_lib();
|
||||
PFN_vkGetInstanceProcAddr gipa = lib ? (PFN_vkGetInstanceProcAddr)lib_sym(lib, "vkGetInstanceProcAddr") : NULL;
|
||||
if (!gipa) {
|
||||
printf("\"loader\": false, \"devices\": []}\n");
|
||||
return 0;
|
||||
}
|
||||
printf("\"loader\": true, ");
|
||||
PFN_vkCreateInstance create = (PFN_vkCreateInstance)gipa(NULL, "vkCreateInstance");
|
||||
VkApplicationInfo app = {.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO, .pApplicationName = "totk-install",
|
||||
.apiVersion = VK_API_VERSION_1_3};
|
||||
VkInstanceCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, .pApplicationInfo = &app};
|
||||
VkInstance inst = VK_NULL_HANDLE;
|
||||
VkResult r = create ? create(&ci, NULL, &inst) : VK_ERROR_INITIALIZATION_FAILED;
|
||||
if (r != VK_SUCCESS) {
|
||||
printf("\"instance\": %d, \"devices\": []}\n", (int)r);
|
||||
return 0;
|
||||
}
|
||||
PFN_vkEnumeratePhysicalDevices enumerate = (PFN_vkEnumeratePhysicalDevices)gipa(inst, "vkEnumeratePhysicalDevices");
|
||||
PFN_vkGetPhysicalDeviceProperties props = (PFN_vkGetPhysicalDeviceProperties)gipa(inst, "vkGetPhysicalDeviceProperties");
|
||||
PFN_vkDestroyInstance destroy = (PFN_vkDestroyInstance)gipa(inst, "vkDestroyInstance");
|
||||
VkPhysicalDevice devs[16];
|
||||
uint32_t n = 16;
|
||||
if (enumerate(inst, &n, devs) < 0) n = 0;
|
||||
printf("\"instance\": 0, \"devices\": [");
|
||||
for (uint32_t i = 0; i < n; i++) {
|
||||
VkPhysicalDeviceProperties p;
|
||||
props(devs[i], &p);
|
||||
printf("%s{\"name\": ", i ? ", " : "");
|
||||
json_string(p.deviceName);
|
||||
printf(", \"api\": [%u, %u, %u], \"type\": %d}", VK_API_VERSION_MAJOR(p.apiVersion),
|
||||
VK_API_VERSION_MINOR(p.apiVersion), VK_API_VERSION_PATCH(p.apiVersion), (int)p.deviceType);
|
||||
}
|
||||
printf("]}\n");
|
||||
destroy(inst, NULL);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
"""totk.ini: set the game language and keep everything else (keys and syntax as runtime/cfg.c reads them).
|
||||
|
||||
A new file starts from totk.ini.example, whose lines are all commented defaults. The language goes into `lang`
|
||||
(a Switch language code, runtime/ov_system.c): an active `lang = ...` line is rewritten in place, else the commented
|
||||
`# lang = ...` line of the example is activated, else the line is added to [language] (created at the end if
|
||||
missing). Other lines, comments and line order stay as they are.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
# runtime/ov_system.c: the codes TOTK_LANG accepts.
|
||||
LANGUAGES = ('ja', 'en-US', 'en-GB', 'fr', 'fr-CA', 'de', 'it', 'es', 'es-419', 'nl', 'pt', 'pt-BR', 'ru', 'ko',
|
||||
'zh-CN', 'zh-TW', 'zh-Hans', 'zh-Hant')
|
||||
# Their text falls back to the Switch shared fonts (runtime/ov_system.c), which the installer does not extract.
|
||||
NEED_SHARED_FONTS = ('ko', 'zh-CN', 'zh-TW', 'zh-Hans', 'zh-Hant')
|
||||
|
||||
ACTIVE = re.compile(r'^(\s*)(?:totk_)?lang(\s*=\s*)(.*?)(\s*)$', re.IGNORECASE)
|
||||
COMMENTED = re.compile(r'^\s*[#;]\s*(?:totk_)?lang\s*=.*$', re.IGNORECASE)
|
||||
SECTION = re.compile(r'^\s*\[(.*)\]\s*$')
|
||||
|
||||
|
||||
def set_language(text, lang):
|
||||
"""text with `lang = <lang>`; see the module docstring."""
|
||||
lines = text.splitlines()
|
||||
for i, line in enumerate(lines):
|
||||
m = ACTIVE.match(line)
|
||||
if m:
|
||||
lines[i] = f'{m.group(1)}lang{m.group(2)}{lang}'
|
||||
return '\n'.join(lines) + '\n'
|
||||
for i, line in enumerate(lines):
|
||||
if COMMENTED.match(line):
|
||||
lines[i] = f'lang = {lang}'
|
||||
return '\n'.join(lines) + '\n'
|
||||
for i, line in enumerate(lines):
|
||||
m = SECTION.match(line)
|
||||
if m and m.group(1).strip().lower() == 'language':
|
||||
lines.insert(i + 1, f'lang = {lang}')
|
||||
return '\n'.join(lines) + '\n'
|
||||
if lines and lines[-1].strip():
|
||||
lines.append('')
|
||||
lines += ['[language]', f'lang = {lang}']
|
||||
return '\n'.join(lines) + '\n'
|
||||
|
||||
|
||||
def get_language(text):
|
||||
for line in text.splitlines():
|
||||
m = ACTIVE.match(line)
|
||||
if m:
|
||||
return m.group(3)
|
||||
return None
|
||||
|
||||
|
||||
def write(dest, lang, example=None):
|
||||
"""Write dest/totk.ini with lang; an existing file keeps its other lines. Returns the path."""
|
||||
path = Path(dest) / 'totk.ini'
|
||||
if path.is_file():
|
||||
text = path.read_text(encoding='utf-8-sig')
|
||||
elif example and Path(example).is_file():
|
||||
text = Path(example).read_text(encoding='utf-8')
|
||||
else:
|
||||
text = ''
|
||||
new = set_language(text, lang)
|
||||
if path.is_file() and new == text:
|
||||
return path
|
||||
tmp = path.with_name('totk.ini.tmp')
|
||||
tmp.write_text(new, encoding='utf-8', newline='\n')
|
||||
os.replace(tmp, path)
|
||||
return path
|
||||
@@ -0,0 +1,38 @@
|
||||
/* TotK-Setup.exe: starts python\pythonw.exe -m installer.gui with this exe's folder as the working directory.
|
||||
|
||||
No console window. The two message strings come from the catalog (installer/text/<lang>.json, ids launcher.title and
|
||||
launcher.python_missing); tools/make_release.py writes them into launcher_text.h before it compiles this file.
|
||||
Build: llvm/bin/clang -O2 -mwindows launcher.c -o TotK-Setup.exe */
|
||||
#include <windows.h>
|
||||
#include <wchar.h>
|
||||
|
||||
#include "launcher_text.h"
|
||||
|
||||
#define LANGID_RUSSIAN 0x19
|
||||
#define PATH_CHARS (MAX_PATH * 2)
|
||||
|
||||
static void fail(void)
|
||||
{
|
||||
int russian = (GetUserDefaultUILanguage() & 0x3ff) == LANGID_RUSSIAN;
|
||||
MessageBoxW(NULL, russian ? LAUNCHER_PYTHON_MISSING_RU : LAUNCHER_PYTHON_MISSING_EN,
|
||||
russian ? LAUNCHER_TITLE_RU : LAUNCHER_TITLE_EN, MB_OK | MB_ICONERROR);
|
||||
}
|
||||
|
||||
int WINAPI WinMain(HINSTANCE instance, HINSTANCE previous, LPSTR command_line, int show)
|
||||
{
|
||||
wchar_t folder[PATH_CHARS], python[PATH_CHARS], command[PATH_CHARS + 64];
|
||||
DWORD n = GetModuleFileNameW(NULL, folder, PATH_CHARS);
|
||||
STARTUPINFOW si = { sizeof si };
|
||||
PROCESS_INFORMATION pi;
|
||||
(void)instance; (void)previous; (void)command_line; (void)show;
|
||||
if (n == 0 || n >= PATH_CHARS) { fail(); return 1; }
|
||||
while (n > 0 && folder[n - 1] != L'\\') n--;
|
||||
folder[n > 0 ? n - 1 : 0] = 0;
|
||||
if (swprintf(python, PATH_CHARS, L"%ls\\python\\pythonw.exe", folder) < 0 ||
|
||||
GetFileAttributesW(python) == INVALID_FILE_ATTRIBUTES) { fail(); return 1; }
|
||||
if (swprintf(command, PATH_CHARS + 64, L"\"%ls\" -m installer.gui", python) < 0) { fail(); return 1; }
|
||||
if (!CreateProcessW(python, command, NULL, NULL, FALSE, 0, NULL, folder, &si, &pi)) { fail(); return 1; }
|
||||
CloseHandle(pi.hThread);
|
||||
CloseHandle(pi.hProcess);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Where the installer finds what it runs: the one place that maps the release folder and a repository checkout.
|
||||
|
||||
Release (docs/installer.md "Release layout"), root = the unpacked folder:
|
||||
|
||||
python/python.exe runs every tool (numpy, lz4, cryptography, compression.zstd); its ._pth puts .., ../tools
|
||||
and ../recomp on sys.path, so `python -m installer` imports from the release folder
|
||||
recomp/, tools/ the lifter and the scripts; recomp/patches-v121.txt, tools/data/expected-v1.2.1.json
|
||||
llvm/ llvm-mingw (bin/clang.exe)
|
||||
runtime/ libtotkrt.a, the headers and install-side sources (romfs.c); runtime/deps/ holds
|
||||
include/ + lib/ of Vulkan, SDL3, Opus, libvpx and VMA (build_game.py --deps)
|
||||
bin/ totk_xlate.exe (+ totk_xlate.id), hactool.exe, ninja.exe, SDL3.dll, libopus-0.dll
|
||||
data/shader_list.tsv the MacNativeGames shader list
|
||||
totk.ini.example
|
||||
VERSION the release version, one line
|
||||
temp/ the default temp folder (created)
|
||||
|
||||
Dev mode, root = a repository checkout (no python/python.exe): build/py314-embed/py/python.exe (the embeddable
|
||||
Python the release ships, with the same packages), else this Python; toolchain/llvm-mingw-*/ and toolchain/deps-mingw;
|
||||
runtime/ as the sources with build/install-rtlib/libtotkrt.a (tools/build_game.py --make-runtime-lib --out
|
||||
build/install-rtlib); MSYS2 UCRT64 ninja.exe; toolchain/hactool; build/xlate/totk_xlate.exe; the shader list from
|
||||
refs/MacNativeGames; build/install-temp. The version is dev-<commit>.
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
EXE = '.exe' if os.name == 'nt' else ''
|
||||
UCRT = Path(r'C:\msys64\ucrt64\bin')
|
||||
SHADER_LIST_REF = 'origin/HEAD:games/totk/assets/shader_list.tsv' # dev: tools/prepare_shaders.py LIST_REF
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Layout:
|
||||
root: Path
|
||||
dev: bool
|
||||
release: str
|
||||
python: Path # runs the tools
|
||||
tools: Path
|
||||
recomp: Path
|
||||
patches: Path # code patch table for recomp/gen.py
|
||||
expected: Path # expected hashes of the supported update
|
||||
llvm: Path # llvm-mingw tree
|
||||
deps: Path # include/ and lib/ for the link
|
||||
runtime: Path # runtime headers and install-side sources
|
||||
runtime_lib: Path # libtotkrt.a
|
||||
ninja: Path
|
||||
hactool: Path
|
||||
xlate: Path # totk_xlate.exe; a totk_xlate.id beside it is copied too
|
||||
dlls: tuple = field(default_factory=tuple)
|
||||
shader_list: Path | None = None # None: dev, read from refs/MacNativeGames through git
|
||||
ini_example: Path | None = None
|
||||
temp: Path | None = None
|
||||
|
||||
@property
|
||||
def clang(self):
|
||||
return self.llvm / 'bin' / ('clang' + EXE)
|
||||
|
||||
def missing(self):
|
||||
"""Paths the steps need that do not exist."""
|
||||
need = [self.python, self.tools / 'extract.py', self.tools / 'build_game.py', self.tools / 'prepare_shaders.py',
|
||||
self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.patches, self.expected, self.clang,
|
||||
self.deps / 'include', self.deps / 'lib', self.runtime, self.runtime_lib, self.ninja, self.hactool,
|
||||
self.xlate, *self.dlls]
|
||||
if self.shader_list:
|
||||
need.append(self.shader_list)
|
||||
return [str(p) for p in need if not p.exists()]
|
||||
|
||||
|
||||
def is_release(root):
|
||||
return (root / 'python' / ('python' + EXE)).is_file()
|
||||
|
||||
|
||||
def release_layout(root):
|
||||
version = root / 'VERSION'
|
||||
return Layout(
|
||||
root=root, dev=False,
|
||||
release=version.read_text(encoding='utf-8').strip() if version.is_file() else 'unknown',
|
||||
python=root / 'python' / ('python' + EXE),
|
||||
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
|
||||
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
|
||||
llvm=root / 'llvm', deps=root / 'runtime' / 'deps', runtime=root / 'runtime',
|
||||
runtime_lib=root / 'runtime' / 'libtotkrt.a',
|
||||
ninja=root / 'bin' / ('ninja' + EXE), hactool=root / 'bin' / ('hactool' + EXE),
|
||||
xlate=root / 'bin' / ('totk_xlate' + EXE),
|
||||
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll'),
|
||||
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example', temp=root / 'temp')
|
||||
|
||||
|
||||
def dev_layout(root):
|
||||
commit = subprocess.run(['git', '-C', str(root), 'rev-parse', '--short', 'HEAD'], capture_output=True,
|
||||
text=True).stdout.strip()
|
||||
embed = root / 'build' / 'py314-embed' / 'py' / ('python' + EXE)
|
||||
trees = sorted(p.parent.parent for p in (root / 'toolchain').glob('llvm-mingw-*/llvm-mingw-*/bin/clang' + EXE))
|
||||
deps = root / 'toolchain' / 'deps-mingw'
|
||||
return Layout(
|
||||
root=root, dev=True, release=f'dev-{commit or "unknown"}',
|
||||
python=embed if embed.is_file() else Path(sys.executable),
|
||||
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
|
||||
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
|
||||
llvm=trees[-1] if trees else root / 'toolchain' / 'llvm-mingw-missing', deps=deps,
|
||||
runtime=root / 'runtime', runtime_lib=root / 'build' / 'install-rtlib' / 'libtotkrt.a',
|
||||
ninja=UCRT / ('ninja' + EXE), hactool=root / 'toolchain' / 'hactool' / ('hactool' + EXE),
|
||||
xlate=root / 'build' / 'xlate' / ('totk_xlate' + EXE),
|
||||
dlls=tuple(sorted((deps / 'bin').glob('*.dll'))),
|
||||
shader_list=None, ini_example=root / 'totk.ini.example', temp=root / 'build' / 'install-temp')
|
||||
|
||||
|
||||
def find(root=None):
|
||||
"""The layout of root (default: the folder above this package)."""
|
||||
root = Path(root or Path(__file__).resolve().parent.parent).resolve()
|
||||
return release_layout(root) if is_release(root) else dev_layout(root)
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Which steps run, in what order, and the bookkeeping around them: fresh install, resume and update are one rule.
|
||||
|
||||
Each step has a key: a fingerprint of everything its output depends on, computed before anything runs (a step's key
|
||||
includes the keys of the steps it reads from, so a change upstream reaches every step below it). A step is skipped
|
||||
when totk-install.json records it with the same key and its output still matches the record (Step.current, cheap).
|
||||
Otherwise it runs. A transient step (its output lives in the temp folder and only the next step reads it) runs only
|
||||
when the next step runs. `always` steps (check, cleanup) run every time.
|
||||
|
||||
Update mode is the same rule with a newer release: the extraction's key depends only on the expected hashes, so a
|
||||
release that changes the lifter or the runtime keeps romfs/ and exefs/ and rebuilds from lift on. Before keeping the
|
||||
output of a step recorded by another release (or with --verify), the check step calls Step.verify (expensive: the full
|
||||
RomFS hash); a mismatch makes the step run again.
|
||||
|
||||
A failed or cancelled run leaves the finished steps recorded; the next run starts at the first step that is not.
|
||||
"""
|
||||
import datetime
|
||||
import traceback
|
||||
|
||||
from . import state as statefile
|
||||
|
||||
|
||||
class InstallError(Exception):
|
||||
"""A failure with a message id (installer/text/en.json) and its parameters."""
|
||||
|
||||
def __init__(self, id, **params):
|
||||
super().__init__(id)
|
||||
self.id, self.params = id, params
|
||||
|
||||
|
||||
class Step:
|
||||
id = ''
|
||||
transient = False
|
||||
always = False
|
||||
|
||||
def key(self, ctx):
|
||||
"""Fingerprint of the step's inputs; may read ctx.keys of earlier steps."""
|
||||
return ''
|
||||
|
||||
def current(self, ctx, record):
|
||||
"""True when the recorded output is still in place (cheap checks only)."""
|
||||
return True
|
||||
|
||||
def verify(self, ctx, record):
|
||||
"""True when the recorded output passes the full check (hashes); called before keeping another release's
|
||||
output."""
|
||||
return True
|
||||
|
||||
def run(self, ctx):
|
||||
"""Do the work. Returns fields to record with the step."""
|
||||
return {}
|
||||
|
||||
|
||||
class Context:
|
||||
def __init__(self, layout, dest, temp, reporter, options, state=None, verify=False):
|
||||
self.layout, self.dest, self.temp, self.reporter, self.options = layout, dest, temp, reporter, options
|
||||
self.state = state or statefile.empty(layout.release)
|
||||
self.verify = verify
|
||||
self.mode = 'install' if state is None else 'resume' if state.get('release') == layout.release else 'update'
|
||||
self.keys, self.runs, self.reasons, self.steps, self.forced = {}, {}, {}, [], set()
|
||||
|
||||
def record(self, step_id):
|
||||
return self.state['steps'].get(step_id)
|
||||
|
||||
def will_run(self, step_id):
|
||||
return self.runs.get(step_id, False)
|
||||
|
||||
def save(self):
|
||||
self.dest.mkdir(parents=True, exist_ok=True)
|
||||
statefile.save(self.dest, self.state)
|
||||
|
||||
def plan(self, forced=()):
|
||||
"""Fill keys (once), runs and reasons. forced: step ids that must run."""
|
||||
if not self.keys:
|
||||
for s in self.steps:
|
||||
self.keys[s.id] = s.key(self)
|
||||
for s in self.steps:
|
||||
rec = self.record(s.id)
|
||||
if s.always or s.id in forced:
|
||||
self.runs[s.id] = True
|
||||
else:
|
||||
self.runs[s.id] = not (rec and rec.get('key') == self.keys[s.id] and s.current(self, rec))
|
||||
self.reasons[s.id] = '' if self.runs[s.id] else 'current'
|
||||
for i in range(len(self.steps) - 2, -1, -1):
|
||||
s, after = self.steps[i], self.steps[i + 1]
|
||||
if s.transient and self.runs[s.id] and not self.runs[after.id]:
|
||||
self.runs[s.id], self.reasons[s.id] = False, 'not_needed'
|
||||
self.forced = set(forced)
|
||||
|
||||
def verify_kept(self, progress_step):
|
||||
"""Full check of every kept step output that another release recorded (or of all with --verify). A step that
|
||||
fails it is planned to run. Returns the ids that failed."""
|
||||
failed = []
|
||||
for s in self.steps:
|
||||
rec = self.record(s.id)
|
||||
if self.runs[s.id] or self.reasons[s.id] != 'current' or s.always:
|
||||
continue
|
||||
if not (self.verify or rec.get('release') != self.layout.release):
|
||||
continue
|
||||
if not s.verify(self, rec):
|
||||
failed.append(s.id)
|
||||
if failed:
|
||||
for f in failed:
|
||||
self.reporter.warning('state.output_changed', step=f)
|
||||
self.plan(self.forced | set(failed))
|
||||
return failed
|
||||
|
||||
|
||||
def now_utc():
|
||||
return datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
|
||||
|
||||
|
||||
def run_steps(steps, ctx):
|
||||
"""Run the plan. Returns 'ok', 'failed' or 'cancelled'; events and the log say why."""
|
||||
rep = ctx.reporter
|
||||
ctx.steps = steps
|
||||
current = None
|
||||
try:
|
||||
ctx.plan()
|
||||
rep.emit('plan', run=[s.id for s in steps if ctx.runs[s.id]],
|
||||
skip={s.id: ctx.reasons[s.id] for s in steps if not ctx.runs[s.id]})
|
||||
for s in steps:
|
||||
if not ctx.runs[s.id]:
|
||||
rep.step(s.id, 'skipped', reason=ctx.reasons[s.id])
|
||||
continue
|
||||
current = s.id
|
||||
rep.step(s.id, 'started')
|
||||
t0 = rep.elapsed()
|
||||
fields = s.run(ctx) or {}
|
||||
seconds = round(rep.elapsed() - t0, 1)
|
||||
if not s.always:
|
||||
ctx.state['steps'][s.id] = {'key': ctx.keys[s.id], 'release': ctx.layout.release,
|
||||
'finished': now_utc(), 'seconds': seconds, **fields}
|
||||
ctx.save() # the first save, after the check, creates the install folder
|
||||
rep.step(s.id, 'done', seconds=seconds)
|
||||
current = None
|
||||
ctx.state['release'] = ctx.layout.release
|
||||
ctx.save()
|
||||
return 'ok'
|
||||
except InstallError as e:
|
||||
rep.error(e.id, e.params, step=current)
|
||||
return 'failed'
|
||||
except statefile.StateError as e:
|
||||
rep.error(e.id, {'path': str(e.path)}, step=current)
|
||||
return 'failed'
|
||||
except KeyboardInterrupt:
|
||||
rep.log(f'cancelled during {current}')
|
||||
return 'cancelled'
|
||||
except Exception as e:
|
||||
rep.log(traceback.format_exc())
|
||||
rep.error('internal', {'exception': type(e).__name__, 'detail': str(e)}, step=current)
|
||||
return 'failed'
|
||||
@@ -0,0 +1,54 @@
|
||||
"""totk-install.json in the install folder: what built it and which steps finished.
|
||||
|
||||
{"format": 1, "release": "...", "lang": "en-US",
|
||||
"inputs": {"base": {"name", "size", "sha256"}, "update": {...}},
|
||||
"steps": {"<step>": {"key": "<input fingerprint>", "release": "...", "finished": "<UTC time>", "seconds": 12.3,
|
||||
...the step's own fields}}}
|
||||
|
||||
A step's key fingerprints everything its output depends on (runner.py), so a step whose recorded key equals the
|
||||
current one and whose output still checks out is not run again. The file is replaced atomically after each step.
|
||||
No key value or keys file content is ever stored.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
NAME = 'totk-install.json'
|
||||
FORMAT = 1
|
||||
|
||||
|
||||
class StateError(Exception):
|
||||
def __init__(self, id, path):
|
||||
super().__init__(id)
|
||||
self.id, self.path = id, path
|
||||
|
||||
|
||||
def empty(release):
|
||||
return {'format': FORMAT, 'release': release, 'inputs': {}, 'steps': {}}
|
||||
|
||||
|
||||
def load(dest):
|
||||
"""The state of dest, or None when the folder has none. StateError for an unreadable or newer file."""
|
||||
path = Path(dest) / NAME
|
||||
if not path.is_file():
|
||||
return None
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding='utf-8'))
|
||||
except (OSError, ValueError):
|
||||
raise StateError('state.unreadable', path) from None
|
||||
if not isinstance(data, dict) or not isinstance(data.get('steps'), dict):
|
||||
raise StateError('state.unreadable', path)
|
||||
if data.get('format', 0) > FORMAT:
|
||||
raise StateError('state.newer_format', path)
|
||||
return data
|
||||
|
||||
|
||||
def save(dest, data):
|
||||
path = Path(dest) / NAME
|
||||
tmp = path.with_name(NAME + '.tmp')
|
||||
with open(tmp, 'w', encoding='utf-8', newline='\n') as f:
|
||||
json.dump(data, f, indent=1, ensure_ascii=False)
|
||||
f.write('\n')
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp, path)
|
||||
@@ -0,0 +1,629 @@
|
||||
"""The six setup steps of docs/installer.md: check, extract, lift, compile, prepare, cleanup.
|
||||
|
||||
Extraction runs in this process through tools/extract.py (the package's sibling tools/, which ships with it); lift,
|
||||
compile and prepare run the bundled Python on recomp/gen.py, tools/build_game.py, tools/make_romfs_patches.py and
|
||||
tools/prepare_shaders.py, whose output goes to <temp>/logs/<step>-<tool>.log and is read for progress. No Win32 call
|
||||
here: the host check is a small C program built with the bundled clang (hostcheck.c).
|
||||
|
||||
Install folder: exefs/, romfs/ (romfs.bin, index.json), profile/, totk.exe and its DLLs, romfs-index.bin,
|
||||
romfs-patches/, totk_xlate.exe, the shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json.
|
||||
Temp folder (<temp>/<dest name>-<hash>): check/ (ExeFS of both inputs, the host check), gen/ (C), build/ (objects),
|
||||
logs/ (kept by the cleanup).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from . import ini
|
||||
from .layout import EXE
|
||||
from .runner import InstallError, Step
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / 'tools'))
|
||||
import extract # noqa: E402
|
||||
from extract_exefs import (common_tickets, container_entries, load_keys, nca_ctr_key, nca_inventory, # noqa: E402
|
||||
program_nca)
|
||||
|
||||
GiB = 1 << 30
|
||||
DEST_EXTRA = 4 * GiB # beside the RomFS: shader and pipeline caches (about 2 GB), exefs, totk.exe, seed files
|
||||
TEMP_NEED = 3 * GiB # generated C (about 0.9 GB) and objects
|
||||
UPDATE_VERSION = '1.2.1'
|
||||
HOSTCHECK = Path(__file__).resolve().parent / 'hostcheck.c'
|
||||
STAMP = '.install-key'
|
||||
|
||||
|
||||
# ---- helpers
|
||||
|
||||
def file_sha256(path, progress=None):
|
||||
"""SHA-256 hex of a file; progress(done, total) after each block."""
|
||||
h = hashlib.sha256()
|
||||
total = path.stat().st_size
|
||||
done = 0
|
||||
with open(path, 'rb', buffering=0) as f:
|
||||
buf = bytearray(8 << 20)
|
||||
view = memoryview(buf)
|
||||
while n := f.readinto(buf):
|
||||
h.update(view[:n])
|
||||
done += n
|
||||
if progress:
|
||||
progress(done, total)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def fingerprint(*parts):
|
||||
"""SHA-256 over strings and the contents of files and directory trees (a missing path counts as missing)."""
|
||||
h = hashlib.sha256()
|
||||
for p in parts:
|
||||
if isinstance(p, Path):
|
||||
if p.is_dir():
|
||||
for f in sorted(p.rglob('*')):
|
||||
if f.is_file() and '__pycache__' not in f.parts:
|
||||
h.update(f.relative_to(p).as_posix().encode() + b'\0' + bytes.fromhex(file_sha256(f)))
|
||||
elif p.is_file():
|
||||
h.update(p.name.encode() + b'\0' + bytes.fromhex(file_sha256(p)))
|
||||
else:
|
||||
h.update(b'missing\0' + p.name.encode())
|
||||
else:
|
||||
h.update(str(p).encode())
|
||||
h.update(b'\1')
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def remove(path):
|
||||
if path.is_dir() and not path.is_symlink():
|
||||
shutil.rmtree(path)
|
||||
elif path.exists() or path.is_symlink():
|
||||
path.unlink()
|
||||
|
||||
|
||||
def replace_file(src, dst):
|
||||
"""Copy src over dst through a temporary name, so a cancelled copy never leaves half a file under dst's name."""
|
||||
tmp = dst.with_name(dst.name + '.tmp')
|
||||
shutil.copy2(src, tmp)
|
||||
os.replace(tmp, dst)
|
||||
|
||||
|
||||
def kill_tree(proc):
|
||||
if proc.poll() is not None:
|
||||
return
|
||||
if os.name == 'nt':
|
||||
subprocess.run(['taskkill', '/T', '/F', '/PID', str(proc.pid)], capture_output=True)
|
||||
else:
|
||||
proc.kill()
|
||||
|
||||
|
||||
def run_tool(ctx, step, script, args, parse=None, cwd=None):
|
||||
"""Run a tool script with the bundled Python. Its output goes to <temp>/logs/<step>-<script>.log and, line by line,
|
||||
to parse(line). Returns (exit code, log path)."""
|
||||
logs = ctx.temp / 'logs'
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
log = logs / f'{step}-{Path(script).stem}.log'
|
||||
command = [str(ctx.layout.python), '-c', extract.RUN_SCRIPT, str(script), *map(str, args)]
|
||||
ctx.reporter.log('run ' + ' '.join(command[3:]))
|
||||
env = dict(os.environ, PYTHONUNBUFFERED='1', PYTHONIOENCODING='utf-8')
|
||||
with open(log, 'w', encoding='utf-8', newline='\n') as out:
|
||||
proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=cwd or ctx.layout.root,
|
||||
env=env, text=True, encoding='utf-8', errors='replace', bufsize=1)
|
||||
try:
|
||||
for line in proc.stdout:
|
||||
out.write(line)
|
||||
if parse:
|
||||
parse(line.rstrip())
|
||||
code = proc.wait()
|
||||
except BaseException:
|
||||
kill_tree(proc)
|
||||
raise
|
||||
ctx.reporter.log(f'{Path(script).name}: exit code {code}; output in {log}')
|
||||
return code, log
|
||||
|
||||
|
||||
def free_bytes(path):
|
||||
"""Free bytes on the disk of path (its nearest existing parent) and that parent's device."""
|
||||
p = Path(path)
|
||||
while not p.exists():
|
||||
p = p.parent
|
||||
return shutil.disk_usage(p).free, os.stat(p).st_dev
|
||||
|
||||
|
||||
def expected_update():
|
||||
return extract.load_expected()[UPDATE_VERSION]
|
||||
|
||||
|
||||
def shader_list_text(layout):
|
||||
if layout.shader_list:
|
||||
return layout.shader_list.read_text(encoding='utf-8')
|
||||
refs = layout.root / 'refs' / 'MacNativeGames'
|
||||
r = subprocess.run(['git', '-C', str(refs), 'show', 'origin/HEAD:games/totk/assets/shader_list.tsv'],
|
||||
capture_output=True, encoding='utf-8')
|
||||
if r.returncode:
|
||||
raise InstallError('prepare.shader_list_missing')
|
||||
return r.stdout
|
||||
|
||||
|
||||
class Watch(threading.Thread):
|
||||
"""Polls a measure() of bytes written while a long call runs and reports it as progress."""
|
||||
|
||||
def __init__(self, reporter, step):
|
||||
super().__init__(daemon=True)
|
||||
self.reporter, self.step = reporter, step
|
||||
self.phase, self.measure, self.total = None, None, None
|
||||
self.stopped = threading.Event()
|
||||
|
||||
def set(self, phase, measure=None, total=None):
|
||||
self.phase, self.measure, self.total = phase, measure, total
|
||||
self.reporter.progress(self.step, phase, 0 if total else None, total, 'bytes' if total else None, force=True)
|
||||
|
||||
def run(self):
|
||||
while not self.stopped.wait(1.0):
|
||||
phase, measure, total = self.phase, self.measure, self.total
|
||||
if phase and measure:
|
||||
try:
|
||||
done = min(measure(), total)
|
||||
except OSError:
|
||||
continue
|
||||
self.reporter.progress(self.step, phase, done, total, 'bytes')
|
||||
|
||||
def stop(self):
|
||||
self.stopped.set()
|
||||
|
||||
|
||||
def size_of(paths):
|
||||
return sum(p.stat().st_size for p in paths if p.is_file())
|
||||
|
||||
|
||||
# ---- 1. check
|
||||
|
||||
class Check(Step):
|
||||
id = 'check'
|
||||
always = True
|
||||
|
||||
def run(self, ctx):
|
||||
layout, rep, dest = ctx.layout, ctx.reporter, ctx.dest
|
||||
missing = layout.missing()
|
||||
if missing:
|
||||
raise InstallError('check.release_incomplete', paths=missing[:5])
|
||||
if dest.exists() and not dest.is_dir():
|
||||
raise InstallError('check.dest_not_folder', path=str(dest))
|
||||
if ctx.mode == 'install' and dest.is_dir() and any(dest.iterdir()):
|
||||
raise InstallError('check.dest_not_empty', path=str(dest))
|
||||
lang = ctx.options.lang or ctx.state.get('lang')
|
||||
if not lang:
|
||||
raise InstallError('check.lang_missing')
|
||||
if lang not in ini.LANGUAGES:
|
||||
raise InstallError('check.lang_unknown', lang=lang, known=', '.join(ini.LANGUAGES))
|
||||
if lang in ini.NEED_SHARED_FONTS:
|
||||
rep.warning('warn.lang_needs_fonts', lang=lang)
|
||||
ctx.lang = lang
|
||||
ctx.temp.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
rep.progress('check', 'host', force=True)
|
||||
self.host(ctx)
|
||||
rep.progress('check', 'verify', force=True)
|
||||
ctx.verify_kept('check')
|
||||
ncas = 0
|
||||
if ctx.will_run('extract'):
|
||||
rep.progress('check', 'inputs', force=True)
|
||||
ncas = self.inputs(ctx)
|
||||
self.space(ctx, ncas)
|
||||
if ctx.will_run('extract'):
|
||||
rep.progress('check', 'versions', force=True)
|
||||
self.versions(ctx)
|
||||
return {}
|
||||
|
||||
def host(self, ctx):
|
||||
layout, work = ctx.layout, ctx.temp / 'check'
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
exe = work / ('hostcheck' + EXE)
|
||||
command = [str(layout.clang), '-O1', str(HOSTCHECK), '-I', str(layout.deps / 'include'), '-o', str(exe)]
|
||||
if os.name != 'nt':
|
||||
command.append('-ldl')
|
||||
r = subprocess.run(command, capture_output=True, text=True, errors='replace')
|
||||
if r.returncode:
|
||||
ctx.reporter.log(r.stdout + r.stderr)
|
||||
raise InstallError('check.toolchain_broken', detail=(r.stderr or r.stdout).strip()[-400:])
|
||||
r = subprocess.run([str(exe)], capture_output=True, text=True, errors='replace')
|
||||
try:
|
||||
host = json.loads(r.stdout)
|
||||
except ValueError:
|
||||
raise InstallError('check.host_probe_failed', code=r.returncode) from None
|
||||
ctx.reporter.log(f'host: {r.stdout.strip()}')
|
||||
if not host['x86_64_v3']:
|
||||
raise InstallError('check.cpu_no_avx2')
|
||||
if not host['loader']:
|
||||
raise InstallError('check.vulkan_missing')
|
||||
devices = host['devices']
|
||||
if not devices:
|
||||
raise InstallError('check.vulkan_no_device')
|
||||
if not any(tuple(d['api'][:2]) >= (1, 3) for d in devices):
|
||||
raise InstallError('check.vulkan_too_old',
|
||||
devices=', '.join(f'{d["name"]} ({d["api"][0]}.{d["api"][1]})' for d in devices))
|
||||
ctx.host = host
|
||||
|
||||
def inputs(self, ctx):
|
||||
"""Open both files with the keys as the extraction will. Returns the bytes of both Program NCAs."""
|
||||
o = ctx.options
|
||||
for name in ('base', 'update', 'keys'):
|
||||
path = getattr(o, name)
|
||||
if not path:
|
||||
raise InstallError('check.inputs_needed')
|
||||
if not Path(path).is_file():
|
||||
raise InstallError('check.input_missing', input=name, path=str(path))
|
||||
try:
|
||||
keys = load_keys(Path(o.keys))
|
||||
except (ValueError, OSError, UnicodeDecodeError):
|
||||
raise InstallError('check.keys_unreadable') from None
|
||||
if 'header_key' not in keys:
|
||||
raise InstallError('check.keys_no_header_key')
|
||||
total = 0
|
||||
for name, is_update in (('base', False), ('update', True)):
|
||||
path = Path(getattr(o, name))
|
||||
with path.open('rb') as f:
|
||||
total += self.open_input(f, path.stat().st_size, keys, name, is_update)
|
||||
del keys
|
||||
return total
|
||||
|
||||
@staticmethod
|
||||
def open_input(f, size, keys, name, is_update):
|
||||
try:
|
||||
kind, entries = container_entries(f, size)
|
||||
except ValueError:
|
||||
raise InstallError('check.not_game_file', input=name) from None
|
||||
if is_update and kind != 'nsp':
|
||||
raise InstallError('check.update_not_nsp')
|
||||
try:
|
||||
tickets = common_tickets(f, entries)
|
||||
except ValueError as e:
|
||||
if 'Personalized' in str(e):
|
||||
raise InstallError('check.ticket_personalized', input=name) from None
|
||||
raise InstallError('check.file_unreadable', input=name, detail=str(e)) from None
|
||||
try:
|
||||
records = nca_inventory(f, entries, keys)
|
||||
except ValueError as e:
|
||||
if 'NCA3' in str(e):
|
||||
raise InstallError('check.keys_wrong', input=name) from None
|
||||
raise InstallError('check.file_unreadable', input=name, detail=str(e)) from None
|
||||
try:
|
||||
record, header = program_nca(records, update=is_update)
|
||||
except ValueError as e:
|
||||
if is_update and 'BKTR' in str(e):
|
||||
raise InstallError('check.update_is_base') from None
|
||||
raise InstallError('check.update_not_totk' if is_update else 'check.base_not_totk', detail=str(e)) from None
|
||||
try:
|
||||
nca_ctr_key(header, keys, tickets)
|
||||
except ValueError as e:
|
||||
if 'ticket' in str(e):
|
||||
raise InstallError('check.ticket_missing', input=name) from None
|
||||
raise InstallError('check.keys_too_old', input=name,
|
||||
generation=max(header[0x206], header[0x220])) from None
|
||||
return record['size']
|
||||
|
||||
def space(self, ctx, ncas):
|
||||
dest_need = temp_need = 0
|
||||
if ctx.will_run('extract'):
|
||||
dest_need += expected_update()['romfs']['size'] + ncas
|
||||
romfs = ctx.dest / 'romfs' / 'romfs.bin'
|
||||
if romfs.is_file(): # a resumed extraction replaces it
|
||||
dest_need -= romfs.stat().st_size
|
||||
if ctx.will_run('prepare') or ctx.will_run('extract'):
|
||||
dest_need += DEST_EXTRA
|
||||
if ctx.will_run('lift') or ctx.will_run('compile'):
|
||||
temp_need += TEMP_NEED
|
||||
dest_free, dest_dev = free_bytes(ctx.dest)
|
||||
temp_free, temp_dev = free_bytes(ctx.temp)
|
||||
ctx.reporter.log(f'space: install folder needs {dest_need / GiB:.1f} GiB of {dest_free / GiB:.1f} free, '
|
||||
f'temp folder {temp_need / GiB:.1f} GiB of {temp_free / GiB:.1f}')
|
||||
if dest_dev == temp_dev:
|
||||
dest_need += temp_need
|
||||
temp_need = 0
|
||||
ctx.reporter.emit('space', dest=str(ctx.dest), need=dest_need, free=dest_free, temp=str(ctx.temp),
|
||||
temp_need=temp_need, temp_free=temp_free, same_disk=dest_dev == temp_dev)
|
||||
for path, need, free in ((ctx.dest, dest_need, dest_free), (ctx.temp, temp_need, temp_free)):
|
||||
if need > free:
|
||||
raise InstallError('check.disk_space', path=str(path), need_gb=round(need / 1e9, 1),
|
||||
free_gb=round(free / 1e9, 1))
|
||||
|
||||
def versions(self, ctx):
|
||||
"""Extract both ExeFS into the temp folder (verifying every block) and identify the versions by hash."""
|
||||
o, work = ctx.options, ctx.temp / 'check'
|
||||
logs = ctx.temp / 'logs'
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
versions = extract.load_expected()
|
||||
for name, is_update in (('base', False), ('update', True)):
|
||||
out = work / name
|
||||
remove(out)
|
||||
try:
|
||||
extract.extract_exefs(Path(getattr(o, name)), Path(o.keys), out, logs, update=is_update)
|
||||
except ValueError:
|
||||
raise InstallError('check.exefs_failed', input=name, log=str(logs / f'extract_exefs-{name}.log')) \
|
||||
from None
|
||||
problems = extract.exefs_problems(work / 'base' / 'exefs', versions['1.0.0'])
|
||||
if problems:
|
||||
raise InstallError('check.base_wrong_version', modules=', '.join(problems))
|
||||
try:
|
||||
found = extract.identify_update(work / 'update' / 'exefs', versions)['display_version']
|
||||
except ValueError:
|
||||
found = 'unknown'
|
||||
if found != UPDATE_VERSION:
|
||||
raise InstallError('check.update_wrong_version', found=found, wanted=UPDATE_VERSION)
|
||||
|
||||
|
||||
# ---- 2. extract
|
||||
|
||||
class Extract(Step):
|
||||
id = 'extract'
|
||||
|
||||
def key(self, ctx):
|
||||
# The output is whatever matches the expected hashes, whichever files and tools produced it.
|
||||
return fingerprint('extract-1', ctx.layout.expected)
|
||||
|
||||
def current(self, ctx, record):
|
||||
expected = expected_update()
|
||||
romfs = ctx.dest / 'romfs'
|
||||
if extract.exefs_problems(ctx.dest / 'exefs', expected) or not extract.romfs_verified(romfs, expected):
|
||||
return False
|
||||
st = (romfs / 'romfs.bin').stat()
|
||||
return st.st_size == record.get('romfs_size') and st.st_mtime_ns == record.get('romfs_mtime_ns')
|
||||
|
||||
def verify(self, ctx, record):
|
||||
image = ctx.dest / 'romfs' / 'romfs.bin'
|
||||
rep = ctx.reporter
|
||||
digest = file_sha256(image, lambda d, t: rep.progress('check', 'verify_romfs', d, t, 'bytes'))
|
||||
return digest == expected_update()['romfs']['sha256']
|
||||
|
||||
def run(self, ctx):
|
||||
o, dest, rep = ctx.options, ctx.dest, ctx.reporter
|
||||
expected = expected_update()
|
||||
base, update, keys = Path(o.base), Path(o.update), Path(o.keys)
|
||||
inputs = {}
|
||||
for name, path in (('base', base), ('update', update)):
|
||||
known = ctx.state['inputs'].get(name, {})
|
||||
st = path.stat()
|
||||
if known.get('size') == st.st_size and known.get('mtime_ns') == st.st_mtime_ns and known.get('sha256'):
|
||||
inputs[name] = known
|
||||
continue
|
||||
digest = file_sha256(path, lambda d, t, n=name: rep.progress('extract', 'hash_' + n, d, t, 'bytes'))
|
||||
inputs[name] = {'name': path.name, 'size': st.st_size, 'mtime_ns': st.st_mtime_ns, 'sha256': digest}
|
||||
ctx.state['inputs'] = inputs
|
||||
ctx.save()
|
||||
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
for stale in dest.glob('update-merge-*'): # a killed merge leaves its NCA copies (and key copies) behind
|
||||
remove(stale)
|
||||
rep.progress('extract', 'exefs', force=True)
|
||||
staged = ctx.temp / 'check' / 'update' / 'exefs'
|
||||
if not staged.is_dir():
|
||||
remove(ctx.temp / 'check' / 'update')
|
||||
try:
|
||||
extract.extract_exefs(update, keys, ctx.temp / 'check' / 'update', ctx.temp / 'logs', update=True)
|
||||
except ValueError:
|
||||
raise InstallError('check.exefs_failed', input='update',
|
||||
log=str(ctx.temp / 'logs' / 'extract_exefs-update.log')) from None
|
||||
remove(dest / 'exefs')
|
||||
shutil.move(staged, dest / 'exefs')
|
||||
problems = extract.exefs_problems(dest / 'exefs', expected)
|
||||
if problems:
|
||||
raise InstallError('extract.exefs_mismatch', modules=', '.join(problems))
|
||||
if extract.sha256(dest / 'exefs' / 'main.npdm') != expected['exefs_sha256']['main.npdm']:
|
||||
rep.warning('warn.npdm_differs')
|
||||
|
||||
romfs = dest / 'romfs'
|
||||
if self.id in ctx.forced:
|
||||
remove(romfs) # the full hash failed: index.json still claims the extraction's hash
|
||||
if not extract.romfs_verified(romfs, expected):
|
||||
self.merge(ctx, base, update, keys, romfs, expected)
|
||||
report = json.loads((romfs / 'index.json').read_text())
|
||||
if report['romfs_size'] != expected['romfs']['size'] or report['romfs_sha256'] != expected['romfs']['sha256']:
|
||||
raise InstallError('extract.romfs_mismatch', size=report['romfs_size'], sha256=report['romfs_sha256'][:16])
|
||||
(dest / 'profile').mkdir(exist_ok=True)
|
||||
st = (romfs / 'romfs.bin').stat()
|
||||
return {'romfs_size': st.st_size, 'romfs_mtime_ns': st.st_mtime_ns}
|
||||
|
||||
def merge(self, ctx, base, update, keys, romfs, expected):
|
||||
dest, logs = ctx.dest, ctx.temp / 'logs'
|
||||
ncas = extract.merge_space(base, update, keys)
|
||||
watch = Watch(ctx.reporter, 'extract')
|
||||
|
||||
def phase(name):
|
||||
if name == 'copy':
|
||||
watch.set('copy', lambda: size_of(dest.glob('update-merge-*/*.nca')), ncas)
|
||||
elif name == 'merge':
|
||||
watch.set('merge', lambda: size_of(dest.glob('update-merge-*/result/romfs.bin')),
|
||||
expected['romfs']['size'])
|
||||
else:
|
||||
watch.set(name)
|
||||
|
||||
watch.start()
|
||||
try:
|
||||
extract.merge_update_romfs(base, update, keys, ctx.layout.hactool, romfs, logs, phase=phase)
|
||||
except ValueError as e:
|
||||
raise InstallError('extract.romfs_merge_failed', detail=str(e), log=str(logs / 'merge_romfs-'
|
||||
f'{dest.name}.log')) from None
|
||||
finally:
|
||||
watch.stop()
|
||||
|
||||
|
||||
# ---- 3. lift
|
||||
|
||||
EMITTED = re.compile(r'^emitted (\d+)/(\d+) chunks')
|
||||
DISCOVERED = re.compile(r'^(\w+): \d+ functions \(.*discovery')
|
||||
|
||||
|
||||
class Lift(Step):
|
||||
id = 'lift'
|
||||
transient = True
|
||||
|
||||
def key(self, ctx):
|
||||
return fingerprint('lift-1', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches)
|
||||
|
||||
def current(self, ctx, record):
|
||||
gen = ctx.temp / 'gen'
|
||||
stamp = gen / STAMP
|
||||
return (gen / 'layout.json').is_file() and stamp.is_file() and stamp.read_text().strip() == ctx.keys['lift']
|
||||
|
||||
def run(self, ctx):
|
||||
layout, rep, gen = ctx.layout, ctx.reporter, ctx.temp / 'gen'
|
||||
remove(gen)
|
||||
rep.progress('lift', 'discover', force=True)
|
||||
|
||||
def parse(line):
|
||||
m = EMITTED.match(line)
|
||||
if m:
|
||||
rep.progress('lift', 'emit', int(m.group(1)), int(m.group(2)), 'chunks')
|
||||
|
||||
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py',
|
||||
['--exefs', ctx.dest / 'exefs', '--out', gen, '--patches', layout.patches], parse)
|
||||
if code:
|
||||
raise InstallError('lift.failed', code=code, log=str(log))
|
||||
(gen / STAMP).write_text(ctx.keys['lift'] + '\n')
|
||||
return {}
|
||||
|
||||
|
||||
# ---- 4. compile
|
||||
|
||||
NINJA_LINE = re.compile(r'^\[(\d+)/(\d+)\] ')
|
||||
|
||||
|
||||
class Compile(Step):
|
||||
id = 'compile'
|
||||
|
||||
def key(self, ctx):
|
||||
layout = ctx.layout
|
||||
clang = subprocess.run([str(layout.clang), '--version'], capture_output=True, text=True).stdout
|
||||
tools = [layout.tools / n for n in ('build_game.py', 'romfs_index.py', 'check_content.py')]
|
||||
return fingerprint('compile-1', ctx.keys['lift'], layout.runtime, layout.runtime_lib, layout.deps, *tools,
|
||||
clang, *[d.name for d in layout.dlls])
|
||||
|
||||
def current(self, ctx, record):
|
||||
exe = ctx.dest / ('totk' + EXE)
|
||||
if not exe.is_file() or not (ctx.dest / 'romfs-index.bin').is_file():
|
||||
return False
|
||||
if not all((ctx.dest / d.name).is_file() for d in ctx.layout.dlls):
|
||||
return False
|
||||
return file_sha256(exe) == record.get('exe_sha256')
|
||||
|
||||
def run(self, ctx):
|
||||
layout, rep, dest = ctx.layout, ctx.reporter, ctx.dest
|
||||
build = ctx.temp / 'build'
|
||||
phase = ['compile']
|
||||
|
||||
def parse(line):
|
||||
m = NINJA_LINE.match(line)
|
||||
if m:
|
||||
rep.progress('compile', phase[0], int(m.group(1)), int(m.group(2)), 'objects')
|
||||
elif line.startswith('compile: '):
|
||||
phase[0] = 'link'
|
||||
rep.progress('compile', 'link', force=True)
|
||||
|
||||
rep.progress('compile', 'compile', force=True)
|
||||
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
|
||||
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--deps', layout.deps,
|
||||
'--runtime', layout.runtime, '--runtime-lib', layout.runtime_lib, '--ninja', layout.ninja,
|
||||
'--expected', layout.expected, '--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
||||
'--profile', dest / 'profile'], parse)
|
||||
exe = build / ('totk' + EXE)
|
||||
if code or not exe.is_file():
|
||||
raise InstallError('compile.failed', code=code, log=str(log))
|
||||
try:
|
||||
replace_file(exe, dest / exe.name)
|
||||
except PermissionError:
|
||||
raise InstallError('compile.exe_in_use', path=str(dest / exe.name)) from None
|
||||
replace_file(build / 'romfs-index.bin', dest / 'romfs-index.bin')
|
||||
for dll in layout.dlls:
|
||||
replace_file(dll, dest / dll.name)
|
||||
return {'exe_sha256': file_sha256(dest / exe.name)}
|
||||
|
||||
|
||||
# ---- 5. prepare
|
||||
|
||||
TRANSLATED = re.compile(r'gfx_shader: prepare: (\d+) of (\d+) pairs')
|
||||
|
||||
|
||||
class Prepare(Step):
|
||||
id = 'prepare'
|
||||
|
||||
def key(self, ctx):
|
||||
layout = ctx.layout
|
||||
xlate_id = layout.xlate.with_suffix('.id')
|
||||
tools = [layout.tools / n for n in ('prepare_shaders.py', 'shader_census.py', 'make_romfs_patches.py')]
|
||||
return fingerprint('prepare-1', ctx.keys['compile'], layout.xlate, xlate_id, shader_list_text(layout), *tools)
|
||||
|
||||
def current(self, ctx, record):
|
||||
dest = ctx.dest
|
||||
xlate = dest / ctx.layout.xlate.name
|
||||
return ((dest / 'shader-prepared.bin').is_file() and (dest / 'romfs-patches' / 'manifest.tsv').is_file()
|
||||
and xlate.is_file() and file_sha256(xlate) == record.get('xlate_sha256'))
|
||||
|
||||
def run(self, ctx):
|
||||
layout, rep, dest = ctx.layout, ctx.reporter, ctx.dest
|
||||
xlate = dest / layout.xlate.name
|
||||
if xlate.is_file() and file_sha256(xlate) != file_sha256(layout.xlate):
|
||||
# Another translator: its cache keys differ, so the old translations would only take space.
|
||||
for name in ('shader-cache', 'shader-prepared.bin'):
|
||||
remove(dest / name)
|
||||
replace_file(layout.xlate, xlate)
|
||||
xlate_id = layout.xlate.with_suffix('.id')
|
||||
if xlate_id.is_file():
|
||||
replace_file(xlate_id, xlate.with_suffix('.id'))
|
||||
else:
|
||||
remove(xlate.with_suffix('.id'))
|
||||
|
||||
rep.progress('prepare', 'romfs_patches', force=True)
|
||||
remove(dest / 'romfs-patches')
|
||||
code, log = run_tool(ctx, 'prepare', layout.tools / 'make_romfs_patches.py',
|
||||
['--romfs', dest / 'romfs', '--out', dest / 'romfs-patches'])
|
||||
if code:
|
||||
raise InstallError('prepare.romfs_patches_failed', code=code, log=str(log))
|
||||
|
||||
shader_list = layout.shader_list
|
||||
if not shader_list:
|
||||
shader_list = ctx.temp / 'shader_list.tsv'
|
||||
shader_list.write_text(shader_list_text(layout), encoding='utf-8', newline='\n')
|
||||
|
||||
def parse(line):
|
||||
m = TRANSLATED.search(line)
|
||||
if m:
|
||||
rep.progress('prepare', 'translate', int(m.group(1)), int(m.group(2)), 'shaders')
|
||||
elif 'listed variants' in line:
|
||||
rep.progress('prepare', 'translate', force=True)
|
||||
elif 'gfx_shader: prepare:' in line and 'translated' in line and 'programs' in line:
|
||||
rep.progress('prepare', 'libraries', force=True)
|
||||
|
||||
rep.progress('prepare', 'seed', force=True)
|
||||
code, log = run_tool(ctx, 'prepare', layout.tools / 'prepare_shaders.py',
|
||||
[dest, '--romfs', dest / 'romfs', '--exefs', dest / 'exefs', '--list', shader_list], parse)
|
||||
if code or not (dest / 'shader-prepared.bin').is_file():
|
||||
raise InstallError('prepare.shaders_failed', code=code, log=str(log))
|
||||
return {'xlate_sha256': file_sha256(xlate)}
|
||||
|
||||
|
||||
# ---- 6. clean up
|
||||
|
||||
class Cleanup(Step):
|
||||
id = 'cleanup'
|
||||
always = True
|
||||
|
||||
def run(self, ctx):
|
||||
rep, dest = ctx.reporter, ctx.dest
|
||||
rep.progress('cleanup', 'ini', force=True)
|
||||
try:
|
||||
ini.write(dest, ctx.lang, ctx.layout.ini_example)
|
||||
except OSError as e:
|
||||
raise InstallError('cleanup.ini_failed', detail=str(e)) from None
|
||||
ctx.state['lang'] = ctx.lang
|
||||
(dest / 'profile').mkdir(exist_ok=True)
|
||||
rep.progress('cleanup', 'temp', force=True)
|
||||
for stale in dest.glob('update-merge-*'):
|
||||
remove(stale)
|
||||
for name in ('gen', 'build', 'check', 'shader_list.tsv'):
|
||||
remove(ctx.temp / name)
|
||||
return {}
|
||||
|
||||
|
||||
def all_steps():
|
||||
return [Check(), Extract(), Lift(), Compile(), Prepare(), Cleanup()]
|
||||
@@ -0,0 +1,170 @@
|
||||
{
|
||||
"step.check": "Checking the computer and the game files",
|
||||
"step.extract": "Extracting the game",
|
||||
"step.lift": "Translating the game code to C",
|
||||
"step.compile": "Compiling totk.exe",
|
||||
"step.prepare": "Preparing the shaders",
|
||||
"step.cleanup": "Cleaning up",
|
||||
|
||||
"phase.check.host": "Checking the processor and the graphics card",
|
||||
"phase.check.verify": "Checking the files already installed",
|
||||
"phase.check.verify_romfs": "Checking the installed game data",
|
||||
"phase.check.inputs": "Opening the game files with the keys",
|
||||
"phase.check.versions": "Checking the game versions",
|
||||
"phase.extract.hash_base": "Computing the checksum of the base game",
|
||||
"phase.extract.hash_update": "Computing the checksum of the update",
|
||||
"phase.extract.exefs": "Extracting the game code",
|
||||
"phase.extract.copy": "Copying the game data",
|
||||
"phase.extract.merge": "Applying the update to the game data",
|
||||
"phase.extract.index": "Verifying the game data",
|
||||
"phase.lift.discover": "Finding the functions",
|
||||
"phase.lift.emit": "Writing C code",
|
||||
"phase.compile.compile": "Compiling",
|
||||
"phase.compile.link": "Linking totk.exe",
|
||||
"phase.prepare.romfs_patches": "Writing the 60 fps game data patches",
|
||||
"phase.prepare.seed": "Reading the shaders from the game data",
|
||||
"phase.prepare.translate": "Translating the shaders",
|
||||
"phase.prepare.libraries": "Building the shaders for the graphics driver",
|
||||
"phase.cleanup.ini": "Writing totk.ini",
|
||||
"phase.cleanup.temp": "Removing temporary files",
|
||||
|
||||
"cli.usage": "Invalid command line: {detail}",
|
||||
"internal": "Unexpected error ({exception}): {detail}",
|
||||
"state.unreadable": "{path} cannot be read. Choose another folder or delete that file.",
|
||||
"state.newer_format": "{path} was written by a newer installer. Use that installer or another folder.",
|
||||
"state.output_changed": "Installed files of step '{step}' changed since they were made; that step runs again.",
|
||||
|
||||
"check.release_incomplete": "The installer folder is incomplete; missing: {paths}. Unpack the release again.",
|
||||
"check.dest_not_folder": "The install location {path} is a file, not a folder.",
|
||||
"check.dest_not_empty": "The install folder {path} is not empty and holds no earlier installation. Choose an empty folder.",
|
||||
"check.lang_missing": "Choose the game language.",
|
||||
"check.lang_unknown": "Unknown language '{lang}'. Known: {known}.",
|
||||
"check.toolchain_broken": "The bundled compiler does not work: {detail}",
|
||||
"check.host_probe_failed": "The computer check did not run (exit code {code}).",
|
||||
"check.cpu_no_avx2": "This processor lacks AVX2 (x86-64-v3), which the game needs. Intel Haswell (2013) or AMD Excavator / Zen and newer have it.",
|
||||
"check.vulkan_missing": "No Vulkan driver is installed. Install the latest driver for your graphics card.",
|
||||
"check.vulkan_no_device": "Vulkan finds no graphics card. Install the latest driver for your graphics card.",
|
||||
"check.vulkan_too_old": "The game needs a graphics card with Vulkan 1.3; found: {devices}. Update the graphics driver.",
|
||||
"check.inputs_needed": "The game files are needed: choose the base game, the 1.2.1 update and prod.keys.",
|
||||
"check.input_missing": "File not found ({input}): {path}",
|
||||
"check.keys_unreadable": "The keys file cannot be read. Choose a prod.keys file.",
|
||||
"check.keys_no_header_key": "The keys file has no header_key. Dump prod.keys from your console again.",
|
||||
"check.keys_wrong": "The keys cannot open the {input} file. Use the prod.keys of your console.",
|
||||
"check.keys_too_old": "The keys file is too old for the {input} file (needs key generation {generation}). Dump prod.keys from a console with newer firmware.",
|
||||
"check.not_game_file": "The {input} file is not an XCI or NSP game file.",
|
||||
"check.file_unreadable": "The {input} file cannot be read: {detail}",
|
||||
"check.ticket_personalized": "The {input} file carries a personalized ticket; a common ticket is needed.",
|
||||
"check.ticket_missing": "The {input} file lacks the ticket its content needs.",
|
||||
"check.base_not_totk": "The base game file is not The Legend of Zelda: Tears of the Kingdom ({detail}).",
|
||||
"check.update_not_nsp": "The update must be an NSP file.",
|
||||
"check.update_is_base": "The update file holds a base game. Choose the 1.2.1 update NSP.",
|
||||
"check.update_not_totk": "The update file is not a Tears of the Kingdom update ({detail}).",
|
||||
"check.exefs_failed": "The game code of the {input} file failed its checks; see {log}.",
|
||||
"check.base_wrong_version": "The base game is not version 1.0.0 ({modules} differ).",
|
||||
"check.update_wrong_version": "The update is version {found}; this release needs {wanted}.",
|
||||
"check.disk_space": "Not enough free space for {path}: {need_gb} GB needed, {free_gb} GB free.",
|
||||
|
||||
"extract.exefs_mismatch": "The extracted game code does not match version 1.2.1 ({modules}).",
|
||||
"extract.romfs_merge_failed": "Applying the update to the game data failed: {detail}. Details: {log}",
|
||||
"extract.romfs_mismatch": "The extracted game data does not match version 1.2.1 (size {size}, checksum {sha256}...).",
|
||||
"lift.failed": "Translating the game code failed (exit code {code}). Details: {log}",
|
||||
"compile.failed": "Compiling totk.exe failed (exit code {code}). Details: {log}",
|
||||
"compile.exe_in_use": "{path} is in use. Close the game and run the installer again.",
|
||||
"prepare.shader_list_missing": "The shader list is missing.",
|
||||
"prepare.romfs_patches_failed": "Writing the game data patches failed (exit code {code}). Details: {log}",
|
||||
"prepare.shaders_failed": "Preparing the shaders failed (exit code {code}). Details: {log}",
|
||||
"cleanup.ini_failed": "totk.ini cannot be written: {detail}",
|
||||
|
||||
"warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.",
|
||||
"warn.npdm_differs": "main.npdm differs from the reference 1.2.1 dump; only the code modules are used.",
|
||||
|
||||
"ui.title": "TotK Setup",
|
||||
"ui.intro": "Builds a playable Tears of the Kingdom folder from your own game files. This takes a while and runs once.",
|
||||
"ui.notice": "The folder this setup builds contains the game. It is for your own use only; do not share or upload it.",
|
||||
"ui.keys_private": "The keys file is read in place. Its contents are never shown, copied or stored.",
|
||||
"ui.base": "Base game (NSP or XCI)",
|
||||
"ui.update": "Update 1.2.1 (NSP)",
|
||||
"ui.keys": "prod.keys",
|
||||
"ui.game_language": "Game language",
|
||||
"ui.dest": "Install folder",
|
||||
"ui.browse": "Browse...",
|
||||
"ui.pick_game": "Choose the base game file",
|
||||
"ui.pick_update": "Choose the 1.2.1 update file",
|
||||
"ui.pick_keys": "Choose prod.keys",
|
||||
"ui.pick_dest": "Choose the install folder",
|
||||
"ui.type_game": "Game files (NSP, XCI)",
|
||||
"ui.type_nsp": "NSP files",
|
||||
"ui.type_keys": "Key files (*.keys)",
|
||||
"ui.type_all": "All files",
|
||||
"ui.start": "Start",
|
||||
"ui.cancel": "Cancel",
|
||||
"ui.cancelling": "Cancelling...",
|
||||
"ui.show_log": "Show log",
|
||||
"ui.hide_log": "Hide log",
|
||||
"ui.close": "Close",
|
||||
"ui.back": "Back",
|
||||
"ui.open_folder": "Open folder",
|
||||
"ui.update_mode": "This folder holds an earlier installation. The setup continues or updates it; the game files are asked for only if they are needed again.",
|
||||
"ui.need_dest": "Choose the install folder.",
|
||||
"ui.need_base": "Choose the base game file.",
|
||||
"ui.need_update": "Choose the 1.2.1 update file.",
|
||||
"ui.need_keys": "Choose the prod.keys file.",
|
||||
"ui.need_lang": "Choose the game language.",
|
||||
"ui.missing_base": "The base game file does not exist.",
|
||||
"ui.missing_update": "The update file does not exist.",
|
||||
"ui.missing_keys": "The keys file does not exist.",
|
||||
"ui.running_install": "Installing",
|
||||
"ui.running_resume": "Continuing the earlier setup",
|
||||
"ui.running_update": "Updating the installation",
|
||||
"ui.state_started": "started",
|
||||
"ui.state_done": "done",
|
||||
"ui.state_skipped": "skipped",
|
||||
"ui.state_pending": "waiting",
|
||||
"ui.state_running": "running",
|
||||
"ui.state_failed": "failed",
|
||||
"ui.log_warning": "Warning",
|
||||
"ui.log_error": "Error",
|
||||
"ui.log_space": "Space needed: {need} GB, free: {free} GB",
|
||||
"ui.log_end": "Finished",
|
||||
"ui.result_ok": "The game is ready",
|
||||
"ui.result_failed": "The setup failed",
|
||||
"ui.result_cancelled": "The setup was cancelled",
|
||||
"ui.done_text": "Start totk.exe from {dest}. The setup program and the game files you chose are no longer needed.",
|
||||
"ui.cancelled_text": "The finished steps are kept. Start the setup again with the same folder to continue.",
|
||||
"ui.log_file": "Log file: {path}",
|
||||
"ui.process_died": "The setup stopped unexpectedly (exit code {code}).",
|
||||
"ui.confirm_cancel": "The setup is running. Cancel it and close the window?",
|
||||
"ui.space_one": "Free space needed on {path}: {need} GB (free: {free} GB).",
|
||||
"ui.space_two": "Free space needed on {path}: {need} GB (free: {free} GB); on {temp}: {temp_need} GB (free: {temp_free} GB).",
|
||||
"ui.amount_bytes": "{done} / {total} GB",
|
||||
"ui.amount_chunks": "{done} / {total} parts",
|
||||
"ui.amount_objects": "{done} / {total} files",
|
||||
"ui.amount_shaders": "{done} / {total} shaders",
|
||||
"ui.amount_other": "{done} / {total} {unit}",
|
||||
"ui.seconds": "{n} s",
|
||||
"param.input.base": "base game",
|
||||
"param.input.update": "update",
|
||||
"param.input.keys": "keys",
|
||||
|
||||
"launcher.title": "TotK Setup",
|
||||
"launcher.python_missing": "The setup files are incomplete: the python folder next to TotK-Setup.exe is missing. Unpack the whole archive again and start TotK-Setup.exe from the unpacked folder.",
|
||||
|
||||
"lang.ja": "Japanese",
|
||||
"lang.en-US": "English (US)",
|
||||
"lang.en-GB": "English (UK)",
|
||||
"lang.fr": "French",
|
||||
"lang.fr-CA": "French (Canada)",
|
||||
"lang.de": "German",
|
||||
"lang.it": "Italian",
|
||||
"lang.es": "Spanish",
|
||||
"lang.es-419": "Spanish (Latin America)",
|
||||
"lang.nl": "Dutch",
|
||||
"lang.pt": "Portuguese",
|
||||
"lang.pt-BR": "Portuguese (Brazil)",
|
||||
"lang.ru": "Russian",
|
||||
"lang.ko": "Korean",
|
||||
"lang.zh-CN": "Chinese (Simplified, China)",
|
||||
"lang.zh-TW": "Chinese (Traditional, Taiwan)",
|
||||
"lang.zh-Hans": "Chinese (Simplified)",
|
||||
"lang.zh-Hant": "Chinese (Traditional)"
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
{
|
||||
"step.check": "Проверка компьютера и игровых файлов",
|
||||
"step.extract": "Распаковка игры",
|
||||
"step.lift": "Перевод кода игры в C",
|
||||
"step.compile": "Сборка totk.exe",
|
||||
"step.prepare": "Подготовка шейдеров",
|
||||
"step.cleanup": "Уборка",
|
||||
|
||||
"phase.check.host": "Проверка процессора и видеокарты",
|
||||
"phase.check.verify": "Проверка уже установленных файлов",
|
||||
"phase.check.verify_romfs": "Проверка установленных данных игры",
|
||||
"phase.check.inputs": "Открытие игровых файлов с помощью ключей",
|
||||
"phase.check.versions": "Проверка версий игры",
|
||||
"phase.extract.hash_base": "Подсчёт контрольной суммы базовой игры",
|
||||
"phase.extract.hash_update": "Подсчёт контрольной суммы обновления",
|
||||
"phase.extract.exefs": "Распаковка кода игры",
|
||||
"phase.extract.copy": "Копирование данных игры",
|
||||
"phase.extract.merge": "Применение обновления к данным игры",
|
||||
"phase.extract.index": "Проверка данных игры",
|
||||
"phase.lift.discover": "Поиск функций",
|
||||
"phase.lift.emit": "Запись кода на C",
|
||||
"phase.compile.compile": "Компиляция",
|
||||
"phase.compile.link": "Компоновка totk.exe",
|
||||
"phase.prepare.romfs_patches": "Запись патчей данных игры для 60 кадров/с",
|
||||
"phase.prepare.seed": "Чтение шейдеров из данных игры",
|
||||
"phase.prepare.translate": "Трансляция шейдеров",
|
||||
"phase.prepare.libraries": "Сборка шейдеров для драйвера видеокарты",
|
||||
"phase.cleanup.ini": "Запись totk.ini",
|
||||
"phase.cleanup.temp": "Удаление временных файлов",
|
||||
|
||||
"cli.usage": "Неверные параметры командной строки: {detail}",
|
||||
"internal": "Непредвиденная ошибка ({exception}): {detail}",
|
||||
"state.unreadable": "Не удаётся прочитать {path}. Выберите другую папку или удалите этот файл.",
|
||||
"state.newer_format": "Файл {path} создан более новой версией установки. Используйте её или выберите другую папку.",
|
||||
"state.output_changed": "Установленные файлы шага «{step}» изменились после создания; этот шаг выполнится заново.",
|
||||
|
||||
"check.release_incomplete": "Папка установки неполная, не хватает: {paths}. Распакуйте архив заново.",
|
||||
"check.dest_not_folder": "Место установки {path} — это файл, а не папка.",
|
||||
"check.dest_not_empty": "Папка установки {path} не пуста и не содержит прежней установки. Выберите пустую папку.",
|
||||
"check.lang_missing": "Выберите язык игры.",
|
||||
"check.lang_unknown": "Неизвестный язык «{lang}». Доступные: {known}.",
|
||||
"check.toolchain_broken": "Встроенный компилятор не работает: {detail}",
|
||||
"check.host_probe_failed": "Проверка компьютера не выполнилась (код выхода {code}).",
|
||||
"check.cpu_no_avx2": "Этому процессору не хватает AVX2 (x86-64-v3), который нужен игре. AVX2 есть у Intel Haswell (2013) и новее, у AMD Excavator и Zen и новее.",
|
||||
"check.vulkan_missing": "Драйвер Vulkan не установлен. Установите свежий драйвер видеокарты.",
|
||||
"check.vulkan_no_device": "Vulkan не находит видеокарту. Установите свежий драйвер видеокарты.",
|
||||
"check.vulkan_too_old": "Игре нужна видеокарта с Vulkan 1.3; найдено: {devices}. Обновите драйвер видеокарты.",
|
||||
"check.inputs_needed": "Нужны игровые файлы: выберите базовую игру, обновление 1.2.1 и prod.keys.",
|
||||
"check.input_missing": "Файл не найден ({input}): {path}",
|
||||
"check.keys_unreadable": "Не удаётся прочитать файл ключей. Выберите файл prod.keys.",
|
||||
"check.keys_no_header_key": "В файле ключей нет header_key. Снова получите prod.keys со своей консоли.",
|
||||
"check.keys_wrong": "Эти ключи не открывают файл «{input}». Используйте prod.keys со своей консоли.",
|
||||
"check.keys_too_old": "Файл ключей слишком старый для файла «{input}» (нужно поколение ключей {generation}). Получите prod.keys с консоли с более новой прошивкой.",
|
||||
"check.not_game_file": "Файл «{input}» не является игровым файлом XCI или NSP.",
|
||||
"check.file_unreadable": "Не удаётся прочитать файл «{input}»: {detail}",
|
||||
"check.ticket_personalized": "В файле «{input}» личный тикет; нужен общий тикет.",
|
||||
"check.ticket_missing": "В файле «{input}» нет тикета, который нужен его содержимому.",
|
||||
"check.base_not_totk": "Файл базовой игры — не The Legend of Zelda: Tears of the Kingdom ({detail}).",
|
||||
"check.update_not_nsp": "Обновление должно быть файлом NSP.",
|
||||
"check.update_is_base": "Файл обновления содержит базовую игру. Выберите NSP с обновлением 1.2.1.",
|
||||
"check.update_not_totk": "Файл обновления — не обновление Tears of the Kingdom ({detail}).",
|
||||
"check.exefs_failed": "Код игры из файла «{input}» не прошёл проверку; подробности в {log}.",
|
||||
"check.base_wrong_version": "Базовая игра не версии 1.0.0 (отличаются модули: {modules}).",
|
||||
"check.update_wrong_version": "Версия обновления {found}; этому выпуску нужна {wanted}.",
|
||||
"check.disk_space": "Не хватает свободного места для {path}: нужно {need_gb} ГБ, свободно {free_gb} ГБ.",
|
||||
|
||||
"extract.exefs_mismatch": "Распакованный код игры не соответствует версии 1.2.1 ({modules}).",
|
||||
"extract.romfs_merge_failed": "Не удалось применить обновление к данным игры: {detail}. Подробности: {log}",
|
||||
"extract.romfs_mismatch": "Распакованные данные игры не соответствуют версии 1.2.1 (размер {size}, контрольная сумма {sha256}...).",
|
||||
"lift.failed": "Не удалось перевести код игры (код выхода {code}). Подробности: {log}",
|
||||
"compile.failed": "Не удалось собрать totk.exe (код выхода {code}). Подробности: {log}",
|
||||
"compile.exe_in_use": "Файл {path} занят. Закройте игру и запустите установку снова.",
|
||||
"prepare.shader_list_missing": "Нет списка шейдеров.",
|
||||
"prepare.romfs_patches_failed": "Не удалось записать патчи данных игры (код выхода {code}). Подробности: {log}",
|
||||
"prepare.shaders_failed": "Не удалось подготовить шейдеры (код выхода {code}). Подробности: {log}",
|
||||
"cleanup.ini_failed": "Не удаётся записать totk.ini: {detail}",
|
||||
|
||||
"warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.",
|
||||
"warn.npdm_differs": "main.npdm отличается от эталонного дампа 1.2.1; используются только модули с кодом.",
|
||||
|
||||
"ui.title": "Установка TotK",
|
||||
"ui.intro": "Собирает готовую к запуску папку с Tears of the Kingdom из ваших игровых файлов. Занимает много времени и выполняется один раз.",
|
||||
"ui.notice": "Папка, которую создаёт установка, содержит игру. Она предназначена только для личного использования: не передавайте и не выкладывайте её.",
|
||||
"ui.keys_private": "Файл ключей читается на месте. Его содержимое не показывается, не копируется и не сохраняется.",
|
||||
"ui.base": "Базовая игра (NSP или XCI)",
|
||||
"ui.update": "Обновление 1.2.1 (NSP)",
|
||||
"ui.keys": "prod.keys",
|
||||
"ui.game_language": "Язык игры",
|
||||
"ui.dest": "Папка установки",
|
||||
"ui.browse": "Обзор...",
|
||||
"ui.pick_game": "Выберите файл базовой игры",
|
||||
"ui.pick_update": "Выберите файл обновления 1.2.1",
|
||||
"ui.pick_keys": "Выберите prod.keys",
|
||||
"ui.pick_dest": "Выберите папку установки",
|
||||
"ui.type_game": "Файлы игры (NSP, XCI)",
|
||||
"ui.type_nsp": "Файлы NSP",
|
||||
"ui.type_keys": "Файлы ключей (*.keys)",
|
||||
"ui.type_all": "Все файлы",
|
||||
"ui.start": "Начать",
|
||||
"ui.cancel": "Отмена",
|
||||
"ui.cancelling": "Отмена...",
|
||||
"ui.show_log": "Показать журнал",
|
||||
"ui.hide_log": "Скрыть журнал",
|
||||
"ui.close": "Закрыть",
|
||||
"ui.back": "Назад",
|
||||
"ui.open_folder": "Открыть папку",
|
||||
"ui.update_mode": "В этой папке уже есть установка. Программа продолжит или обновит её; файлы игры понадобятся, только если их снова придётся читать.",
|
||||
"ui.need_dest": "Выберите папку установки.",
|
||||
"ui.need_base": "Выберите файл базовой игры.",
|
||||
"ui.need_update": "Выберите файл обновления 1.2.1.",
|
||||
"ui.need_keys": "Выберите файл prod.keys.",
|
||||
"ui.need_lang": "Выберите язык игры.",
|
||||
"ui.missing_base": "Файл базовой игры не найден.",
|
||||
"ui.missing_update": "Файл обновления не найден.",
|
||||
"ui.missing_keys": "Файл ключей не найден.",
|
||||
"ui.running_install": "Установка",
|
||||
"ui.running_resume": "Продолжение прерванной установки",
|
||||
"ui.running_update": "Обновление установки",
|
||||
"ui.state_started": "начат",
|
||||
"ui.state_done": "готово",
|
||||
"ui.state_skipped": "пропущен",
|
||||
"ui.state_pending": "ожидает",
|
||||
"ui.state_running": "выполняется",
|
||||
"ui.state_failed": "ошибка",
|
||||
"ui.log_warning": "Предупреждение",
|
||||
"ui.log_error": "Ошибка",
|
||||
"ui.log_space": "Нужно места: {need} ГБ, свободно: {free} ГБ",
|
||||
"ui.log_end": "Завершено",
|
||||
"ui.result_ok": "Игра готова",
|
||||
"ui.result_failed": "Установка не удалась",
|
||||
"ui.result_cancelled": "Установка отменена",
|
||||
"ui.done_text": "Запустите totk.exe из папки {dest}. Программа установки и выбранные игровые файлы больше не нужны.",
|
||||
"ui.cancelled_text": "Завершённые шаги сохранены. Чтобы продолжить, снова запустите установку с той же папкой.",
|
||||
"ui.log_file": "Файл журнала: {path}",
|
||||
"ui.process_died": "Установка неожиданно остановилась (код выхода {code}).",
|
||||
"ui.confirm_cancel": "Установка выполняется. Отменить её и закрыть окно?",
|
||||
"ui.space_one": "Нужно свободного места на {path}: {need} ГБ (свободно: {free} ГБ).",
|
||||
"ui.space_two": "Нужно свободного места на {path}: {need} ГБ (свободно: {free} ГБ); на {temp}: {temp_need} ГБ (свободно: {temp_free} ГБ).",
|
||||
"ui.amount_bytes": "{done} / {total} ГБ",
|
||||
"ui.amount_chunks": "{done} / {total} частей",
|
||||
"ui.amount_objects": "{done} / {total} файлов",
|
||||
"ui.amount_shaders": "{done} / {total} шейдеров",
|
||||
"ui.amount_other": "{done} / {total} {unit}",
|
||||
"ui.seconds": "{n} с",
|
||||
"param.input.base": "базовая игра",
|
||||
"param.input.update": "обновление",
|
||||
"param.input.keys": "ключи",
|
||||
|
||||
"launcher.title": "Установка TotK",
|
||||
"launcher.python_missing": "Файлы установки неполные: рядом с TotK-Setup.exe нет папки python. Распакуйте архив целиком заново и запустите TotK-Setup.exe из распакованной папки.",
|
||||
|
||||
"lang.ja": "Японский",
|
||||
"lang.en-US": "Английский (США)",
|
||||
"lang.en-GB": "Английский (Великобритания)",
|
||||
"lang.fr": "Французский",
|
||||
"lang.fr-CA": "Французский (Канада)",
|
||||
"lang.de": "Немецкий",
|
||||
"lang.it": "Итальянский",
|
||||
"lang.es": "Испанский",
|
||||
"lang.es-419": "Испанский (Латинская Америка)",
|
||||
"lang.nl": "Нидерландский",
|
||||
"lang.pt": "Португальский",
|
||||
"lang.pt-BR": "Португальский (Бразилия)",
|
||||
"lang.ru": "Русский",
|
||||
"lang.ko": "Корейский",
|
||||
"lang.zh-CN": "Китайский (упрощённый, КНР)",
|
||||
"lang.zh-TW": "Китайский (традиционный, Тайвань)",
|
||||
"lang.zh-Hans": "Китайский (упрощённый)",
|
||||
"lang.zh-Hant": "Китайский (традиционный)"
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
"""End-to-end run of the installer in dev mode with the game files named in config.local.toml.
|
||||
|
||||
Reads input.base, the 1.2.1 entry of input.updates (the one build/extracted/v121 was extracted from, else the only
|
||||
one) and input.keys, and passes them to `python -m installer` started the way the release launcher starts it: the
|
||||
embeddable Python (build/py314-embed) with this checkout on sys.path. The paths are never printed. Events go to
|
||||
<events> (JSON lines); stdout gets each step's state and time, warnings and errors.
|
||||
|
||||
python tests/installer_e2e.py --dest build/install-test --lang en-US [--events FILE] [--no-inputs] [-- extra args]
|
||||
|
||||
--no-inputs leaves the game files out (update mode on an existing folder).
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
EMBED = ROOT / 'build' / 'py314-embed' / 'py' / 'python.exe'
|
||||
BOOT = ('import runpy, sys; sys.path.insert(0, sys.argv.pop(1)); '
|
||||
"runpy.run_module('installer', run_name='__main__', alter_sys=True)")
|
||||
|
||||
|
||||
def game_inputs():
|
||||
config = tomllib.loads((ROOT / 'config.local.toml').read_text(encoding='utf-8'))['input']
|
||||
updates = [Path(u) for u in config.get('updates', [])]
|
||||
manifest = ROOT / 'build' / 'extracted' / 'v121' / 'extraction-manifest.json'
|
||||
update = None
|
||||
if manifest.is_file():
|
||||
source = json.loads(manifest.read_text())['source']
|
||||
update = next((u for u in updates if str(u.resolve()) == source), None)
|
||||
if update is None and len(updates) == 1:
|
||||
update = updates[0]
|
||||
if update is None:
|
||||
sys.exit('installer_e2e: cannot tell which input.updates entry is 1.2.1')
|
||||
return ['--base', config['base'], '--update', str(update), '--keys', config['keys']]
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--dest', type=Path, required=True)
|
||||
ap.add_argument('--lang')
|
||||
ap.add_argument('--events', type=Path)
|
||||
ap.add_argument('--no-inputs', action='store_true')
|
||||
ap.add_argument('extra', nargs='*')
|
||||
a = ap.parse_args()
|
||||
python = EMBED if EMBED.is_file() else Path(sys.executable)
|
||||
command = [str(python), '-c', BOOT, str(ROOT), '--dest', str(a.dest.resolve())]
|
||||
if a.lang:
|
||||
command += ['--lang', a.lang]
|
||||
if not a.no_inputs:
|
||||
command += game_inputs()
|
||||
command += a.extra
|
||||
events = a.events or a.dest.resolve().parent / f'{a.dest.name}-events.jsonl'
|
||||
with open(events, 'w', encoding='utf-8') as log:
|
||||
proc = subprocess.Popen(command, stdout=subprocess.PIPE, text=True, encoding='utf-8', cwd=ROOT)
|
||||
for line in proc.stdout:
|
||||
log.write(line)
|
||||
log.flush()
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except ValueError:
|
||||
print('non-JSON line on stdout:', line.rstrip())
|
||||
continue
|
||||
if e['type'] == 'step' and e['state'] != 'started':
|
||||
extra = f'{e["seconds"]:.1f} s' if 'seconds' in e else e.get('reason', '')
|
||||
print(f'[{e["t"]:8.1f}] {e["step"]:8} {e["state"]} {extra}', flush=True)
|
||||
elif e['type'] == 'step':
|
||||
print(f'[{e["t"]:8.1f}] {e["step"]:8} started', flush=True)
|
||||
elif e['type'] in ('warning', 'error', 'plan', 'end'):
|
||||
print(f'[{e["t"]:8.1f}] {json.dumps({k: v for k, v in e.items() if k != "t"})}', flush=True)
|
||||
elif e['type'] == 'start':
|
||||
print(f'[{e["t"]:8.1f}] start mode={e["mode"]} release={e["release"]}', flush=True)
|
||||
return proc.wait()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,394 @@
|
||||
"""The installer window: event model, form, catalogs, cancel, a self-closing Tk run and one end-to-end run.
|
||||
|
||||
.venv-py314\\Scripts\\python tests\\installer_gui_test.py
|
||||
|
||||
The end-to-end test runs the real CLI in --verify mode on a finished install (every step skips) and folds its events
|
||||
through the same code the window uses. The install is build/inst-gui/e2e-install: hard links to build/install-test, so
|
||||
nothing there is written, except the state file and totk.ini, which a run rewrites and which are copied. The run hashes
|
||||
the 17 GB RomFS; it is skipped when the hash differs from the expected one in all three tries. On the developer machine
|
||||
(RAM with a bad bit) a buffered read of a file sometimes returns flipped bits from the page cache while the file on
|
||||
disk is intact (an unbuffered read of install-test's romfs.bin gives the expected hash), so such a skip is the machine,
|
||||
not the installer. It is also skipped when build/install-test is absent.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
from installer import state as statefile # noqa: E402
|
||||
from installer.guilogic import (DONE, FAILED, PENDING, RUNNING, SKIPPED, Catalog, Form, Model, Process, # noqa: E402
|
||||
amount, describe_event, parse_line, windows_ui_language)
|
||||
|
||||
INSTALLED = ROOT / 'build' / 'install-test'
|
||||
FIXTURE = ROOT / 'build' / 'inst-gui' / 'e2e-install'
|
||||
COPIED = ('totk.ini', statefile.NAME) # written by a run, so never linked
|
||||
START = {'type': 'start', 'release': 'r1', 'mode': 'install', 'dest': 'D:\\T', 'log': 'D:\\T.log',
|
||||
'steps': ['check', 'extract', 'lift', 'compile', 'prepare', 'cleanup']}
|
||||
|
||||
|
||||
def fold(*events):
|
||||
m = Model()
|
||||
for e in events:
|
||||
m.apply(e)
|
||||
return m
|
||||
|
||||
|
||||
class ParseTest(unittest.TestCase):
|
||||
def test_event_line(self):
|
||||
self.assertEqual(parse_line('{"type": "step", "step": "check", "state": "started"}\n')['step'], 'check')
|
||||
|
||||
def test_noise_is_not_an_event(self):
|
||||
for line in ('', 'Traceback (most recent call last):', '{broken', '[1, 2]', '{"no_type": 1}', '{"type": 3}'):
|
||||
self.assertIsNone(parse_line(line), line)
|
||||
|
||||
def test_unicode_survives(self):
|
||||
e = parse_line(json.dumps({'type': 'warning', 'id': 'x', 'params': {'p': 'Привет'}}, ensure_ascii=False))
|
||||
self.assertEqual(e['params']['p'], 'Привет')
|
||||
|
||||
|
||||
class ModelTest(unittest.TestCase):
|
||||
def test_steps_follow_the_events(self):
|
||||
m = fold(START, {'type': 'plan', 'run': ['check', 'extract'], 'skip': {'lift': 'current', 'compile': 'current'}},
|
||||
{'type': 'step', 'step': 'check', 'state': 'started'})
|
||||
self.assertEqual([m.states[s] for s in m.steps],
|
||||
[RUNNING, PENDING, SKIPPED, SKIPPED, PENDING, PENDING])
|
||||
self.assertEqual(m.current, 'check')
|
||||
m.apply({'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 1.0})
|
||||
self.assertEqual(m.states['check'], DONE)
|
||||
self.assertIsNone(m.current)
|
||||
m.apply({'type': 'step', 'step': 'extract', 'state': 'skipped', 'reason': 'current'})
|
||||
self.assertEqual(m.states['extract'], SKIPPED)
|
||||
|
||||
def test_progress_fraction(self):
|
||||
m = fold(START, {'type': 'step', 'step': 'extract', 'state': 'started'},
|
||||
{'type': 'progress', 'step': 'extract', 'phase': 'copy', 'done': 5, 'total': 20, 'unit': 'bytes'})
|
||||
self.assertEqual((m.current, m.phase, m.fraction()), ('extract', 'copy', 0.25))
|
||||
m.apply({'type': 'progress', 'step': 'extract', 'phase': 'merge'})
|
||||
self.assertIsNone(m.fraction()) # a phase without size is indeterminate
|
||||
m.apply({'type': 'progress', 'step': 'extract', 'phase': 'copy', 'done': 30, 'total': 20})
|
||||
self.assertEqual(m.fraction(), 1.0)
|
||||
|
||||
def test_a_new_step_resets_progress(self):
|
||||
m = fold(START, {'type': 'progress', 'step': 'check', 'phase': 'inputs', 'done': 1, 'total': 2},
|
||||
{'type': 'step', 'step': 'extract', 'state': 'started'})
|
||||
self.assertIsNone(m.phase)
|
||||
self.assertIsNone(m.fraction())
|
||||
|
||||
def test_error_marks_the_step(self):
|
||||
m = fold(START, {'type': 'step', 'step': 'check', 'state': 'started'},
|
||||
{'type': 'error', 'id': 'check.disk_space', 'step': 'check', 'params': {'path': 'D:\\T'}},
|
||||
{'type': 'end', 'result': 'failed', 'seconds': 2})
|
||||
self.assertEqual(m.states['check'], FAILED)
|
||||
self.assertEqual(m.error[0], 'check.disk_space')
|
||||
self.assertEqual(m.result, 'failed')
|
||||
|
||||
def test_space_text(self):
|
||||
m = fold({'type': 'space', 'dest': 'D:\\T', 'need': 36_400_000_000, 'free': 100_000_000_000, 'temp': 'C:\\t',
|
||||
'temp_need': 0, 'temp_free': 5, 'same_disk': True})
|
||||
text = m.need_text(Catalog('en'))
|
||||
self.assertIn('36.4', text)
|
||||
self.assertIn('100.0', text)
|
||||
self.assertEqual(Model().need_text(Catalog('en')), '')
|
||||
m = fold({'type': 'space', 'dest': 'D:\\T', 'need': 20_000_000_000, 'free': 1, 'temp': 'C:\\t',
|
||||
'temp_need': 2_000_000_000, 'temp_free': 3, 'same_disk': False})
|
||||
self.assertIn('C:\\t', m.need_text(Catalog('en')))
|
||||
|
||||
def test_warnings_collect(self):
|
||||
m = fold(START, {'type': 'warning', 'id': 'warn.lang_needs_fonts', 'params': {'lang': 'ko'}})
|
||||
self.assertEqual(m.warnings, [('warn.lang_needs_fonts', {'lang': 'ko'})])
|
||||
|
||||
def test_finish_without_end_event(self):
|
||||
m = fold(START, {'type': 'step', 'step': 'lift', 'state': 'started'})
|
||||
m.finish(-1073741510)
|
||||
self.assertEqual(m.result, 'failed')
|
||||
self.assertEqual(m.states['lift'], FAILED)
|
||||
self.assertEqual(m.error[0], 'ui.process_died')
|
||||
|
||||
def test_finish_after_cancel(self):
|
||||
m = fold(START, {'type': 'step', 'step': 'lift', 'state': 'started'}, {'type': 'end', 'result': 'cancelled'})
|
||||
m.finish(3)
|
||||
self.assertEqual(m.result, 'cancelled')
|
||||
self.assertIsNone(m.error)
|
||||
self.assertEqual(m.states['lift'], PENDING) # not finished; the next run resumes here
|
||||
|
||||
def test_exit_code_3_alone_is_cancelled(self):
|
||||
m = Model()
|
||||
m.finish(3)
|
||||
self.assertEqual(m.result, 'cancelled')
|
||||
|
||||
def test_unknown_event_is_ignored(self):
|
||||
m = fold(START, {'type': 'future_thing', 'x': 1})
|
||||
self.assertEqual(m.states['check'], PENDING)
|
||||
|
||||
|
||||
class RunStatesTest(unittest.TestCase):
|
||||
OK = [START, {'type': 'plan', 'run': ['check', 'extract'], 'skip': {'lift': 'current', 'compile': 'current',
|
||||
'prepare': 'current', 'cleanup': 'not_needed'}},
|
||||
{'type': 'step', 'step': 'check', 'state': 'started'}, {'type': 'step', 'step': 'check', 'state': 'done'},
|
||||
{'type': 'step', 'step': 'extract', 'state': 'started'}, {'type': 'step', 'step': 'extract', 'state': 'done'},
|
||||
{'type': 'step', 'step': 'lift', 'state': 'skipped'}, {'type': 'step', 'step': 'compile', 'state': 'skipped'},
|
||||
{'type': 'step', 'step': 'prepare', 'state': 'skipped'}, {'type': 'step', 'step': 'cleanup', 'state': 'done'},
|
||||
{'type': 'end', 'result': 'ok', 'seconds': 1}]
|
||||
FAILED = [START, {'type': 'step', 'step': 'check', 'state': 'started'}, {'type': 'step', 'step': 'check', 'state': 'done'},
|
||||
{'type': 'step', 'step': 'extract', 'state': 'started'},
|
||||
{'type': 'error', 'id': 'extract.romfs_mismatch', 'step': 'extract', 'params': {}},
|
||||
{'type': 'warning', 'id': 'warn.lang_needs_fonts', 'params': {'lang': 'ko'}},
|
||||
{'type': 'end', 'result': 'failed', 'seconds': 1}]
|
||||
|
||||
def test_ok_run_has_only_done_and_skipped_steps(self):
|
||||
m = fold(*self.OK)
|
||||
m.finish(0)
|
||||
self.assertEqual(m.result, 'ok')
|
||||
self.assertIsNone(m.error)
|
||||
self.assertEqual({m.states[s] for s in m.steps}, {DONE, SKIPPED})
|
||||
|
||||
def test_a_new_run_keeps_nothing_of_the_failed_one(self):
|
||||
m = fold(*self.FAILED)
|
||||
m.finish(1)
|
||||
self.assertEqual(m.states['extract'], FAILED)
|
||||
for e in self.OK: # the same model object fed a second run (the window makes a new one, but even then)
|
||||
m.apply(e)
|
||||
m.finish(0)
|
||||
self.assertEqual({m.states[s] for s in m.steps}, {DONE, SKIPPED})
|
||||
self.assertIsNone(m.error)
|
||||
self.assertEqual(m.warnings, [])
|
||||
self.assertEqual(m.result, 'ok')
|
||||
|
||||
|
||||
class FormTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = Path(self.tmp.name)
|
||||
for name in ('base.nsp', 'upd.nsp', 'prod.keys'):
|
||||
(self.dir / name).write_bytes(b'x')
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def form(self, **kw):
|
||||
values = dict(base=str(self.dir / 'base.nsp'), update=str(self.dir / 'upd.nsp'), keys=str(self.dir / 'prod.keys'),
|
||||
lang='en-US', dest=str(self.dir / 'out'))
|
||||
values.update(kw)
|
||||
return Form(**values)
|
||||
|
||||
def test_complete_form(self):
|
||||
self.assertEqual(self.form().problems(), [])
|
||||
|
||||
def test_empty_form(self):
|
||||
self.assertEqual(Form().problems(), ['ui.need_dest', 'ui.need_base', 'ui.need_update', 'ui.need_keys',
|
||||
'ui.need_lang'])
|
||||
|
||||
def test_missing_files(self):
|
||||
self.assertEqual(self.form(base=str(self.dir / 'no.nsp'), keys=str(self.dir / 'no.keys')).problems(),
|
||||
['ui.missing_base', 'ui.missing_keys'])
|
||||
|
||||
def test_unknown_language(self):
|
||||
self.assertEqual(self.form(lang='xx').problems(), ['ui.need_lang'])
|
||||
|
||||
def test_command(self):
|
||||
cmd = self.form().command('py.exe', root=ROOT, temp=self.dir / 't', verify=True)
|
||||
self.assertEqual(cmd[:3], ['py.exe', '-m', 'installer'])
|
||||
for flag in ('--base', '--update', '--keys', '--lang', '--dest', '--root', '--temp', '--verify'):
|
||||
self.assertIn(flag, cmd)
|
||||
self.assertEqual(cmd[cmd.index('--lang') + 1], 'en-US')
|
||||
|
||||
def test_installed_folder_needs_only_itself(self):
|
||||
out = self.dir / 'out'
|
||||
statefile.save(self._mkdir(out), statefile.empty('r0') | {'lang': 'ru'})
|
||||
form = Form(dest=str(out))
|
||||
self.assertTrue(form.update_mode())
|
||||
self.assertEqual(form.problems(), [])
|
||||
cmd = form.command('py.exe')
|
||||
self.assertEqual(cmd, ['py.exe', '-m', 'installer', '--dest', str(out)])
|
||||
|
||||
def test_installed_folder_still_checks_given_files(self):
|
||||
out = self.dir / 'out'
|
||||
statefile.save(self._mkdir(out), statefile.empty('r0'))
|
||||
self.assertEqual(Form(dest=str(out), base=str(self.dir / 'no.nsp')).problems(), ['ui.missing_base'])
|
||||
|
||||
def test_broken_state_is_a_new_install(self):
|
||||
out = self._mkdir(self.dir / 'out')
|
||||
(out / statefile.NAME).write_text('{nope', encoding='utf-8')
|
||||
self.assertFalse(Form(dest=str(out)).update_mode())
|
||||
|
||||
@staticmethod
|
||||
def _mkdir(path):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
class CatalogTest(unittest.TestCase):
|
||||
def test_russian_covers_english_with_the_same_placeholders(self):
|
||||
en, ru = Catalog('en').english, Catalog('ru').texts
|
||||
self.assertEqual(sorted(set(en) - set(ru)), [])
|
||||
self.assertEqual(sorted(set(ru) - set(en)), [])
|
||||
holes = [k for k in en if sorted(re.findall(r'\{\w+\}', en[k])) != sorted(re.findall(r'\{\w+\}', ru[k]))]
|
||||
self.assertEqual(holes, [])
|
||||
|
||||
def test_every_id_the_window_uses_exists(self):
|
||||
cat = Catalog('en')
|
||||
ids = ['ui.' + n for n in ('title', 'start', 'cancel', 'open_folder', 'notice', 'space_one', 'space_two',
|
||||
'process_died', 'result_ok', 'result_failed', 'result_cancelled')]
|
||||
ids += [f'step.{s}' for s in ('check', 'extract', 'lift', 'compile', 'prepare', 'cleanup')]
|
||||
ids += ['ui.state_' + n for n in ('started', 'done', 'skipped', 'pending', 'running', 'failed')]
|
||||
for lang in ('en', 'ru'):
|
||||
cat.set_language(lang)
|
||||
self.assertEqual([i for i in ids if not cat.has(i)], [])
|
||||
|
||||
def test_codes_become_names(self):
|
||||
for lang, korean in (('en', 'Korean'), ('ru', 'Корейский')):
|
||||
cat = Catalog(lang)
|
||||
self.assertIn(korean, cat('warn.lang_needs_fonts', lang='ko'))
|
||||
self.assertNotIn(' ko ', cat('warn.lang_needs_fonts', lang='ko'))
|
||||
self.assertNotIn('extract', cat('state.output_changed', step='extract'))
|
||||
for word in ('base', 'update', 'keys'):
|
||||
self.assertIn(cat('param.input.' + word), cat('check.keys_wrong', input=word))
|
||||
self.assertNotIn('en-US', cat('check.lang_unknown', lang='xx', known='ja, en-US'))
|
||||
self.assertEqual(Catalog('en')('check.lang_unknown', lang='xx', known='ja, en-US'),
|
||||
"Unknown language 'xx'. Known: Japanese, English (US).")
|
||||
|
||||
def test_fallback_and_params(self):
|
||||
cat = Catalog('ru')
|
||||
self.assertIn('36.4', cat('ui.log_space', need='36.4', free='1.0'))
|
||||
self.assertEqual(cat('no.such.id'), 'no.such.id')
|
||||
self.assertIn('a, b', cat('check.release_incomplete', paths=['a', 'b']))
|
||||
cat.set_language('de') # unknown language falls back to English
|
||||
self.assertEqual(cat.lang, 'en')
|
||||
|
||||
def test_describe_and_amount(self):
|
||||
cat = Catalog('en')
|
||||
self.assertEqual(amount(cat, 6e9, 16e9, 'bytes'), '6.0 / 16.0 GB')
|
||||
self.assertEqual(amount(cat, 1, 9, 'shaders'), '1 / 9 shaders')
|
||||
self.assertEqual(amount(cat, 1, 9, 'widgets'), '1 / 9 widgets')
|
||||
line = describe_event(cat, {'type': 'step', 'step': 'check', 'state': 'done', 'seconds': 1.5})
|
||||
self.assertIn('1.5 s', line)
|
||||
self.assertEqual(describe_event(cat, {'type': 'start'}), '')
|
||||
|
||||
def test_windows_language_is_a_known_value(self):
|
||||
self.assertIn(windows_ui_language(), ('en', 'ru'))
|
||||
|
||||
|
||||
BREAK_CHILD = 'import time\nprint("ready", flush=True)\ntime.sleep(60)\n'
|
||||
|
||||
|
||||
class CancelTest(unittest.TestCase):
|
||||
def test_ctrl_break_ends_the_process_group(self):
|
||||
p = Process([sys.executable, '-c', BREAK_CHILD])
|
||||
try:
|
||||
self.assertEqual(p.lines.get(timeout=20).strip(), 'ready')
|
||||
t0 = time.monotonic()
|
||||
p.cancel()
|
||||
code = None
|
||||
while code is None and time.monotonic() - t0 < 20:
|
||||
code = p.poll()
|
||||
time.sleep(0.05)
|
||||
self.assertIsNotNone(code, 'the child ignored Ctrl+Break')
|
||||
self.assertNotEqual(code, 0)
|
||||
finally:
|
||||
p.kill()
|
||||
|
||||
|
||||
class SmokeTest(unittest.TestCase):
|
||||
def test_every_screen_in_both_languages(self):
|
||||
try:
|
||||
import tkinter # noqa: F401
|
||||
except ImportError:
|
||||
self.skipTest('no tkinter')
|
||||
r = subprocess.run([sys.executable, '-m', 'installer.gui', '--selftest'], cwd=ROOT, timeout=120,
|
||||
capture_output=True, text=True)
|
||||
self.assertEqual(r.returncode, 0, r.stderr[-2000:])
|
||||
|
||||
|
||||
def make_fixture():
|
||||
"""build/inst-gui/e2e-install: links to install-test (see the module docstring); rebuilt when absent."""
|
||||
if (FIXTURE / statefile.NAME).is_file():
|
||||
return FIXTURE
|
||||
shutil.rmtree(FIXTURE, ignore_errors=True)
|
||||
for folder, _dirs, files in os.walk(INSTALLED):
|
||||
target = FIXTURE / Path(folder).relative_to(INSTALLED)
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
for name in files:
|
||||
src, dst = Path(folder) / name, target / name
|
||||
if name in COPIED and target == FIXTURE:
|
||||
shutil.copy2(src, dst)
|
||||
else:
|
||||
os.link(src, dst)
|
||||
return FIXTURE
|
||||
|
||||
|
||||
@unittest.skipUnless((INSTALLED / statefile.NAME).is_file(), 'build/install-test is absent')
|
||||
class EndToEndTest(unittest.TestCase):
|
||||
@staticmethod
|
||||
def run_cli(form, temp, tries=1):
|
||||
"""Runs the CLI in --verify mode through Process, up to `tries` times while the RomFS hash reads wrong."""
|
||||
for attempt in range(tries):
|
||||
p = Process(form.command(sys.executable, root=ROOT, temp=temp, verify=True), cwd=ROOT)
|
||||
model, stray = Model(), []
|
||||
t0, code = time.monotonic(), None
|
||||
while code is None and time.monotonic() - t0 < 600:
|
||||
try:
|
||||
line = p.lines.get(timeout=0.2)
|
||||
except queue.Empty:
|
||||
line = ''
|
||||
if line:
|
||||
event = parse_line(line)
|
||||
if event:
|
||||
model.apply(event)
|
||||
elif line.strip():
|
||||
stray.append(line)
|
||||
code = p.poll()
|
||||
p.kill()
|
||||
model.finish(code)
|
||||
if not any(w[0] == 'state.output_changed' for w in model.warnings):
|
||||
break
|
||||
return code, model, stray, time.monotonic() - t0
|
||||
|
||||
def test_verify_run_skips_every_step(self):
|
||||
form = Form(dest=str(make_fixture()))
|
||||
self.assertTrue(form.update_mode())
|
||||
code, model, stray, elapsed = self.run_cli(form, ROOT / 'build' / 'inst-gui' / 'e2e-temp', tries=3)
|
||||
print(f'\nend-to-end: exit {code}, result {model.result}, {elapsed:.1f} s, states {model.states}')
|
||||
if any(w[0] == 'state.output_changed' for w in model.warnings):
|
||||
self.skipTest('the RomFS hash read wrong three times (page cache of the faulty RAM, see the module docstring)')
|
||||
self.assertEqual(code, 0, stray[-5:] or model.error)
|
||||
self.assertEqual(model.result, 'ok')
|
||||
self.assertIn(model.mode, ('resume', 'update'))
|
||||
self.assertEqual({s: model.states[s] for s in model.steps if s not in ('check', 'cleanup')},
|
||||
{s: SKIPPED for s in model.steps if s not in ('check', 'cleanup')})
|
||||
self.assertIsNotNone(model.space)
|
||||
self.assertEqual(stray, [])
|
||||
|
||||
def test_cancel_during_the_check(self):
|
||||
cmd = Form(dest=str(make_fixture())).command(sys.executable, root=ROOT, temp=ROOT / 'build' / 'inst-gui' / 'e2e-temp',
|
||||
verify=True)
|
||||
p = Process(cmd, cwd=ROOT)
|
||||
model, t0, code, cancelled = Model(), time.monotonic(), None, False
|
||||
while code is None and time.monotonic() - t0 < 120:
|
||||
try:
|
||||
event = parse_line(p.lines.get(timeout=0.2))
|
||||
except queue.Empty:
|
||||
event = None
|
||||
if event:
|
||||
model.apply(event)
|
||||
if event.get('phase') == 'verify_romfs' and not cancelled:
|
||||
cancelled = True
|
||||
p.cancel()
|
||||
code = p.poll()
|
||||
p.kill()
|
||||
model.finish(code)
|
||||
self.assertTrue(cancelled)
|
||||
self.assertEqual((code, model.result), (3, 'cancelled'))
|
||||
self.assertIsNone(model.error)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,264 @@
|
||||
"""installer/ bookkeeping with fake steps: fresh install, resume after a failure or cancel, transient steps, update
|
||||
with a newer release, totk.ini and the event stream. No game files, no tools; stdlib only.
|
||||
|
||||
Run: python tests/installer_test.py
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
from installer import ini, runner, state # noqa: E402
|
||||
from installer.events import Reporter # noqa: E402
|
||||
from installer.layout import Layout # noqa: E402
|
||||
|
||||
|
||||
class Fake(runner.Step):
|
||||
"""A step whose output is <dir>/<id>.out holding its key; dir is the temp folder for a transient step."""
|
||||
|
||||
def __init__(self, id, transient=False, always=False, previous=None):
|
||||
self.id, self.transient, self.always, self.previous = id, transient, always, previous
|
||||
|
||||
def out(self, ctx):
|
||||
return (ctx.temp if self.transient else ctx.dest) / f'{self.id}.out'
|
||||
|
||||
def key(self, ctx):
|
||||
upstream = ctx.keys[self.previous] if self.previous else ''
|
||||
return f'{self.id}:{ctx.options.inputs.get(self.id, "v1")}:{upstream}'
|
||||
|
||||
def current(self, ctx, record):
|
||||
p = self.out(ctx)
|
||||
return p.is_file() and p.read_text() == ctx.keys[self.id]
|
||||
|
||||
def verify(self, ctx, record):
|
||||
ctx.options.verified.append(self.id)
|
||||
return ctx.options.verify_ok.get(self.id, True)
|
||||
|
||||
def run(self, ctx):
|
||||
o = ctx.options
|
||||
o.ran.append(self.id)
|
||||
if self.id in ctx.forced:
|
||||
o.forced.append(self.id)
|
||||
if self.id == 'check':
|
||||
ctx.verify_kept(self.id)
|
||||
failure = o.fail.pop(self.id, None)
|
||||
if failure == 'error':
|
||||
raise runner.InstallError('fake.failed', step=self.id)
|
||||
if failure == 'cancel':
|
||||
raise KeyboardInterrupt
|
||||
if self.id == 'cleanup':
|
||||
for p in ctx.temp.glob('*.out'):
|
||||
p.unlink()
|
||||
return {}
|
||||
p = self.out(ctx)
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(ctx.keys[self.id])
|
||||
return {'note': self.id}
|
||||
|
||||
|
||||
def fake_steps():
|
||||
return [Fake('check', always=True), Fake('extract'), Fake('lift', transient=True, previous='extract'),
|
||||
Fake('compile', previous='lift'), Fake('prepare', previous='compile'), Fake('cleanup', always=True)]
|
||||
|
||||
|
||||
class Bed:
|
||||
"""A dest and temp folder plus one run of the fake steps per call."""
|
||||
|
||||
def __init__(self, base):
|
||||
self.dest, self.temp = base / 'dest', base / 'temp'
|
||||
self.temp.mkdir()
|
||||
|
||||
def run(self, release='1.0', inputs=None, fail=None, verify_ok=None, verify=False):
|
||||
layout = Layout(root=Path('.'), dev=True, release=release, python=Path('python'), tools=Path('tools'),
|
||||
recomp=Path('recomp'), patches=Path('p'), expected=Path('e'), llvm=Path('llvm'),
|
||||
deps=Path('deps'), runtime=Path('rt'), runtime_lib=Path('lib'), ninja=Path('ninja'),
|
||||
hactool=Path('hactool'), xlate=Path('xlate'))
|
||||
options = SimpleNamespace(inputs=inputs or {}, fail=dict(fail or {}), verify_ok=verify_ok or {}, ran=[],
|
||||
verified=[], forced=[])
|
||||
out = io.StringIO()
|
||||
ctx = runner.Context(layout, self.dest, self.temp, Reporter(out=out), options, state=state.load(self.dest),
|
||||
verify=verify)
|
||||
mode = ctx.mode
|
||||
result = runner.run_steps(fake_steps(), ctx)
|
||||
events = [json.loads(line) for line in out.getvalue().splitlines()]
|
||||
return SimpleNamespace(result=result, ran=options.ran, verified=options.verified, forced=options.forced,
|
||||
events=events, mode=mode,
|
||||
state=state.load(self.dest))
|
||||
|
||||
@staticmethod
|
||||
def skipped(r):
|
||||
return {e['step']: e['reason'] for e in r.events if e['type'] == 'step' and e['state'] == 'skipped'}
|
||||
|
||||
|
||||
ALL = ['check', 'extract', 'lift', 'compile', 'prepare', 'cleanup']
|
||||
|
||||
|
||||
class RunnerTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.bed = Bed(Path(self.tmp.name))
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_fresh_install_runs_everything_and_records_it(self):
|
||||
r = self.bed.run()
|
||||
self.assertEqual((r.result, r.mode, r.ran), ('ok', 'install', ALL))
|
||||
self.assertEqual(set(r.state['steps']), {'extract', 'lift', 'compile', 'prepare'})
|
||||
self.assertEqual(r.state['steps']['compile']['note'], 'compile')
|
||||
self.assertEqual(r.state['release'], '1.0')
|
||||
started = [e['step'] for e in r.events if e['type'] == 'step' and e['state'] == 'started']
|
||||
done = [e['step'] for e in r.events if e['type'] == 'step' and e['state'] == 'done']
|
||||
self.assertEqual((started, done), (ALL, ALL))
|
||||
|
||||
def test_rerun_skips_finished_steps_and_unneeded_transient(self):
|
||||
self.bed.run()
|
||||
r = self.bed.run()
|
||||
self.assertEqual((r.result, r.mode, r.ran), ('ok', 'resume', ['check', 'cleanup']))
|
||||
self.assertEqual(self.bed.skipped(r), {'extract': 'current', 'lift': 'not_needed', 'compile': 'current',
|
||||
'prepare': 'current'})
|
||||
self.assertEqual(r.verified, []) # same release: no full checks
|
||||
|
||||
def test_resume_after_failure(self):
|
||||
r = self.bed.run(fail={'compile': 'error'})
|
||||
self.assertEqual((r.result, r.ran), ('failed', ['check', 'extract', 'lift', 'compile']))
|
||||
error = [e for e in r.events if e['type'] == 'error'][0]
|
||||
self.assertEqual((error['id'], error['step'], error['params']), ('fake.failed', 'compile', {'step': 'compile'}))
|
||||
self.assertEqual(set(r.state['steps']), {'extract', 'lift'})
|
||||
r = self.bed.run()
|
||||
# lift's output is still in the temp folder (no cleanup ran), so only compile and later run.
|
||||
self.assertEqual((r.result, r.mode, r.ran), ('ok', 'resume', ['check', 'compile', 'prepare', 'cleanup']))
|
||||
|
||||
def test_resume_after_cancel_with_transient_output_gone(self):
|
||||
r = self.bed.run(fail={'prepare': 'cancel'})
|
||||
self.assertEqual(r.result, 'cancelled')
|
||||
self.assertNotIn('prepare', r.state['steps'])
|
||||
for p in self.bed.temp.glob('*.out'):
|
||||
p.unlink()
|
||||
r = self.bed.run()
|
||||
self.assertEqual(r.ran, ['check', 'prepare', 'cleanup'])
|
||||
self.assertEqual(self.bed.skipped(r)['lift'], 'not_needed')
|
||||
|
||||
def test_failed_check_writes_nothing(self):
|
||||
r = self.bed.run(fail={'check': 'error'})
|
||||
self.assertEqual(r.result, 'failed')
|
||||
self.assertFalse(self.bed.dest.exists())
|
||||
|
||||
def test_removed_output_runs_again(self):
|
||||
self.bed.run()
|
||||
(self.bed.dest / 'compile.out').unlink()
|
||||
r = self.bed.run()
|
||||
self.assertEqual(r.ran, ['check', 'lift', 'compile', 'cleanup'])
|
||||
|
||||
def test_update_with_new_lifter_keeps_extraction(self):
|
||||
self.bed.run(release='1.0')
|
||||
r = self.bed.run(release='1.1', inputs={'lift': 'v2'})
|
||||
self.assertEqual((r.result, r.mode), ('ok', 'update'))
|
||||
self.assertEqual(r.ran, ['check', 'lift', 'compile', 'prepare', 'cleanup'])
|
||||
self.assertEqual(r.verified, ['extract']) # kept from another release: full check
|
||||
self.assertEqual(r.state['steps']['extract']['release'], '1.0')
|
||||
self.assertEqual(r.state['steps']['compile']['release'], '1.1')
|
||||
self.assertEqual(r.state['release'], '1.1')
|
||||
|
||||
def test_update_that_changes_nothing_skips_the_rebuild(self):
|
||||
self.bed.run(release='1.0')
|
||||
r = self.bed.run(release='1.1')
|
||||
self.assertEqual(r.ran, ['check', 'cleanup'])
|
||||
self.assertEqual(r.verified, ['extract', 'compile', 'prepare'])
|
||||
|
||||
def test_update_with_changed_kept_files_extracts_again(self):
|
||||
self.bed.run(release='1.0')
|
||||
r = self.bed.run(release='1.1', verify_ok={'extract': False})
|
||||
self.assertEqual(r.ran, ['check', 'extract', 'cleanup'])
|
||||
self.assertEqual(r.forced, ['extract']) # the step knows not to trust what it left behind
|
||||
warnings = [e for e in r.events if e['type'] == 'warning']
|
||||
self.assertEqual([(w['id'], w['params']) for w in warnings], [('state.output_changed', {'step': 'extract'})])
|
||||
|
||||
def test_verify_option_checks_same_release(self):
|
||||
self.bed.run()
|
||||
r = self.bed.run(verify=True)
|
||||
self.assertEqual(r.verified, ['extract', 'compile', 'prepare'])
|
||||
|
||||
def test_state_errors(self):
|
||||
self.bed.dest.mkdir()
|
||||
(self.bed.dest / state.NAME).write_text('{not json')
|
||||
with self.assertRaises(state.StateError) as e:
|
||||
state.load(self.bed.dest)
|
||||
self.assertEqual(e.exception.id, 'state.unreadable')
|
||||
(self.bed.dest / state.NAME).write_text(json.dumps({'format': 99, 'steps': {}}))
|
||||
with self.assertRaises(state.StateError) as e:
|
||||
state.load(self.bed.dest)
|
||||
self.assertEqual(e.exception.id, 'state.newer_format')
|
||||
|
||||
|
||||
class IniTest(unittest.TestCase):
|
||||
EXAMPLE = '[display]\n# vk_present = mailbox\n\n[language]\n# locale = en-US\n# lang = en-US\n# x = 1\n'
|
||||
|
||||
def test_example_line_is_activated(self):
|
||||
out = ini.set_language(self.EXAMPLE, 'ru')
|
||||
self.assertIn('\nlang = ru\n', out)
|
||||
self.assertNotIn('# lang', out)
|
||||
self.assertIn('# locale = en-US', out)
|
||||
|
||||
def test_active_line_is_rewritten_and_the_rest_kept(self):
|
||||
text = '[display]\nvk_present = fifo\n[language]\n LANG = ja \nfps60 = 0\n'
|
||||
out = ini.set_language(text, 'de')
|
||||
self.assertEqual(out, '[display]\nvk_present = fifo\n[language]\n lang = de\nfps60 = 0\n')
|
||||
self.assertEqual(ini.get_language(out), 'de')
|
||||
|
||||
def test_missing_section_is_added(self):
|
||||
self.assertEqual(ini.set_language('fps60 = 0\n', 'fr'), 'fps60 = 0\n\n[language]\nlang = fr\n')
|
||||
self.assertEqual(ini.set_language('[language]\nlocale = de-DE\n', 'fr'),
|
||||
'[language]\nlang = fr\nlocale = de-DE\n')
|
||||
|
||||
def test_write_keeps_user_lines(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
d = Path(d)
|
||||
example = d / 'totk.ini.example'
|
||||
example.write_text(self.EXAMPLE)
|
||||
path = ini.write(d / '', 'en-GB', example)
|
||||
self.assertEqual(ini.get_language(path.read_text()), 'en-GB')
|
||||
path.write_text(path.read_text().replace('# vk_present = mailbox', 'vk_present = fifo'))
|
||||
ini.write(d, 'ja', example)
|
||||
text = path.read_text()
|
||||
self.assertIn('vk_present = fifo', text)
|
||||
self.assertEqual(ini.get_language(text), 'ja')
|
||||
|
||||
|
||||
class EventsTest(unittest.TestCase):
|
||||
def test_lines_are_json_and_the_log_is_english(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
out = io.StringIO()
|
||||
rep = Reporter(out=out, log_path=Path(d) / 'x.log')
|
||||
rep.step('check', 'started')
|
||||
rep.progress('extract', 'copy', 0, 10, 'bytes')
|
||||
rep.progress('extract', 'copy', 5, 10, 'bytes') # throttled
|
||||
rep.progress('extract', 'copy', 10, 10, 'bytes') # last one always goes out
|
||||
rep.error('check.disk_space', {'path': 'D:\\x', 'need_gb': 40, 'free_gb': 3}, step='check')
|
||||
rep.close()
|
||||
events = [json.loads(line) for line in out.getvalue().splitlines()]
|
||||
self.assertEqual([e['type'] for e in events], ['step', 'progress', 'progress', 'error'])
|
||||
self.assertEqual([e.get('done') for e in events[1:3]], [0, 10])
|
||||
self.assertTrue(all('t' in e for e in events))
|
||||
log = (Path(d) / 'x.log').read_text(encoding='utf-8')
|
||||
self.assertIn('40 GB needed, 3 GB free', log)
|
||||
self.assertIn('[check.disk_space]', log)
|
||||
|
||||
def test_catalog_has_every_id_the_code_uses(self):
|
||||
catalog = json.loads((ROOT / 'installer' / 'text' / 'en.json').read_text(encoding='utf-8'))
|
||||
used = set()
|
||||
for path in (ROOT / 'installer').glob('*.py'):
|
||||
used |= set(re.findall(r"InstallError\('([a-z_.]+)'", path.read_text()))
|
||||
used |= set(re.findall(r"warning\('([a-z_.]+)'", path.read_text()))
|
||||
used |= {f'phase.{s}.{p}' for s, p in re.findall(r"progress\('(\w+)', '(\w+)'[,)]", path.read_text())}
|
||||
self.assertEqual(sorted(i for i in used if i not in catalog), [])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+90
-10
@@ -7,6 +7,8 @@ recompiled only when it, a header it includes or its flags change.
|
||||
.venv\\Scripts\\python tools\\build_game.py --modules subsdk0 # one module (links if it has no calls into others)
|
||||
.venv\\Scripts\\python tools\\build_game.py --jobs 8
|
||||
.venv\\Scripts\\python tools\\build_game.py --out build/v121-t2 # a separate build directory
|
||||
.venv\\Scripts\\python tools\\build_game.py --toolchain toolchain/llvm-mingw-<version>/llvm-mingw-<version>-ucrt-x86_64 \\
|
||||
--out build/v121-llvmmingw # llvm-mingw instead of MSYS2 UCRT64
|
||||
|
||||
Every runtime/*.c is compiled. build/<version>/modules.c carries the layout, the linked function tables, the
|
||||
expected SHA-256 of each ExeFS file (tools/data/expected-<display version>.json) and the default game paths
|
||||
@@ -34,6 +36,16 @@ Compile jobs share a memory budget (MEM_BUDGET, default the smaller of 50 GiB an
|
||||
big machine does not run out of memory on the largest chunks. Each phase reports its wall time and the peak memory
|
||||
of its processes (Windows job object accounting: committed private bytes).
|
||||
|
||||
--ninja PATH names the Ninja to run; without MSYS2 the bin/ninja.exe of the release folder (tools/make_release.py) is used.
|
||||
--toolchain DIR builds with an llvm-mingw tree (DIR/bin/clang.exe) instead of MSYS2 UCRT64 clang. Ninja still comes
|
||||
from MSYS2. The libraries and headers MSYS2 supplies (Vulkan, SDL3, Opus, libvpx, VMA) then come from --deps DIR
|
||||
(default toolchain/deps-mingw: include/, lib/ with the import libraries and libvpx.a, bin/ with the DLLs), and the C++
|
||||
runtime is libc++ from llvm-mingw. Such a build never reuses another build's generated objects.
|
||||
|
||||
Split build. --make-runtime-lib compiles only runtime/ into <out>/libtotkrt.a (what CI prebuilds). --runtime-lib PATH
|
||||
then compiles only the generated C and the install-time files (modules.c and INSTALL_SIDE, which bake the player's
|
||||
game hashes in) and links them with that library. The library holds no game-derived data.
|
||||
|
||||
Standard library only.
|
||||
"""
|
||||
import argparse
|
||||
@@ -67,6 +79,15 @@ RT_FLAGS = ['-O2', '-g', '-Wall', '-fno-strict-aliasing', '-march=x86-64-v3']
|
||||
LINK_FLAGS = ['-fuse-ld=lld', '-Wl,--stack,16777216', '-Wl,--Map=$builddir/totk.map'] # the map feeds tools/prof_report.py
|
||||
LINK_LIBS = ['-lpsapi', '-lbcrypt', '-lvulkan-1', '-luser32', '-lgdi32', '-ldwmapi', '-lwinmm', '-lSDL3', '-lopus', '-l:libvpx.a', '-l:libstdc++.a', '-l:libwinpthread.a'] # Vulkan back end of M4; libvpx static (movies)
|
||||
VMA_INCLUDE = UCRT.parent / 'include' / 'vma' # vk_mem_alloc.h; MSYS2 package vulkan-memory-allocator
|
||||
# --toolchain: an llvm-mingw tree. The libraries, headers and DLLs MSYS2 supplies come from the deps directory instead.
|
||||
LLVM_LINK_LIBS = ['-lpsapi', '-lbcrypt', '-luser32', '-lgdi32', '-ldwmapi', '-lwinmm', '-lvulkan-1', '-lSDL3', '-lopus',
|
||||
'-l:libvpx.a', '-l:libc++.a', '-l:libc++abi.a', '-l:libunwind.a', '-l:libwinpthread.a']
|
||||
DEFAULT_DEPS = ROOT / 'toolchain' / 'deps-mingw'
|
||||
INSTALL_SIDE = {'romfs.c'} # runtime sources that include romfs_config.h: compiled per install, not in the library
|
||||
LIB_NAME = 'libtotkrt.a'
|
||||
AR = UCRT / 'llvm-ar.exe'
|
||||
TOOLCHAIN_BIN = None # llvm-mingw bin directory, set by main() for --toolchain
|
||||
DEPS = None # the deps directory of an llvm-mingw build
|
||||
# tools/data/expected-<display version>.json for each build version name.
|
||||
DISPLAY_VERSIONS = {'v0': '1.0.0', 'v112': '1.1.2', 'v121': '1.2.1', 'v143': '1.4.3'}
|
||||
|
||||
@@ -182,11 +203,14 @@ def module_glue(layout, linked, hashes, paths):
|
||||
return '\n'.join(out)
|
||||
|
||||
|
||||
def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False):
|
||||
def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=None, make_lib=False):
|
||||
"""build.ninja text, and the object lists per module. With prebuilt the generated objects already exist in
|
||||
out/obj and are link inputs without a compile edge."""
|
||||
out/obj and are link inputs without a compile edge. runtime_lib: a prebuilt library replaces the runtime objects
|
||||
except INSTALL_SIDE. make_lib: build only that library, from the runtime objects except INSTALL_SIDE."""
|
||||
inc = f'-I {flag_path(RUNTIME)} -I {flag_path(gen)}'
|
||||
rt_inc = f'{inc} -I {flag_path(out)} -I {flag_path(VMA_INCLUDE)}' # romfs_config.h, vk_mem_alloc.h
|
||||
vma = f'{flag_path(DEPS / "include")} -I {flag_path(DEPS / "include" / "vma")}' if DEPS else flag_path(VMA_INCLUDE)
|
||||
rt_inc = f'{inc} -I {flag_path(out)} -I {vma}' # romfs_config.h, vk_mem_alloc.h (and the deps headers)
|
||||
libs = [f'-L{flag_path(DEPS / "lib")}'] + LLVM_LINK_LIBS if DEPS else LINK_LIBS
|
||||
lines = [
|
||||
'# Written by tools/build_game.py; do not edit.',
|
||||
'ninja_required_version = 1.10',
|
||||
@@ -195,7 +219,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False):
|
||||
f'genflags = {" ".join(GEN_FLAGS)} {inc}',
|
||||
f'rtflags = {" ".join(RT_FLAGS)} {rt_inc}',
|
||||
f'ldflags = {" ".join(LINK_FLAGS)}',
|
||||
f'libs = {" ".join(LINK_LIBS)}',
|
||||
f'libs = {" ".join(libs)}',
|
||||
'',
|
||||
'pool gen',
|
||||
f' depth = {pool_depth}',
|
||||
@@ -229,14 +253,35 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False):
|
||||
lines.append(' pool = gen')
|
||||
objs[mod].append(o)
|
||||
rt = []
|
||||
for src in runtime_sources(skip) + [out / 'modules.c']:
|
||||
sources = runtime_sources(skip)
|
||||
if make_lib:
|
||||
sources = [p for p in sources if p.name not in INSTALL_SIDE]
|
||||
elif runtime_lib:
|
||||
sources = [p for p in sources if p.name in INSTALL_SIDE] + [out / 'modules.c']
|
||||
else:
|
||||
sources += [out / 'modules.c']
|
||||
for src in sources:
|
||||
o = runtime_object(out, src)
|
||||
lines.append(f'build {ninja_path(o)}: cc_rt {ninja_path(src)}')
|
||||
rt.append(o)
|
||||
objs['runtime'] = rt
|
||||
every = [o for v in objs.values() for o in v]
|
||||
exe = out / 'totk.exe'
|
||||
lines += ['', f'build {ninja_path(exe)}: link ' + ' '.join(ninja_path(o) for o in every),
|
||||
if make_lib:
|
||||
exe = out / LIB_NAME
|
||||
lines += ['', 'rule ar', f' command = {AR.as_posix()} rcs $out $in', ' description = AR $out', '', f'build {ninja_path(exe)}: ar ' + ' '.join(ninja_path(o) for o in every),
|
||||
'build objects: phony ' + ' '.join(ninja_path(o) for o in every), f'default {ninja_path(exe)}', '']
|
||||
return '\n'.join(lines), objs, exe
|
||||
rule = 'link'
|
||||
if runtime_lib:
|
||||
# Whole archive: the weak probe stubs in gfx_stubs.c would otherwise satisfy references to the probes before the
|
||||
# linker looks into the library, and the real probes would stay out of totk.exe.
|
||||
rule = 'link_lib'
|
||||
lines += ['', 'rule link_lib',
|
||||
f' command = $cc $ldflags @$out.rsp -Wl,--whole-archive {flag_path(runtime_lib)} -Wl,--no-whole-archive $libs -o $out',
|
||||
' rspfile = $out.rsp', ' rspfile_content = $in_newline', ' description = LINK $out']
|
||||
lines += ['', f'build {ninja_path(exe)}: {rule} ' + ' '.join(ninja_path(o) for o in every) +
|
||||
(f' | {ninja_path(runtime_lib)}' if runtime_lib else ''),
|
||||
'build objects: phony ' + ' '.join(ninja_path(o) for o in every), f'default {ninja_path(exe)}', '']
|
||||
return '\n'.join(lines), objs, exe
|
||||
|
||||
@@ -517,6 +562,16 @@ def main():
|
||||
ap.add_argument('--romfs', type=Path, help="totk.exe's default RomFS directory with romfs.bin and index.json "
|
||||
'(default build/extracted/<version>/romfs)')
|
||||
ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile)")
|
||||
ap.add_argument('--toolchain', type=Path, help='llvm-mingw directory (with bin/clang.exe) to build with instead of '
|
||||
'MSYS2 UCRT64 clang')
|
||||
ap.add_argument('--ninja', type=Path, help='ninja.exe to build with (default: MSYS2 UCRT64, else bin/ninja.exe of the '
|
||||
'release folder)')
|
||||
ap.add_argument('--deps', type=Path, help=f'with --toolchain: include/, lib/ and bin/ of Vulkan, SDL3, Opus, libvpx '
|
||||
f'and VMA (default {DEFAULT_DEPS.relative_to(ROOT).as_posix()})')
|
||||
ap.add_argument('--make-runtime-lib', action='store_true',
|
||||
help=f'build only <out>/{LIB_NAME} from runtime/ (no generated C, no totk.exe)')
|
||||
ap.add_argument('--runtime-lib', type=Path, help=f'link against this {LIB_NAME} instead of compiling runtime/ '
|
||||
f'({", ".join(sorted(INSTALL_SIDE))} and modules.c still compile)')
|
||||
ap.add_argument('--expected', type=Path,
|
||||
help='expected hashes (default tools/data/expected-<display version>.json)')
|
||||
args = ap.parse_args()
|
||||
@@ -533,6 +588,18 @@ def main():
|
||||
sys.exit(f'no expected hashes known for {args.version}; pass --expected')
|
||||
expected = ROOT / 'tools' / 'data' / f'expected-v{DISPLAY_VERSIONS[args.version]}.json'
|
||||
hashes = json.loads(Path(expected).read_text())['exefs_sha256']
|
||||
global CLANG, AR, NINJA, TOOLCHAIN_BIN, DEPS
|
||||
if args.ninja:
|
||||
NINJA = args.ninja.resolve()
|
||||
elif not NINJA.is_file() and (ROOT / 'bin' / 'ninja.exe').is_file():
|
||||
NINJA = ROOT / 'bin' / 'ninja.exe' # the release folder (tools/make_release.py)
|
||||
dlls = SDL3_DLLS
|
||||
if args.toolchain:
|
||||
TOOLCHAIN_BIN = (args.toolchain / 'bin').resolve()
|
||||
CLANG = TOOLCHAIN_BIN / 'clang.exe'
|
||||
AR = TOOLCHAIN_BIN / 'llvm-ar.exe'
|
||||
DEPS = (args.deps or DEFAULT_DEPS).resolve()
|
||||
dlls = sorted((DEPS / 'bin').glob('*.dll'))
|
||||
for tool in (CLANG, NINJA):
|
||||
if not tool.is_file():
|
||||
sys.exit(f'{tool} not found: install MSYS2 UCRT64 clang, lld and ninja (docs/toolchain-msys2-python.md)')
|
||||
@@ -566,8 +633,12 @@ def main():
|
||||
skip = {s for s in args.skip_runtime.split(',') if s}
|
||||
main_out = (ROOT / 'build' / args.version).resolve()
|
||||
pairs = None
|
||||
if args.full_build:
|
||||
if args.make_runtime_lib:
|
||||
mods, linked, why = {}, [], ''
|
||||
elif args.full_build:
|
||||
why = '--full-build'
|
||||
elif args.toolchain:
|
||||
why = f'--toolchain {flag_path(args.toolchain)}'
|
||||
elif out == main_out:
|
||||
why = ''
|
||||
else:
|
||||
@@ -580,10 +651,14 @@ def main():
|
||||
unlink_shared(out, mods)
|
||||
if why:
|
||||
print(f'build_game: compiling all generated C: {why}')
|
||||
text, objs, exe = ninja_file(gen, out, mods, jobs, skip, pairs is not None)
|
||||
runtime_lib = args.runtime_lib.resolve() if args.runtime_lib else None
|
||||
if runtime_lib and not runtime_lib.is_file():
|
||||
sys.exit(f'{runtime_lib} not found: build it with --make-runtime-lib')
|
||||
text, objs, exe = ninja_file(gen, out, mods, jobs, skip, pairs is not None, runtime_lib,
|
||||
args.make_runtime_lib)
|
||||
write_if_changed(out / 'build.ninja', text)
|
||||
|
||||
env = dict(os.environ, PATH=str(UCRT) + os.pathsep + os.environ.get('PATH', ''))
|
||||
env = dict(os.environ, PATH=str(TOOLCHAIN_BIN or UCRT) + os.pathsep + os.environ.get('PATH', ''))
|
||||
base = [str(NINJA), '-f', str(out / 'build.ninja'), '-j', str(jobs)]
|
||||
nfiles = sum(len(v) for v in mods.values())
|
||||
print(f'build_game: {nfiles} generated files in {len(linked)} modules ({", ".join(linked)}), {jobs} jobs', flush=True)
|
||||
@@ -598,13 +673,18 @@ def main():
|
||||
verify_objects(out, objs, pairs is not None, main_out, pairs, base, env)
|
||||
if args.compile_only:
|
||||
return
|
||||
if args.make_runtime_lib:
|
||||
exe.unlink(missing_ok=True) # ar rcs would keep members of deleted sources
|
||||
code, dt, peak_one, _ = run_measured(base + [flag_path(exe)], env)
|
||||
print(f'link: {dt:.1f} s, peak {mib(peak_one)}')
|
||||
if code:
|
||||
print(f'build_game: link failed, {flag_path(exe)} is the old one (a running totk.exe from this dir locks it)',
|
||||
file=sys.stderr)
|
||||
sys.exit(code)
|
||||
for dll in SDL3_DLLS: # dynamic SDL3 (zlib) and its libiconv (LGPL, unmodified DLL), found next to totk.exe
|
||||
if args.make_runtime_lib:
|
||||
print(f'{below_root(exe)}: {mib(exe.stat().st_size)}')
|
||||
return
|
||||
for dll in dlls: # dynamic SDL3 (zlib) and its libiconv (LGPL, unmodified DLL), found next to totk.exe
|
||||
shutil.copy2(dll, exe.parent / dll.name)
|
||||
sections, image = pe_sections(exe)
|
||||
print(f'{below_root(exe)}: file {mib(exe.stat().st_size)}, '
|
||||
|
||||
@@ -47,7 +47,9 @@ def list_files(staged: bool) -> list[Path]:
|
||||
return [Path(p) for p in out.decode("utf-8").split("\0") if p]
|
||||
|
||||
|
||||
def problems_for(path: Path) -> list[str]:
|
||||
def problems_for(path: Path, allow_large: bool = False) -> list[str]:
|
||||
"""allow_large: files over MAX_BYTES (toolchain binaries in a release folder) are checked for the generated marker
|
||||
only, not refused."""
|
||||
found = []
|
||||
if path.suffix.lower() in BLOCKED_SUFFIXES:
|
||||
found.append(f"blocked file type {path.suffix}")
|
||||
@@ -56,6 +58,10 @@ def problems_for(path: Path) -> list[str]:
|
||||
if not path.is_file():
|
||||
return found
|
||||
size = path.stat().st_size
|
||||
if size > MAX_BYTES and allow_large:
|
||||
if GENERATED_MARKER.encode() in path.read_bytes():
|
||||
found.append("generated from game code")
|
||||
return found
|
||||
if size > MAX_BYTES:
|
||||
found.append(f"{size // 1024} KiB, over the {MAX_BYTES // 1024} KiB limit")
|
||||
return found
|
||||
|
||||
+59
-18
@@ -9,7 +9,9 @@ Steps, each skipped when its verified output already exists:
|
||||
Steps 1 and 3 run for every file in `input.updates`. Each update's version is identified by the SHA-256 of its `main`,
|
||||
not by its file name. Output: build/extracted/v0/{exefs,romfs} and build/extracted/v<version>/{exefs,romfs}.
|
||||
|
||||
Runs under .venv-py314. Key values are read by the tools from the keys file and never printed.
|
||||
--base, --update, --keys and --work name the files and the output directory instead of the config (the installer
|
||||
imports the functions instead). Runs under .venv-py314. Key values are read by the tools from the keys file and never
|
||||
printed.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
@@ -45,6 +47,11 @@ def say(message):
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
def shown(path):
|
||||
"""path relative to the repository when it is below it (log lines), else as given."""
|
||||
return path.relative_to(ROOT) if path.is_relative_to(ROOT) else path
|
||||
|
||||
|
||||
def sha256(path):
|
||||
with path.open('rb') as f:
|
||||
return hashlib.file_digest(f, 'sha256').hexdigest()
|
||||
@@ -62,10 +69,17 @@ def version_dir(expected):
|
||||
return 'v0' if expected['version'] == 0 else 'v' + expected['display_version'].replace('.', '')
|
||||
|
||||
|
||||
# The embeddable Python (a ._pth file) does not put a script's directory on sys.path, so a vendored script that imports
|
||||
# its siblings fails when started by path. This stub adds the directory, then runs the script as __main__.
|
||||
RUN_SCRIPT = ('import os, runpy, sys; p = sys.argv[1]; sys.path.insert(0, os.path.dirname(p)); sys.argv = sys.argv[1:]; '
|
||||
"runpy.run_path(p, run_name='__main__')")
|
||||
|
||||
|
||||
def run_logged(command, log):
|
||||
args = [str(c) for c in command]
|
||||
with log.open('w') as out:
|
||||
code = subprocess.run([str(c) for c in command], stdout=out, stderr=subprocess.STDOUT).returncode
|
||||
require(code == 0, f'{Path(command[1]).name} failed; see {log.relative_to(ROOT)}')
|
||||
code = subprocess.run([args[0], '-c', RUN_SCRIPT, *args[1:]], stdout=out, stderr=subprocess.STDOUT).returncode
|
||||
require(code == 0, f'{Path(command[1]).name} failed; see {shown(log)}')
|
||||
|
||||
|
||||
def exefs_problems(exefs, expected):
|
||||
@@ -125,13 +139,9 @@ def extract_base_romfs(out, keys, logs):
|
||||
'--manifest', out / 'extraction-manifest.json', '--output', romfs], logs / 'extract_romfs-v0.log')
|
||||
|
||||
|
||||
def merge_update_romfs(base, update, keys_path, hactool, output, logs):
|
||||
"""extract_update_romfs.merge() without its Darwin-only bounded view of the base: the base Program NCA is copied out.
|
||||
|
||||
Needs room for both Program NCAs (about 17 GB together) and the merged image (about 17 GB).
|
||||
"""
|
||||
if output.exists():
|
||||
shutil.rmtree(output)
|
||||
def program_ncas(base, update, keys_path):
|
||||
"""(tickets, [(source, record, header)] of the base and the update Program NCA). Raises ValueError for a file
|
||||
that is no TotK base game or update, or keys that cannot open it."""
|
||||
keys = load_keys(keys_path)
|
||||
tickets = {}
|
||||
inputs = []
|
||||
@@ -146,6 +156,28 @@ def merge_update_romfs(base, update, keys_path, hactool, output, logs):
|
||||
rights = header[0x230:0x240]
|
||||
require(rights == bytes(16) or rights in tickets, 'Missing common ticket')
|
||||
inputs.append((source, record, header))
|
||||
return tickets, inputs
|
||||
|
||||
|
||||
def merge_space(base, update, keys_path):
|
||||
"""Bytes merge_update_romfs needs free next to its output, beyond the merged image itself: the two Program NCAs."""
|
||||
_, inputs = program_ncas(base, update, keys_path)
|
||||
return inputs[0][1]['size'] + inputs[1][1]['size']
|
||||
|
||||
|
||||
MERGE_PHASES = {'copy': ' copying the base and update Program NCAs', 'merge': ' merging with hactool',
|
||||
'index': ' hashing and indexing the merged image'}
|
||||
|
||||
|
||||
def merge_update_romfs(base, update, keys_path, hactool, output, logs, phase=lambda name: say(MERGE_PHASES[name])):
|
||||
"""extract_update_romfs.merge() without its Darwin-only bounded view of the base: the base Program NCA is copied out.
|
||||
|
||||
Needs room for both Program NCAs (about 17 GB together) and the merged image (about 17 GB). phase(name) is called
|
||||
as each part of MERGE_PHASES starts.
|
||||
"""
|
||||
if output.exists():
|
||||
shutil.rmtree(output)
|
||||
tickets, inputs = program_ncas(base, update, keys_path)
|
||||
sections = [i for i in range(4) if inputs[1][2][0x400 + i * 512 + 4] == 4]
|
||||
require(len(sections) == 1, 'Expected one BKTR section')
|
||||
need = inputs[0][1]['size'] + inputs[1][1]['size'] + 18 * (1 << 30)
|
||||
@@ -160,13 +192,13 @@ def merge_update_romfs(base, update, keys_path, hactool, output, logs):
|
||||
f.write(f'{rights.hex()} = {key.hex()}\n')
|
||||
hactool_keys = tmp / 'hactool.keys'
|
||||
hactool_prod_keys(keys_path, hactool_keys)
|
||||
say(' copying the base and update Program NCAs')
|
||||
phase('copy')
|
||||
base_nca, update_nca = tmp / 'base.nca', tmp / 'update.nca'
|
||||
for (source, record, _), target in zip(inputs, (base_nca, update_nca)):
|
||||
copy_range(source, record_offset(record), record['size'], target)
|
||||
result = tmp / 'result'
|
||||
result.mkdir()
|
||||
say(' merging with hactool')
|
||||
phase('merge')
|
||||
command = [str(hactool), '-x', '-y', '--disablekeywarns', '--suppresskeys', '-k', str(hactool_keys),
|
||||
f'--basenca={base_nca}', f'--section{sections[0]}={result / "romfs.bin"}', str(update_nca)]
|
||||
environment = dict(os.environ, MNG_TITLE_KEYS=str(title_keys))
|
||||
@@ -175,10 +207,10 @@ def merge_update_romfs(base, update, keys_path, hactool, output, logs):
|
||||
log_path.write_text(log)
|
||||
checked = '\n'.join(line for line in log.splitlines() if not UNSIGNED.match(line))
|
||||
require(completed.returncode == 0 and not HACTOOL_ERROR.search(checked),
|
||||
f'hactool merge failed; see {log_path.relative_to(ROOT)}')
|
||||
f'hactool merge failed; see {shown(log_path)}')
|
||||
require(all(re.search(rf'Level {i} \(GOOD\)', log) for i in range(6)), 'hactool did not verify all six IVFC levels')
|
||||
image = result / 'romfs.bin'
|
||||
say(' hashing and indexing the merged image')
|
||||
phase('index')
|
||||
files = index_romfs(image)
|
||||
report = dict(status='pass', romfs_size=image.stat().st_size, romfs_sha256=sha256(image), files=files,
|
||||
integrity='BKTR merge; all six IVFC levels verified')
|
||||
@@ -190,12 +222,21 @@ def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument('--config', type=Path, default=ROOT / 'config.local.toml')
|
||||
parser.add_argument('--hactool', type=Path, default=ROOT / 'toolchain' / 'hactool' / 'hactool.exe')
|
||||
parser.add_argument('--base', type=Path, help='base game (default: input.base of the config)')
|
||||
parser.add_argument('--update', type=Path, action='append',
|
||||
help='an update NSP, repeatable (default: input.updates of the config)')
|
||||
parser.add_argument('--keys', type=Path, help='prod.keys (default: input.keys of the config)')
|
||||
parser.add_argument('--work', type=Path, help='output directory with extracted/ and logs/ (default: work.dir of '
|
||||
'the config, else build)')
|
||||
args = parser.parse_args()
|
||||
|
||||
config = tomllib.loads(args.config.read_text(encoding='utf-8'))
|
||||
base, keys = Path(config['input']['base']), Path(config['input']['keys'])
|
||||
updates = config['input'].get('updates', [])
|
||||
work = ROOT / config.get('work', {}).get('dir', 'build')
|
||||
config = {}
|
||||
if not (args.base and args.keys and args.update and args.work):
|
||||
config = tomllib.loads(args.config.read_text(encoding='utf-8'))
|
||||
given = config.get('input', {})
|
||||
base, keys = args.base or Path(given['base']), args.keys or Path(given['keys'])
|
||||
updates = args.update or given.get('updates', [])
|
||||
work = args.work or ROOT / config.get('work', {}).get('dir', 'build')
|
||||
extracted, logs = work / 'extracted', work / 'logs'
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
versions = load_expected()
|
||||
|
||||
@@ -0,0 +1,509 @@
|
||||
"""Assemble the release folder of docs/installer.md ("Release layout") from pinned downloads.
|
||||
|
||||
.venv-py314\\Scripts\\python tools\\make_release.py --out build/release/TotK-Setup [--zip build/release/TotK-Setup.zip]
|
||||
.venv-py314\\Scripts\\python tools\\make_release.py --out build/release/TotK-Setup --prove
|
||||
|
||||
Needs Python 3.14 (tarfile reads the MSYS2 .zst packages), uv on PATH and network access on the first run. Downloads are
|
||||
cached in toolchain/release-dl/ and checked against the sha256 in PINS; a mismatch stops the build. The folder holds:
|
||||
|
||||
python/ python.org embeddable 3.14.9 + the packages of tools/requirements-installer.txt + tkinter (the Tcl/Tk 9.0
|
||||
files come out of the python.org installer's own component MSI, unpacked with msiexec /a and expand.exe;
|
||||
the installer is never run)
|
||||
llvm/ llvm-mingw, x86_64 only, the files clang and lld need to compile and link totk.exe
|
||||
runtime/ libtotkrt.a (build_game.py --make-runtime-lib with llvm/), the runtime headers, romfs.c (compiled per
|
||||
install), deps/ (headers and import libraries of SDL3, Opus, libvpx, Vulkan and VMA)
|
||||
bin/ totk_xlate.exe (+ .id), hactool.exe, SDL3.dll (upstream), libopus-0.dll, ninja.exe
|
||||
recomp/, tools/, data/, installer/, licenses/, THIRD_PARTY_NOTICES.md, totk.ini.example
|
||||
VERSION git describe of the checkout (else the short hash); the installer's release string
|
||||
TotK-Setup.exe installer/launcher/launcher.c built with llvm/ (-mwindows): starts python/pythonw.exe -m installer.gui
|
||||
|
||||
--prove compiles generated/v121 and links build/release-check/totk.exe with nothing but the folder's python/, llvm/,
|
||||
runtime/ and bin/ (PATH holds the Windows system directory and those), then runs the four probes there. It needs a
|
||||
generated/v121 and build/extracted/v121 from the usual game build (they are read, nothing is written to them).
|
||||
|
||||
The folder contains nothing derived from the game: check_content runs over it before the script finishes. Then the
|
||||
folder's own python runs a check-only installer pass (no game files: it must get through the release, toolchain and host
|
||||
checks and stop at the missing game files) and the window's --selftest.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import check_content # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/, generated/, toolchain/ and refs/ are junctions in a worktree
|
||||
DL = MAIN_ROOT / 'toolchain' / 'release-dl'
|
||||
WORK = MAIN_ROOT / 'toolchain' / 'release-work' # unpacked downloads
|
||||
|
||||
MSYS2 = 'https://repo.msys2.org/mingw/ucrt64/'
|
||||
SDL_VERSION = '3.4.18'
|
||||
LLVM_VERSION = '20261020'
|
||||
MNG_COMMIT = 'b75be8cfa704df3701e86130c8ced268e79d6b11' # MacNativeGames, the shader list (refs/MacNativeGames)
|
||||
# name: (url, sha256)
|
||||
PINS = {
|
||||
'python-embed': ('https://www.python.org/ftp/python/3.14.9/python-3.14.9-embed-amd64.zip',
|
||||
'3c01d7cfc666f746a6f064f60fb2156ab0ec464d3d738ab796371a46a9bb24a2'),
|
||||
'python-installer': ('https://www.python.org/ftp/python/3.14.9/python-3.14.9-amd64.exe',
|
||||
'a247a0b612244a803f507b39fb7121e3602e4160d59c3b99d98e735289c53394'),
|
||||
'llvm-mingw': (f'https://github.com/mstorsjo/llvm-mingw/releases/download/{LLVM_VERSION}/'
|
||||
f'llvm-mingw-{LLVM_VERSION}-ucrt-x86_64.zip',
|
||||
'fd5462b7f5d93b2ff981fcd3f9c7e427e47f3203c64ea91c27bcc444eebd74cd'),
|
||||
'sdl3-dll': (f'https://github.com/libsdl-org/SDL/releases/download/release-{SDL_VERSION}/SDL3-{SDL_VERSION}-win32-x64.zip',
|
||||
'75c2c0fc74e7d1206aaedc22893a35887332f4f85f5c8b74fddc10d177985f2c'),
|
||||
'sdl3-devel': (f'https://github.com/libsdl-org/SDL/releases/download/release-{SDL_VERSION}/SDL3-devel-{SDL_VERSION}-mingw.tar.gz',
|
||||
'049b9dd711e3fb3f8d84feb998dfec54a75e17cf539c4a80ff4d058d839771cf'),
|
||||
'ninja': ('https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-win.zip',
|
||||
'07fc8261b42b20e71d1720b39068c2e14ffcee6396b76fb7a795fb460b78dc65'),
|
||||
'opus': (MSYS2 + 'mingw-w64-ucrt-x86_64-opus-1.6.1-2-any.pkg.tar.zst',
|
||||
'bb114bcf26225f50f874497ebfd6b8224fff2a20bda3f8ee45cd07ce5f02657b'),
|
||||
'libvpx': (MSYS2 + 'mingw-w64-ucrt-x86_64-libvpx-1.17.0-1-any.pkg.tar.zst',
|
||||
'8ff6919dbb88ea26ce28d22fd81d6f0d6e200cf56a4dd81ffc2d33124fe98024'),
|
||||
'libvpx-src': ('https://github.com/webmproject/libvpx/archive/refs/tags/v1.17.0.tar.gz', # LICENSE and PATENTS only
|
||||
'1020f184046187baa2985dbde38e0691f49c44088bca7a1842b0236c6081dc0a'),
|
||||
'vulkan-headers': (MSYS2 + 'mingw-w64-ucrt-x86_64-vulkan-headers-1~1.4.363.0-1-any.pkg.tar.zst',
|
||||
'be602b2657cace531316e78b1eda4087e8f7f84ebb877f08fba198a2db06406a'),
|
||||
'vulkan-loader': (MSYS2 + 'mingw-w64-ucrt-x86_64-vulkan-loader-1~1.4.363.0-1-any.pkg.tar.zst', # import library only
|
||||
'065076fe13d39ab97eea4740596426d5ae741b5044e48e1c9495d04835c92dbe'),
|
||||
'vma': (MSYS2 + 'mingw-w64-ucrt-x86_64-vulkan-memory-allocator-3.4.0-1-any.pkg.tar.zst',
|
||||
'44c81162337915afecda3185f66fd98268a54454580b2717080c9dfef57fe781'),
|
||||
}
|
||||
# The component MSI inside the python.org installer that carries Tcl/Tk and tkinter (Burn names its payloads a0..a27).
|
||||
TCLTK_PAYLOAD = ('a12', 'ac87609db3f715404dd41a989c2f1706ea1512f3d86217f938d11c63af8eb505')
|
||||
SDL3_DLL_SHA256 = '2b4b128d8d510cc2d442693cf848c050d700d43dc667afe9e5fd82b0570d667e' # SDL3.dll inside the SDL zip
|
||||
|
||||
# llvm-mingw: what clang and lld need to compile and link totk.exe for x86_64-w64-mingw32 (dynamic LLVM, libc++, compiler-rt).
|
||||
LLVM_BIN = {'clang.exe', 'clang++.exe', 'clang-23.exe', 'ld.lld.exe', 'llvm-ar.exe', 'llvm-ranlib.exe', 'llvm-nm.exe',
|
||||
'llvm-objdump.exe', 'llvm-objcopy.exe', 'llvm-strip.exe', 'llvm-readobj.exe', 'llvm-dlltool.exe',
|
||||
'llvm-rc.exe', 'llvm-cvtres.exe', 'llvm-windres.exe', 'llvm-wrapper.exe', 'mingw32-common.cfg',
|
||||
'x86_64-w64-windows-gnu.cfg', 'x86_64-pc-windows-gnu.cfg', 'x86_64-w64-mingw32-clang.exe',
|
||||
'x86_64-w64-mingw32-clang++.exe', 'libLLVM-23.dll', 'libclang-cpp.dll', 'libc++.dll', 'libunwind.dll',
|
||||
'libwinpthread-1.dll', 'libffi-8.dll'}
|
||||
LLVM_SKIP_DIRS = {'aarch64-w64-mingw32', 'armv7-w64-mingw32', 'i686-w64-mingw32', 'busybox', 'python', 'share'}
|
||||
LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'}
|
||||
|
||||
# tools/ files the installer steps run (and what they import), copied from the repo.
|
||||
TOOLS_FILES = ['build_game.py', 'check_content.py', 'extract.py', 'make_romfs_patches.py', 'prepare_shaders.py',
|
||||
'romfs_index.py', 'shader_census.py']
|
||||
TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes
|
||||
RUNTIME_SUFFIXES = {'.h', '.inc'}
|
||||
|
||||
PROBES = ['--probe-kernel', '--probe-vkexec', '--probe-nvn-tex', '--probe-shader']
|
||||
|
||||
|
||||
def sha256_file(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, 'rb') as f:
|
||||
for block in iter(lambda: f.read(1 << 20), b''):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def fetch(name):
|
||||
"""The pinned download, from the cache or the network, checked against its sha256."""
|
||||
url, want = PINS[name]
|
||||
DL.mkdir(parents=True, exist_ok=True)
|
||||
path = DL / Path(url.split('?')[0]).name.replace('%7E', '~')
|
||||
if name == 'libvpx-src':
|
||||
path = DL / 'libvpx-1.17.0.tar.gz'
|
||||
if not path.is_file():
|
||||
print(f'download {url}')
|
||||
with urllib.request.urlopen(url) as r, open(path, 'wb') as f:
|
||||
shutil.copyfileobj(r, f)
|
||||
got = sha256_file(path)
|
||||
if got != want:
|
||||
sys.exit(f'{path}: sha256 {got}, pinned {want}')
|
||||
return path
|
||||
|
||||
|
||||
def unpack(name, member_filter=None):
|
||||
"""Extract a pinned archive into toolchain/release-work/<name> (once) and return that directory."""
|
||||
dest = WORK / name
|
||||
stamp = dest / '.unpacked'
|
||||
if stamp.is_file():
|
||||
return dest
|
||||
shutil.rmtree(dest, ignore_errors=True)
|
||||
dest.mkdir(parents=True)
|
||||
path = fetch(name)
|
||||
if path.suffix == '.zip':
|
||||
with zipfile.ZipFile(path) as z:
|
||||
z.extractall(dest)
|
||||
else:
|
||||
with tarfile.open(path) as t:
|
||||
t.extractall(dest, filter='data')
|
||||
stamp.write_text('')
|
||||
return dest
|
||||
|
||||
|
||||
def copy_tree(src, dst, skip=lambda rel: False):
|
||||
"""Copy src into dst file by file (sorted), leaving out what skip(relative posix path) names."""
|
||||
for p in sorted(src.rglob('*')):
|
||||
rel = p.relative_to(src)
|
||||
if p.is_dir() or skip(rel.as_posix()):
|
||||
continue
|
||||
(dst / rel).parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(p, dst / rel)
|
||||
|
||||
|
||||
def clean_env(*path_dirs):
|
||||
"""A process environment without MSYS2 and the user's tools: the Windows system directory and path_dirs only."""
|
||||
system = os.environ.get('SystemRoot', r'C:\Windows')
|
||||
keep = ('SYSTEMROOT', 'SYSTEMDRIVE', 'TEMP', 'TMP', 'USERPROFILE', 'LOCALAPPDATA', 'APPDATA', 'COMSPEC', 'PATHEXT',
|
||||
'NUMBER_OF_PROCESSORS') # os.environ keys are upper case on Windows
|
||||
env = {k: v for k, v in os.environ.items() if k.upper() in keep}
|
||||
env['PATH'] = os.pathsep.join([*(str(Path(d)) for d in path_dirs), rf'{system}\System32', system])
|
||||
return env
|
||||
|
||||
|
||||
def run(cmd, **kw):
|
||||
print('+', ' '.join(str(c) for c in cmd), flush=True)
|
||||
return subprocess.run([str(c) for c in cmd], check=True, **kw)
|
||||
|
||||
|
||||
def mib(n):
|
||||
return f'{n / (1 << 20):.1f} MiB'
|
||||
|
||||
|
||||
def dir_size(p):
|
||||
return sum(f.stat().st_size for f in Path(p).rglob('*') if f.is_file())
|
||||
|
||||
|
||||
# ---- python/ -------------------------------------------------------------------------------------------------------
|
||||
|
||||
def tcltk_files():
|
||||
"""Directory holding the python.org installer's tcltk payload, unpacked (not installed) into toolchain/release-work."""
|
||||
dest = WORK / 'tcltk'
|
||||
if (dest / 'DLLs' / '_tkinter.pyd').is_file():
|
||||
return dest
|
||||
exe = fetch('python-installer')
|
||||
data = exe.read_bytes()
|
||||
cabs = []
|
||||
pos = data.find(b'MSCF\0\0\0\0')
|
||||
while pos >= 0:
|
||||
cabs.append((struct.unpack_from('<I', data, pos + 8)[0], pos))
|
||||
pos = data.find(b'MSCF\0\0\0\0', pos + 1)
|
||||
size, pos = max(cabs) # the attached container holds all component MSIs
|
||||
shutil.rmtree(WORK / 'tcltk-src', ignore_errors=True)
|
||||
src = WORK / 'tcltk-src'
|
||||
src.mkdir(parents=True)
|
||||
(src / 'container.cab').write_bytes(data[pos:pos + size])
|
||||
name, want = TCLTK_PAYLOAD
|
||||
expand = Path(os.environ.get('SystemRoot', r'C:\Windows')) / 'System32' / 'expand.exe'
|
||||
run([expand, '-F:' + name, src / 'container.cab', src], stdout=subprocess.DEVNULL)
|
||||
got = sha256_file(src / name)
|
||||
if got != want:
|
||||
sys.exit(f'{src / name}: sha256 {got}, pinned {want} (not the tcltk component of python 3.14.9)')
|
||||
msi = src / (name + '.msi')
|
||||
(src / name).rename(msi)
|
||||
shutil.rmtree(dest, ignore_errors=True)
|
||||
# Administrative image: copies the MSI's files to TARGETDIR, installs nothing and shows no window.
|
||||
run(['msiexec', '/a', msi, '/qn', f'TARGETDIR={dest}'])
|
||||
if not (dest / 'DLLs' / '_tkinter.pyd').is_file():
|
||||
sys.exit(f'{dest}: no _tkinter.pyd after msiexec /a')
|
||||
return dest
|
||||
|
||||
|
||||
def build_python(out):
|
||||
py = out / 'python'
|
||||
with zipfile.ZipFile(fetch('python-embed')) as z:
|
||||
z.extractall(py)
|
||||
(py / 'python314._pth').write_text('python314.zip\n.\nLib/site-packages\n..\n../tools\n../recomp\n', encoding='utf-8')
|
||||
# The embeddable Python ignores the script directory and PYTHONPATH once a ._pth file exists, so the folder's own code
|
||||
# (installer/, and tools/*.py and recomp/*.py importing their siblings) has to be on the list.
|
||||
env = dict(os.environ, UV_CACHE_DIR=str(MAIN_ROOT / 'toolchain' / 'uv-cache'))
|
||||
run(['uv', 'pip', 'install', '--target', py / 'Lib' / 'site-packages', '--python', sys.executable,
|
||||
'--python-platform', 'windows', '--python-version', '3.14', '--only-binary', ':all:',
|
||||
'-r', ROOT / 'tools' / 'requirements-installer.txt'], env=env)
|
||||
shutil.rmtree(py / 'Lib' / 'site-packages' / 'bin', ignore_errors=True) # numpy/cffi console scripts
|
||||
tk = tcltk_files()
|
||||
for dll in ('_tkinter.pyd', 'tcl90.dll', 'tcl9tk90.dll', 'zlib1.dll'):
|
||||
shutil.copy2(tk / 'DLLs' / dll, py / dll)
|
||||
copy_tree(tk / 'Lib' / 'tkinter', py / 'tkinter', lambda r: r.startswith('__pycache__'))
|
||||
copy_tree(tk / 'tcl', py / 'tcl', lambda r: r.endswith(('.lib', '.sh')) or r.startswith('nmake'))
|
||||
check = subprocess.run([str(py / 'python.exe'), '-I', '-c',
|
||||
'import tkinter, numpy, lz4.block, cryptography, cffi, pycparser; '
|
||||
'print("python ok: Tcl", tkinter.Tcl().eval("info patchlevel"), "numpy", numpy.__version__)'],
|
||||
env=clean_env(py), capture_output=True, text=True)
|
||||
if check.returncode:
|
||||
sys.exit('bundled python check failed:\n' + check.stderr)
|
||||
print(check.stdout.strip())
|
||||
|
||||
|
||||
# ---- llvm/ ---------------------------------------------------------------------------------------------------------
|
||||
|
||||
def build_llvm(out):
|
||||
src = unpack('llvm-mingw') / f'llvm-mingw-{LLVM_VERSION}-ucrt-x86_64'
|
||||
dst = out / 'llvm'
|
||||
|
||||
def skip(rel):
|
||||
top = rel.split('/')[0]
|
||||
if top in LLVM_SKIP_DIRS:
|
||||
return True
|
||||
if top == 'bin':
|
||||
return rel not in {f'bin/{n}' for n in LLVM_BIN}
|
||||
if top == 'lib': # lib/clang/<major>/include and the x86_64 Windows builtins; no sanitizers, no other targets
|
||||
if rel.startswith('lib/clang/') and '/include/' in rel:
|
||||
return False
|
||||
return not rel.endswith('/lib/windows/libclang_rt.builtins-x86_64.a')
|
||||
return Path(rel).suffix.lower() in LLVM_SKIP_SUFFIXES or rel.startswith('include/ddk/')
|
||||
|
||||
copy_tree(src, dst, skip)
|
||||
return dst
|
||||
|
||||
|
||||
# ---- bin/ and deps ---------------------------------------------------------------------------------------------------
|
||||
|
||||
def build_bin(out):
|
||||
b = out / 'bin'
|
||||
b.mkdir(parents=True, exist_ok=True)
|
||||
for f in ('totk_xlate.exe', 'totk_xlate.id'):
|
||||
shutil.copy2(MAIN_ROOT / 'build' / 'xlate' / f, b / f)
|
||||
shutil.copy2(MAIN_ROOT / 'toolchain' / 'hactool' / 'hactool.exe', b / 'hactool.exe')
|
||||
dll = unpack('sdl3-dll') / 'SDL3.dll'
|
||||
if sha256_file(dll) != SDL3_DLL_SHA256:
|
||||
sys.exit(f'{dll}: not the pinned SDL3.dll')
|
||||
shutil.copy2(dll, b / 'SDL3.dll')
|
||||
shutil.copy2(unpack('opus') / 'ucrt64' / 'bin' / 'libopus-0.dll', b / 'libopus-0.dll')
|
||||
shutil.copy2(unpack('ninja') / 'ninja.exe', b / 'ninja.exe')
|
||||
|
||||
|
||||
def build_deps(out):
|
||||
"""runtime/deps: the headers and import libraries build_game.py --deps expects (include/, lib/)."""
|
||||
d = out / 'runtime' / 'deps'
|
||||
(d / 'include').mkdir(parents=True, exist_ok=True)
|
||||
(d / 'lib').mkdir(parents=True, exist_ok=True)
|
||||
sdl = unpack('sdl3-devel') / f'SDL3-{SDL_VERSION}' / 'x86_64-w64-mingw32'
|
||||
copy_tree(sdl / 'include', d / 'include')
|
||||
shutil.copy2(sdl / 'lib' / 'libSDL3.dll.a', d / 'lib')
|
||||
for pkg, dirs, libs in (('opus', ['opus'], ['libopus.dll.a']), ('libvpx', ['vpx'], ['libvpx.a']),
|
||||
('vulkan-headers', ['vulkan', 'vk_video'], []), ('vulkan-loader', [], ['libvulkan-1.dll.a']),
|
||||
('vma', ['vma'], [])):
|
||||
root = unpack(pkg) / 'ucrt64'
|
||||
for inc in dirs:
|
||||
copy_tree(root / 'include' / inc, d / 'include' / inc)
|
||||
for lib in libs:
|
||||
shutil.copy2(root / 'lib' / lib, d / 'lib')
|
||||
|
||||
|
||||
def collect_licenses(out):
|
||||
lic = out / 'licenses'
|
||||
lic.mkdir(parents=True, exist_ok=True)
|
||||
pairs = [
|
||||
(unpack('opus') / 'ucrt64' / 'share' / 'licenses' / 'opus' / 'COPYING', 'opus-COPYING.txt'),
|
||||
(unpack('vulkan-headers') / 'ucrt64' / 'share' / 'licenses' / 'vulkan-headers' / 'LICENSE', 'vulkan-headers-LICENSE.txt'),
|
||||
(unpack('vulkan-loader') / 'ucrt64' / 'share' / 'licenses' / 'vulkan-loader' / 'LICENSE', 'vulkan-loader-LICENSE.txt'),
|
||||
(unpack('vma') / 'ucrt64' / 'share' / 'licenses' / 'VulkanMemoryAllocator' / 'LICENSE', 'vulkan-memory-allocator-LICENSE.txt'),
|
||||
(unpack('sdl3-dll') / 'LICENSE.txt', 'SDL3-LICENSE.txt'),
|
||||
(MAIN_ROOT / 'toolchain' / 'hactool' / 'LICENSE', 'hactool-LICENSE.txt'),
|
||||
(MAIN_ROOT / 'refs' / 'MacNativeGames' / 'LICENSE', 'MacNativeGames-LICENSE.txt'),
|
||||
]
|
||||
for src, name in pairs:
|
||||
shutil.copy2(src, lic / name)
|
||||
with tarfile.open(fetch('libvpx-src')) as t: # the MSYS2 package carries no licence text; the notice needs both
|
||||
for member, name in (('libvpx-1.17.0/LICENSE', 'libvpx-LICENSE.txt'), ('libvpx-1.17.0/PATENTS', 'libvpx-PATENTS.txt')):
|
||||
(lic / name).write_bytes(t.extractfile(member).read())
|
||||
for name in ('LICENSE.TXT',):
|
||||
shutil.copy2(out / 'llvm' / name, lic / 'llvm-mingw-LICENSE.txt')
|
||||
|
||||
|
||||
# ---- runtime/, recomp/, tools/, data/ ---------------------------------------------------------------------------------
|
||||
|
||||
def runtime_lib(out, llvm):
|
||||
"""Build libtotkrt.a with the folder's llvm/ and copy it, the headers and romfs.c into runtime/."""
|
||||
build = MAIN_ROOT / 'build' / 'release-check' / 'rtlib'
|
||||
gen = MAIN_ROOT / 'generated' / 'v121'
|
||||
run([sys.executable, ROOT / 'tools' / 'build_game.py', '--make-runtime-lib', '--toolchain', llvm,
|
||||
'--deps', out / 'runtime' / 'deps', '--ninja', out / 'bin' / 'ninja.exe', '--gen', gen, '--out', build],
|
||||
env=clean_env(llvm / 'bin', out / 'bin'))
|
||||
rt = out / 'runtime'
|
||||
shutil.copy2(build / 'libtotkrt.a', rt / 'libtotkrt.a')
|
||||
for p in sorted((ROOT / 'runtime').rglob('*')):
|
||||
if p.suffix in RUNTIME_SUFFIXES and p.is_file():
|
||||
(rt / p.relative_to(ROOT / 'runtime')).parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(p, rt / p.relative_to(ROOT / 'runtime'))
|
||||
shutil.copy2(ROOT / 'runtime' / 'romfs.c', rt / 'romfs.c') # build_game.py INSTALL_SIDE
|
||||
|
||||
|
||||
def build_sources(out):
|
||||
copy_tree(ROOT / 'recomp', out / 'recomp', lambda r: not r.endswith(('.py', '.txt')) or '__pycache__' in r)
|
||||
for f in TOOLS_FILES:
|
||||
(out / 'tools').mkdir(exist_ok=True)
|
||||
shutil.copy2(ROOT / 'tools' / f, out / 'tools' / f)
|
||||
for d in TOOLS_DIRS:
|
||||
copy_tree(ROOT / 'tools' / d, out / 'tools' / d, lambda r: '__pycache__' in r or r.endswith('.md'))
|
||||
(out / 'data').mkdir(exist_ok=True)
|
||||
text = subprocess.run(['git', '-C', MAIN_ROOT / 'refs' / 'MacNativeGames', 'show',
|
||||
f'{MNG_COMMIT}:games/totk/assets/shader_list.tsv'], check=True, capture_output=True).stdout
|
||||
(out / 'data' / 'shader_list.tsv').write_bytes(text)
|
||||
copy_tree(ROOT / 'installer', out / 'installer', lambda r: '__pycache__' in r or r.startswith('launcher/'))
|
||||
shutil.copy2(ROOT / 'totk.ini.example', out / 'totk.ini.example')
|
||||
shutil.copy2(ROOT / 'THIRD_PARTY_NOTICES.md', out / 'THIRD_PARTY_NOTICES.md')
|
||||
(out / 'VERSION').write_text(version() + '\n', encoding='utf-8')
|
||||
|
||||
|
||||
def version():
|
||||
"""git describe of the checkout, else its short hash."""
|
||||
for args in (['describe', '--tags', '--always'], ['rev-parse', '--short', 'HEAD']):
|
||||
r = subprocess.run(['git', '-C', ROOT, *args], capture_output=True, text=True)
|
||||
if r.returncode == 0 and r.stdout.strip():
|
||||
return r.stdout.strip()
|
||||
sys.exit('no git version: not a git checkout')
|
||||
|
||||
|
||||
# ---- TotK-Setup.exe --------------------------------------------------------------------------------------------------
|
||||
|
||||
def c_wide(text):
|
||||
"""A C wide string literal of text: ASCII as is, everything else as a universal character name."""
|
||||
bs = chr(92)
|
||||
out = []
|
||||
for ch in text:
|
||||
if ch in (bs, '"'):
|
||||
out.append(bs + ch)
|
||||
elif 32 <= ord(ch) < 127:
|
||||
out.append(ch)
|
||||
elif ord(ch) < 0x10000:
|
||||
out.append(f'{bs}u{ord(ch):04x}')
|
||||
else:
|
||||
sys.exit(f'launcher text: {ch!r} is outside the BMP')
|
||||
return 'L"' + ''.join(out) + '"'
|
||||
|
||||
|
||||
def launcher_header(path):
|
||||
"""launcher_text.h: the launcher's two messages in English and Russian, taken from the installer's catalog."""
|
||||
lines = ['/* generated by tools/make_release.py from installer/text/{en,ru}.json */']
|
||||
for lang in ('en', 'ru'):
|
||||
texts = json.loads((ROOT / 'installer' / 'text' / f'{lang}.json').read_text(encoding='utf-8'))
|
||||
for key in ('title', 'python_missing'):
|
||||
lines.append(f'#define LAUNCHER_{key.upper()}_{lang.upper()} {c_wide(texts["launcher." + key])}')
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text('\n'.join(lines) + '\n', encoding='ascii')
|
||||
|
||||
|
||||
def build_launcher(out, llvm):
|
||||
work = MAIN_ROOT / 'build' / 'release-check' / 'launcher'
|
||||
launcher_header(work / 'launcher_text.h')
|
||||
run([llvm / 'bin' / 'clang.exe', '-O2', '-mwindows', '-Wall', '-Wl,-s', '-I', work,
|
||||
ROOT / 'installer' / 'launcher' / 'launcher.c', '-o', out / 'TotK-Setup.exe'], env=clean_env(llvm / 'bin'))
|
||||
|
||||
|
||||
# ---- checks ----------------------------------------------------------------------------------------------------------
|
||||
|
||||
def check_folder(out):
|
||||
"""check_content over every file of the release folder; a hit stops the build."""
|
||||
bad = []
|
||||
for p in sorted(out.rglob('*')):
|
||||
if p.is_file():
|
||||
bad += [f'{p.relative_to(out)}: {problem}' for problem in check_content.problems_for(p, allow_large=True)]
|
||||
if bad:
|
||||
sys.exit('game-derived content in the release folder:\n' + '\n'.join(bad))
|
||||
print(f'check_content: {sum(1 for p in out.rglob("*") if p.is_file())} files, no hit')
|
||||
|
||||
|
||||
def check_bundle(out):
|
||||
"""The folder's own python runs the installer without game files and the window's selftest."""
|
||||
chk = MAIN_ROOT / 'build' / 'release-check' / 'bundle'
|
||||
shutil.rmtree(chk, ignore_errors=True)
|
||||
env = clean_env(out / 'python')
|
||||
py = out / 'python' / 'python.exe'
|
||||
r = subprocess.run([py, '-m', 'installer', '--lang', 'en-US', '--dest', chk / 'install', '--temp', chk / 'temp'],
|
||||
cwd=out, env=env, capture_output=True, text=True, timeout=600)
|
||||
events = [json.loads(line) for line in r.stdout.splitlines() if line.startswith('{')]
|
||||
errors = [e['id'] for e in events if e['type'] == 'error']
|
||||
started = any(e['type'] == 'start' and e.get('release') == (out / 'VERSION').read_text().strip() for e in events)
|
||||
steps = [e['phase'] for e in events if e['type'] == 'progress']
|
||||
if errors != ['check.inputs_needed'] or not started or 'inputs' not in steps:
|
||||
sys.exit(f'bundle check: exit {r.returncode}, errors {errors}, release read {started}\n{r.stdout[-2000:]}\n{r.stderr[-2000:]}')
|
||||
print(f'bundle check: the installer ran the host check from the folder and stopped at the game files (exit {r.returncode})')
|
||||
r = subprocess.run([py, '-m', 'installer.gui', '--selftest'], cwd=out, env=env, capture_output=True, text=True, timeout=300)
|
||||
if r.returncode:
|
||||
sys.exit(f'bundle check: window selftest exit {r.returncode}\n{r.stdout[-2000:]}\n{r.stderr[-2000:]}')
|
||||
print('bundle check: the window built every screen in both languages and closed')
|
||||
|
||||
|
||||
def sizes(out):
|
||||
rows = [(p.name if p.is_dir() else p.name, dir_size(p) if p.is_dir() else p.stat().st_size) for p in sorted(out.iterdir())]
|
||||
for name, n in rows:
|
||||
print(f' {name:26} {mib(n):>12}')
|
||||
print(f' {"total":26} {mib(sum(n for _, n in rows)):>12}')
|
||||
|
||||
|
||||
def write_zip(out, path):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(path, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as z:
|
||||
for p in sorted(out.rglob('*')):
|
||||
if p.is_file():
|
||||
info = zipfile.ZipInfo((Path(out.name) / p.relative_to(out)).as_posix(), (1980, 1, 1, 0, 0, 0))
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
info.external_attr = 0o644 << 16
|
||||
z.writestr(info, p.read_bytes(), compresslevel=6)
|
||||
print(f'{path}: {mib(path.stat().st_size)}')
|
||||
|
||||
|
||||
def prove(out):
|
||||
"""Compile generated/v121 and link totk.exe with the folder's tools only, then run the probes next to it."""
|
||||
out = out.resolve()
|
||||
gen = MAIN_ROOT / 'generated' / 'v121'
|
||||
ext = MAIN_ROOT / 'build' / 'extracted' / 'v121'
|
||||
chk = MAIN_ROOT / 'build' / 'release-check'
|
||||
env = clean_env(out / 'python', out / 'llvm' / 'bin', out / 'bin')
|
||||
t0 = time.time()
|
||||
run([out / 'python' / 'python.exe', out / 'tools' / 'build_game.py', '--toolchain', out / 'llvm',
|
||||
'--deps', out / 'runtime' / 'deps', '--runtime', out / 'runtime', '--runtime-lib', out / 'runtime' / 'libtotkrt.a',
|
||||
'--ninja', out / 'bin' / 'ninja.exe', '--gen', gen, '--out', chk, '--exefs', ext / 'exefs', '--romfs', ext / 'romfs',
|
||||
'--profile', chk / 'profile'], env=env)
|
||||
print(f'prove: build {time.time() - t0:.0f} s')
|
||||
for dll in ('SDL3.dll', 'libopus-0.dll'):
|
||||
shutil.copy2(out / 'bin' / dll, chk / dll)
|
||||
failed = []
|
||||
for probe in PROBES:
|
||||
r = subprocess.run([str(chk / 'totk.exe'), probe], cwd=chk, env=clean_env(chk), capture_output=True, text=True,
|
||||
timeout=900)
|
||||
tail = (r.stdout.strip().splitlines() or [''])[-1]
|
||||
print(f'{probe}: exit {r.returncode}: {tail}')
|
||||
if r.returncode:
|
||||
failed.append(probe)
|
||||
if failed:
|
||||
sys.exit(f'probes failed: {failed}')
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--out', type=Path, required=True, help='release folder (replaced)')
|
||||
ap.add_argument('--zip', type=Path, help='also write the folder as this zip')
|
||||
ap.add_argument('--prove', action='store_true', help='only run the compile-and-probe check on an existing folder')
|
||||
args = ap.parse_args()
|
||||
out = args.out.resolve()
|
||||
if args.prove:
|
||||
prove(out)
|
||||
return
|
||||
if out.exists():
|
||||
shutil.rmtree(out)
|
||||
out.mkdir(parents=True)
|
||||
build_python(out)
|
||||
llvm = build_llvm(out)
|
||||
build_bin(out)
|
||||
build_deps(out)
|
||||
runtime_lib(out, llvm)
|
||||
build_sources(out)
|
||||
build_launcher(out, llvm)
|
||||
collect_licenses(out)
|
||||
check_folder(out)
|
||||
check_bundle(out)
|
||||
sizes(out)
|
||||
if args.zip:
|
||||
write_zip(out, args.zip.resolve())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -16,10 +16,12 @@ pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "sta
|
||||
a 12-core machine (inventory 1, translation 3, library builds 0.6) and 1.3 GB of shader cache; the seed files can be
|
||||
deleted afterwards.
|
||||
|
||||
Game-derived output stays in the build directory (ignored). The list is read from refs/ at preparation time; nothing
|
||||
of it is copied into the repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314:
|
||||
Game-derived output stays in the build directory (ignored). The list is read at preparation time from tools/data/shader_list.tsv when that file exists (the installer archive
|
||||
bundles it; it stays out of the repository), else from refs/ through git; nothing of it is copied into the
|
||||
repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14:
|
||||
|
||||
.venv-py314\\Scripts\\python tools\\prepare_shaders.py build\\v121 [--version v121] [--list <tsv>] [--no-run]
|
||||
[--romfs DIR --exefs DIR] # instead of build\\extracted\\<version>
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
@@ -50,9 +52,16 @@ CONTROL = 0x880
|
||||
ANS_REC = 4 + 32 + 32 + 147 + 1
|
||||
|
||||
|
||||
BUNDLED_LIST = ROOT / 'tools' / 'data' / 'shader_list.tsv' # the installer archive carries a copy (no git, no refs/)
|
||||
if not BUNDLED_LIST.is_file():
|
||||
BUNDLED_LIST = ROOT / 'data' / 'shader_list.tsv' # release folder layout (tools/make_release.py)
|
||||
|
||||
|
||||
def read_list(path):
|
||||
if path:
|
||||
return Path(path).read_text(encoding='utf-8')
|
||||
if BUNDLED_LIST.is_file():
|
||||
return BUNDLED_LIST.read_text(encoding='utf-8')
|
||||
return subprocess.run(['git', '-C', str(ROOT / 'refs' / 'MacNativeGames'), 'show', ':'.join(LIST_REF)],
|
||||
check=True, capture_output=True, encoding='utf-8').stdout
|
||||
|
||||
@@ -111,13 +120,15 @@ def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('build', type=Path, help='build directory holding totk.exe (build\\v121)')
|
||||
ap.add_argument('--version', default='v121', help='build/extracted/<version>')
|
||||
ap.add_argument('--list', help='a shader_list.tsv instead of the one in refs/MacNativeGames')
|
||||
ap.add_argument('--romfs', type=Path, help='RomFS directory (default build/extracted/<version>/romfs)')
|
||||
ap.add_argument('--exefs', type=Path, help='ExeFS directory (default build/extracted/<version>/exefs)')
|
||||
ap.add_argument('--list', help='a shader_list.tsv instead of tools/data/shader_list.tsv or refs/MacNativeGames')
|
||||
ap.add_argument('--no-run', action='store_true', help='write the seed files only')
|
||||
a = ap.parse_args()
|
||||
t0 = time.monotonic()
|
||||
variants, prio = parse_list(read_list(a.list))
|
||||
extracted = ROOT / 'build' / 'extracted' / a.version
|
||||
shaders, _ = inventory(extracted / 'romfs', extracted / 'exefs')
|
||||
shaders, _ = inventory(a.romfs or extracted / 'romfs', a.exefs or extracted / 'exefs')
|
||||
t1 = time.monotonic()
|
||||
progs, recs, skipped = {}, [], {'not in this version': 0, 'stage': 0}
|
||||
for key, deps in variants:
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# Packages of the release's bundled Python 3.14.9 (tools/make_release.py): what the installer steps import.
|
||||
# numpy: recomp/gen.py. lz4: recomp/nso.py, tools/shader_census.py. cryptography, cffi, pycparser: tools/extract.py and the vendored extract scripts.
|
||||
numpy==2.5.4
|
||||
lz4==4.4.5
|
||||
cryptography==50.0.2
|
||||
cffi==2.1.1
|
||||
pycparser==3.11
|
||||
Reference in new issue
Block a user