Merge game version 1.4.3: loader, patch table, SDK 20.5 layouts and probes, the installer choosing 1.2.1 or 1.4.3 by hash, and shader preparation of every shader's default variant plus the list rows MacNativeGames pruned

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-10-26 20:03:45 +03:00
commit cdecd014f6
32 files changed
+691 -177

No files matched your search

+3 -2
View File
@@ -43,9 +43,10 @@ Setup and verify commands: [docs/toolchain-windows.md](docs/toolchain-windows.md
- `tests\diff\atomic.py`: exclusive load/store sequences and ARMv8.1 atomics; needs no game files.
- `tests\diff\func.py`: whole leaf functions as the generator emits them, loops and jump tables included.
- `tests\excl_mt.py`: the exclusive monitor helpers of `cpu.h` under contention and ABA, on several threads; needs no game files.
- Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle.
- Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle (on 1.4.3 too: `--exefs build\extracted\v143\exefs`).
- Game version 1.4.3: the same commands with `v143` (`--patches recomp\patches-v143.txt`, `build_game.py --version v143`, `make_romfs_patches.py --version v143`). Its builds default to `build\profile-v143`; `run_game.py` runs it on copies of the 1.2.1 save backups.
- Shader census (`.venv-py314`): `tools\shader_census.py --version v121 [--translate]` extracts every Maxwell shader to `build/v121/shaders/` and runs it through `build\xlate\totk_xlate.exe` and `spirv-val`; failure groups go to `failures.txt`. `tools\xlate_smoke.py` checks the translator alone, without game files.
- Shader preparation (`.venv-py314`, once per build dir and translator): `tools\prepare_shaders.py build\v121` translates the variants of the MacNativeGames shared shader list (read from `refs/` at run time) into the build's shader cache and builds their stage libraries into the driver cache (`totk.exe --prepare-shaders`, about 5 min, 1.3 GB); play then translates and compiles only variants the list lacks.
- Shader preparation (`.venv-py314`, once per build dir and translator): `tools\prepare_shaders.py build\v121` translates every game shader's default variant and the variants of the MacNativeGames shared shader list (read from `refs/` at run time, with the rows its history dropped: `tools/shader_list.py`) into the build's shader cache and builds their stage libraries into the driver cache (`totk.exe --prepare-shaders`; 1.4.3: 29,587 variants, 7 min on 24 threads, 1.7 GB); play then translates and compiles only variants neither covers.
- Open work with its evidence: [docs/backlog.md](docs/backlog.md).
- Installer (`installer/`, `tools/make_release.py`), which builds the game on a player's machine: [docs/installer.md](docs/installer.md).
- Running, measuring or frame-dumping the game, save backups, task worktrees and their builds: [docs/agent-playbook.md](docs/agent-playbook.md).
+1 -1
View File
@@ -24,7 +24,7 @@ MacNativeGames cannot run on x86-64 as is: its CPU path executes ARM64 code nati
- ExeFS and RomFS extraction: MacNativeGames' Python extractor (Python 3.14 for `compression.zstd`, `.venv-py314`).
- Update merging: hactool at `1d64a83`, with MacNativeGames' BKTR patches (`toolchain/hactool`).
- Verification: MacNativeGames' `expected-v*.json` hashes of ExeFS, RomFS and fonts.
- Primary version: 1.2.1, the update most players have. MacNativeGames supports it with the same build path as the others. 1.0.0, 1.1.2 and 1.2.1 share `sdk`, `subsdk0` and `rtld` byte for byte, so the lifted SDK modules and the host's SDK boundary serve all three; only `main` differs. 1.4.3 replaced all three modules (new SDK 20.5.22, RELR relocations, audio renderer REV15) and is a separate target; work on it has started on a branch and is not merged.
- Primary version: 1.2.1, the update most players have. MacNativeGames supports it with the same build path as the others. 1.0.0, 1.1.2 and 1.2.1 share `sdk`, `subsdk0` and `rtld` byte for byte, so the lifted SDK modules and the host's SDK boundary serve all three; only `main` differs. 1.4.3 replaced all three modules (new SDK 20.5.22, RELR relocations, audio renderer REV15) and is a second target on the same build path: `--version v143`, `recomp/patches-v143.txt` (state in backlog item 39).
- Windows driver: `tools/extract.py` runs the vendored extractors (`tools/vendor/macnativegames/`) and identifies the update by the SHA-256 of its `main`. hactool has no Darwin `funopen` here, so the base Program NCA is copied out of the NSP before the BKTR merge (about 17 GB of temporary space).
### 2. CPU: lift ARM64 to C
+18 -16
View File
@@ -173,22 +173,24 @@ Each platform item gets a written design first, agreed with the maintainer.
every render target ourselves); MacNativeGames has neither. The software keyboard could move onto
SavageWindRecomp's `osk.mm`.
39. **Game version 1.4.3** (branch `task/v143`, paused). The 1.4.3 files extract and verify, and the generator runs on
them unchanged. Estimate: one day to the title screen, two to three to play at 60 fps. Steps:
1. `stg` no-op lift, DT_RELR in `recomp/nso.py` and `runtime/loader.c`, symbol count from DT_GNU_HASH for `sdk`
and `subsdk0`.
2. Jump tables: 26 disagreements and 83 unresolved sites down to 0 unexplained, with 1.2.1 still at 0.
3. Tools take a version instead of `v121`; re-measure the `--probe-loader` table (`loader.c` `expect[]`) and
`boot.c` `expected_inits`; new boot anchors (`boot.c` `CHECKPOINT_RET`, `NVN_RET`).
4. RELRO (`nn::ro::ProtectRelro`) and the new SDK imports; run to the title screen.
5. `recomp/patches-v143.txt` from MacNativeGames `game_patches_v143.c` (fps60, bounce, cloth; the real-time step
rewritten for the `sys+0x1da` coefficient); the player position of `frame_rate.c` `player_pos`;
`make_romfs_patches.py --version` (`Common.Product.140`).
6. Audio REV15 is mostly in place (`audren.c`, `audio_mixer.c`): fix the REV11 self-test, check the override
signatures (biquad, font decode), and listen for render differences in play.
40. **Community ExeFS patches** (`.ips`, `.pchtxt` for 1.2.1): apply them through the generator's patch table
(`recomp/patches.py`). RomFS mods already load (`romfs.c`, `TOTK_ROMFS_MODS`). Code-module mods (exlaunch
`subsdk9`) need porting one by one.
39. **Game version 1.4.3**. Builds, boots and plays: the user played it from their own saves on 2026-10-26 without
problems; the field save runs at 60 fps with game speed 1.000, REV15 audio without underruns, 0 unimplemented NVN
calls, 0 validation errors; every probe passes on 1.4.3 and on 1.2.1; the jump-table check reports 0 disagreements
on both; the installer takes either update. Saves carry over both ways. `prepare_shaders.py` seeds every shader's
default variant, as MacNativeGames' seed does, and reads the list with the V rows its history dropped
(`tools/shader_list.py`; the current list alone has 134 rows of 1.4.3 shaders): 29,587 variants, among them all
1,095 that the field and intro runs met in play. Open:
1. MacNativeGames types vertex inputs and colour outputs by the bound format, so one variant serves any format;
ours still differ by those types (347 of 19,468 variants met on 1.2.1). No frame-time gain, only fewer
translations on first sight.
2. The installer end-to-end test (`tests/installer_e2e.py`) runs only on 1.2.1.
3. MacNativeGames' 1.4.3 output, render and fixed resolution patches (`game_patches_v143.c`) are not ported; 1.2.1
has none of them either.
4. The opening cutscene (Zelda under Hyrule Castle) shows hard light bands on 1.4.3 and on 1.2.1 alike; whether
that is the game's look or ours is not checked.
40. **Community ExeFS patches** (`.ips`, `.pchtxt`, each for one game version): apply them through the generator's
patch table (`recomp/patches.py`). RomFS mods already load (`romfs.c`, `TOTK_ROMFS_MODS`). Code-module mods
(exlaunch `subsdk9`) need porting one by one.
41. **Frame rates above 60 and VRR.**
42. **Map tour**: teleport Link across the map by writing position, time of day and weather in guest memory, to fill
the caches on a first run and to serve as an automatic test (crashes, validation, fps per region).
+7 -3
View File
@@ -9,7 +9,7 @@ holds only the game; the installer and its tools stay outside it.
1. Downloads `TotK-Setup-<version>-windows.zip` from GitHub Releases (none is published yet), unpacks it anywhere and
starts `TotK-Setup.exe`.
2. Picks:
- the base game (XCI or NSP) and the 1.2.1 update (NSP);
- the base game (XCI or NSP) and the 1.2.1 or 1.4.3 update (NSP);
- their `prod.keys`;
- the game language and the install folder.
3. Waits for the setup, one time. The window shows each step and its progress.
@@ -33,14 +33,18 @@ The player runs a newer installer on the same folder. It reads `totk-install.jso
`profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the shaders. A newer
release that changes nothing game-derived skips the rebuild.
Given another update file than the recorded one (1.2.1, then 1.4.3), the installer extracts and builds again from it and
keeps `profile/`. Saves carry over both ways: 1.4.3 converts a 1.2.1 save on load, and 1.2.1 loads the saves 1.4.3 wrote.
## Setup steps
1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 (by hash, as
1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 or 1.4.3 (by hash, as
`tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a
first install, 20 GiB after it), the CPU has
AVX2 (`-march=x86-64-v3`), a Vulkan 1.3 GPU exists. Each failure is one clear message.
2. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
3. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt`) into a temp folder next to the installer.
3. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt` or `patches-v143.txt`) into a temp folder next
to the installer.
4. **Compile** the generated C, `modules.c` and `romfs.c` (they bake per-install layout and RomFS hashes) and link
them with the prebuilt runtime library into `totk.exe` (`build_game.py --runtime-lib`; about 3 min on 24 cores).
5. **Prepare** the RomFS patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`,
+1 -1
View File
@@ -41,7 +41,7 @@ def cancel(signum, frame):
def main(argv=None):
ap = Parser(prog='python -m installer', description=__doc__.splitlines()[0])
ap.add_argument('--base', type=Path, help='base game, XCI or NSP')
ap.add_argument('--update', type=Path, help='1.2.1 update NSP')
ap.add_argument('--update', type=Path, help='1.2.1 or 1.4.3 update NSP')
ap.add_argument('--keys', type=Path, help='prod.keys')
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
ap.add_argument('--dest', type=Path, required=True, help='install folder')
+20 -10
View File
@@ -4,7 +4,8 @@ Release (docs/installer.md "Release layout"), root = the unpacked folder:
python/python.exe runs every tool (numpy, lz4, cryptography, compression.zstd); its ._pth puts .., ../tools
and ../recomp on sys.path, so `python -m installer` imports from the release folder
recomp/, tools/ the lifter and the scripts; recomp/patches-v121.txt, tools/data/expected-v1.2.1.json
recomp/, tools/ the lifter and the scripts; recomp/patches-<v>.txt, tools/data/expected-v<version>.json for
each supported update (SUPPORTED)
llvm/ llvm-mingw (bin/clang.exe)
runtime/ libtotkrt.a, the headers, install-side sources (romfs.c) and totk.ico; runtime/deps/ holds
include/ + lib/ of Vulkan, SDL3, Opus, libvpx and VMA (build_game.py --deps)
@@ -28,7 +29,12 @@ from pathlib import Path
EXE = '.exe' if os.name == 'nt' else ''
UCRT = Path(r'C:\msys64\ucrt64\bin')
SHADER_LIST_REF = 'origin/HEAD:games/totk/assets/shader_list.tsv' # dev: tools/prepare_shaders.py LIST_REF
SUPPORTED = ('1.2.1', '1.4.3') # updates the installer builds; the check identifies the player's by hash
def version_tag(version):
"""'1.4.3' -> 'v143': the version name of build_game.py, make_romfs_patches.py and the patch tables."""
return 'v' + version.replace('.', '')
@dataclass(frozen=True)
@@ -39,8 +45,6 @@ class Layout:
python: Path # runs the tools
tools: Path
recomp: Path
patches: Path # code patch table for recomp/gen.py
expected: Path # expected hashes of the supported update
llvm: Path # llvm-mingw tree
deps: Path # include/ and lib/ for the link
runtime: Path # runtime headers and install-side sources
@@ -57,12 +61,20 @@ class Layout:
def clang(self):
return self.llvm / 'bin' / ('clang' + EXE)
def patches(self, version):
"""The code patch table for recomp/gen.py."""
return self.recomp / f'patches-{version_tag(version)}.txt'
def expected(self, version):
"""The expected hashes of the update."""
return self.tools / 'data' / f'expected-v{version}.json'
def missing(self):
"""Paths the steps need that do not exist."""
need = [self.python, self.tools / 'extract.py', self.tools / 'build_game.py', self.tools / 'prepare_shaders.py',
self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.patches, self.expected, self.clang,
self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.clang,
self.deps / 'include', self.deps / 'lib', self.runtime, self.runtime_lib, self.ninja, self.hactool,
self.xlate, *self.dlls]
self.xlate, *self.dlls, *(f(v) for v in SUPPORTED for f in (self.patches, self.expected))]
if self.shader_list:
need.append(self.shader_list)
return [str(p) for p in need if not p.exists()]
@@ -78,8 +90,7 @@ def release_layout(root):
root=root, dev=False,
release=version.read_text(encoding='utf-8').strip() if version.is_file() else 'unknown',
python=root / 'python' / ('python' + EXE),
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
tools=root / 'tools', recomp=root / 'recomp',
llvm=root / 'llvm', deps=root / 'runtime' / 'deps', runtime=root / 'runtime',
runtime_lib=root / 'runtime' / 'libtotkrt.a',
ninja=root / 'bin' / ('ninja' + EXE), hactool=root / 'bin' / ('hactool' + EXE),
@@ -97,8 +108,7 @@ def dev_layout(root):
return Layout(
root=root, dev=True, release=f'dev-{commit or "unknown"}',
python=embed if embed.is_file() else Path(sys.executable),
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
tools=root / 'tools', recomp=root / 'recomp',
llvm=trees[-1] if trees else root / 'toolchain' / 'llvm-mingw-missing', deps=deps,
runtime=root / 'runtime', runtime_lib=root / 'build' / 'install-rtlib' / 'libtotkrt.a',
ninja=UCRT / ('ninja' + EXE), hactool=root / 'toolchain' / 'hactool' / ('hactool' + EXE),
+46 -24
View File
@@ -22,18 +22,18 @@ import time
from pathlib import Path
from . import ini
from .layout import EXE
from .layout import EXE, SUPPORTED, version_tag
from .runner import InstallError, Step
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / 'tools'))
import extract # noqa: E402
import shader_list # noqa: E402
from extract_exefs import (common_tickets, container_entries, load_keys, nca_ctr_key, nca_inventory, # noqa: E402
program_nca)
GiB = 1 << 30
DEST_EXTRA = 4 * GiB # beside the RomFS: shader and pipeline caches (about 2 GB), exefs, totk.exe, seed files
TEMP_NEED = 3 * GiB # generated C (about 0.9 GB) and objects
UPDATE_VERSION = '1.2.1'
HOSTCHECK = Path(__file__).resolve().parent / 'hostcheck.c'
STAMP = '.install-key'
@@ -138,19 +138,23 @@ def raise_if_damaged(error, name, path):
raise InstallError('check.input_damaged', input=name, path=str(path), detail=str(error)) from None
def expected_update():
return extract.load_expected()[UPDATE_VERSION]
def game_version(ctx):
"""The update this install is built from: what the check identified, else the recorded one (an install made before
1.4.3 support records none: 1.2.1)."""
return ctx.state.get('version', SUPPORTED[0])
def expected_update(ctx):
return extract.load_expected()[game_version(ctx)]
def shader_list_text(layout):
if layout.shader_list:
return layout.shader_list.read_text(encoding='utf-8')
refs = layout.root / 'refs' / 'MacNativeGames'
r = subprocess.run(['git', '-C', str(refs), 'show', 'origin/HEAD:games/totk/assets/shader_list.tsv'],
capture_output=True, encoding='utf-8')
if r.returncode:
try:
return shader_list.history(layout.root / 'refs' / 'MacNativeGames')
except subprocess.CalledProcessError:
raise InstallError('prepare.shader_list_missing')
return r.stdout
class Watch(threading.Thread):
@@ -316,7 +320,8 @@ class Check(Step):
def space(self, ctx, ncas):
dest_need = temp_need = 0
if ctx.will_run('extract'):
dest_need += expected_update()['romfs']['size'] + ncas
# the update is identified after this estimate: the larger RomFS of the supported ones
dest_need += max(extract.load_expected()[v]['romfs']['size'] for v in SUPPORTED) + ncas
romfs = ctx.dest / 'romfs' / 'romfs.bin'
if romfs.is_file(): # a resumed extraction replaces it
dest_need -= romfs.stat().st_size
@@ -361,8 +366,14 @@ class Check(Step):
found = extract.identify_update(work / 'update' / 'exefs', versions)['display_version']
except ValueError:
found = 'unknown'
if found != UPDATE_VERSION:
raise InstallError('check.update_wrong_version', found=found, wanted=UPDATE_VERSION)
if found not in SUPPORTED:
raise InstallError('check.update_wrong_version', found=found, wanted=' / '.join(SUPPORTED))
if found != game_version(ctx): # another update than the recorded one: every step builds again from it
ctx.reporter.log(f'check: update {found} (recorded {game_version(ctx)})')
ctx.state['version'] = found
ctx.keys = {}
ctx.plan(ctx.forced | {'extract', 'lift', 'compile', 'prepare'})
ctx.state['version'] = found
# ---- 2. extract
@@ -372,10 +383,17 @@ class Extract(Step):
def key(self, ctx):
# The output is whatever matches the expected hashes, whichever files and tools produced it.
return fingerprint('extract-1', ctx.layout.expected)
return fingerprint('extract-1', ctx.layout.expected(game_version(ctx)))
def current(self, ctx, record):
expected = expected_update()
# Another update file than the recorded one (1.2.1 -> 1.4.3) extracts again; the check then identifies it.
update, known = getattr(ctx.options, 'update', None), ctx.state['inputs'].get('update')
if update and known and Path(update).is_file():
st = Path(update).stat()
if (st.st_size, st.st_mtime_ns) != (known.get('size'), known.get('mtime_ns')) \
and file_sha256(Path(update)) != known.get('sha256'):
return False
expected = expected_update(ctx)
romfs = ctx.dest / 'romfs'
if extract.exefs_problems(ctx.dest / 'exefs', expected) or not extract.romfs_verified(romfs, expected):
return False
@@ -386,11 +404,11 @@ class Extract(Step):
image = ctx.dest / 'romfs' / 'romfs.bin'
rep = ctx.reporter
digest = file_sha256(image, lambda d, t: rep.progress('check', 'verify_romfs', d, t, 'bytes'))
return digest == expected_update()['romfs']['sha256']
return digest == expected_update(ctx)['romfs']['sha256']
def run(self, ctx):
o, dest, rep = ctx.options, ctx.dest, ctx.reporter
expected = expected_update()
expected = expected_update(ctx)
base, update, keys = Path(o.base), Path(o.update), Path(o.keys)
inputs = {}
for name, path in (('base', base), ('update', update)):
@@ -422,9 +440,9 @@ class Extract(Step):
shutil.move(staged, dest / 'exefs')
problems = extract.exefs_problems(dest / 'exefs', expected)
if problems:
raise InstallError('extract.exefs_mismatch', modules=', '.join(problems))
raise InstallError('extract.exefs_mismatch', version=game_version(ctx), modules=', '.join(problems))
if extract.sha256(dest / 'exefs' / 'main.npdm') != expected['exefs_sha256']['main.npdm']:
rep.warning('warn.npdm_differs')
rep.warning('warn.npdm_differs', version=game_version(ctx))
romfs = dest / 'romfs'
if self.id in ctx.forced:
@@ -433,7 +451,8 @@ class Extract(Step):
self.merge(ctx, base, update, keys, romfs, expected)
report = json.loads((romfs / 'index.json').read_text())
if report['romfs_size'] != expected['romfs']['size'] or report['romfs_sha256'] != expected['romfs']['sha256']:
raise InstallError('extract.romfs_mismatch', size=report['romfs_size'], sha256=report['romfs_sha256'][:16])
raise InstallError('extract.romfs_mismatch', version=game_version(ctx), size=report['romfs_size'],
sha256=report['romfs_sha256'][:16])
(dest / 'profile').mkdir(exist_ok=True)
st = (romfs / 'romfs.bin').stat()
return {'romfs_size': st.st_size, 'romfs_mtime_ns': st.st_mtime_ns}
@@ -476,7 +495,7 @@ class Lift(Step):
transient = True
def key(self, ctx):
return fingerprint('lift-1', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches)
return fingerprint('lift-1', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches(game_version(ctx)))
def current(self, ctx, record):
gen = ctx.temp / 'gen'
@@ -494,7 +513,7 @@ class Lift(Step):
rep.progress('lift', 'emit', int(m.group(1)), int(m.group(2)), 'chunks')
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py',
['--exefs', ctx.dest / 'exefs', '--out', gen, '--patches', layout.patches], parse)
['--exefs', ctx.dest / 'exefs', '--out', gen, '--patches', layout.patches(game_version(ctx))], parse)
if code:
raise InstallError('lift.failed', code=code, log=str(log))
(gen / STAMP).write_text(ctx.keys['lift'] + '\n')
@@ -541,7 +560,8 @@ class Compile(Step):
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--deps', layout.deps,
'--runtime', layout.runtime, '--runtime-lib', layout.runtime_lib, '--ninja', layout.ninja,
'--expected', layout.expected, '--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
'--profile', dest / 'profile'], parse)
exe = build / ('totk' + EXE)
if code or not exe.is_file():
@@ -593,7 +613,8 @@ class Prepare(Step):
rep.progress('prepare', 'romfs_patches', force=True)
remove(dest / 'romfs-patches')
code, log = run_tool(ctx, 'prepare', layout.tools / 'make_romfs_patches.py',
['--romfs', dest / 'romfs', '--out', dest / 'romfs-patches'])
['--version', version_tag(game_version(ctx)), '--romfs', dest / 'romfs', '--out',
dest / 'romfs-patches'])
if code:
raise InstallError('prepare.romfs_patches_failed', code=code, log=str(log))
@@ -613,7 +634,8 @@ class Prepare(Step):
rep.progress('prepare', 'seed', force=True)
code, log = run_tool(ctx, 'prepare', layout.tools / 'prepare_shaders.py',
[dest, '--romfs', dest / 'romfs', '--exefs', dest / 'exefs', '--list', shader_list], parse)
[dest, '--version', version_tag(game_version(ctx)), '--romfs', dest / 'romfs', '--exefs',
dest / 'exefs', '--list', shader_list], parse)
if code or not (dest / 'shader-prepared.bin').is_file():
raise InstallError('prepare.shaders_failed', code=code, log=str(log))
return {'xlate_sha256': file_sha256(xlate)}
+9 -9
View File
@@ -46,7 +46,7 @@
"check.vulkan_missing": "No Vulkan driver is installed. Install the latest driver for your graphics card.",
"check.vulkan_no_device": "Vulkan finds no graphics card. Install the latest driver for your graphics card.",
"check.vulkan_too_old": "The game needs a graphics card with Vulkan 1.3; found: {devices}. Update the graphics driver.",
"check.inputs_needed": "The game files are needed: choose the base game, the 1.2.1 update and prod.keys.",
"check.inputs_needed": "The game files are needed: choose the base game, the 1.2.1 or 1.4.3 update and prod.keys.",
"check.input_missing": "File not found ({input}): {path}",
"check.keys_unreadable": "The keys file cannot be read. Choose a prod.keys file.",
"check.keys_no_header_key": "The keys file has no header_key. Dump prod.keys from your console again.",
@@ -58,17 +58,17 @@
"check.ticket_missing": "The {input} file lacks the ticket its content needs.",
"check.base_not_totk": "The base game file is not The Legend of Zelda: Tears of the Kingdom ({detail}).",
"check.update_not_nsp": "The update must be an NSP file.",
"check.update_is_base": "The update file holds a base game. Choose the 1.2.1 update NSP.",
"check.update_is_base": "The update file holds a base game. Choose the 1.2.1 or 1.4.3 update NSP.",
"check.update_not_totk": "The update file is not a Tears of the Kingdom update ({detail}).",
"check.exefs_failed": "The game code of the {input} file failed its checks; see {log}.",
"check.input_damaged": "The {input} file is damaged or incomplete: {path} ({detail}). Copy that file again from its source, or dump it again, then run the setup again.",
"check.base_wrong_version": "The base game is not version 1.0.0 ({modules} differ).",
"check.update_wrong_version": "The update is version {found}; this release needs {wanted}.",
"check.update_wrong_version": "The update is version {found}; this release takes {wanted}.",
"check.disk_space": "Not enough free space for {path}: {need_gb} GB needed, {free_gb} GB free.",
"extract.exefs_mismatch": "The extracted game code does not match version 1.2.1 ({modules}).",
"extract.exefs_mismatch": "The extracted game code does not match version {version} ({modules}).",
"extract.romfs_merge_failed": "Applying the update to the game data failed: {detail}. Details: {log}",
"extract.romfs_mismatch": "The extracted game data does not match version 1.2.1 (size {size}, checksum {sha256}...).",
"extract.romfs_mismatch": "The extracted game data does not match version {version} (size {size}, checksum {sha256}...).",
"lift.failed": "Translating the game code failed (exit code {code}). Details: {log}",
"compile.failed": "Compiling totk.exe failed (exit code {code}). Details: {log}",
"compile.exe_in_use": "{path} is in use. Close the game and run the installer again.",
@@ -78,20 +78,20 @@
"cleanup.ini_failed": "totk.ini cannot be written: {detail}",
"warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.",
"warn.npdm_differs": "main.npdm differs from the reference 1.2.1 dump; only the code modules are used.",
"warn.npdm_differs": "main.npdm differs from the reference {version} dump; only the code modules are used.",
"ui.title": "TotK Setup",
"ui.intro": "Builds a playable Tears of the Kingdom folder from your own game files. This takes a while and runs once.",
"ui.notice": "The folder this setup builds contains the game. It is for your own use only; do not share or upload it.",
"ui.keys_private": "The keys file is read in place. Its contents are never shown, copied or stored.",
"ui.base": "Base game (NSP or XCI)",
"ui.update": "Update 1.2.1 (NSP)",
"ui.update": "Update 1.2.1 or 1.4.3 (NSP)",
"ui.keys": "prod.keys",
"ui.game_language": "Game language",
"ui.dest": "Install folder",
"ui.browse": "Browse...",
"ui.pick_game": "Choose the base game file",
"ui.pick_update": "Choose the 1.2.1 update file",
"ui.pick_update": "Choose the 1.2.1 or 1.4.3 update file",
"ui.pick_keys": "Choose prod.keys",
"ui.pick_dest": "Choose the install folder",
"ui.type_game": "Game files (NSP, XCI)",
@@ -109,7 +109,7 @@
"ui.update_mode": "This folder holds an earlier installation. The setup continues or updates it; the game files are asked for only if they are needed again.",
"ui.need_dest": "Choose the install folder.",
"ui.need_base": "Choose the base game file.",
"ui.need_update": "Choose the 1.2.1 update file.",
"ui.need_update": "Choose the 1.2.1 or 1.4.3 update file.",
"ui.need_keys": "Choose the prod.keys file.",
"ui.need_lang": "Choose the game language.",
"ui.missing_base": "The base game file does not exist.",
+9 -9
View File
@@ -46,7 +46,7 @@
"check.vulkan_missing": "Драйвер Vulkan не установлен. Установите свежий драйвер видеокарты.",
"check.vulkan_no_device": "Vulkan не находит видеокарту. Установите свежий драйвер видеокарты.",
"check.vulkan_too_old": "Игре нужна видеокарта с Vulkan 1.3; найдено: {devices}. Обновите драйвер видеокарты.",
"check.inputs_needed": "Нужны игровые файлы: выберите базовую игру, обновление 1.2.1 и prod.keys.",
"check.inputs_needed": "Нужны игровые файлы: выберите базовую игру, обновление 1.2.1 или 1.4.3 и prod.keys.",
"check.input_missing": "Файл не найден ({input}): {path}",
"check.keys_unreadable": "Не удаётся прочитать файл ключей. Выберите файл prod.keys.",
"check.keys_no_header_key": "В файле ключей нет header_key. Снова получите prod.keys со своей консоли.",
@@ -58,17 +58,17 @@
"check.ticket_missing": "В файле «{input}» нет тикета, который нужен его содержимому.",
"check.base_not_totk": "Файл базовой игры — не The Legend of Zelda: Tears of the Kingdom ({detail}).",
"check.update_not_nsp": "Обновление должно быть файлом NSP.",
"check.update_is_base": "Файл обновления содержит базовую игру. Выберите NSP с обновлением 1.2.1.",
"check.update_is_base": "Файл обновления содержит базовую игру. Выберите NSP с обновлением 1.2.1 или 1.4.3.",
"check.update_not_totk": "Файл обновления — не обновление Tears of the Kingdom ({detail}).",
"check.exefs_failed": "Код игры из файла «{input}» не прошёл проверку; подробности в {log}.",
"check.input_damaged": "Файл «{input}» повреждён или скопирован не полностью: {path} ({detail}). Скопируйте этот файл заново из источника или сделайте дамп ещё раз, затем снова запустите установку.",
"check.base_wrong_version": "Базовая игра не версии 1.0.0 (отличаются модули: {modules}).",
"check.update_wrong_version": "Версия обновления {found}; этому выпуску нужна {wanted}.",
"check.update_wrong_version": "Версия обновления {found}; этот выпуск принимает {wanted}.",
"check.disk_space": "Не хватает свободного места для {path}: нужно {need_gb} ГБ, свободно {free_gb} ГБ.",
"extract.exefs_mismatch": "Распакованный код игры не соответствует версии 1.2.1 ({modules}).",
"extract.exefs_mismatch": "Распакованный код игры не соответствует версии {version} ({modules}).",
"extract.romfs_merge_failed": "Не удалось применить обновление к данным игры: {detail}. Подробности: {log}",
"extract.romfs_mismatch": "Распакованные данные игры не соответствуют версии 1.2.1 (размер {size}, контрольная сумма {sha256}...).",
"extract.romfs_mismatch": "Распакованные данные игры не соответствуют версии {version} (размер {size}, контрольная сумма {sha256}...).",
"lift.failed": "Не удалось перевести код игры (код выхода {code}). Подробности: {log}",
"compile.failed": "Не удалось собрать totk.exe (код выхода {code}). Подробности: {log}",
"compile.exe_in_use": "Файл {path} занят. Закройте игру и запустите установку снова.",
@@ -78,20 +78,20 @@
"cleanup.ini_failed": "Не удаётся записать totk.ini: {detail}",
"warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.",
"warn.npdm_differs": "main.npdm отличается от эталонного дампа 1.2.1; используются только модули с кодом.",
"warn.npdm_differs": "main.npdm отличается от эталонного дампа {version}; используются только модули с кодом.",
"ui.title": "Установка TotK",
"ui.intro": "Собирает готовую к запуску папку с Tears of the Kingdom из ваших игровых файлов. Занимает много времени и выполняется один раз.",
"ui.notice": "Папка, которую создаёт установка, содержит игру. Она предназначена только для личного использования: не передавайте и не выкладывайте её.",
"ui.keys_private": "Файл ключей читается на месте. Его содержимое не показывается, не копируется и не сохраняется.",
"ui.base": "Базовая игра (NSP или XCI)",
"ui.update": "Обновление 1.2.1 (NSP)",
"ui.update": "Обновление 1.2.1 или 1.4.3 (NSP)",
"ui.keys": "prod.keys",
"ui.game_language": "Язык игры",
"ui.dest": "Папка установки",
"ui.browse": "Обзор...",
"ui.pick_game": "Выберите файл базовой игры",
"ui.pick_update": "Выберите файл обновления 1.2.1",
"ui.pick_update": "Выберите файл обновления 1.2.1 или 1.4.3",
"ui.pick_keys": "Выберите prod.keys",
"ui.pick_dest": "Выберите папку установки",
"ui.type_game": "Файлы игры (NSP, XCI)",
@@ -109,7 +109,7 @@
"ui.update_mode": "В этой папке уже есть установка. Программа продолжит или обновит её; файлы игры понадобятся, только если их снова придётся читать.",
"ui.need_dest": "Выберите папку установки.",
"ui.need_base": "Выберите файл базовой игры.",
"ui.need_update": "Выберите файл обновления 1.2.1.",
"ui.need_update": "Выберите файл обновления 1.2.1 или 1.4.3.",
"ui.need_keys": "Выберите файл prod.keys.",
"ui.need_lang": "Выберите язык игры.",
"ui.missing_base": "Файл базовой игры не найден.",
+12
View File
@@ -383,8 +383,20 @@ class Lifter:
return self.ldst_pair(w)
if bits(w, 27, 3) == 0b111:
return self.ldst_reg(w)
if (w & 0xFFE00000) == 0xD9200000 and bits(w, 10, 2):
return self.stg(w)
raise Unsupported(f'ldst {w:08x}')
def stg(self, w):
"""STG (post-index, signed offset, pre-index). Horizon runs without MTE, so storing an allocation tag
changes no memory; only the base writeback remains (1.4.3 sdk+0x640050)."""
mode, rn = bits(w, 10, 2), bits(w, 5, 5)
if mode == 2:
return Insn('op', '/* stg */')
off = sext(bits(w, 12, 9), 9) << 4
dst = 'c->sp' if rn == 31 else f'c->x[{rn}]'
return Insn('op', f'{dst} = {XS(rn)} + {hx(off & 0xFFFFFFFFFFFFFFFF)}; /* stg */')
def ldst_excl(self, w, pc):
size, o2, L, o1, rs, o0, rn, rt = (bits(w, 30, 2), bits(w, 23, 1), bits(w, 22, 1), bits(w, 21, 1),
bits(w, 16, 5), bits(w, 15, 1), bits(w, 5, 5), bits(w, 0, 5))
+38 -4
View File
@@ -8,6 +8,7 @@ import lz4.block
DT_NULL, DT_NEEDED, DT_PLTRELSZ, DT_HASH, DT_STRTAB, DT_SYMTAB, DT_RELA, DT_RELASZ = 0, 1, 2, 4, 5, 6, 7, 8
DT_STRSZ, DT_JMPREL, DT_RELACOUNT = 10, 23, 0x6FFFFFF9
DT_RELRSZ, DT_RELR, DT_GNU_HASH = 0x23, 0x24, 0x6FFFFEF5
R_AARCH64_ABS64, R_AARCH64_GLOB_DAT, R_AARCH64_JUMP_SLOT, R_AARCH64_RELATIVE = 257, 1025, 1026, 1027
@@ -61,12 +62,12 @@ class Nso:
strtab = dyn.get(DT_STRTAB, 0)
self.strtab = strtab
self.needed = [self.cstr(strtab + n) for n in needed]
# dynsym count from DT_HASH nchain
# dynsym count from DT_HASH nchain, or from DT_GNU_HASH (1.4.3 sdk and subsdk0 have no DT_HASH)
self.syms = []
if DT_HASH in dyn and DT_SYMTAB in dyn:
nchain = self.u32(dyn[DT_HASH] + 4)
nsyms = self.symbol_count()
if nsyms and DT_SYMTAB in dyn:
st = dyn[DT_SYMTAB]
for i in range(nchain):
for i in range(nsyms):
o = st + i * 24
name, info, other, shndx = struct.unpack_from('<IBBH', self.image, o)
value, size = struct.unpack_from('<QQ', self.image, o + 8)
@@ -78,6 +79,39 @@ class Nso:
for i in range(n):
r_off, r_info, r_add = struct.unpack_from('<QQq', self.image, b + i * 24)
self.relocs.append((r_off, r_info & 0xFFFFFFFF, r_info >> 32, r_add))
# DT_RELR (1.4.3): packed RELATIVE relocations whose addend is the word in place. An even entry is an address
# (relocate it, continue after it); an odd entry is a bitmap of the next 63 words.
if DT_RELR in dyn:
where = 0
for i in range(dyn[DT_RELRSZ] // 8):
e = self.u64(dyn[DT_RELR] + i * 8)
if e & 1 == 0:
self.relocs.append((e, R_AARCH64_RELATIVE, 0, self.u64(e)))
where = e + 8
continue
for k in range(63):
if e >> (k + 1) & 1:
a = where + k * 8
self.relocs.append((a, R_AARCH64_RELATIVE, 0, self.u64(a)))
where += 63 * 8
def symbol_count(self):
"""Number of .dynsym entries: DT_HASH nchain, else one past the last symbol a DT_GNU_HASH chain reaches."""
dyn = self.dyn
if DT_HASH in dyn:
return self.u32(dyn[DT_HASH] + 4)
if DT_GNU_HASH not in dyn:
return 0
h = dyn[DT_GNU_HASH]
nbuckets, symoffset, bloom_size = self.u32(h), self.u32(h + 4), self.u32(h + 8)
buckets = h + 16 + bloom_size * 8
last = max([self.u32(buckets + 4 * i) for i in range(nbuckets)] + [0])
if last < symoffset:
return symoffset
chains = buckets + 4 * nbuckets
while not self.u32(chains + 4 * (last - symoffset)) & 1:
last += 1
return last + 1
def cstr(self, off):
end = self.image.index(b'\0', off)
+46
View File
@@ -0,0 +1,46 @@
# Code patches for TotK 1.4.3 main (recomp/patches.py). Ported from MacNativeGames
# games/totk/runtime/game_patches_v143.c (MIT, Copyright (c) 2026 MacNativeGames contributors) at upstream 73c4821,
# each original word checked against the 1.4.3 main. MacNativeGames branches to veneers after the text; here a
# veneer's body is the site's replacement. The patch page holds three floats the runtime writes at every present
# (frame_rate.c): +0 the 30 Hz ratio (seconds * 30), +4 the frame step in seconds, +8 the step in 60 Hz ticks.
set fps60 TOTK_FPS60 1
set realtime TOTK_REALTIME 1
set cloth TOTK_CLOTH_REALTIME 1
set bounce TOTK_HIGHFPS_FIX 1
# 60 fps: the adaptive present-interval controller (main+0x2db174, 1.2.1 0x8f26c8) gets the minimum interval 1.
site fps60 main 2db204 6b08015f 7100055f # clamp: cmp w10, #1 (was cmp w10, w8 = minimum)
site fps60 main 2db208 1a89b10a 1a9fa52a # clamp: csinc w10, w9, wzr, ge (desired = max(desired, 1))
site fps60 main 2db22c b940468a d503201f # lower trigger: nop the stored-candidate load
site fps60 main 2db360 6b0801bf 710005bf # raise path: cmp w13, #1
site fps60 main 2db36c 1a88c1ac 1a9fc5ac # raise path: csinc w12, w13, wzr, gt (next lower candidate floors at 1)
site fps60 main 2db340 b9400288 52800028 # lower path: mov w8, #1 (minimum interval 1)
site fps60 main 2db34c 2a0a03f5 1a9fa555 # lower path: csinc w21, w10, wzr, ge (result never below 1)
# Real-time step. The frame-step producer feeds both FrameRateMgr call sites with s8 = nominal interval * a 0.5/1.0
# coefficient (sys+0x1da bit 0, fcsel above the site); the nominal interval comes from the patch page, the coefficient
# and FrameRateMgr's own arithmetic stay. x8 is free: the ldxr after the site loads it. The global ratio after the
# controller call (1.2.1 0x8f2394) reads the ratio from the page instead of 60 / interval * 0.5.
site realtime main 3e5658 1e210808 @page:x8 bd400900 1e210808 # fmul s8, s0, s1: s0 = [page+8] first
site realtime main 2daa54 bd425260 @page:x9 # ldr s0, [x19, #0x250] -> adrp x9, page
site realtime main 2daa58 52a84e08 bd400120 # mov w8, #60.0 -> ldr s0, [x9] (global ratio)
site realtime main 2daa6c 5e21d800 d503201f # scvtf s0, s0 -> nop
site realtime main 2daa70 1e201820 d503201f # fdiv s0, s1, s0 -> nop
site realtime main 2daa74 1e2c1001 d503201f # fmov s1, #0.5 -> nop
site realtime main 2daa78 1e210800 d503201f # fmul s0, s0, s1 -> nop
# Hair and cloth: the actor animation update builds a second clock {seconds = factor / 30, ratio = factor} per
# rendered frame. 1.4.3 inlines it in two actor paths, where the factor was just stored to the ratio slot: the
# replacement stores measured ratio * factor there and computes measured seconds * factor. Two other callers share the
# out-of-line update (main+0x2a0e40), whose s2 (seconds) and s3 (ratio) are scaled by the measured ratio; its third
# caller already passes measured time and stays. x8 and s4 are free at all four sites.
site cloth main 246eac 1e200940 @page:x8 bd400100 1e200940 bd0367e0 bd400500 1e200940 # fmul s0, s10, s0
site cloth main 345904 1e210801 @page:x8 bd400101 1e210801 bd0117e1 bd400501 1e210801 # fmul s1, s0, s1
site cloth main 24944c 94015e7d @page:x8 bd400104 1e240842 1e240863 94015e7d # bl 0x2a0e40
site cloth main 28bc98 9400546a @page:x8 bd400104 1e240842 1e240863 9400546a # bl 0x2a0e40
# Trampolines: the contact code scales the bounce vector by 0.1 when time object +0x28 (30 / (ratio * time scale)) is
# over 100; compare +0x28 * +0x38 (30 / time scale) instead, the same value at 30 Hz and independent of the frame rate.
# s1 is set again by the fmov two instructions on.
site bounce main 562a64 bd402920 bd402920 bd403921 1e210800 # ldr s0, [x9, #0x28]; ldr s1, [x9, #0x38]; fmul s0, s0, s1
+15 -9
View File
@@ -4,8 +4,10 @@
// The boot driver: the guest main thread runs the game's own nn::init::Start
// with three host callbacks, the second of which runs DT_INIT of main, subsdk0 and sdk; Start then calls nnMain.
// --boot checks two checkpoints on the way and ends at the first nvnBootstrapLoader call:
// SetFocusHandlingMode called from main+0xe7a954 in nnMain, every module's constructor flag reads 2
// nvnBootstrapLoader called from main+0xe7eb2c in the graphics bring-up; logs the run statistics, exits 0
// SetFocusHandlingMode called from main+0xe7a954 (1.4.3: +0xbdb48c) in nnMain, every module's constructor flag
// reads 2
// nvnBootstrapLoader called from main+0xe7eb2c (1.4.3: +0xbe0360) in the graphics bring-up; logs the run
// statistics, exits 0
// With --frames N the run continues past that call; rt_frame_presented ends it at the Nth present.
#include <windows.h>
#include <stdio.h>
@@ -18,13 +20,17 @@
#include "vk.h"
#define SYM_INIT_START "_ZN2nn4init5StartEmmPFvvES2_S2_"
#define CHECKPOINT_RET 0xe7a954ull // main: return address of nnMain's SetFocusHandlingMode call (section 0.4)
#define NVN_RET 0xe7eb2cull // main: return address of the first nvnBootstrapLoader call
// main: return addresses of nnMain's SetFocusHandlingMode call (section 0.4; 1.4.3: MNG scripts/verify_boot.py:9) and
// of the first nvnBootstrapLoader call, per game version.
#define CHECKPOINT_RET (RT_V143 ? 0xbdb48cull : 0xe7a954ull)
#define NVN_RET (RT_V143 ? 0xbe0360ull : 0xe7eb2cull)
#define CTOR_DONE 2 // a module's constructor flag after its DT_INIT ran the init_array
// The init_array sizes of 1.2.1 (section 0.1); MacNativeGames' boot check expects the same.
static const struct { const char *name; uint64_t entries; } expected_inits[] = {
{"main", 10673}, {"subsdk0", 45}, {"sdk", 116}};
// The init_array sizes (1.2.1: section 0.1; 1.4.3: DT_INIT_ARRAYSZ); MacNativeGames' boot check expects the same.
typedef struct { const char *name; uint64_t entries; } InitCount;
static const InitCount inits_121[] = {{"main", 10673}, {"subsdk0", 45}, {"sdk", 116}};
static const InitCount inits_143[] = {{"main", 10717}, {"subsdk0", 45}, {"sdk", 83}};
#define expected_inits (RT_V143 ? inits_143 : inits_121)
static volatile LONG inits_done, checkpoint_seen;
@@ -63,7 +69,7 @@ static void cb_loader_ready(Cpu *c) {
// afterwards (MacNativeGames loader.c:458-462).
static void cb_module_inits(Cpu *c) {
if (InterlockedCompareExchange(&inits_done, 0, 0)) rt_fatal("boot: module constructors requested twice");
for (unsigned k = 0; k < sizeof expected_inits / sizeof *expected_inits; k++) {
for (unsigned k = 0; k < sizeof inits_121 / sizeof *inits_121; k++) {
const RtLoaded *m = module_named(expected_inits[k].name);
uint64_t entries = m->init_array_size / 8;
if (!m->init) rt_fatal("boot: %s has no DT_INIT", m->name);
@@ -73,7 +79,7 @@ static void cb_module_inits(Cpu *c) {
printf("boot: DT_INIT %s state=%u (%llu)\n", m->name, state, (unsigned long long)entries);
if (state != CTOR_DONE) rt_fatal("boot: %s DT_INIT left its constructor flag at %u, not 2", m->name, state);
if (entries != expected_inits[k].entries)
rt_fatal("boot: %s has %llu init_array entries, 1.2.1 has %llu", m->name, (unsigned long long)entries,
rt_fatal("boot: %s has %llu init_array entries, this game version has %llu", m->name, (unsigned long long)entries,
(unsigned long long)expected_inits[k].entries);
}
InterlockedExchange(&inits_done, 1);
+18 -13
View File
@@ -2,22 +2,23 @@
// contributors): runtime_frame_step_update (2338bb0: no 1/120 s floor) and the speed survey's player chain
// (world_survey.c player_actor).
// Frame rate: 60 fps and the real-time frame step, the host half of the patch sets fps60, realtime and cloth
// (build table patches-v121.txt; docs/research/macnativegames.md section 6.2).
// (build tables patches-v121.txt, patches-v143.txt; docs/research/macnativegames.md section 6.2).
//
// TotK's adaptive present-interval controller (main+0x8f26c8) turns the main thread's measured work into a number of
// 60 Hz ticks clamped to [minimum, maximum]; shipping data sets the minimum to 2 (30 fps), the set fps60 lowers it to
// 1. The controller still raises the interval when a frame does not fit. Shipping code then steps the game by whole
// vsyncs (VFR step main+0x7ede6c: ratio = interval * 0.5, seconds = interval / 60; global ratio at main+0x8f2394), so
// game speed is right only when frames take exactly their interval. With the set realtime both read the patch page
// instead: +0 ratio = seconds * 30, +4 seconds, the moving average of the last 5 display intervals, at most 1/15 s
// instead: +0 ratio = seconds * 30, +4 seconds (1.4.3: +8 interval = seconds * 60, the producer's nominal interval
// before its half-step coefficient), the moving average of the last 5 display intervals, at most 1/15 s
// (slower frames slow the game down, as on the console), no lower bound, so a higher frame-rate cap only changes the
// pacer (nvn_api_obj.c VSYNC_HZ). The set cloth gives hair and cloth the same clock and needs realtime.
//
// The sample is the display time the pacer gives each present (nvnQueuePresentTexture), so the step follows the
// frames as shown, pacing waits included. With the step on, the pacer ignores the game's interval and holds a frame
// only to the frame-rate cap (nvn_api_obj.c present_period): the controller's 2 or 3 vsyncs would only add latency.
// One 64-bit store per present; the guest reads the two floats on its own
// threads, and both stay finite and in range whichever store they see.
// One 64-bit and one 32-bit store per present; the guest reads the floats on its own threads, and each stays finite
// and in range whichever store it sees.
//
// TOTK_TRACE=perf adds a line per 120 presents: fps, the game seconds stepped against wall seconds, and with
// TOTK_SPEED_PROBE=1 the player's position and ground speed (an independent check: Link runs at the same speed at
@@ -40,16 +41,19 @@
#define STEP_FIRST_S (1.0 / 60.0)
static _Atomic uint64_t *step_word; // patch page +0: ratio (low float), seconds (high float)
static _Atomic uint32_t *step_interval; // patch page +8: seconds * 60 (patches-v143.txt realtime)
static uint64_t last_ns;
static double samples[STEP_SAMPLES];
static unsigned sample_count, sample_next;
static uint64_t step_bits(float seconds) {
float ratio = seconds * 30.0f;
uint32_t lo, hi;
static void step_store(float seconds) {
float ratio = seconds * 30.0f, interval = seconds * 60.0f;
uint32_t lo, hi, iv;
memcpy(&lo, &ratio, 4);
memcpy(&hi, &seconds, 4);
return (uint64_t)lo | (uint64_t)hi << 32;
memcpy(&iv, &interval, 4);
atomic_store_explicit(step_word, (uint64_t)lo | (uint64_t)hi << 32, memory_order_release);
atomic_store_explicit(step_interval, iv, memory_order_release);
}
void frame_step_init(void) {
@@ -64,7 +68,8 @@ void frame_step_init(void) {
}
if (rt_patch_enabled(realtime)) {
step_word = (_Atomic uint64_t *)rt_patch_page();
atomic_store(step_word, step_bits((float)STEP_FIRST_S));
step_interval = (_Atomic uint32_t *)((uint8_t *)rt_patch_page() + 8);
step_store((float)STEP_FIRST_S);
}
rt_log("frame rate: present interval %s, %s", rt_patch_enabled(fps60) ? "1..3 (60 fps)" : "2..3 (shipping 30 fps)",
step_word ? "real-time step (5-frame average of display intervals, at most 1/15 s)" : "whole-vsync steps");
@@ -106,10 +111,10 @@ static uint64_t read_ptr(uint64_t a) {
uint64_t p = 0;
return read_guest(a, &p, 8) ? p : 0;
}
// 1.2.1: [main+0x4728538] +0x1e8 scene, its component 13 lists the resident actors; the one named "Player" has its
// position at +0x2b4 (MacNativeGames world_survey.c, game_version.h TOTK_GAME_V121).
// 1.2.1: [main+0x4728538] (1.4.3: [main+0x3add238]) +0x1e8 scene, its component 13 lists the resident actors; the one
// named "Player" has its position at +0x2b4 (MacNativeGames world_survey.c, game_version.h TOTK_GAME_V121/V143).
static int player_pos(float xyz[3]) {
uint64_t module = read_ptr(MOD_MAIN_BASE + 0x4728538), scene = module ? read_ptr(module + 0x1e8) : 0;
uint64_t module = read_ptr(MOD_MAIN_BASE + (RT_V143 ? 0x3add238 : 0x4728538)), scene = module ? read_ptr(module + 0x1e8) : 0;
int32_t count = 0;
if (!scene || !read_guest(scene + 0x50, &count, 4) || count < 14 || count > 256) return 0;
uint64_t comps = read_ptr(scene + 0x58), resident = comps ? read_ptr(comps + 13 * 8) : 0;
@@ -177,7 +182,7 @@ void frame_step_present(uint64_t display_ns, int interval) {
double sum = 0;
for (unsigned i = 0; i < sample_count; i++) sum += samples[i];
step = sum / sample_count;
atomic_store_explicit(step_word, step_bits((float)step), memory_order_release);
step_store((float)step);
}
}
last_ns = display_ns;
+2
View File
@@ -163,6 +163,8 @@ typedef struct { const char *file, *sha256; } RtFileHash;
extern const RtFileHash rt_exefs_sha256[]; // tools/data/expected-<version>.json "exefs_sha256"
extern const int rt_exefs_sha256_count;
extern const char rt_default_exefs[], rt_default_romfs[], rt_default_profile[];
extern const int rt_game_version; // the game build: 121 (1.2.1) or 143 (1.4.3)
#define RT_V143 (rt_game_version >= 143)
// ---- kernel (kernel.c, T2)
typedef struct RtThread RtThread;
+13 -6
View File
@@ -715,13 +715,17 @@ static uint64_t stub(const char *sym) {
// mutex look owned by the caller (__cxa_guard_acquire in nn::sm::Initialize aborts);
// - sdk's full DT_INIT needs nn::os initialised (its TLS slot constructor reaches ThreadManager), which is the boot
// sequence's job (T6). The audio client only needs its two sf heap constructors: 16 KiB ExpHeaps on static buffers,
// stored at sdk+0xd37e90 and sdk+0xd3bec0 (CreateAudioRendererManagerByHipc reads the second).
// stored at sdk+0xd37e90 and sdk+0xd3bec0 (CreateAudioRendererManagerByHipc reads the second). The 1.4.3 SDK creates
// that heap lazily; its OpenAudioRenderer takes nn::audio::detail::g_RendererResourceSemaphore first, which the
// constructor at sdk+0x221fb0 initializes (two renderers).
static void sdk_prepare(Cpu *c) {
Cpu *again;
ST32(LD64(c->tpidrro + 0x1F8) + 0x1B0, kern_main_thread(&again));
const RtLoaded *sdk = ldr_module(2);
static const uint64_t heap_ctors[] = {0x1dcbd0, 0x1de8a0};
for (unsigned k = 0; k < sizeof heap_ctors / sizeof *heap_ctors; k++) {
static const uint64_t ctors_121[] = {0x1dcbd0, 0x1de8a0}, ctors_143[] = {0x221fb0};
const uint64_t *heap_ctors = RT_V143 ? ctors_143 : ctors_121;
unsigned nctors = RT_V143 ? sizeof ctors_143 / sizeof *ctors_143 : sizeof ctors_121 / sizeof *ctors_121;
for (unsigned k = 0; k < nctors; k++) {
int found = 0;
for (uint64_t e = 0; e < sdk->init_array_size; e += 8)
found |= LD64(sdk->init_array + e) == sdk->base + heap_ctors[k];
@@ -740,8 +744,8 @@ static void probe_sdk_client(Cpu *c) {
rt_call(c, stub(SYM_INIT_PARAM), 1, (uint64_t)(uintptr_t)param);
printf("sdk: InitializeAudioRendererParameter -> %d Hz x%d, mix buffers %d, revision word %#x (REV%d)\n", param[0],
param[1], param[2], (uint32_t)param[13], revision_number_of((uint32_t)param[13]));
CHECK(param[0] == 48000 && param[1] == 240 && (uint32_t)param[13] == g_param.revision,
"SDK default parameter (TotK 1.2.1 builds REV11 requests)");
CHECK(param[0] == 48000 && param[1] == 240 && (uint32_t)param[13] == AUDREN_REV(RT_V143 ? 15 : 11),
"SDK default parameter (TotK 1.2.1 builds REV11 requests, 1.4.3 REV15)");
param[2] = g_param.mix_buffer_count, param[3] = g_param.submix_count, param[4] = g_param.voice_count;
param[5] = g_param.sink_count, param[6] = g_param.effect_count;
uint64_t work = rt_call(c, stub(SYM_WORK_SIZE), 1, (uint64_t)(uintptr_t)param);
@@ -784,7 +788,10 @@ static void probe_sdk_client(Cpu *c) {
"lifted client (sm: kept open by the SDK: %ld session)\n", audren_renderers_live(),
(unsigned long long)(ipc_sessions_released() - released0), (unsigned long long)requests,
ipc_sessions_live());
CHECK(audren_renderers_live() == 0 && ipc_sessions_released() - released0 >= 3 && ipc_sessions_live() == 1,
// The 1.2.1 SDK closes the renderer, device and manager sessions and keeps sm; the 1.4.3 SDK keeps the
// audren:u manager it created lazily, beside sm.
CHECK(audren_renderers_live() == 0 && ipc_sessions_released() - released0 >= (RT_V143 ? 1u : 3u) &&
ipc_sessions_live() == (RT_V143 ? 2 : 1),
"SDK CloseAudioRenderer");
rt_trace = trace;
}
+67 -9
View File
@@ -19,7 +19,7 @@ uint64_t hostfn_register_observer(const char *name, GuestFn fn, uint64_t origina
enum { DT_NULL = 0, DT_PLTRELSZ = 2, DT_HASH = 4, DT_STRTAB = 5, DT_SYMTAB = 6, DT_RELA = 7, DT_RELASZ = 8,
DT_INIT = 12, DT_FINI = 13, DT_JMPREL = 23, DT_INIT_ARRAY = 25, DT_FINI_ARRAY = 26, DT_INIT_ARRAYSZ = 27,
DT_FINI_ARRAYSZ = 28 };
DT_FINI_ARRAYSZ = 28, DT_RELRSZ = 0x23, DT_RELR = 0x24, DT_GNU_HASH = 0x6ffffef5 };
enum { R_ABS64 = 257, R_GLOB_DAT = 1025, R_JUMP_SLOT = 1026, R_RELATIVE = 1027 };
enum { K_RELATIVE, K_ABS64, K_GLOB_DAT, K_JUMP_SLOT, K_COUNT };
enum { STB_WEAK = 2, STT_FUNC = 2 };
@@ -39,6 +39,8 @@ typedef struct {
uint32_t nsyms;
const Elf64Rela *rela, *jmprel;
uint64_t nrela, njmprel;
const uint64_t *relr; // DT_RELR (1.4.3): packed RELATIVE relocations
uint64_t nrelr, relr_count;
uint32_t *hash, hash_mask; // defined symbols by name
Res *res_plain, *res_jump; // per symbol; JUMP_SLOT may bind an observer
uint32_t *funcs, nfuncs; // defined functions sorted by value (ldr_describe)
@@ -219,12 +221,15 @@ static void parse_dynamic(Mod *m) {
uint64_t mod0 = base + rd32(base + 4);
if (memcmp((const void *)(uintptr_t)mod0, "MOD0", 4)) rt_fatal("loader: %s has no MOD0 header", m->pub.name);
uint64_t dyn = mod0 + (int64_t)(int32_t)rd32(mod0 + 4);
uint64_t rela = 0, relasz = 0, jmprel = 0, pltsz = 0, hash = 0;
uint64_t rela = 0, relasz = 0, jmprel = 0, pltsz = 0, hash = 0, gnu_hash = 0, relr = 0, relrsz = 0;
for (const uint64_t *d = (const uint64_t *)(uintptr_t)dyn; d[0] != DT_NULL; d += 2) {
switch (d[0]) {
case DT_SYMTAB: m->syms = (const Elf64Sym *)(uintptr_t)(base + d[1]); break;
case DT_STRTAB: m->strtab = (const char *)(uintptr_t)(base + d[1]); break;
case DT_HASH: hash = base + d[1]; break;
case DT_GNU_HASH: gnu_hash = base + d[1]; break;
case DT_RELR: relr = base + d[1]; break;
case DT_RELRSZ: relrsz = d[1]; break;
case DT_RELA: rela = base + d[1]; break;
case DT_RELASZ: relasz = d[1]; break;
case DT_JMPREL: jmprel = base + d[1]; break;
@@ -234,8 +239,26 @@ static void parse_dynamic(Mod *m) {
case DT_INIT_ARRAYSZ: m->pub.init_array_size = d[1]; break;
}
}
if (!m->syms || !m->strtab || !hash) rt_fatal("loader: %s lacks DT_SYMTAB, DT_STRTAB or DT_HASH", m->pub.name);
m->nsyms = rd32(hash + 4); // nchain
if (!m->syms || !m->strtab || (!hash && !gnu_hash))
rt_fatal("loader: %s lacks DT_SYMTAB, DT_STRTAB or a symbol hash", m->pub.name);
if (hash) {
m->nsyms = rd32(hash + 4); // nchain
} else {
// DT_GNU_HASH only (1.4.3 sdk, subsdk0): one past the last symbol the highest bucket's chain reaches.
uint32_t nbuckets = rd32(gnu_hash), symoffset = rd32(gnu_hash + 4), bloom = rd32(gnu_hash + 8);
uint64_t buckets = gnu_hash + 16 + 8ull * bloom, chains = buckets + 4ull * nbuckets;
uint32_t last = 0;
for (uint32_t i = 0; i < nbuckets; i++)
if (rd32(buckets + 4ull * i) > last) last = rd32(buckets + 4ull * i);
if (last < symoffset) {
m->nsyms = symoffset;
} else {
while (!(rd32(chains + 4ull * (last - symoffset)) & 1)) last++;
m->nsyms = last + 1;
}
}
m->relr = (const uint64_t *)(uintptr_t)relr;
m->nrelr = relrsz / 8;
m->rela = (const Elf64Rela *)(uintptr_t)rela;
m->nrela = relasz / sizeof(Elf64Rela);
m->jmprel = (const Elf64Rela *)(uintptr_t)jmprel;
@@ -363,6 +386,33 @@ static void relocate_table(Mod *m, const Elf64Rela *r, uint64_t n) {
}
}
// DT_RELR: an even entry is an offset to relocate (the next bitmap starts after it); an odd entry is a bitmap of the
// 63 words that follow. Each relocated word holds its addend, as R_AARCH64_RELATIVE with an implicit addend.
static void relocate_relr(Mod *m) {
uint64_t base = m->pub.base, where = 0;
for (uint64_t i = 0; i < m->nrelr; i++) {
uint64_t e = m->relr[i];
for (int k = -1; k < 63; k++) {
uint64_t off;
if (k < 0) {
if (e & 1) continue;
off = e;
where = e + 8;
} else {
if (!(e & 1) || !((e >> (k + 1)) & 1)) continue;
off = where + 8ull * k;
}
if (off < m->pub.data_off || off > m->pub.size - 8)
rt_fatal("loader: %s RELR entry %llu writes +%#llx, outside the data segment", m->pub.name,
(unsigned long long)i, (unsigned long long)off);
ST64(base + off, base + LD64(base + off));
m->kinds[K_RELATIVE]++;
m->relr_count++;
}
if (e & 1) where += 63 * 8;
}
}
// Import table statistics: every undefined named symbol, bound by guest definition (overrides aside).
static void count_imports(Mod *m) {
for (uint32_t i = 1; i < m->nsyms; i++) {
@@ -416,6 +466,7 @@ void ldr_load(const char *exefs_dir) {
for (int i = 0; i < nmods; i++) {
relocate_table(&mods[i], mods[i].rela, mods[i].nrela);
relocate_table(&mods[i], mods[i].jmprel, mods[i].njmprel);
relocate_relr(&mods[i]);
count_imports(&mods[i]);
mods[i].pub.ctor_flag = decode_ctor_flag(&mods[i]);
}
@@ -425,7 +476,7 @@ void ldr_load(const char *exefs_dir) {
const Mod *m = &mods[i];
rt_log("loaded %-8s at %#llx size %#llx: %llu relocations, %u imports", m->pub.name,
(unsigned long long)m->pub.base, (unsigned long long)m->pub.size,
(unsigned long long)(m->nrela + m->njmprel), m->imports);
(unsigned long long)(m->nrela + m->njmprel + m->relr_count), m->imports);
}
}
@@ -515,16 +566,23 @@ uint64_t ldr_plt_stub(const char *module, const char *name) {
}
// ---- --probe-loader
// Expected values, measured on the 1.2.1 modules.
static const struct {
// Expected values, measured on the 1.2.1 and 1.4.3 modules (1.4.3 relocation totals match MNG v1.4.3-port.md:20).
typedef struct {
const char *name;
uint64_t kinds[K_COUNT], total, sym_import, sym_own;
uint32_t imports, to[MOD_COUNT], weak, init_off, init_array;
} expect[] = {
} Expect;
static const Expect expect_121[] = {
{"main", {508816, 1415, 32, 641}, 510904, 2088, 0, 675, {0, 16, 647}, 12, 0x30, 10673},
{"subsdk0", {1623, 12820, 1030, 3924}, 19397, 6576, 11198, 667, {9, 0, 654}, 4, 0x30, 45},
{"sdk", {10917, 13976, 1207, 8932}, 35032, 16, 24099, 13, {8, 0, 0}, 5, 0x30, 116},
};
static const Expect expect_143[] = {
{"main", {499078, 1393, 28, 650}, 501149, 2071, 0, 680, {0, 16, 656}, 8, 0x10b4f10, 10717},
{"subsdk0", {3424, 10927, 905, 3817}, 19073, 6557, 9092, 663, {9, 0, 654}, 0, 0x20, 45},
{"sdk", {15013, 10135, 1079, 7832}, 34059, 10, 19036, 8, {8, 0, 0}, 0, 0x400, 83},
};
#define expect (RT_V143 ? expect_143 : expect_121)
static int failures;
#define CHECK(cond, ...) do { if (!(cond)) { failures++; printf(" FAIL: " __VA_ARGS__); putchar('\n'); } } while (0)
@@ -614,7 +672,7 @@ int probe_loader(void) {
const RtConfig *cfg = rt_config();
hostfn_add_override(PROBE_SYMBOL, probe_override);
ldr_load(cfg->exefs);
if (nmods != (int)(sizeof expect / sizeof *expect)) {
if (nmods != (int)(sizeof expect_121 / sizeof *expect_121)) {
printf("probe-loader: FAILED (%d modules, expected 3)\n", nmods);
return 1;
}
+12 -5
View File
@@ -123,8 +123,9 @@ const pool_t *nvn_pool_find(uint64_t addr, uint64_t len) {
return p;
}
// Physical and virtual pools (MacNativeGames totk_vpool.c:32-52 checks). A CPU-invisible one gets a GPU window range;
// a CPU-visible one would need aliased guest storage, which this port does not support (section 3.3).
// Physical and virtual pools (MacNativeGames totk_vpool.c:32-52 checks). A CPU-invisible one gets a GPU window range.
// A CPU-visible physical pool (1.4.3: flags 0x18c, GPU_NO_ACCESS) keeps the game's storage for the CPU; the back end
// gives it device memory as any physical pool (vk_memory.c vk_pool_init), which the GPU reaches through virtual pools.
static int pool_place(pool_t *o) {
int phys = (o->flags & NVN_MEMORY_POOL_FLAGS_PHYSICAL) != 0, virt = (o->flags & NVN_MEMORY_POOL_FLAGS_VIRTUAL) != 0;
if (!phys && !virt) return 1;
@@ -136,9 +137,15 @@ static int pool_place(pool_t *o) {
}
if (virt && o->mem) { rt_log("nvn: virtual pool given storage %p", (void *)o->mem); return 0; }
if (phys && !o->mem) { rt_log("nvn: physical pool without storage"); return 0; }
if (!(o->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS))
rt_fatal("nvn: CPU-visible %s pool (flags %#x, size %#llx) needs aliased guest storage",
kind, o->flags, (unsigned long long)o->size);
if (!(o->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS)) {
if (virt) {
rt_log("nvn: CPU-visible virtual pool (flags %#x, size %#llx) refused", o->flags, (unsigned long long)o->size);
return 0;
}
rt_log("nvn: CPU-visible physical pool %p (flags %#x, size %#llx) keeps the game's storage", (void *)o->mem,
o->flags, (unsigned long long)o->size);
return 1;
}
uint64_t a = gva_alloc(o->size);
if (!a) {
rt_log("nvn: GPU address window full, %s pool of %#llx bytes refused", kind, (unsigned long long)o->size);
+2
View File
@@ -854,6 +854,8 @@ static void ov_get_output_md(Cpu *c) {
HOST_OVERRIDE("_ZN5movie7DecoderC1ENS_11DecoderTypeENS_19DecoderOutputFormatENS_11DecoderModeEi", ov_ctor);
HOST_OVERRIDE("_ZN5movie7DecoderD2Ev", ov_d2);
// 1.4.3's vtable and a PLT call name the complete-object destructor; without virtual bases it is D2.
HOST_OVERRIDE("_ZN5movie7DecoderD1Ev", ov_d2);
HOST_OVERRIDE("_ZN5movie7DecoderD0Ev", ov_d0);
HOST_OVERRIDE("_ZN5movie7Decoder9ConfigureEPNS_9MediaDataE", ov_configure);
HOST_OVERRIDE("_ZN5movie7Decoder20RegisterClientEventsEPNS_13DecoderEventsE", ov_register_events);
+10 -4
View File
@@ -75,6 +75,7 @@ static Ov ovs[] = {
// SDK-internal (section 5.2)
{"nndiagStartup", "nndiagStartup"},
{"aoc.Initialize", "_ZN2nn3aoc10InitializeEv"},
{"aoc.ListAddOnContent", "_ZN2nn3aoc16ListAddOnContentEPiii"},
{"err.InitializeContextWriterApi", "_ZN2nn3err26InitializeContextWriterApiEv"},
{"pctl.Initialize", "_ZN2nn4pctl10InitializeEv"},
{"diag.InitializeApplicationAbortObserver", "_ZN2nn4diag34InitializeApplicationAbortObserverEv"},
@@ -374,6 +375,7 @@ static void probe_applet(void) {
CALL("err.ShowError", 0x2a8);
CALL("nndiagStartup", 0);
CALL("aoc.Initialize", 0);
CHECK(CALL("aoc.ListAddOnContent", U(out), 0, 8) == 0, "aoc: no add-on content");
CALL("err.InitializeContextWriterApi", 0);
CALL("pctl.Initialize", 0);
CALL("diag.InitializeApplicationAbortObserver", 0);
@@ -628,10 +630,12 @@ static void probe_media_gfx(void) {
"vsync event %s\n", display, layer, window, vsync ? "bound (auto-clear)" : "FAILED");
}
// ---- software keyboard (swkbd_win.c): ShowKeyboardArg built by the 1.2.1 SDK's own setters, as main+0x1b2b3a0 does
// ---- software keyboard (swkbd_win.c): ShowKeyboardArg built by the SDK's own setters, as the horse naming does
enum { KB_TYPE = 0x00, KB_DISABLED = 0x1c, KB_HEADER = 0x24, KB_GUIDE = 0x1a8, KB_MAX_LEN = 0x3ac, KB_MIN_LEN = 0x3b0,
KB_UTF8 = 0x3bd, KB_INIT_LEN = 0x3c4, ARG_WORK = 0x4d0, ARG_WORK_SIZE = 0x4d8, ARG_SIZE = 0x500,
STRING_SIZE = 0x7d4, SWKBD_CANCELED = 0x29f };
KB_UTF8 = 0x3bd, KB_INIT_LEN = 0x3c4, STRING_SIZE = 0x7d4, SWKBD_CANCELED = 0x29f };
#define ARG_WORK (RT_V143 ? 0x518 : 0x4d0) // SDK 20.5 grew KeyboardConfig (swkbd_win.c)
#define ARG_WORK_SIZE (ARG_WORK + 8)
#define ARG_SIZE (RT_V143 ? 0x548 : 0x500)
static uint64_t sdk(const char *name) {
uint64_t a = ldr_symbol(name);
@@ -964,7 +968,9 @@ int probe_services(void) {
for (int k = 0; k < NOBS; k++)
if (!strcmp(observers[k], n)) obs_bound++;
}
for (int k = 0; k < NOV; k++) CHECK(ovs[k].bound, "override %s matches no relocation", ovs[k].name);
for (int k = 0; k < NOV; k++) // 1.2.1 lists no add-on content
CHECK(ovs[k].bound || (!RT_V143 && !strcmp(ovs[k].key, "aoc.ListAddOnContent")), "override %s matches no relocation",
ovs[k].name);
for (int k = 0; k < NOBS; k++) CHECK(hostfn_observer(observers[k]), "observer %s is not linked", observers[k]);
CHECK(obs_bound == NOBS, "%d of %d observers match a JUMP_SLOT", obs_bound, NOBS);
+8
View File
@@ -301,6 +301,13 @@ static void ov_show_error(Cpu *c) {
// ==== SDK-internal initialisation (services.c:208-218)
static void ov_diag_startup(Cpu *c) { (void)c; rt_log("diag: host diagnostics"); }
static void ov_aoc_initialize(Cpu *c) { (void)c; rt_log("aoc: no add-on content"); }
// 1.4.3's main lists add-on content (MNG services.c host_aoc_list): there is none.
// int ListAddOnContent(AddOnContentIndex *out, int offset, int count)
static void ov_aoc_list(Cpu *c) {
int offset = (int)arg_x(c, 1), count = (int)arg_x(c, 2);
if (offset < 0 || count < 0) rt_fatal("aoc: ListAddOnContent range %d+%d", offset, count);
ret_x(c, 0);
}
static void ov_err_context(Cpu *c) { (void)c; rt_log("err: error contexts stay local"); }
static void ov_pctl_initialize(Cpu *c) { (void)c; rt_log("pctl: no parental controls"); }
static void ov_abort_observer(Cpu *c) { (void)c; rt_log("diag: the host crash reporter observes aborts"); }
@@ -820,6 +827,7 @@ HOST_OVERRIDE("_ZN2nn3err9ShowErrorENS_6ResultE", ov_show_error);
HOST_OVERRIDE("nndiagStartup", ov_diag_startup);
HOST_OVERRIDE("_ZN2nn3aoc10InitializeEv", ov_aoc_initialize);
HOST_OVERRIDE("_ZN2nn3aoc16ListAddOnContentEPiii", ov_aoc_list);
HOST_OVERRIDE("_ZN2nn3err26InitializeContextWriterApiEv", ov_err_context);
HOST_OVERRIDE("_ZN2nn4pctl10InitializeEv", ov_pctl_initialize);
HOST_OVERRIDE("_ZN2nn4diag34InitializeApplicationAbortObserverEv", ov_abort_observer);
+2 -1
View File
@@ -6,7 +6,8 @@
//
// The patch page is the first 4 KiB page after main's image, inside the module window memory.c reserved: replacement
// code reaches it with one adrp, as MacNativeGames' data page after main (games/totk/runtime/game_patches.c). Its
// layout belongs to the sets that use it (frame_rate.c: +0 the 30 Hz ratio, +4 the frame step in seconds).
// layout belongs to the sets that use it (frame_rate.c: +0 the 30 Hz ratio, +4 the frame step in seconds, +8 the
// step in 60 Hz ticks).
// Probe modes never call rt_patches_init: every set stays off.
#include "cfg.h"
#include "platform/platform.h"
+5 -3
View File
@@ -30,12 +30,14 @@
#include "ui_text.h"
#include "vk.h"
// KeyboardConfig (0x4d0 bytes) then the work buffer {pointer, size}: offsets read from the SDK setters (MNG) and
// checked against the 1.2.1 SDK by --probe-services.
// KeyboardConfig then the work buffer {pointer, size}: offsets read from the SDK setters (MNG) and checked against
// the SDK by --probe-services. SDK 20.5 (1.4.3) put a text-check callback at +0x4c8 and grew KeyboardConfig from
// 0x4d0 to 0x518 bytes (SetInitialText, SetUserWordList read the work buffer there); the fields below +0x3d8 stay.
#define ARG_WORK (RT_V143 ? 0x518 : 0x4d0)
#define ARG_WORK_SIZE (ARG_WORK + 8)
enum {
KB_TYPE = 0x00, KB_OK_TEXT = 0x04, KB_DISABLED = 0x1c, KB_HEADER = 0x24, KB_SUB = 0xa6, KB_GUIDE = 0x1a8,
KB_MAX_LEN = 0x3ac, KB_MIN_LEN = 0x3b0, KB_PASSWORD = 0x3b4, KB_UTF8 = 0x3bd, KB_INIT_OFF = 0x3c0, KB_INIT_LEN = 0x3c4,
ARG_WORK = 0x4d0, ARG_WORK_SIZE = 0x4d8,
KB_TYPE_NUMPAD = 1, KB_LIMIT = 500, SWKBD_CANCELED = 0x29f,
};
enum { B_A = 0, B_B = 1, B_PLUS = 10 }; // input.h bits
+3 -1
View File
@@ -242,7 +242,9 @@ static void destroy_rec(VkCtx *c, PoolRec *r) {
int vk_pool_init(pool_t *p) {
VkCtx *c = vk_ctx();
if (!c) { rt_log("vk: MemoryPoolInitialize before vk_init"); return 0; }
// The GPU never sees a GPU_NO_ACCESS pool, except a physical one (0x189) through the virtual pools that map it.
// The GPU never sees a GPU_NO_ACCESS pool, except a physical one (0x189; 1.4.3 also 0x18c, CPU-visible) through the
// virtual pools that map it. Either gets device-local memory: 1.4.3's CPU never writes its 0x18c pool's storage
// (unchanged over 3600 frames of the intro), so the guest storage and the GPU copy need no sync.
if ((p->flags & NVN_MEMORY_POOL_FLAGS_GPU_NO_ACCESS) && !(p->flags & NVN_MEMORY_POOL_FLAGS_PHYSICAL)) return 1;
PoolRec *r = calloc(1, sizeof *r);
if (!r) rt_fatal("vk: out of memory for a pool record");
+1 -1
View File
@@ -76,7 +76,7 @@ class Bed:
def run(self, release='1.0', inputs=None, fail=None, verify_ok=None, verify=False):
layout = Layout(root=Path('.'), dev=True, release=release, python=Path('python'), tools=Path('tools'),
recomp=Path('recomp'), patches=Path('p'), expected=Path('e'), llvm=Path('llvm'),
recomp=Path('recomp'), llvm=Path('llvm'),
deps=Path('deps'), runtime=Path('rt'), runtime_lib=Path('lib'), ninja=Path('ninja'),
hactool=Path('hactool'), xlate=Path('xlate'))
options = SimpleNamespace(inputs=inputs or {}, fail=dict(fail or {}), verify_ok=verify_ok or {}, ran=[],
+234 -6
View File
@@ -55,6 +55,10 @@ class Oracle:
mu.reg_write(A.UC_ARM64_REG_CPACR_EL1, 0x300000)
self.mu = mu
self.set_at = self.regs = None
self.start = self.cmp = None # the first instruction of the bounds-check block, the bounds cmp
self.guard = None # _taken: the b.ls/b.lo into the dispatch block (set_at)
self.count = None # indices the bounds check lets through
self.exclude = frozenset() # indices the emulation skips (oracle_wrong: excluded on every way in)
self.watch = self.seen = None # (address, register): record the register when the emulation reaches address
def word(self, a):
@@ -65,7 +69,7 @@ class Oracle:
def jump_table(self, br_pc, entry, end):
"""Targets, or None. Leaves the index set point and the registers set there in self.set_at, self.regs."""
self.set_at = self.regs = None
self.set_at = self.regs = self.start = self.cmp = self.guard = self.count = None
r = self._fallthrough(br_pc, entry, end)
if r is None:
r = self._taken(br_pc, entry, end)
@@ -127,6 +131,8 @@ class Oracle:
targets = []
try:
for i in range(count):
if i in self.exclude:
continue
self._reset(entry, start)
if set_at != start and not self._step_to(start, set_at):
return None
@@ -159,7 +165,11 @@ class Oracle:
count = limit + 1 if bits(w, 0, 4) == 9 else limit
if count <= 0 or count > 4096:
continue
self.set_at, self.regs = s, [bits(cw, 5, 5)]
self.set_at, self.regs, self.guard, self.cmp, self.count = s, [bits(cw, 5, 5)], g, g - 4, count
b = g - 4
while b - 4 >= entry and g - 4 - (b - 4) <= 4 * 12 and self.kind(b - 4) in ('op', 'asm'):
b -= 4
self.start = b
return self._emulate(br_pc, entry, end, s, [bits(cw, 5, 5)], count, s)
return None
@@ -197,11 +207,13 @@ class Oracle:
while start - 4 >= entry and cmp - (start - 4) <= 4 * 12 and self.kind(start - 4) in ('op', 'asm'):
start -= 4
regs = self._index_regs(start, cmp, bits(w, 5, 5))
self.set_at, self.regs = cmp, regs
self.set_at, self.regs, self.start, self.cmp, self.count = cmp, regs, start, cmp, count
rn = bits(self.word(br_pc), 5, 5)
targets = []
try:
for i in range(count):
if i in self.exclude:
continue
self._reset(entry, start)
if not self._step_to(start, cmp):
return None
@@ -397,13 +409,218 @@ def _reads(ins):
return {n for n in (_regno(ins.reg_name(x)) for x in r) if n is not None}
def oracle_wrong(fc, br, oracle):
def _branch_target(ins):
"""Target of a direct b, b.cond, cbz/cbnz or tbz/tbnz, else None."""
if ins is None or not ins.operands or not (ins.mnemonic in ('b', 'cbz', 'cbnz', 'tbz', 'tbnz')
or ins.mnemonic.startswith('b.')):
return None
op = ins.operands[-1]
return op.imm if op.type == capstone.arm64.ARM64_OP_IMM else None
def _ways_in(fc, lo, hi, tables):
"""Direct branches of the function into [lo, hi], or None when the range is entered otherwise too: by falling
through into lo, at the function entry, or as a target of another table."""
mod = fc.mod
if lo <= fc.entry or any(lo <= t <= hi for ts in tables.values() if ts != 'all' for t in ts):
return None
prev = _insn(mod, lo - 4)
if prev is None or prev.mnemonic not in ('b', 'br', 'ret'):
return None
out = []
for a in range(fc.entry, fc.end, 4):
ins = _insn(mod, a)
t = _branch_target(ins)
if t is not None and lo <= t <= hi:
out.append((a, ins, t))
return out
# Instructions an index slice may hold: integer arithmetic that reads only general registers and writes the flags at
# most (the flags-reading forms adc/sbc/csel/ccmp/... stay out).
_FLAG_SETTERS = {'cmp', 'cmn', 'tst', 'ands', 'bics', 'adds', 'subs', 'negs'}
_PURE_ALU = {'mov', 'movz', 'movn', 'movk', 'add', 'sub', 'and', 'orr', 'eor', 'bic', 'orn', 'eon', 'lsl', 'lsr',
'asr', 'ror', 'ubfx', 'sbfx', 'ubfiz', 'sbfiz', 'bfi', 'bfxil', 'uxtb', 'uxth', 'uxtw', 'sxtb', 'sxth',
'sxtw', 'neg', 'mvn', 'mul', 'madd', 'msub', 'udiv', 'sdiv', 'ubfm', 'sbfm', 'bfm'} | _FLAG_SETTERS
_DEPTH = 4 # predecessor blocks above the dispatch region
def _ends_block(ins):
return ins is None or _branch_target(ins) is not None or ins.mnemonic in ('br', 'ret', 'brk', 'udf')
class _Ways:
"""Direct branches and fall-throughs of one function, with the addresses control may enter from elsewhere."""
def __init__(self, fc, tables):
mod = self.mod = fc.mod
self.entry, self.into = fc.entry, {}
for a in range(fc.entry, fc.end, 4):
ins = _insn(mod, a)
t = _branch_target(ins)
if t is not None and fc.entry <= t < fc.end:
self.into.setdefault(t, []).append((a, ins))
self.roots = {fc.entry} | set(fc.cfg.unknown)
self.roots |= {t for ts in tables.values() if ts != 'all' for t in ts}
self.leaders = self.roots | set(self.into)
def block_start(self, a):
s = a
while s > self.entry and s not in self.leaders and not _ends_block(_insn(self.mod, s - 4)):
s -= 4
return s
def preds(self, s):
"""(branch or last instruction, insn, taken) for every way into s."""
out = [(a, ins, True) for a, ins in self.into.get(s, ())]
if s > self.entry:
p = _insn(self.mod, s - 4)
if p is not None and p.mnemonic not in ('b', 'br', 'ret', 'brk', 'udf'):
out.append((s - 4, p, False))
return out
def _edge_holds(oracle, ways, start, src, ins, taken, reg, k):
"""Whether the way out of the block [start, src] over the branch at src can be taken with x<reg> = k at src:
False only when the branch condition is a function of x<reg> alone (its slice, back to the last write of x<reg>,
is plain arithmetic and flag setting on x<reg>), evaluated by Unicorn; True otherwise."""
if _branch_target(ins) is None or ins.mnemonic == 'b':
return True
mod = ways.mod
def gp_reads(i): # general registers read, zr dropped; None when sp or a non-general register is read
try:
r, _ = i.regs_access()
except capstone.CsError:
return None
if i.mnemonic in ('cmp', 'cmn', 'tst'): # capstone lists their first operand as written, not read
r = list(r) + [op.reg for op in i.operands if op.type == capstone.arm64.ARM64_OP_REG]
out = set()
for x in r:
name = i.reg_name(x)
if name in ('xzr', 'wzr', 'nzcv'):
continue
n = _regno(name)
if n is None or n == 31:
return None
out.add(n)
return out
def sets_flags(i):
try:
return i.mnemonic in _FLAG_SETTERS or 'nzcv' in (i.reg_name(r) for r in i.regs_access()[1])
except capstone.CsError:
return True
if ins.mnemonic.startswith('b.'):
setter = next((x for x in range(src - 4, start - 4, -4) if sets_flags(_insn(mod, x))), None)
if setter is None or _insn(mod, setter).mnemonic not in _FLAG_SETTERS or any(reg in _writes(_insn(mod, x)) for x in range(setter, src, 4)):
return True
need, top, sl = gp_reads(_insn(mod, setter)), setter, [setter]
else:
need, top, sl = gp_reads(ins), src, []
if need is None:
return True
for x in range(top - 4, start - 4, -4):
i = _insn(mod, x)
w = _writes(i)
if reg in w:
break # x<reg> is k from here on
if not w & need:
continue
r = gp_reads(i)
if i.mnemonic not in _PURE_ALU or r is None:
return True
sl.append(x)
need = (need - w) | r
if need - {reg}:
return True
mu = oracle.mu
try:
for r in range(29):
mu.reg_write(A.UC_ARM64_REG_X0 + r, SCRATCH_MEM + 0x80000)
mu.reg_write(A.UC_ARM64_REG_X0 + reg if reg < 29 else A.UC_ARM64_REG_X29, k)
for x in reversed(sl):
mu.emu_start(x, 0, count=1)
mu.emu_start(src, 0, count=1)
went = mu.reg_read(A.UC_ARM64_REG_PC) != src + 4
except uc.UcError:
return True
return went == taken
def _way_live(oracle, ways, src, ins, taken, reg, k, depth, seen):
"""Whether the way over src can carry x<reg> = k: its edge holds (_edge_holds) and its block either writes x<reg>
or is entered with k (_reaches)."""
start = ways.block_start(src)
if not _edge_holds(oracle, ways, start, src, ins, taken, reg, k):
return False
if any(reg in _writes(_insn(ways.mod, x)) for x in range(start, src + 4, 4)):
return True
return _reaches(oracle, ways, start, reg, k, depth, seen)
def _reaches(oracle, ways, s, reg, k, depth, seen):
"""Whether control can enter s with x<reg> = k, over at most `depth` predecessor blocks."""
if s in ways.roots or depth == 0 or s in seen:
return True
seen = seen | {s}
return any(_way_live(oracle, ways, src, ins, taken, reg, k, depth - 1, seen)
for src, ins, taken in ways.preds(s))
def excluded_indices(fc, br, oracle, tables, count):
"""Indices < count that cannot reach br: no way into the dispatch region [oracle.start, br] (its first instruction
and every branch target inside it) can carry them. The index is the register of the oracle's bounds cmp, which no
instruction from such an entry to br may write. Ways in are the module's direct branches and fall-throughs by
capstone; the function entry, exception landing pads and every resolver table target count as entries from
anywhere. Branch conditions are evaluated only where they depend on the index alone (_edge_holds). When the
oracle enters the dispatch block by the taken bounds branch (_taken), that branch must be its only way in."""
mod, lo = fc.mod, oracle.start
if lo is None:
return frozenset()
if oracle.guard is not None:
ways = _ways_in(fc, oracle.set_at, br, tables)
if ways is None or [a for a, _, _ in ways] != [oracle.guard]:
return frozenset()
reg = bits(oracle.word(oracle.cmp), 5, 5)
ways = _Ways(fc, tables)
# The region falls through from lo to br (_fallthrough), or is the bounds-check block whose b.ls/b.lo is the only
# way into the dispatch block (_taken).
hi = br if oracle.guard is None else oracle.cmp
if oracle.guard is not None and any(reg in _writes(_insn(mod, x)) for x in range(oracle.set_at, br, 4)):
return frozenset()
entries = [lo] + [t for t in range(lo + 4, hi + 4, 4) if t in ways.leaders]
for e in entries:
if e in ways.roots or any(reg in _writes(_insn(mod, x)) for x in range(e, hi, 4)):
return frozenset()
out = set()
for k in range(count):
live = False
for e in entries:
for src, ins, taken in ways.preds(e):
if lo <= src < hi:
continue # inside the region: its own conditions are ignored
if _way_live(oracle, ways, src, ins, taken, reg, k, _DEPTH, frozenset()):
live = True
break
if live:
break
if not live:
out.add(k)
return frozenset(out)
def oracle_wrong(fc, br, oracle, ours=None):
"""Why the oracle's targets for br are not the table's, from evidence independent of the resolver; or None.
- the index the table is read with does not depend on the registers the oracle sets (it comes from another
register or is reloaded from memory, which the oracle leaves at its scratch value);
- the table base the oracle uses (its replay of ADRP/ADR/ADD-imm from the entry) is not the one constant that
every reaching definition gives.
every reaching definition gives;
- the oracle enumerates an index that no way into the dispatch region lets through (excluded_indices: a case
peeled off before the table by `cmp wI, #k; b.ne`, `cbnz wI` or a range check on wI, possibly some blocks up),
and without it the oracle gives the resolver's targets.
"""
mod = fc.mod
s = fc.cfg.block(br)
@@ -427,6 +644,17 @@ def oracle_wrong(fc, br, oracle):
seen, oracle.watch, oracle.seen = oracle.seen, None, None
if seen is not None and seen != value:
return f'oracle base x{reg} = {seen:#x}, every reaching definition gives {value:#x}'
if ours is not None:
oracle.jump_table(br, fc.entry, fc.end)
ex = excluded_indices(fc, br, oracle, fc.tables, oracle.count) if oracle.count else frozenset()
if ex:
oracle.exclude = ex
try:
again = oracle.jump_table(br, fc.entry, fc.end)
finally:
oracle.exclude = frozenset()
if again == ours:
return 'index ' + ', '.join(str(k) for k in sorted(ex)) + ' unreachable on every way in'
return None
@@ -535,7 +763,7 @@ def main():
else:
key = 'computed, unresolved' if comp else None
if key in ('disagree', 'only oracle'):
why = oracle_wrong(fc, br, oracle)
why = oracle_wrong(fc, br, oracle, ours)
if why:
key = 'oracle wrong' if key == 'disagree' else 'only oracle, oracle wrong'
if key or raw is not None:
+10 -6
View File
@@ -187,9 +187,9 @@ def runtime_object(out, src):
return out / 'obj' / 'runtime' / rel.with_suffix('.o')
def module_glue(layout, linked, hashes, paths):
def module_glue(layout, linked, hashes, paths, version):
"""C for build/<version>/modules.c: the layout table, the linked modules' function tables (dispatch.h), the
expected ExeFS hashes and the default game paths (host.h)."""
expected ExeFS hashes, the game version and the default game paths (host.h)."""
out = [f'// Module glue written by tools/build_game.py; do not edit. {GENERATED_MARKER}',
'#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
for name, base, size in layout:
@@ -213,7 +213,8 @@ def module_glue(layout, linked, hashes, paths):
out += ['};', f'const int rt_exefs_sha256_count = {len(hashes)};', '']
for name in ('exefs', 'romfs', 'profile'):
out.append(f'const char rt_default_{name}[] = {c_string(paths[name].as_posix())};')
out.append('')
number = int(DISPLAY_VERSIONS[version].replace('.', '')) if version in DISPLAY_VERSIONS else 0 # 1.4.3 -> 143
out += [f'const int rt_game_version = {number};', '']
return '\n'.join(out)
@@ -584,7 +585,9 @@ def main():
help="totk.exe's default ExeFS directory (default build/extracted/<version>/exefs)")
ap.add_argument('--romfs', type=Path, help="totk.exe's default RomFS directory with romfs.bin and index.json "
'(default build/extracted/<version>/romfs)')
ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile)")
ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile for v121, "
"build/profile-<version> otherwise, so trying another version leaves "
"the main profile alone; saves load on 1.2.1 and 1.4.3 alike)")
ap.add_argument('--toolchain', type=Path, help='llvm-mingw directory (with bin/clang.exe) to build with instead of '
'MSYS2 UCRT64 clang')
ap.add_argument('--ninja', type=Path, help='ninja.exe to build with (default: MSYS2 UCRT64, else bin/ninja.exe of the '
@@ -604,7 +607,8 @@ def main():
extracted = ROOT / 'build' / 'extracted' / args.version
paths = {'exefs': (args.exefs or extracted / 'exefs').resolve(),
'romfs': (args.romfs or extracted / 'romfs').resolve(),
'profile': (args.profile or ROOT / 'build' / 'profile').resolve()}
'profile': (args.profile or ROOT / 'build' / ('profile' if args.version == 'v121'
else f'profile-{args.version}')).resolve()}
expected = args.expected
if not expected:
if args.version not in DISPLAY_VERSIONS:
@@ -645,7 +649,7 @@ def main():
missing = [n for n in names if n not in hashes]
if missing:
sys.exit(f'{expected}: no exefs_sha256 for {missing}')
write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths))
write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths, args.version))
if (paths['romfs'] / 'index.json').is_file():
count, size, dt = romfs_index.write(paths['romfs'], out)
print(f'romfs index: {count} files, {mib(size)}, {dt:.1f} s')
+4 -4
View File
@@ -41,6 +41,7 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import check_content # noqa: E402
import shader_list # noqa: E402
ROOT = Path(__file__).resolve().parent.parent
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/, generated/, toolchain/ and refs/ are junctions in a worktree
@@ -95,7 +96,7 @@ LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'}
# tools/ files the installer steps run (and what they import), copied from the repo.
TOOLS_FILES = ['build_game.py', 'check_content.py', 'extract.py', 'make_romfs_patches.py', 'prepare_shaders.py',
'romfs_index.py', 'shader_census.py']
'romfs_index.py', 'shader_census.py', 'shader_list.py']
TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes
RUNTIME_SUFFIXES = {'.h', '.inc', '.ico'} # .ico: the installer window icon
@@ -343,9 +344,8 @@ def build_sources(out):
for d in TOOLS_DIRS:
copy_tree(ROOT / 'tools' / d, out / 'tools' / d, lambda r: '__pycache__' in r or r.endswith('.md'))
(out / 'data').mkdir(exist_ok=True)
text = subprocess.run(['git', '-C', MAIN_ROOT / 'refs' / 'MacNativeGames', 'show',
f'{MNG_COMMIT}:games/totk/assets/shader_list.tsv'], check=True, capture_output=True).stdout
(out / 'data' / 'shader_list.tsv').write_bytes(text)
text = shader_list.history(MAIN_ROOT / 'refs' / 'MacNativeGames', MNG_COMMIT)
(out / 'data' / 'shader_list.tsv').write_text(text, encoding='utf-8', newline='\n')
copy_tree(ROOT / 'installer', out / 'installer', lambda r: '__pycache__' in r or r.startswith('launcher/'))
shutil.copy2(ROOT / 'totk.ini.example', out / 'totk.ini.example')
shutil.copy2(ROOT / 'THIRD_PARTY_NOTICES.md', out / 'THIRD_PARTY_NOTICES.md')
+13 -8
View File
@@ -1,17 +1,18 @@
# Adapted from MacNativeGames games/totk/scripts/make_romfs_patches.py @ ebe621d (MIT, Copyright (c) 2026
# MacNativeGames contributors).
"""Build the 60 fps RomFS overlay for TotK 1.2.1 (runtime/romfs.c, TOTK_ROMFS_PATCHES).
"""Build the 60 fps RomFS overlay for TotK 1.2.1 or 1.4.3 (runtime/romfs.c, TOTK_ROMFS_PATCHES).
UI/LayoutArchive/Common.Product.110.Nin_NX_NVN.blarc.zs: the weapon / shield / material quick menus capture the game
UI/LayoutArchive/Common.Product.<110 or 140>.Nin_NX_NVN.blarc.zs: the weapon / shield / material quick menus capture the game
picture for their background when the panes C_CaptureBG_00, C_CaptureClearBG_00 and P_CaptureClearBG_00 become
visible at animation frame 1.0 (anim/ShortCutBG_00_In.bflan), AutoBuilder the same with C_CaptureBG_00
(anim/AutoBuilder_00_In.bflan). UI animations advance 0.5 frames per game frame at 60 fps, so the capture lands after
the world stopped drawing and the background can be black. The visibility keys move to frame 0.0: the live world is
captured at once (upstream checked weapon, shield, material and ability wheel at 60 fps, and 30 fps).
The game-over flow fix of the same upstream script is 1.0.0 only (1.2.1 runs DmF_SY_GameOver.110) and is left out.
The game-over flow fix of the same upstream script is 1.0.0 only (1.2.1 and 1.4.3 run DmF_SY_GameOver.110) and is
left out.
Input: build/extracted/v121/romfs (romfs.bin and index.json, verified by tools/extract.py). Output: <out>/<path> and
<out>/manifest.tsv (path, size, SHA-256 of the overlay, SHA-256 of the original). Run with .venv-py314.
Input: build/extracted/<version>/romfs (romfs.bin and index.json, verified by tools/extract.py). Output:
build/<version>/romfs-patches/<path> by default, and <out>/manifest.tsv (path, size, SHA-256 of the overlay, SHA-256 of the original). Run with .venv-py314.
"""
import argparse
import compression.zstd as zstd
@@ -22,9 +23,12 @@ from pathlib import Path
root = Path(__file__).resolve().parents[1]
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--romfs', type=Path, default=root / 'build/extracted/v121/romfs')
ap.add_argument('--out', type=Path, default=root / 'build/v121/romfs-patches')
ap.add_argument('--version', choices=('v121', 'v143'), default='v121')
ap.add_argument('--romfs', type=Path, help='default build/extracted/<version>/romfs')
ap.add_argument('--out', type=Path, help='default build/<version>/romfs-patches')
args = ap.parse_args()
args.romfs = args.romfs or root / 'build/extracted' / args.version / 'romfs'
args.out = args.out or root / 'build' / args.version / 'romfs-patches'
index = json.loads((args.romfs / 'index.json').read_text())
assert index['status'] == 'pass'
files = {f['path']: f for f in index['files']}
@@ -73,7 +77,8 @@ def flvi_keys(data, base):
return out
ARCHIVE = 'UI/LayoutArchive/Common.Product.110.Nin_NX_NVN.blarc.zs'
COMMON = {'v121': '110', 'v143': '140'} # the version in the name of each game's Common archive
ARCHIVE = f'UI/LayoutArchive/Common.Product.{COMMON[args.version]}.Nin_NX_NVN.blarc.zs'
MEMBERS = {
'anim/ShortCutBG_00_In.bflan': ('C_CaptureBG_00', 'C_CaptureClearBG_00', 'P_CaptureClearBG_00'),
'anim/AutoBuilder_00_In.bflan': ('C_CaptureBG_00',),
+11 -7
View File
@@ -1,16 +1,16 @@
"""Package a built totk.exe into a folder that runs from anywhere: build/v121-portable by default.
"""Package a built totk.exe into a folder that runs from anywhere: build/<version>-portable by default.
The folder holds the exe, its DLLs, the shader translator (NativeAOT: no .NET), the RomFS index and patches, the
extracted ExeFS (the game code the exe loads), the shared fonts, the prepared shader and pipeline files. There is
no config file: totk.exe finds the exefs, romfs, profile and shared-fonts folders next to itself. Without --with-romfs
the 16 GB RomFS image stays out; copy build/extracted/v121/romfs/romfs.bin to <folder>/romfs/ or set TOTK_ROMFS
the 16 GB RomFS image stays out; copy build/extracted/<version>/romfs/romfs.bin to <folder>/romfs/ or set TOTK_ROMFS
(a path, relative ones to the exe's folder).
The folder contains game code and data derived from the game: for your own machines only, never publish or share it.
Saves go to <folder>/profile.
.venv/Scripts/python tools/package_build.py [--build build/v121] [--exe PATH/totk.exe] [--out build/v121-portable]
[--with-romfs] [--no-shader-cache]
.venv/Scripts/python tools/package_build.py [--version v121] [--build build/v121] [--exe PATH/totk.exe]
[--out build/v121-portable] [--with-romfs] [--no-shader-cache]
"""
import argparse
import os
@@ -48,14 +48,18 @@ def size_of(path):
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--build', type=Path, default=ROOT / 'build' / 'v121', help='build dir with the prepared state')
ap.add_argument('--version', default='v121', help='game version (v121 or v143): the defaults below follow it')
ap.add_argument('--build', type=Path, help='build dir with the prepared state (default build/<version>)')
ap.add_argument('--exe', type=Path, help='totk.exe to package (default: <build>/totk.exe); its DLLs are taken from --build')
ap.add_argument('--xlate', type=Path, default=ROOT / 'build' / 'xlate' / 'totk_xlate.exe')
ap.add_argument('--extracted', type=Path, default=ROOT / 'build' / 'extracted' / 'v121')
ap.add_argument('--out', type=Path, default=ROOT / 'build' / 'v121-portable')
ap.add_argument('--extracted', type=Path, help='default build/extracted/<version>')
ap.add_argument('--out', type=Path, help='default build/<version>-portable')
ap.add_argument('--with-romfs', action='store_true', help='also copy romfs.bin (about 16 GB)')
ap.add_argument('--no-shader-cache', action='store_true', help='leave out shader-cache (about 2 GB): translates again')
a = ap.parse_args()
a.build = a.build or ROOT / 'build' / a.version
a.extracted = a.extracted or ROOT / 'build' / 'extracted' / a.version
a.out = a.out or ROOT / 'build' / f'{a.version}-portable'
build, out = a.build.resolve(), a.out.resolve()
exe = (a.exe or build / 'totk.exe').resolve()
+23 -15
View File
@@ -1,24 +1,26 @@
"""Prepare the shader variants of the MacNativeGames shared shader list for a build, once, before play.
"""Prepare the default variant of every game shader and the variants of the MacNativeGames shared shader list for a
build, once, before play.
The list (refs/MacNativeGames games/totk/assets/shader_list.tsv, MIT; docs/shader-list.md there) names shader variants
players met: a shader key (sha256 of the NVN control section and code, as tools/shader_census.py names shaders) and
the translator answers it was translated for. This tool reads the shaders from the extracted RomFS and ExeFS
(tools/shader_census.py inventory), keeps the list's variants whose shader is in this game version, and writes them
next to the build's totk.exe in the record layouts of runtime/gfx_shader.c ("answers met in play"):
(tools/shader_census.py inventory), keeps the list's variants whose shader is in this game version, adds each shader's
default variant (every texture slot a 2D texture, float vertex inputs and colour outputs, as MacNativeGames' seed
translates every shader for its default bind state: prepare_shader_seed.py all_default), and writes them next to the
build's totk.exe in the record layouts of runtime/gfx_shader.c ("answers met in play"):
seed-programs.bin "TOTKSPRG": stage, length, c1 length, runtime code hash, control section, code, c1
seed-answers.bin "TOTKSANS": stage, code hash, input id (zero: the runtime computes it), answers, priority
Priority (the pad byte): 0 shaders the list's warm rows flag as title/menu (t), 1 other warm rows (S, C, R), 2 the
rest. Then it runs `totk.exe --prepare-shaders`, which translates every record into the build's shader cache, writes
rest; a default variant takes its shader's priority. Then it runs `totk.exe --prepare-shaders`, which translates every record into the build's shader cache, writes
shader-prepared.bin (one record per variant, in priority order) and builds every variant's stage library or compute
pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "stage library prebuild"). About 5 min on
a 12-core machine (inventory 1, translation 3, library builds 0.6) and 1.3 GB of shader cache; the seed files can be
deleted afterwards.
pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "stage library prebuild"). The seed files
can be deleted afterwards.
Game-derived output stays in the build directory (ignored). The list is read at preparation time from tools/data/shader_list.tsv when that file exists (the installer archive
bundles it; it stays out of the repository), else from refs/ through git; nothing of it is copied into the
repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14:
bundles it; it stays out of the repository), else from refs/ through git with the rows its history dropped
(tools/shader_list.py); nothing of it is copied into the repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14:
.venv-py314\\Scripts\\python tools\\prepare_shaders.py build\\v121 [--version v121] [--list <tsv>] [--no-run]
[--romfs DIR --exefs DIR] # instead of build\\extracted\\<version>
@@ -36,9 +38,9 @@ ROOT = Path(__file__).resolve().parents[1]
while not (ROOT / 'tools' / 'shader_census.py').is_file() and ROOT != ROOT.parent:
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / 'tools'))
import shader_list # noqa: E402
from shader_census import inventory # noqa: E402
LIST_REF = ('origin/HEAD', 'games/totk/assets/shader_list.tsv')
STAGES = {'vs': 0, 'ps': 4, 'cs': 5} # runtime/gfx.h GFX_STAGE_*
TEX = [None, '1d', '1darray', 'buffer', '2d', '2darray', '2dms', '2dmsarray', '3d', 'cube', 'cubearray']
FMT = [None, 'R8Unorm', 'R8Snorm', 'R8Uint', 'R8Sint', 'R16Float', 'R16Unorm', 'R16Snorm', 'R16Uint', 'R16Sint',
@@ -48,6 +50,7 @@ FMT = [None, 'R8Unorm', 'R8Snorm', 'R8Uint', 'R8Sint', 'R16Float', 'R16Unorm', '
'R16G16B16A16Uint', 'R16G16B16A16Sint', 'R32G32B32A32Float', 'R32G32B32A32Uint', 'R32G32B32A32Sint',
'R10G10B10A2Unorm', 'R10G10B10A2Uint', 'R11G11B10Float'] # gfx_shader.c fmt_names
TYPES = {'float': 0, 'sint': 1, 'uint': 2}
DEFAULT = {'tex': {str(slot): '2d' for slot in range(32)}} # answers outside a shader's use mask are dropped
CONTROL = 0x880
ANS_REC = 4 + 32 + 32 + 147 + 1
@@ -62,8 +65,7 @@ def read_list(path):
return Path(path).read_text(encoding='utf-8')
if BUNDLED_LIST.is_file():
return BUNDLED_LIST.read_text(encoding='utf-8')
return subprocess.run(['git', '-C', str(ROOT / 'refs' / 'MacNativeGames'), 'show', ':'.join(LIST_REF)],
check=True, capture_output=True, encoding='utf-8').stdout
return shader_list.history(ROOT / 'refs' / 'MacNativeGames')
def parse_list(text):
@@ -131,7 +133,9 @@ def main():
extracted = ROOT / 'build' / 'extracted' / a.version
shaders, _ = inventory(a.romfs or extracted / 'romfs', a.exefs or extracted / 'exefs')
t1 = time.monotonic()
progs, recs, skipped = {}, [], {'not in this version': 0, 'stage': 0}
progs, recs, seen, skipped = {}, [], set(), {'not in this version': 0, 'stage': 0}
listed = len(variants)
variants += [(key, DEFAULT) for key in sorted(shaders)]
for key, deps in variants:
s = shaders.get(key)
if not s:
@@ -148,7 +152,11 @@ def main():
h = code_hash(stage, control, code, length, c1)
progs[key] = (h, struct.pack('<III', stage, length, len(c1)) + h + control + code[:length] + c1)
h = progs[key][0]
rec = struct.pack('<I', stage) + h + bytes(32) + answers(deps, stage, control) + bytes([prio.get(key, 2)])
packed = answers(deps, stage, control)
if (h, packed) in seen: # a list row of the default bind state
continue
seen.add((h, packed))
rec = struct.pack('<I', stage) + h + bytes(32) + packed + bytes([prio.get(key, 2)])
assert len(rec) == ANS_REC
recs.append((prio.get(key, 2), rec))
recs.sort(key=lambda r: r[0]) # stable: list order within a priority
@@ -162,7 +170,7 @@ def main():
for _, rec in recs:
f.write(rec)
by = [sum(1 for p, _ in recs if p == i) for i in range(3)]
print(f'{len(variants)} listed variants: {len(recs)} of {len(progs)} shaders in {a.version} '
print(f'{listed} listed variants and {len(shaders)} default ones: {len(recs)} of {len(progs)} shaders in {a.version} '
f'(priority t {by[0]}, warm {by[1]}, rest {by[2]}), skipped {skipped}; inventory {t1 - t0:.1f} s, '
f'seed written in {time.monotonic() - t0:.1f} s', flush=True)
if a.no_run:
+28
View File
@@ -0,0 +1,28 @@
"""The MacNativeGames shared shader list (refs/MacNativeGames games/totk/assets/shader_list.tsv, MIT; docs/shader-list.md
there) as our tools read it: every row of one revision, plus the V rows of every older revision that it dropped.
MacNativeGames prunes V rows its own runtime no longer needs: default variants, which its seed translates for every
shader (1.4.3 on 2026-10-22, 1.0.0 to 1.2.1 on 2026-10-25), and variants that differ only in vertex input or colour
output types (2026-10-26). Our answers (runtime/gfx_shader.c) still tell those apart, so a dropped row names a variant
our play meets: at 1340bee the list alone has 134 rows of 1.4.3 shaders, its history 24,839, and of 1.2.1 6,771 and
25,238. A row no draw asks for costs one translation. No copy of the list goes into the repository.
"""
import subprocess
PATH = 'games/totk/assets/shader_list.tsv'
def git(repo, *args):
return subprocess.run(['git', '-C', str(repo), *args], check=True, capture_output=True, encoding='utf-8').stdout
def history(repo, rev='origin/HEAD'):
"""rev's list, then each V row of an older revision that no newer one has, newest first."""
lines = git(repo, 'show', f'{rev}:{PATH}').splitlines()
seen = set(lines)
for h in git(repo, 'log', '--format=%H', rev, '--', PATH).split()[1:]:
for line in git(repo, 'show', f'{h}:{PATH}').splitlines():
if line.startswith('V\t') and line not in seen:
seen.add(line)
lines.append(line)
return '\n'.join(lines) + '\n'