mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-18 20:15:56 +00:00
Merge game version 1.4.3: loader, patch table, SDK 20.5 layouts and probes, the installer choosing 1.2.1 or 1.4.3 by hash, and shader preparation of every shader's default variant plus the list rows MacNativeGames pruned
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
cdecd014f6
32 files changed
+691
-177
No files matched your search
@@ -43,9 +43,10 @@ Setup and verify commands: [docs/toolchain-windows.md](docs/toolchain-windows.md
|
||||
- `tests\diff\atomic.py`: exclusive load/store sequences and ARMv8.1 atomics; needs no game files.
|
||||
- `tests\diff\func.py`: whole leaf functions as the generator emits them, loops and jump tables included.
|
||||
- `tests\excl_mt.py`: the exclusive monitor helpers of `cpu.h` under contention and ABA, on several threads; needs no game files.
|
||||
- Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle.
|
||||
- Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle (on 1.4.3 too: `--exefs build\extracted\v143\exefs`).
|
||||
- Game version 1.4.3: the same commands with `v143` (`--patches recomp\patches-v143.txt`, `build_game.py --version v143`, `make_romfs_patches.py --version v143`). Its builds default to `build\profile-v143`; `run_game.py` runs it on copies of the 1.2.1 save backups.
|
||||
- Shader census (`.venv-py314`): `tools\shader_census.py --version v121 [--translate]` extracts every Maxwell shader to `build/v121/shaders/` and runs it through `build\xlate\totk_xlate.exe` and `spirv-val`; failure groups go to `failures.txt`. `tools\xlate_smoke.py` checks the translator alone, without game files.
|
||||
- Shader preparation (`.venv-py314`, once per build dir and translator): `tools\prepare_shaders.py build\v121` translates the variants of the MacNativeGames shared shader list (read from `refs/` at run time) into the build's shader cache and builds their stage libraries into the driver cache (`totk.exe --prepare-shaders`, about 5 min, 1.3 GB); play then translates and compiles only variants the list lacks.
|
||||
- Shader preparation (`.venv-py314`, once per build dir and translator): `tools\prepare_shaders.py build\v121` translates every game shader's default variant and the variants of the MacNativeGames shared shader list (read from `refs/` at run time, with the rows its history dropped: `tools/shader_list.py`) into the build's shader cache and builds their stage libraries into the driver cache (`totk.exe --prepare-shaders`; 1.4.3: 29,587 variants, 7 min on 24 threads, 1.7 GB); play then translates and compiles only variants neither covers.
|
||||
- Open work with its evidence: [docs/backlog.md](docs/backlog.md).
|
||||
- Installer (`installer/`, `tools/make_release.py`), which builds the game on a player's machine: [docs/installer.md](docs/installer.md).
|
||||
- Running, measuring or frame-dumping the game, save backups, task worktrees and their builds: [docs/agent-playbook.md](docs/agent-playbook.md).
|
||||
|
||||
@@ -24,7 +24,7 @@ MacNativeGames cannot run on x86-64 as is: its CPU path executes ARM64 code nati
|
||||
- ExeFS and RomFS extraction: MacNativeGames' Python extractor (Python 3.14 for `compression.zstd`, `.venv-py314`).
|
||||
- Update merging: hactool at `1d64a83`, with MacNativeGames' BKTR patches (`toolchain/hactool`).
|
||||
- Verification: MacNativeGames' `expected-v*.json` hashes of ExeFS, RomFS and fonts.
|
||||
- Primary version: 1.2.1, the update most players have. MacNativeGames supports it with the same build path as the others. 1.0.0, 1.1.2 and 1.2.1 share `sdk`, `subsdk0` and `rtld` byte for byte, so the lifted SDK modules and the host's SDK boundary serve all three; only `main` differs. 1.4.3 replaced all three modules (new SDK 20.5.22, RELR relocations, audio renderer REV15) and is a separate target; work on it has started on a branch and is not merged.
|
||||
- Primary version: 1.2.1, the update most players have. MacNativeGames supports it with the same build path as the others. 1.0.0, 1.1.2 and 1.2.1 share `sdk`, `subsdk0` and `rtld` byte for byte, so the lifted SDK modules and the host's SDK boundary serve all three; only `main` differs. 1.4.3 replaced all three modules (new SDK 20.5.22, RELR relocations, audio renderer REV15) and is a second target on the same build path: `--version v143`, `recomp/patches-v143.txt` (state in backlog item 39).
|
||||
- Windows driver: `tools/extract.py` runs the vendored extractors (`tools/vendor/macnativegames/`) and identifies the update by the SHA-256 of its `main`. hactool has no Darwin `funopen` here, so the base Program NCA is copied out of the NSP before the BKTR merge (about 17 GB of temporary space).
|
||||
|
||||
### 2. CPU: lift ARM64 to C
|
||||
|
||||
+18
-16
@@ -173,22 +173,24 @@ Each platform item gets a written design first, agreed with the maintainer.
|
||||
every render target ourselves); MacNativeGames has neither. The software keyboard could move onto
|
||||
SavageWindRecomp's `osk.mm`.
|
||||
|
||||
39. **Game version 1.4.3** (branch `task/v143`, paused). The 1.4.3 files extract and verify, and the generator runs on
|
||||
them unchanged. Estimate: one day to the title screen, two to three to play at 60 fps. Steps:
|
||||
1. `stg` no-op lift, DT_RELR in `recomp/nso.py` and `runtime/loader.c`, symbol count from DT_GNU_HASH for `sdk`
|
||||
and `subsdk0`.
|
||||
2. Jump tables: 26 disagreements and 83 unresolved sites down to 0 unexplained, with 1.2.1 still at 0.
|
||||
3. Tools take a version instead of `v121`; re-measure the `--probe-loader` table (`loader.c` `expect[]`) and
|
||||
`boot.c` `expected_inits`; new boot anchors (`boot.c` `CHECKPOINT_RET`, `NVN_RET`).
|
||||
4. RELRO (`nn::ro::ProtectRelro`) and the new SDK imports; run to the title screen.
|
||||
5. `recomp/patches-v143.txt` from MacNativeGames `game_patches_v143.c` (fps60, bounce, cloth; the real-time step
|
||||
rewritten for the `sys+0x1da` coefficient); the player position of `frame_rate.c` `player_pos`;
|
||||
`make_romfs_patches.py --version` (`Common.Product.140`).
|
||||
6. Audio REV15 is mostly in place (`audren.c`, `audio_mixer.c`): fix the REV11 self-test, check the override
|
||||
signatures (biquad, font decode), and listen for render differences in play.
|
||||
40. **Community ExeFS patches** (`.ips`, `.pchtxt` for 1.2.1): apply them through the generator's patch table
|
||||
(`recomp/patches.py`). RomFS mods already load (`romfs.c`, `TOTK_ROMFS_MODS`). Code-module mods (exlaunch
|
||||
`subsdk9`) need porting one by one.
|
||||
39. **Game version 1.4.3**. Builds, boots and plays: the user played it from their own saves on 2026-10-26 without
|
||||
problems; the field save runs at 60 fps with game speed 1.000, REV15 audio without underruns, 0 unimplemented NVN
|
||||
calls, 0 validation errors; every probe passes on 1.4.3 and on 1.2.1; the jump-table check reports 0 disagreements
|
||||
on both; the installer takes either update. Saves carry over both ways. `prepare_shaders.py` seeds every shader's
|
||||
default variant, as MacNativeGames' seed does, and reads the list with the V rows its history dropped
|
||||
(`tools/shader_list.py`; the current list alone has 134 rows of 1.4.3 shaders): 29,587 variants, among them all
|
||||
1,095 that the field and intro runs met in play. Open:
|
||||
1. MacNativeGames types vertex inputs and colour outputs by the bound format, so one variant serves any format;
|
||||
ours still differ by those types (347 of 19,468 variants met on 1.2.1). No frame-time gain, only fewer
|
||||
translations on first sight.
|
||||
2. The installer end-to-end test (`tests/installer_e2e.py`) runs only on 1.2.1.
|
||||
3. MacNativeGames' 1.4.3 output, render and fixed resolution patches (`game_patches_v143.c`) are not ported; 1.2.1
|
||||
has none of them either.
|
||||
4. The opening cutscene (Zelda under Hyrule Castle) shows hard light bands on 1.4.3 and on 1.2.1 alike; whether
|
||||
that is the game's look or ours is not checked.
|
||||
40. **Community ExeFS patches** (`.ips`, `.pchtxt`, each for one game version): apply them through the generator's
|
||||
patch table (`recomp/patches.py`). RomFS mods already load (`romfs.c`, `TOTK_ROMFS_MODS`). Code-module mods
|
||||
(exlaunch `subsdk9`) need porting one by one.
|
||||
41. **Frame rates above 60 and VRR.**
|
||||
42. **Map tour**: teleport Link across the map by writing position, time of day and weather in guest memory, to fill
|
||||
the caches on a first run and to serve as an automatic test (crashes, validation, fps per region).
|
||||
+7
-3
@@ -9,7 +9,7 @@ holds only the game; the installer and its tools stay outside it.
|
||||
1. Downloads `TotK-Setup-<version>-windows.zip` from GitHub Releases (none is published yet), unpacks it anywhere and
|
||||
starts `TotK-Setup.exe`.
|
||||
2. Picks:
|
||||
- the base game (XCI or NSP) and the 1.2.1 update (NSP);
|
||||
- the base game (XCI or NSP) and the 1.2.1 or 1.4.3 update (NSP);
|
||||
- their `prod.keys`;
|
||||
- the game language and the install folder.
|
||||
3. Waits for the setup, one time. The window shows each step and its progress.
|
||||
@@ -33,14 +33,18 @@ The player runs a newer installer on the same folder. It reads `totk-install.jso
|
||||
`profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the shaders. A newer
|
||||
release that changes nothing game-derived skips the rebuild.
|
||||
|
||||
Given another update file than the recorded one (1.2.1, then 1.4.3), the installer extracts and builds again from it and
|
||||
keeps `profile/`. Saves carry over both ways: 1.4.3 converts a 1.2.1 save on load, and 1.2.1 loads the saves 1.4.3 wrote.
|
||||
|
||||
## Setup steps
|
||||
|
||||
1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 (by hash, as
|
||||
1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 or 1.4.3 (by hash, as
|
||||
`tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a
|
||||
first install, 20 GiB after it), the CPU has
|
||||
AVX2 (`-march=x86-64-v3`), a Vulkan 1.3 GPU exists. Each failure is one clear message.
|
||||
2. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
|
||||
3. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt`) into a temp folder next to the installer.
|
||||
3. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt` or `patches-v143.txt`) into a temp folder next
|
||||
to the installer.
|
||||
4. **Compile** the generated C, `modules.c` and `romfs.c` (they bake per-install layout and RomFS hashes) and link
|
||||
them with the prebuilt runtime library into `totk.exe` (`build_game.py --runtime-lib`; about 3 min on 24 cores).
|
||||
5. **Prepare** the RomFS patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`,
|
||||
|
||||
+1
-1
@@ -41,7 +41,7 @@ def cancel(signum, frame):
|
||||
def main(argv=None):
|
||||
ap = Parser(prog='python -m installer', description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--base', type=Path, help='base game, XCI or NSP')
|
||||
ap.add_argument('--update', type=Path, help='1.2.1 update NSP')
|
||||
ap.add_argument('--update', type=Path, help='1.2.1 or 1.4.3 update NSP')
|
||||
ap.add_argument('--keys', type=Path, help='prod.keys')
|
||||
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
|
||||
ap.add_argument('--dest', type=Path, required=True, help='install folder')
|
||||
|
||||
+20
-10
@@ -4,7 +4,8 @@ Release (docs/installer.md "Release layout"), root = the unpacked folder:
|
||||
|
||||
python/python.exe runs every tool (numpy, lz4, cryptography, compression.zstd); its ._pth puts .., ../tools
|
||||
and ../recomp on sys.path, so `python -m installer` imports from the release folder
|
||||
recomp/, tools/ the lifter and the scripts; recomp/patches-v121.txt, tools/data/expected-v1.2.1.json
|
||||
recomp/, tools/ the lifter and the scripts; recomp/patches-<v>.txt, tools/data/expected-v<version>.json for
|
||||
each supported update (SUPPORTED)
|
||||
llvm/ llvm-mingw (bin/clang.exe)
|
||||
runtime/ libtotkrt.a, the headers, install-side sources (romfs.c) and totk.ico; runtime/deps/ holds
|
||||
include/ + lib/ of Vulkan, SDL3, Opus, libvpx and VMA (build_game.py --deps)
|
||||
@@ -28,7 +29,12 @@ from pathlib import Path
|
||||
|
||||
EXE = '.exe' if os.name == 'nt' else ''
|
||||
UCRT = Path(r'C:\msys64\ucrt64\bin')
|
||||
SHADER_LIST_REF = 'origin/HEAD:games/totk/assets/shader_list.tsv' # dev: tools/prepare_shaders.py LIST_REF
|
||||
SUPPORTED = ('1.2.1', '1.4.3') # updates the installer builds; the check identifies the player's by hash
|
||||
|
||||
|
||||
def version_tag(version):
|
||||
"""'1.4.3' -> 'v143': the version name of build_game.py, make_romfs_patches.py and the patch tables."""
|
||||
return 'v' + version.replace('.', '')
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -39,8 +45,6 @@ class Layout:
|
||||
python: Path # runs the tools
|
||||
tools: Path
|
||||
recomp: Path
|
||||
patches: Path # code patch table for recomp/gen.py
|
||||
expected: Path # expected hashes of the supported update
|
||||
llvm: Path # llvm-mingw tree
|
||||
deps: Path # include/ and lib/ for the link
|
||||
runtime: Path # runtime headers and install-side sources
|
||||
@@ -57,12 +61,20 @@ class Layout:
|
||||
def clang(self):
|
||||
return self.llvm / 'bin' / ('clang' + EXE)
|
||||
|
||||
def patches(self, version):
|
||||
"""The code patch table for recomp/gen.py."""
|
||||
return self.recomp / f'patches-{version_tag(version)}.txt'
|
||||
|
||||
def expected(self, version):
|
||||
"""The expected hashes of the update."""
|
||||
return self.tools / 'data' / f'expected-v{version}.json'
|
||||
|
||||
def missing(self):
|
||||
"""Paths the steps need that do not exist."""
|
||||
need = [self.python, self.tools / 'extract.py', self.tools / 'build_game.py', self.tools / 'prepare_shaders.py',
|
||||
self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.patches, self.expected, self.clang,
|
||||
self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.clang,
|
||||
self.deps / 'include', self.deps / 'lib', self.runtime, self.runtime_lib, self.ninja, self.hactool,
|
||||
self.xlate, *self.dlls]
|
||||
self.xlate, *self.dlls, *(f(v) for v in SUPPORTED for f in (self.patches, self.expected))]
|
||||
if self.shader_list:
|
||||
need.append(self.shader_list)
|
||||
return [str(p) for p in need if not p.exists()]
|
||||
@@ -78,8 +90,7 @@ def release_layout(root):
|
||||
root=root, dev=False,
|
||||
release=version.read_text(encoding='utf-8').strip() if version.is_file() else 'unknown',
|
||||
python=root / 'python' / ('python' + EXE),
|
||||
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
|
||||
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
|
||||
tools=root / 'tools', recomp=root / 'recomp',
|
||||
llvm=root / 'llvm', deps=root / 'runtime' / 'deps', runtime=root / 'runtime',
|
||||
runtime_lib=root / 'runtime' / 'libtotkrt.a',
|
||||
ninja=root / 'bin' / ('ninja' + EXE), hactool=root / 'bin' / ('hactool' + EXE),
|
||||
@@ -97,8 +108,7 @@ def dev_layout(root):
|
||||
return Layout(
|
||||
root=root, dev=True, release=f'dev-{commit or "unknown"}',
|
||||
python=embed if embed.is_file() else Path(sys.executable),
|
||||
tools=root / 'tools', recomp=root / 'recomp', patches=root / 'recomp' / 'patches-v121.txt',
|
||||
expected=root / 'tools' / 'data' / 'expected-v1.2.1.json',
|
||||
tools=root / 'tools', recomp=root / 'recomp',
|
||||
llvm=trees[-1] if trees else root / 'toolchain' / 'llvm-mingw-missing', deps=deps,
|
||||
runtime=root / 'runtime', runtime_lib=root / 'build' / 'install-rtlib' / 'libtotkrt.a',
|
||||
ninja=UCRT / ('ninja' + EXE), hactool=root / 'toolchain' / 'hactool' / ('hactool' + EXE),
|
||||
|
||||
+46
-24
@@ -22,18 +22,18 @@ import time
|
||||
from pathlib import Path
|
||||
|
||||
from . import ini
|
||||
from .layout import EXE
|
||||
from .layout import EXE, SUPPORTED, version_tag
|
||||
from .runner import InstallError, Step
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / 'tools'))
|
||||
import extract # noqa: E402
|
||||
import shader_list # noqa: E402
|
||||
from extract_exefs import (common_tickets, container_entries, load_keys, nca_ctr_key, nca_inventory, # noqa: E402
|
||||
program_nca)
|
||||
|
||||
GiB = 1 << 30
|
||||
DEST_EXTRA = 4 * GiB # beside the RomFS: shader and pipeline caches (about 2 GB), exefs, totk.exe, seed files
|
||||
TEMP_NEED = 3 * GiB # generated C (about 0.9 GB) and objects
|
||||
UPDATE_VERSION = '1.2.1'
|
||||
HOSTCHECK = Path(__file__).resolve().parent / 'hostcheck.c'
|
||||
STAMP = '.install-key'
|
||||
|
||||
@@ -138,19 +138,23 @@ def raise_if_damaged(error, name, path):
|
||||
raise InstallError('check.input_damaged', input=name, path=str(path), detail=str(error)) from None
|
||||
|
||||
|
||||
def expected_update():
|
||||
return extract.load_expected()[UPDATE_VERSION]
|
||||
def game_version(ctx):
|
||||
"""The update this install is built from: what the check identified, else the recorded one (an install made before
|
||||
1.4.3 support records none: 1.2.1)."""
|
||||
return ctx.state.get('version', SUPPORTED[0])
|
||||
|
||||
|
||||
def expected_update(ctx):
|
||||
return extract.load_expected()[game_version(ctx)]
|
||||
|
||||
|
||||
def shader_list_text(layout):
|
||||
if layout.shader_list:
|
||||
return layout.shader_list.read_text(encoding='utf-8')
|
||||
refs = layout.root / 'refs' / 'MacNativeGames'
|
||||
r = subprocess.run(['git', '-C', str(refs), 'show', 'origin/HEAD:games/totk/assets/shader_list.tsv'],
|
||||
capture_output=True, encoding='utf-8')
|
||||
if r.returncode:
|
||||
try:
|
||||
return shader_list.history(layout.root / 'refs' / 'MacNativeGames')
|
||||
except subprocess.CalledProcessError:
|
||||
raise InstallError('prepare.shader_list_missing')
|
||||
return r.stdout
|
||||
|
||||
|
||||
class Watch(threading.Thread):
|
||||
@@ -316,7 +320,8 @@ class Check(Step):
|
||||
def space(self, ctx, ncas):
|
||||
dest_need = temp_need = 0
|
||||
if ctx.will_run('extract'):
|
||||
dest_need += expected_update()['romfs']['size'] + ncas
|
||||
# the update is identified after this estimate: the larger RomFS of the supported ones
|
||||
dest_need += max(extract.load_expected()[v]['romfs']['size'] for v in SUPPORTED) + ncas
|
||||
romfs = ctx.dest / 'romfs' / 'romfs.bin'
|
||||
if romfs.is_file(): # a resumed extraction replaces it
|
||||
dest_need -= romfs.stat().st_size
|
||||
@@ -361,8 +366,14 @@ class Check(Step):
|
||||
found = extract.identify_update(work / 'update' / 'exefs', versions)['display_version']
|
||||
except ValueError:
|
||||
found = 'unknown'
|
||||
if found != UPDATE_VERSION:
|
||||
raise InstallError('check.update_wrong_version', found=found, wanted=UPDATE_VERSION)
|
||||
if found not in SUPPORTED:
|
||||
raise InstallError('check.update_wrong_version', found=found, wanted=' / '.join(SUPPORTED))
|
||||
if found != game_version(ctx): # another update than the recorded one: every step builds again from it
|
||||
ctx.reporter.log(f'check: update {found} (recorded {game_version(ctx)})')
|
||||
ctx.state['version'] = found
|
||||
ctx.keys = {}
|
||||
ctx.plan(ctx.forced | {'extract', 'lift', 'compile', 'prepare'})
|
||||
ctx.state['version'] = found
|
||||
|
||||
|
||||
# ---- 2. extract
|
||||
@@ -372,10 +383,17 @@ class Extract(Step):
|
||||
|
||||
def key(self, ctx):
|
||||
# The output is whatever matches the expected hashes, whichever files and tools produced it.
|
||||
return fingerprint('extract-1', ctx.layout.expected)
|
||||
return fingerprint('extract-1', ctx.layout.expected(game_version(ctx)))
|
||||
|
||||
def current(self, ctx, record):
|
||||
expected = expected_update()
|
||||
# Another update file than the recorded one (1.2.1 -> 1.4.3) extracts again; the check then identifies it.
|
||||
update, known = getattr(ctx.options, 'update', None), ctx.state['inputs'].get('update')
|
||||
if update and known and Path(update).is_file():
|
||||
st = Path(update).stat()
|
||||
if (st.st_size, st.st_mtime_ns) != (known.get('size'), known.get('mtime_ns')) \
|
||||
and file_sha256(Path(update)) != known.get('sha256'):
|
||||
return False
|
||||
expected = expected_update(ctx)
|
||||
romfs = ctx.dest / 'romfs'
|
||||
if extract.exefs_problems(ctx.dest / 'exefs', expected) or not extract.romfs_verified(romfs, expected):
|
||||
return False
|
||||
@@ -386,11 +404,11 @@ class Extract(Step):
|
||||
image = ctx.dest / 'romfs' / 'romfs.bin'
|
||||
rep = ctx.reporter
|
||||
digest = file_sha256(image, lambda d, t: rep.progress('check', 'verify_romfs', d, t, 'bytes'))
|
||||
return digest == expected_update()['romfs']['sha256']
|
||||
return digest == expected_update(ctx)['romfs']['sha256']
|
||||
|
||||
def run(self, ctx):
|
||||
o, dest, rep = ctx.options, ctx.dest, ctx.reporter
|
||||
expected = expected_update()
|
||||
expected = expected_update(ctx)
|
||||
base, update, keys = Path(o.base), Path(o.update), Path(o.keys)
|
||||
inputs = {}
|
||||
for name, path in (('base', base), ('update', update)):
|
||||
@@ -422,9 +440,9 @@ class Extract(Step):
|
||||
shutil.move(staged, dest / 'exefs')
|
||||
problems = extract.exefs_problems(dest / 'exefs', expected)
|
||||
if problems:
|
||||
raise InstallError('extract.exefs_mismatch', modules=', '.join(problems))
|
||||
raise InstallError('extract.exefs_mismatch', version=game_version(ctx), modules=', '.join(problems))
|
||||
if extract.sha256(dest / 'exefs' / 'main.npdm') != expected['exefs_sha256']['main.npdm']:
|
||||
rep.warning('warn.npdm_differs')
|
||||
rep.warning('warn.npdm_differs', version=game_version(ctx))
|
||||
|
||||
romfs = dest / 'romfs'
|
||||
if self.id in ctx.forced:
|
||||
@@ -433,7 +451,8 @@ class Extract(Step):
|
||||
self.merge(ctx, base, update, keys, romfs, expected)
|
||||
report = json.loads((romfs / 'index.json').read_text())
|
||||
if report['romfs_size'] != expected['romfs']['size'] or report['romfs_sha256'] != expected['romfs']['sha256']:
|
||||
raise InstallError('extract.romfs_mismatch', size=report['romfs_size'], sha256=report['romfs_sha256'][:16])
|
||||
raise InstallError('extract.romfs_mismatch', version=game_version(ctx), size=report['romfs_size'],
|
||||
sha256=report['romfs_sha256'][:16])
|
||||
(dest / 'profile').mkdir(exist_ok=True)
|
||||
st = (romfs / 'romfs.bin').stat()
|
||||
return {'romfs_size': st.st_size, 'romfs_mtime_ns': st.st_mtime_ns}
|
||||
@@ -476,7 +495,7 @@ class Lift(Step):
|
||||
transient = True
|
||||
|
||||
def key(self, ctx):
|
||||
return fingerprint('lift-1', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches)
|
||||
return fingerprint('lift-1', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches(game_version(ctx)))
|
||||
|
||||
def current(self, ctx, record):
|
||||
gen = ctx.temp / 'gen'
|
||||
@@ -494,7 +513,7 @@ class Lift(Step):
|
||||
rep.progress('lift', 'emit', int(m.group(1)), int(m.group(2)), 'chunks')
|
||||
|
||||
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py',
|
||||
['--exefs', ctx.dest / 'exefs', '--out', gen, '--patches', layout.patches], parse)
|
||||
['--exefs', ctx.dest / 'exefs', '--out', gen, '--patches', layout.patches(game_version(ctx))], parse)
|
||||
if code:
|
||||
raise InstallError('lift.failed', code=code, log=str(log))
|
||||
(gen / STAMP).write_text(ctx.keys['lift'] + '\n')
|
||||
@@ -541,7 +560,8 @@ class Compile(Step):
|
||||
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
|
||||
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--deps', layout.deps,
|
||||
'--runtime', layout.runtime, '--runtime-lib', layout.runtime_lib, '--ninja', layout.ninja,
|
||||
'--expected', layout.expected, '--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
||||
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
|
||||
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
||||
'--profile', dest / 'profile'], parse)
|
||||
exe = build / ('totk' + EXE)
|
||||
if code or not exe.is_file():
|
||||
@@ -593,7 +613,8 @@ class Prepare(Step):
|
||||
rep.progress('prepare', 'romfs_patches', force=True)
|
||||
remove(dest / 'romfs-patches')
|
||||
code, log = run_tool(ctx, 'prepare', layout.tools / 'make_romfs_patches.py',
|
||||
['--romfs', dest / 'romfs', '--out', dest / 'romfs-patches'])
|
||||
['--version', version_tag(game_version(ctx)), '--romfs', dest / 'romfs', '--out',
|
||||
dest / 'romfs-patches'])
|
||||
if code:
|
||||
raise InstallError('prepare.romfs_patches_failed', code=code, log=str(log))
|
||||
|
||||
@@ -613,7 +634,8 @@ class Prepare(Step):
|
||||
|
||||
rep.progress('prepare', 'seed', force=True)
|
||||
code, log = run_tool(ctx, 'prepare', layout.tools / 'prepare_shaders.py',
|
||||
[dest, '--romfs', dest / 'romfs', '--exefs', dest / 'exefs', '--list', shader_list], parse)
|
||||
[dest, '--version', version_tag(game_version(ctx)), '--romfs', dest / 'romfs', '--exefs',
|
||||
dest / 'exefs', '--list', shader_list], parse)
|
||||
if code or not (dest / 'shader-prepared.bin').is_file():
|
||||
raise InstallError('prepare.shaders_failed', code=code, log=str(log))
|
||||
return {'xlate_sha256': file_sha256(xlate)}
|
||||
|
||||
@@ -46,7 +46,7 @@
|
||||
"check.vulkan_missing": "No Vulkan driver is installed. Install the latest driver for your graphics card.",
|
||||
"check.vulkan_no_device": "Vulkan finds no graphics card. Install the latest driver for your graphics card.",
|
||||
"check.vulkan_too_old": "The game needs a graphics card with Vulkan 1.3; found: {devices}. Update the graphics driver.",
|
||||
"check.inputs_needed": "The game files are needed: choose the base game, the 1.2.1 update and prod.keys.",
|
||||
"check.inputs_needed": "The game files are needed: choose the base game, the 1.2.1 or 1.4.3 update and prod.keys.",
|
||||
"check.input_missing": "File not found ({input}): {path}",
|
||||
"check.keys_unreadable": "The keys file cannot be read. Choose a prod.keys file.",
|
||||
"check.keys_no_header_key": "The keys file has no header_key. Dump prod.keys from your console again.",
|
||||
@@ -58,17 +58,17 @@
|
||||
"check.ticket_missing": "The {input} file lacks the ticket its content needs.",
|
||||
"check.base_not_totk": "The base game file is not The Legend of Zelda: Tears of the Kingdom ({detail}).",
|
||||
"check.update_not_nsp": "The update must be an NSP file.",
|
||||
"check.update_is_base": "The update file holds a base game. Choose the 1.2.1 update NSP.",
|
||||
"check.update_is_base": "The update file holds a base game. Choose the 1.2.1 or 1.4.3 update NSP.",
|
||||
"check.update_not_totk": "The update file is not a Tears of the Kingdom update ({detail}).",
|
||||
"check.exefs_failed": "The game code of the {input} file failed its checks; see {log}.",
|
||||
"check.input_damaged": "The {input} file is damaged or incomplete: {path} ({detail}). Copy that file again from its source, or dump it again, then run the setup again.",
|
||||
"check.base_wrong_version": "The base game is not version 1.0.0 ({modules} differ).",
|
||||
"check.update_wrong_version": "The update is version {found}; this release needs {wanted}.",
|
||||
"check.update_wrong_version": "The update is version {found}; this release takes {wanted}.",
|
||||
"check.disk_space": "Not enough free space for {path}: {need_gb} GB needed, {free_gb} GB free.",
|
||||
|
||||
"extract.exefs_mismatch": "The extracted game code does not match version 1.2.1 ({modules}).",
|
||||
"extract.exefs_mismatch": "The extracted game code does not match version {version} ({modules}).",
|
||||
"extract.romfs_merge_failed": "Applying the update to the game data failed: {detail}. Details: {log}",
|
||||
"extract.romfs_mismatch": "The extracted game data does not match version 1.2.1 (size {size}, checksum {sha256}...).",
|
||||
"extract.romfs_mismatch": "The extracted game data does not match version {version} (size {size}, checksum {sha256}...).",
|
||||
"lift.failed": "Translating the game code failed (exit code {code}). Details: {log}",
|
||||
"compile.failed": "Compiling totk.exe failed (exit code {code}). Details: {log}",
|
||||
"compile.exe_in_use": "{path} is in use. Close the game and run the installer again.",
|
||||
@@ -78,20 +78,20 @@
|
||||
"cleanup.ini_failed": "totk.ini cannot be written: {detail}",
|
||||
|
||||
"warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.",
|
||||
"warn.npdm_differs": "main.npdm differs from the reference 1.2.1 dump; only the code modules are used.",
|
||||
"warn.npdm_differs": "main.npdm differs from the reference {version} dump; only the code modules are used.",
|
||||
|
||||
"ui.title": "TotK Setup",
|
||||
"ui.intro": "Builds a playable Tears of the Kingdom folder from your own game files. This takes a while and runs once.",
|
||||
"ui.notice": "The folder this setup builds contains the game. It is for your own use only; do not share or upload it.",
|
||||
"ui.keys_private": "The keys file is read in place. Its contents are never shown, copied or stored.",
|
||||
"ui.base": "Base game (NSP or XCI)",
|
||||
"ui.update": "Update 1.2.1 (NSP)",
|
||||
"ui.update": "Update 1.2.1 or 1.4.3 (NSP)",
|
||||
"ui.keys": "prod.keys",
|
||||
"ui.game_language": "Game language",
|
||||
"ui.dest": "Install folder",
|
||||
"ui.browse": "Browse...",
|
||||
"ui.pick_game": "Choose the base game file",
|
||||
"ui.pick_update": "Choose the 1.2.1 update file",
|
||||
"ui.pick_update": "Choose the 1.2.1 or 1.4.3 update file",
|
||||
"ui.pick_keys": "Choose prod.keys",
|
||||
"ui.pick_dest": "Choose the install folder",
|
||||
"ui.type_game": "Game files (NSP, XCI)",
|
||||
@@ -109,7 +109,7 @@
|
||||
"ui.update_mode": "This folder holds an earlier installation. The setup continues or updates it; the game files are asked for only if they are needed again.",
|
||||
"ui.need_dest": "Choose the install folder.",
|
||||
"ui.need_base": "Choose the base game file.",
|
||||
"ui.need_update": "Choose the 1.2.1 update file.",
|
||||
"ui.need_update": "Choose the 1.2.1 or 1.4.3 update file.",
|
||||
"ui.need_keys": "Choose the prod.keys file.",
|
||||
"ui.need_lang": "Choose the game language.",
|
||||
"ui.missing_base": "The base game file does not exist.",
|
||||
|
||||
@@ -46,7 +46,7 @@
|
||||
"check.vulkan_missing": "Драйвер Vulkan не установлен. Установите свежий драйвер видеокарты.",
|
||||
"check.vulkan_no_device": "Vulkan не находит видеокарту. Установите свежий драйвер видеокарты.",
|
||||
"check.vulkan_too_old": "Игре нужна видеокарта с Vulkan 1.3; найдено: {devices}. Обновите драйвер видеокарты.",
|
||||
"check.inputs_needed": "Нужны игровые файлы: выберите базовую игру, обновление 1.2.1 и prod.keys.",
|
||||
"check.inputs_needed": "Нужны игровые файлы: выберите базовую игру, обновление 1.2.1 или 1.4.3 и prod.keys.",
|
||||
"check.input_missing": "Файл не найден ({input}): {path}",
|
||||
"check.keys_unreadable": "Не удаётся прочитать файл ключей. Выберите файл prod.keys.",
|
||||
"check.keys_no_header_key": "В файле ключей нет header_key. Снова получите prod.keys со своей консоли.",
|
||||
@@ -58,17 +58,17 @@
|
||||
"check.ticket_missing": "В файле «{input}» нет тикета, который нужен его содержимому.",
|
||||
"check.base_not_totk": "Файл базовой игры — не The Legend of Zelda: Tears of the Kingdom ({detail}).",
|
||||
"check.update_not_nsp": "Обновление должно быть файлом NSP.",
|
||||
"check.update_is_base": "Файл обновления содержит базовую игру. Выберите NSP с обновлением 1.2.1.",
|
||||
"check.update_is_base": "Файл обновления содержит базовую игру. Выберите NSP с обновлением 1.2.1 или 1.4.3.",
|
||||
"check.update_not_totk": "Файл обновления — не обновление Tears of the Kingdom ({detail}).",
|
||||
"check.exefs_failed": "Код игры из файла «{input}» не прошёл проверку; подробности в {log}.",
|
||||
"check.input_damaged": "Файл «{input}» повреждён или скопирован не полностью: {path} ({detail}). Скопируйте этот файл заново из источника или сделайте дамп ещё раз, затем снова запустите установку.",
|
||||
"check.base_wrong_version": "Базовая игра не версии 1.0.0 (отличаются модули: {modules}).",
|
||||
"check.update_wrong_version": "Версия обновления {found}; этому выпуску нужна {wanted}.",
|
||||
"check.update_wrong_version": "Версия обновления {found}; этот выпуск принимает {wanted}.",
|
||||
"check.disk_space": "Не хватает свободного места для {path}: нужно {need_gb} ГБ, свободно {free_gb} ГБ.",
|
||||
|
||||
"extract.exefs_mismatch": "Распакованный код игры не соответствует версии 1.2.1 ({modules}).",
|
||||
"extract.exefs_mismatch": "Распакованный код игры не соответствует версии {version} ({modules}).",
|
||||
"extract.romfs_merge_failed": "Не удалось применить обновление к данным игры: {detail}. Подробности: {log}",
|
||||
"extract.romfs_mismatch": "Распакованные данные игры не соответствуют версии 1.2.1 (размер {size}, контрольная сумма {sha256}...).",
|
||||
"extract.romfs_mismatch": "Распакованные данные игры не соответствуют версии {version} (размер {size}, контрольная сумма {sha256}...).",
|
||||
"lift.failed": "Не удалось перевести код игры (код выхода {code}). Подробности: {log}",
|
||||
"compile.failed": "Не удалось собрать totk.exe (код выхода {code}). Подробности: {log}",
|
||||
"compile.exe_in_use": "Файл {path} занят. Закройте игру и запустите установку снова.",
|
||||
@@ -78,20 +78,20 @@
|
||||
"cleanup.ini_failed": "Не удаётся записать totk.ini: {detail}",
|
||||
|
||||
"warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.",
|
||||
"warn.npdm_differs": "main.npdm отличается от эталонного дампа 1.2.1; используются только модули с кодом.",
|
||||
"warn.npdm_differs": "main.npdm отличается от эталонного дампа {version}; используются только модули с кодом.",
|
||||
|
||||
"ui.title": "Установка TotK",
|
||||
"ui.intro": "Собирает готовую к запуску папку с Tears of the Kingdom из ваших игровых файлов. Занимает много времени и выполняется один раз.",
|
||||
"ui.notice": "Папка, которую создаёт установка, содержит игру. Она предназначена только для личного использования: не передавайте и не выкладывайте её.",
|
||||
"ui.keys_private": "Файл ключей читается на месте. Его содержимое не показывается, не копируется и не сохраняется.",
|
||||
"ui.base": "Базовая игра (NSP или XCI)",
|
||||
"ui.update": "Обновление 1.2.1 (NSP)",
|
||||
"ui.update": "Обновление 1.2.1 или 1.4.3 (NSP)",
|
||||
"ui.keys": "prod.keys",
|
||||
"ui.game_language": "Язык игры",
|
||||
"ui.dest": "Папка установки",
|
||||
"ui.browse": "Обзор...",
|
||||
"ui.pick_game": "Выберите файл базовой игры",
|
||||
"ui.pick_update": "Выберите файл обновления 1.2.1",
|
||||
"ui.pick_update": "Выберите файл обновления 1.2.1 или 1.4.3",
|
||||
"ui.pick_keys": "Выберите prod.keys",
|
||||
"ui.pick_dest": "Выберите папку установки",
|
||||
"ui.type_game": "Файлы игры (NSP, XCI)",
|
||||
@@ -109,7 +109,7 @@
|
||||
"ui.update_mode": "В этой папке уже есть установка. Программа продолжит или обновит её; файлы игры понадобятся, только если их снова придётся читать.",
|
||||
"ui.need_dest": "Выберите папку установки.",
|
||||
"ui.need_base": "Выберите файл базовой игры.",
|
||||
"ui.need_update": "Выберите файл обновления 1.2.1.",
|
||||
"ui.need_update": "Выберите файл обновления 1.2.1 или 1.4.3.",
|
||||
"ui.need_keys": "Выберите файл prod.keys.",
|
||||
"ui.need_lang": "Выберите язык игры.",
|
||||
"ui.missing_base": "Файл базовой игры не найден.",
|
||||
|
||||
@@ -383,8 +383,20 @@ class Lifter:
|
||||
return self.ldst_pair(w)
|
||||
if bits(w, 27, 3) == 0b111:
|
||||
return self.ldst_reg(w)
|
||||
if (w & 0xFFE00000) == 0xD9200000 and bits(w, 10, 2):
|
||||
return self.stg(w)
|
||||
raise Unsupported(f'ldst {w:08x}')
|
||||
|
||||
def stg(self, w):
|
||||
"""STG (post-index, signed offset, pre-index). Horizon runs without MTE, so storing an allocation tag
|
||||
changes no memory; only the base writeback remains (1.4.3 sdk+0x640050)."""
|
||||
mode, rn = bits(w, 10, 2), bits(w, 5, 5)
|
||||
if mode == 2:
|
||||
return Insn('op', '/* stg */')
|
||||
off = sext(bits(w, 12, 9), 9) << 4
|
||||
dst = 'c->sp' if rn == 31 else f'c->x[{rn}]'
|
||||
return Insn('op', f'{dst} = {XS(rn)} + {hx(off & 0xFFFFFFFFFFFFFFFF)}; /* stg */')
|
||||
|
||||
def ldst_excl(self, w, pc):
|
||||
size, o2, L, o1, rs, o0, rn, rt = (bits(w, 30, 2), bits(w, 23, 1), bits(w, 22, 1), bits(w, 21, 1),
|
||||
bits(w, 16, 5), bits(w, 15, 1), bits(w, 5, 5), bits(w, 0, 5))
|
||||
|
||||
+38
-4
@@ -8,6 +8,7 @@ import lz4.block
|
||||
|
||||
DT_NULL, DT_NEEDED, DT_PLTRELSZ, DT_HASH, DT_STRTAB, DT_SYMTAB, DT_RELA, DT_RELASZ = 0, 1, 2, 4, 5, 6, 7, 8
|
||||
DT_STRSZ, DT_JMPREL, DT_RELACOUNT = 10, 23, 0x6FFFFFF9
|
||||
DT_RELRSZ, DT_RELR, DT_GNU_HASH = 0x23, 0x24, 0x6FFFFEF5
|
||||
R_AARCH64_ABS64, R_AARCH64_GLOB_DAT, R_AARCH64_JUMP_SLOT, R_AARCH64_RELATIVE = 257, 1025, 1026, 1027
|
||||
|
||||
|
||||
@@ -61,12 +62,12 @@ class Nso:
|
||||
strtab = dyn.get(DT_STRTAB, 0)
|
||||
self.strtab = strtab
|
||||
self.needed = [self.cstr(strtab + n) for n in needed]
|
||||
# dynsym count from DT_HASH nchain
|
||||
# dynsym count from DT_HASH nchain, or from DT_GNU_HASH (1.4.3 sdk and subsdk0 have no DT_HASH)
|
||||
self.syms = []
|
||||
if DT_HASH in dyn and DT_SYMTAB in dyn:
|
||||
nchain = self.u32(dyn[DT_HASH] + 4)
|
||||
nsyms = self.symbol_count()
|
||||
if nsyms and DT_SYMTAB in dyn:
|
||||
st = dyn[DT_SYMTAB]
|
||||
for i in range(nchain):
|
||||
for i in range(nsyms):
|
||||
o = st + i * 24
|
||||
name, info, other, shndx = struct.unpack_from('<IBBH', self.image, o)
|
||||
value, size = struct.unpack_from('<QQ', self.image, o + 8)
|
||||
@@ -78,6 +79,39 @@ class Nso:
|
||||
for i in range(n):
|
||||
r_off, r_info, r_add = struct.unpack_from('<QQq', self.image, b + i * 24)
|
||||
self.relocs.append((r_off, r_info & 0xFFFFFFFF, r_info >> 32, r_add))
|
||||
# DT_RELR (1.4.3): packed RELATIVE relocations whose addend is the word in place. An even entry is an address
|
||||
# (relocate it, continue after it); an odd entry is a bitmap of the next 63 words.
|
||||
if DT_RELR in dyn:
|
||||
where = 0
|
||||
for i in range(dyn[DT_RELRSZ] // 8):
|
||||
e = self.u64(dyn[DT_RELR] + i * 8)
|
||||
if e & 1 == 0:
|
||||
self.relocs.append((e, R_AARCH64_RELATIVE, 0, self.u64(e)))
|
||||
where = e + 8
|
||||
continue
|
||||
for k in range(63):
|
||||
if e >> (k + 1) & 1:
|
||||
a = where + k * 8
|
||||
self.relocs.append((a, R_AARCH64_RELATIVE, 0, self.u64(a)))
|
||||
where += 63 * 8
|
||||
|
||||
def symbol_count(self):
|
||||
"""Number of .dynsym entries: DT_HASH nchain, else one past the last symbol a DT_GNU_HASH chain reaches."""
|
||||
dyn = self.dyn
|
||||
if DT_HASH in dyn:
|
||||
return self.u32(dyn[DT_HASH] + 4)
|
||||
if DT_GNU_HASH not in dyn:
|
||||
return 0
|
||||
h = dyn[DT_GNU_HASH]
|
||||
nbuckets, symoffset, bloom_size = self.u32(h), self.u32(h + 4), self.u32(h + 8)
|
||||
buckets = h + 16 + bloom_size * 8
|
||||
last = max([self.u32(buckets + 4 * i) for i in range(nbuckets)] + [0])
|
||||
if last < symoffset:
|
||||
return symoffset
|
||||
chains = buckets + 4 * nbuckets
|
||||
while not self.u32(chains + 4 * (last - symoffset)) & 1:
|
||||
last += 1
|
||||
return last + 1
|
||||
|
||||
def cstr(self, off):
|
||||
end = self.image.index(b'\0', off)
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Code patches for TotK 1.4.3 main (recomp/patches.py). Ported from MacNativeGames
|
||||
# games/totk/runtime/game_patches_v143.c (MIT, Copyright (c) 2026 MacNativeGames contributors) at upstream 73c4821,
|
||||
# each original word checked against the 1.4.3 main. MacNativeGames branches to veneers after the text; here a
|
||||
# veneer's body is the site's replacement. The patch page holds three floats the runtime writes at every present
|
||||
# (frame_rate.c): +0 the 30 Hz ratio (seconds * 30), +4 the frame step in seconds, +8 the step in 60 Hz ticks.
|
||||
|
||||
set fps60 TOTK_FPS60 1
|
||||
set realtime TOTK_REALTIME 1
|
||||
set cloth TOTK_CLOTH_REALTIME 1
|
||||
set bounce TOTK_HIGHFPS_FIX 1
|
||||
|
||||
# 60 fps: the adaptive present-interval controller (main+0x2db174, 1.2.1 0x8f26c8) gets the minimum interval 1.
|
||||
site fps60 main 2db204 6b08015f 7100055f # clamp: cmp w10, #1 (was cmp w10, w8 = minimum)
|
||||
site fps60 main 2db208 1a89b10a 1a9fa52a # clamp: csinc w10, w9, wzr, ge (desired = max(desired, 1))
|
||||
site fps60 main 2db22c b940468a d503201f # lower trigger: nop the stored-candidate load
|
||||
site fps60 main 2db360 6b0801bf 710005bf # raise path: cmp w13, #1
|
||||
site fps60 main 2db36c 1a88c1ac 1a9fc5ac # raise path: csinc w12, w13, wzr, gt (next lower candidate floors at 1)
|
||||
site fps60 main 2db340 b9400288 52800028 # lower path: mov w8, #1 (minimum interval 1)
|
||||
site fps60 main 2db34c 2a0a03f5 1a9fa555 # lower path: csinc w21, w10, wzr, ge (result never below 1)
|
||||
|
||||
# Real-time step. The frame-step producer feeds both FrameRateMgr call sites with s8 = nominal interval * a 0.5/1.0
|
||||
# coefficient (sys+0x1da bit 0, fcsel above the site); the nominal interval comes from the patch page, the coefficient
|
||||
# and FrameRateMgr's own arithmetic stay. x8 is free: the ldxr after the site loads it. The global ratio after the
|
||||
# controller call (1.2.1 0x8f2394) reads the ratio from the page instead of 60 / interval * 0.5.
|
||||
site realtime main 3e5658 1e210808 @page:x8 bd400900 1e210808 # fmul s8, s0, s1: s0 = [page+8] first
|
||||
site realtime main 2daa54 bd425260 @page:x9 # ldr s0, [x19, #0x250] -> adrp x9, page
|
||||
site realtime main 2daa58 52a84e08 bd400120 # mov w8, #60.0 -> ldr s0, [x9] (global ratio)
|
||||
site realtime main 2daa6c 5e21d800 d503201f # scvtf s0, s0 -> nop
|
||||
site realtime main 2daa70 1e201820 d503201f # fdiv s0, s1, s0 -> nop
|
||||
site realtime main 2daa74 1e2c1001 d503201f # fmov s1, #0.5 -> nop
|
||||
site realtime main 2daa78 1e210800 d503201f # fmul s0, s0, s1 -> nop
|
||||
|
||||
# Hair and cloth: the actor animation update builds a second clock {seconds = factor / 30, ratio = factor} per
|
||||
# rendered frame. 1.4.3 inlines it in two actor paths, where the factor was just stored to the ratio slot: the
|
||||
# replacement stores measured ratio * factor there and computes measured seconds * factor. Two other callers share the
|
||||
# out-of-line update (main+0x2a0e40), whose s2 (seconds) and s3 (ratio) are scaled by the measured ratio; its third
|
||||
# caller already passes measured time and stays. x8 and s4 are free at all four sites.
|
||||
site cloth main 246eac 1e200940 @page:x8 bd400100 1e200940 bd0367e0 bd400500 1e200940 # fmul s0, s10, s0
|
||||
site cloth main 345904 1e210801 @page:x8 bd400101 1e210801 bd0117e1 bd400501 1e210801 # fmul s1, s0, s1
|
||||
site cloth main 24944c 94015e7d @page:x8 bd400104 1e240842 1e240863 94015e7d # bl 0x2a0e40
|
||||
site cloth main 28bc98 9400546a @page:x8 bd400104 1e240842 1e240863 9400546a # bl 0x2a0e40
|
||||
|
||||
# Trampolines: the contact code scales the bounce vector by 0.1 when time object +0x28 (30 / (ratio * time scale)) is
|
||||
# over 100; compare +0x28 * +0x38 (30 / time scale) instead, the same value at 30 Hz and independent of the frame rate.
|
||||
# s1 is set again by the fmov two instructions on.
|
||||
site bounce main 562a64 bd402920 bd402920 bd403921 1e210800 # ldr s0, [x9, #0x28]; ldr s1, [x9, #0x38]; fmul s0, s0, s1
|
||||
+15
-9
@@ -4,8 +4,10 @@
|
||||
// The boot driver: the guest main thread runs the game's own nn::init::Start
|
||||
// with three host callbacks, the second of which runs DT_INIT of main, subsdk0 and sdk; Start then calls nnMain.
|
||||
// --boot checks two checkpoints on the way and ends at the first nvnBootstrapLoader call:
|
||||
// SetFocusHandlingMode called from main+0xe7a954 in nnMain, every module's constructor flag reads 2
|
||||
// nvnBootstrapLoader called from main+0xe7eb2c in the graphics bring-up; logs the run statistics, exits 0
|
||||
// SetFocusHandlingMode called from main+0xe7a954 (1.4.3: +0xbdb48c) in nnMain, every module's constructor flag
|
||||
// reads 2
|
||||
// nvnBootstrapLoader called from main+0xe7eb2c (1.4.3: +0xbe0360) in the graphics bring-up; logs the run
|
||||
// statistics, exits 0
|
||||
// With --frames N the run continues past that call; rt_frame_presented ends it at the Nth present.
|
||||
#include <windows.h>
|
||||
#include <stdio.h>
|
||||
@@ -18,13 +20,17 @@
|
||||
#include "vk.h"
|
||||
|
||||
#define SYM_INIT_START "_ZN2nn4init5StartEmmPFvvES2_S2_"
|
||||
#define CHECKPOINT_RET 0xe7a954ull // main: return address of nnMain's SetFocusHandlingMode call (section 0.4)
|
||||
#define NVN_RET 0xe7eb2cull // main: return address of the first nvnBootstrapLoader call
|
||||
// main: return addresses of nnMain's SetFocusHandlingMode call (section 0.4; 1.4.3: MNG scripts/verify_boot.py:9) and
|
||||
// of the first nvnBootstrapLoader call, per game version.
|
||||
#define CHECKPOINT_RET (RT_V143 ? 0xbdb48cull : 0xe7a954ull)
|
||||
#define NVN_RET (RT_V143 ? 0xbe0360ull : 0xe7eb2cull)
|
||||
#define CTOR_DONE 2 // a module's constructor flag after its DT_INIT ran the init_array
|
||||
|
||||
// The init_array sizes of 1.2.1 (section 0.1); MacNativeGames' boot check expects the same.
|
||||
static const struct { const char *name; uint64_t entries; } expected_inits[] = {
|
||||
{"main", 10673}, {"subsdk0", 45}, {"sdk", 116}};
|
||||
// The init_array sizes (1.2.1: section 0.1; 1.4.3: DT_INIT_ARRAYSZ); MacNativeGames' boot check expects the same.
|
||||
typedef struct { const char *name; uint64_t entries; } InitCount;
|
||||
static const InitCount inits_121[] = {{"main", 10673}, {"subsdk0", 45}, {"sdk", 116}};
|
||||
static const InitCount inits_143[] = {{"main", 10717}, {"subsdk0", 45}, {"sdk", 83}};
|
||||
#define expected_inits (RT_V143 ? inits_143 : inits_121)
|
||||
|
||||
static volatile LONG inits_done, checkpoint_seen;
|
||||
|
||||
@@ -63,7 +69,7 @@ static void cb_loader_ready(Cpu *c) {
|
||||
// afterwards (MacNativeGames loader.c:458-462).
|
||||
static void cb_module_inits(Cpu *c) {
|
||||
if (InterlockedCompareExchange(&inits_done, 0, 0)) rt_fatal("boot: module constructors requested twice");
|
||||
for (unsigned k = 0; k < sizeof expected_inits / sizeof *expected_inits; k++) {
|
||||
for (unsigned k = 0; k < sizeof inits_121 / sizeof *inits_121; k++) {
|
||||
const RtLoaded *m = module_named(expected_inits[k].name);
|
||||
uint64_t entries = m->init_array_size / 8;
|
||||
if (!m->init) rt_fatal("boot: %s has no DT_INIT", m->name);
|
||||
@@ -73,7 +79,7 @@ static void cb_module_inits(Cpu *c) {
|
||||
printf("boot: DT_INIT %s state=%u (%llu)\n", m->name, state, (unsigned long long)entries);
|
||||
if (state != CTOR_DONE) rt_fatal("boot: %s DT_INIT left its constructor flag at %u, not 2", m->name, state);
|
||||
if (entries != expected_inits[k].entries)
|
||||
rt_fatal("boot: %s has %llu init_array entries, 1.2.1 has %llu", m->name, (unsigned long long)entries,
|
||||
rt_fatal("boot: %s has %llu init_array entries, this game version has %llu", m->name, (unsigned long long)entries,
|
||||
(unsigned long long)expected_inits[k].entries);
|
||||
}
|
||||
InterlockedExchange(&inits_done, 1);
|
||||
|
||||
+18
-13
@@ -2,22 +2,23 @@
|
||||
// contributors): runtime_frame_step_update (2338bb0: no 1/120 s floor) and the speed survey's player chain
|
||||
// (world_survey.c player_actor).
|
||||
// Frame rate: 60 fps and the real-time frame step, the host half of the patch sets fps60, realtime and cloth
|
||||
// (build table patches-v121.txt; docs/research/macnativegames.md section 6.2).
|
||||
// (build tables patches-v121.txt, patches-v143.txt; docs/research/macnativegames.md section 6.2).
|
||||
//
|
||||
// TotK's adaptive present-interval controller (main+0x8f26c8) turns the main thread's measured work into a number of
|
||||
// 60 Hz ticks clamped to [minimum, maximum]; shipping data sets the minimum to 2 (30 fps), the set fps60 lowers it to
|
||||
// 1. The controller still raises the interval when a frame does not fit. Shipping code then steps the game by whole
|
||||
// vsyncs (VFR step main+0x7ede6c: ratio = interval * 0.5, seconds = interval / 60; global ratio at main+0x8f2394), so
|
||||
// game speed is right only when frames take exactly their interval. With the set realtime both read the patch page
|
||||
// instead: +0 ratio = seconds * 30, +4 seconds, the moving average of the last 5 display intervals, at most 1/15 s
|
||||
// instead: +0 ratio = seconds * 30, +4 seconds (1.4.3: +8 interval = seconds * 60, the producer's nominal interval
|
||||
// before its half-step coefficient), the moving average of the last 5 display intervals, at most 1/15 s
|
||||
// (slower frames slow the game down, as on the console), no lower bound, so a higher frame-rate cap only changes the
|
||||
// pacer (nvn_api_obj.c VSYNC_HZ). The set cloth gives hair and cloth the same clock and needs realtime.
|
||||
//
|
||||
// The sample is the display time the pacer gives each present (nvnQueuePresentTexture), so the step follows the
|
||||
// frames as shown, pacing waits included. With the step on, the pacer ignores the game's interval and holds a frame
|
||||
// only to the frame-rate cap (nvn_api_obj.c present_period): the controller's 2 or 3 vsyncs would only add latency.
|
||||
// One 64-bit store per present; the guest reads the two floats on its own
|
||||
// threads, and both stay finite and in range whichever store they see.
|
||||
// One 64-bit and one 32-bit store per present; the guest reads the floats on its own threads, and each stays finite
|
||||
// and in range whichever store it sees.
|
||||
//
|
||||
// TOTK_TRACE=perf adds a line per 120 presents: fps, the game seconds stepped against wall seconds, and with
|
||||
// TOTK_SPEED_PROBE=1 the player's position and ground speed (an independent check: Link runs at the same speed at
|
||||
@@ -40,16 +41,19 @@
|
||||
#define STEP_FIRST_S (1.0 / 60.0)
|
||||
|
||||
static _Atomic uint64_t *step_word; // patch page +0: ratio (low float), seconds (high float)
|
||||
static _Atomic uint32_t *step_interval; // patch page +8: seconds * 60 (patches-v143.txt realtime)
|
||||
static uint64_t last_ns;
|
||||
static double samples[STEP_SAMPLES];
|
||||
static unsigned sample_count, sample_next;
|
||||
|
||||
static uint64_t step_bits(float seconds) {
|
||||
float ratio = seconds * 30.0f;
|
||||
uint32_t lo, hi;
|
||||
static void step_store(float seconds) {
|
||||
float ratio = seconds * 30.0f, interval = seconds * 60.0f;
|
||||
uint32_t lo, hi, iv;
|
||||
memcpy(&lo, &ratio, 4);
|
||||
memcpy(&hi, &seconds, 4);
|
||||
return (uint64_t)lo | (uint64_t)hi << 32;
|
||||
memcpy(&iv, &interval, 4);
|
||||
atomic_store_explicit(step_word, (uint64_t)lo | (uint64_t)hi << 32, memory_order_release);
|
||||
atomic_store_explicit(step_interval, iv, memory_order_release);
|
||||
}
|
||||
|
||||
void frame_step_init(void) {
|
||||
@@ -64,7 +68,8 @@ void frame_step_init(void) {
|
||||
}
|
||||
if (rt_patch_enabled(realtime)) {
|
||||
step_word = (_Atomic uint64_t *)rt_patch_page();
|
||||
atomic_store(step_word, step_bits((float)STEP_FIRST_S));
|
||||
step_interval = (_Atomic uint32_t *)((uint8_t *)rt_patch_page() + 8);
|
||||
step_store((float)STEP_FIRST_S);
|
||||
}
|
||||
rt_log("frame rate: present interval %s, %s", rt_patch_enabled(fps60) ? "1..3 (60 fps)" : "2..3 (shipping 30 fps)",
|
||||
step_word ? "real-time step (5-frame average of display intervals, at most 1/15 s)" : "whole-vsync steps");
|
||||
@@ -106,10 +111,10 @@ static uint64_t read_ptr(uint64_t a) {
|
||||
uint64_t p = 0;
|
||||
return read_guest(a, &p, 8) ? p : 0;
|
||||
}
|
||||
// 1.2.1: [main+0x4728538] +0x1e8 scene, its component 13 lists the resident actors; the one named "Player" has its
|
||||
// position at +0x2b4 (MacNativeGames world_survey.c, game_version.h TOTK_GAME_V121).
|
||||
// 1.2.1: [main+0x4728538] (1.4.3: [main+0x3add238]) +0x1e8 scene, its component 13 lists the resident actors; the one
|
||||
// named "Player" has its position at +0x2b4 (MacNativeGames world_survey.c, game_version.h TOTK_GAME_V121/V143).
|
||||
static int player_pos(float xyz[3]) {
|
||||
uint64_t module = read_ptr(MOD_MAIN_BASE + 0x4728538), scene = module ? read_ptr(module + 0x1e8) : 0;
|
||||
uint64_t module = read_ptr(MOD_MAIN_BASE + (RT_V143 ? 0x3add238 : 0x4728538)), scene = module ? read_ptr(module + 0x1e8) : 0;
|
||||
int32_t count = 0;
|
||||
if (!scene || !read_guest(scene + 0x50, &count, 4) || count < 14 || count > 256) return 0;
|
||||
uint64_t comps = read_ptr(scene + 0x58), resident = comps ? read_ptr(comps + 13 * 8) : 0;
|
||||
@@ -177,7 +182,7 @@ void frame_step_present(uint64_t display_ns, int interval) {
|
||||
double sum = 0;
|
||||
for (unsigned i = 0; i < sample_count; i++) sum += samples[i];
|
||||
step = sum / sample_count;
|
||||
atomic_store_explicit(step_word, step_bits((float)step), memory_order_release);
|
||||
step_store((float)step);
|
||||
}
|
||||
}
|
||||
last_ns = display_ns;
|
||||
|
||||
@@ -163,6 +163,8 @@ typedef struct { const char *file, *sha256; } RtFileHash;
|
||||
extern const RtFileHash rt_exefs_sha256[]; // tools/data/expected-<version>.json "exefs_sha256"
|
||||
extern const int rt_exefs_sha256_count;
|
||||
extern const char rt_default_exefs[], rt_default_romfs[], rt_default_profile[];
|
||||
extern const int rt_game_version; // the game build: 121 (1.2.1) or 143 (1.4.3)
|
||||
#define RT_V143 (rt_game_version >= 143)
|
||||
|
||||
// ---- kernel (kernel.c, T2)
|
||||
typedef struct RtThread RtThread;
|
||||
|
||||
+13
-6
@@ -715,13 +715,17 @@ static uint64_t stub(const char *sym) {
|
||||
// mutex look owned by the caller (__cxa_guard_acquire in nn::sm::Initialize aborts);
|
||||
// - sdk's full DT_INIT needs nn::os initialised (its TLS slot constructor reaches ThreadManager), which is the boot
|
||||
// sequence's job (T6). The audio client only needs its two sf heap constructors: 16 KiB ExpHeaps on static buffers,
|
||||
// stored at sdk+0xd37e90 and sdk+0xd3bec0 (CreateAudioRendererManagerByHipc reads the second).
|
||||
// stored at sdk+0xd37e90 and sdk+0xd3bec0 (CreateAudioRendererManagerByHipc reads the second). The 1.4.3 SDK creates
|
||||
// that heap lazily; its OpenAudioRenderer takes nn::audio::detail::g_RendererResourceSemaphore first, which the
|
||||
// constructor at sdk+0x221fb0 initializes (two renderers).
|
||||
static void sdk_prepare(Cpu *c) {
|
||||
Cpu *again;
|
||||
ST32(LD64(c->tpidrro + 0x1F8) + 0x1B0, kern_main_thread(&again));
|
||||
const RtLoaded *sdk = ldr_module(2);
|
||||
static const uint64_t heap_ctors[] = {0x1dcbd0, 0x1de8a0};
|
||||
for (unsigned k = 0; k < sizeof heap_ctors / sizeof *heap_ctors; k++) {
|
||||
static const uint64_t ctors_121[] = {0x1dcbd0, 0x1de8a0}, ctors_143[] = {0x221fb0};
|
||||
const uint64_t *heap_ctors = RT_V143 ? ctors_143 : ctors_121;
|
||||
unsigned nctors = RT_V143 ? sizeof ctors_143 / sizeof *ctors_143 : sizeof ctors_121 / sizeof *ctors_121;
|
||||
for (unsigned k = 0; k < nctors; k++) {
|
||||
int found = 0;
|
||||
for (uint64_t e = 0; e < sdk->init_array_size; e += 8)
|
||||
found |= LD64(sdk->init_array + e) == sdk->base + heap_ctors[k];
|
||||
@@ -740,8 +744,8 @@ static void probe_sdk_client(Cpu *c) {
|
||||
rt_call(c, stub(SYM_INIT_PARAM), 1, (uint64_t)(uintptr_t)param);
|
||||
printf("sdk: InitializeAudioRendererParameter -> %d Hz x%d, mix buffers %d, revision word %#x (REV%d)\n", param[0],
|
||||
param[1], param[2], (uint32_t)param[13], revision_number_of((uint32_t)param[13]));
|
||||
CHECK(param[0] == 48000 && param[1] == 240 && (uint32_t)param[13] == g_param.revision,
|
||||
"SDK default parameter (TotK 1.2.1 builds REV11 requests)");
|
||||
CHECK(param[0] == 48000 && param[1] == 240 && (uint32_t)param[13] == AUDREN_REV(RT_V143 ? 15 : 11),
|
||||
"SDK default parameter (TotK 1.2.1 builds REV11 requests, 1.4.3 REV15)");
|
||||
param[2] = g_param.mix_buffer_count, param[3] = g_param.submix_count, param[4] = g_param.voice_count;
|
||||
param[5] = g_param.sink_count, param[6] = g_param.effect_count;
|
||||
uint64_t work = rt_call(c, stub(SYM_WORK_SIZE), 1, (uint64_t)(uintptr_t)param);
|
||||
@@ -784,7 +788,10 @@ static void probe_sdk_client(Cpu *c) {
|
||||
"lifted client (sm: kept open by the SDK: %ld session)\n", audren_renderers_live(),
|
||||
(unsigned long long)(ipc_sessions_released() - released0), (unsigned long long)requests,
|
||||
ipc_sessions_live());
|
||||
CHECK(audren_renderers_live() == 0 && ipc_sessions_released() - released0 >= 3 && ipc_sessions_live() == 1,
|
||||
// The 1.2.1 SDK closes the renderer, device and manager sessions and keeps sm; the 1.4.3 SDK keeps the
|
||||
// audren:u manager it created lazily, beside sm.
|
||||
CHECK(audren_renderers_live() == 0 && ipc_sessions_released() - released0 >= (RT_V143 ? 1u : 3u) &&
|
||||
ipc_sessions_live() == (RT_V143 ? 2 : 1),
|
||||
"SDK CloseAudioRenderer");
|
||||
rt_trace = trace;
|
||||
}
|
||||
|
||||
+67
-9
@@ -19,7 +19,7 @@ uint64_t hostfn_register_observer(const char *name, GuestFn fn, uint64_t origina
|
||||
|
||||
enum { DT_NULL = 0, DT_PLTRELSZ = 2, DT_HASH = 4, DT_STRTAB = 5, DT_SYMTAB = 6, DT_RELA = 7, DT_RELASZ = 8,
|
||||
DT_INIT = 12, DT_FINI = 13, DT_JMPREL = 23, DT_INIT_ARRAY = 25, DT_FINI_ARRAY = 26, DT_INIT_ARRAYSZ = 27,
|
||||
DT_FINI_ARRAYSZ = 28 };
|
||||
DT_FINI_ARRAYSZ = 28, DT_RELRSZ = 0x23, DT_RELR = 0x24, DT_GNU_HASH = 0x6ffffef5 };
|
||||
enum { R_ABS64 = 257, R_GLOB_DAT = 1025, R_JUMP_SLOT = 1026, R_RELATIVE = 1027 };
|
||||
enum { K_RELATIVE, K_ABS64, K_GLOB_DAT, K_JUMP_SLOT, K_COUNT };
|
||||
enum { STB_WEAK = 2, STT_FUNC = 2 };
|
||||
@@ -39,6 +39,8 @@ typedef struct {
|
||||
uint32_t nsyms;
|
||||
const Elf64Rela *rela, *jmprel;
|
||||
uint64_t nrela, njmprel;
|
||||
const uint64_t *relr; // DT_RELR (1.4.3): packed RELATIVE relocations
|
||||
uint64_t nrelr, relr_count;
|
||||
uint32_t *hash, hash_mask; // defined symbols by name
|
||||
Res *res_plain, *res_jump; // per symbol; JUMP_SLOT may bind an observer
|
||||
uint32_t *funcs, nfuncs; // defined functions sorted by value (ldr_describe)
|
||||
@@ -219,12 +221,15 @@ static void parse_dynamic(Mod *m) {
|
||||
uint64_t mod0 = base + rd32(base + 4);
|
||||
if (memcmp((const void *)(uintptr_t)mod0, "MOD0", 4)) rt_fatal("loader: %s has no MOD0 header", m->pub.name);
|
||||
uint64_t dyn = mod0 + (int64_t)(int32_t)rd32(mod0 + 4);
|
||||
uint64_t rela = 0, relasz = 0, jmprel = 0, pltsz = 0, hash = 0;
|
||||
uint64_t rela = 0, relasz = 0, jmprel = 0, pltsz = 0, hash = 0, gnu_hash = 0, relr = 0, relrsz = 0;
|
||||
for (const uint64_t *d = (const uint64_t *)(uintptr_t)dyn; d[0] != DT_NULL; d += 2) {
|
||||
switch (d[0]) {
|
||||
case DT_SYMTAB: m->syms = (const Elf64Sym *)(uintptr_t)(base + d[1]); break;
|
||||
case DT_STRTAB: m->strtab = (const char *)(uintptr_t)(base + d[1]); break;
|
||||
case DT_HASH: hash = base + d[1]; break;
|
||||
case DT_GNU_HASH: gnu_hash = base + d[1]; break;
|
||||
case DT_RELR: relr = base + d[1]; break;
|
||||
case DT_RELRSZ: relrsz = d[1]; break;
|
||||
case DT_RELA: rela = base + d[1]; break;
|
||||
case DT_RELASZ: relasz = d[1]; break;
|
||||
case DT_JMPREL: jmprel = base + d[1]; break;
|
||||
@@ -234,8 +239,26 @@ static void parse_dynamic(Mod *m) {
|
||||
case DT_INIT_ARRAYSZ: m->pub.init_array_size = d[1]; break;
|
||||
}
|
||||
}
|
||||
if (!m->syms || !m->strtab || !hash) rt_fatal("loader: %s lacks DT_SYMTAB, DT_STRTAB or DT_HASH", m->pub.name);
|
||||
m->nsyms = rd32(hash + 4); // nchain
|
||||
if (!m->syms || !m->strtab || (!hash && !gnu_hash))
|
||||
rt_fatal("loader: %s lacks DT_SYMTAB, DT_STRTAB or a symbol hash", m->pub.name);
|
||||
if (hash) {
|
||||
m->nsyms = rd32(hash + 4); // nchain
|
||||
} else {
|
||||
// DT_GNU_HASH only (1.4.3 sdk, subsdk0): one past the last symbol the highest bucket's chain reaches.
|
||||
uint32_t nbuckets = rd32(gnu_hash), symoffset = rd32(gnu_hash + 4), bloom = rd32(gnu_hash + 8);
|
||||
uint64_t buckets = gnu_hash + 16 + 8ull * bloom, chains = buckets + 4ull * nbuckets;
|
||||
uint32_t last = 0;
|
||||
for (uint32_t i = 0; i < nbuckets; i++)
|
||||
if (rd32(buckets + 4ull * i) > last) last = rd32(buckets + 4ull * i);
|
||||
if (last < symoffset) {
|
||||
m->nsyms = symoffset;
|
||||
} else {
|
||||
while (!(rd32(chains + 4ull * (last - symoffset)) & 1)) last++;
|
||||
m->nsyms = last + 1;
|
||||
}
|
||||
}
|
||||
m->relr = (const uint64_t *)(uintptr_t)relr;
|
||||
m->nrelr = relrsz / 8;
|
||||
m->rela = (const Elf64Rela *)(uintptr_t)rela;
|
||||
m->nrela = relasz / sizeof(Elf64Rela);
|
||||
m->jmprel = (const Elf64Rela *)(uintptr_t)jmprel;
|
||||
@@ -363,6 +386,33 @@ static void relocate_table(Mod *m, const Elf64Rela *r, uint64_t n) {
|
||||
}
|
||||
}
|
||||
|
||||
// DT_RELR: an even entry is an offset to relocate (the next bitmap starts after it); an odd entry is a bitmap of the
|
||||
// 63 words that follow. Each relocated word holds its addend, as R_AARCH64_RELATIVE with an implicit addend.
|
||||
static void relocate_relr(Mod *m) {
|
||||
uint64_t base = m->pub.base, where = 0;
|
||||
for (uint64_t i = 0; i < m->nrelr; i++) {
|
||||
uint64_t e = m->relr[i];
|
||||
for (int k = -1; k < 63; k++) {
|
||||
uint64_t off;
|
||||
if (k < 0) {
|
||||
if (e & 1) continue;
|
||||
off = e;
|
||||
where = e + 8;
|
||||
} else {
|
||||
if (!(e & 1) || !((e >> (k + 1)) & 1)) continue;
|
||||
off = where + 8ull * k;
|
||||
}
|
||||
if (off < m->pub.data_off || off > m->pub.size - 8)
|
||||
rt_fatal("loader: %s RELR entry %llu writes +%#llx, outside the data segment", m->pub.name,
|
||||
(unsigned long long)i, (unsigned long long)off);
|
||||
ST64(base + off, base + LD64(base + off));
|
||||
m->kinds[K_RELATIVE]++;
|
||||
m->relr_count++;
|
||||
}
|
||||
if (e & 1) where += 63 * 8;
|
||||
}
|
||||
}
|
||||
|
||||
// Import table statistics: every undefined named symbol, bound by guest definition (overrides aside).
|
||||
static void count_imports(Mod *m) {
|
||||
for (uint32_t i = 1; i < m->nsyms; i++) {
|
||||
@@ -416,6 +466,7 @@ void ldr_load(const char *exefs_dir) {
|
||||
for (int i = 0; i < nmods; i++) {
|
||||
relocate_table(&mods[i], mods[i].rela, mods[i].nrela);
|
||||
relocate_table(&mods[i], mods[i].jmprel, mods[i].njmprel);
|
||||
relocate_relr(&mods[i]);
|
||||
count_imports(&mods[i]);
|
||||
mods[i].pub.ctor_flag = decode_ctor_flag(&mods[i]);
|
||||
}
|
||||
@@ -425,7 +476,7 @@ void ldr_load(const char *exefs_dir) {
|
||||
const Mod *m = &mods[i];
|
||||
rt_log("loaded %-8s at %#llx size %#llx: %llu relocations, %u imports", m->pub.name,
|
||||
(unsigned long long)m->pub.base, (unsigned long long)m->pub.size,
|
||||
(unsigned long long)(m->nrela + m->njmprel), m->imports);
|
||||
(unsigned long long)(m->nrela + m->njmprel + m->relr_count), m->imports);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -515,16 +566,23 @@ uint64_t ldr_plt_stub(const char *module, const char *name) {
|
||||
}
|
||||
|
||||
// ---- --probe-loader
|
||||
// Expected values, measured on the 1.2.1 modules.
|
||||
static const struct {
|
||||
// Expected values, measured on the 1.2.1 and 1.4.3 modules (1.4.3 relocation totals match MNG v1.4.3-port.md:20).
|
||||
typedef struct {
|
||||
const char *name;
|
||||
uint64_t kinds[K_COUNT], total, sym_import, sym_own;
|
||||
uint32_t imports, to[MOD_COUNT], weak, init_off, init_array;
|
||||
} expect[] = {
|
||||
} Expect;
|
||||
static const Expect expect_121[] = {
|
||||
{"main", {508816, 1415, 32, 641}, 510904, 2088, 0, 675, {0, 16, 647}, 12, 0x30, 10673},
|
||||
{"subsdk0", {1623, 12820, 1030, 3924}, 19397, 6576, 11198, 667, {9, 0, 654}, 4, 0x30, 45},
|
||||
{"sdk", {10917, 13976, 1207, 8932}, 35032, 16, 24099, 13, {8, 0, 0}, 5, 0x30, 116},
|
||||
};
|
||||
static const Expect expect_143[] = {
|
||||
{"main", {499078, 1393, 28, 650}, 501149, 2071, 0, 680, {0, 16, 656}, 8, 0x10b4f10, 10717},
|
||||
{"subsdk0", {3424, 10927, 905, 3817}, 19073, 6557, 9092, 663, {9, 0, 654}, 0, 0x20, 45},
|
||||
{"sdk", {15013, 10135, 1079, 7832}, 34059, 10, 19036, 8, {8, 0, 0}, 0, 0x400, 83},
|
||||
};
|
||||
#define expect (RT_V143 ? expect_143 : expect_121)
|
||||
|
||||
static int failures;
|
||||
#define CHECK(cond, ...) do { if (!(cond)) { failures++; printf(" FAIL: " __VA_ARGS__); putchar('\n'); } } while (0)
|
||||
@@ -614,7 +672,7 @@ int probe_loader(void) {
|
||||
const RtConfig *cfg = rt_config();
|
||||
hostfn_add_override(PROBE_SYMBOL, probe_override);
|
||||
ldr_load(cfg->exefs);
|
||||
if (nmods != (int)(sizeof expect / sizeof *expect)) {
|
||||
if (nmods != (int)(sizeof expect_121 / sizeof *expect_121)) {
|
||||
printf("probe-loader: FAILED (%d modules, expected 3)\n", nmods);
|
||||
return 1;
|
||||
}
|
||||
|
||||
+12
-5
@@ -123,8 +123,9 @@ const pool_t *nvn_pool_find(uint64_t addr, uint64_t len) {
|
||||
return p;
|
||||
}
|
||||
|
||||
// Physical and virtual pools (MacNativeGames totk_vpool.c:32-52 checks). A CPU-invisible one gets a GPU window range;
|
||||
// a CPU-visible one would need aliased guest storage, which this port does not support (section 3.3).
|
||||
// Physical and virtual pools (MacNativeGames totk_vpool.c:32-52 checks). A CPU-invisible one gets a GPU window range.
|
||||
// A CPU-visible physical pool (1.4.3: flags 0x18c, GPU_NO_ACCESS) keeps the game's storage for the CPU; the back end
|
||||
// gives it device memory as any physical pool (vk_memory.c vk_pool_init), which the GPU reaches through virtual pools.
|
||||
static int pool_place(pool_t *o) {
|
||||
int phys = (o->flags & NVN_MEMORY_POOL_FLAGS_PHYSICAL) != 0, virt = (o->flags & NVN_MEMORY_POOL_FLAGS_VIRTUAL) != 0;
|
||||
if (!phys && !virt) return 1;
|
||||
@@ -136,9 +137,15 @@ static int pool_place(pool_t *o) {
|
||||
}
|
||||
if (virt && o->mem) { rt_log("nvn: virtual pool given storage %p", (void *)o->mem); return 0; }
|
||||
if (phys && !o->mem) { rt_log("nvn: physical pool without storage"); return 0; }
|
||||
if (!(o->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS))
|
||||
rt_fatal("nvn: CPU-visible %s pool (flags %#x, size %#llx) needs aliased guest storage",
|
||||
kind, o->flags, (unsigned long long)o->size);
|
||||
if (!(o->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS)) {
|
||||
if (virt) {
|
||||
rt_log("nvn: CPU-visible virtual pool (flags %#x, size %#llx) refused", o->flags, (unsigned long long)o->size);
|
||||
return 0;
|
||||
}
|
||||
rt_log("nvn: CPU-visible physical pool %p (flags %#x, size %#llx) keeps the game's storage", (void *)o->mem,
|
||||
o->flags, (unsigned long long)o->size);
|
||||
return 1;
|
||||
}
|
||||
uint64_t a = gva_alloc(o->size);
|
||||
if (!a) {
|
||||
rt_log("nvn: GPU address window full, %s pool of %#llx bytes refused", kind, (unsigned long long)o->size);
|
||||
|
||||
@@ -854,6 +854,8 @@ static void ov_get_output_md(Cpu *c) {
|
||||
|
||||
HOST_OVERRIDE("_ZN5movie7DecoderC1ENS_11DecoderTypeENS_19DecoderOutputFormatENS_11DecoderModeEi", ov_ctor);
|
||||
HOST_OVERRIDE("_ZN5movie7DecoderD2Ev", ov_d2);
|
||||
// 1.4.3's vtable and a PLT call name the complete-object destructor; without virtual bases it is D2.
|
||||
HOST_OVERRIDE("_ZN5movie7DecoderD1Ev", ov_d2);
|
||||
HOST_OVERRIDE("_ZN5movie7DecoderD0Ev", ov_d0);
|
||||
HOST_OVERRIDE("_ZN5movie7Decoder9ConfigureEPNS_9MediaDataE", ov_configure);
|
||||
HOST_OVERRIDE("_ZN5movie7Decoder20RegisterClientEventsEPNS_13DecoderEventsE", ov_register_events);
|
||||
|
||||
+10
-4
@@ -75,6 +75,7 @@ static Ov ovs[] = {
|
||||
// SDK-internal (section 5.2)
|
||||
{"nndiagStartup", "nndiagStartup"},
|
||||
{"aoc.Initialize", "_ZN2nn3aoc10InitializeEv"},
|
||||
{"aoc.ListAddOnContent", "_ZN2nn3aoc16ListAddOnContentEPiii"},
|
||||
{"err.InitializeContextWriterApi", "_ZN2nn3err26InitializeContextWriterApiEv"},
|
||||
{"pctl.Initialize", "_ZN2nn4pctl10InitializeEv"},
|
||||
{"diag.InitializeApplicationAbortObserver", "_ZN2nn4diag34InitializeApplicationAbortObserverEv"},
|
||||
@@ -374,6 +375,7 @@ static void probe_applet(void) {
|
||||
CALL("err.ShowError", 0x2a8);
|
||||
CALL("nndiagStartup", 0);
|
||||
CALL("aoc.Initialize", 0);
|
||||
CHECK(CALL("aoc.ListAddOnContent", U(out), 0, 8) == 0, "aoc: no add-on content");
|
||||
CALL("err.InitializeContextWriterApi", 0);
|
||||
CALL("pctl.Initialize", 0);
|
||||
CALL("diag.InitializeApplicationAbortObserver", 0);
|
||||
@@ -628,10 +630,12 @@ static void probe_media_gfx(void) {
|
||||
"vsync event %s\n", display, layer, window, vsync ? "bound (auto-clear)" : "FAILED");
|
||||
}
|
||||
|
||||
// ---- software keyboard (swkbd_win.c): ShowKeyboardArg built by the 1.2.1 SDK's own setters, as main+0x1b2b3a0 does
|
||||
// ---- software keyboard (swkbd_win.c): ShowKeyboardArg built by the SDK's own setters, as the horse naming does
|
||||
enum { KB_TYPE = 0x00, KB_DISABLED = 0x1c, KB_HEADER = 0x24, KB_GUIDE = 0x1a8, KB_MAX_LEN = 0x3ac, KB_MIN_LEN = 0x3b0,
|
||||
KB_UTF8 = 0x3bd, KB_INIT_LEN = 0x3c4, ARG_WORK = 0x4d0, ARG_WORK_SIZE = 0x4d8, ARG_SIZE = 0x500,
|
||||
STRING_SIZE = 0x7d4, SWKBD_CANCELED = 0x29f };
|
||||
KB_UTF8 = 0x3bd, KB_INIT_LEN = 0x3c4, STRING_SIZE = 0x7d4, SWKBD_CANCELED = 0x29f };
|
||||
#define ARG_WORK (RT_V143 ? 0x518 : 0x4d0) // SDK 20.5 grew KeyboardConfig (swkbd_win.c)
|
||||
#define ARG_WORK_SIZE (ARG_WORK + 8)
|
||||
#define ARG_SIZE (RT_V143 ? 0x548 : 0x500)
|
||||
|
||||
static uint64_t sdk(const char *name) {
|
||||
uint64_t a = ldr_symbol(name);
|
||||
@@ -964,7 +968,9 @@ int probe_services(void) {
|
||||
for (int k = 0; k < NOBS; k++)
|
||||
if (!strcmp(observers[k], n)) obs_bound++;
|
||||
}
|
||||
for (int k = 0; k < NOV; k++) CHECK(ovs[k].bound, "override %s matches no relocation", ovs[k].name);
|
||||
for (int k = 0; k < NOV; k++) // 1.2.1 lists no add-on content
|
||||
CHECK(ovs[k].bound || (!RT_V143 && !strcmp(ovs[k].key, "aoc.ListAddOnContent")), "override %s matches no relocation",
|
||||
ovs[k].name);
|
||||
for (int k = 0; k < NOBS; k++) CHECK(hostfn_observer(observers[k]), "observer %s is not linked", observers[k]);
|
||||
CHECK(obs_bound == NOBS, "%d of %d observers match a JUMP_SLOT", obs_bound, NOBS);
|
||||
|
||||
|
||||
@@ -301,6 +301,13 @@ static void ov_show_error(Cpu *c) {
|
||||
// ==== SDK-internal initialisation (services.c:208-218)
|
||||
static void ov_diag_startup(Cpu *c) { (void)c; rt_log("diag: host diagnostics"); }
|
||||
static void ov_aoc_initialize(Cpu *c) { (void)c; rt_log("aoc: no add-on content"); }
|
||||
// 1.4.3's main lists add-on content (MNG services.c host_aoc_list): there is none.
|
||||
// int ListAddOnContent(AddOnContentIndex *out, int offset, int count)
|
||||
static void ov_aoc_list(Cpu *c) {
|
||||
int offset = (int)arg_x(c, 1), count = (int)arg_x(c, 2);
|
||||
if (offset < 0 || count < 0) rt_fatal("aoc: ListAddOnContent range %d+%d", offset, count);
|
||||
ret_x(c, 0);
|
||||
}
|
||||
static void ov_err_context(Cpu *c) { (void)c; rt_log("err: error contexts stay local"); }
|
||||
static void ov_pctl_initialize(Cpu *c) { (void)c; rt_log("pctl: no parental controls"); }
|
||||
static void ov_abort_observer(Cpu *c) { (void)c; rt_log("diag: the host crash reporter observes aborts"); }
|
||||
@@ -820,6 +827,7 @@ HOST_OVERRIDE("_ZN2nn3err9ShowErrorENS_6ResultE", ov_show_error);
|
||||
|
||||
HOST_OVERRIDE("nndiagStartup", ov_diag_startup);
|
||||
HOST_OVERRIDE("_ZN2nn3aoc10InitializeEv", ov_aoc_initialize);
|
||||
HOST_OVERRIDE("_ZN2nn3aoc16ListAddOnContentEPiii", ov_aoc_list);
|
||||
HOST_OVERRIDE("_ZN2nn3err26InitializeContextWriterApiEv", ov_err_context);
|
||||
HOST_OVERRIDE("_ZN2nn4pctl10InitializeEv", ov_pctl_initialize);
|
||||
HOST_OVERRIDE("_ZN2nn4diag34InitializeApplicationAbortObserverEv", ov_abort_observer);
|
||||
|
||||
+2
-1
@@ -6,7 +6,8 @@
|
||||
//
|
||||
// The patch page is the first 4 KiB page after main's image, inside the module window memory.c reserved: replacement
|
||||
// code reaches it with one adrp, as MacNativeGames' data page after main (games/totk/runtime/game_patches.c). Its
|
||||
// layout belongs to the sets that use it (frame_rate.c: +0 the 30 Hz ratio, +4 the frame step in seconds).
|
||||
// layout belongs to the sets that use it (frame_rate.c: +0 the 30 Hz ratio, +4 the frame step in seconds, +8 the
|
||||
// step in 60 Hz ticks).
|
||||
// Probe modes never call rt_patches_init: every set stays off.
|
||||
#include "cfg.h"
|
||||
#include "platform/platform.h"
|
||||
|
||||
+5
-3
@@ -30,12 +30,14 @@
|
||||
#include "ui_text.h"
|
||||
#include "vk.h"
|
||||
|
||||
// KeyboardConfig (0x4d0 bytes) then the work buffer {pointer, size}: offsets read from the SDK setters (MNG) and
|
||||
// checked against the 1.2.1 SDK by --probe-services.
|
||||
// KeyboardConfig then the work buffer {pointer, size}: offsets read from the SDK setters (MNG) and checked against
|
||||
// the SDK by --probe-services. SDK 20.5 (1.4.3) put a text-check callback at +0x4c8 and grew KeyboardConfig from
|
||||
// 0x4d0 to 0x518 bytes (SetInitialText, SetUserWordList read the work buffer there); the fields below +0x3d8 stay.
|
||||
#define ARG_WORK (RT_V143 ? 0x518 : 0x4d0)
|
||||
#define ARG_WORK_SIZE (ARG_WORK + 8)
|
||||
enum {
|
||||
KB_TYPE = 0x00, KB_OK_TEXT = 0x04, KB_DISABLED = 0x1c, KB_HEADER = 0x24, KB_SUB = 0xa6, KB_GUIDE = 0x1a8,
|
||||
KB_MAX_LEN = 0x3ac, KB_MIN_LEN = 0x3b0, KB_PASSWORD = 0x3b4, KB_UTF8 = 0x3bd, KB_INIT_OFF = 0x3c0, KB_INIT_LEN = 0x3c4,
|
||||
ARG_WORK = 0x4d0, ARG_WORK_SIZE = 0x4d8,
|
||||
KB_TYPE_NUMPAD = 1, KB_LIMIT = 500, SWKBD_CANCELED = 0x29f,
|
||||
};
|
||||
enum { B_A = 0, B_B = 1, B_PLUS = 10 }; // input.h bits
|
||||
|
||||
+3
-1
@@ -242,7 +242,9 @@ static void destroy_rec(VkCtx *c, PoolRec *r) {
|
||||
int vk_pool_init(pool_t *p) {
|
||||
VkCtx *c = vk_ctx();
|
||||
if (!c) { rt_log("vk: MemoryPoolInitialize before vk_init"); return 0; }
|
||||
// The GPU never sees a GPU_NO_ACCESS pool, except a physical one (0x189) through the virtual pools that map it.
|
||||
// The GPU never sees a GPU_NO_ACCESS pool, except a physical one (0x189; 1.4.3 also 0x18c, CPU-visible) through the
|
||||
// virtual pools that map it. Either gets device-local memory: 1.4.3's CPU never writes its 0x18c pool's storage
|
||||
// (unchanged over 3600 frames of the intro), so the guest storage and the GPU copy need no sync.
|
||||
if ((p->flags & NVN_MEMORY_POOL_FLAGS_GPU_NO_ACCESS) && !(p->flags & NVN_MEMORY_POOL_FLAGS_PHYSICAL)) return 1;
|
||||
PoolRec *r = calloc(1, sizeof *r);
|
||||
if (!r) rt_fatal("vk: out of memory for a pool record");
|
||||
|
||||
@@ -76,7 +76,7 @@ class Bed:
|
||||
|
||||
def run(self, release='1.0', inputs=None, fail=None, verify_ok=None, verify=False):
|
||||
layout = Layout(root=Path('.'), dev=True, release=release, python=Path('python'), tools=Path('tools'),
|
||||
recomp=Path('recomp'), patches=Path('p'), expected=Path('e'), llvm=Path('llvm'),
|
||||
recomp=Path('recomp'), llvm=Path('llvm'),
|
||||
deps=Path('deps'), runtime=Path('rt'), runtime_lib=Path('lib'), ninja=Path('ninja'),
|
||||
hactool=Path('hactool'), xlate=Path('xlate'))
|
||||
options = SimpleNamespace(inputs=inputs or {}, fail=dict(fail or {}), verify_ok=verify_ok or {}, ran=[],
|
||||
|
||||
+234
-6
@@ -55,6 +55,10 @@ class Oracle:
|
||||
mu.reg_write(A.UC_ARM64_REG_CPACR_EL1, 0x300000)
|
||||
self.mu = mu
|
||||
self.set_at = self.regs = None
|
||||
self.start = self.cmp = None # the first instruction of the bounds-check block, the bounds cmp
|
||||
self.guard = None # _taken: the b.ls/b.lo into the dispatch block (set_at)
|
||||
self.count = None # indices the bounds check lets through
|
||||
self.exclude = frozenset() # indices the emulation skips (oracle_wrong: excluded on every way in)
|
||||
self.watch = self.seen = None # (address, register): record the register when the emulation reaches address
|
||||
|
||||
def word(self, a):
|
||||
@@ -65,7 +69,7 @@ class Oracle:
|
||||
|
||||
def jump_table(self, br_pc, entry, end):
|
||||
"""Targets, or None. Leaves the index set point and the registers set there in self.set_at, self.regs."""
|
||||
self.set_at = self.regs = None
|
||||
self.set_at = self.regs = self.start = self.cmp = self.guard = self.count = None
|
||||
r = self._fallthrough(br_pc, entry, end)
|
||||
if r is None:
|
||||
r = self._taken(br_pc, entry, end)
|
||||
@@ -127,6 +131,8 @@ class Oracle:
|
||||
targets = []
|
||||
try:
|
||||
for i in range(count):
|
||||
if i in self.exclude:
|
||||
continue
|
||||
self._reset(entry, start)
|
||||
if set_at != start and not self._step_to(start, set_at):
|
||||
return None
|
||||
@@ -159,7 +165,11 @@ class Oracle:
|
||||
count = limit + 1 if bits(w, 0, 4) == 9 else limit
|
||||
if count <= 0 or count > 4096:
|
||||
continue
|
||||
self.set_at, self.regs = s, [bits(cw, 5, 5)]
|
||||
self.set_at, self.regs, self.guard, self.cmp, self.count = s, [bits(cw, 5, 5)], g, g - 4, count
|
||||
b = g - 4
|
||||
while b - 4 >= entry and g - 4 - (b - 4) <= 4 * 12 and self.kind(b - 4) in ('op', 'asm'):
|
||||
b -= 4
|
||||
self.start = b
|
||||
return self._emulate(br_pc, entry, end, s, [bits(cw, 5, 5)], count, s)
|
||||
return None
|
||||
|
||||
@@ -197,11 +207,13 @@ class Oracle:
|
||||
while start - 4 >= entry and cmp - (start - 4) <= 4 * 12 and self.kind(start - 4) in ('op', 'asm'):
|
||||
start -= 4
|
||||
regs = self._index_regs(start, cmp, bits(w, 5, 5))
|
||||
self.set_at, self.regs = cmp, regs
|
||||
self.set_at, self.regs, self.start, self.cmp, self.count = cmp, regs, start, cmp, count
|
||||
rn = bits(self.word(br_pc), 5, 5)
|
||||
targets = []
|
||||
try:
|
||||
for i in range(count):
|
||||
if i in self.exclude:
|
||||
continue
|
||||
self._reset(entry, start)
|
||||
if not self._step_to(start, cmp):
|
||||
return None
|
||||
@@ -397,13 +409,218 @@ def _reads(ins):
|
||||
return {n for n in (_regno(ins.reg_name(x)) for x in r) if n is not None}
|
||||
|
||||
|
||||
def oracle_wrong(fc, br, oracle):
|
||||
def _branch_target(ins):
|
||||
"""Target of a direct b, b.cond, cbz/cbnz or tbz/tbnz, else None."""
|
||||
if ins is None or not ins.operands or not (ins.mnemonic in ('b', 'cbz', 'cbnz', 'tbz', 'tbnz')
|
||||
or ins.mnemonic.startswith('b.')):
|
||||
return None
|
||||
op = ins.operands[-1]
|
||||
return op.imm if op.type == capstone.arm64.ARM64_OP_IMM else None
|
||||
|
||||
|
||||
def _ways_in(fc, lo, hi, tables):
|
||||
"""Direct branches of the function into [lo, hi], or None when the range is entered otherwise too: by falling
|
||||
through into lo, at the function entry, or as a target of another table."""
|
||||
mod = fc.mod
|
||||
if lo <= fc.entry or any(lo <= t <= hi for ts in tables.values() if ts != 'all' for t in ts):
|
||||
return None
|
||||
prev = _insn(mod, lo - 4)
|
||||
if prev is None or prev.mnemonic not in ('b', 'br', 'ret'):
|
||||
return None
|
||||
out = []
|
||||
for a in range(fc.entry, fc.end, 4):
|
||||
ins = _insn(mod, a)
|
||||
t = _branch_target(ins)
|
||||
if t is not None and lo <= t <= hi:
|
||||
out.append((a, ins, t))
|
||||
return out
|
||||
|
||||
|
||||
# Instructions an index slice may hold: integer arithmetic that reads only general registers and writes the flags at
|
||||
# most (the flags-reading forms adc/sbc/csel/ccmp/... stay out).
|
||||
_FLAG_SETTERS = {'cmp', 'cmn', 'tst', 'ands', 'bics', 'adds', 'subs', 'negs'}
|
||||
_PURE_ALU = {'mov', 'movz', 'movn', 'movk', 'add', 'sub', 'and', 'orr', 'eor', 'bic', 'orn', 'eon', 'lsl', 'lsr',
|
||||
'asr', 'ror', 'ubfx', 'sbfx', 'ubfiz', 'sbfiz', 'bfi', 'bfxil', 'uxtb', 'uxth', 'uxtw', 'sxtb', 'sxth',
|
||||
'sxtw', 'neg', 'mvn', 'mul', 'madd', 'msub', 'udiv', 'sdiv', 'ubfm', 'sbfm', 'bfm'} | _FLAG_SETTERS
|
||||
_DEPTH = 4 # predecessor blocks above the dispatch region
|
||||
|
||||
|
||||
def _ends_block(ins):
|
||||
return ins is None or _branch_target(ins) is not None or ins.mnemonic in ('br', 'ret', 'brk', 'udf')
|
||||
|
||||
|
||||
class _Ways:
|
||||
"""Direct branches and fall-throughs of one function, with the addresses control may enter from elsewhere."""
|
||||
|
||||
def __init__(self, fc, tables):
|
||||
mod = self.mod = fc.mod
|
||||
self.entry, self.into = fc.entry, {}
|
||||
for a in range(fc.entry, fc.end, 4):
|
||||
ins = _insn(mod, a)
|
||||
t = _branch_target(ins)
|
||||
if t is not None and fc.entry <= t < fc.end:
|
||||
self.into.setdefault(t, []).append((a, ins))
|
||||
self.roots = {fc.entry} | set(fc.cfg.unknown)
|
||||
self.roots |= {t for ts in tables.values() if ts != 'all' for t in ts}
|
||||
self.leaders = self.roots | set(self.into)
|
||||
|
||||
def block_start(self, a):
|
||||
s = a
|
||||
while s > self.entry and s not in self.leaders and not _ends_block(_insn(self.mod, s - 4)):
|
||||
s -= 4
|
||||
return s
|
||||
|
||||
def preds(self, s):
|
||||
"""(branch or last instruction, insn, taken) for every way into s."""
|
||||
out = [(a, ins, True) for a, ins in self.into.get(s, ())]
|
||||
if s > self.entry:
|
||||
p = _insn(self.mod, s - 4)
|
||||
if p is not None and p.mnemonic not in ('b', 'br', 'ret', 'brk', 'udf'):
|
||||
out.append((s - 4, p, False))
|
||||
return out
|
||||
|
||||
|
||||
def _edge_holds(oracle, ways, start, src, ins, taken, reg, k):
|
||||
"""Whether the way out of the block [start, src] over the branch at src can be taken with x<reg> = k at src:
|
||||
False only when the branch condition is a function of x<reg> alone (its slice, back to the last write of x<reg>,
|
||||
is plain arithmetic and flag setting on x<reg>), evaluated by Unicorn; True otherwise."""
|
||||
if _branch_target(ins) is None or ins.mnemonic == 'b':
|
||||
return True
|
||||
mod = ways.mod
|
||||
|
||||
def gp_reads(i): # general registers read, zr dropped; None when sp or a non-general register is read
|
||||
try:
|
||||
r, _ = i.regs_access()
|
||||
except capstone.CsError:
|
||||
return None
|
||||
if i.mnemonic in ('cmp', 'cmn', 'tst'): # capstone lists their first operand as written, not read
|
||||
r = list(r) + [op.reg for op in i.operands if op.type == capstone.arm64.ARM64_OP_REG]
|
||||
out = set()
|
||||
for x in r:
|
||||
name = i.reg_name(x)
|
||||
if name in ('xzr', 'wzr', 'nzcv'):
|
||||
continue
|
||||
n = _regno(name)
|
||||
if n is None or n == 31:
|
||||
return None
|
||||
out.add(n)
|
||||
return out
|
||||
|
||||
def sets_flags(i):
|
||||
try:
|
||||
return i.mnemonic in _FLAG_SETTERS or 'nzcv' in (i.reg_name(r) for r in i.regs_access()[1])
|
||||
except capstone.CsError:
|
||||
return True
|
||||
|
||||
if ins.mnemonic.startswith('b.'):
|
||||
setter = next((x for x in range(src - 4, start - 4, -4) if sets_flags(_insn(mod, x))), None)
|
||||
if setter is None or _insn(mod, setter).mnemonic not in _FLAG_SETTERS or any(reg in _writes(_insn(mod, x)) for x in range(setter, src, 4)):
|
||||
return True
|
||||
need, top, sl = gp_reads(_insn(mod, setter)), setter, [setter]
|
||||
else:
|
||||
need, top, sl = gp_reads(ins), src, []
|
||||
if need is None:
|
||||
return True
|
||||
for x in range(top - 4, start - 4, -4):
|
||||
i = _insn(mod, x)
|
||||
w = _writes(i)
|
||||
if reg in w:
|
||||
break # x<reg> is k from here on
|
||||
if not w & need:
|
||||
continue
|
||||
r = gp_reads(i)
|
||||
if i.mnemonic not in _PURE_ALU or r is None:
|
||||
return True
|
||||
sl.append(x)
|
||||
need = (need - w) | r
|
||||
if need - {reg}:
|
||||
return True
|
||||
mu = oracle.mu
|
||||
try:
|
||||
for r in range(29):
|
||||
mu.reg_write(A.UC_ARM64_REG_X0 + r, SCRATCH_MEM + 0x80000)
|
||||
mu.reg_write(A.UC_ARM64_REG_X0 + reg if reg < 29 else A.UC_ARM64_REG_X29, k)
|
||||
for x in reversed(sl):
|
||||
mu.emu_start(x, 0, count=1)
|
||||
mu.emu_start(src, 0, count=1)
|
||||
went = mu.reg_read(A.UC_ARM64_REG_PC) != src + 4
|
||||
except uc.UcError:
|
||||
return True
|
||||
return went == taken
|
||||
|
||||
|
||||
def _way_live(oracle, ways, src, ins, taken, reg, k, depth, seen):
|
||||
"""Whether the way over src can carry x<reg> = k: its edge holds (_edge_holds) and its block either writes x<reg>
|
||||
or is entered with k (_reaches)."""
|
||||
start = ways.block_start(src)
|
||||
if not _edge_holds(oracle, ways, start, src, ins, taken, reg, k):
|
||||
return False
|
||||
if any(reg in _writes(_insn(ways.mod, x)) for x in range(start, src + 4, 4)):
|
||||
return True
|
||||
return _reaches(oracle, ways, start, reg, k, depth, seen)
|
||||
|
||||
|
||||
def _reaches(oracle, ways, s, reg, k, depth, seen):
|
||||
"""Whether control can enter s with x<reg> = k, over at most `depth` predecessor blocks."""
|
||||
if s in ways.roots or depth == 0 or s in seen:
|
||||
return True
|
||||
seen = seen | {s}
|
||||
return any(_way_live(oracle, ways, src, ins, taken, reg, k, depth - 1, seen)
|
||||
for src, ins, taken in ways.preds(s))
|
||||
|
||||
|
||||
def excluded_indices(fc, br, oracle, tables, count):
|
||||
"""Indices < count that cannot reach br: no way into the dispatch region [oracle.start, br] (its first instruction
|
||||
and every branch target inside it) can carry them. The index is the register of the oracle's bounds cmp, which no
|
||||
instruction from such an entry to br may write. Ways in are the module's direct branches and fall-throughs by
|
||||
capstone; the function entry, exception landing pads and every resolver table target count as entries from
|
||||
anywhere. Branch conditions are evaluated only where they depend on the index alone (_edge_holds). When the
|
||||
oracle enters the dispatch block by the taken bounds branch (_taken), that branch must be its only way in."""
|
||||
mod, lo = fc.mod, oracle.start
|
||||
if lo is None:
|
||||
return frozenset()
|
||||
if oracle.guard is not None:
|
||||
ways = _ways_in(fc, oracle.set_at, br, tables)
|
||||
if ways is None or [a for a, _, _ in ways] != [oracle.guard]:
|
||||
return frozenset()
|
||||
reg = bits(oracle.word(oracle.cmp), 5, 5)
|
||||
ways = _Ways(fc, tables)
|
||||
# The region falls through from lo to br (_fallthrough), or is the bounds-check block whose b.ls/b.lo is the only
|
||||
# way into the dispatch block (_taken).
|
||||
hi = br if oracle.guard is None else oracle.cmp
|
||||
if oracle.guard is not None and any(reg in _writes(_insn(mod, x)) for x in range(oracle.set_at, br, 4)):
|
||||
return frozenset()
|
||||
entries = [lo] + [t for t in range(lo + 4, hi + 4, 4) if t in ways.leaders]
|
||||
for e in entries:
|
||||
if e in ways.roots or any(reg in _writes(_insn(mod, x)) for x in range(e, hi, 4)):
|
||||
return frozenset()
|
||||
out = set()
|
||||
for k in range(count):
|
||||
live = False
|
||||
for e in entries:
|
||||
for src, ins, taken in ways.preds(e):
|
||||
if lo <= src < hi:
|
||||
continue # inside the region: its own conditions are ignored
|
||||
if _way_live(oracle, ways, src, ins, taken, reg, k, _DEPTH, frozenset()):
|
||||
live = True
|
||||
break
|
||||
if live:
|
||||
break
|
||||
if not live:
|
||||
out.add(k)
|
||||
return frozenset(out)
|
||||
|
||||
|
||||
def oracle_wrong(fc, br, oracle, ours=None):
|
||||
"""Why the oracle's targets for br are not the table's, from evidence independent of the resolver; or None.
|
||||
|
||||
- the index the table is read with does not depend on the registers the oracle sets (it comes from another
|
||||
register or is reloaded from memory, which the oracle leaves at its scratch value);
|
||||
- the table base the oracle uses (its replay of ADRP/ADR/ADD-imm from the entry) is not the one constant that
|
||||
every reaching definition gives.
|
||||
every reaching definition gives;
|
||||
- the oracle enumerates an index that no way into the dispatch region lets through (excluded_indices: a case
|
||||
peeled off before the table by `cmp wI, #k; b.ne`, `cbnz wI` or a range check on wI, possibly some blocks up),
|
||||
and without it the oracle gives the resolver's targets.
|
||||
"""
|
||||
mod = fc.mod
|
||||
s = fc.cfg.block(br)
|
||||
@@ -427,6 +644,17 @@ def oracle_wrong(fc, br, oracle):
|
||||
seen, oracle.watch, oracle.seen = oracle.seen, None, None
|
||||
if seen is not None and seen != value:
|
||||
return f'oracle base x{reg} = {seen:#x}, every reaching definition gives {value:#x}'
|
||||
if ours is not None:
|
||||
oracle.jump_table(br, fc.entry, fc.end)
|
||||
ex = excluded_indices(fc, br, oracle, fc.tables, oracle.count) if oracle.count else frozenset()
|
||||
if ex:
|
||||
oracle.exclude = ex
|
||||
try:
|
||||
again = oracle.jump_table(br, fc.entry, fc.end)
|
||||
finally:
|
||||
oracle.exclude = frozenset()
|
||||
if again == ours:
|
||||
return 'index ' + ', '.join(str(k) for k in sorted(ex)) + ' unreachable on every way in'
|
||||
return None
|
||||
|
||||
|
||||
@@ -535,7 +763,7 @@ def main():
|
||||
else:
|
||||
key = 'computed, unresolved' if comp else None
|
||||
if key in ('disagree', 'only oracle'):
|
||||
why = oracle_wrong(fc, br, oracle)
|
||||
why = oracle_wrong(fc, br, oracle, ours)
|
||||
if why:
|
||||
key = 'oracle wrong' if key == 'disagree' else 'only oracle, oracle wrong'
|
||||
if key or raw is not None:
|
||||
|
||||
+10
-6
@@ -187,9 +187,9 @@ def runtime_object(out, src):
|
||||
return out / 'obj' / 'runtime' / rel.with_suffix('.o')
|
||||
|
||||
|
||||
def module_glue(layout, linked, hashes, paths):
|
||||
def module_glue(layout, linked, hashes, paths, version):
|
||||
"""C for build/<version>/modules.c: the layout table, the linked modules' function tables (dispatch.h), the
|
||||
expected ExeFS hashes and the default game paths (host.h)."""
|
||||
expected ExeFS hashes, the game version and the default game paths (host.h)."""
|
||||
out = [f'// Module glue written by tools/build_game.py; do not edit. {GENERATED_MARKER}',
|
||||
'#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
|
||||
for name, base, size in layout:
|
||||
@@ -213,7 +213,8 @@ def module_glue(layout, linked, hashes, paths):
|
||||
out += ['};', f'const int rt_exefs_sha256_count = {len(hashes)};', '']
|
||||
for name in ('exefs', 'romfs', 'profile'):
|
||||
out.append(f'const char rt_default_{name}[] = {c_string(paths[name].as_posix())};')
|
||||
out.append('')
|
||||
number = int(DISPLAY_VERSIONS[version].replace('.', '')) if version in DISPLAY_VERSIONS else 0 # 1.4.3 -> 143
|
||||
out += [f'const int rt_game_version = {number};', '']
|
||||
return '\n'.join(out)
|
||||
|
||||
|
||||
@@ -584,7 +585,9 @@ def main():
|
||||
help="totk.exe's default ExeFS directory (default build/extracted/<version>/exefs)")
|
||||
ap.add_argument('--romfs', type=Path, help="totk.exe's default RomFS directory with romfs.bin and index.json "
|
||||
'(default build/extracted/<version>/romfs)')
|
||||
ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile)")
|
||||
ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile for v121, "
|
||||
"build/profile-<version> otherwise, so trying another version leaves "
|
||||
"the main profile alone; saves load on 1.2.1 and 1.4.3 alike)")
|
||||
ap.add_argument('--toolchain', type=Path, help='llvm-mingw directory (with bin/clang.exe) to build with instead of '
|
||||
'MSYS2 UCRT64 clang')
|
||||
ap.add_argument('--ninja', type=Path, help='ninja.exe to build with (default: MSYS2 UCRT64, else bin/ninja.exe of the '
|
||||
@@ -604,7 +607,8 @@ def main():
|
||||
extracted = ROOT / 'build' / 'extracted' / args.version
|
||||
paths = {'exefs': (args.exefs or extracted / 'exefs').resolve(),
|
||||
'romfs': (args.romfs or extracted / 'romfs').resolve(),
|
||||
'profile': (args.profile or ROOT / 'build' / 'profile').resolve()}
|
||||
'profile': (args.profile or ROOT / 'build' / ('profile' if args.version == 'v121'
|
||||
else f'profile-{args.version}')).resolve()}
|
||||
expected = args.expected
|
||||
if not expected:
|
||||
if args.version not in DISPLAY_VERSIONS:
|
||||
@@ -645,7 +649,7 @@ def main():
|
||||
missing = [n for n in names if n not in hashes]
|
||||
if missing:
|
||||
sys.exit(f'{expected}: no exefs_sha256 for {missing}')
|
||||
write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths))
|
||||
write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths, args.version))
|
||||
if (paths['romfs'] / 'index.json').is_file():
|
||||
count, size, dt = romfs_index.write(paths['romfs'], out)
|
||||
print(f'romfs index: {count} files, {mib(size)}, {dt:.1f} s')
|
||||
|
||||
@@ -41,6 +41,7 @@ from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import check_content # noqa: E402
|
||||
import shader_list # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/, generated/, toolchain/ and refs/ are junctions in a worktree
|
||||
@@ -95,7 +96,7 @@ LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'}
|
||||
|
||||
# tools/ files the installer steps run (and what they import), copied from the repo.
|
||||
TOOLS_FILES = ['build_game.py', 'check_content.py', 'extract.py', 'make_romfs_patches.py', 'prepare_shaders.py',
|
||||
'romfs_index.py', 'shader_census.py']
|
||||
'romfs_index.py', 'shader_census.py', 'shader_list.py']
|
||||
TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes
|
||||
RUNTIME_SUFFIXES = {'.h', '.inc', '.ico'} # .ico: the installer window icon
|
||||
|
||||
@@ -343,9 +344,8 @@ def build_sources(out):
|
||||
for d in TOOLS_DIRS:
|
||||
copy_tree(ROOT / 'tools' / d, out / 'tools' / d, lambda r: '__pycache__' in r or r.endswith('.md'))
|
||||
(out / 'data').mkdir(exist_ok=True)
|
||||
text = subprocess.run(['git', '-C', MAIN_ROOT / 'refs' / 'MacNativeGames', 'show',
|
||||
f'{MNG_COMMIT}:games/totk/assets/shader_list.tsv'], check=True, capture_output=True).stdout
|
||||
(out / 'data' / 'shader_list.tsv').write_bytes(text)
|
||||
text = shader_list.history(MAIN_ROOT / 'refs' / 'MacNativeGames', MNG_COMMIT)
|
||||
(out / 'data' / 'shader_list.tsv').write_text(text, encoding='utf-8', newline='\n')
|
||||
copy_tree(ROOT / 'installer', out / 'installer', lambda r: '__pycache__' in r or r.startswith('launcher/'))
|
||||
shutil.copy2(ROOT / 'totk.ini.example', out / 'totk.ini.example')
|
||||
shutil.copy2(ROOT / 'THIRD_PARTY_NOTICES.md', out / 'THIRD_PARTY_NOTICES.md')
|
||||
|
||||
@@ -1,17 +1,18 @@
|
||||
# Adapted from MacNativeGames games/totk/scripts/make_romfs_patches.py @ ebe621d (MIT, Copyright (c) 2026
|
||||
# MacNativeGames contributors).
|
||||
"""Build the 60 fps RomFS overlay for TotK 1.2.1 (runtime/romfs.c, TOTK_ROMFS_PATCHES).
|
||||
"""Build the 60 fps RomFS overlay for TotK 1.2.1 or 1.4.3 (runtime/romfs.c, TOTK_ROMFS_PATCHES).
|
||||
|
||||
UI/LayoutArchive/Common.Product.110.Nin_NX_NVN.blarc.zs: the weapon / shield / material quick menus capture the game
|
||||
UI/LayoutArchive/Common.Product.<110 or 140>.Nin_NX_NVN.blarc.zs: the weapon / shield / material quick menus capture the game
|
||||
picture for their background when the panes C_CaptureBG_00, C_CaptureClearBG_00 and P_CaptureClearBG_00 become
|
||||
visible at animation frame 1.0 (anim/ShortCutBG_00_In.bflan), AutoBuilder the same with C_CaptureBG_00
|
||||
(anim/AutoBuilder_00_In.bflan). UI animations advance 0.5 frames per game frame at 60 fps, so the capture lands after
|
||||
the world stopped drawing and the background can be black. The visibility keys move to frame 0.0: the live world is
|
||||
captured at once (upstream checked weapon, shield, material and ability wheel at 60 fps, and 30 fps).
|
||||
The game-over flow fix of the same upstream script is 1.0.0 only (1.2.1 runs DmF_SY_GameOver.110) and is left out.
|
||||
The game-over flow fix of the same upstream script is 1.0.0 only (1.2.1 and 1.4.3 run DmF_SY_GameOver.110) and is
|
||||
left out.
|
||||
|
||||
Input: build/extracted/v121/romfs (romfs.bin and index.json, verified by tools/extract.py). Output: <out>/<path> and
|
||||
<out>/manifest.tsv (path, size, SHA-256 of the overlay, SHA-256 of the original). Run with .venv-py314.
|
||||
Input: build/extracted/<version>/romfs (romfs.bin and index.json, verified by tools/extract.py). Output:
|
||||
build/<version>/romfs-patches/<path> by default, and <out>/manifest.tsv (path, size, SHA-256 of the overlay, SHA-256 of the original). Run with .venv-py314.
|
||||
"""
|
||||
import argparse
|
||||
import compression.zstd as zstd
|
||||
@@ -22,9 +23,12 @@ from pathlib import Path
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--romfs', type=Path, default=root / 'build/extracted/v121/romfs')
|
||||
ap.add_argument('--out', type=Path, default=root / 'build/v121/romfs-patches')
|
||||
ap.add_argument('--version', choices=('v121', 'v143'), default='v121')
|
||||
ap.add_argument('--romfs', type=Path, help='default build/extracted/<version>/romfs')
|
||||
ap.add_argument('--out', type=Path, help='default build/<version>/romfs-patches')
|
||||
args = ap.parse_args()
|
||||
args.romfs = args.romfs or root / 'build/extracted' / args.version / 'romfs'
|
||||
args.out = args.out or root / 'build' / args.version / 'romfs-patches'
|
||||
index = json.loads((args.romfs / 'index.json').read_text())
|
||||
assert index['status'] == 'pass'
|
||||
files = {f['path']: f for f in index['files']}
|
||||
@@ -73,7 +77,8 @@ def flvi_keys(data, base):
|
||||
return out
|
||||
|
||||
|
||||
ARCHIVE = 'UI/LayoutArchive/Common.Product.110.Nin_NX_NVN.blarc.zs'
|
||||
COMMON = {'v121': '110', 'v143': '140'} # the version in the name of each game's Common archive
|
||||
ARCHIVE = f'UI/LayoutArchive/Common.Product.{COMMON[args.version]}.Nin_NX_NVN.blarc.zs'
|
||||
MEMBERS = {
|
||||
'anim/ShortCutBG_00_In.bflan': ('C_CaptureBG_00', 'C_CaptureClearBG_00', 'P_CaptureClearBG_00'),
|
||||
'anim/AutoBuilder_00_In.bflan': ('C_CaptureBG_00',),
|
||||
|
||||
+11
-7
@@ -1,16 +1,16 @@
|
||||
"""Package a built totk.exe into a folder that runs from anywhere: build/v121-portable by default.
|
||||
"""Package a built totk.exe into a folder that runs from anywhere: build/<version>-portable by default.
|
||||
|
||||
The folder holds the exe, its DLLs, the shader translator (NativeAOT: no .NET), the RomFS index and patches, the
|
||||
extracted ExeFS (the game code the exe loads), the shared fonts, the prepared shader and pipeline files. There is
|
||||
no config file: totk.exe finds the exefs, romfs, profile and shared-fonts folders next to itself. Without --with-romfs
|
||||
the 16 GB RomFS image stays out; copy build/extracted/v121/romfs/romfs.bin to <folder>/romfs/ or set TOTK_ROMFS
|
||||
the 16 GB RomFS image stays out; copy build/extracted/<version>/romfs/romfs.bin to <folder>/romfs/ or set TOTK_ROMFS
|
||||
(a path, relative ones to the exe's folder).
|
||||
|
||||
The folder contains game code and data derived from the game: for your own machines only, never publish or share it.
|
||||
Saves go to <folder>/profile.
|
||||
|
||||
.venv/Scripts/python tools/package_build.py [--build build/v121] [--exe PATH/totk.exe] [--out build/v121-portable]
|
||||
[--with-romfs] [--no-shader-cache]
|
||||
.venv/Scripts/python tools/package_build.py [--version v121] [--build build/v121] [--exe PATH/totk.exe]
|
||||
[--out build/v121-portable] [--with-romfs] [--no-shader-cache]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
@@ -48,14 +48,18 @@ def size_of(path):
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--build', type=Path, default=ROOT / 'build' / 'v121', help='build dir with the prepared state')
|
||||
ap.add_argument('--version', default='v121', help='game version (v121 or v143): the defaults below follow it')
|
||||
ap.add_argument('--build', type=Path, help='build dir with the prepared state (default build/<version>)')
|
||||
ap.add_argument('--exe', type=Path, help='totk.exe to package (default: <build>/totk.exe); its DLLs are taken from --build')
|
||||
ap.add_argument('--xlate', type=Path, default=ROOT / 'build' / 'xlate' / 'totk_xlate.exe')
|
||||
ap.add_argument('--extracted', type=Path, default=ROOT / 'build' / 'extracted' / 'v121')
|
||||
ap.add_argument('--out', type=Path, default=ROOT / 'build' / 'v121-portable')
|
||||
ap.add_argument('--extracted', type=Path, help='default build/extracted/<version>')
|
||||
ap.add_argument('--out', type=Path, help='default build/<version>-portable')
|
||||
ap.add_argument('--with-romfs', action='store_true', help='also copy romfs.bin (about 16 GB)')
|
||||
ap.add_argument('--no-shader-cache', action='store_true', help='leave out shader-cache (about 2 GB): translates again')
|
||||
a = ap.parse_args()
|
||||
a.build = a.build or ROOT / 'build' / a.version
|
||||
a.extracted = a.extracted or ROOT / 'build' / 'extracted' / a.version
|
||||
a.out = a.out or ROOT / 'build' / f'{a.version}-portable'
|
||||
|
||||
build, out = a.build.resolve(), a.out.resolve()
|
||||
exe = (a.exe or build / 'totk.exe').resolve()
|
||||
|
||||
+23
-15
@@ -1,24 +1,26 @@
|
||||
"""Prepare the shader variants of the MacNativeGames shared shader list for a build, once, before play.
|
||||
"""Prepare the default variant of every game shader and the variants of the MacNativeGames shared shader list for a
|
||||
build, once, before play.
|
||||
|
||||
The list (refs/MacNativeGames games/totk/assets/shader_list.tsv, MIT; docs/shader-list.md there) names shader variants
|
||||
players met: a shader key (sha256 of the NVN control section and code, as tools/shader_census.py names shaders) and
|
||||
the translator answers it was translated for. This tool reads the shaders from the extracted RomFS and ExeFS
|
||||
(tools/shader_census.py inventory), keeps the list's variants whose shader is in this game version, and writes them
|
||||
next to the build's totk.exe in the record layouts of runtime/gfx_shader.c ("answers met in play"):
|
||||
(tools/shader_census.py inventory), keeps the list's variants whose shader is in this game version, adds each shader's
|
||||
default variant (every texture slot a 2D texture, float vertex inputs and colour outputs, as MacNativeGames' seed
|
||||
translates every shader for its default bind state: prepare_shader_seed.py all_default), and writes them next to the
|
||||
build's totk.exe in the record layouts of runtime/gfx_shader.c ("answers met in play"):
|
||||
|
||||
seed-programs.bin "TOTKSPRG": stage, length, c1 length, runtime code hash, control section, code, c1
|
||||
seed-answers.bin "TOTKSANS": stage, code hash, input id (zero: the runtime computes it), answers, priority
|
||||
|
||||
Priority (the pad byte): 0 shaders the list's warm rows flag as title/menu (t), 1 other warm rows (S, C, R), 2 the
|
||||
rest. Then it runs `totk.exe --prepare-shaders`, which translates every record into the build's shader cache, writes
|
||||
rest; a default variant takes its shader's priority. Then it runs `totk.exe --prepare-shaders`, which translates every record into the build's shader cache, writes
|
||||
shader-prepared.bin (one record per variant, in priority order) and builds every variant's stage library or compute
|
||||
pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "stage library prebuild"). About 5 min on
|
||||
a 12-core machine (inventory 1, translation 3, library builds 0.6) and 1.3 GB of shader cache; the seed files can be
|
||||
deleted afterwards.
|
||||
pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "stage library prebuild"). The seed files
|
||||
can be deleted afterwards.
|
||||
|
||||
Game-derived output stays in the build directory (ignored). The list is read at preparation time from tools/data/shader_list.tsv when that file exists (the installer archive
|
||||
bundles it; it stays out of the repository), else from refs/ through git; nothing of it is copied into the
|
||||
repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14:
|
||||
bundles it; it stays out of the repository), else from refs/ through git with the rows its history dropped
|
||||
(tools/shader_list.py); nothing of it is copied into the repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14:
|
||||
|
||||
.venv-py314\\Scripts\\python tools\\prepare_shaders.py build\\v121 [--version v121] [--list <tsv>] [--no-run]
|
||||
[--romfs DIR --exefs DIR] # instead of build\\extracted\\<version>
|
||||
@@ -36,9 +38,9 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
while not (ROOT / 'tools' / 'shader_census.py').is_file() and ROOT != ROOT.parent:
|
||||
ROOT = ROOT.parent
|
||||
sys.path.insert(0, str(ROOT / 'tools'))
|
||||
import shader_list # noqa: E402
|
||||
from shader_census import inventory # noqa: E402
|
||||
|
||||
LIST_REF = ('origin/HEAD', 'games/totk/assets/shader_list.tsv')
|
||||
STAGES = {'vs': 0, 'ps': 4, 'cs': 5} # runtime/gfx.h GFX_STAGE_*
|
||||
TEX = [None, '1d', '1darray', 'buffer', '2d', '2darray', '2dms', '2dmsarray', '3d', 'cube', 'cubearray']
|
||||
FMT = [None, 'R8Unorm', 'R8Snorm', 'R8Uint', 'R8Sint', 'R16Float', 'R16Unorm', 'R16Snorm', 'R16Uint', 'R16Sint',
|
||||
@@ -48,6 +50,7 @@ FMT = [None, 'R8Unorm', 'R8Snorm', 'R8Uint', 'R8Sint', 'R16Float', 'R16Unorm', '
|
||||
'R16G16B16A16Uint', 'R16G16B16A16Sint', 'R32G32B32A32Float', 'R32G32B32A32Uint', 'R32G32B32A32Sint',
|
||||
'R10G10B10A2Unorm', 'R10G10B10A2Uint', 'R11G11B10Float'] # gfx_shader.c fmt_names
|
||||
TYPES = {'float': 0, 'sint': 1, 'uint': 2}
|
||||
DEFAULT = {'tex': {str(slot): '2d' for slot in range(32)}} # answers outside a shader's use mask are dropped
|
||||
CONTROL = 0x880
|
||||
ANS_REC = 4 + 32 + 32 + 147 + 1
|
||||
|
||||
@@ -62,8 +65,7 @@ def read_list(path):
|
||||
return Path(path).read_text(encoding='utf-8')
|
||||
if BUNDLED_LIST.is_file():
|
||||
return BUNDLED_LIST.read_text(encoding='utf-8')
|
||||
return subprocess.run(['git', '-C', str(ROOT / 'refs' / 'MacNativeGames'), 'show', ':'.join(LIST_REF)],
|
||||
check=True, capture_output=True, encoding='utf-8').stdout
|
||||
return shader_list.history(ROOT / 'refs' / 'MacNativeGames')
|
||||
|
||||
|
||||
def parse_list(text):
|
||||
@@ -131,7 +133,9 @@ def main():
|
||||
extracted = ROOT / 'build' / 'extracted' / a.version
|
||||
shaders, _ = inventory(a.romfs or extracted / 'romfs', a.exefs or extracted / 'exefs')
|
||||
t1 = time.monotonic()
|
||||
progs, recs, skipped = {}, [], {'not in this version': 0, 'stage': 0}
|
||||
progs, recs, seen, skipped = {}, [], set(), {'not in this version': 0, 'stage': 0}
|
||||
listed = len(variants)
|
||||
variants += [(key, DEFAULT) for key in sorted(shaders)]
|
||||
for key, deps in variants:
|
||||
s = shaders.get(key)
|
||||
if not s:
|
||||
@@ -148,7 +152,11 @@ def main():
|
||||
h = code_hash(stage, control, code, length, c1)
|
||||
progs[key] = (h, struct.pack('<III', stage, length, len(c1)) + h + control + code[:length] + c1)
|
||||
h = progs[key][0]
|
||||
rec = struct.pack('<I', stage) + h + bytes(32) + answers(deps, stage, control) + bytes([prio.get(key, 2)])
|
||||
packed = answers(deps, stage, control)
|
||||
if (h, packed) in seen: # a list row of the default bind state
|
||||
continue
|
||||
seen.add((h, packed))
|
||||
rec = struct.pack('<I', stage) + h + bytes(32) + packed + bytes([prio.get(key, 2)])
|
||||
assert len(rec) == ANS_REC
|
||||
recs.append((prio.get(key, 2), rec))
|
||||
recs.sort(key=lambda r: r[0]) # stable: list order within a priority
|
||||
@@ -162,7 +170,7 @@ def main():
|
||||
for _, rec in recs:
|
||||
f.write(rec)
|
||||
by = [sum(1 for p, _ in recs if p == i) for i in range(3)]
|
||||
print(f'{len(variants)} listed variants: {len(recs)} of {len(progs)} shaders in {a.version} '
|
||||
print(f'{listed} listed variants and {len(shaders)} default ones: {len(recs)} of {len(progs)} shaders in {a.version} '
|
||||
f'(priority t {by[0]}, warm {by[1]}, rest {by[2]}), skipped {skipped}; inventory {t1 - t0:.1f} s, '
|
||||
f'seed written in {time.monotonic() - t0:.1f} s', flush=True)
|
||||
if a.no_run:
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
"""The MacNativeGames shared shader list (refs/MacNativeGames games/totk/assets/shader_list.tsv, MIT; docs/shader-list.md
|
||||
there) as our tools read it: every row of one revision, plus the V rows of every older revision that it dropped.
|
||||
|
||||
MacNativeGames prunes V rows its own runtime no longer needs: default variants, which its seed translates for every
|
||||
shader (1.4.3 on 2026-10-22, 1.0.0 to 1.2.1 on 2026-10-25), and variants that differ only in vertex input or colour
|
||||
output types (2026-10-26). Our answers (runtime/gfx_shader.c) still tell those apart, so a dropped row names a variant
|
||||
our play meets: at 1340bee the list alone has 134 rows of 1.4.3 shaders, its history 24,839, and of 1.2.1 6,771 and
|
||||
25,238. A row no draw asks for costs one translation. No copy of the list goes into the repository.
|
||||
"""
|
||||
import subprocess
|
||||
|
||||
PATH = 'games/totk/assets/shader_list.tsv'
|
||||
|
||||
|
||||
def git(repo, *args):
|
||||
return subprocess.run(['git', '-C', str(repo), *args], check=True, capture_output=True, encoding='utf-8').stdout
|
||||
|
||||
|
||||
def history(repo, rev='origin/HEAD'):
|
||||
"""rev's list, then each V row of an older revision that no newer one has, newest first."""
|
||||
lines = git(repo, 'show', f'{rev}:{PATH}').splitlines()
|
||||
seen = set(lines)
|
||||
for h in git(repo, 'log', '--format=%H', rev, '--', PATH).split()[1:]:
|
||||
for line in git(repo, 'show', f'{h}:{PATH}').splitlines():
|
||||
if line.startswith('V\t') and line not in seen:
|
||||
seen.add(line)
|
||||
lines.append(line)
|
||||
return '\n'.join(lines) + '\n'
|
||||
Reference in new issue
Block a user