mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-20 04:35:57 +00:00
The MacNativeGames BKTR and title-key changes and our UTF-8 manifest lived only in the maintainer's toolchain folder, so nobody else could build the hactool that extraction and the installer need. The script's exe matches the one in use byte for byte apart from its build time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
156 lines
7.9 KiB
Markdown
156 lines
7.9 KiB
Markdown
# Building from source
|
|
|
|
Windows 10 or later, an x86-64 CPU with AVX2, and a GPU with Vulkan 1.3 and `VK_EXT_external_memory_host`. Allow about
|
|
60 GB for the work folder. Extraction wants about 55 GB free while it runs and keeps about 35 GB afterwards.
|
|
|
|
Players who only want to play can use the [installer](installer.md) instead: it builds the game on their machine
|
|
without this toolchain.
|
|
|
|
## Toolchain
|
|
|
|
Versions are the ones the build was verified with; newer patch releases are fine.
|
|
|
|
| Tool | Version | Install | Used for |
|
|
|---|---|---|---|
|
|
| MSYS2 UCRT64 | clang/lld 23.1.3 | msys2.org installer to `C:\msys64`, packages below | compiling the generated C and the runtime |
|
|
| Visual Studio Build Tools | 2026, C++ workload, Windows SDK 10.0.26100 | winget `Microsoft.VisualStudio.BuildTools` with `--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended` | the linker .NET NativeAOT calls |
|
|
| .NET SDK | 10.0 | winget `Microsoft.DotNet.SDK.10` | the shader translator (`totk_xlate.exe`) |
|
|
| Vulkan SDK | 1.4.363 | winget `KhronosGroup.VulkanSDK` | validation layers, `vulkaninfo` |
|
|
| uv | 0.13 | standalone installer from astral.sh | the two Python environments |
|
|
| hactool | 1.4.0 @ `1d64a83` | `.venv\Scripts\python tools\build_hactool.py`, see below | NCA and update extraction |
|
|
|
|
MSYS2 packages, after `pacman -Syuu` (run it twice; the first pass updates the core and exits):
|
|
|
|
```sh
|
|
pacman -S --needed --noconfirm mingw-w64-ucrt-x86_64-{clang,lld,ninja,gcc,make,pkgconf,sdl3,opus,libvpx,vulkan-devel,\
|
|
vulkan-memory-allocator,vulkan-validation-layers,shaderc,spirv-tools} base-devel git unzip
|
|
```
|
|
|
|
`ffmpeg` from MSYS2 is a GPLv3 build and is never linked; the movies go through libvpx.
|
|
|
|
PATH:
|
|
|
|
- Add `C:\Program Files (x86)\Microsoft Visual Studio\Installer` to the user PATH: NativeAOT needs `vswhere.exe`, or
|
|
the publish fails at the link step (MSB3073).
|
|
- Another `cmake` or `ninja` earlier on PATH (a 32-bit winlibs MinGW, for example) is picked up by accident. The build
|
|
scripts call Ninja from `C:\msys64\ucrt64\bin` themselves; from a shell, put `C:\msys64\ucrt64\bin` first.
|
|
|
|
### hactool
|
|
|
|
`tools/build_hactool.py` fetches SciresM/hactool at `1d64a83` into `toolchain\src\hactool`, applies the two patches in
|
|
`tools/hactool/` and builds a static `toolchain\hactool\hactool.exe` with MSYS2 UCRT64 gcc (a few seconds):
|
|
|
|
- `portable.patch`: the parts of MacNativeGames' `build_hactool()` that run on Windows, the `MNG_TITLE_KEYS` variable and
|
|
the two BKTR fixes in `nca.c` for the subsection table padding and the subsection layout of newer cumulative updates;
|
|
- `unicode.patch`: a manifest with `<activeCodePage>UTF-8</activeCodePage>`, so paths under a non-ASCII user profile
|
|
reach it intact.
|
|
|
|
hactool reads `-k <prod.keys>`; MSYS2 shells set `HOME`, which hides its default key location, so pass `-k` there.
|
|
|
|
## Build
|
|
|
|
From the repository root, in PowerShell.
|
|
|
|
1. Python environments (once). `.venv` runs the lifter and the build; `.venv-py314` runs the extraction scripts, which
|
|
need the stdlib `compression.zstd` of Python 3.14:
|
|
|
|
```powershell
|
|
uv venv --python 3.12 .venv
|
|
uv pip install --python .venv/Scripts/python.exe -r tools/requirements.txt
|
|
uv venv --python 3.14 .venv-py314
|
|
uv pip install --python .venv-py314/Scripts/python.exe -r tools/requirements-py314.txt
|
|
```
|
|
|
|
2. Copy `config.example.toml` to `config.local.toml` and set the paths to your base game, the 1.2.1 update and
|
|
`prod.keys`. `config.local.toml` is ignored by git. The tools read the keys file themselves.
|
|
|
|
3. Extract and verify. This writes `build\extracted\v0` (1.0.0) and `build\extracted\v121` (1.2.1); a hash mismatch
|
|
stops the run:
|
|
|
|
```powershell
|
|
.venv-py314\Scripts\python tools\extract.py
|
|
```
|
|
|
|
4. Shader translator (once; no game files). This publishes `build\xlate\totk_xlate.exe`:
|
|
|
|
```powershell
|
|
.venv\Scripts\python tools\build_xlate.py
|
|
```
|
|
|
|
5. Lift the modules to C, write the RomFS half of the frame-rate patches, then compile and link `build\v121\totk.exe`:
|
|
|
|
```powershell
|
|
.venv\Scripts\python recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt
|
|
.venv-py314\Scripts\python tools\make_romfs_patches.py
|
|
.venv\Scripts\python tools\build_game.py
|
|
```
|
|
|
|
Lifting takes about 15 s, a clean compile about 200 s on 24 threads; later builds recompile only what changed.
|
|
`--patches` builds in the 60 fps and real-time frame step patches, switchable at run time. The link copies
|
|
`SDL3.dll`, `libiconv-2.dll` and `libopus-0.dll` next to the exe.
|
|
|
|
6. Prepare the shaders (once per build folder and translator). It needs `refs\MacNativeGames` (`tools\fetch_refs.sh`)
|
|
for the shared shader list:
|
|
|
|
```powershell
|
|
.venv-py314\Scripts\python tools\prepare_shaders.py build\v121
|
|
```
|
|
|
|
Without this step the first launch compiles shaders and pipelines during play and stutters until the caches fill.
|
|
The AMD driver keeps its pipeline cache per exe path and holds a limited amount: preparing another build folder can
|
|
evict this one's. `totk.exe --warm-shaders` rebuilds it from the shader cache.
|
|
|
|
7. Run. Keep the window in the foreground; an unfocused window stalls the game.
|
|
|
|
```powershell
|
|
build\v121\totk.exe
|
|
```
|
|
|
|
Settings live in `totk.ini` next to `totk.exe` ([totk.ini.example](../totk.ini.example) lists them). Saves go to
|
|
`build\profile`.
|
|
|
|
### Portable folder
|
|
|
|
`.venv\Scripts\python tools\package_build.py` writes `build\v121-portable`, which runs on another of your machines
|
|
without MSYS2, Python or .NET. `romfs.bin` stays out unless you pass `--with-romfs`; otherwise put it in `romfs\` next to
|
|
the exe or point `TOTK_ROMFS` at its folder. Run `totk.exe --prepare-shaders` once in the copied folder. The folder
|
|
holds code and data derived from your game: for your own machines only.
|
|
|
|
### Game version 1.4.3
|
|
|
|
Add the 1.4.3 update to `updates` in `config.local.toml`; `extract.py` writes `build\extracted\v143`. Then the same
|
|
commands with `v143`: `gen.py --exefs build\extracted\v143\exefs --out generated\v143 --patches recomp\patches-v143.txt`,
|
|
`make_romfs_patches.py --version v143` and `build_game.py --version v143`.
|
|
|
|
## Checks
|
|
|
|
Before a commit:
|
|
|
|
- `python tools/check_content.py --staged`: nothing game-derived is staged. CI runs the same check;
|
|
`git config core.hooksPath .githooks` runs it before each local commit.
|
|
- `build\v121\totk.exe --selftest`: layout, dispatch table and host-function registry, without game files.
|
|
|
|
The lifter (`recomp/`) and the CPU headers (`runtime/cpu.h`, `runtime/simd.h`) are checked against Unicorn. Every lifter
|
|
change passes these, all run with `.venv`:
|
|
|
|
| Test | Checks | Game files |
|
|
|---|---|---|
|
|
| `tests\diff\lift.py` | instructions sampled from the extracted modules; `--coverage` lists every word the lifter rejects or traps | yes |
|
|
| `tests\diff\simd.py` | the whole FP/SIMD encoding space | no |
|
|
| `tests\diff\atomic.py` | exclusive load/store sequences and ARMv8.1 atomics | no |
|
|
| `tests\diff\func.py` | whole leaf functions as the generator emits them, loops and jump tables included | yes |
|
|
| `tests\excl_mt.py` | the exclusive monitor helpers under contention and ABA, on several threads | no |
|
|
| `tests\jumptables_check.py` | `recomp/jumptables.py` against a Unicorn oracle; must report 0 disagreements, on 1.4.3 too | yes |
|
|
|
|
Runtime probes, each a mode of `totk.exe` (the list is at the top of `runtime/main.c`): `--probe-kernel`,
|
|
`--probe-vkexec`, `--probe-nvn-tex` and `--probe-shader` cover most runtime changes.
|
|
|
|
Shaders: `.venv-py314\Scripts\python tools\shader_census.py --version v121 --translate` extracts every Maxwell shader
|
|
to `build\v121\shaders\` and runs it through the translator and `spirv-val`; failure groups go to `failures.txt`.
|
|
`tools\xlate_smoke.py` checks the translator alone, without game files.
|
|
|
|
## Reverse engineering (optional)
|
|
|
|
- Ghidra 12.1.2 with Adubbz's Ghidra-Switch-Loader in `toolchain\ghidra\` (JDK 21) opens the game modules (NSO).
|
|
- `tools\fetch_refs.sh` clones the upstream projects this port builds on into `refs\`, read-only.
|