diff --git a/.claude/skills/run-game/SKILL.md b/.claude/skills/run-game/SKILL.md index 2efc4122..a216440e 100644 --- a/.claude/skills/run-game/SKILL.md +++ b/.claude/skills/run-game/SKILL.md @@ -22,7 +22,7 @@ prints `: rc=`. The game window opens behind the user's windows witho | `user-save-2026-10-21-monsters` | Moblins and Zonai constructs attack at once; use with `--script tools/input/fight.txt` | | `user-save-2026-10-21` | Hyrule field facing the castle, rainy; distant sky over the castle | | `user-save-2026-10-23-pillars` | Hyrule field by standing stone pillars, about 17:40, a Zonai construct beside Link; use with `--script tools/input/backrun.txt` (runs backward from 1550: phantom rocks, a one-frame glitch at about 1605) | -| `user-save-2026-10-24-brightbloom` | Cave, about 20:40: a Brightbloom plant at the lower left (its bulb floats beside the stem, `build/v121-bloom/NOTES.md`), teal Moblin and Bokoblin ahead or Zonai constructs (runs differ) | +| `user-save-2026-10-24-brightbloom` | Cave, about 20:40: a Brightbloom plant at the lower left (its bulb sits in its cup; it floated beside the stem before ddd0ce8), teal Moblin and Bokoblin ahead or Zonai constructs (runs differ) | | `user-save-2026-10-22-island` | Sky island edge; use with `--script tools/input/island.txt` (runs off, skydives, opens the paraglider with B at frame 1975; X does nothing in a dive) | `--frames` counts presents. Pick `1500 + the frames you need in-world`. diff --git a/AGENTS.md b/AGENTS.md index fb714a5b..24f78f25 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,7 +46,8 @@ Setup and verify commands: [docs/toolchain-windows.md](docs/toolchain-windows.md - Game code build (`.venv`): `recomp\gen.py --exefs build\extracted\v121\exefs --out generated\v121 --patches recomp\patches-v121.txt` writes the C (about 15 s). The patch table holds the 60 fps and real-time frame step code patches, switchable at run time (`recomp/patches.py`); `.venv-py314\Scripts\python tools\make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. `tools\build_game.py` compiles and links `build\v121\totk.exe` with Ninja (about 200 s clean, incremental after). `recomp/layout.py` alone decides module bases. Jump tables are resolved by `recomp/jumptables.py`; after changing it, `tests\jumptables_check.py` must report 0 disagreements with its Unicorn oracle. - Shader census (`.venv-py314`): `tools\shader_census.py --version v121 [--translate]` extracts every Maxwell shader to `build/v121/shaders/` and runs it through `build\xlate\totk_xlate.exe` and `spirv-val`; failure groups go to `failures.txt`. `tools\xlate_smoke.py` checks the translator alone, without game files. - Shader preparation (`.venv-py314`, once per build dir and translator): `tools\prepare_shaders.py build\v121` translates the variants of the MacNativeGames shared shader list (read from `refs/` at run time) into the build's shader cache and builds their stage libraries into the driver cache (`totk.exe --prepare-shaders`, about 5 min, 1.3 GB); play then translates and compiles only variants the list lacks. -- The plan by stage and the proposed fixes with their evidence: [docs/backlog.md](docs/backlog.md). +- Open work with its evidence: [docs/backlog.md](docs/backlog.md). +- Installer (`installer/`, `tools/make_release.py`), which builds the game on a player's machine: [docs/installer.md](docs/installer.md). - Running, measuring or frame-dumping the game, save backups, task worktrees and their builds: [docs/agent-playbook.md](docs/agent-playbook.md). - Project skills (`.claude/skills/`): `private-build` before changing runtime/ or tools/ code and building totk.exe (one git worktree per task); `run-game` to run a build into gameplay from a save. - hactool: `toolchain/hactool/hactool.exe`. From an MSYS2 shell pass `-k ` explicitly; the shell's `HOME` hides the default key location. diff --git a/README.md b/README.md index 4023ff25..40768b0e 100644 --- a/README.md +++ b/README.md @@ -12,54 +12,23 @@ > > Extraction, the generated C and `totk.exe` happen on your machine and stay out of this repository. The program you build contains the game, so it is for your own use. Do not share it. -The Switch code is lifted from ARM64 to C ahead of time, compiled with Clang, and linked with a host that answers the system calls. NVN, the Switch graphics API, is implemented on Vulkan. Maxwell shaders are translated to SPIR-V the first time the game uses them, then read from a cache. +The Switch code is lifted from ARM64 to C ahead of time, compiled with Clang, and linked with a host that answers the system calls. NVN, the Switch graphics API, is implemented on Vulkan. Maxwell shaders are translated to SPIR-V the first time the game uses them, then read from a cache. The design is described in [docs/architecture.md](docs/architecture.md). ## Status -This is an early build. Version 1.2.1 starts, shows the title screen, loads saves and runs in the world at 60 fps, with keyboard or gamepad input and with sound. Graphics, stability and speed are still being worked on: visual bugs and crashes are expected. You build it from source; nothing is packaged for download. +This is an early build: version 1.2.1 starts, loads saves and runs in the world at 60 fps with keyboard, mouse or gamepad input and with sound, and it has been tested on AMD, NVIDIA and Intel GPUs. Visual bugs and crashes are expected; the known ones are listed in [docs/backlog.md](docs/backlog.md). No release has been published yet, and macOS, Linux, other game versions and an in-game settings menu are not supported yet. -It is developed and measured on one machine (Ryzen 9 7900X, Radeon RX 9070). NVIDIA and older AMD GPUs are expected to work but are barely tested; [docs/tester-guide.md](docs/tester-guide.md) says what to send back. - -To report a problem, open an issue and follow [CONTRIBUTING.md](CONTRIBUTING.md). Include the GPU and driver, the log, and the steps that get there. Screenshots are fine. - -### Known problems - -- For one frame, a rock or pillar can appear in the wrong place, or nearby objects vanish, most often while running or falling from a sky island. The leading suspect is thread ordering the game expects from the Switch scheduler; see [docs/backlog.md](docs/backlog.md). -- Without `tools\prepare_shaders.py` (build step 6) the first launch compiles shaders and pipelines and stutters until the caches fill. With it, a fight on a cleared driver cache had 1 slow frame instead of 37. -- Newly streamed areas still have single slow frames while their textures upload. -- Far terrain can turn flat gray or brown for a few frames after Link is knocked down. - -### Not in yet - -- macOS and Linux -- Running 1.0.0, 1.1.2 or 1.4.3 (the extractor recognises those updates; the host is built for 1.2.1) -- Button remapping -- A package that builds the game code on a player's machine. Today you compile it yourself +To report a problem, open an issue and follow [CONTRIBUTING.md](CONTRIBUTING.md). Testers: [docs/tester-guide.md](docs/tester-guide.md) says what to send back. ## Features -**Runs natively** -- **This is not emulation.** The 1.2.1 `main`, `sdk` and `subsdk0` modules (about 96 MB of image, a little under 300,000 functions) are compiled into `totk.exe`. -- Title screen, save loading and play, including the overworld, combat and the map. -- 60 fps with the game running at real-time speed: MacNativeGames' 60 fps and frame-step patches are built in and can be switched off (`TOTK_FPS60=0`, `TOTK_REALTIME=0`). -- F11 switches between the window and borderless fullscreen. -- Keyboard and the first SDL3 gamepad, merged into one controller. Buttons are taken by position, so the south face button is B. The key map is in [docs/controls.md](docs/controls.md). Keys count only while the game window is in front. -- Sound through the game's own audio renderer, played with SDL3, including Opus voice. -- Language from the Windows locale, or from `TOTK_LOCALE`. A locale the game does not ship becomes `en-US`. `TOTK_LANG` sets a Switch code directly (`en-US`, `ja`, `zh-Hant`, …). -- Docked mode by default, which reports the boost performance configuration. `TOTK_OPERATION_MODE=handheld` reports the normal one. -- A resizable window that opens at 1920×1080. The game's picture is scaled to the client area. -- Presentation uses mailbox when the driver offers it, and vsync (FIFO) otherwise. `TOTK_VK_PRESENT=fifo` or `immediate` selects those. -- A shader cache and a Vulkan pipeline cache. The first launch translates and compiles; the next launch reuses both. -- Saves under `build/profile`, created on first use. - -## How it works - -The design and the research it rests on are in [docs/architecture.md](docs/architecture.md). - -1. **Input.** Your base game and the 1.2.1 update are decrypted and unpacked locally with your own `prod.keys`. File hashes are checked against known retail dumps. -2. **CPU.** `main`, `sdk` and `subsdk0` are lifted to C, one function per guest function. NEON and floating point are lowered to C that follows ARM rounding. Guest addresses are host addresses. Jump tables are resolved by a symbolic pass in the generator. -3. **Host.** Threads, memory, the RomFS, saves, time, account, language and the audio renderer are implemented on Windows. Every guest thread is a host thread, but only three run game code at once: the host models the Switch's three application cores and its priority preemption, because the game keeps per-core data without locks and orders some work by thread priority alone. Input and audio output go through SDL3. -4. **Graphics.** The NVN front end records what the game submits. A Vulkan 1.3 back end replays it, uploading the memory the CPU wrote and writing GPU results back when the CPU reads them. The Ryujinx Maxwell translator, built as `totk_xlate.exe`, turns shaders into SPIR-V. Pipelines are linked from precompiled stages on first use (`VK_EXT_graphics_pipeline_library`) and optimised in the background. +- Native code, not emulation: the 1.2.1 `main`, `sdk` and `subsdk0` modules are compiled into `totk.exe`. +- 60 fps with the game running at real-time speed, using MacNativeGames' frame-rate patches (`TOTK_FPS60=0` and `TOTK_REALTIME=0` switch them off). +- Keyboard, mouse and the first gamepad, merged into one controller, with the mouse on the camera and every button remappable: [docs/controls.md](docs/controls.md). +- Rumble with adjustable strength, and motion controls on gamepads that have a gyro. +- Sound through the game's own audio renderer, including voice. +- Language from the Windows locale, docked mode by default, a resizable window, and F11 for borderless fullscreen. +- Shader and pipeline caches, so later launches reuse what the first one compiled. ## How this project is made @@ -70,7 +39,7 @@ Most of the code and documentation here is written by AI coding agents (Claude C - Players and testers: [controls](docs/controls.md), [tester guide](docs/tester-guide.md). - How it works: [architecture](docs/architecture.md), with research notes on the upstream projects, Maxwell GPU behaviour and what other ports learned in [docs/research/](docs/research/). - What is being worked on: [backlog](docs/backlog.md). -- Building: [toolchain-windows.md](docs/toolchain-windows.md), [toolchain-msys2-python.md](docs/toolchain-msys2-python.md). +- Building: [toolchain-windows.md](docs/toolchain-windows.md), [toolchain-msys2-python.md](docs/toolchain-msys2-python.md), and the [installer](docs/installer.md) that builds the game on a player's machine. - Coding agents: [AGENTS.md](AGENTS.md) and [docs/agent-playbook.md](docs/agent-playbook.md). ## Building from source @@ -116,21 +85,23 @@ From the repository root, in PowerShell: .venv\Scripts\python tools\build_game.py ``` - `--patches` builds in the 60 fps and frame-step patches; `make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. The generator takes well under a minute. A clean compile takes a few minutes and is incremental after that. The link copies `SDL3.dll`, `libiconv-2.dll` and `libopus-0.dll` next to the exe. `vulkan-1.dll` comes from Windows. + `--patches` builds in the 60 fps and frame-step patches; `make_romfs_patches.py` writes their RomFS half to `build\v121\romfs-patches`. The link copies `SDL3.dll`, `libiconv-2.dll` and `libopus-0.dll` next to the exe. -6. Prepare the shaders (once per build; about 5 minutes and 1.3 GB). It needs `refs\MacNativeGames` (`tools\fetch_refs.sh`) for the shared shader list: +6. Prepare the shaders (once per build). It needs `refs\MacNativeGames` (`tools\fetch_refs.sh`) for the shared shader list: ```powershell .venv-py314\Scripts\python tools\prepare_shaders.py build\v121 ``` -7. Package a portable folder (optional; about 2.4 GB without the RomFS): + Without this step the first launch compiles shaders and pipelines during play and stutters until the caches fill. + +7. Package a portable folder (optional): ```powershell .venv\Scripts\python tools\package_build.py ``` - This writes `build\v121-portable` with `totk.exe`, its DLLs, the shader translator (a native exe, no .NET), the ExeFS, the RomFS index and patches, the shared fonts and the shader and pipeline caches. Copy the folder to another machine of yours; it needs no MSYS2, Python or .NET, only Vulkan 1.3 drivers. `romfs.bin` stays out (16 GB): `--with-romfs` copies it, or put it in `romfs\` next to the exe, or set `TOTK_ROMFS` (also settable in `totk.ini`) to its folder. Saves go to `profile\` in the folder (`TOTK_PROFILE_DIR` moves it, `TOTK_EXEFS` moves the ExeFS). The folder holds game code and data derived from your game: it is for your own machines only, never share it. The driver keeps its pipeline library cache per exe path, so after copying run `totk.exe --prepare-shaders` once in the new folder (about a minute; it needs no Python), or the first fights stutter. + This writes `build\v121-portable`, which runs on another of your machines without MSYS2, Python or .NET. `romfs.bin` stays out unless you pass `--with-romfs`; otherwise put it in `romfs\` next to the exe or point `TOTK_ROMFS` at its folder. After copying, run `totk.exe --prepare-shaders` once in the new folder, because the driver keeps its pipeline cache per exe path. The folder holds game code and data derived from your game: it is for your own machines only, never share it. 8. Run: @@ -138,23 +109,11 @@ From the repository root, in PowerShell: build\v121\totk.exe ``` - Keep the window in the foreground; an unfocused window stalls the game. Close the window to quit. The first launch spends time on shaders and pipelines. Later launches read `build\v121\shader-cache` and `build\v121\pipeline-cache.bin`. + Keep the window in the foreground; an unfocused window stalls the game. Close the window to quit. -`build\v121\totk.exe --selftest` checks the layout and the host tables and needs no game files. `--frames N` exits after N presented frames; `--frames N --shot out.png` writes that frame. The other modes are development probes (`--boot`, `--probe-kernel`, `--probe-fs`, and the rest printed by the exe when the arguments are wrong). +`build\v121\totk.exe --selftest` checks the build without game files. -Where files live, all under the repository and all ignored by git: - -- Extracted code and data: `build\extracted\v121\exefs` and `build\extracted\v121\romfs` -- Generated C: `generated\v121` -- The program: `build\v121\totk.exe`, with the three DLLs beside it -- Saves, temp and cache: `build\profile` (`--profile` on `totk.exe` or on `tools\build_game.py` points elsewhere) -- Shader cache: `build\v121\shader-cache` (`TOTK_SHADER_CACHE` overrides the directory) -- Pipeline cache: `build\v121\pipeline-cache.bin` (`TOTK_PIPELINE_CACHE` overrides the file) -- Translator: `build\xlate\totk_xlate.exe` (`TOTK_XLATE` overrides the path) - -Settings live in `totk.ini` next to `totk.exe`; copy [totk.ini.example](totk.ini.example) and edit it. A `TOTK_*` environment variable overrides the file (`TOTK_FPS60=0` beats `fps60 = 1`), and the startup log lists the file read and each override. The `TOTK_*` names in this README are also the ini keys without the prefix. - -`TOTK_AUDIO=none` mixes without opening a device. `TOTK_GPU=null` runs the graphics front end with no GPU, for tests. +Settings live in `totk.ini` next to `totk.exe`; copy [totk.ini.example](totk.ini.example), which lists them, and edit it. A `TOTK_*` environment variable of the same name overrides the file (`TOTK_FPS60=0` beats `fps60 = 1`). Saves go to `build\profile`, created on first use. Everything the build writes stays under `build\` and `generated\`, which git ignores. ## Contributing @@ -171,6 +130,7 @@ This project stands on two MIT codebases and the projects they came from. Code t - **[hactool](https://github.com/SciresM/hactool)** by SciresM (ISC) extracts NCA and update containers. - **[SDL 3](https://libsdl.org)** (zlib) is the gamepad and the audio output. - **[libopus](https://opus-codec.org)** (BSD) decodes voice. +- **[libvpx](https://chromium.googlesource.com/webm/libvpx)** (BSD) decodes the cutscene movies. - **[Vulkan Memory Allocator](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator)** (MIT) allocates GPU memory. - **[deko3d](https://github.com/devkitPro/deko3d)** (zlib) is the source of the block-linear image layout, via MacNativeGames, and a reference for Maxwell command semantics. - **[SavageWindRecomp](https://github.com/Revivedx/SavageWindRecomp)** (MIT), a static recompilation of *Breath of the Wild*, is the model for preemption at indirect calls and a reference for frame pacing and the platform split. @@ -195,6 +155,7 @@ Third-party code keeps its own license. The full list, with the commit each copy | SDL 3 | 3.4.18 | zlib | `SDL3.dll` next to `totk.exe` (gamepad, audio) | | libiconv | 1.19 | LGPL 2.1+ | `libiconv-2.dll`, unmodified, a dependency of SDL 3 | | libopus | 1.6.1 | BSD 3-clause | `libopus-0.dll` next to `totk.exe` (voice) | +| libvpx | 1.17.0 | BSD 3-clause, with the WebM patent grant | linked statically into `totk.exe` (movies) | | Vulkan Memory Allocator | 3.4.0 | MIT | compiled into `totk.exe` | | Ryujinx shader translator | 1.3.2 | MIT | `build\xlate\totk_xlate.exe`, a separate process | | GCC runtime (`libstdc++`) | 16.2.0 | GPL 3.0 with the GCC Runtime Library Exception | linked into `totk.exe` | diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index e8fc765f..ad327b53 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -6,20 +6,20 @@ Code copied or adapted from another project is listed here with its source, the |---|---|---|---| | MacNativeGames extraction scripts and verification hashes | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11`, Copyright (c) 2026 MacNativeGames contributors | MIT | `tools/vendor/macnativegames/` (unmodified), `tools/data/expected-v*.json` | | LAS Recomp lifter, code generator, FP/SIMD lowering, CPU headers and differential tests | https://github.com/peiikai/LASRecomp @ `da35b83a9611d046a9a681aae668d7085954d37f`, Copyright (c) 2026 Peiikai | MIT | `recomp/lifter.py`, `recomp/simd_c.py`, `recomp/nso.py`, `recomp/gen.py`, `recomp/module.py`, `tests/jumptables_check.py`, `runtime/cpu.h`, `runtime/simd.h`, `runtime/simd_fp.h`, `tests/diff/` | -| LAS Recomp host runtime: NSO loader, host-function table, guest-space reservation and re-execution, guest thread start | https://github.com/peiikai/LASRecomp @ `da35b83a9611d046a9a681aae668d7085954d37f`, Copyright (c) 2026 Peiikai | MIT | `runtime/loader.c`, `runtime/hostfn.c`, `runtime/memory.c`; M3 tasks T2 and T6: `runtime/kernel.c`, `runtime/boot.c`; M4 T2: `runtime/vk_device.c`, `runtime/gfx_window.c`, M4 T5a: `runtime/tex_astc.c` (`runtime/nvn/astc.c`; its decode ported to GLSL in `runtime/shaders/astc_decode.comp`, compiled into `runtime/astc_decode_spv.h`, and its tables copied into `runtime/vk_astc.c`), `runtime/vk_formats.c` (format table of `runtime/nvn/vulkan/textures.c`), M4 T5b: `runtime/vk_memory.c` (page write generations of `runtime/nvn/vulkan/exec.c`, deferred destruction of `textures.c`), `runtime/vk_texture.c` (image creation, uploads and samplers of `runtime/nvn/vulkan/textures.c`), M4 T6a: `runtime/vk_sync.c` (timeline submissions and waits of `runtime/nvn/vulkan/exec.c`), M4 T6b: `runtime/vk_pipeline.c` (blend, compare and stencil mappings, vertex format table with its 3-component fallback and pipeline cache file of `runtime/nvn/vulkan/exec.c` and `device.c`), M4 T6c1: `runtime/vk_exec.c` (pass begin and end, scissor clip and attachment clears of `runtime/nvn/vulkan/exec.c`), M4 T6c3: `runtime/vk_exec_xfer.c` (copies of `runtime/nvn/vulkan/exec.c`), M4 T6c2: `runtime/vk_exec_draw.c` (uniform buffer spans, texture and sampler descriptors and dispatch flow of `runtime/nvn/vulkan/exec.c`); in-game menu: `runtime/ui/menu.cpp` (Dear ImGui Vulkan set-up of `runtime/nvn/vulkan/imgui_glue.cpp`) | -| MacNativeGames TotK host runtime: dynamic linking, identity check, module permissions, guest memory and SVCs, threads, IPC, audio renderer, system services, RomFS and save storage, boot sequence | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/runtime/`, `games/totk/scripts/prepare_runtime.py`), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/loader.c`, `runtime/memory.c`; M3 tasks T2-T6: `runtime/kernel.c`, `runtime/svc.c`, `runtime/ipc.c`, `runtime/ipc.h`, `runtime/audren.c`, `runtime/audren.h`, `runtime/audio_mixer.c`, `runtime/audio_effects.c`, `runtime/audio_effects.h`, `runtime/audio_reverb.c`, `runtime/audio_reverb.h` (M5; SDL3 output replaces CoreAudio), `runtime/ov_system.c` (also `shared_font.c`: the BFTTF layout behind nn::pl), `runtime/ov_input.c` (also `hid_feedback.c`: vibration and six-axis handles and calls), `runtime/motion_fusion.c` (six-axis fusion of `motion_fusion.c`), `runtime/ov_media.c` (M5: hardware Opus decoder over libopus), `runtime/swkbd_win.c` (`swkbd_host.c`: ShowKeyboardArg offsets, UTF-8/UTF-16 conversion, disabled-key filter, result codes; Win32 window replaces the macOS text bar), `runtime/ov_gfx.c`, `runtime/romfs.c`, `runtime/romfs_path.h`, `runtime/fs_host.c`, `runtime/fs.h`, `runtime/boot.c`; M4 T2: `runtime/gfx_window.c` | +| LAS Recomp host runtime: NSO loader, host-function table, guest-space reservation and re-execution, guest thread start | https://github.com/peiikai/LASRecomp @ `da35b83a9611d046a9a681aae668d7085954d37f`, Copyright (c) 2026 Peiikai | MIT | `runtime/loader.c`, `runtime/hostfn.c`, `runtime/memory.c`; `runtime/kernel.c`, `runtime/boot.c`; `runtime/vk_device.c`, `runtime/gfx_window.c`, `runtime/tex_astc.c` (`runtime/nvn/astc.c`; its decode ported to GLSL in `runtime/shaders/astc_decode.comp`, compiled into `runtime/astc_decode_spv.h`, and its tables copied into `runtime/vk_astc.c`), `runtime/vk_formats.c` (format table of `runtime/nvn/vulkan/textures.c`), `runtime/vk_memory.c` (page write generations of `runtime/nvn/vulkan/exec.c`, deferred destruction of `textures.c`), `runtime/vk_texture.c` (image creation, uploads and samplers of `runtime/nvn/vulkan/textures.c`), `runtime/vk_sync.c` (timeline submissions and waits of `runtime/nvn/vulkan/exec.c`), `runtime/vk_pipeline.c` (blend, compare and stencil mappings, vertex format table with its 3-component fallback and pipeline cache file of `runtime/nvn/vulkan/exec.c` and `device.c`), `runtime/vk_exec.c` (pass begin and end, scissor clip and attachment clears of `runtime/nvn/vulkan/exec.c`), `runtime/vk_exec_xfer.c` (copies of `runtime/nvn/vulkan/exec.c`), `runtime/vk_exec_draw.c` (uniform buffer spans, texture and sampler descriptors and dispatch flow of `runtime/nvn/vulkan/exec.c`); in-game menu: `runtime/ui/menu.cpp` (Dear ImGui Vulkan set-up of `runtime/nvn/vulkan/imgui_glue.cpp`) | +| MacNativeGames TotK host runtime: dynamic linking, identity check, module permissions, guest memory and SVCs, threads, IPC, audio renderer, system services, RomFS and save storage, boot sequence | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/runtime/`, `games/totk/scripts/prepare_runtime.py`), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/loader.c`, `runtime/memory.c`; `runtime/kernel.c`, `runtime/svc.c`, `runtime/ipc.c`, `runtime/ipc.h`, `runtime/audren.c`, `runtime/audren.h`, `runtime/audio_mixer.c`, `runtime/audio_effects.c`, `runtime/audio_effects.h`, `runtime/audio_reverb.c`, `runtime/audio_reverb.h` (SDL3 output replaces CoreAudio), `runtime/ov_system.c` (also `shared_font.c`: the BFTTF layout behind nn::pl), `runtime/ov_input.c` (also `hid_feedback.c`: vibration and six-axis handles and calls), `runtime/motion_fusion.c` (six-axis fusion of `motion_fusion.c`), `runtime/ov_media.c` (hardware Opus decoder over libopus), `runtime/swkbd_win.c` (`swkbd_host.c`: ShowKeyboardArg offsets, UTF-8/UTF-16 conversion, disabled-key filter, result codes; Win32 window replaces the macOS text bar), `runtime/ov_gfx.c`, `runtime/romfs.c`, `runtime/romfs_path.h`, `runtime/fs_host.c`, `runtime/fs.h`, `runtime/boot.c`; `runtime/gfx_window.c` | | MacNativeGames TotK frame-rate patches: 60 fps present-interval controller, real-time frame step and its present sampling, hair and cloth clock, trampoline slow-motion check (instruction words and 1.2.1 sites), player position chain; quick-menu capture background RomFS overlay | https://github.com/ly05010419/MacNativeGames @ `ebe621de99bfc48f3c42f2d2aa73e04fbcc6eae2` (`games/totk/runtime/game_patches.c`, `game_version.h`, `world_survey.c`, `games/totk/scripts/make_romfs_patches.py`), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/frame_rate.c`, `recomp/patches-v121.txt`, `tools/make_romfs_patches.py` | | MacNativeGames TotK virtual-core preemption: asynchronous preemption of an owner in top-level game code, priority inheritance through guest mutexes, the SDK's preemption-disable protocol (TLR+0x100/+0x102, SVC 0x36), the core-wait diagnostics | https://github.com/ly05010419/MacNativeGames @ `33e8fa83052ea1ae09a3583137008e9797e12950` (`games/totk/runtime/threads.c`, commits 8bf0893 and 19730cd), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/kernel.c` (SuspendThread and a register-saving stub replace SIGUSR2), `runtime/hostfn.c`, `runtime/svc.c` (host depth) | | MacNativeGames RomFS index builder (`TOTKRFS1` layout) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/scripts/prepare_romfs_backend.py`), Copyright (c) 2026 MacNativeGames contributors | MIT | `tools/romfs_index.py` | -| d2r-mobile host thread environment (TLR, TLS, thread object), the origin of MacNativeGames' thread layer; d2r-mobile NVN host object layouts, command record format and NVN front end, as adapted for TotK in MacNativeGames' `d2r_gfx` copy | https://github.com/jhaizhou-ops/d2r-mobile @ `53e7c5bb9ca3ea53e2160005be8fbe920cd9cb53` (`dm/host/thread.c`, `dm/host/hle_thread.c`); the NVN layouts and front end via https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/nvn_obj.h`, the two types of `nvn_tex.h`, `hle_nvn.c`, `totk_nvn_ext.c`, `nvn_impl.h`, `nvn_hmap.h`, `nvn_pool_index.h`, the request checks and mapping semantics of `totk_vpool.c`, the generation stamps of `nvn_dirty.c`, texture mirroring and texel copies of `nvn_mtl.m`; licence in `games/totk/third_party/d2r_gfx/LICENSE`), Copyright (c) 2026 jhaizhou | MIT | M3 task T2: `runtime/kernel.c`; M4 task T1: `runtime/nvn_obj.h`; M4 task T3c: `runtime/nvn_api.h`, `runtime/nvn_api_core.c`, `runtime/nvn_hmap.h`, `runtime/nvn_pool_index.h`; M4 task T3d: `runtime/nvn_api_obj.c` (also `nvn_pace.h`); M4 task T3e: `runtime/nvn_api_cmd.c`, `runtime/nvn_ext.c`; M4 task T5b: `runtime/vk_memory.c`, `runtime/vk_texture.c`; M4 task T6a: `runtime/vk_sync.c` (side effects written after GPU completion, `hle_nvn.c`); M4 task T6b: `runtime/vk_pipeline.c` (pipeline key and state mapping of `nvn_mtl.m` `pso_key_fill` / `pso_desc`, its `vfmt` vertex formats and the missing-varying fix-up of `msl_varyings`); M4 task T6c1: `runtime/vk_exec.c`, `runtime/vk_exec.h` (op loop, pure-state ops and state defaults of `nvn_mtl.m` `mtl_exec` / `state_op` / `state_reset`); M4 task T6c3: `runtime/vk_exec_xfer.c` (copy, clear, event, counter and conditional-render semantics of `nvn_mtl.m` `copy_t2b` / `copy_b2t` / `copy_t2t` / `clear_tex` / `cond_eval`); M4 task T6c2: `runtime/vk_exec_draw.c` (bind ops, draw and dispatch flow, vertex streams of `nvn_mtl.m` `state_op` / `bind_vertex_streams` / `draw_inner` / `dispatch`) | -| nxnative NVN front end (d2r-mobile's successor): texture view depth-stencil mode bits, stencil-view rule, `Downsample` and indirect draw and dispatch records | https://github.com/seedds/nxnative @ `d5b380b3a80495609c2e751bb994911a638935ad` (`dm/gfx/nvn/hle_nvn.c`, `nvn_obj.h`, `nvn_mtl.m`), Copyright (c) 2026 jhaizhou | MIT | M5: `runtime/nvn_obj.h`, `runtime/nvn_ext.c`, `runtime/nvn_api_cmd.c`, `runtime/vk_exec_draw.c` | +| d2r-mobile host thread environment (TLR, TLS, thread object), the origin of MacNativeGames' thread layer; d2r-mobile NVN host object layouts, command record format and NVN front end, as adapted for TotK in MacNativeGames' `d2r_gfx` copy | https://github.com/jhaizhou-ops/d2r-mobile @ `53e7c5bb9ca3ea53e2160005be8fbe920cd9cb53` (`dm/host/thread.c`, `dm/host/hle_thread.c`); the NVN layouts and front end via https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/nvn_obj.h`, the two types of `nvn_tex.h`, `hle_nvn.c`, `totk_nvn_ext.c`, `nvn_impl.h`, `nvn_hmap.h`, `nvn_pool_index.h`, the request checks and mapping semantics of `totk_vpool.c`, the generation stamps of `nvn_dirty.c`, texture mirroring and texel copies of `nvn_mtl.m`; licence in `games/totk/third_party/d2r_gfx/LICENSE`), Copyright (c) 2026 jhaizhou | MIT | `runtime/kernel.c`; `runtime/nvn_obj.h`; `runtime/nvn_api.h`, `runtime/nvn_api_core.c`, `runtime/nvn_hmap.h`, `runtime/nvn_pool_index.h`; `runtime/nvn_api_obj.c` (also `nvn_pace.h`); `runtime/nvn_api_cmd.c`, `runtime/nvn_ext.c`; `runtime/vk_memory.c`, `runtime/vk_texture.c`; `runtime/vk_sync.c` (side effects written after GPU completion, `hle_nvn.c`); `runtime/vk_pipeline.c` (pipeline key and state mapping of `nvn_mtl.m` `pso_key_fill` / `pso_desc`, its `vfmt` vertex formats and the missing-varying fix-up of `msl_varyings`); `runtime/vk_exec.c`, `runtime/vk_exec.h` (op loop, pure-state ops and state defaults of `nvn_mtl.m` `mtl_exec` / `state_op` / `state_reset`); `runtime/vk_exec_xfer.c` (copy, clear, event, counter and conditional-render semantics of `nvn_mtl.m` `copy_t2b` / `copy_b2t` / `copy_t2t` / `clear_tex` / `cond_eval`); `runtime/vk_exec_draw.c` (bind ops, draw and dispatch flow, vertex streams of `nvn_mtl.m` `state_op` / `bind_vertex_streams` / `draw_inner` / `dispatch`) | +| nxnative NVN front end (d2r-mobile's successor): texture view depth-stencil mode bits, stencil-view rule, `Downsample` and indirect draw and dispatch records | https://github.com/seedds/nxnative @ `d5b380b3a80495609c2e751bb994911a638935ad` (`dm/gfx/nvn/hle_nvn.c`, `nvn_obj.h`, `nvn_mtl.m`), Copyright (c) 2026 jhaizhou | MIT | `runtime/nvn_obj.h`, `runtime/nvn_ext.c`, `runtime/nvn_api_cmd.c`, `runtime/vk_exec_draw.c` | | NVN type definitions reconstructed from Darksiders II DWARF information, with MacNativeGames' TotK size annotations | h1k421, via https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/include/nvn.h`), Copyright (c) 2020 h1k421 | MIT | `runtime/nvn.h` (unmodified) | -| Ryujinx.Audio voice state and effects, as ported in MacNativeGames' audio renderer | https://github.com/Ryujinx/Ryujinx, via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/runtime/audio_renderer.c`, `audio_effects.c`, `audio_reverb.c`; licence text in `games/totk/third_party/licenses/Ryujinx-Audio.txt`), Copyright (c) Ryujinx Team and Contributors | MIT | M3 task T3: `runtime/audren.c` | -| REA agent skill `reverse-engineer-anything` | https://github.com/morluto/rea @ `799e8265b254e225dc1dabdd25e56f0ab7f86bd3`, Copyright (c) 2026 morluto | MIT | `.claude/skills/reverse-engineer-anything/`, `.agents/skills/reverse-engineer-anything/` (pinned in `skills-lock.json`) | -| Ryujinx shader translator 1.3.2 (`Ryujinx.Graphics.Shader`, `Spv.Generator`, `Ryujinx.Common`) with MacNativeGames' four optimizer patches and `totk_xlate` tool | https://github.com/Ryujinx/Ryujinx, via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`vendor/ryujinx`, `games/totk/tools/totk_xlate/`), Copyright (c) Ryujinx Team and Contributors; tool Copyright (c) 2026 jhaizhou | MIT | `tools/vendor/ryujinx/` (unmodified), `tools/xlate/`, `runtime/gfx_shader.c` (job layout from `tools/shader_census.py`; M4 T6b: c2 handle ranges of `totk_xlate/Service.cs` for the reflection), `runtime/vk_pipeline.c` (M4 T6b: the translator's set/binding and support-buffer conventions) | -| deko3d block-linear image layout, as ported in MacNativeGames' `nvn_tex.c` (Apple parts dropped, layout math unchanged; licence text in `games/totk/third_party/d2r_gfx/LICENSE.deko3d`, reproduced in the file header) | https://github.com/devkitPro/deko3d (`source/dk_image.cpp`), via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/nvn_tex.c`, `nvn_tex.h`), Copyright (C) 2018-2020 fincs | zlib | M4 task T3b: `runtime/nvn_tex.c`, `runtime/nvn_tex.h` | -| MacNativeGames texture auto-layout goldens (40 original-SDK size and block-height cases) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/tests/texture_auto_layout/probe.c`), Copyright (c) 2026 jhaizhou | MIT | M4 task T3b: `runtime/nvn_tex_probe.c`; T3f: `runtime/nvn_probe.c` (four of the cases, through the procs) | -| MacNativeGames block-linear deswizzler (address function; copy loop split out as `tex_bl_copy`) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/tex/nvn_deswizzle.c`), Copyright (c) 2026 jhaizhou | MIT | M4 task T5a: `runtime/tex_deswizzle.c`, `runtime/tex.h` | +| Ryujinx.Audio voice state and effects, as ported in MacNativeGames' audio renderer | https://github.com/Ryujinx/Ryujinx, via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/runtime/audio_renderer.c`, `audio_effects.c`, `audio_reverb.c`; licence text in `games/totk/third_party/licenses/Ryujinx-Audio.txt`), Copyright (c) Ryujinx Team and Contributors | MIT | `runtime/audren.c` | +| REA agent skill `reverse-engineer-anything` | https://github.com/morluto/rea @ `799e8265b254e225dc1dabdd25e56f0ab7f86bd3`, Copyright (c) 2026 morluto | MIT | `.claude/skills/reverse-engineer-anything/`, `.agents/skills/reverse-engineer-anything/` (pinned in `.claude/skills/skills-lock.json`) | +| Ryujinx shader translator 1.3.2 (`Ryujinx.Graphics.Shader`, `Spv.Generator`, `Ryujinx.Common`) with MacNativeGames' four optimizer patches and `totk_xlate` tool | https://github.com/Ryujinx/Ryujinx, via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`vendor/ryujinx`, `games/totk/tools/totk_xlate/`), Copyright (c) Ryujinx Team and Contributors; tool Copyright (c) 2026 jhaizhou | MIT | `tools/vendor/ryujinx/` (unmodified), `tools/xlate/`, `runtime/gfx_shader.c` (job layout from `tools/shader_census.py`; c2 handle ranges of `totk_xlate/Service.cs` for the reflection), `runtime/vk_pipeline.c` (the translator's set/binding and support-buffer conventions) | +| deko3d block-linear image layout, as ported in MacNativeGames' `nvn_tex.c` (Apple parts dropped, layout math unchanged; licence text in `games/totk/third_party/d2r_gfx/LICENSE.deko3d`, reproduced in the file header) | https://github.com/devkitPro/deko3d (`source/dk_image.cpp`), via MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/nvn/nvn_tex.c`, `nvn_tex.h`), Copyright (C) 2018-2020 fincs | zlib | `runtime/nvn_tex.c`, `runtime/nvn_tex.h` | +| MacNativeGames texture auto-layout goldens (40 original-SDK size and block-height cases) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/tests/texture_auto_layout/probe.c`), Copyright (c) 2026 jhaizhou | MIT | `runtime/nvn_tex_probe.c`; `runtime/nvn_probe.c` (four of the cases, through the procs) | +| MacNativeGames block-linear deswizzler (address function; copy loop split out as `tex_bl_copy`) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`games/totk/third_party/d2r_gfx/gfx/tex/nvn_deswizzle.c`), Copyright (c) 2026 jhaizhou | MIT | `runtime/tex_deswizzle.c`, `runtime/tex.h` | | SDL 3.4.18, dynamic link, `SDL3.dll` copied next to `totk.exe`. Developer builds use the MSYS2 build (`mingw-w64-ucrt-x86_64-sdl3`, which imports `libiconv-2.dll`, LGPL-2.1, unmodified, copied beside it); the installer release ships the upstream release `SDL3-3.4.18-win32-x64.zip`, which imports no libiconv | https://libsdl.org, Copyright (c) 1997-2026 Sam Lantinga | zlib | `runtime/input_win.c` (gamepad input, rumble, gyro and accelerometer) | | libopus (MSYS2 `mingw-w64-ucrt-x86_64-opus`), dynamic link, `libopus-0.dll` copied next to `totk.exe` | https://opus-codec.org, Copyright (c) 2001-2023 Xiph.Org, Skype Limited, Octasic, Jean-Marc Valin, Timothy B. Terriberry, CSIRO, Gregory Maxwell, Mark Borgerding, Erik de Castro Lopo, Mozilla, Amazon | BSD-3-Clause | `runtime/ov_media.c` (nn::codec hardware Opus decoder) | | Dear ImGui 1.91.9 (core and Vulkan back end, unmodified), as vendored by LAS Recomp | https://github.com/ocornut/imgui, via https://github.com/peiikai/LASRecomp @ `da35b83a9611d046a9a681aae668d7085954d37f` (`third_party/imgui/`), Copyright (c) 2014-2025 Omar Cornut | MIT | `runtime/third_party/imgui/` (compiled into `totk.exe` for the in-game menu, `runtime/ui/`) | diff --git a/docs/agent-playbook.md b/docs/agent-playbook.md index f20cee66..9eed6083 100644 --- a/docs/agent-playbook.md +++ b/docs/agent-playbook.md @@ -8,8 +8,7 @@ Read this before you run the game, take a frame dump or measure performance. Thi Start, build, hand back and clean up a task with the `private-build` skill (`.claude/skills/private-build/SKILL.md`): a git worktree on `task/` and a build dir `build/v121-`. The pipeline cache, `pipeline-keys.bin`, `shader-cache`, `shader-programs.bin` and `shader-answers.bin` live next to each `totk.exe`, so a task build starts cold: seed it by copying `build/v121`'s five before timing anything. A build seeded -from an older one stalls on pipeline and shader builds, and its slow frames say nothing about the change. (Until 2026-10-22 every task build wrote into -`build/v121`'s caches, and task builds broke each other's runs.) +from an older one stalls on pipeline and shader builds, and its slow frames say nothing about the change. ## Running @@ -39,7 +38,7 @@ other `totk.exe` or `ninja.exe` runs. The log's first and last lines record the - `TOTK_PROFILE= TOTK_PROFILE_FRAMES=lo-hi`: sampling profiler (`runtime/prof_win.c`), read with `tools/prof_report.py `. With `TOTK_PROFILE_RENDER=1` it samples the render thread only and keeps call chains: the report adds inclusive time per function and the exe callers of the driver and system DLLs. -- `tools/regress.py --base build/v121-m4 [--routes fight,rock,island,field] [--repeat 3] [--cold] [--validation] +- `tools/regress.py [--base ] [--routes fight,rock,island,field] [--repeat 3] [--cold] [--validation] [--out dir]`: runs the routes through `run_game.py --quiet`, scores logs and `TOTK_SHOTS` frames (`tools/regress_score.py`, tested by `tests/regress_score_test.py`) and prints median and worst per metric. Run the candidate alone and judge it against the targets: max fps, no slow frames (`perf: slow frame`), no black or flat frames, no new validation errors. @@ -52,7 +51,7 @@ other `totk.exe` or `ninja.exe` runs. The log's first and last lines record the - `fight.txt`: 59.1 fps mean, 60 median, replay 13.6 ms, 0 CPU-side slow frames; the slow frames left are pipeline and shader-translation waits. - `--validation` runs work in the world (1800 frames checked). The layer prints each message kind 10 times, so it - gives no totals; the known messages left are in the backlog correctness queue, item 8. + gives no totals; the known messages left are in the backlog, item 6. ## Frame dumps @@ -63,7 +62,7 @@ copies and clears. `TOTK_DUMP_BUF=addr:size[:off:stride:w:h]` dumps a guest buff - A draw's argument lines come before its pass header. - Pass numbers shift between runs. Find a pass by its draw count instead: the g-buffer pass has about 1990 draws. -RenderDoc cannot capture: the driver hides `VK_EXT_external_memory_host`. +RenderDoc cannot capture: it hides `VK_EXT_external_memory_host` ([controls.md](controls.md), "RenderDoc capture"). ## Other GPUs @@ -72,27 +71,28 @@ Vulkan code queries the feature, extension, format or limit it uses and keeps a it is missing (`vk_pipeline.c` provoking vertex and subgroup size pinning are examples). Never hard-code what this GPU reports: alignments, subgroup size, depth formats, `maxPushDescriptors`. -Numbers from this machine do not carry over. Measured on the user's other machines (2026-10-24/25, logs and analysis -in `build/mem-plan.md`): +Numbers from this machine do not carry over. The user's other machines (the test machines of +[backlog.md](backlog.md)) measured: - GTX 1050 Ti (4 GiB, 214 MiB BAR, no ReBAR; i7 8th-10th gen) and RTX 3090 Ti: GPU time 110-120 ms per frame in the world against 4.3 ms here, because the GPU reads imported host pools over PCIe. The VRAM mirror (cd6f490) brings - the 1050 Ti to 10-11 ms; that machine is now CPU-bound at 25-45 fps (replay 25-30 ms per frame, backlog 0b). + the 1050 Ti to 10-11 ms; that machine is now CPU-bound at 25-45 fps (replay 25-30 ms per frame, backlog item 8). - The same write-watch scan costs about 3x the CPU time it costs here. -- NVIDIA has no ASTC: first texture uploads decode on the CPU, 0.4-0.7 s load frames (backlog 5a.2). +- NVIDIA and AMD desktop GPUs have no ASTC: textures decode in a compute shader (`TOTK_ASTC`). Before that, CPU + decodes made 0.4-0.7 s load frames on the 1050 Ti. - `cannot alias its pool memory` fills its 16-line log limit on NVIDIA and on the RX 9070 alike (1920x1080, 1600x900, - 800x450 render targets); a suspect for the black patches of backlog 0a (`build/alias-plan.md`). + 800x450 render targets); a suspect for the black patches of backlog item 2. -A run on another machine goes through the user: copy `build/release/TotK-perf-log.vbs` next to the `totk.exe` you -hand over. It starts the game with `TOTK_TRACE=perf` and writes `logs\perf--