Add contributor and agent guides for pull requests, running and debugging

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-11-09 14:44:59 +03:00
1 parent f9ed079614
commit bdcfb42c07
6 files changed
+409 -7

No files matched your search

+117
View File
@@ -0,0 +1,117 @@
# Debugging the port: method for agents
A game run costs minutes and a full rebuild costs several more, while reading code costs seconds. Order the work so the
cheap steps decide as much as they can before the game starts. Each step ends on its stated condition.
## Steps
1. **Evidence.** Hold one artifact that shows the symptom: the player's screenshot, log or input recording, a
flight-recorder dump, a frame dump ([running.md](running.md), "Observing"). Done when you can point at the wrong
pixel, line or frame.
2. **Where it reproduces.** OS, GPU vendor and generation, driver, game update, settings, mods, the save. Done when you
know whether your hardware can show the bug at all. NVIDIA bugs do not reproduce on AMD; RDNA1/2 bugs do not
reproduce on RDNA3/4 (another driver branch); a Ryzen 7000 iGPU is RDNA2 and runs that branch. When your hardware
cannot show it, say so and work from the code and the reporter's logs; runs that cannot decide anything are skipped.
3. **Known already?** Search [symptoms.md](symptoms.md), [pr-review-kit/04-knowledge.md](../pr-review-kit/04-knowledge.md)
and [maxwell.md](../maxwell.md) sections 6-7 with the words the player used (black, magenta, flicker, slow motion,
missing, stale, streaks, stuck) and the object (sky, grass, shadow, fog, HUD, weapon). Then the upstream notes:
`refs/nxnative/docs`, `refs/MacNativeGames/docs` and the SavageWindRecomp issue tracker (`tools/fetch_refs.sh`
fetches `refs/`). Done when every hit is either ruled out or is your first hypothesis. A hit on a class we already
fixed: confirm that fix still holds before deriving anything new.
4. **Read the code.** The game's function in Ghidra ([building.md](../building.md), "Reverse engineering") or in
`generated/<ver>/`, and our runtime path in `runtime/`. Done when the code either rules the hypothesis out or names
the exact condition a run must show. A run confirms what the code could not rule out.
5. **The cheapest falsifying check.** An existing switch (a `totk.ini` setting or a `TOTK_*` variable), a dump, a trace
line, a probe. When it refutes the hypothesis, stop: write down what was checked and why it fell, and start again
at step 3 with the next hypothesis. No build, no benchmark for a refuted idea.
6. **Repro red, then green.** The same repro shows the bug before the fix and its absence after. A fix without a
reproduced failure is a guess.
7. **Fix the cause.** A crash that goes away with a feature off points at the bug in that feature; fix the feature.
A fix that one setup needs is gated on that setup, and everyone else keeps today's behaviour.
## Traps
- **One-off failures.** A failure that happens once and not again: rerun before you debug it as a code bug. A crash or
a forced shutdown leaves zero-filled cache files, and memory errors on the host flip bits in files and objects.
- **Every build fails the same way at once.** Suspect the inputs and caches (game files, the OS file cache, shader and
pipeline caches), not the code. Bisecting code is pointless when the old builds fail too.
- **Compile errors in generated C** on a line the generator could not have written are a flipped bit. Regenerate and
compare hashes; `recomp/` is fine.
- **A crash that looks like a race.** Try the save first: some saves crash the same way on a real Switch.
- **Theories about locks and threads.** Read the game's own protocol in Ghidra before running anything. Ten runs on
lock theories once went where one read of the game's double-buffer code would have narrowed it.
- **Numbers from a fast machine.** A change neutral on a fast desktop can matter on a weak laptop or the Steam Deck,
in either direction. Say which hardware your numbers come from.
## Bug classes of a recompiled Switch game
The port runs the game's own code on another OS and another GPU. Most bugs come from places where the host differs
from the console.
### Variable frame step and epsilon-gated timers
The console steps the game by whole 30 Hz units, so a countdown started at a whole value reaches exactly 0. With the
real-time step (`realtime = 1`, the default) the step is a measured ratio, and the last step can leave a remainder of
about 1e-8. Many of the game's updaters step a timer only while it is above `FLT_EPSILON`; a remainder in
(0, `FLT_EPSILON`] is never stepped again. A consumer that tests the same field with `> 0` then sees it active
forever, until a load reinitialises the object. Issue #18 (weapons locked after a Fuse) was this.
Symptom words: stuck until a reload, a teleport or a shrine; locked after an action; works sometimes; only above
30 fps. To check a report: find the field the stuck behaviour reads, then compare the updater's threshold with the
consumer's test. A change to the frame step, the pacer or the fps limit is checked for this class.
### Thread scheduling: Horizon is not a desktop OS
- Horizon runs a woken better-priority thread at once on its core, before the thread that woke it goes on. A host
runs both in parallel, so state can be read one step early or freed while another thread still uses it.
- Threads created with ideal core -2 start pinned to one core on Horizon. The game keys per-core data by
`GetCurrentProcessorNumber` without locks and expects that pinning.
- The game has hundreds of spin loops without a system call. On the console a spinning thread holds one core of three;
on the host it burns a host core and can keep an equal-priority waiter out.
- ARMv8 `stlr` followed by `ldar` is ordered (RCsc). C11 release/acquire lets x86 reorder that store and load. A
two-thread flag handshake that loses a wake points here; lifter changes to barriers and atomics pass
`tests\diff\atomic.py` and `tests\excl_mt.py`.
- A scheduling change is judged by how long a ready, better thread waits for a core it may use, plus the slow-frame
count. A check that made one bug rarer once added 50-75 ms hitches.
### Maxwell behaviour versus Vulkan defaults
The port must do what the Switch GPU does: last-vertex provoking, (0,0,0,1) for unwritten vertex outputs, colour
targets outside the shader's output map keep their texels, the G-buffer pass samples its own targets. The full list
with sources: [maxwell.md](../maxwell.md), section 7.
### Memory shared by the CPU and the GPU
- The game fills textures and buffers with the CPU and samples them on the GPU without the flush a desktop API needs.
Missing uploads show as magenta, brown or stale textures.
- The game reads GPU-rendered textures with the CPU (grass maps, cut and burn masks). Missing write-backs show as
effects that never happen: cut grass stays up, fire does not spread.
- Memory only the GPU writes looks idle to write-watch tracking: a store from a shader leaves no fault and no dirty bit.
- Anything kept per texture, per frame or per object without a free grows host memory until a commit failure after
hours. Watch private memory over a few idle minutes.
### Device lost and GPU hangs
- The `vk: pass marks` lines name the pass in flight. Validation names the misuse when there is one.
- Submits and presents share one queue under one lock. A present from its own thread on a second queue gave device
lost on NVIDIA before Turing in another port.
- Caches from another driver or GPU (pipeline caches, translated shaders) gave black windows after driver updates;
every cache is validated on load and dropped when vendor, device or `pipelineCacheUUID` change.
- A first-time shader stall over 2 s on a slow GPU trips the Windows GPU timeout (display event 4101). Large one-shot
submissions (a boot-time decode of every texture) do the same.
- Point lists from vertex shaders that write no point size are undefined without `VK_KHR_maintenance5`.
### Windows and Wine host costs
What the hot path must not do, with measured costs: [pr-review-kit/02-performance.md](../pr-review-kit/02-performance.md),
"What the hot path must not do". For debugging, also:
- `SuspendThread` returns at once, but `GetThreadContext` waits until the thread leaves the kernel. A sampler or a
debugger stalls for as long as the stopped thread's file open or driver call runs.
- A thread stopped between its `sub rsp` and its stack probe has its stack pointer inside the guard page. Check
`PAGE_GUARD` with `VirtualQuery` before reading another thread's stack.
- Paths go through the W APIs with UTF-8 to UTF-16 conversion. An A API call breaks on a path with non-Latin letters,
such as a Windows user name in Cyrillic or Chinese.
- Wine emulates `GetWriteWatch` with `mprotect` and one fault per first write. Anything relying on cheap OS memory
tricks (write-watch, protection changes, fault handlers) costs more under Proton.
- `win32u.dll` frames in a host stack are kernel graphics calls: memory allocation or submission.
+90
View File
@@ -0,0 +1,90 @@
# Running and observing the game as an agent
How an agent runs a build into gameplay without a person at the controls, captures what the game shows, and reads the
results without flooding its context. Setting names and file formats live in [settings.md](../settings.md) and
[controls.md](../controls.md); this page says which to use when. How to judge a performance change:
[pr-review-kit/02-performance.md](../pr-review-kit/02-performance.md).
## Before the first run
- **A build dir per change.** Build into its own folder (`tools/build_game.py --out build/<ver>-<name>`) so a
comparison has two exes side by side. After every build check the exe's modification time: a failed link leaves the
old `totk.exe` in place and you test old code.
- **Prepared shaders in that folder.** `tools/prepare_shaders.py build/<dir> --version <ver>` ([building.md](../building.md)
step 6). Without it the game compiles shaders during play, and every timing and every "a draw is missing" shot of the
first minutes says nothing about your change. Shader and pipeline caches are not copied between build folders.
- **The same `totk.ini` on both sides** of any comparison. A folder without one runs at the defaults; check the log's
`resolution:` line of each run before comparing them.
- **Saves.** `tools/run_game.py` copies the profile `build/profile`, swaps in a save from `build/save-backup/<name>/`
and deletes the copy afterwards, so the real profile is never written. To make a backup, copy the user folder from
`build/profile/save/` into `build/save-backup/<name>/`. Pick a save that starts where the change acts.
## Running
```powershell
.venv\Scripts\python tools\run_game.py build\<dir> --save <name> --seconds 30 [--script <input.txt>] [--shot out.png] [--env KEY=VALUE] [--quiet] [--tag <name>]
```
- `--seconds` is game time since the world loaded (the log line `frame step: world clock starts` marks the start), at
any frame rate. Measure and script by game time; a frame count at another fps is another scene.
- `--script` replays input ([controls.md](../controls.md), "Scripted input"). A new script starts with the presses of
`tools/input/load_save.txt`, which load the first save from the title. `TOTK_INPUT_RECORD` turns a person's play into
a script; ask the user to record the repro rather than guessing the button sequence.
- `--shot` saves the last frame and builds pipelines synchronously: use it for pictures, never for timing.
- `--quiet` waits until no other `totk.exe` or `ninja.exe` runs. Every timing you report comes from a `--quiet` run.
- The log goes to `<dir>\logs\<tag>.log`; the tool prints `<log>: rc=<code>`.
Run one game at a time: two instances share the GPU's memory and the CPU, and both runs' numbers are void.
Running `totk.exe` yourself: set `TOTK_ASSUME_FOCUS=1` (an unfocused window otherwise pauses the game),
`TOTK_INPUT=none` (the user's keyboard and pad stay out) and `TOTK_BACKGROUND_WINDOW=1`. Stop only the process you
started, by PID: `taskkill /IM totk.exe` also closes the user's game.
**Check that the run reached the world** before trusting it: the world clock line is in the log and the shot shows the
scene you expected. Title menus on a slow boot eat presses, and runs of one script drift apart after 10-15 game seconds
(enemy AI, physics, weather). Two runs are comparable only when shots at the same game second show the same scene.
## Observing
| You need | Use |
|---|---|
| pictures over a time span | `TOTK_SHOTS=w<lo>-<hi>:<step>` (world seconds) and `TOTK_SHOTS_DIR`; 480x270 by default |
| the frames around a hitch or a one-frame glitch | `TOTK_FLIGHT=1`: keeps the last 600 frames with timings and thumbnails, dumps on F8, on a frame over `TOTK_FLIGHT_SPIKE_MS` or at `TOTK_FLIGHT_AT=<frame>`; `python tools/flight_view.py <dump dir>` lists the worst frames and writes a contact sheet |
| every render pass of one frame | `TOTK_DUMP=<frame>` and `TOTK_DUMP_DIR`: a PNG per pass target plus `passes.txt` (draws, bound textures with swizzles, storage buffers, copies, clears) and `images.txt`; `TOTK_DUMP_BUF` adds a guest buffer |
| Vulkan misuse | `TOTK_VK_VALIDATION=1` (needs the Vulkan SDK), run into the world; compare the message kinds with a run without your change. The layer caps repeats, so counts are not totals |
| which pass the GPU was in at a device lost | the `vk: pass marks` lines; on by default on AMD, `TOTK_PASS_CRUMBS=1` elsewhere |
| a crash | the crash report at the end of the log: exception, guest and host addresses, registers |
| a run on a large card that acts like a small one | `TOTK_VRAM_LIMIT_MIB=<MiB>` |
Frame dumps: a draw's argument lines come before its pass header. Pass numbers shift between runs, so find a pass by
its draw count or its targets, not by its number. RenderDoc cannot capture this renderer ([controls.md](../controls.md),
"RenderDoc capture").
## Measuring
| What | How |
|---|---|
| fps, worst frame, render-thread waits, GPU busy time, VRAM, `bound: gpu\|cpu\|pace` | `--env TOTK_TRACE=perf,sched`, then `python tools\perf_summary.py <log> <lo> <hi>` |
| slow frames | count the `perf: slow frame` lines; each names what the render thread was blocked on (pipeline, translation, texture, fence, pacing) |
| the frame's critical path, by thread and state | `TOTK_CRIT=w<lo>-<hi>` and `TOTK_CRIT_OUT=<absolute path>` with `sched` tracing, then `python tools\crit_path.py <file>` |
| where CPU time goes | `TOTK_PROFILE=<path>` and `TOTK_PROFILE_FRAMES=w<lo>-<hi>`, then `python tools\prof_report.py <profile> <totk.map>`; `TOTK_PROFILE_RENDER=1` samples the render thread with call chains |
| a switch on a static scene | a setting set to `ab` with `TOTK_AB=<seconds>` alternates the two sides in blocks of game time; `python tools\ab_summary.py <log> --skip <seconds>`. On a moving scene compare whole runs instead |
A single run proves little: runs of one script on one build spread by a third in their 1% lows. Compare medians of at
least two whole runs per side, on a quiet machine.
## Probes
Each probe is a mode of `totk.exe`, run from the build folder; the list with what each covers is at the top of
`runtime/main.c`. `--selftest` needs no game files. `--probe-kernel`, `--probe-vkexec`, `--probe-nvn-tex` and
`--probe-shader` cover most runtime changes and run in seconds: run them before any game run.
## Reading output without filling the context
- Game logs run to megabytes. `grep` the lines you need, `tail` the end, pipe `perf_summary.py` through `grep SUMMARY`.
- Generated C runs to hundreds of thousands of lines per file. `grep -n` the function, then print its line range.
- Look at screenshots downscaled: paste several into one 1280x720 grid and read the grid once. Write down what it shows
instead of opening it again.
- Send long builds and runs to the background with their output in a file; read the last lines when they end.
- Keep a notes file in the build folder from the first finding: what is ruled out, the leading hypothesis, the next
step, temporary debug edits with file and line. A fresh session continues from it.
+71
View File
@@ -0,0 +1,71 @@
# Symptoms and their usual causes
What a player reports, the causes this port and other Switch ports (nxnative, MacNativeGames, SavageWindRecomp,
LAS Recomp) found behind it, and the first check. Search by the player's words and by the object. Which of these are
already fixed here, and where: [pr-review-kit/04-knowledge.md](../pr-review-kit/04-knowledge.md), "Fixed bug classes".
GPU semantics behind the graphics rows: [maxwell.md](../maxwell.md). The method around this table:
[debugging.md](debugging.md).
## Graphics
| Symptom | Usual causes | First check |
|---|---|---|
| magenta, brown or stale textures; caves or sky wrong | the CPU filled the texture and the GPU never got the upload (no flush on the console) | frame dump: the texture's upload count and mirror state in `passes.txt` |
| effects that never happen: cut grass stays up, fire does not spread | the CPU reads a GPU-rendered texture that was never written back | which texture the game reads after its fence; write-back of that target |
| black scene, black dialog background or loading map | deferred binds read at record time; a 3D target rejected; a texture compare ignoring the channel mapping | frame dump: the first pass whose target goes black |
| lime silhouettes, rectangles in the sky | a target outside the shader's output map overwritten instead of kept | dump: the target's previous contents versus after the pass |
| dark or magenta 16x16 blocks, missing characters | a pass samples its own render targets (G-buffer feedback) | feedback-loop layout and barriers on that pass |
| tile rectangles, wrong data in tiles | storage buffers bound in the wrong slot order | `passes.txt`: storage buffers with their c0 slots |
| streaks on cliffs, characters stretched into streaks | primitive restart ignored; integer vertex inputs against a float format | validation run; the draw's topology and vertex formats in the dump |
| garbage geometry on title or load screens | 8-bit indices widened on the CPU before a copy in the same submission wrote them | whether a copy to the index buffer precedes the draw in one submission |
| black tiles, masks not cleared | clears with a partial channel or stencil mask skipped or done in full | `passes.txt` clears and their masks |
| garbage in feedback effects at first use | new GPU-only images start undefined; the console starts them at zero | clear on creation and compare |
| empty clouds or fog for the whole session | a one-shot draw skipped while its pipeline built, never drawn again | `perf: slow frame` lines naming pipeline waits; the draw's first frame |
| fog that never parts, flicker in tile rows | driver constants (c0: dispatch size, point size, base instance) not filled | the c0 contents the translator expects for that stage |
| no sun shadows, wrong lighting | unwritten vertex outputs read as zero instead of (0,0,0,1) | the shader's output map |
| one flat ambient colour | a 1-layer 2D array sampled as 2D | the view type the shader expects |
| upside-down text | lower-left window origin | origin mode in the log |
| flicker or a texture swapped with a similar one | the newest texture picked at shared storage instead of the one with the matching view | texture lookup by id in the dump |
## Timing and frame pacing
| Symptom | Usual causes | First check |
|---|---|---|
| stuck until a reload or teleport, locked after an action | a timer remainder below `FLT_EPSILON` with the real-time step ([debugging.md](debugging.md)) | `realtime = 0` in `totk.ini`; if the bug is gone, find the timer field |
| slow motion in GPU-heavy scenes, dynamic resolution never drops | the engine's GPU time comes from report timestamps; wrong timestamps make the frame look free | what the engine reads as GPU time and what the report writes |
| frame counter wanders 50-60 | two 60 Hz clocks (vsync event and present acquire) drifting apart | whether both follow one display tick |
| a 30 Hz thread runs at 27 Hz, slow cutscenes | host sleep overshoot: Windows waitable timers wake 0.3-0.5 ms late | `TOTK_TRACE=sched` sleep asked versus took |
| 20 fps after a scheduler change | a yield back-off that starts too early | yield handling against the console's |
| climbing or movement wrong only above some fps | game code that assumes a fixed step count per frame | compare at 30, 60 and the failing rate |
## Threads and crashes
| Symptom | Usual causes | First check |
|---|---|---|
| state read one step early, one frame of inconsistent data | a woken better-priority thread does not run before the waker goes on | sched trace around the handoff |
| per-core data corrupted, rare crash in a container walk | threads the console pins to one core run on any host core | the thread's core mask from its creator; a run on one core |
| rare hang or lost wake | address-arbiter signal values or wake order differ from Horizon | the arbiter path against mesosphere |
| a flag handshake loses a wake | `stlr`/`ldar` ordering lost in x86 release/acquire | the guest code's barrier pattern |
| a host core at 100% while the game waits | game spin loops without system calls | profile: which guest function spins |
| crash on world load | sizes the game reads from the driver (command memory used) differ from the console's | the sizes the game asks for against the console's |
| crash with one save only | the save itself; some crash on a real Switch | the same save on another build and, if possible, on hardware |
## Device lost, hangs, black window
| Symptom | Usual causes | First check |
|---|---|---|
| device lost a few frames in, GPU writes to address 0 (older NVIDIA) | present from another thread or queue | every submit and present on one queue under one lock |
| device lost or garbage on one AMD generation | views the driver branch mishandles (BC block views of 3D textures) | `vk: pass marks`, validation; the formats in the pass |
| device lost on one scene, every run, one vendor | a shader or pass that driver cannot run | pass marks name the pass; skip shaders by hash to bisect |
| GPU timeout after 1-2 min, single frames over 2 s | first-time shader stalls or a huge one-shot submission on a slow GPU | prepared shaders; submission sizes at boot |
| GPU hang at boot with pipeline libraries | point lists without a point size | the shader writes point size or the device clamps it |
| black window after a driver update | caches from another driver | the log line `vk_pipeline: ... written by another device or driver`; delete the caches next to `totk.exe` |
## Memory
| Symptom | Usual causes | First check |
|---|---|---|
| host memory grows in play until a commit failure after hours | state kept per texture init or per frame and never freed | private memory over a few idle minutes |
| multi-GB peaks while travelling | upload staging grows inside one submission | staging bytes per submission |
| out of memory after minutes | dead textures re-entering tables | live images and allocator bytes over time |
| stutter as VRAM fills | new memory blocks allocated on the render thread (8-23 ms each) | profile the render thread over the hitch: `vkAllocateMemory` under the allocator's lock |