totk-pc-recomp/docs/agents/debugging.md

Debugging the port: method for agents

A game run costs minutes and a full rebuild costs several more, while reading code costs seconds. Order the work so the cheap steps decide as much as they can before the game starts. Each step ends on its stated condition.

Steps

1. Evidence. Hold one artifact that shows the symptom: the player's screenshot, log or input recording, a flight-recorder dump, a frame dump (running.md, "Observing"). Done when you can point at the wrong pixel, line or frame.
2. Where it reproduces. OS, GPU vendor and generation, driver, game update, settings, mods, the save. Done when you know whether your hardware can show the bug at all. NVIDIA bugs do not reproduce on AMD; RDNA1/2 bugs do not reproduce on RDNA3/4 (another driver branch); a Ryzen 7000 iGPU is RDNA2 and runs that branch. When your hardware cannot show it, say so and work from the code and the reporter's logs; runs that cannot decide anything are skipped.
3. Known already? Search symptoms.md, pr-review-kit/04-knowledge.md and maxwell.md sections 6-7 with the words the player used (black, magenta, flicker, slow motion, missing, stale, streaks, stuck) and the object (sky, grass, shadow, fog, HUD, weapon). Then the upstream notes: refs/nxnative/docs, refs/MacNativeGames/docs and the SavageWindRecomp issue tracker (tools/fetch_refs.sh fetches refs/). Done when every hit is either ruled out or is your first hypothesis. A hit on a class we already fixed: confirm that fix still holds before deriving anything new.
4. Read the code. The game's function in Ghidra (building.md, "Reverse engineering") or in generated/<ver>/, and our runtime path in runtime/. Done when the code either rules the hypothesis out or names the exact condition a run must show. A run confirms what the code could not rule out.
5. The cheapest falsifying check. An existing switch (a totk.ini setting or a TOTK_* variable), a dump, a trace line, a probe. When it refutes the hypothesis, stop: write down what was checked and why it fell, and start again at step 3 with the next hypothesis. No build, no benchmark for a refuted idea.
6. Repro red, then green. The same repro shows the bug before the fix and its absence after. A fix without a reproduced failure is a guess.
7. Fix the cause. A crash that goes away with a feature off points at the bug in that feature; fix the feature. A fix that one setup needs is gated on that setup, and everyone else keeps today's behaviour.

Traps

• One-off failures. A failure that happens once and not again: rerun before you debug it as a code bug. A crash or a forced shutdown leaves zero-filled cache files, and memory errors on the host flip bits in files and objects.
• Every build fails the same way at once. Suspect the inputs and caches (game files, the OS file cache, shader and pipeline caches), not the code. Bisecting code is pointless when the old builds fail too.
• Compile errors in generated C on a line the generator could not have written are a flipped bit. Regenerate and compare hashes; recomp/ is fine.
• A crash that looks like a race. Try the save first: some saves crash the same way on a real Switch.
• Theories about locks and threads. Read the game's own protocol in Ghidra before running anything. Ten runs on lock theories once went where one read of the game's double-buffer code would have narrowed it.
• Numbers from a fast machine. A change neutral on a fast desktop can matter on a weak laptop or the Steam Deck, in either direction. Say which hardware your numbers come from.

Bug classes of a recompiled Switch game

The port runs the game's own code on another OS and another GPU. Most bugs come from places where the host differs from the console.

Variable frame step and epsilon-gated timers

The console steps the game by whole 30 Hz units, so a countdown started at a whole value reaches exactly 0. With the real-time step (realtime = 1, the default) the step is a measured ratio, and the last step can leave a remainder of about 1e-8. Many of the game's updaters step a timer only while it is above FLT_EPSILON; a remainder in (0, FLT_EPSILON] is never stepped again. A consumer that tests the same field with > 0 then sees it active forever, until a load reinitialises the object. Issue #18 (weapons locked after a Fuse) was this.

Symptom words: stuck until a reload, a teleport or a shrine; locked after an action; works sometimes; only above 30 fps. To check a report: find the field the stuck behaviour reads, then compare the updater's threshold with the consumer's test. A change to the frame step, the pacer or the fps limit is checked for this class.

Thread scheduling: Horizon is not a desktop OS

• Horizon runs a woken better-priority thread at once on its core, before the thread that woke it goes on. A host runs both in parallel, so state can be read one step early or freed while another thread still uses it.
• Threads created with ideal core -2 start pinned to one core on Horizon. The game keys per-core data by GetCurrentProcessorNumber without locks and expects that pinning.
• The game has hundreds of spin loops without a system call. On the console a spinning thread holds one core of three; on the host it burns a host core and can keep an equal-priority waiter out.
• ARMv8 stlr followed by ldar is ordered (RCsc). C11 release/acquire lets x86 reorder that store and load. A two-thread flag handshake that loses a wake points here; lifter changes to barriers and atomics pass tests\diff\atomic.py and tests\excl_mt.py.
• A scheduling change is judged by how long a ready, better thread waits for a core it may use, plus the slow-frame count. A check that made one bug rarer once added 50-75 ms hitches.

Maxwell behaviour versus Vulkan defaults

The port must do what the Switch GPU does: last-vertex provoking, (0,0,0,1) for unwritten vertex outputs, colour targets outside the shader's output map keep their texels, the G-buffer pass samples its own targets. The full list with sources: maxwell.md, section 7.

Memory shared by the CPU and the GPU

• The game fills textures and buffers with the CPU and samples them on the GPU without the flush a desktop API needs. Missing uploads show as magenta, brown or stale textures.
• The game reads GPU-rendered textures with the CPU (grass maps, cut and burn masks). Missing write-backs show as effects that never happen: cut grass stays up, fire does not spread.
• Memory only the GPU writes looks idle to write-watch tracking: a store from a shader leaves no fault and no dirty bit.
• Anything kept per texture, per frame or per object without a free grows host memory until a commit failure after hours. Watch private memory over a few idle minutes.

Device lost and GPU hangs

• The vk: pass marks lines name the pass in flight. Validation names the misuse when there is one.
• Submits and presents share one queue under one lock. A present from its own thread on a second queue gave device lost on NVIDIA before Turing in another port.
• Caches from another driver or GPU (pipeline caches, translated shaders) gave black windows after driver updates; every cache is validated on load and dropped when vendor, device or pipelineCacheUUID change.
• A first-time shader stall over 2 s on a slow GPU trips the Windows GPU timeout (display event 4101). Large one-shot submissions (a boot-time decode of every texture) do the same.
• Point lists from vertex shaders that write no point size are undefined without VK_KHR_maintenance5.

Windows and Wine host costs

What the hot path must not do, with measured costs: pr-review-kit/02-performance.md, "What the hot path must not do". For debugging, also:

• SuspendThread returns at once, but GetThreadContext waits until the thread leaves the kernel. A sampler or a debugger stalls for as long as the stopped thread's file open or driver call runs.
• A thread stopped between its sub rsp and its stack probe has its stack pointer inside the guard page. Check PAGE_GUARD with VirtualQuery before reading another thread's stack.
• Paths go through the W APIs with UTF-8 to UTF-16 conversion. An A API call breaks on a path with non-Latin letters, such as a Windows user name in Cyrillic or Chinese.
• Wine emulates GetWriteWatch with mprotect and one fault per first write. Anything relying on cheap OS memory tricks (write-watch, protection changes, fault handlers) costs more under Proton.
• win32u.dll frames in a host stack are kernel graphics calls: memory allocation or submission.