diff --git a/AGENTS.md b/AGENTS.md index e8e4d628..fe94242d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,6 +27,7 @@ Read the note before working on its area; each cites `file:line` in `refs/`. - Other ports and recompilers: [docs/research/prior-art.md](docs/research/prior-art.md) - Sizes, function and instruction counts, OS touch-points of the 1.2.1 modules: [docs/research/totk-v121-inventory.md](docs/research/totk-v121-inventory.md) - Maxwell GPU semantics our Vulkan path must match, TotK bugs other ports and emulators fixed: [docs/research/maxwell.md](docs/research/maxwell.md) +- Windows host costs (timers, file I/O and flushes, thread sampling, VMA allocation, bit flips, run-to-run spread): read before adding a wait, file access or allocation on a game or render thread, or comparing frame times: [docs/research/windows-host.md](docs/research/windows-host.md) - Symptoms other ports already solved (artifacts, crashes, pacing, scheduling) and whether we have the fix; search it before investigating any symptom: [docs/research/port-lessons.md](docs/research/port-lessons.md) ## Toolchain diff --git a/docs/research/port-lessons.md b/docs/research/port-lessons.md index 7c866b9b..196ff60a 100644 --- a/docs/research/port-lessons.md +++ b/docs/research/port-lessons.md @@ -38,7 +38,7 @@ new commits and `git diff origin/HEAD -- docs` their notes. Add lessons he |---|---|---|---|---| | T1 | GPU-bound scenes run in slow motion; dynamic resolution never drops | The Zelda engine's GPU time is the `TIMESTAMP` report at the end of the frame minus `TIMESTAMP_TOP` at its start, read after the frame fence; the interval controller and dynamic resolution use it. One time per submission made it 0-6 ticks (BotW). Fix: each report gets the GPU start or end of its command segment (`gpu_timestamps segment`, `DM_NVN_GPU_TS`; 04-graphics.md) | open: every report gets the host time at fence completion (`vk_exec_xfer.c` `write_counter`), so the difference is microseconds. Counter word 0 is right (conditional clears need it). 1.2.1's interval controller does not read it: `[x1+0x118]` (`0x8f26c8`) is `GetSystemTick` minus the frame start (`f_70008f24bc`); dynamic resolution not checked | find a reader of the report difference; then `vkCmdWriteTimestamp` (top of pipe for `TIMESTAMP_TOP`) converted to 19.2 MHz ticks with calibrated timestamps | | T2 | frame-rate counter wanders 50-60 | the vsync event and the window acquire each kept a 60 Hz clock and drifted apart; both follow one display tick now (03-bringup.md) | open: `gfx_window.c` `vsync_main` and the pacer of `nvn_api_obj.c` (`pace`, `present_period`) run separate clocks | derive acquire slots from the vsync thread's tick | -| T3 | a 30 Hz thread runs at 27 Hz, slow cutscenes | host sleep overshoot; `sleep_precise` | unknown: high-resolution waitable timer (`kernel.c` `sleep_ns`), overshoot not measured | measure `SleepThread` at 1, 5, 16 and 33 ms | +| T3 | a 30 Hz thread runs at 27 Hz, slow cutscenes | host sleep overshoot; `sleep_precise` | partly: sleeps up to 1 ms spin (`kernel.c` `sleep_ns`; 20 us sleeps took 460-515 us before, [windows-host.md](windows-host.md)); longer ones use the high-resolution waitable timer, 0.3-0.5 ms late | measure `SleepThread` at 5, 16 and 33 ms with `TOTK_TRACE=sched` | ## Open: thread scheduling diff --git a/docs/research/windows-host.md b/docs/research/windows-host.md new file mode 100644 index 00000000..fe483966 --- /dev/null +++ b/docs/research/windows-host.md @@ -0,0 +1,83 @@ +# Windows host behaviour + +What the Windows host does differently from the console and from MacNativeGames' macOS host, measured on machine A +(Ryzen 9 7900X, RX 9070, NVMe) in 2026-10. Each entry: the behaviour, the numbers, where the runtime handles it, and +the rule for new code. Linux equivalents of the fixes: [backlog.md](../backlog.md) item 35. + +## Rules + +- The game threads (guest main, job workers) and the render thread never open, read, write or flush a file, and + never wait for a thread that does. Move file I/O to a background thread and hand results over in memory. +- Short waits spin; waits over 1 ms use the high-resolution timer. +- Every frame-time claim needs three runs per build: single runs of one recording differ by up to three times. + +## Timers and time + +- **Waitable timer overshoot.** A high-resolution waitable timer ends a wait 0.3-0.5 ms late at best: the SDK's + 20 us polling sleeps (main thread and job workers, about 1.5 per frame each) took 460-515 us. `kernel.c` + `sleep_ns` spins on the performance counter up to `SLEEP_SPIN_NS` (1 ms), the virtual core already given up. + macOS `nanosleep` is accurate, so MacNativeGames never had this. Measure with `TOTK_TRACE=sched` ("sleep asked / + took" per thread). +- **A 1 ms timer thread does not help.** A run with a thread waking every 1 ms (the sampler of `TOTK_PROFILE_MAIN`) + had more slow frames than one without, not fewer: no evidence for core parking or C-state wake-up costs here. +- **Counter conversion.** `QueryPerformanceCounter` ticks to ns or to the guest's 19.2 MHz counter as + `(unsigned __int128)t * num / den` compiles to `__udivti3`, a library call; it took a tenth of the guest threads' + running time in a fight, as the scheduler and the guest's counter reads convert many thousands of times a second. + `platform.h` `PlatScale` multiplies by a precomputed integer and 64-bit fraction (branch `task/perf2`). A lazily + set two-field scale shared by threads must be published behind a flag stored after it. + +## Files + +- **Small-file opens stall behind other writers.** Opening a 20-byte file in a 139k-file cache folder takes 0.1 ms + alone (p99 0.3 ms) but took 11-40 ms in play, while the translator, the pipeline cache and the save wrote and + flushed on the same volume. Defender's real-time scan was off; the stall is the file system. `gfx_shader.c` reads + all `.use` masks on a background thread at start (45k files in 12 s). +- **Flushes are expensive.** A save Commit (copy or hard-link 82 files, `FlushFileBuffers` on each file and directory, + two `MoveFileExW` with `MOVEFILE_WRITE_THROUGH`, rotate backups) took 93-135 ms: compare 23-41, hard links 17-31, + copy 0-18, directory flush 22-24, oldest backup deletion 16. `fs_host.c` runs it on a thread; game writes wait for it. +- **Saving a cache holds its lock.** `vkGetPipelineCacheData` plus the file write took about 60 ms; done on the present + thread under the pipeline lock, every 20 s autosave froze the replay. `vk_pipeline.c` saves on its own thread + under a separate lock. +- **A crash leaves zero-filled files.** After bugchecks, thousands of `.spv` and `.res.json` files were zero-filled + (3647 of 21924 once). Validate cached files on load (`gfx_shader.c` checks the `.res.json` shape) instead of + trusting their presence. +- **Unicode paths.** Use the W APIs with UTF-8 to UTF-16 conversion (`fs_wide`); exes that take paths carry an + `activeCodePage` UTF-8 manifest ([installer.md](../installer.md), Constraints). + +## Threads and sampling + +- **Stopping a thread waits for user mode.** `SuspendThread` returns at once, but `GetThreadContext` waits until the + thread has left the kernel; a thread inside a long kernel call (a file open, a driver call) stalls the sampler for + that long. `prof_win.c` records such stops over 2 ms as `tstall`: they name the thread's kernel call. +- **Guard page.** A thread stopped between its `sub rsp` and the stack probe has Rsp inside its guard page; copying + the stack from Rsp raised `STATUS_GUARD_PAGE_VIOLATION` and ended the process. Check `PAGE_GUARD` with + `VirtualQuery` before reading another thread's stack. +- **Idle threads.** `QueryThreadCycleTime` tells without suspending whether a thread ran since the last look; the + sampler repeats an idle thread's last chain instead of stopping it. +- **Three virtual cores are the CPU budget.** TotK keys per-core data by `GetCurrentProcessorNumber` without locks, + so guest threads share three virtual cores whatever the host has. In a fight the guest main thread and five job + workers keep 2.6 of 3 busy and ready threads queue for a core: guest code speed decides the frame rate there. + +## Vulkan and the driver (AMD, Windows) + +- **New memory blocks.** VMA's `vkAllocateMemory` of a 256 MiB device-local block took 8-11 ms (memory type 3: + 11-19 ms, 650 MiB: 23 ms) under VMA's block lock; texture creation on the render thread paid it four times in the + first minute of play while VRAM grew from 3.0 to 4.0 GB. Each allocation over 2 ms is logged by `vk_device.c` + `timed_allocate` (branch `task/perf2`). +- **Driver pipeline cache.** Keyed by the exe path and limited in size; preparing another build dir can evict this + one's libraries ([backlog.md](../backlog.md) item 12, [agent-playbook.md](../agent-playbook.md)). +- `win32u.dll` frames in a host stack are kernel graphics calls (WDDM), usually memory allocation or submission. + +## Measuring + +- **Run-to-run spread.** One recording replayed on near-identical builds gave 1% lows from 31.5 to 50 fps; three runs + of one build agreed within 46-50. Compare medians of three `--quiet` runs, never one run against another. +- **Replays need game time.** With the real-time frame step, input keyed by presents lands at other game times at + another frame rate; recordings key input by the world clock (`frame_rate.c`), and a load of any length no longer + shifts it. The fight still diverges after about 15 game seconds (enemy AI, physics). +- **Bit flips on machine A.** Its RAM flips bits (bugcheck 0x139 and others, several a day): a cached file, a + hard-linked object or a release can carry a flip without an error. Compare two independent builds; objects embed + the toolchain path of the folder they were built in, so compare their `.text`, `.data` and `.rdata`, not whole + files. A file that reads wrong only through the file cache: hash it with `FILE_FLAG_NO_BUFFERING` too. +- **Swapping exes.** `build_game.py` does not relink when only `totk.exe` was replaced by a copy; delete the exe + before rebuilding to restore a build dir.