From a3d876e2d2ee76f8a881c72a3863e2cd96d926c3 Mon Sep 17 00:00:00 2001 From: ProSep <9l0mhe7e@prosep.anonaddy.me> Date: Thu, 5 Nov 2026 01:40:00 +0100 Subject: [PATCH] Let sync_ahead lag the 1.4.3 frame wait too On 1.4.3 the per-frame wait is the second of the two nvnSyncWait calls in f_70002db8ac (return address 0x70002db99c). Like on 1.2.1, its sync comes from a two-entry ring indexed per frame; the first call waits on one object's sync behind a mode check. wdiasjunior found the site and measured it on an RX 7800 XT (#676): about 67-70 to 81-83 fps on Great Sky Island. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 2 +- runtime/nvn_api_core.c | 13 +++++++++---- totk.ini.example | 5 ++--- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index ae4bc03f..6c43d4a2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,7 +85,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`). - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. -- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `TOTK_SHADOW=1` on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, so 1.2.1 only for now); results the game reads back from the GPU after it can be a frame old. +- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `TOTK_SHADOW=1` on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.2.1 and 1.4.3); results the game reads back from the GPU after it can be a frame old. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 3e284f9b..9007902a 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -471,7 +471,7 @@ NVN_API(nvnQueueFenceSync) { s->fenced = 1; s->fence = nvn_fence_last(); } -// TOTK_SYNC_AHEAD=1: one more frame in flight. Each frame the game waits (one nvnSyncWait, 1.2.1 lr 0x7000b23dfc) for +// TOTK_SYNC_AHEAD=1: one more frame in flight. Each frame the game waits (one nvnSyncWait, frame_wait_lr) for // the frame it presented last; on the Switch that work is long done by then, here it is still in the replay and on the // GPU (about 11 ms from the submit), and the wait sets the frame time. With the switch on, that wait, and only that // one, waits for the frame before it instead. What the game reads back of GPU results after the wait can then be a @@ -479,8 +479,13 @@ NVN_API(nvnQueueFenceSync) { // queued for the GPU or wait (nvn_sync_ahead_guard). The wait still lasts until the replay thread submitted the asked // value, so no copy of that value itself picks up those writes. pa_fence: the newest fence at the latest present and // the one before, present thread only for writes. pa_guard: the newest value the game was told is done early. -#define FRAME_WAIT_LR_121 0x7000b23dfcull static uint64_t pa_fence[2], pa_guard; +// The return address of that wait. 1.2.1: 0x7000b23dfc. 1.4.3: 0x70002db99c, the second of the two nvnSyncWait calls +// in f_70002db8ac; like on 1.2.1 its sync comes from a two-entry ring indexed per frame, and the first call waits on +// one object's sync behind a mode check. 0: no site known, the switch stays off. +static uint64_t frame_wait_lr(void) { + return rt_game_version == 121 ? 0x7000b23dfcull : rt_game_version == 143 ? 0x70002db99cull : 0; +} static int sync_ahead(void) { static int on = -1; if (on < 0) { @@ -493,7 +498,7 @@ static int sync_ahead(void) { "the game waits for the GPU as before"); on = 0; } - if (on && rt_game_version != 121) { + if (on && !frame_wait_lr()) { rt_log("nvn: TOTK_SYNC_AHEAD: no frame wait site is known for this game version; the game waits for the " "GPU as before"); on = 0; @@ -518,7 +523,7 @@ NVN_API(nvnSyncWait) { const GfxBackend *b = gfx_backend(); uint64_t want = s->fence; // Only the game's per-frame wait lags, and not a poll (timeout 0). - if (timeout && c->x[30] == FRAME_WAIT_LR_121 && sync_ahead() && b->fence_done() < want) { + if (timeout && c->x[30] == frame_wait_lr() && sync_ahead() && b->fence_done() < want) { uint64_t p0 = __atomic_load_n(&pa_fence[0], __ATOMIC_ACQUIRE); uint64_t p1 = __atomic_load_n(&pa_fence[1], __ATOMIC_ACQUIRE); if (want <= p0 && want > p1) want = p1; diff --git a/totk.ini.example b/totk.ini.example index 7dc995ff..ce70a413 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -51,9 +51,8 @@ # instead. On Great Sky Island (Linux, RX 9070 XT) with shadow_defer = 1 this took about 73 to 100 fps. Costs one more # frame of input delay. The frame rate swings less from frame to frame when uncapped than before, but more than # without it. GPU results the game reads back after that wait, for example occlusion queries, can be a frame old. -# Works on 1.2.1 only for now; other game versions ignore it. Needs shadow storage, which is on by default on Linux -# and Wine and needs the environment variable TOTK_SHADOW=1 on native Windows; without it the setting is ignored and -# the log says so. +# Works on 1.2.1 and 1.4.3; 1.1.0 ignores it. Needs shadow storage, which is on by default on Linux and Wine and needs +# the environment variable TOTK_SHADOW=1 on native Windows; without it the setting is ignored and the log says so. # sync_ahead = 0 # With sync_ahead = 1: when the game changes memory the GPU still reads, queue the copy into the GPU's shadow of that # memory for the GPU to run in order, 0 or 1. Default: on. With 0 the replay thread waits for the GPU to finish