diff --git a/docs/architecture.md b/docs/architecture.md index 0c123786..ae4bc03f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,6 +85,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`). - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. +- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `TOTK_SHADOW=1` on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, so 1.2.1 only for now); results the game reads back from the GPU after it can be a frame old. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/runtime/cfg.c b/runtime/cfg.c index b913d3b8..0b47be91 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -56,10 +56,10 @@ static const char *const known[] = { "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", - "SHADER_MEMO", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", + "SHADER_MEMO", "SHADOW_DEFER", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "STICK_SENS", "SWKBD_ANSWER", - "SWKBD_TEST", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_HEAP_EXTRA_MB", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UB_OVERLAY", "UB_OVERLAY_OUT", "UPSCALER", "VB_USED", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64", + "SWKBD_TEST", "SYNC_AHEAD", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_HEAP_EXTRA_MB", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UB_OVERLAY", "UB_OVERLAY_OUT", "UPSCALER", "VB_USED", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64", "VK_PRESENT", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VRAM_LIMIT_MIB", "VS_OUTS", "WALK_KEY", "WALK_SCALE", "WB_EXACT", "WB_FENCE_LIMIT", "WB_MERGE", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_KEEP", "XLATE_PROCS", "ZSTD_CHECK", "ZSTD_NATIVE", @@ -101,7 +101,7 @@ static const struct { const char *key, *section; } sections[] = { {"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"}, {"TEX_ALIAS", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"}, {"BIND_SKIP", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"TEX_MEMO", "performance"}, {"VB_USED", "performance"}, {"DESC_TEMPLATE", "performance"}, - {"PUSH_SKIP", "performance"}, {"SPLIT_DRAWS", "performance"}, {"SPLIT_US", "performance"}, {"VK_VALIDATION", "performance"}, + {"SYNC_AHEAD", "performance"}, {"SHADOW_DEFER", "performance"}, {"PUSH_SKIP", "performance"}, {"SPLIT_DRAWS", "performance"}, {"SPLIT_US", "performance"}, {"VK_VALIDATION", "performance"}, {"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"}, {"TRACE", "diagnostics"}, {"FLIGHT", "diagnostics"}, {"PASS_CRUMBS", "diagnostics"}, {"FLIGHT_SPIKE_MS", "diagnostics"}, {"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, @@ -163,6 +163,7 @@ static const CfgMeta metas[] = { {"DISCORD_IMAGE", "totk_512x512", NULL, LIVE}, // Advanced: scheduling {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, + {"SYNC_AHEAD", "0", "0|1", RESTART}, {"SHADOW_DEFER", "1", "0|1", RESTART}, {"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART}, // shaders and pipelines {"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "fullscreen", "strict|any|fullscreen", RESTART}, diff --git a/runtime/nvn_api.h b/runtime/nvn_api.h index b74253ab..8bf68216 100644 --- a/runtime/nvn_api.h +++ b/runtime/nvn_api.h @@ -43,6 +43,15 @@ _Static_assert(sizeof(nvn_mapping_t) == 0x28, "NVNmappingRequest stride"); const pool_t *nvn_pool_find(uint64_t addr, uint64_t len); // GPU-visible pool holding [addr, addr+len], NULL if none uint64_t nvn_new_serial(void); // next tex_t / prog_t serial, from 1 +// TOTK_SYNC_AHEAD (nvn_api_core.c): nvnQueuePresentTexture notes the newest fence at each present; the newest +// previous-frame value the game was told is done although it may not be (0: none). The back end's shadow copies +// wait for it or are queued for the GPU (vk_sync.c prologue), so the GPU never reads CPU writes the game made under +// that assumption too early. +void nvn_sync_ahead_present(void); +uint64_t nvn_sync_ahead_guard(void); +int vk_mem_shadow_on(void); // vk_memory.c: 1 with shadow storage on (valid once the Vulkan back end is up) +void vk_mem_shadow_fence(void); // vk_memory.c: returns once every shadow sync that read the old guard is done +void gfx_replay_wait_value(uint64_t v); // vk_backend.c: the replay thread submitted value v uint64_t nvn_fence_last(void); // fence value of the newest submit (0 before the first) int nvn_depth_minus_w(void); // 1 after DeviceSetDepthMode(NEAR_IS_MINUS_W), else 0 void nvn_guest_written(uint64_t addr, uint64_t size); // the host or the game wrote pool storage: back end re-reads diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 51f7f6bd..3e284f9b 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -471,6 +471,44 @@ NVN_API(nvnQueueFenceSync) { s->fenced = 1; s->fence = nvn_fence_last(); } +// TOTK_SYNC_AHEAD=1: one more frame in flight. Each frame the game waits (one nvnSyncWait, 1.2.1 lr 0x7000b23dfc) for +// the frame it presented last; on the Switch that work is long done by then, here it is still in the replay and on the +// GPU (about 11 ms from the submit), and the wait sets the frame time. With the switch on, that wait, and only that +// one, waits for the frame before it instead. What the game reads back of GPU results after the wait can then be a +// frame old. The game's CPU writes to memory the GPU still reads reach the GPU only through shadow copies, which are +// queued for the GPU or wait (nvn_sync_ahead_guard). The wait still lasts until the replay thread submitted the asked +// value, so no copy of that value itself picks up those writes. pa_fence: the newest fence at the latest present and +// the one before, present thread only for writes. pa_guard: the newest value the game was told is done early. +#define FRAME_WAIT_LR_121 0x7000b23dfcull +static uint64_t pa_fence[2], pa_guard; +static int sync_ahead(void) { + static int on = -1; + if (on < 0) { + on = cfg_bool("TOTK_SYNC_AHEAD"); + // Without shadow storage the GPU reads the game's own memory, which the game writes ahead: nothing guards + // that. The Vulkan back end sets the mode while it starts, before the game's first present or wait; the null + // back end never does. + if (on && !vk_mem_shadow_on()) { + rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage (TOTK_SHADOW=1 on native Windows); " + "the game waits for the GPU as before"); + on = 0; + } + if (on && rt_game_version != 121) { + rt_log("nvn: TOTK_SYNC_AHEAD: no frame wait site is known for this game version; the game waits for the " + "GPU as before"); + on = 0; + } + if (on) rt_log("nvn: TOTK_SYNC_AHEAD: a wait for the frame presented last waits for the frame before it"); + } + return on; +} +void nvn_sync_ahead_present(void) { + if (!sync_ahead()) return; + __atomic_store_n(&pa_fence[1], __atomic_load_n(&pa_fence[0], __ATOMIC_RELAXED), __ATOMIC_RELEASE); + __atomic_store_n(&pa_fence[0], nvn_fence_last(), __ATOMIC_RELEASE); +} +uint64_t nvn_sync_ahead_guard(void) { return __atomic_load_n(&pa_guard, __ATOMIC_ACQUIRE); } + // NVNsyncWaitResult nvnSyncWait(const NVNsync *s, uint64_t timeout_ns): a sync never fenced reports a timeout at once; // a fenced one is released only after the back end completed that fence value NVN_API(nvnSyncWait) { @@ -478,13 +516,34 @@ NVN_API(nvnSyncWait) { uint64_t timeout = arg_x(c, 1); if (!s->fenced) { ret_x(c, NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); return; } const GfxBackend *b = gfx_backend(); - if (b->fence_done() >= s->fence) { b->cpu_synced(s->fence); ret_x(c, NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED); return; } + uint64_t want = s->fence; + // Only the game's per-frame wait lags, and not a poll (timeout 0). + if (timeout && c->x[30] == FRAME_WAIT_LR_121 && sync_ahead() && b->fence_done() < want) { + uint64_t p0 = __atomic_load_n(&pa_fence[0], __ATOMIC_ACQUIRE); + uint64_t p1 = __atomic_load_n(&pa_fence[1], __ATOMIC_ACQUIRE); + if (want <= p0 && want > p1) want = p1; + } + if (want != s->fence) { // the replay of the asked value is done first: the guard then covers every copy of it + { + NVN_BLOCKING(); + gfx_replay_wait_value(s->fence); + } + uint64_t g = __atomic_load_n(&pa_guard, __ATOMIC_RELAXED); + while (s->fence > g && + !__atomic_compare_exchange_n(&pa_guard, &g, s->fence, 1, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) {} + vk_mem_shadow_fence(); // a shadow sync that read the old guard finishes before the game writes again + } + if (b->fence_done() >= want) { + b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); + ret_x(c, NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED); + return; + } int ok; { NVN_BLOCKING(); - ok = b->fence_wait(s->fence, timeout); + ok = b->fence_wait(want, timeout); } - if (ok) b->cpu_synced(s->fence); + if (ok) b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); } // void nvnEventBuilderSetDefaults(NVNeventBuilder *b) diff --git a/runtime/nvn_api_obj.c b/runtime/nvn_api_obj.c index 0324d6e9..644acc8d 100644 --- a/runtime/nvn_api_obj.c +++ b/runtime/nvn_api_obj.c @@ -713,6 +713,7 @@ NVN_API(nvnQueuePresentTexture) { const tex_t *t = (const tex_t *)o->tex[idx]; uint64_t period = present_period(o), now = kern_now_ns(); uint64_t slot = now + pace(&o->pace, now, period); + nvn_sync_ahead_present(); gfx_backend()->present(t, o->crop, slot); o->ticket[idx] = o->presents; o->avail[idx] = cfg_bool("TOTK_FRAME_QUEUE") ? slot : slot + period; diff --git a/runtime/tex_probe.c b/runtime/tex_probe.c index 552677de..c8a22782 100644 --- a/runtime/tex_probe.c +++ b/runtime/tex_probe.c @@ -355,7 +355,7 @@ static int test_import(pool_t *pool) { GCHECK(n == 4 && pages[0] == a + 0x30000 && pages[3] == a + 0x33000, "CPU writes to 4 pages: %llu reported", (unsigned long long)n); vk_mem_resolve(a + 0x30000, 0x4000, &sp); - vk_mem_shadow_sync(); + vk_mem_shadow_sync(0); copy_wait(sp.buffer, sp.offset, sb, 0, 0x4000); vmaInvalidateAllocation(c->vma, sa, 0, VK_WHOLE_SIZE); GCHECK(!memcmp(s, m + 0x30000, 0x4000), "GPU read of CPU-written heap differs"); diff --git a/runtime/vk.h b/runtime/vk.h index abe40f57..104f7fd1 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -280,7 +280,13 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb); // Shadow storage (vk_memory.c): vk_sync.c calls vk_mem_shadow_sync right before each submit; the replay notes every GPU // write into CPU-visible memory at record time (vk_mem_gpu_pending) and reports it at the fence with vk_gpu_written, // which copies the bytes back into guest memory before telling the texture cache (vk_guest_written). -void vk_mem_shadow_sync(void); +void vk_mem_shadow_sync(uint64_t fence); // fence: the value of the part about to be submitted (0 untracked) +// TOTK_SHADOW_DEFER: shadow updates that may not be written by the CPU while the GPU still reads the shadow are queued +// by vk_mem_shadow_sync as ranges of guest memory. vk_mem_shadow_deferred says whether there are any; +// vk_mem_shadow_prologue records the copies (staging -> shadow buffers) into the prologue command buffer, before +// vk_mem_mirror_prologue's copies, which read the shadow. +int vk_mem_shadow_deferred(void); +void vk_mem_shadow_prologue(VkCommandBuffer cb, uint64_t fence); void vk_mem_gpu_pending(uint64_t addr, uint64_t size); void vk_gpu_written(uint64_t addr, uint64_t size); int vk_mem_flush_add_probe(void); // --probe-vkexec: the flush range list; 0 or the number of the failed check diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index 5749de6b..c0a30b10 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -159,6 +159,7 @@ static PlatCond rcv = PLAT_COND_INIT; // item queued, item taken, item done static ReplayItem rq[RQ_CAP]; static int rq_head, rq_n, r_busy, r_started; static uint64_t r_presents_queued, r_presents_done, r_submit_done, game_frames; +static uint64_t r_value_done; // under rl: the newest timeline value the replay thread submitted // Under rl: calls, calls that waited and the time waited per cause, since the last replay_log line and over the run. static uint64_t dr_calls[DR_N], dr_waits[DR_N], dr_ns[DR_N], dr_tcalls[DR_N], dr_twaits[DR_N], dr_tns[DR_N]; // TOTK_REPLAY_CHECK: ReadTexels compared with a read after a full drain (under rl): checked, differing, differing @@ -191,6 +192,15 @@ uint64_t gfx_replayed_submits(void) { return v; } +// TOTK_SYNC_AHEAD (nvn_api_core.c nvnSyncWait): waits until the replay thread submitted value v, so every shadow copy +// of v was made before the game writes memory v reads. The caller has given its core up. +void gfx_replay_wait_value(uint64_t v) { + if (!gfx_replay_async()) return; + plat_mutex_lock(&rl); + while (r_value_done < v) plat_cond_wait(&rcv, &rl); + plat_mutex_unlock(&rl); +} + static uint64_t submit_sync(const uint64_t *handles, int n) { vk_sync_start(); vk_present_drain(); @@ -225,6 +235,7 @@ static void replay_main(void *arg) { present_image(it.tex, it.has_crop ? it.crop : NULL, it.at, it.queued, frame, it.value); } plat_mutex_lock(&rl); + if (it.kind != RI_RELEASE) r_value_done = it.value; if (it.kind == RI_SUBMIT) r_submit_done = it.submit_no; else r_presents_done++; r_busy = 0; diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 6c6b1a8b..75928616 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -71,6 +71,9 @@ typedef struct PoolRec { uint8_t *smap; // its mapping; offsets as in buf (from import_lo) uint32_t sgen; // gen_lock: every page write stamped up to this generation is in the shadow volatile uint8_t stouched; // on the touched list: the GPU may use it in the submission being recorded + uint64_t suse; // shadow_lock: the newest timeline value whose submission reads the shadow + uint64_t sdefer; // shadow_lock: the newest value whose prologue carries a deferred copy into the + // shadow (TOTK_SHADOW_DEFER); 0 never } PoolRec; static void mirror_forget(PoolRec *r); // pool_lock exclusive: off the mirrored list @@ -78,6 +81,7 @@ static int shadow_on; // shadow mode: imported pools get shad static int shadow_pool(VkCtx *c, PoolRec *r, uint64_t addr, uint64_t size); static void shadow_touch(PoolRec *r); static void shadow_forget(PoolRec *r); +static int shadow_had_deferred(PoolRec *r); static void mirror_retiring(PoolRec *r); // vk_pool_fini: kept until mirror_retire (or vk_mem_shutdown) static void mirror_retire(void *arg); // completion thread: destroys a finalized mirrored pool's record static void grow(void **arr, int *cap, size_t elem); @@ -335,7 +339,10 @@ void vk_pool_fini(pool_t *p) { vk_mem_collect(); // A mirrored pool's record and buffers outlive the submission being recorded, whose sync copies and write-backs may // name them (the game frees a pool only once the GPU is done with it, but its CPU writes may still be queued). - if (r->mbuf) mirror_retiring(r), vk_completion_add(vk_sync_issued() + 1, mirror_retire, r); + // The same holds for a shadow that got a deferred copy: the queue (dq) may still name it until that prologue is + // recorded. sdefer is read after shadow_forget, which took shadow_lock: a sync either ran before (sdefer set) or + // will not see the pool. + if (r->mbuf || shadow_had_deferred(r)) mirror_retiring(r), vk_completion_add(vk_sync_issued() + 1, mirror_retire, r); else destroy_rec(c, r); } @@ -878,6 +885,18 @@ static int nspend, cspend; static int64_t s_live_n, s_live_bytes; // shadow_lock static uint64_t s_syncs, s_copied, s_queries, s_untracked, s_wb_bytes, s_ticks; // shadow_lock: since the last log +// TOTK_SHADOW_DEFER (with TOTK_SYNC_AHEAD): a shadow update the CPU must not write yet is queued instead and executed +// by the GPU, in queue order, at the start of the prologue of the part submitted next (vk_mem_shadow_prologue). +// defer_on: the switch is on. sdefer_now: the pool being synced is deferred, and sdefer_fence the value its copies +// belong to. Replay thread (sub_lock): the queue. +typedef struct { PoolRec *r; uint64_t addr, len; } DRange; +static int defer_on; +static int sdefer_now; +static uint64_t sdefer_fence; +static DRange *dq; // guest ranges to copy into their pools' shadows +static int ndq, cdq; +static uint64_t s_defer_pools, s_defer_bytes, s_defer_copies, s_wait_ticks, s_waits; // shadow_lock: since the last log + static int shadow_pool(VkCtx *c, PoolRec *r, uint64_t addr, uint64_t size) { r->import_lo = addr & ~0xfffull; r->import_hi = (addr + size + 0xfff) & ~0xfffull; @@ -927,6 +946,13 @@ static void shadow_forget(PoolRec *r) { plat_mutex_unlock(&shadow_lock); } +static int shadow_had_deferred(PoolRec *r) { + plat_mutex_lock(&shadow_lock); + int d = r->sdefer != 0; + plat_mutex_unlock(&shadow_lock); + return d; +} + // shadow_lock held: copies guest [a, b) into the shadow, leaving out the bytes of GPU writes not copied back yet // (spend from index i on). static void shadow_copy(PoolRec *r, uint64_t a, uint64_t b, int i) { @@ -936,6 +962,13 @@ static void shadow_copy(PoolRec *r, uint64_t a, uint64_t b, int i) { if (spend[i].hi < b) shadow_copy(r, spend[i].hi, b, i + 1); return; } + if (sdefer_now) { // the GPU copies it later: the bytes are read from guest memory when the prologue is recorded + if (ndq == cdq) grow((void **)&dq, &cdq, sizeof *dq); + dq[ndq++] = (DRange){r, a, b - a}; + r->sdefer = sdefer_fence; + s_defer_bytes += b - a; + return; + } memcpy(r->smap + (a - r->import_lo), (const void *)(uintptr_t)a, b - a); s_copied += b - a; } @@ -980,13 +1013,76 @@ static void shadow_sync_pool(PoolRec *r) { if (run) shadow_copy(r, run, hi, 0); } -void vk_mem_shadow_sync(void) { - if (!shadow_on) return; - uint64_t t0 = qpc(); +// TOTK_SYNC_AHEAD (nvn_api_core.c): the game may have been told a value is done (at most the guard) that the GPU still +// runs, and then write memory that value reads. The CPU copies its writes into a shadow, so while the guard is not done +// a shadow with any reader in flight must not be written by the CPU: that reader may be the guarded value or an older +// one (the parts of one submission share a value), and a pool records only its newest reader. A shadow with no reader +// in flight is copied at once. +// With TOTK_SHADOW_DEFER the update of such a shadow is queued instead (this function). Without it the replay thread +// waits (shadow_wait_guard). + +// shadow_lock held: whether a shadow this sync is about to update has a reader in flight. +static int shadow_reader_busy(uint64_t done) { + for (int i = 0; i < nstouch; i++) + if (stouch[i]->suse > done) return 1; + return 0; +} + +// shadow_lock held (released while waiting), no TOTK_SHADOW_DEFER: waits until the guard is done when a shadow to +// update has a reader in flight, then re-reads the guard under the lock, since the game may have raised it meanwhile. The wait +// runs without shadow_lock: the completion thread needs it for the fence callbacks (vk_gpu_written). *g and *done are +// current on return. +static void shadow_wait_guard(uint64_t fence, uint64_t *g, uint64_t *done) { + while (*g > *done && (!fence || *g < fence) && shadow_reader_busy(*done)) { + uint64_t want = *g; + plat_mutex_unlock(&shadow_lock); + uint64_t t0 = qpc(); + vk_sync_fence_wait(want, UINT64_MAX); + uint64_t dt = qpc() - t0; + plat_mutex_lock(&shadow_lock); + s_wait_ticks += dt, s_waits++; + *g = nvn_sync_ahead_guard(); + *done = vk_sync_fence_done(); + } +} + +// shadow_lock held, TOTK_SHADOW_DEFER: the pool's update goes through the queue when a reader of it is in flight while +// the guard is (see above), or when an earlier deferred copy into it has not executed yet: a direct memcpy must never +// land before it, or the queued copy would overwrite newer bytes with older ones. +static int shadow_defers(const PoolRec *r, uint64_t g, uint64_t done) { + if (r->sdefer > done) return 1; + return g > done && r->suse > done; +} + +// nvnSyncWait raised the guard: a sync that read the old guard holds shadow_lock until its copies are made, so taking +// the lock once here makes the game return from the wait only after such a sync finished. Later syncs see the new +// guard. +void vk_mem_shadow_fence(void) { plat_mutex_lock(&shadow_lock); + plat_mutex_unlock(&shadow_lock); +} + +int vk_mem_shadow_on(void) { return shadow_on; } + +void vk_mem_shadow_sync(uint64_t fence) { + if (!shadow_on) return; + plat_mutex_lock(&shadow_lock); + // The guard is read under the lock: see vk_mem_shadow_fence. + uint64_t g = nvn_sync_ahead_guard(), done = g ? vk_sync_fence_done() : 0; + if (!defer_on && nstouch) shadow_wait_guard(fence, &g, &done); + uint64_t t0 = qpc(); if (nstouch) { plat_mutex_lock(&gen_lock); - for (int i = 0; i < nstouch; i++) shadow_sync_pool(stouch[i]), stouch[i]->stouched = 0; + for (int i = 0; i < nstouch; i++) { + PoolRec *r = stouch[i]; + sdefer_now = defer_on && fence && shadow_defers(r, g, done); + sdefer_fence = fence; + int before = ndq; + shadow_sync_pool(r), r->stouched = 0; + if (ndq > before) s_defer_pools++; + sdefer_now = 0; + if (fence > r->suse) r->suse = fence; + } plat_mutex_unlock(&gen_lock); s_syncs += (uint64_t)nstouch; nstouch = 0; @@ -995,6 +1091,8 @@ void vk_mem_shadow_sync(void) { plat_mutex_unlock(&shadow_lock); } +int vk_mem_shadow_deferred(void) { return ndq; } + void vk_mem_gpu_pending(uint64_t addr, uint64_t size) { if (!shadow_on || !size) return; plat_mutex_lock(&shadow_lock); @@ -1026,11 +1124,14 @@ static void shadow_log(uint64_t frame) { if (!shadow_on) return; plat_mutex_lock(&shadow_lock); rt_log("mem: shadow at frame %llu: %lld pools %.1f MiB; since the last: %llu pool syncs, %llu queries, %.1f MiB " - "copied in (%llu untracked), %.1f MiB written back, %.2f ms, %d GPU writes pending", + "copied in (%llu untracked), %.1f MiB written back, %.2f ms, %d GPU writes pending; deferred (%s): %llu pools, " + "%.1f MiB in %llu copies; replay waited for a reader %llu times, %.2f ms", (unsigned long long)frame, (long long)s_live_n, s_live_bytes / 1048576.0, (unsigned long long)s_syncs, (unsigned long long)s_queries, s_copied / 1048576.0, (unsigned long long)s_untracked, s_wb_bytes / 1048576.0, - qpc_ms(s_ticks), nspend); + qpc_ms(s_ticks), nspend, defer_on ? "on" : "off", (unsigned long long)s_defer_pools, + s_defer_bytes / 1048576.0, (unsigned long long)s_defer_copies, (unsigned long long)s_waits, qpc_ms(s_wait_ticks)); s_syncs = s_queries = s_copied = s_untracked = s_wb_bytes = s_ticks = 0; + s_defer_pools = s_defer_bytes = s_defer_copies = s_wait_ticks = s_waits = 0; plat_mutex_unlock(&shadow_lock); } @@ -1242,6 +1343,8 @@ static void mirror_mode(int discrete) { #else shadow_on = !sh || !*sh || strcmp(sh, "0"); #endif + defer_on = shadow_on && cfg_bool("TOTK_SYNC_AHEAD") && cfg_bool("TOTK_SHADOW_DEFER"); + if (defer_on) rt_log("mem: TOTK_SHADOW_DEFER: shadow updates that wait for a reader are queued for the GPU instead"); if (shadow_on) rt_log("mem: shadow storage: CPU-visible pools are host-visible copies of guest memory, not imports%s", sh && *sh ? " (TOTK_SHADOW)" : ""); rt_log("mem: mode %s (%s): %s", mode == 2 ? "verify" : mode ? "mirror" : "import", why, @@ -2309,6 +2412,61 @@ void *vk_staging(VkDeviceSize size, VkBuffer *buf, VmaAllocation *alloc) { return info.pMappedData; } +typedef struct { VkBuffer b; VmaAllocation a; } StgFree; +static void free_shadow_stg(void *arg) { + StgFree *f = arg; + vmaDestroyBuffer(vk_ctx()->vma, f->b, f->a); + free(f); +} + +// Records the deferred shadow copies (vk_mem_shadow_sync queued the ranges): the guest bytes go into staging buffers +// now, the GPU copies them into the pools' shadows. The barrier before waits for every earlier command, writes and +// reads alike (a read of the shadow must be over before the copy overwrites it); the one after makes the copies visible +// to the rest of the queue, including the mirror copies that follow in this command buffer. The staging buffers are +// freed when the value completes. Replay thread, sub_lock held. +void vk_mem_shadow_prologue(VkCommandBuffer cb, uint64_t fence) { + if (!ndq) return; + mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT | VK_ACCESS_2_MEMORY_READ_BIT, + VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); + const uint64_t lim = STG_BLOCK / 2; + uint64_t copies = 0; + int i = 0; + uint64_t at_i = 0; // bytes of dq[i] already recorded + while (i < ndq) { + uint64_t total = 0, jo = at_i; // how much the next staging buffer holds + for (int j = i; j < ndq && total < lim;) { + uint64_t left = dq[j].len - jo, take = left < lim - total ? left : lim - total; + total += take; + if (take == left) j++, jo = 0; + else jo += take; + } + VkBuffer sb; + VmaAllocation sa; + uint8_t *p = vk_staging(total, &sb, &sa); + for (uint64_t at = 0; at < total;) { + const DRange *d = &dq[i]; + uint64_t left = d->len - at_i, take = left < total - at ? left : total - at; + memcpy(p + at, (const void *)(uintptr_t)(d->addr + at_i), take); + VkBufferCopy bc = {at, d->addr + at_i - d->r->import_lo, take}; + vkCmdCopyBuffer(cb, sb, d->r->buf, 1, &bc); + copies++; + at += take, at_i += take; + if (at_i == d->len) i++, at_i = 0; + } + vmaFlushAllocation(vk_ctx()->vma, sa, 0, VK_WHOLE_SIZE); + StgFree *f = malloc(sizeof *f); + if (!f) rt_fatal("vk: out of memory"); + f->b = sb, f->a = sa; + vk_completion_add(fence, free_shadow_stg, f); + } + mbarrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); + ndq = 0; + plat_mutex_lock(&shadow_lock); + s_defer_copies += copies; + plat_mutex_unlock(&shadow_lock); +} + uint64_t vk_upload_flush(int wait) { plat_once(&cmd_once, cmd_init, NULL); plat_rmutex_lock(&cmd_lock); @@ -2438,7 +2596,7 @@ void vk_mem_shutdown(void) { plat_mutex_lock(&pool_lock); for (int i = 0; i < pool_index.n; i++) destroy_rec(c, pool_index.e[i].buf); pool_index.n = 0; - nm = npend = nwbl = nsnl = snl_copied = 0; // mirrors went with their pools + nm = npend = nwbl = nsnl = snl_copied = ndq = 0; // mirrors went with their pools plat_mutex_lock(&snap_pool_lock); for (int i = 0; i < nsnap_pool; i++) vmaDestroyBuffer(c->vma, snap_pool[i].buf, snap_pool[i].alloc); nsnap_pool = 0, snap_pool_bytes = 0; diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index e3162d12..d1c57883 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -350,12 +350,15 @@ static CmdSlot *take_slot(VkCtx *c, uint64_t fence) { // part was recorded, sub_lock held; may move slots[]. // First brings the shadow storage up to date with the CPU's writes, which needs no command buffer. static VkCommandBuffer prologue(VkCtx *c, uint64_t fence) { - vk_mem_shadow_sync(); // shadow storage: the CPU's writes go in before the part is submitted - if (!vk_mem_mirror_pending()) return VK_NULL_HANDLE; + // Shadow storage: the CPU's writes go in before the part is submitted (TOTK_SYNC_AHEAD: queued for the GPU or + // made after the GPU finished the readers the game may think done, vk_memory.c). + vk_mem_shadow_sync(fence); + if (!vk_mem_mirror_pending() && !vk_mem_shadow_deferred()) return VK_NULL_HANDLE; int i = claim_slot(c, fence); // before reading slots: the claim may move it VkCommandBuffer cb = slots[i].cb; rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts int pair = ts_open(cb); + vk_mem_shadow_prologue(cb, fence); // before the mirror copies, which read the shadow these bring up to date vk_mem_mirror_prologue(cb); ts_close(cb, pair, fence); if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); diff --git a/totk.ini.example b/totk.ini.example index 33e7dc4a..7dc995ff 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -45,6 +45,21 @@ # Windows timer resolution of 1 ms for steadier frame pacing (0 = the Windows default). # timer_1ms = 1 +# One frame ahead of the GPU. +# Let the game build the next frame while the GPU still draws the previous one, 0 or 1. Default: off. Each frame the +# game waits once for the GPU to finish the frame it presented last; with 1 that wait covers the frame before it +# instead. On Great Sky Island (Linux, RX 9070 XT) with shadow_defer = 1 this took about 73 to 100 fps. Costs one more +# frame of input delay. The frame rate swings less from frame to frame when uncapped than before, but more than +# without it. GPU results the game reads back after that wait, for example occlusion queries, can be a frame old. +# Works on 1.2.1 only for now; other game versions ignore it. Needs shadow storage, which is on by default on Linux +# and Wine and needs the environment variable TOTK_SHADOW=1 on native Windows; without it the setting is ignored and +# the log says so. +# sync_ahead = 0 +# With sync_ahead = 1: when the game changes memory the GPU still reads, queue the copy into the GPU's shadow of that +# memory for the GPU to run in order, 0 or 1. Default: on. With 0 the replay thread waits for the GPU to finish +# reading instead, which costs part of the gain. +# shadow_defer = 1 + # Shaders and pipelines. # Draws whose pipeline is still building: default waits on a reserved worker; draw is the previous scheduling. # pipe_wait = default