From 1ab3856fdd94f6ee5fa8d534132ee6de50ad8771 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 09:19:54 +0300 Subject: [PATCH 1/7] TOTK_SYNC_AHEAD=check measures what a one-frame-late frame wait without shadow storage would race, with the wait left exact The replay notes each GPU-timeline access of guest memory with its submission's value (in-place bindings of imported pools without a mirror, mirror sync copies, GPU deswizzle sources, mirror write-backs); at the frame wait the notes of the frame the late wait would leave running form its read set, snapped with the write watch and checked again at the next frame-wait entry. Invalidates and small-target write-backs of GPU results from that frame count as stale readbacks. One summary line per second and a run summary with the largest offenders by pool (runtime/vk_ahead.c). --probe-vkmem checks the detector with a synthetic write after the wait; tests/sync_ahead_check.py checks that check mode keeps the wait exact. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 1 + runtime/cfg.c | 2 +- runtime/nvn_api_core.c | 35 ++- runtime/tex_probe.c | 4 + runtime/vk.h | 16 +- runtime/vk_ahead.c | 561 ++++++++++++++++++++++++++++++++++++++ runtime/vk_ahead.h | 30 ++ runtime/vk_backend.c | 1 + runtime/vk_exec_draw.c | 9 + runtime/vk_exec_xfer.c | 6 + runtime/vk_fg.cpp | 2 +- runtime/vk_memory.c | 67 ++++- runtime/vk_sync.c | 2 +- runtime/vk_texture.c | 3 + tests/sync_ahead_check.py | 40 ++- 15 files changed, 752 insertions(+), 27 deletions(-) create mode 100644 runtime/vk_ahead.c create mode 100644 runtime/vk_ahead.h diff --git a/docs/architecture.md b/docs/architecture.md index 70d5ddad..a7ffb229 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -91,6 +91,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. - Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. - Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.1.0, 1.2.1 and 1.4.3). GPU writes into pool storage come back with their fence, so a buffer the game reads back every frame after that wait holds the result of the frame before; a buffer it has not read lately holds none, and its `nvnMemoryPoolInvalidateMappedRange` waits for the write (`vk_mem_shadow_read`): TotK reads single pixels through a buffer per request, and in the Depths zeros in one of them put Link in gloom. The exit line `mem: the game waited for a GPU write it reads` counts these waits and their time. On 1.4.3 the patch set `qstamp` (`recomp/patches-v143.txt`) lets the game take its depth query results one frame older; it follows the switch and is off when the switch is refused. The fence write-back of small render targets (`vk_texture.c`, `write_back_n`) does not wait for the GPU with `TOTK_WB_EAGER=1` (`wb_eager`, the default, but only while frame-ahead is in effect: with `sync_ahead` off or refused the setting does nothing and the write-back reads the image when it runs): `vk_sync_submit` has `vk_texture_eager_record` copy the targets that value rendered into staging buffers inside the same command buffer, so once the value has passed the write-back only copies from staging into the target's storage. Only a target that was written back within the last 64 timeline values is copied (the first write-back of a target takes the blocking path), and the staging of a target past that window, no longer wanted or released goes through the deferred free (`vk_defer_buffer_until`). A snapshot is used only when its value is the one that last rendered the target, nothing recorded a write to the image since (`wr_fence`) and the GPU has finished it; otherwise the write-back reads the image as before, and the exit line `vk: wb_eager:` counts the reasons. With frame-ahead in effect the exit line `vk: fence write-backs waited for the GPU` counts the write-backs that waited for a copy and the time they waited, for every value of the setting. `check` also reads the old way and compares. +- Frame-ahead check. `TOTK_SYNC_AHEAD=check` keeps the frame wait exact and measures what a one-frame-late wait without shadow storage would race (`runtime/vk_ahead.c`). The replay notes, with each submission's value, every access of guest memory the GPU makes after the recording: in-place bindings of imported pools without a VRAM mirror (`VK_AHEAD_READ`), the mirror's sync copies (`vk_mem_mirror_prologue`), GPU deswizzle sources (`vk_texture.c` `upload_gpu`) and the mirror write-backs (`vk_mem_mirror_end`). At the frame wait for value A, whose late form would have waited for L, the notes of values in (L, A] form the frame's read set: merged per class, gap-joined into at most 512 write-watch queries that stamp the writes made so far into the shared page generations (`vk_mem_ahead_observe`). At the next frame-wait entry the queries run again, and every page of the set stamped since is a CPU write the late wait would have raced; memory without a write watch is compared by page hashes. GPU writes into CPU-visible memory and reports are listed with their value: an invalidate of a range written by a value in (L, A] of the last frame wait, and the fence write-back of a small target rendered by one, count as stale readbacks. One line `sync-ahead check:` per second of host time and a run summary with the largest offenders by pool at exit. `--probe-vkmem` checks the detector on a watched pool (`vk_ahead_selftest`); `tests/sync_ahead_check.py` checks that the wait stays exact. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/runtime/cfg.c b/runtime/cfg.c index 9349ac04..16abeaba 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -217,7 +217,7 @@ static const CfgMeta metas[] = { {"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_CHUNKS", "0", "0..64", RESTART}, {"REPLAY_CHUNK_DRAWS", "128", "1..100000", RESTART}, {"REPLAY_STATE_CHECK", "0", "0|1", RESTART}, {"REPLAY_SKEW_US", "0", "0..100000", RESTART}, - {"SYNC_AHEAD", "0", "0|1", RESTART}, {"SHADOW_DEFER", "1", "0|1", RESTART}, + {"SYNC_AHEAD", "0", "0|1|check", RESTART}, {"SHADOW_DEFER", "1", "0|1", RESTART}, {"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART}, // shaders and pipelines {"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "fullscreen", "strict|any|fullscreen", RESTART}, diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 96c65e2b..187bee54 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -15,6 +15,7 @@ #include "host.h" #include "nvn_api.h" #include "nvn_pool_index.h" +#include "vk_ahead.h" #include "platform/platform.h" #define DEV_MAGIC MAGIC('D', 'E', 'V', 0) @@ -494,10 +495,13 @@ static uint64_t frame_wait_lr(void) { if (rt_game_version == 110) return 0x7000b39aa4ull; return rt_game_version == 121 ? 0x7000b23dfcull : rt_game_version == 143 ? 0x70002db99cull : 0; } -static int sync_ahead(void) { - static int on = -1; - if (on < 0) { - on = cfg_bool("TOTK_SYNC_AHEAD"); +// TOTK_SYNC_AHEAD=check (vk_ahead.c): the wait stays exact; the presents and the wait note what the one-frame-late +// wait would have waited for, and the back end counts what that would have raced. sync_mode: 0 off, 1 on, 2 check. +static int sync_mode(void) { + static int mode = -1; + if (mode < 0) { + const char *v = cfg_value("TOTK_SYNC_AHEAD"); + int on = strcmp(v, "0") != 0 && strcmp(v, "check") != 0, check = !strcmp(v, "check"); // Without shadow storage the GPU reads the game's own memory, which the game writes ahead: nothing guards // that. The Vulkan back end sets the mode while it starts, before the game's first present or wait; the null // back end never does. @@ -506,20 +510,22 @@ static int sync_ahead(void) { "the game waits for the GPU as before"); on = 0; } - if (on && !frame_wait_lr()) { + if ((on || check) && !frame_wait_lr()) { rt_log("nvn: TOTK_SYNC_AHEAD: no frame wait site is known for this game version; the game waits for the " "GPU as before"); - on = 0; + on = check = 0; } // The depth query patch (set qstamp, patches-v143.txt) is switched by TOTK_SYNC_AHEAD too: a refused switch // leaves the game's own frame number check, as with the switch off. if (!on) rt_patch_switch(rt_patch_set("qstamp"), 0); if (on) rt_log("nvn: TOTK_SYNC_AHEAD: a wait for the frame presented last waits for the frame before it"); + mode = on ? 1 : check ? 2 : 0; } - return on; + return mode; } +static int sync_ahead(void) { return sync_mode() == 1; } void nvn_sync_ahead_present(void) { - if (!sync_ahead()) return; + if (!sync_mode()) return; __atomic_store_n(&pa_fence[1], __atomic_load_n(&pa_fence[0], __ATOMIC_RELAXED), __ATOMIC_RELEASE); __atomic_store_n(&pa_fence[0], nvn_fence_last(), __ATOMIC_RELEASE); } @@ -533,12 +539,15 @@ NVN_API(nvnSyncWait) { uint64_t timeout = arg_x(c, 1); if (!s->fenced) { ret_x(c, NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); return; } const GfxBackend *b = gfx_backend(); - uint64_t want = s->fence; + uint64_t want = s->fence, lag = s->fence; // Only the game's per-frame wait lags, and not a poll (timeout 0). - if (timeout && c->x[30] == frame_wait_lr() && sync_ahead() && b->fence_done() < want) { + int mode = timeout && c->x[30] == frame_wait_lr() ? sync_mode() : 0; + if (mode == 2) vk_ahead_frame_entry(); + if (mode && b->fence_done() < want) { uint64_t p0 = __atomic_load_n(&pa_fence[0], __ATOMIC_ACQUIRE); uint64_t p1 = __atomic_load_n(&pa_fence[1], __ATOMIC_ACQUIRE); - if (want <= p0 && want > p1) want = p1; + if (want <= p0 && want > p1) lag = p1; + if (mode == 1) want = lag; } if (want != s->fence) { // the replay of the asked value is done first: the guard then covers every copy of it { @@ -551,7 +560,9 @@ NVN_API(nvnSyncWait) { vk_mem_shadow_fence(); // a shadow sync that read the old guard finishes before the game writes again } if (b->fence_done() >= want) { + if (mode == 2) vk_ahead_frame_lag(s->fence, lag); b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); + if (mode == 2) vk_ahead_frame_done(1); ret_x(c, NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED); return; } @@ -560,7 +571,9 @@ NVN_API(nvnSyncWait) { NVN_BLOCKING(); ok = b->fence_wait(want, timeout); } + if (mode == 2) vk_ahead_frame_lag(s->fence, lag); if (ok) b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); + if (mode == 2) vk_ahead_frame_done(ok); ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); } // void nvnEventBuilderSetDefaults(NVNeventBuilder *b) diff --git a/runtime/tex_probe.c b/runtime/tex_probe.c index 80eb91c5..06396205 100644 --- a/runtime/tex_probe.c +++ b/runtime/tex_probe.c @@ -372,6 +372,10 @@ static int test_import(pool_t *pool) { GCHECK(vk_mem_changed(RT_KERNEL_BASE, 0x1000, 0, &g3) == -1, "the kernel region is reported as tracked"); printf("probe-vkmem: write watch: CPU writes reported page-exact; generations see CPU and explicit writes\n"); g_gpu_checks += 7; + int ah = vk_ahead_selftest(a, pool->size); + GCHECK(!ah, "sync-ahead check self test failed at check %d", ah); + printf("probe-vkmem: sync-ahead check: a write after the frame wait into the read set is flagged, one before it is not\n"); + g_gpu_checks++; return 0; } diff --git a/runtime/vk.h b/runtime/vk.h index 55094c1d..9600b528 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -8,6 +8,7 @@ #pragma clang diagnostic pop #include "nvn_obj.h" // pool_t, tex_t, view_t, samp_par, NVNcopyRegion (memory and textures block) #include "gfx.h" // GfxShader, GfxShaderRes, GfxShaderAnswers (pipelines block) +#include "vk_ahead.h" // TOTK_SYNC_AHEAD=check hooks (vk_ahead_on) // Required (device choice fails without them): Vulkan 1.3; VK_KHR_swapchain, VK_KHR_push_descriptor, // VK_EXT_depth_clip_control, VK_EXT_external_memory_host; features timelineSemaphore, dynamicRendering, @@ -265,6 +266,19 @@ void vk_mem_use(VkMemSpan *sp); void vk_mem_pick(VkMemSpan *sp); #define VK_MEM_USE(sp) do { if (vk_mirror_on && (sp).pool) vk_mem_use(&(sp)); } while (0) #define VK_MEM_PICK(sp) do { if (vk_mirror_on && (sp).pool) vk_mem_pick(&(sp)); } while (0) +// TOTK_SYNC_AHEAD=check (vk_ahead.c): VK_AHEAD_READ after VK_MEM_USE / VK_MEM_PICK notes a binding or copy of +// [addr, addr+size) that the GPU reads (use: VK_MUSE_*; GPU_WRITE for a write) in place in guest memory: an imported +// pool without a mirror or shadow. One test of vk_ahead_on when the mode is off. +void vk_ahead_span(const VkMemSpan *sp, uint64_t addr, uint64_t size, int use); +#define VK_AHEAD_READ(sp, addr, size, use) \ + do { if (__builtin_expect(vk_ahead_on, 0)) vk_ahead_span(&(sp), (addr), (size), (use)); } while (0) +int vk_mem_span_inplace(const VkMemSpan *sp); // the span is guest memory itself (no mirror, no shadow, no device memory) +// The write watch over the ranges: every write made so far stamped into the page generations; untracked[i] 1 where +// range i has no write watch. Returns the newest generation. +uint32_t vk_mem_ahead_observe(const uint64_t (*iv)[2], int n, uint8_t *untracked); +// Bytes of [lo, hi) in pages stamped after generation since; *first: the first such byte. +uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, uint64_t *first); +int vk_mem_pool_info(uint64_t addr, uint64_t *lo, uint64_t *size, uint32_t *flags, int *mirrored); // 0: no pool void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size); // A shader's storage-buffer write: goes back through the write-back merge (vk_memory.c), which keeps the words the GPU // did not change. @@ -303,7 +317,7 @@ void vk_mem_mirror_begin(void); int vk_mem_mirror_end(VkCommandBuffer cb, uint64_t fence); void vk_mem_mirror_seal(void); // before vk_mem_shadow_sync: the copies the next vk_mem_mirror_prologue takes int vk_mem_mirror_pending(void); -void vk_mem_mirror_prologue(VkCommandBuffer cb); +void vk_mem_mirror_prologue(VkCommandBuffer cb, uint64_t fence); // fence: the value of the part it precedes // Shadow storage (vk_memory.c): vk_sync.c calls vk_mem_shadow_sync right before each submit; the replay notes every GPU // write into CPU-visible memory at record time (vk_mem_gpu_pending) and reports it at the fence with vk_gpu_written, // which copies the bytes back into guest memory before telling the texture cache (vk_guest_written). diff --git a/runtime/vk_ahead.c b/runtime/vk_ahead.c new file mode 100644 index 00000000..d11e7a0b --- /dev/null +++ b/runtime/vk_ahead.c @@ -0,0 +1,561 @@ +// TOTK_SYNC_AHEAD=check: what answering the game's frame wait one frame late without shadow storage would race, with +// the wait left exact. In that form (Savage Wind's overlap) the game is told frame N is done when only N-1 is, and the +// GPU runs N while the game builds N+1, so every byte of guest memory N's GPU work reads after its recording (or writes +// back into guest memory) must not change between the wait for N and the next frame wait, which then waits for N; and +// every GPU result the game reads back after the wait for N must not come from N. +// - Read set: the replay notes each GPU-timeline access of guest memory with its submission's value (vk_ahead_read): +// in-place bindings of imported pools without a VRAM mirror, the mirror's sync copies (the pages and flush ranges +// copied in the prologue), GPU deswizzle sources and the mirror write-backs. Uniform snapshots and CPU texture uploads +// copy at record time, before the wait: not noted. +// - At the frame wait for value A whose one-frame-late form would have waited for L < A, the notes of values in (L, A] +// are N's read set: merged per class and over all classes, and gap-joined into at most IV_MAX write-watch queries. The +// queries stamp every write made so far into the shared page generations (vk_memory.c vk_mem_ahead_observe); G is the +// newest generation then. At the next frame-wait entry the same queries run again, and every page of the set stamped +// after G is a CPU write after the wait that the late form would have raced: counted in bytes by class, and the first +// offenders logged by pool. A range in memory without a write watch is hashed by page instead. +// - Readbacks: GPU writes into CPU-visible memory and reports are listed with their value (vk_ahead_gpu_write). An +// invalidate of a range a GPU write of a value in (L, A] of the last frame wait lands in would have read stale data; +// so would the fence write-back of a small target last rendered by such a value (vk_ahead_small). +// Bounded: fixed tables, a note list that is compacted when full, one frame set at a time. One log line per second of +// host time and a summary at exit, prefixed "sync-ahead check:". +#include +#include +#include + +#include "cfg.h" +#include "host.h" +#include "nvn_api.h" +#include "vk.h" +#include "vk_ahead.h" +#include "vk_exec.h" +#include "platform/platform.h" + +int vk_ahead_on; + +static const char *const cls_name[AH_N + 1] = {"vb", "ib", "ubo", "ssbo", "ind", "copy", "msync", "deswz", "wb", "wbm", + "all"}; + +#define NOTE_MAX 65536 // notes not yet taken by a frame wait +#define SET_MAX 8192 // merged ranges of one frame set, per class and the union together +#define IV_MAX 512 // write-watch queries per set (as Savage Wind's gsnap list) +#define IV_GAP 0x10000ull // ranges closer than this share a query +#define HASH_MAX 4096 // pages hashed per set (memory without a write watch) +#define GW_MAX 8192 // listed GPU writes and reports +#define RD_N 64 // invalidated ranges remembered (as vk_memory.c sread) +#define RD_AGE 64 // ... known when invalidated within this many values +#define SMALL_N 64 // small targets remembered (Savage Wind: one-off = not read back in 16 frames, 64 entries) +#define SMALL_FRAMES 16 +#define OFF_N 32 // offender table +#define OFF_LOG 24 // first-sighting lines + +typedef struct { uint64_t lo, hi, fence; uint8_t cls; } Note; +typedef struct { uint64_t lo, hi; uint8_t cls, untracked; uint32_t hash0, nhash; } SetR; +typedef struct { uint64_t lo, hi, fence; uint8_t kind; } GWrite; + +static PlatMutex ah_lock = PLAT_MUTEX_INIT; // notes, gw, rd, small, lag context +static Note *notes; +static int nnotes; +static uint64_t n_compact; +static GWrite gw[GW_MAX]; +static int ngw; +static uint64_t gw_over; +static uint64_t win_lo, win_hi; // (L, A] of the last frame wait; win_lo == win_hi: none +static int lag_live; // inside cpu_synced of a frame wait (vk_ahead_frame_lag .. done) +static uint64_t cur_asked, cur_lagged; +static uint64_t frame_no; +static struct { uint64_t addr, size, at; } rd[RD_N]; +static struct { uint64_t addr, frame; } small_t[SMALL_N]; + +// Game thread only (the frame wait): +static Note *take; +static SetR set[SET_MAX]; +static int nset; +static uint64_t iv[IV_MAX][2]; +static uint8_t iv_untr[IV_MAX]; +static int niv, armed; +static uint32_t set_gen; +static uint64_t hashes[HASH_MAX]; +static int nhash; + +typedef struct { + uint64_t frames, lagged, read[AH_N + 1], hit[AH_N + 1], hit_frames[AH_N + 1], hit_max[AH_N + 1]; + uint64_t inval, inval_stale, inval_stale_unknown, inval_stale_bytes, late[AHW_N], small, small_stale, small_oneoff; + uint64_t cost_ns, set_joined, set_lost, untracked, hashed, hash_skipped, ivq; +} Stat; +static Stat win, tot; // ah_lock for the readback counters, game thread for the rest (summed under ah_lock) + +typedef struct { uint64_t pool, size, bytes, frames, first; uint32_t flags; int cls, mirrored; } Offender; +static Offender off[OFF_N]; +static int noff, nofflog; +static uint64_t last_log_ns; + +void vk_ahead_init(void) { + vk_ahead_on = !strcmp(cfg_value("TOTK_SYNC_AHEAD"), "check"); + if (!vk_ahead_on) return; + notes = malloc(sizeof *notes * NOTE_MAX); + take = malloc(sizeof *take * NOTE_MAX); + if (!notes || !take) rt_fatal("vk: out of memory (sync-ahead check)"); + rt_log("sync-ahead check: the frame wait stays exact; logging what a one-frame-late wait without shadow storage " + "would race (%s)", vk_mem_shadow_on() ? "shadow storage is on: the counts describe the in-place form only " + "approximately" : "in place"); +} + +static uint64_t note_fence(uint64_t fence) { + if (fence) return fence; + uint64_t f = vk_exec_recording_fence(); + return f ? f : vk_sync_issued() + 1; +} + +static int note_cmp(const void *x, const void *y) { + const Note *a = x, *b = y; + if (a->cls != b->cls) return a->cls < b->cls ? -1 : 1; + return a->lo < b->lo ? -1 : a->lo > b->lo; +} + +// ah_lock held, list full: overlapping and touching notes of one class merge (the newest value kept). +static void compact(void) { + qsort(notes, (size_t)nnotes, sizeof *notes, note_cmp); + int o = 0; + for (int i = 1; i < nnotes; i++) { + Note *p = ¬es[o]; + if (notes[i].cls == p->cls && notes[i].lo <= p->hi) { + if (notes[i].hi > p->hi) p->hi = notes[i].hi; + if (notes[i].fence > p->fence) p->fence = notes[i].fence; + } else notes[++o] = notes[i]; + } + nnotes = o + 1; + n_compact++; + if (nnotes > NOTE_MAX * 3 / 4) { // still full: join each class's neighbours (coarser, never smaller) + o = 0; + for (int i = 1; i < nnotes; i++) { + Note *p = ¬es[o]; + if (notes[i].cls == p->cls && notes[i].lo - p->hi <= IV_GAP * 16) { + if (notes[i].hi > p->hi) p->hi = notes[i].hi; + if (notes[i].fence > p->fence) p->fence = notes[i].fence; + } else notes[++o] = notes[i]; + } + nnotes = o + 1; + } +} + +void vk_ahead_read(uint64_t addr, uint64_t size, int cls, uint64_t fence) { + if (!vk_ahead_on || !size || (unsigned)cls >= AH_N || addr + size < addr) return; + fence = note_fence(fence); + plat_mutex_lock(&ah_lock); + Note *l = nnotes ? ¬es[nnotes - 1] : NULL; + if (l && l->cls == cls && l->fence == fence && addr <= l->hi && addr + size >= l->lo) { + if (addr < l->lo) l->lo = addr; + if (addr + size > l->hi) l->hi = addr + size; + } else { + if (nnotes == NOTE_MAX) compact(); + notes[nnotes++] = (Note){addr, addr + size, fence, (uint8_t)cls}; + } + plat_mutex_unlock(&ah_lock); +} + +void vk_ahead_span(const VkMemSpan *sp, uint64_t addr, uint64_t size, int use) { + if (!vk_mem_span_inplace(sp)) return; + int cls = use == VK_MUSE_VB ? AH_VB : use == VK_MUSE_IB ? AH_IB : use == VK_MUSE_UBO ? AH_UBO + : use == VK_MUSE_IND ? AH_IND : use == VK_MUSE_COPY_SRC ? AH_COPY : use == VK_MUSE_GPU_WRITE ? AH_WB : AH_SSBO; + vk_ahead_read(addr, size, cls, 0); +} + +void vk_ahead_gpu_write(uint64_t addr, uint64_t size, int kind, uint64_t fence) { + if (!vk_ahead_on || !size) return; + fence = note_fence(fence); + plat_mutex_lock(&ah_lock); + int i = ngw; + while (i > 0 && i > ngw - 8 && !(gw[i - 1].lo == addr && gw[i - 1].hi == addr + size && gw[i - 1].kind == kind)) i--; + if (i > 0 && i > ngw - 8) { + if (fence > gw[i - 1].fence) gw[i - 1].fence = fence; + } else { + if (ngw == GW_MAX) { // the oldest goes + memmove(gw, gw + 1, sizeof *gw * (GW_MAX - 1)); + ngw--, gw_over++; + } + gw[ngw++] = (GWrite){addr, addr + size, fence, (uint8_t)kind}; + } + plat_mutex_unlock(&ah_lock); +} + +void vk_ahead_invalidate(uint64_t addr, uint64_t size) { + if (!vk_ahead_on || !size) return; + plat_mutex_lock(&ah_lock); + uint64_t now = vk_sync_issued(), stale = 0; + for (int i = 0; i < ngw; i++) + if (gw[i].fence > win_lo && gw[i].fence <= win_hi && gw[i].lo < addr + size && gw[i].hi > addr) { + uint64_t a = gw[i].lo > addr ? gw[i].lo : addr, b = gw[i].hi < addr + size ? gw[i].hi : addr + size; + stale += b - a; + } + int e = 0; // the range's entry, else the oldest + for (int i = 0; i < RD_N; i++) { + if (rd[i].addr == addr && rd[i].size == size) { e = i; break; } + if (rd[i].at < rd[e].at) e = i; + } + int known = rd[e].addr == addr && rd[e].size == size && rd[e].at + RD_AGE >= now; + rd[e].addr = addr, rd[e].size = size, rd[e].at = now; + win.inval++; + if (stale) win.inval_stale++, win.inval_stale_bytes += stale, win.inval_stale_unknown += !known; + plat_mutex_unlock(&ah_lock); +} + +void vk_ahead_small(uint64_t addr, uint64_t gpu_fence) { + if (!vk_ahead_on) return; + plat_mutex_lock(&ah_lock); + if (lag_live) { + int e = 0; + for (int i = 0; i < SMALL_N; i++) { + if (small_t[i].addr == addr) { e = i; break; } + if (small_t[i].frame < small_t[e].frame) e = i; + } + int oneoff = small_t[e].addr != addr || small_t[e].frame + SMALL_FRAMES < frame_no; + small_t[e].addr = addr, small_t[e].frame = frame_no; + win.small++; + if (gpu_fence > cur_lagged) win.small_stale++, win.small_oneoff += oneoff; + } + plat_mutex_unlock(&ah_lock); +} + +// ---- the frame set + +static int lo_cmp(const void *x, const void *y) { + const SetR *a = x, *b = y; + return a->lo < b->lo ? -1 : a->lo > b->lo; +} + +static uint64_t hash_bytes(uint64_t lo, uint64_t hi) { + uint64_t h = 0x9E3779B97F4A7C15ull ^ (hi - lo); + const uint8_t *p = (const uint8_t *)(uintptr_t)lo; + uint64_t n = hi - lo, i = 0; + for (; i + 8 <= n; i += 8) { + uint64_t w; + memcpy(&w, p + i, 8); + h = (h ^ w) * 0x100000001B3ull, h ^= h >> 29; + } + for (; i < n; i++) h = (h ^ p[i]) * 0x100000001B3ull; + return h; +} + +// Readable now (committed, not freed by the game since): hashing a page that is gone would fault. +static int readable(uint64_t lo, uint64_t hi) { + for (uint64_t a = lo, b; a < hi; a = b) { + int st = plat_mem_state(a, hi, &b); + if (st != PLAT_PROT_READ && st != PLAT_PROT_RW) return 0; + if (b <= a) return 0; + } + return 1; +} + +// The pages of set[i] (in memory without a write watch) hashed into hashes[], or compared with them (check). +static uint64_t hash_pass(SetR *s, int check) { + uint64_t changed = 0; + if (!check) s->hash0 = (uint32_t)nhash, s->nhash = 0; + uint32_t k = s->hash0; + for (uint64_t a = s->lo; a < s->hi;) { + uint64_t b = (a & ~0xfffull) + 0x1000; + if (b > s->hi) b = s->hi; + if (!check) { // 0: the page is not compared (the table is full or the page not readable) + if (nhash == HASH_MAX) { win.hash_skipped += s->hi - a; break; } + int ok = readable(a, b); + hashes[nhash++] = ok ? hash_bytes(a, b) | 1 : 0; + s->nhash++; + if (ok) win.hashed += b - a; + else win.hash_skipped += b - a; + } else { + if (k >= s->hash0 + s->nhash) break; + if (hashes[k] && readable(a, b) && hashes[k] != (hash_bytes(a, b) | 1)) changed += b - a; + k++; + } + a = b; + } + return changed; +} + +static void offender(int cls, uint64_t addr, uint64_t bytes) { + uint64_t lo = 0, size = 0; + uint32_t flags = 0; + int mirrored = 0; + vk_mem_pool_info(addr, &lo, &size, &flags, &mirrored); + int i = 0; + while (i < noff && !(off[i].cls == cls && off[i].pool == lo)) i++; + if (i == noff) { + if (noff == OFF_N) { // the smallest goes + int m = 0; + for (int k = 1; k < noff; k++) if (off[k].bytes < off[m].bytes) m = k; + i = m; + } else noff++; + off[i] = (Offender){lo, size, 0, 0, addr - lo, flags, cls, mirrored}; + if (nofflog++ < OFF_LOG) + rt_log("sync-ahead check: frame %llu: %s range written after the wait: pool %#llx (%.1f MiB, flags %#x, %s) " + "+%#llx, %llu bytes", (unsigned long long)frame_no, cls_name[cls], (unsigned long long)lo, + size / 1048576.0, flags, mirrored ? "mirrored" : "in place", (unsigned long long)(addr - lo), + (unsigned long long)bytes); + } + off[i].bytes += bytes, off[i].frames++; +} + +// The armed set against the writes since its snap. +static void check_set(void) { + vk_mem_ahead_observe((const uint64_t(*)[2])iv, niv, iv_untr); + win.ivq += (uint64_t)niv; + uint64_t hit[AH_N + 1] = {0}; + for (int i = 0; i < nset; i++) { + SetR *s = &set[i]; + uint64_t first = s->lo, b = s->untracked ? hash_pass(s, 1) : vk_mem_ahead_written(s->lo, s->hi, set_gen, &first); + if (!b) continue; + hit[s->cls] += b; + if (s->cls != AH_N) offender(s->cls, first, b); + } + for (int c = 0; c <= AH_N; c++) { + win.hit[c] += hit[c]; + if (hit[c]) win.hit_frames[c]++; + if (hit[c] > win.hit_max[c]) win.hit_max[c] = hit[c]; + } + armed = 0; +} + +// take[0..n): notes of values in (L, A]. Merged per class, then over all classes (class AH_N), then the queries. +static void build_set(int n) { + qsort(take, (size_t)n, sizeof *take, note_cmp); + nset = 0; + for (int i = 0; i < n; i++) { + SetR *p = nset ? &set[nset - 1] : NULL; + if (p && p->cls == take[i].cls && take[i].lo <= p->hi) { + if (take[i].hi > p->hi) p->hi = take[i].hi; + continue; + } + if (nset == SET_MAX / 2) { // full: the rest joins the last range of its class, or is left out + if (p && p->cls == take[i].cls) { + win.set_joined += take[i].lo - p->hi; + if (take[i].hi > p->hi) p->hi = take[i].hi; + } else win.set_lost += take[i].hi - take[i].lo; + continue; + } + set[nset++] = (SetR){take[i].lo, take[i].hi, take[i].cls, 0, 0, 0}; + } + int nc = nset; + for (int i = 0; i < nc; i++) win.read[set[i].cls] += set[i].hi - set[i].lo; + // The union over all classes, after them. + SetR *u = set + nc; + memcpy(u, set, sizeof *set * (size_t)nc); + qsort(u, (size_t)nc, sizeof *u, lo_cmp); + int nu = 0; + for (int i = 0; i < nc; i++) { + if (nu && u[i].lo <= u[nu - 1].hi) { + if (u[i].hi > u[nu - 1].hi) u[nu - 1].hi = u[i].hi; + } else u[nu++] = (SetR){u[i].lo, u[i].hi, AH_N, 0, 0, 0}; + } + nset = nc + nu; + for (int i = 0; i < nu; i++) win.read[AH_N] += u[i].hi - u[i].lo; + // Queries: the union gap-joined; a gap that does not fit doubles until it does. + uint64_t gap = IV_GAP; + for (;;) { + niv = 0; + for (int i = 0; i < nu && niv <= IV_MAX; i++) { + if (niv && u[i].lo - iv[niv - 1][1] <= gap) { + if (u[i].hi > iv[niv - 1][1]) iv[niv - 1][1] = u[i].hi; + } else if (niv < IV_MAX) iv[niv][0] = u[i].lo, iv[niv][1] = u[i].hi, niv++; + else { niv = IV_MAX + 1; break; } + } + if (niv <= IV_MAX) break; + gap *= 2; + } +} + +// Whether [lo, hi) lies in a query the write watch does not cover. +static int untracked_at(uint64_t lo) { + int a = 0, b = niv; + while (a < b) { + int m = (a + b) / 2; + if (iv[m][1] <= lo) a = m + 1; + else b = m; + } + return a < niv && iv[a][0] <= lo && iv_untr[a]; +} + +static void snap_set(void) { + set_gen = vk_mem_ahead_observe((const uint64_t(*)[2])iv, niv, iv_untr); + win.ivq += (uint64_t)niv; + nhash = 0; + for (int i = 0; i < nset; i++) { + SetR *s = &set[i]; + s->untracked = (uint8_t)untracked_at(s->lo); + if (!s->untracked) continue; + win.untracked += s->cls == AH_N ? s->hi - s->lo : 0; + hash_pass(s, 0); + } + armed = 1; +} + +static void log_line(const Stat *s, const char *what) { + char b[2048]; // about 1,200 bytes with every class + int k = 0; + double f = s->frames ? (double)s->frames : 1; + k += snprintf(b + k, sizeof b - (size_t)k, "sync-ahead check: %s %llu frames (%llu lagged): written after the wait " + "KiB/frame", what, (unsigned long long)s->frames, (unsigned long long)s->lagged); + for (int c = 0; c <= AH_N; c++) + if (s->hit[c] || c == AH_N) + k += snprintf(b + k, sizeof b - (size_t)k, " %s %.1f (in %llu frames, max %.0f)", cls_name[c], + s->hit[c] / 1024.0 / f, (unsigned long long)s->hit_frames[c], s->hit_max[c] / 1024.0); + k += snprintf(b + k, sizeof b - (size_t)k, "; read KiB/frame"); + for (int c = 0; c <= AH_N; c++) + if (s->read[c] || c == AH_N) k += snprintf(b + k, sizeof b - (size_t)k, " %s %.1f", cls_name[c], s->read[c] / 1024.0 / f); + k += snprintf(b + k, sizeof b - (size_t)k, + "; stale: invalidates %llu of %llu (%llu not read lately, %.1f KiB), late reports counter %llu samples " + "%llu event %llu, writes %llu, small write-backs %llu of %llu (%llu one-off)", + (unsigned long long)s->inval_stale, (unsigned long long)s->inval, (unsigned long long)s->inval_stale_unknown, + s->inval_stale_bytes / 1024.0, (unsigned long long)s->late[AHW_COUNTER], (unsigned long long)s->late[AHW_SAMPLES], + (unsigned long long)s->late[AHW_EVENT], (unsigned long long)s->late[AHW_WRITE], + (unsigned long long)s->small_stale, (unsigned long long)s->small, (unsigned long long)s->small_oneoff); + snprintf(b + k, sizeof b - (size_t)k, "; game-thread cost %.3f ms/frame, %.1f queries/frame, untracked %.1f " + "KiB/frame (hashed %.1f, skipped %.1f), set over its size: %.1f KiB joined, %.1f KiB left out; note compactions %llu, gpu-write list " + "overflow %llu", s->cost_ns / 1e6 / f, s->ivq / f, s->untracked / 1024.0 / f, s->hashed / 1024.0 / f, + s->hash_skipped / 1024.0 / f, s->set_joined / 1024.0, s->set_lost / 1024.0, (unsigned long long)n_compact, + (unsigned long long)gw_over); + rt_log("%s", b); +} + +static void stat_add(Stat *d, const Stat *s) { + const uint64_t *x = (const uint64_t *)s; + uint64_t *y = (uint64_t *)d; + for (size_t i = 0; i < sizeof *s / sizeof(uint64_t); i++) y[i] += x[i]; + for (int c = 0; c <= AH_N; c++) { // maxima, not sums + d->hit_max[c] -= s->hit_max[c]; + if (s->hit_max[c] > d->hit_max[c]) d->hit_max[c] = s->hit_max[c]; + } +} + +void vk_ahead_frame_entry(void) { + if (!vk_ahead_on || !armed) return; + uint64_t t0 = plat_now_ns(); + check_set(); + win.cost_ns += plat_now_ns() - t0; +} + +void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged) { + if (!vk_ahead_on) return; + plat_mutex_lock(&ah_lock); + frame_no++; + cur_asked = asked, cur_lagged = lagged, lag_live = 1; + plat_mutex_unlock(&ah_lock); +} + +void vk_ahead_frame_done(int ok) { + if (!vk_ahead_on) return; + uint64_t t0 = plat_now_ns(); + plat_mutex_lock(&ah_lock); + lag_live = 0; + uint64_t a = cur_asked, l = cur_lagged; + int n = 0, k = 0; + if (ok) { + for (int i = 0; i < nnotes; i++) { + if (notes[i].fence > a) notes[k++] = notes[i]; + else if (notes[i].fence > l) take[n++] = notes[i]; + } + nnotes = k; + k = 0; + for (int i = 0; i < ngw; i++) { // writes done even with the lag go; the late ones are counted + if (gw[i].fence > l && gw[i].fence <= a) win.late[gw[i].kind]++; + if (gw[i].fence > l) gw[k++] = gw[i]; + } + ngw = k; + win_lo = l, win_hi = a; + } + plat_mutex_unlock(&ah_lock); + if (ok) { + win.frames++; + if (l < a) win.lagged++; + build_set(n); + snap_set(); + } + uint64_t now = plat_now_ns(); + win.cost_ns += now - t0; + if (now - last_log_ns >= 1000000000ull && win.frames) { + plat_mutex_lock(&ah_lock); + Stat w = win; + memset(&win, 0, sizeof win); + plat_mutex_unlock(&ah_lock); + if (last_log_ns) log_line(&w, "1 s:"); + stat_add(&tot, &w); + last_log_ns = now; + } +} + +static int off_cmp(const void *x, const void *y) { + const Offender *a = x, *b = y; + return a->bytes < b->bytes ? 1 : a->bytes > b->bytes ? -1 : 0; +} + +void vk_ahead_log_final(void) { + static int32_t done; + if (!vk_ahead_on || plat_xchg32(&done, 1)) return; + plat_mutex_lock(&ah_lock); + stat_add(&tot, &win); + memset(&win, 0, sizeof win); + plat_mutex_unlock(&ah_lock); + log_line(&tot, "run:"); + qsort(off, (size_t)noff, sizeof *off, off_cmp); + for (int i = 0; i < noff && i < 12; i++) + rt_log("sync-ahead check: offender %d: %s pool %#llx (%.1f MiB, flags %#x, %s) first at +%#llx: %.1f KiB in %llu " + "frames", i + 1, cls_name[off[i].cls], (unsigned long long)off[i].pool, off[i].size / 1048576.0, off[i].flags, + off[i].mirrored ? "mirrored" : "in place", (unsigned long long)off[i].first, off[i].bytes / 1024.0, + (unsigned long long)off[i].frames); +} + +// --probe-vkmem (tex_probe.c test_import): [a, a+size) is write-watched guest memory of an imported pool, at least +// 0x40000 bytes, that nothing else touches meanwhile. A frame's read set gets a CPU write before its wait (not counted) +// and one after it (counted in its class and the union, not in another class); a frame the late wait would not have +// lagged keeps no set; an invalidate of a GPU write of the lagged frame counts as stale. 0: passed, else the failed check. +int vk_ahead_selftest(uint64_t a, uint64_t size) { + if (size < 0x40000) return 1; + int was = vk_ahead_on; + Note *n0 = notes, *t0 = take; + Note nbuf[64], tbuf[64]; + if (!notes) notes = nbuf, take = tbuf; // the checks add far fewer than 64 notes + vk_ahead_on = 1; + plat_mutex_lock(&ah_lock); + nnotes = ngw = 0; + plat_mutex_unlock(&ah_lock); + armed = 0; + Stat saved = win; + uint64_t log_was = last_log_ns; + last_log_ns = plat_now_ns(); // no summary line (it would reset the counters) while the checks run + memset(&win, 0, sizeof win); + volatile uint8_t *m = (volatile uint8_t *)(uintptr_t)a; + uint64_t f = vk_sync_issued() + 1000; // values no real submission has + int bad = 0; + vk_ahead_read(a + 0x10000, 0x8000, AH_MSYNC, f); + vk_ahead_read(a + 0x20000, 0x1000, AH_VB, f); + vk_ahead_read(a + 0x28000, 0x1000, AH_VB, f + 1); // a later frame's: stays in the list + vk_ahead_gpu_write(a + 0x30000, 16, AHW_COUNTER, f); + m[0x10010] = (uint8_t)(m[0x10010] + 1); // before the wait: the frame wrote it before the GPU read it + vk_ahead_frame_lag(f, f - 1); + vk_ahead_frame_done(1); + if (!armed || win.read[AH_MSYNC] != 0x8000 || win.read[AH_VB] != 0x1000 || win.read[AH_N] != 0x9000) bad = 2; + else if (win.late[AHW_COUNTER] != 1) bad = 3; + vk_ahead_invalidate(a + 0x30000, 0x100); + if (!bad && (win.inval != 1 || win.inval_stale != 1)) bad = 4; + m[0x12345] = (uint8_t)(m[0x12345] + 1); // after the wait, in the msync range: the late wait would race it + m[0x3a000] = (uint8_t)(m[0x3a000] + 1); // after the wait, outside the set + vk_ahead_frame_entry(); + if (!bad && (armed || win.hit[AH_MSYNC] != 0x1000 || win.hit[AH_VB] || win.hit[AH_N] != 0x1000)) bad = 5; + plat_mutex_lock(&ah_lock); + int left = nnotes; + plat_mutex_unlock(&ah_lock); + if (!bad && left != 1) bad = 6; + vk_ahead_frame_lag(f + 1, f + 1); // not lagged: nothing at risk + vk_ahead_frame_done(1); + if (!bad && win.read[AH_N] != 0x9000) bad = 7; + m[0x28010] = (uint8_t)(m[0x28010] + 1); + vk_ahead_frame_entry(); + if (!bad && win.hit[AH_N] != 0x1000) bad = 8; + plat_mutex_lock(&ah_lock); + nnotes = ngw = 0, win_lo = win_hi = 0; + plat_mutex_unlock(&ah_lock); + armed = 0, noff = 0, nofflog = 0; + win = saved; + last_log_ns = log_was; + notes = n0, take = t0; + vk_ahead_on = was; + return bad; +} diff --git a/runtime/vk_ahead.h b/runtime/vk_ahead.h new file mode 100644 index 00000000..48149cb0 --- /dev/null +++ b/runtime/vk_ahead.h @@ -0,0 +1,30 @@ +// TOTK_SYNC_AHEAD=check (vk_ahead.c): measures what answering the frame wait one frame late without shadow storage +// would race, while the wait stays exact. Off: vk_ahead_on is 0 and every hook is one test of it. +#pragma once +#include + +// Classes of GPU-timeline accesses of guest memory a submission makes after its recording: in-place bindings of an +// imported pool without a mirror (vertex, index, uniform, storage, indirect, copy source), the VRAM mirror's sync copies, +// the GPU deswizzle of a texture's storage, and the mirror write-backs (plain copies, word merges). +enum { AH_VB, AH_IB, AH_UBO, AH_SSBO, AH_IND, AH_COPY, AH_MSYNC, AH_DESWZ, AH_WB, AH_WBM, AH_N }; +// Kinds of GPU results the CPU may read back: buffer writes, counter reports, samples-passed reports, event signals. +enum { AHW_WRITE, AHW_COUNTER, AHW_SAMPLES, AHW_EVENT, AHW_N }; + +extern int vk_ahead_on; +void vk_ahead_init(void); // vk_memory.c mirror_mode, once at device start: reads TOTK_SYNC_AHEAD +// [addr, addr+size) of guest memory is read (or, for WB and WBM, written) on the GPU timeline by the submission of value +// fence (0: the one being recorded, else the next one). +void vk_ahead_read(uint64_t addr, uint64_t size, int cls, uint64_t fence); +// The GPU writes [addr, addr+size) of CPU-visible memory, or a report lands there, at the completion of value fence +// (0: as above). +void vk_ahead_gpu_write(uint64_t addr, uint64_t size, int kind, uint64_t fence); +void vk_ahead_invalidate(uint64_t addr, uint64_t size); // nvnMemoryPoolInvalidateMappedRange +void vk_ahead_small(uint64_t addr, uint64_t gpu_fence); // the fence write-back of a small target (vk_texture.c) +// The game's per-frame wait (nvn_api_core.c nvnSyncWait at frame_wait_lr): entry checks the previous frame's read set; +// lag before cpu_synced gives the value a one-frame-late wait would have waited for (lagged == asked: no lag); done after +// it records this frame's read set. ok 0: the wait timed out. +void vk_ahead_frame_entry(void); +void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged); +void vk_ahead_frame_done(int ok); +void vk_ahead_log_final(void); // at exit +int vk_ahead_selftest(uint64_t addr, uint64_t size); // --probe-vkmem: 0 passed diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index 64b61ed2..bde23fbe 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -478,6 +478,7 @@ void gfx_exit_log(void) { if (!plat_xchg32(&fs_done, 1)) fs_log(&fs_run, "over the run"); vk_texture_eager_log(); vk_mem_shadow_read_log(); + vk_ahead_log_final(); if (gfx_replay_async() && r_started && !plat_xchg32(&done, 1)) replay_log(game_frames, 1); } diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index 305d67a3..e7f27738 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -1277,6 +1277,7 @@ static int bind_range(VkExec *ex, uint64_t addr, uint64_t size, uint64_t align, } if (sp.offset % align == 0) { VK_MEM_USE(sp); + VK_AHEAD_READ(sp, addr, n, use); if (written) ub_written(ex, addr, n); *out = (VkDescriptorBufferInfo){sp.buffer, sp.offset, n}; if (written) @@ -2966,6 +2967,7 @@ static void bind_vertex_buffers(VkExec *ex) { if (r->address && vk_mem_resolve(r->address, 1, &sp)) { VK_MEM_USE(sp); nb = sp.buffer, no = sp.offset, nsz = r->size && r->size < sp.avail ? r->size : sp.avail; + VK_AHEAD_READ(sp, r->address, nsz, VK_MUSE_VB); ncls = sp.cls, npool = sp.pool; } D->B.vin[b] = *r, D->B.vres |= 1u << b; @@ -3788,6 +3790,7 @@ static uint32_t mo_vertex_buffers(VkExec *ex) { if (!r->address || !vk_mem_resolve(r->address, 1, &sp)) continue; VK_MEM_USE(sp); v->b[b] = sp.buffer, v->o[b] = sp.offset, v->sz[b] = r->size && r->size < sp.avail ? r->size : sp.avail; + VK_AHEAD_READ(sp, r->address, v->sz[b], VK_MUSE_VB); } memcpy(MQ.vin, ex->vbuf, sizeof MQ.vin); MQ.vgen = gen; @@ -4422,6 +4425,7 @@ static void op_draw_idx(VkExec *ex, const p_draw_idx *d) { return warn_once(W_INDEX, "index buffer %#llx is not aligned to its index size: draw skipped", (unsigned long long)d->ibuf); } else { VK_MEM_USE(sp); + VK_AHEAD_READ(sp, d->ibuf, bytes, VK_MUSE_IB); vkCmdBindIndexBuffer(ex->cb, sp.buffer, sp.offset, d->itype == NVN_INDEX_TYPE_UNSIGNED_SHORT ? VK_INDEX_TYPE_UINT16 : VK_INDEX_TYPE_UINT32); ex->recorded++; vk_mem_count(sp.cls, VK_MUSE_IB, bytes); @@ -4450,6 +4454,8 @@ static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { ex->fr.draws++; VK_MEM_USE(ip); VK_MEM_USE(ib); + VK_AHEAD_READ(ip, d->ind, sizeof(VkDrawIndexedIndirectCommand), VK_MUSE_IND); + VK_AHEAD_READ(ib, d->ibuf, ib.avail < 0x10000 ? ib.avail : 0x10000, VK_MUSE_IB); // the count is the GPU's: 64 KiB vkCmdBindIndexBuffer(ex->cb, ib.buffer, ib.offset, d->itype == NVN_INDEX_TYPE_UNSIGNED_SHORT ? VK_INDEX_TYPE_UINT16 : VK_INDEX_TYPE_UINT32); vkCmdDrawIndexedIndirect(ex->cb, ip.buffer, ip.offset, 1, sizeof(VkDrawIndexedIndirectCommand)); ex->recorded += 2; @@ -4504,6 +4510,7 @@ static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { if (!p || !info) return warn_once(W_PIPELINE, "a compute pipeline could not be created: dispatches with it are skipped"); vk_exec_barrier(ex); VK_MEM_USE(ip); + VK_AHEAD_READ(ip, d->ind, sizeof(VkDispatchIndirectCommand), VK_MUSE_IND); bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p); D->c0_dim[0] = D->c0_dim[1] = D->c0_dim[2] = 0; D->B.c0_valid[2] = 0; // a fresh c0 range: the copy below must not change one an earlier dispatch reads @@ -4759,6 +4766,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { uint8_t *d = scratch(ex, u->size, &b, &o); MX_CPY(d, u->data, u->size); VK_MEM_PICK(sp); + VK_AHEAD_READ(sp, u->dst, u->size, VK_MUSE_GPU_WRITE); ov_add(u->dst, u->dst + u->size, b, o, &sp); vk_mem_gpu_wrote(&sp, u->size); vk_usnap_drop(u->dst, u->size); @@ -4776,6 +4784,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { vk_exec_barrier(ex); // earlier draws finished with the old contents; overlaid updates' copies go first MX_CPY(scratch(ex, u->size, &b, &o), u->data, u->size); VK_MEM_PICK(sp); + VK_AHEAD_READ(sp, u->dst, u->size, VK_MUSE_GPU_WRITE); VkBufferCopy cp = {o, sp.offset, u->size}; vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp); vk_mem_gpu_wrote(&sp, u->size); diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index d3b35b19..bb8723ac 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -277,6 +277,7 @@ static void signal_event(VkExec *ex, const p_signal_event *p) { EventWrite *e = malloc(sizeof *e); if (!e) rt_fatal("vk: out of memory"); *e = (EventWrite){p->val, p->mode, p->value}; + if (vk_ahead_on) vk_ahead_gpu_write((uintptr_t)p->val, sizeof *p->val, AHW_EVENT, ex->fence); vk_completion_add(ex->fence, event_cb, e); } @@ -290,6 +291,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) { if (!vk_mem_resolve(p->addr, sizeof(NVNcounterData), &m)) rt_fatal("vk: counter report at %#llx is in no memory pool", (unsigned long long)p->addr); vk_mem_count(m.cls, VK_MUSE_GPU_WRITE, sizeof(NVNcounterData)); + if (vk_ahead_on) vk_ahead_gpu_write(p->addr, sizeof(NVNcounterData), samples ? AHW_SAMPLES : AHW_COUNTER, ex->fence); if (samples) oq_report(ex, p->addr); else { uint64_t *a = malloc(sizeof *a); @@ -432,6 +434,7 @@ static void copy_buffer_image(VkExec *ex, const p_copy_b2t *p, int to_image) { VkBufferImageCopy c; VkMemSpan m; if (!buffer_side(ex, &s, p->src, &size, &c, &m)) return; + VK_AHEAD_READ(m, p->src, size, to_image ? VK_MUSE_COPY_SRC : VK_MUSE_GPU_WRITE); vk_mem_count(m.cls, to_image ? VK_MUSE_COPY_SRC : VK_MUSE_GPU_WRITE, size); vk_exec_pass_end(ex); vk_exec_barrier(ex); @@ -497,6 +500,8 @@ static void copy_b2b(VkExec *ex, const p_copy_b2b *p) { if (n > d.avail) n = d.avail; VK_MEM_PICK(s); VK_MEM_PICK(d); + VK_AHEAD_READ(s, src, n, VK_MUSE_COPY_SRC); + VK_AHEAD_READ(d, dst, n, VK_MUSE_GPU_WRITE); if (s.buffer == d.buffer && s.offset < d.offset + n && d.offset < s.offset + n) { warn_once(W_OVERLAP, "buffer copy %#llx -> %#llx overlaps within one buffer: skipped", (unsigned long long)p->src, (unsigned long long)p->dst); @@ -537,6 +542,7 @@ static void clear_buf(VkExec *ex, const p_clear_buf *p) { if (!n) break; if (!started) vk_exec_pass_end(ex), vk_exec_barrier(ex), started = 1; VK_MEM_PICK(d); + VK_AHEAD_READ(d, dst, n, VK_MUSE_GPU_WRITE); vkCmdFillBuffer(ex->cb, d.buffer, d.offset, n, p->value); vk_mem_gpu_wrote(&d, n); ex->recorded++; diff --git a/runtime/vk_fg.cpp b/runtime/vk_fg.cpp index 7ac91fca..c74216e0 100644 --- a/runtime/vk_fg.cpp +++ b/runtime/vk_fg.cpp @@ -295,7 +295,7 @@ static void start(void) { } g.win_fmt = VK_FORMAT_R8G8B8A8_UNORM; g.workers = cfg_int("TOTK_REPLAY_WORKERS"); - g.sync_ahead = cfg_bool("TOTK_SYNC_AHEAD"); + g.sync_ahead = cfg_bool("TOTK_SYNC_AHEAD") && strcmp(cfg_value("TOTK_SYNC_AHEAD"), "check") != 0; // check: exact g.hdr = vk_hdr_wanted(); g.test = knob_int("TOTK_FG_TEST"); g.debug = knob_int("TOTK_FG_DEBUG"); diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index baaabef5..cfc10d78 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -982,6 +982,56 @@ static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) { } } +// ---------------------------------------------------------------- TOTK_SYNC_AHEAD=check (vk_ahead.c) +int vk_mem_span_inplace(const VkMemSpan *sp) { + const PoolRec *r = sp->pool; + return r && !sp->alloc && r->kind == POOL_IMPORT && !r->smap && sp->buffer == r->buf; +} + +uint32_t vk_mem_ahead_observe(const uint64_t (*iv)[2], int n, uint8_t *untracked) { + plat_mutex_lock(&gen_lock); + uint32_t g = ++gen_now; + for (int i = 0; i < n; i++) { + uint64_t lo = iv[i][0] & ~0xfffull, hi = (iv[i][1] + 0xfff) & ~0xfffull; + int frozen = hi > lo && hi <= RT_ADDRESS_LIMIT; // a frozen region changed in no page since its freeze + for (uint64_t a = lo & ~((1ull << REG_SHIFT) - 1); a < hi && frozen; a += 1ull << REG_SHIFT) frozen = mem_frozen(a); + untracked[i] = (uint8_t)(!frozen && (hi <= lo || hi > RT_ADDRESS_LIMIT || !query_stamp(lo, hi, g))); + } + g = gen_now; + plat_mutex_unlock(&gen_lock); + return g; +} + +uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, uint64_t *first) { + uint64_t bytes = 0; + if (hi <= lo || hi > RT_ADDRESS_LIMIT) return 0; + plat_mutex_lock_shared(&gen_lock); + for (uint64_t r = lo >> REG_SHIFT; r <= (hi - 1) >> REG_SHIFT; r++) { + if (reg_max[r] <= since) continue; + uint64_t a = r << REG_SHIFT > lo ? r << REG_SHIFT : lo & ~0xfffull; + uint64_t b = (r + 1) << REG_SHIFT < hi ? (r + 1) << REG_SHIFT : hi; + const uint32_t *ch = gen_chunk[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk + if (!ch) continue; + for (uint64_t x = a; x < b; x += 1ull << GEN_PAGE_SHIFT) { + if (ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)] <= since) continue; + uint64_t s = x > lo ? x : lo, e = x + (1ull << GEN_PAGE_SHIFT) < hi ? x + (1ull << GEN_PAGE_SHIFT) : hi; + if (!bytes && first) *first = s; + bytes += e - s; + } + } + plat_mutex_unlock_shared(&gen_lock); + return bytes; +} + +int vk_mem_pool_info(uint64_t addr, uint64_t *lo, uint64_t *size, uint32_t *flags, int *mirrored) { + plat_mutex_lock_shared(&pool_lock); + const pidx_ent *e = pidx_find(&pool_index, (uintptr_t)addr, 1); + const PoolRec *r = e ? e->buf : NULL; + if (r) *lo = r->lo, *size = r->size, *flags = r->flags, *mirrored = r->mstate == MS_ON; + plat_mutex_unlock_shared(&pool_lock); + return r != NULL; +} + // ---------------------------------------------------------------- shadow storage // Where importing guest memory costs too much (Linux: re-protecting pages that amdgpu imported as a userptr // invalidates the buffer, about 1.3 ms per 16 MiB at the next submit, and the import's write pinning clears the @@ -1322,6 +1372,7 @@ static uint64_t s_read_waits, s_read_ticks; // shadow_lock: since the last log static uint64_t r_read_waits, r_read_ticks, r_read_max; // shadow_lock: over the run (vk_mem_shadow_read_log) void vk_mem_shadow_read(uint64_t addr, uint64_t size) { + if (vk_ahead_on) vk_ahead_invalidate(addr, size); if (!shadow_on || !size || !nvn_sync_ahead_guard()) return; plat_mutex_lock(&shadow_lock); uint64_t now = vk_sync_issued(); @@ -1396,6 +1447,7 @@ int vk_mem_shadow_deferred(void) { return ndq; } // Record time: the range's base is the shadow's bytes, except where an overlapping pending range has one: the GPU may // have written those shadow bytes since. void vk_mem_gpu_pending(uint64_t addr, uint64_t size) { + if (vk_ahead_on) vk_ahead_gpu_write(addr, size, AHW_WRITE, 0); if (!shadow_on || !size) return; uint64_t lo = addr, hi = addr + size; plat_mutex_lock_shared(&pool_lock); @@ -1689,12 +1741,14 @@ static void mirror_mode(int discrete) { #else shadow_on = !sh || !*sh || strcmp(sh, "0"); #endif - defer_on = shadow_on && cfg_bool("TOTK_SYNC_AHEAD") && cfg_bool("TOTK_SHADOW_DEFER"); + defer_on = shadow_on && cfg_bool("TOTK_SYNC_AHEAD") && strcmp(cfg_value("TOTK_SYNC_AHEAD"), "check") && + cfg_bool("TOTK_SHADOW_DEFER"); // check: the wait stays exact, nothing is deferred if (defer_on) rt_log("mem: TOTK_SHADOW_DEFER: shadow updates that wait for a reader are queued for the GPU instead"); if (shadow_on) rt_log("mem: shadow storage: CPU-visible pools are host-visible copies of guest memory, not imports%s", sh && *sh ? " (TOTK_SHADOW)" : ""); rt_log("mem: mode %s (%s): %s", mode == 2 ? "verify" : mode ? "mirror" : "import", why, mode ? "pools used as buffers get a device-local mirror" : "every CPU-visible pool is imported host memory"); + vk_ahead_init(); // Import is the integrated-GPU mode. On a discrete card every draw reads the pools over PCIe (the comment above // this section has the measurements), which is the difference between playable and 5 fps when it is not intended. if (!mode && discrete) @@ -2309,6 +2363,9 @@ static void freeze_pass(void) { void vk_mem_mirror_begin(void) { if (!vk_mirror_on) { // import mode: the texture checks' snapshots still skip frozen regions + // TOTK_SYNC_AHEAD=check notes in-place bindings where the replay resolves them: a new generation per submission + // makes it resolve them again in each, as mirror mode does. + if (vk_ahead_on) __atomic_add_fetch(&mirror_gen, 1, __ATOMIC_RELEASE); if (!freeze_idle) return; plat_mutex_lock(&gen_lock); ww_ep++; @@ -2370,12 +2427,14 @@ int vk_mem_mirror_pending(void) { return n; } -void vk_mem_mirror_prologue(VkCommandBuffer cb) { +void vk_mem_mirror_prologue(VkCommandBuffer cb, uint64_t fence) { plat_mutex_lock(&mirror_lock); if (!pend_seal && snl_copied == snl_seal) { plat_mutex_unlock(&mirror_lock); return; } + if (vk_ahead_on) // the copies read guest memory when the GPU runs them + for (int i = 0; i < pend_seal; i++) vk_ahead_read(pend[i].r->import_lo + pend[i].off, pend[i].size, AH_MSYNC, fence); mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); copy_ranges(cb, pend, pend_seal, 1, &mrep.upload_bytes, &mrep.upload_copies); @@ -2489,6 +2548,8 @@ int vk_mem_mirror_end(VkCommandBuffer cb, uint64_t fence) { mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_STORAGE_READ_BIT); int m = wb_merge_record(cb); + if (m && vk_ahead_on) // the merge writes guest memory when the GPU runs it + for (int i = 0; i < nsnl; i++) vk_ahead_read(snl[i].r->import_lo + snl[i].off, snl[i].size, AH_WBM, fence); if (!m) { // no range list: copy them plainly for (int i = 0; i < nsnl; i++) range_add(&wbl, &nwbl, &cwbl, 0, snl[i].r, snl[i].off, snl[i].size); } @@ -2508,6 +2569,8 @@ int vk_mem_mirror_end(VkCommandBuffer cb, uint64_t fence) { mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); for (int i = 0; i < nwbl; i++) wbl[i].r->wb_sub = msub; + if (vk_ahead_on) + for (int i = 0; i < nwbl; i++) vk_ahead_read(wbl[i].r->import_lo + wbl[i].off, wbl[i].size, AH_WB, fence); mrep.wb_ranges += (uint64_t)nwbl; rec += copy_ranges(cb, wbl, nwbl, 0, &mrep.wb_bytes, &mrep.wb_copies) + 2; mbarrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index f48e237f..a5ad6e3d 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -385,7 +385,7 @@ static VkCommandBuffer prologue(VkCtx *c, uint64_t fence, int direct) { } int pair = ts_open(cb); vk_mem_shadow_prologue(cb, fence); // before the mirror copies, which read the shadow these bring up to date - vk_mem_mirror_prologue(cb); + vk_mem_mirror_prologue(cb, fence); ts_close(cb, pair, fence); if (direct || !rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); return cb; diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index daad15bd..b7761fac 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -1203,6 +1203,7 @@ static int upload_gpu(Tex *e) { vk_upload_end(); if (vk_upload_held() > UPLOAD_HELD_MAX) vk_upload_flush(0); free(dr), free(rg), free(ar); + if (vk_ahead_on) vk_ahead_read(e->pub.addr, t->L.size, AH_DESWZ, 0); // the batch reads the pool when it runs return 1; } @@ -2584,6 +2585,8 @@ void vk_texture_write_back_small(uint64_t fence) { } write_back_n(hit, k); for (int i = 0; i < k; i++) perf_count(PERF_CNT_WB); + if (vk_ahead_on) + for (int i = 0; i < k; i++) vk_ahead_small(hit[i]->pub.addr, hit[i]->gpu_fence); for (int i = k; i < n; i++) perf_count(PERF_CNT_WB_SKIP); free(hit); unlock(); diff --git a/tests/sync_ahead_check.py b/tests/sync_ahead_check.py index ef6d0571..a8499799 100644 --- a/tests/sync_ahead_check.py +++ b/tests/sync_ahead_check.py @@ -1,4 +1,4 @@ -"""Exercise the production sync-ahead wait selection with a deterministic back end. +"""Exercise the production sync-ahead wait selection with a deterministic back end (TOTK_SYNC_AHEAD 0, 1, check). python tests/sync_ahead_check.py [--cc path/to/clang] """ @@ -35,7 +35,19 @@ static int wait_for(uint64_t value, uint64_t timeout) { typedef struct { uint64_t (*fence_done)(void); void (*cpu_synced)(uint64_t); int (*fence_wait)(uint64_t, uint64_t); } GfxBackend; static const GfxBackend backend = {done, cpu_synced, wait_for}; static const GfxBackend *gfx_backend(void) { return &backend; } -static int cfg_bool(const char *name) { assert(!strcmp(name, "TOTK_SYNC_AHEAD")); return enabled; } +// enabled: 0 off, 1 on, 2 check (the wait stays exact; vk_ahead.c is told what the late wait would have waited for) +static const char *cfg_value(const char *name) { + assert(!strcmp(name, "TOTK_SYNC_AHEAD")); + return enabled == 2 ? "check" : enabled ? "1" : "0"; +} +static int entry_calls, lag_calls, done_calls, done_ok; +static uint64_t lag_asked, lag_value; +static void vk_ahead_frame_entry(void) { assert(!lag_calls && !done_calls); entry_calls++; } +static void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged) { + assert(entry_calls == 1 && !sync_calls); // before cpu_synced: the small write-backs see the lag + lag_calls++, lag_asked = asked, lag_value = lagged; +} +static void vk_ahead_frame_done(int ok) { assert(lag_calls == 1); done_calls++, done_ok = ok; } static int vk_mem_shadow_on(void) { return shadow; } static uint64_t nvn_fence_last(void) { return last; } static void rt_log(const char *format, ...) { (void)format; } @@ -66,9 +78,10 @@ int main(int argc, char **argv) { qstamp_on = rt_game_version == 143 && enabled; last = 20; nvn_sync_ahead_present(); last = 40; nvn_sync_ahead_present(); - assert(pa_fence[0] == (enabled && shadow && supported ? 40 : 0)); - assert(pa_fence[1] == (enabled && shadow && supported ? 20 : 0)); - assert(qstamp_on == (rt_game_version == 143 && enabled && shadow)); // a refused switch leaves the game's check + int on = enabled == 1 && shadow && supported, check = enabled == 2 && supported; + assert(pa_fence[0] == (on || check ? 40 : 0)); + assert(pa_fence[1] == (on || check ? 20 : 0)); + assert(qstamp_on == (rt_game_version == 143 && on)); // a refused switch and check leave the game's check int relaxed_cases = 0, polls = 0, other_waits = 0; for (int i = 0; i < 100000; i++) { sync_t sync = {.fenced = random_word() % 2, .fence = random_word() % 65}; @@ -77,16 +90,23 @@ int main(int argc, char **argv) { completed = random_word() % 65; uint64_t initial_done = completed, target = sync.fence; succeed = random_word() % 2; - relaxed = sync.fenced && enabled && shadow && supported && timeout && lr == site && - completed < target && target > 20 && target <= 40; + int lagcase = sync.fenced && timeout && lr == site && completed < target && target > 20 && target <= 40; + relaxed = on && lagcase; + int checked = check && sync.fenced && timeout && lr == site; relaxed_cases += relaxed; polls += timeout == 0; other_waits += lr != site; if (relaxed) target = 20; replay_calls = shadow_calls = sync_calls = wait_calls = 0; + entry_calls = lag_calls = done_calls = 0; synced = waited = waited_timeout = replay_value = UINT64_MAX; pa_guard = 0; Cpu cpu = {0}; cpu.x[0] = (uintptr_t)&sync; cpu.x[1] = timeout; cpu.x[30] = lr; nvnSyncWait(&cpu); assert(replay_calls == relaxed && shadow_calls == relaxed); + assert(entry_calls == checked && lag_calls == checked && done_calls == checked); + if (checked) { // exact wait, the late one's value told to the check + assert(lag_asked == sync.fence && lag_value == (lagcase ? 20 : sync.fence)); + assert(done_ok == (initial_done >= target || succeed)); + } assert(pa_guard == (relaxed ? sync.fence : 0)); if (relaxed) assert(replay_value == sync.fence); if (!sync.fenced) { @@ -102,9 +122,9 @@ int main(int argc, char **argv) { } } assert(polls > 0 && other_waits > 0); - assert((relaxed_cases > 0) == (enabled && shadow && supported)); + assert((relaxed_cases > 0) == on); // Raising the guard for an older request must not lower a guard already published by another waiter. - if (enabled && shadow && supported) { + if (on) { sync_t sync = {.fenced = 1, .fence = 35}; completed = 25; pa_guard = 39; relaxed = 1; Cpu cpu = {0}; cpu.x[0] = (uintptr_t)&sync; cpu.x[1] = UINT64_MAX; cpu.x[30] = site; @@ -130,7 +150,7 @@ def main(): path.write_text(HARNESS.replace("PRODUCTION", source[start:end])) subprocess.run([args.cc, "-std=c11", "-O2", "-Wall", "-Werror", str(path), "-o", str(exe)], check=True) for version in (110, 121, 143, 999): - for enabled in (0, 1): + for enabled in (0, 1, 2): for shadow in (0, 1): subprocess.run([str(exe), str(version), str(enabled), str(shadow)], check=True) From f5fbe9df151829f37b2f18dce21ca12bf5a73114 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 09:42:18 +0300 Subject: [PATCH 2/7] The sync-ahead check counts every frame as if the GPU were behind, and compares bytes as well as written pages Whether the game writes what the GPU still reads does not depend on this machine's timing, so check mode now takes the late wait's value at every frame wait (the frames where the GPU really was behind are counted apart). The set's bytes are copied at the snap: a written page counts its changed bytes too, and the mirror sync copies are narrowed to the bytes the frame binds from the mirror (class bound), since a page written only outside them races nothing it reads. Co-Authored-By: Claude Opus 5.5 --- runtime/nvn_api_core.c | 9 +- runtime/vk.h | 9 +- runtime/vk_ahead.c | 187 +++++++++++++++++++++++++++++--------- runtime/vk_ahead.h | 12 ++- runtime/vk_memory.c | 8 +- tests/sync_ahead_check.py | 9 +- 6 files changed, 173 insertions(+), 61 deletions(-) diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 187bee54..5d6a1879 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -541,9 +541,10 @@ NVN_API(nvnSyncWait) { const GfxBackend *b = gfx_backend(); uint64_t want = s->fence, lag = s->fence; // Only the game's per-frame wait lags, and not a poll (timeout 0). - int mode = timeout && c->x[30] == frame_wait_lr() ? sync_mode() : 0; + int mode = timeout && c->x[30] == frame_wait_lr() ? sync_mode() : 0, behind = b->fence_done() < want; if (mode == 2) vk_ahead_frame_entry(); - if (mode && b->fence_done() < want) { + // check: every frame as if the GPU were behind (the question is what the game writes, not this machine's timing) + if (mode == 2 || (mode && behind)) { uint64_t p0 = __atomic_load_n(&pa_fence[0], __ATOMIC_ACQUIRE); uint64_t p1 = __atomic_load_n(&pa_fence[1], __ATOMIC_ACQUIRE); if (want <= p0 && want > p1) lag = p1; @@ -560,7 +561,7 @@ NVN_API(nvnSyncWait) { vk_mem_shadow_fence(); // a shadow sync that read the old guard finishes before the game writes again } if (b->fence_done() >= want) { - if (mode == 2) vk_ahead_frame_lag(s->fence, lag); + if (mode == 2) vk_ahead_frame_lag(s->fence, lag, behind); b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); if (mode == 2) vk_ahead_frame_done(1); ret_x(c, NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED); @@ -571,7 +572,7 @@ NVN_API(nvnSyncWait) { NVN_BLOCKING(); ok = b->fence_wait(want, timeout); } - if (mode == 2) vk_ahead_frame_lag(s->fence, lag); + if (mode == 2) vk_ahead_frame_lag(s->fence, lag, behind); if (ok) b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); if (mode == 2) vk_ahead_frame_done(ok); ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); diff --git a/runtime/vk.h b/runtime/vk.h index 9600b528..b9179a1e 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -272,12 +272,15 @@ void vk_mem_pick(VkMemSpan *sp); void vk_ahead_span(const VkMemSpan *sp, uint64_t addr, uint64_t size, int use); #define VK_AHEAD_READ(sp, addr, size, use) \ do { if (__builtin_expect(vk_ahead_on, 0)) vk_ahead_span(&(sp), (addr), (size), (use)); } while (0) -int vk_mem_span_inplace(const VkMemSpan *sp); // the span is guest memory itself (no mirror, no shadow, no device memory) +// 1: the span is guest memory itself (no mirror, no shadow, no device memory); 2: an imported pool's VRAM mirror; else 0 +int vk_mem_span_inplace(const VkMemSpan *sp); // The write watch over the ranges: every write made so far stamped into the page generations; untracked[i] 1 where // range i has no write watch. Returns the newest generation. uint32_t vk_mem_ahead_observe(const uint64_t (*iv)[2], int n, uint8_t *untracked); -// Bytes of [lo, hi) in pages stamped after generation since; *first: the first such byte. -uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, uint64_t *first); +// Bytes of [lo, hi) in pages stamped after generation since; fn (may be NULL) gets each such page's part of the range, +// in address order. +uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, void (*fn)(uint64_t, uint64_t, void *), + void *arg); int vk_mem_pool_info(uint64_t addr, uint64_t *lo, uint64_t *size, uint32_t *flags, int *mirrored); // 0: no pool void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size); // A shader's storage-buffer write: goes back through the write-back merge (vk_memory.c), which keeps the words the GPU diff --git a/runtime/vk_ahead.c b/runtime/vk_ahead.c index d11e7a0b..dc0b24e0 100644 --- a/runtime/vk_ahead.c +++ b/runtime/vk_ahead.c @@ -33,7 +33,7 @@ int vk_ahead_on; static const char *const cls_name[AH_N + 1] = {"vb", "ib", "ubo", "ssbo", "ind", "copy", "msync", "deswz", "wb", "wbm", - "all"}; + "bound", "all"}; #define NOTE_MAX 65536 // notes not yet taken by a frame wait #define SET_MAX 8192 // merged ranges of one frame set, per class and the union together @@ -47,9 +47,10 @@ static const char *const cls_name[AH_N + 1] = {"vb", "ib", "ubo", "ssbo", "ind", #define SMALL_FRAMES 16 #define OFF_N 32 // offender table #define OFF_LOG 24 // first-sighting lines +#define COPY_MAX (32ull << 20) // bytes of a set's union copied at its snap, for the byte compare typedef struct { uint64_t lo, hi, fence; uint8_t cls; } Note; -typedef struct { uint64_t lo, hi; uint8_t cls, untracked; uint32_t hash0, nhash; } SetR; +typedef struct { uint64_t lo, hi, coff; uint8_t cls, untracked; uint32_t hash0, nhash; } SetR; // coff: in copybuf typedef struct { uint64_t lo, hi, fence; uint8_t kind; } GWrite; static PlatMutex ah_lock = PLAT_MUTEX_INIT; // notes, gw, rd, small, lag context @@ -72,19 +73,24 @@ static SetR set[SET_MAX]; static int nset; static uint64_t iv[IV_MAX][2]; static uint8_t iv_untr[IV_MAX]; -static int niv, armed; +static int niv, armed, nclass; // set: [0, nclass) per class by (class, address), then the union by address +static int bound0, bound1; // set[bound0, bound1): the BOUND ranges +static uint8_t *copybuf; static uint32_t set_gen; static uint64_t hashes[HASH_MAX]; static int nhash; typedef struct { - uint64_t frames, lagged, read[AH_N + 1], hit[AH_N + 1], hit_frames[AH_N + 1], hit_max[AH_N + 1]; + // hit: bytes in pages written after the wait; hitb: bytes that changed; mb: MSYNC bytes N also binds (BOUND) + uint64_t frames, lagged, behind, read[AH_N + 1], hit[AH_N + 1], hit_frames[AH_N + 1], hit_max[AH_N + 1]; + uint64_t hitb[AH_N + 1], hitb_frames[AH_N + 1], hitb_max[AH_N + 1], read_mb, hit_mb, hit_mb_frames, hit_mb_max; + uint64_t uncopied; uint64_t inval, inval_stale, inval_stale_unknown, inval_stale_bytes, late[AHW_N], small, small_stale, small_oneoff; uint64_t cost_ns, set_joined, set_lost, untracked, hashed, hash_skipped, ivq; } Stat; static Stat win, tot; // ah_lock for the readback counters, game thread for the rest (summed under ah_lock) -typedef struct { uint64_t pool, size, bytes, frames, first; uint32_t flags; int cls, mirrored; } Offender; +typedef struct { uint64_t pool, size, bytes, changed, frames, first; uint32_t flags; int cls, mirrored; } Offender; static Offender off[OFF_N]; static int noff, nofflog; static uint64_t last_log_ns; @@ -154,7 +160,9 @@ void vk_ahead_read(uint64_t addr, uint64_t size, int cls, uint64_t fence) { } void vk_ahead_span(const VkMemSpan *sp, uint64_t addr, uint64_t size, int use) { - if (!vk_mem_span_inplace(sp)) return; + int k = vk_mem_span_inplace(sp); + if (!k || (k == 2 && use == VK_MUSE_GPU_WRITE)) return; // a mirror's writes go back as WB / WBM + if (k == 2) { vk_ahead_read(addr, size, AH_BOUND, 0); return; } int cls = use == VK_MUSE_VB ? AH_VB : use == VK_MUSE_IB ? AH_IB : use == VK_MUSE_UBO ? AH_UBO : use == VK_MUSE_IND ? AH_IND : use == VK_MUSE_COPY_SRC ? AH_COPY : use == VK_MUSE_GPU_WRITE ? AH_WB : AH_SSBO; vk_ahead_read(addr, size, cls, 0); @@ -271,7 +279,7 @@ static uint64_t hash_pass(SetR *s, int check) { return changed; } -static void offender(int cls, uint64_t addr, uint64_t bytes) { +static void offender(int cls, uint64_t addr, uint64_t bytes, uint64_t changed) { uint64_t lo = 0, size = 0; uint32_t flags = 0; int mirrored = 0; @@ -284,33 +292,88 @@ static void offender(int cls, uint64_t addr, uint64_t bytes) { for (int k = 1; k < noff; k++) if (off[k].bytes < off[m].bytes) m = k; i = m; } else noff++; - off[i] = (Offender){lo, size, 0, 0, addr - lo, flags, cls, mirrored}; + off[i] = (Offender){lo, size, 0, 0, 0, addr - lo, flags, cls, mirrored}; if (nofflog++ < OFF_LOG) rt_log("sync-ahead check: frame %llu: %s range written after the wait: pool %#llx (%.1f MiB, flags %#x, %s) " - "+%#llx, %llu bytes", (unsigned long long)frame_no, cls_name[cls], (unsigned long long)lo, - size / 1048576.0, flags, mirrored ? "mirrored" : "in place", (unsigned long long)(addr - lo), - (unsigned long long)bytes); + "+%#llx, %llu bytes in written pages, %llu changed", (unsigned long long)frame_no, cls_name[cls], + (unsigned long long)lo, size / 1048576.0, flags, mirrored ? "mirrored" : "in place", + (unsigned long long)(addr - lo), (unsigned long long)bytes, (unsigned long long)changed); } - off[i].bytes += bytes, off[i].frames++; + off[i].bytes += bytes, off[i].changed += changed, off[i].frames++; } +// The union range holding a (set[nclass, nset) by address), NULL: none. +static const SetR *union_at(uint64_t a) { + int lo = nclass, hi = nset; + while (lo < hi) { + int m = (lo + hi) / 2; + if (set[m].hi <= a) lo = m + 1; + else hi = m; + } + return lo < nset && set[lo].lo <= a ? &set[lo] : NULL; +} + +// Bytes of [x, y) that differ from the copy taken at the snap (0 where there is none or the memory is gone). +static uint64_t changed_bytes(uint64_t x, uint64_t y) { + const SetR *u = union_at(x); + if (!u || u->coff == UINT64_MAX || y > u->hi || !readable(x, y)) return 0; + const uint8_t *m = (const uint8_t *)(uintptr_t)x, *c = copybuf + u->coff + (x - u->lo); + uint64_t n = y - x, d = 0, i = 0; + for (; i + 8 <= n; i += 8) { + uint64_t a, b; + memcpy(&a, m + i, 8), memcpy(&b, c + i, 8); + if (a == b) continue; + for (int k = 0; k < 8; k++) d += m[i + k] != c[i + k]; + } + for (; i < n; i++) d += m[i] != c[i]; + return d; +} + +typedef struct { int cls; uint64_t first, changed, changed_bound; } Seg; + +// A written page's part [x, y) of a set range: its changed bytes, and for MSYNC those inside a BOUND range too. +static void seg_cb(uint64_t x, uint64_t y, void *arg) { + Seg *g = arg; + if (g->first == UINT64_MAX) g->first = x; + g->changed += changed_bytes(x, y); + if (g->cls != AH_MSYNC) return; + int a = bound0, b = bound1; // the first BOUND range ending after x + while (a < b) { + int m = (a + b) / 2; + if (set[m].hi <= x) a = m + 1; + else b = m; + } + for (int i = a; i < bound1 && set[i].lo < y; i++) { + uint64_t p = set[i].lo > x ? set[i].lo : x, q = set[i].hi < y ? set[i].hi : y; + if (p < q) g->changed_bound += changed_bytes(p, q); + } +} + +static void max_into(uint64_t *m, uint64_t v) { if (v > *m) *m = v; } + // The armed set against the writes since its snap. static void check_set(void) { vk_mem_ahead_observe((const uint64_t(*)[2])iv, niv, iv_untr); win.ivq += (uint64_t)niv; - uint64_t hit[AH_N + 1] = {0}; + uint64_t hit[AH_N + 1] = {0}, hitb[AH_N + 1] = {0}, mb = 0; for (int i = 0; i < nset; i++) { SetR *s = &set[i]; - uint64_t first = s->lo, b = s->untracked ? hash_pass(s, 1) : vk_mem_ahead_written(s->lo, s->hi, set_gen, &first); + if (s->cls == AH_BOUND) continue; + Seg g = {s->cls, UINT64_MAX, 0, 0}; + uint64_t b; + if (s->untracked) b = g.changed = hash_pass(s, 1), g.first = s->lo; + else b = vk_mem_ahead_written(s->lo, s->hi, set_gen, seg_cb, &g); if (!b) continue; - hit[s->cls] += b; - if (s->cls != AH_N) offender(s->cls, first, b); + hit[s->cls] += b, hitb[s->cls] += g.changed, mb += g.changed_bound; + if (s->cls != AH_N) offender(s->cls, g.first, b, g.changed); } for (int c = 0; c <= AH_N; c++) { - win.hit[c] += hit[c]; - if (hit[c]) win.hit_frames[c]++; - if (hit[c] > win.hit_max[c]) win.hit_max[c] = hit[c]; + win.hit[c] += hit[c], win.hitb[c] += hitb[c]; + win.hit_frames[c] += hit[c] != 0, win.hitb_frames[c] += hitb[c] != 0; + max_into(&win.hit_max[c], hit[c]), max_into(&win.hitb_max[c], hitb[c]); } + win.hit_mb += mb, win.hit_mb_frames += mb != 0; + max_into(&win.hit_mb_max, mb); armed = 0; } @@ -331,19 +394,36 @@ static void build_set(int n) { } else win.set_lost += take[i].hi - take[i].lo; continue; } - set[nset++] = (SetR){take[i].lo, take[i].hi, take[i].cls, 0, 0, 0}; + set[nset++] = (SetR){take[i].lo, take[i].hi, UINT64_MAX, take[i].cls, 0, 0, 0}; } int nc = nset; - for (int i = 0; i < nc; i++) win.read[set[i].cls] += set[i].hi - set[i].lo; - // The union over all classes, after them. - SetR *u = set + nc; - memcpy(u, set, sizeof *set * (size_t)nc); - qsort(u, (size_t)nc, sizeof *u, lo_cmp); - int nu = 0; + nclass = nc, bound0 = bound1 = 0; for (int i = 0; i < nc; i++) { + win.read[set[i].cls] += set[i].hi - set[i].lo; + if (set[i].cls == AH_BOUND) bound1 = i + 1; + else if (set[i].cls < AH_BOUND) bound0 = i + 1; + } + if (bound1 < bound0) bound1 = bound0; + // MSYNC bytes N also binds: both lists are sorted by address. + for (int i = 0, j = bound0; i < nc && j < bound1; i++) { + if (set[i].cls != AH_MSYNC) continue; + while (j < bound1 && set[j].hi <= set[i].lo) j++; + for (int k = j; k < bound1 && set[k].lo < set[i].hi; k++) { + uint64_t p = set[k].lo > set[i].lo ? set[k].lo : set[i].lo, q = set[k].hi < set[i].hi ? set[k].hi : set[i].hi; + if (p < q) win.read_mb += q - p; + } + } + // The union over the classes that race (not BOUND), after them. + SetR *u = set + nc; + int nr = 0; + for (int i = 0; i < nc; i++) + if (set[i].cls != AH_BOUND) u[nr++] = set[i]; + qsort(u, (size_t)nr, sizeof *u, lo_cmp); + int nu = 0; + for (int i = 0; i < nr; i++) { if (nu && u[i].lo <= u[nu - 1].hi) { if (u[i].hi > u[nu - 1].hi) u[nu - 1].hi = u[i].hi; - } else u[nu++] = (SetR){u[i].lo, u[i].hi, AH_N, 0, 0, 0}; + } else u[nu++] = (SetR){u[i].lo, u[i].hi, UINT64_MAX, AH_N, 0, 0, 0}; } nset = nc + nu; for (int i = 0; i < nu; i++) win.read[AH_N] += u[i].hi - u[i].lo; @@ -377,8 +457,19 @@ static void snap_set(void) { set_gen = vk_mem_ahead_observe((const uint64_t(*)[2])iv, niv, iv_untr); win.ivq += (uint64_t)niv; nhash = 0; + if (!copybuf && !(copybuf = malloc(COPY_MAX))) rt_fatal("vk: out of memory (sync-ahead check)"); + uint64_t used = 0; + for (int i = nclass; i < nset; i++) { + SetR *s = &set[i]; + uint64_t n = s->hi - s->lo; + if (used + n <= COPY_MAX && readable(s->lo, s->hi)) { + memcpy(copybuf + used, (const void *)(uintptr_t)s->lo, n); + s->coff = used, used += n; + } else win.uncopied += n; + } for (int i = 0; i < nset; i++) { SetR *s = &set[i]; + if (s->cls == AH_BOUND) continue; s->untracked = (uint8_t)untracked_at(s->lo); if (!s->untracked) continue; win.untracked += s->cls == AH_N ? s->hi - s->lo : 0; @@ -391,15 +482,20 @@ static void log_line(const Stat *s, const char *what) { char b[2048]; // about 1,200 bytes with every class int k = 0; double f = s->frames ? (double)s->frames : 1; - k += snprintf(b + k, sizeof b - (size_t)k, "sync-ahead check: %s %llu frames (%llu lagged): written after the wait " - "KiB/frame", what, (unsigned long long)s->frames, (unsigned long long)s->lagged); + k += snprintf(b + k, sizeof b - (size_t)k, "sync-ahead check: %s %llu frames (%llu lagged, the GPU " + "behind in %llu): written after the wait KiB/frame", what, (unsigned long long)s->frames, + (unsigned long long)s->lagged, (unsigned long long)s->behind); for (int c = 0; c <= AH_N; c++) if (s->hit[c] || c == AH_N) - k += snprintf(b + k, sizeof b - (size_t)k, " %s %.1f (in %llu frames, max %.0f)", cls_name[c], - s->hit[c] / 1024.0 / f, (unsigned long long)s->hit_frames[c], s->hit_max[c] / 1024.0); + k += snprintf(b + k, sizeof b - (size_t)k, " %s %.1f (in %llu frames, max %.0f; changed %.1f in %llu, max %.0f)", + cls_name[c], s->hit[c] / 1024.0 / f, (unsigned long long)s->hit_frames[c], s->hit_max[c] / 1024.0, + s->hitb[c] / 1024.0 / f, (unsigned long long)s->hitb_frames[c], s->hitb_max[c] / 1024.0); + k += snprintf(b + k, sizeof b - (size_t)k, " msync&bound changed %.1f (in %llu, max %.0f)", s->hit_mb / 1024.0 / f, + (unsigned long long)s->hit_mb_frames, s->hit_mb_max / 1024.0); k += snprintf(b + k, sizeof b - (size_t)k, "; read KiB/frame"); for (int c = 0; c <= AH_N; c++) if (s->read[c] || c == AH_N) k += snprintf(b + k, sizeof b - (size_t)k, " %s %.1f", cls_name[c], s->read[c] / 1024.0 / f); + k += snprintf(b + k, sizeof b - (size_t)k, " msync&bound %.1f", s->read_mb / 1024.0 / f); k += snprintf(b + k, sizeof b - (size_t)k, "; stale: invalidates %llu of %llu (%llu not read lately, %.1f KiB), late reports counter %llu samples " "%llu event %llu, writes %llu, small write-backs %llu of %llu (%llu one-off)", @@ -408,10 +504,10 @@ static void log_line(const Stat *s, const char *what) { (unsigned long long)s->late[AHW_EVENT], (unsigned long long)s->late[AHW_WRITE], (unsigned long long)s->small_stale, (unsigned long long)s->small, (unsigned long long)s->small_oneoff); snprintf(b + k, sizeof b - (size_t)k, "; game-thread cost %.3f ms/frame, %.1f queries/frame, untracked %.1f " - "KiB/frame (hashed %.1f, skipped %.1f), set over its size: %.1f KiB joined, %.1f KiB left out; note compactions %llu, gpu-write list " - "overflow %llu", s->cost_ns / 1e6 / f, s->ivq / f, s->untracked / 1024.0 / f, s->hashed / 1024.0 / f, - s->hash_skipped / 1024.0 / f, s->set_joined / 1024.0, s->set_lost / 1024.0, (unsigned long long)n_compact, - (unsigned long long)gw_over); + "KiB/frame (hashed %.1f, skipped %.1f), not copied %.1f KiB, set over its size: %.1f KiB joined, %.1f KiB " + "left out; note compactions %llu, gpu-write list overflow %llu", s->cost_ns / 1e6 / f, s->ivq / f, + s->untracked / 1024.0 / f, s->hashed / 1024.0 / f, s->hash_skipped / 1024.0 / f, s->uncopied / 1024.0, + s->set_joined / 1024.0, s->set_lost / 1024.0, (unsigned long long)n_compact, (unsigned long long)gw_over); rt_log("%s", b); } @@ -420,9 +516,10 @@ static void stat_add(Stat *d, const Stat *s) { uint64_t *y = (uint64_t *)d; for (size_t i = 0; i < sizeof *s / sizeof(uint64_t); i++) y[i] += x[i]; for (int c = 0; c <= AH_N; c++) { // maxima, not sums - d->hit_max[c] -= s->hit_max[c]; - if (s->hit_max[c] > d->hit_max[c]) d->hit_max[c] = s->hit_max[c]; + d->hit_max[c] -= s->hit_max[c], max_into(&d->hit_max[c], s->hit_max[c]); + d->hitb_max[c] -= s->hitb_max[c], max_into(&d->hitb_max[c], s->hitb_max[c]); } + d->hit_mb_max -= s->hit_mb_max, max_into(&d->hit_mb_max, s->hit_mb_max); } void vk_ahead_frame_entry(void) { @@ -432,8 +529,9 @@ void vk_ahead_frame_entry(void) { win.cost_ns += plat_now_ns() - t0; } -void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged) { +void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged, int behind) { if (!vk_ahead_on) return; + if (behind && lagged < asked) win.behind++; plat_mutex_lock(&ah_lock); frame_no++; cur_asked = asked, cur_lagged = lagged, lag_live = 1; @@ -496,9 +594,10 @@ void vk_ahead_log_final(void) { log_line(&tot, "run:"); qsort(off, (size_t)noff, sizeof *off, off_cmp); for (int i = 0; i < noff && i < 12; i++) - rt_log("sync-ahead check: offender %d: %s pool %#llx (%.1f MiB, flags %#x, %s) first at +%#llx: %.1f KiB in %llu " - "frames", i + 1, cls_name[off[i].cls], (unsigned long long)off[i].pool, off[i].size / 1048576.0, off[i].flags, - off[i].mirrored ? "mirrored" : "in place", (unsigned long long)off[i].first, off[i].bytes / 1024.0, + rt_log("sync-ahead check: offender %d: %s pool %#llx (%.1f MiB, flags %#x, %s) first at +%#llx: %.1f KiB in written " + "pages, %.1f KiB changed, in %llu frames", i + 1, cls_name[off[i].cls], (unsigned long long)off[i].pool, + off[i].size / 1048576.0, off[i].flags, off[i].mirrored ? "mirrored" : "in place", + (unsigned long long)off[i].first, off[i].bytes / 1024.0, off[i].changed / 1024.0, (unsigned long long)off[i].frames); } @@ -527,9 +626,10 @@ int vk_ahead_selftest(uint64_t a, uint64_t size) { vk_ahead_read(a + 0x10000, 0x8000, AH_MSYNC, f); vk_ahead_read(a + 0x20000, 0x1000, AH_VB, f); vk_ahead_read(a + 0x28000, 0x1000, AH_VB, f + 1); // a later frame's: stays in the list + vk_ahead_read(a + 0x12300, 0x100, AH_BOUND, f); // N binds 256 bytes of the synced range vk_ahead_gpu_write(a + 0x30000, 16, AHW_COUNTER, f); m[0x10010] = (uint8_t)(m[0x10010] + 1); // before the wait: the frame wrote it before the GPU read it - vk_ahead_frame_lag(f, f - 1); + vk_ahead_frame_lag(f, f - 1, 1); vk_ahead_frame_done(1); if (!armed || win.read[AH_MSYNC] != 0x8000 || win.read[AH_VB] != 0x1000 || win.read[AH_N] != 0x9000) bad = 2; else if (win.late[AHW_COUNTER] != 1) bad = 3; @@ -539,11 +639,12 @@ int vk_ahead_selftest(uint64_t a, uint64_t size) { m[0x3a000] = (uint8_t)(m[0x3a000] + 1); // after the wait, outside the set vk_ahead_frame_entry(); if (!bad && (armed || win.hit[AH_MSYNC] != 0x1000 || win.hit[AH_VB] || win.hit[AH_N] != 0x1000)) bad = 5; + if (!bad && (win.hitb[AH_MSYNC] != 1 || win.hitb[AH_N] != 1 || win.hit_mb != 1 || win.read_mb != 0x100)) bad = 9; plat_mutex_lock(&ah_lock); int left = nnotes; plat_mutex_unlock(&ah_lock); if (!bad && left != 1) bad = 6; - vk_ahead_frame_lag(f + 1, f + 1); // not lagged: nothing at risk + vk_ahead_frame_lag(f + 1, f + 1, 0); // not lagged: nothing at risk vk_ahead_frame_done(1); if (!bad && win.read[AH_N] != 0x9000) bad = 7; m[0x28010] = (uint8_t)(m[0x28010] + 1); diff --git a/runtime/vk_ahead.h b/runtime/vk_ahead.h index 48149cb0..8d4a1b9a 100644 --- a/runtime/vk_ahead.h +++ b/runtime/vk_ahead.h @@ -5,8 +5,9 @@ // Classes of GPU-timeline accesses of guest memory a submission makes after its recording: in-place bindings of an // imported pool without a mirror (vertex, index, uniform, storage, indirect, copy source), the VRAM mirror's sync copies, -// the GPU deswizzle of a texture's storage, and the mirror write-backs (plain copies, word merges). -enum { AH_VB, AH_IB, AH_UBO, AH_SSBO, AH_IND, AH_COPY, AH_MSYNC, AH_DESWZ, AH_WB, AH_WBM, AH_N }; +// the GPU deswizzle of a texture's storage, and the mirror write-backs (plain copies, word merges). BOUND: a binding of +// a mirrored pool, the bytes the GPU reads from the mirror; not a race of its own, it narrows MSYNC to what N reads. +enum { AH_VB, AH_IB, AH_UBO, AH_SSBO, AH_IND, AH_COPY, AH_MSYNC, AH_DESWZ, AH_WB, AH_WBM, AH_BOUND, AH_N }; // Kinds of GPU results the CPU may read back: buffer writes, counter reports, samples-passed reports, event signals. enum { AHW_WRITE, AHW_COUNTER, AHW_SAMPLES, AHW_EVENT, AHW_N }; @@ -21,10 +22,11 @@ void vk_ahead_gpu_write(uint64_t addr, uint64_t size, int kind, uint64_t fence); void vk_ahead_invalidate(uint64_t addr, uint64_t size); // nvnMemoryPoolInvalidateMappedRange void vk_ahead_small(uint64_t addr, uint64_t gpu_fence); // the fence write-back of a small target (vk_texture.c) // The game's per-frame wait (nvn_api_core.c nvnSyncWait at frame_wait_lr): entry checks the previous frame's read set; -// lag before cpu_synced gives the value a one-frame-late wait would have waited for (lagged == asked: no lag); done after -// it records this frame's read set. ok 0: the wait timed out. +// lag before cpu_synced gives the value a one-frame-late wait would have waited for (lagged == asked: no lag), counted +// whether or not the GPU had finished asked (behind: it had not, the late wait would have returned early on this +// machine); done after it records this frame's read set. ok 0: the wait timed out. void vk_ahead_frame_entry(void); -void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged); +void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged, int behind); void vk_ahead_frame_done(int ok); void vk_ahead_log_final(void); // at exit int vk_ahead_selftest(uint64_t addr, uint64_t size); // --probe-vkmem: 0 passed diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index cfc10d78..1e17ee7c 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -985,7 +985,8 @@ static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) { // ---------------------------------------------------------------- TOTK_SYNC_AHEAD=check (vk_ahead.c) int vk_mem_span_inplace(const VkMemSpan *sp) { const PoolRec *r = sp->pool; - return r && !sp->alloc && r->kind == POOL_IMPORT && !r->smap && sp->buffer == r->buf; + if (!r || sp->alloc || r->kind != POOL_IMPORT || r->smap) return 0; + return sp->buffer == r->buf ? 1 : r->mbuf && sp->buffer == r->mbuf ? 2 : 0; } uint32_t vk_mem_ahead_observe(const uint64_t (*iv)[2], int n, uint8_t *untracked) { @@ -1002,7 +1003,8 @@ uint32_t vk_mem_ahead_observe(const uint64_t (*iv)[2], int n, uint8_t *untracked return g; } -uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, uint64_t *first) { +uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, void (*fn)(uint64_t, uint64_t, void *), + void *arg) { uint64_t bytes = 0; if (hi <= lo || hi > RT_ADDRESS_LIMIT) return 0; plat_mutex_lock_shared(&gen_lock); @@ -1015,7 +1017,7 @@ uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, uint64_t for (uint64_t x = a; x < b; x += 1ull << GEN_PAGE_SHIFT) { if (ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)] <= since) continue; uint64_t s = x > lo ? x : lo, e = x + (1ull << GEN_PAGE_SHIFT) < hi ? x + (1ull << GEN_PAGE_SHIFT) : hi; - if (!bytes && first) *first = s; + if (fn) fn(s, e, arg); bytes += e - s; } } diff --git a/tests/sync_ahead_check.py b/tests/sync_ahead_check.py index a8499799..192e6b56 100644 --- a/tests/sync_ahead_check.py +++ b/tests/sync_ahead_check.py @@ -42,10 +42,11 @@ static const char *cfg_value(const char *name) { } static int entry_calls, lag_calls, done_calls, done_ok; static uint64_t lag_asked, lag_value; +static int lag_behind; static void vk_ahead_frame_entry(void) { assert(!lag_calls && !done_calls); entry_calls++; } -static void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged) { +static void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged, int behind) { assert(entry_calls == 1 && !sync_calls); // before cpu_synced: the small write-backs see the lag - lag_calls++, lag_asked = asked, lag_value = lagged; + lag_calls++, lag_asked = asked, lag_value = lagged, lag_behind = behind; } static void vk_ahead_frame_done(int ok) { assert(lag_calls == 1); done_calls++, done_ok = ok; } static int vk_mem_shadow_on(void) { return shadow; } @@ -93,6 +94,7 @@ int main(int argc, char **argv) { int lagcase = sync.fenced && timeout && lr == site && completed < target && target > 20 && target <= 40; relaxed = on && lagcase; int checked = check && sync.fenced && timeout && lr == site; + int checkcase = checked && target > 20 && target <= 40; // check: as if the GPU were behind relaxed_cases += relaxed; polls += timeout == 0; other_waits += lr != site; if (relaxed) target = 20; replay_calls = shadow_calls = sync_calls = wait_calls = 0; @@ -104,7 +106,8 @@ int main(int argc, char **argv) { assert(replay_calls == relaxed && shadow_calls == relaxed); assert(entry_calls == checked && lag_calls == checked && done_calls == checked); if (checked) { // exact wait, the late one's value told to the check - assert(lag_asked == sync.fence && lag_value == (lagcase ? 20 : sync.fence)); + assert(lag_asked == sync.fence && lag_value == (checkcase ? 20 : sync.fence)); + assert(lag_behind == (initial_done < sync.fence)); assert(done_ok == (initial_done >= target || succeed)); } assert(pa_guard == (relaxed ? sync.fence : 0)); From eec2314584efe67fb1ea29f5b6d0bf494f7a4732 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 09:47:44 +0300 Subject: [PATCH 3/7] The sync-ahead check skips repeated notes without the lock and looks up memory state once per range A binding the replay resolves again in the same submission hits a per-thread table of the last notes, and the byte compare walks the memory state once per union range instead of once per written page: the check's game-thread cost on the islands bench went from 5-9 to about 3 ms a frame. The log line gives the notes taken and the time of each step. Co-Authored-By: Claude Opus 5.5 --- runtime/vk_ahead.c | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/runtime/vk_ahead.c b/runtime/vk_ahead.c index dc0b24e0..9ae8192b 100644 --- a/runtime/vk_ahead.c +++ b/runtime/vk_ahead.c @@ -50,7 +50,8 @@ static const char *const cls_name[AH_N + 1] = {"vb", "ib", "ubo", "ssbo", "ind", #define COPY_MAX (32ull << 20) // bytes of a set's union copied at its snap, for the byte compare typedef struct { uint64_t lo, hi, fence; uint8_t cls; } Note; -typedef struct { uint64_t lo, hi, coff; uint8_t cls, untracked; uint32_t hash0, nhash; } SetR; // coff: in copybuf +// coff: in copybuf; rd: union ranges, readable at the check (-1 not looked at yet: one VirtualQuery walk per range) +typedef struct { uint64_t lo, hi, coff; uint8_t cls, untracked; int8_t rd; uint32_t hash0, nhash; } SetR; typedef struct { uint64_t lo, hi, fence; uint8_t kind; } GWrite; static PlatMutex ah_lock = PLAT_MUTEX_INIT; // notes, gw, rd, small, lag context @@ -84,7 +85,7 @@ typedef struct { // hit: bytes in pages written after the wait; hitb: bytes that changed; mb: MSYNC bytes N also binds (BOUND) uint64_t frames, lagged, behind, read[AH_N + 1], hit[AH_N + 1], hit_frames[AH_N + 1], hit_max[AH_N + 1]; uint64_t hitb[AH_N + 1], hitb_frames[AH_N + 1], hitb_max[AH_N + 1], read_mb, hit_mb, hit_mb_frames, hit_mb_max; - uint64_t uncopied; + uint64_t uncopied, notes, t_check, t_build, t_snap; // notes taken; ns in the three steps uint64_t inval, inval_stale, inval_stale_unknown, inval_stale_bytes, late[AHW_N], small, small_stale, small_oneoff; uint64_t cost_ns, set_joined, set_lost, untracked, hashed, hash_skipped, ivq; } Stat; @@ -144,9 +145,17 @@ static void compact(void) { } } +// Per thread: the notes made last, by range, so a binding the replay resolves again in the same submission (most do, +// draw after draw) costs no lock and no entry in the list. +#define SEEN_N 1024 +static _Thread_local Note seen[SEEN_N]; + void vk_ahead_read(uint64_t addr, uint64_t size, int cls, uint64_t fence) { if (!vk_ahead_on || !size || (unsigned)cls >= AH_N || addr + size < addr) return; fence = note_fence(fence); + Note *sn = &seen[((addr * 0x9E3779B97F4A7C15ull) ^ size ^ (uint64_t)cls << 7) >> 54 & (SEEN_N - 1)]; + if (sn->lo == addr && sn->hi == addr + size && sn->fence == fence && sn->cls == cls) return; + *sn = (Note){addr, addr + size, fence, (uint8_t)cls}; plat_mutex_lock(&ah_lock); Note *l = nnotes ? ¬es[nnotes - 1] : NULL; if (l && l->cls == cls && l->fence == fence && addr <= l->hi && addr + size >= l->lo) { @@ -303,7 +312,7 @@ static void offender(int cls, uint64_t addr, uint64_t bytes, uint64_t changed) { } // The union range holding a (set[nclass, nset) by address), NULL: none. -static const SetR *union_at(uint64_t a) { +static SetR *union_at(uint64_t a) { int lo = nclass, hi = nset; while (lo < hi) { int m = (lo + hi) / 2; @@ -315,8 +324,10 @@ static const SetR *union_at(uint64_t a) { // Bytes of [x, y) that differ from the copy taken at the snap (0 where there is none or the memory is gone). static uint64_t changed_bytes(uint64_t x, uint64_t y) { - const SetR *u = union_at(x); - if (!u || u->coff == UINT64_MAX || y > u->hi || !readable(x, y)) return 0; + SetR *u = union_at(x); + if (!u || u->coff == UINT64_MAX || y > u->hi) return 0; + if (u->rd < 0) u->rd = (int8_t)readable(u->lo, u->hi); + if (!u->rd) return 0; const uint8_t *m = (const uint8_t *)(uintptr_t)x, *c = copybuf + u->coff + (x - u->lo); uint64_t n = y - x, d = 0, i = 0; for (; i + 8 <= n; i += 8) { @@ -394,7 +405,7 @@ static void build_set(int n) { } else win.set_lost += take[i].hi - take[i].lo; continue; } - set[nset++] = (SetR){take[i].lo, take[i].hi, UINT64_MAX, take[i].cls, 0, 0, 0}; + set[nset++] = (SetR){take[i].lo, take[i].hi, UINT64_MAX, take[i].cls, 0, -1, 0, 0}; } int nc = nset; nclass = nc, bound0 = bound1 = 0; @@ -423,7 +434,7 @@ static void build_set(int n) { for (int i = 0; i < nr; i++) { if (nu && u[i].lo <= u[nu - 1].hi) { if (u[i].hi > u[nu - 1].hi) u[nu - 1].hi = u[i].hi; - } else u[nu++] = (SetR){u[i].lo, u[i].hi, UINT64_MAX, AH_N, 0, 0, 0}; + } else u[nu++] = (SetR){u[i].lo, u[i].hi, UINT64_MAX, AH_N, 0, -1, 0, 0}; } nset = nc + nu; for (int i = 0; i < nu; i++) win.read[AH_N] += u[i].hi - u[i].lo; @@ -505,9 +516,11 @@ static void log_line(const Stat *s, const char *what) { (unsigned long long)s->small_stale, (unsigned long long)s->small, (unsigned long long)s->small_oneoff); snprintf(b + k, sizeof b - (size_t)k, "; game-thread cost %.3f ms/frame, %.1f queries/frame, untracked %.1f " "KiB/frame (hashed %.1f, skipped %.1f), not copied %.1f KiB, set over its size: %.1f KiB joined, %.1f KiB " - "left out; note compactions %llu, gpu-write list overflow %llu", s->cost_ns / 1e6 / f, s->ivq / f, + "left out; note compactions %llu, gpu-write list overflow %llu; per frame %.0f notes, check %.3f ms, build %.3f ms, " + "snap %.3f ms", s->cost_ns / 1e6 / f, s->ivq / f, s->untracked / 1024.0 / f, s->hashed / 1024.0 / f, s->hash_skipped / 1024.0 / f, s->uncopied / 1024.0, - s->set_joined / 1024.0, s->set_lost / 1024.0, (unsigned long long)n_compact, (unsigned long long)gw_over); + s->set_joined / 1024.0, s->set_lost / 1024.0, (unsigned long long)n_compact, (unsigned long long)gw_over, + s->notes / f, s->t_check / 1e6 / f, s->t_build / 1e6 / f, s->t_snap / 1e6 / f); rt_log("%s", b); } @@ -526,7 +539,8 @@ void vk_ahead_frame_entry(void) { if (!vk_ahead_on || !armed) return; uint64_t t0 = plat_now_ns(); check_set(); - win.cost_ns += plat_now_ns() - t0; + uint64_t dt = plat_now_ns() - t0; + win.cost_ns += dt, win.t_check += dt; } void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged, int behind) { @@ -563,8 +577,11 @@ void vk_ahead_frame_done(int ok) { if (ok) { win.frames++; if (l < a) win.lagged++; + uint64_t t1 = plat_now_ns(); build_set(n); + uint64_t t2 = plat_now_ns(); snap_set(); + win.notes += (uint64_t)n, win.t_build += t2 - t1, win.t_snap += plat_now_ns() - t2; } uint64_t now = plat_now_ns(); win.cost_ns += now - t0; From df10f20fe42438bb3ab31f8a1d06a916e0389a60 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 10:19:26 +0300 Subject: [PATCH 4/7] The sync-ahead check's offender table counts a frame once per pool and class Co-Authored-By: Claude Opus 5.5 --- runtime/vk_ahead.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/runtime/vk_ahead.c b/runtime/vk_ahead.c index 9ae8192b..64c56d78 100644 --- a/runtime/vk_ahead.c +++ b/runtime/vk_ahead.c @@ -91,7 +91,7 @@ typedef struct { } Stat; static Stat win, tot; // ah_lock for the readback counters, game thread for the rest (summed under ah_lock) -typedef struct { uint64_t pool, size, bytes, changed, frames, first; uint32_t flags; int cls, mirrored; } Offender; +typedef struct { uint64_t pool, size, bytes, changed, frames, first, last; uint32_t flags; int cls, mirrored; } Offender; static Offender off[OFF_N]; static int noff, nofflog; static uint64_t last_log_ns; @@ -301,14 +301,15 @@ static void offender(int cls, uint64_t addr, uint64_t bytes, uint64_t changed) { for (int k = 1; k < noff; k++) if (off[k].bytes < off[m].bytes) m = k; i = m; } else noff++; - off[i] = (Offender){lo, size, 0, 0, 0, addr - lo, flags, cls, mirrored}; + off[i] = (Offender){lo, size, 0, 0, 0, addr - lo, 0, flags, cls, mirrored}; if (nofflog++ < OFF_LOG) rt_log("sync-ahead check: frame %llu: %s range written after the wait: pool %#llx (%.1f MiB, flags %#x, %s) " "+%#llx, %llu bytes in written pages, %llu changed", (unsigned long long)frame_no, cls_name[cls], (unsigned long long)lo, size / 1048576.0, flags, mirrored ? "mirrored" : "in place", (unsigned long long)(addr - lo), (unsigned long long)bytes, (unsigned long long)changed); } - off[i].bytes += bytes, off[i].changed += changed, off[i].frames++; + off[i].bytes += bytes, off[i].changed += changed; + if (off[i].last != frame_no || !off[i].frames) off[i].frames++, off[i].last = frame_no; // once per frame } // The union range holding a (set[nclass, nset) by address), NULL: none. From b558317a7a70c811a3bb1101a1807380876c1bcd Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 10:42:03 +0300 Subject: [PATCH 5/7] TOTK_SYNC_AHEAD=1 works without shadow storage: the frame wait is answered one frame late with the GPU reading the game's memory in place, as Savage Wind does With shadow off the back end turns on guards, each with its own switch: the mirror syncs copy the synced pages into staging on the CPU at submit and wait for an earlier write-back into the same range first (TOTK_SYNC_AHEAD_MSYNC), a texture the CPU rewrote takes the CPU deswizzle (TOTK_SYNC_AHEAD_DESWZ), an invalidate of a range not read lately waits for the pending GPU write into it, from a bounded list pruned at completion (TOTK_SYNC_AHEAD_READS), and the frame wait waits up to the value of a one-off small target (Savage Wind's 16-frame, 64-entry rule, TOTK_SYNC_AHEAD_ONESHOT) or, off by default, of a samples-passed report (TOTK_SYNC_AHEAD_QUERIES). TOTK_SYNC_AHEAD_CHECK=1 runs the check-mode detector with the late wait itself. The shadow form and the default are unchanged. Co-Authored-By: Claude Opus 5.5 --- runtime/nvn_api.h | 2 +- runtime/nvn_api_core.c | 45 +++++--- runtime/vk.h | 4 + runtime/vk_ahead.c | 68 +++++++++++- runtime/vk_ahead.h | 19 +++- runtime/vk_backend.c | 1 + runtime/vk_exec_xfer.c | 1 + runtime/vk_memory.c | 226 ++++++++++++++++++++++++++++++++++++-- runtime/vk_texture.c | 53 ++++++++- tests/sync_ahead_check.py | 51 ++++++--- 10 files changed, 418 insertions(+), 52 deletions(-) diff --git a/runtime/nvn_api.h b/runtime/nvn_api.h index a5bcbe3e..82dce1dd 100644 --- a/runtime/nvn_api.h +++ b/runtime/nvn_api.h @@ -49,7 +49,7 @@ uint64_t nvn_new_serial(void); // next tex_t / prog_ // that assumption too early. void nvn_sync_ahead_present(void); uint64_t nvn_sync_ahead_guard(void); -int nvn_sync_ahead_on(void); // 1 when the frame-ahead wait is in effect: TOTK_SYNC_AHEAD on and not refused (no shadow storage, no known wait site) +int nvn_sync_ahead_on(void); // 1 when the frame-ahead wait is in effect: TOTK_SYNC_AHEAD=1 and a wait site known for this version int vk_mem_shadow_on(void); // vk_memory.c: 1 with shadow storage on (valid once the Vulkan back end is up) void vk_mem_shadow_fence(void); // vk_memory.c: returns once every shadow sync that read the old guard is done // vk_memory.c: the game is about to read [addr, addr+size) of pool storage; waits for a GPU write into it that the game diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 5d6a1879..afa2da6c 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -483,10 +483,17 @@ NVN_API(nvnQueueFenceSync) { // GPU (about 11 ms from the submit), and the wait sets the frame time. With the switch on, that wait, and only that // one, waits for the frame before it instead. What the game reads back of GPU results after the wait can then be a // frame old; a buffer that holds no older result is waited for when the game invalidates it (vk_memory.c -// vk_mem_shadow_read). The game's CPU writes to memory the GPU still reads reach the GPU only through shadow copies, which are -// queued for the GPU or wait (nvn_sync_ahead_guard). The wait still lasts until the replay thread submitted the asked -// value, so no copy of that value itself picks up those writes. pa_fence: the newest fence at the latest present and -// the one before, present thread only for writes. pa_guard: the newest value the game was told is done early. +// vk_mem_shadow_read). The wait still lasts until the replay thread submitted the asked value, so every copy the +// replay makes of the game's memory at record time (uniform snapshots, CPU texture uploads, the staged mirror syncs) +// holds the asked frame's bytes before the game writes the next. Two forms, by shadow storage: +// - shadow on: the game's CPU writes to memory the GPU still reads reach the GPU only through shadow copies, which are +// queued for the GPU or wait (nvn_sync_ahead_guard). +// - shadow off (in place, Savage Wind's form): the GPU reads the game's own memory; the back end's guards (vk_ahead.c +// vk_ahead_init) keep its GPU-timeline reads off memory the game writes after the wait, and a value whose result the +// game reads at once (vk_ahead_oneshot: a one-off small target, a samples-passed report with +// TOTK_SYNC_AHEAD_QUERIES) is waited for, only up to that value. +// pa_fence: the newest fence at the latest present and the one before, present thread only for writes. pa_guard: the +// newest value the game was told is done early. static uint64_t pa_fence[2], pa_guard; // The return address of that wait. 1.1.0: 0x7000b39aa4, 1.2.1: 0x7000b23dfc, 1.4.3: 0x70002db99c. The second // nvnSyncWait in 1.1.0 f_7000b39a00 and 1.4.3 f_70002db8ac uses the per-frame sync, like on 1.2.1; the first @@ -502,14 +509,6 @@ static int sync_mode(void) { if (mode < 0) { const char *v = cfg_value("TOTK_SYNC_AHEAD"); int on = strcmp(v, "0") != 0 && strcmp(v, "check") != 0, check = !strcmp(v, "check"); - // Without shadow storage the GPU reads the game's own memory, which the game writes ahead: nothing guards - // that. The Vulkan back end sets the mode while it starts, before the game's first present or wait; the null - // back end never does. - if (on && !vk_mem_shadow_on()) { - rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage (shadow = 1 in totk.ini or TOTK_SHADOW=1 on native Windows); " - "the game waits for the GPU as before"); - on = 0; - } if ((on || check) && !frame_wait_lr()) { rt_log("nvn: TOTK_SYNC_AHEAD: no frame wait site is known for this game version; the game waits for the " "GPU as before"); @@ -518,7 +517,11 @@ static int sync_mode(void) { // The depth query patch (set qstamp, patches-v143.txt) is switched by TOTK_SYNC_AHEAD too: a refused switch // leaves the game's own frame number check, as with the switch off. if (!on) rt_patch_switch(rt_patch_set("qstamp"), 0); - if (on) rt_log("nvn: TOTK_SYNC_AHEAD: a wait for the frame presented last waits for the frame before it"); + // The Vulkan back end decides the mode while it starts (vk_memory.c mirror_mode), after it knows whether shadow + // storage is on, before the game's first present or wait. + if (on) + rt_log("nvn: TOTK_SYNC_AHEAD: a wait for the frame presented last waits for the frame before it (%s)", + vk_mem_shadow_on() ? "shadow storage" : "in place, with the back end's guards"); mode = on ? 1 : check ? 2 : 0; } return mode; @@ -542,12 +545,18 @@ NVN_API(nvnSyncWait) { uint64_t want = s->fence, lag = s->fence; // Only the game's per-frame wait lags, and not a poll (timeout 0). int mode = timeout && c->x[30] == frame_wait_lr() ? sync_mode() : 0, behind = b->fence_done() < want; - if (mode == 2) vk_ahead_frame_entry(); + // The detector (vk_ahead.c): check mode, or the switch on with TOTK_SYNC_AHEAD_CHECK=1. + int det = mode && vk_ahead_on; + if (det) vk_ahead_frame_entry(); // check: every frame as if the GPU were behind (the question is what the game writes, not this machine's timing) if (mode == 2 || (mode && behind)) { uint64_t p0 = __atomic_load_n(&pa_fence[0], __ATOMIC_ACQUIRE); uint64_t p1 = __atomic_load_n(&pa_fence[1], __ATOMIC_ACQUIRE); if (want <= p0 && want > p1) lag = p1; + if (mode == 1 && lag < want) { // a value in (lag, want] whose result the game reads at once: up to it + uint64_t o = vk_ahead_oneshot(lag, want); + if (o > lag) lag = o; + } if (mode == 1) want = lag; } if (want != s->fence) { // the replay of the asked value is done first: the guard then covers every copy of it @@ -561,9 +570,9 @@ NVN_API(nvnSyncWait) { vk_mem_shadow_fence(); // a shadow sync that read the old guard finishes before the game writes again } if (b->fence_done() >= want) { - if (mode == 2) vk_ahead_frame_lag(s->fence, lag, behind); + if (det) vk_ahead_frame_lag(s->fence, lag, behind); b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); - if (mode == 2) vk_ahead_frame_done(1); + if (det) vk_ahead_frame_done(1); ret_x(c, NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED); return; } @@ -572,9 +581,9 @@ NVN_API(nvnSyncWait) { NVN_BLOCKING(); ok = b->fence_wait(want, timeout); } - if (mode == 2) vk_ahead_frame_lag(s->fence, lag, behind); + if (det) vk_ahead_frame_lag(s->fence, lag, behind); if (ok) b->cpu_synced(want == s->fence ? s->fence : b->fence_done()); - if (mode == 2) vk_ahead_frame_done(ok); + if (det) vk_ahead_frame_done(ok); ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); } // void nvnEventBuilderSetDefaults(NVNeventBuilder *b) diff --git a/runtime/vk.h b/runtime/vk.h index b9179a1e..02c5a766 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -282,6 +282,7 @@ uint32_t vk_mem_ahead_observe(const uint64_t (*iv)[2], int n, uint8_t *untracked uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, void (*fn)(uint64_t, uint64_t, void *), void *arg); int vk_mem_pool_info(uint64_t addr, uint64_t *lo, uint64_t *size, uint32_t *flags, int *mirrored); // 0: no pool +void vk_mem_ahead_log(void); // TOTK_SYNC_AHEAD=1 in place: the guards' waits and copies over the run, at exit void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size); // A shader's storage-buffer write: goes back through the write-back merge (vk_memory.c), which keeps the words the GPU // did not change. @@ -531,6 +532,9 @@ uint64_t vk_texture_last_write(const tex_t *t); // timeline value of the newe size_t vk_texture_texel_span(const tex_t *t, const view_t *v, const NVNcopyRegion *r, uint64_t row, uint64_t img); // bytes ReadTexels spans; 0 for a bad region int vk_texture_small_dirty(void); // a small mirrored target holds GPU texels its storage lacks void vk_texture_write_back_small(uint64_t fence); // writes those rendered by work up to fence back to storage +// TOTK_SYNC_AHEAD=1 in place: the newest value in (lo, hi] that rendered a dirty small target not written back in the +// last 16 frames (a one-off read), else 0 +uint64_t vk_texture_oneshot(uint64_t lo, uint64_t hi); // vk_sync_submit, after the replay recorded value fence into cb (no render pass open, cb the recorder's part): copies // the small targets that value rendered into staging so their write-back later needs no GPU wait (TOTK_WB_EAGER). // cb VK_NULL_HANDLE: the value recorded no part, the candidates for it are dropped. diff --git a/runtime/vk_ahead.c b/runtime/vk_ahead.c index 64c56d78..ed69e732 100644 --- a/runtime/vk_ahead.c +++ b/runtime/vk_ahead.c @@ -96,15 +96,70 @@ static Offender off[OFF_N]; static int noff, nofflog; static uint64_t last_log_ns; +VkAheadGuards vk_ahead_g; +static uint64_t oneshot_q; // the newest value with a samples-passed report (TOTK_SYNC_AHEAD_QUERIES) +static uint64_t n_oneshot_tex, n_oneshot_q; // game thread: frame waits made exact up to a one-off value + void vk_ahead_init(void) { - vk_ahead_on = !strcmp(cfg_value("TOTK_SYNC_AHEAD"), "check"); + int check = !strcmp(cfg_value("TOTK_SYNC_AHEAD"), "check"); + memset(&vk_ahead_g, 0, sizeof vk_ahead_g); + if (!check && !vk_mem_shadow_on() && nvn_sync_ahead_on()) { + VkAheadGuards *g = &vk_ahead_g; + g->lag = 1; + g->msync = rt_knob("SYNC_AHEAD_MSYNC", "1"); + g->deswz = rt_knob("SYNC_AHEAD_DESWZ", "1"); + g->reads = rt_knob("SYNC_AHEAD_READS", "1"); + g->oneshot = rt_knob("SYNC_AHEAD_ONESHOT", "1"); + // Off: TotK reads its samples-passed reports two updates after it asked (1.2.1 f_70010c3298: a query's state + // goes 1 -> 2 or 3 at one update, 2 -> 3 at the next, and the report is read in state 3), by then the frame + // wait one frame late has covered that frame. Savage Wind (BOTW reads them right after the wait) has it on Mali. + g->queries = rt_knob("SYNC_AHEAD_QUERIES", "0"); + rt_log("sync-ahead: in place (no shadow storage), memory %s; guards: mirror syncs staged %d, deswizzle of " + "rewritten textures on the CPU %d, invalidates wait for the writer %d, one-off small targets %d, " + "samples-passed reports %d", vk_mirror_on ? "mirror" : "import", g->msync, g->deswz, g->reads, g->oneshot, + g->queries); + if (!vk_mirror_on) + rt_log("sync-ahead: warning: import memory: vertex, index and storage data are read in place on the GPU, " + "no guard covers them"); + } + vk_ahead_on = check || (vk_ahead_g.lag && rt_knob("SYNC_AHEAD_CHECK", "0")); if (!vk_ahead_on) return; notes = malloc(sizeof *notes * NOTE_MAX); take = malloc(sizeof *take * NOTE_MAX); if (!notes || !take) rt_fatal("vk: out of memory (sync-ahead check)"); - rt_log("sync-ahead check: the frame wait stays exact; logging what a one-frame-late wait without shadow storage " - "would race (%s)", vk_mem_shadow_on() ? "shadow storage is on: the counts describe the in-place form only " - "approximately" : "in place"); + if (vk_ahead_g.lag) + rt_log("sync-ahead check: with the late wait itself: logging what the CPU writes after the wait into memory " + "frames in flight still read on the GPU timeline, after the guards"); + else + rt_log("sync-ahead check: the frame wait stays exact; logging what a one-frame-late wait without shadow storage " + "would race (%s)", vk_mem_shadow_on() ? "shadow storage is on: the counts describe the in-place form only " + "approximately" : "in place"); +} + +void vk_ahead_query(uint64_t fence) { + uint64_t v = __atomic_load_n(&oneshot_q, __ATOMIC_RELAXED); + while (fence > v && !__atomic_compare_exchange_n(&oneshot_q, &v, fence, 1, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) {} +} + +uint64_t vk_ahead_oneshot(uint64_t lo, uint64_t hi) { + if (!vk_ahead_g.lag) return 0; + uint64_t v = 0; + if (vk_ahead_g.queries) { + uint64_t q = __atomic_load_n(&oneshot_q, __ATOMIC_ACQUIRE); + if (q > lo && q <= hi && q > vk_sync_fence_done()) v = q, n_oneshot_q++; + } + if (vk_ahead_g.oneshot && v < hi) { + uint64_t t = vk_texture_oneshot(v > lo ? v : lo, hi); + if (t) v = t, n_oneshot_tex++; + } + return v; +} + +void vk_ahead_log_guards(void) { + if (!vk_ahead_g.lag) return; + vk_mem_ahead_log(); + rt_log("sync-ahead: frame waits made exact up to a one-off value: %llu for small targets, %llu for " + "samples-passed reports", (unsigned long long)n_oneshot_tex, (unsigned long long)n_oneshot_q); } static uint64_t note_fence(uint64_t fence) { @@ -198,9 +253,10 @@ void vk_ahead_gpu_write(uint64_t addr, uint64_t size, int kind, uint64_t fence) void vk_ahead_invalidate(uint64_t addr, uint64_t size) { if (!vk_ahead_on || !size) return; plat_mutex_lock(&ah_lock); - uint64_t now = vk_sync_issued(), stale = 0; + // with the late wait itself, after the guard: a write done by now is no stale read + uint64_t now = vk_sync_issued(), stale = 0, done = vk_ahead_g.lag ? vk_sync_fence_done() : 0; for (int i = 0; i < ngw; i++) - if (gw[i].fence > win_lo && gw[i].fence <= win_hi && gw[i].lo < addr + size && gw[i].hi > addr) { + if (gw[i].fence > win_lo && gw[i].fence <= win_hi && gw[i].fence > done && gw[i].lo < addr + size && gw[i].hi > addr) { uint64_t a = gw[i].lo > addr ? gw[i].lo : addr, b = gw[i].hi < addr + size ? gw[i].hi : addr + size; stale += b - a; } diff --git a/runtime/vk_ahead.h b/runtime/vk_ahead.h index 8d4a1b9a..2c02dac0 100644 --- a/runtime/vk_ahead.h +++ b/runtime/vk_ahead.h @@ -1,5 +1,7 @@ // TOTK_SYNC_AHEAD=check (vk_ahead.c): measures what answering the frame wait one frame late without shadow storage -// would race, while the wait stays exact. Off: vk_ahead_on is 0 and every hook is one test of it. +// would race, while the wait stays exact. Off: vk_ahead_on is 0 and every hook is one test of it. The same detector +// runs with the late wait itself (TOTK_SYNC_AHEAD=1, shadow off) under TOTK_SYNC_AHEAD_CHECK=1. +// TOTK_SYNC_AHEAD=1 without shadow storage: vk_ahead_g.lag, and the guards of that form (vk_ahead_init). #pragma once #include @@ -12,7 +14,19 @@ enum { AH_VB, AH_IB, AH_UBO, AH_SSBO, AH_IND, AH_COPY, AH_MSYNC, AH_DESWZ, AH_WB enum { AHW_WRITE, AHW_COUNTER, AHW_SAMPLES, AHW_EVENT, AHW_N }; extern int vk_ahead_on; -void vk_ahead_init(void); // vk_memory.c mirror_mode, once at device start: reads TOTK_SYNC_AHEAD +// The in-place form's switches, read once by vk_ahead_init; every guard is 0 without lag. A/B: TOTK_SYNC_AHEAD_. +// msync: the mirror syncs copy the synced pages into staging on the CPU at submit (vk_memory.c vk_mem_mirror_prologue); +// deswz: a texture re-uploaded after the CPU rewrote it takes the CPU deswizzle (vk_texture.c upload_gpu); reads: an +// invalidate of a range not read lately waits for the pending GPU write into it (vk_memory.c vk_mem_shadow_read); +// oneshot: a one-off small target is waited for, up to its value (vk_ahead_oneshot); queries (default 0): so is a +// samples-passed report. +typedef struct { int lag, msync, deswz, reads, oneshot, queries; } VkAheadGuards; +extern VkAheadGuards vk_ahead_g; +void vk_ahead_init(void); // vk_memory.c mirror_mode, once at device start, shadow mode known: reads TOTK_SYNC_AHEAD +// The newest value in (lo, hi] whose GPU result the game reads right after the frame wait (0: none): the in-place +// frame wait waits up to it. Game thread. +uint64_t vk_ahead_oneshot(uint64_t lo, uint64_t hi); +void vk_ahead_query(uint64_t fence); // a samples-passed report into CPU-visible memory, by the value being recorded // [addr, addr+size) of guest memory is read (or, for WB and WBM, written) on the GPU timeline by the submission of value // fence (0: the one being recorded, else the next one). void vk_ahead_read(uint64_t addr, uint64_t size, int cls, uint64_t fence); @@ -29,4 +43,5 @@ void vk_ahead_frame_entry(void); void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged, int behind); void vk_ahead_frame_done(int ok); void vk_ahead_log_final(void); // at exit +void vk_ahead_log_guards(void); // at exit: what the in-place form's waits did int vk_ahead_selftest(uint64_t addr, uint64_t size); // --probe-vkmem: 0 passed diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index bde23fbe..8efaa1bb 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -479,6 +479,7 @@ void gfx_exit_log(void) { vk_texture_eager_log(); vk_mem_shadow_read_log(); vk_ahead_log_final(); + vk_ahead_log_guards(); if (gfx_replay_async() && r_started && !plat_xchg32(&done, 1)) replay_log(game_frames, 1); } diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index bb8723ac..cd001345 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -292,6 +292,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) { rt_fatal("vk: counter report at %#llx is in no memory pool", (unsigned long long)p->addr); vk_mem_count(m.cls, VK_MUSE_GPU_WRITE, sizeof(NVNcounterData)); if (vk_ahead_on) vk_ahead_gpu_write(p->addr, sizeof(NVNcounterData), samples ? AHW_SAMPLES : AHW_COUNTER, ex->fence); + if (samples && vk_ahead_g.queries) vk_ahead_query(ex->fence); // the in-place frame wait waits up to it if (samples) oq_report(ex, p->addr); else { uint64_t *a = malloc(sizeof *a); diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 1e17ee7c..109fef9d 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -1373,7 +1373,80 @@ static SRead sread[SREAD_N]; // shadow_lock: the ranges the gam static uint64_t s_read_waits, s_read_ticks; // shadow_lock: since the last log static uint64_t r_read_waits, r_read_ticks, r_read_max; // shadow_lock: over the run (vk_mem_shadow_read_log) +// ---------------------------------------------------------------- TOTK_SYNC_AHEAD=1 in place (vk_ahead.h) +// Without shadow storage a GPU write into CPU-visible memory lands in guest memory on the GPU timeline: directly, or +// through the mirror's write-back at the end of its submission (vk_mem_mirror_end). Once the frame wait returns a frame +// early those bytes can still be on their way. aw: the writes recorded and not known done, {lo, hi, fence}; pruned +// when a value completes (aw_prune), bounded: a full list joins a new write with its nearest entry (wider, never +// narrower, so a reader waits for at least what it must). Used by the invalidates below and the staged mirror syncs. +#define AW_N 2048 +typedef struct { uint64_t lo, hi, fence; } AWrite; +static PlatMutex aw_lock = PLAT_MUTEX_INIT; // after every other lock of this file +static AWrite aw[AW_N]; +static int naw, aw_peak; +static uint64_t aw_joined; +// Run totals (aw_lock): invalidate waits, staged sync bytes and calls, CPU time of the copies, waits for an earlier +// write-back into a synced range +static uint64_t ah_rd_waits, ah_rd_ns, ah_rd_max, ah_stage_bytes, ah_stage_n, ah_stage_ns, ah_ov_waits, ah_ov_ns, ah_ov_max; +static uint64_t ah_inplace_bytes; // mirror sync bytes still copied from guest memory on the GPU (TOTK_SYNC_AHEAD_MSYNC=0) + +static void aw_prune(uint64_t done) { // aw_lock + int k = 0; + for (int i = 0; i < naw; i++) + if (aw[i].fence > done) aw[k++] = aw[i]; + naw = k; +} + +static void aw_add(uint64_t lo, uint64_t hi, uint64_t fence) { + if (hi <= lo) return; + plat_mutex_lock(&aw_lock); + for (int i = naw; i-- > 0 && i >= naw - 16;) // the same range again, as the replay notes it draw after draw + if (aw[i].lo == lo && aw[i].hi == hi) { + if (fence > aw[i].fence) aw[i].fence = fence; + plat_mutex_unlock(&aw_lock); + return; + } + if (naw == AW_N) aw_prune(vk_sync_fence_done()); + if (naw == AW_N) { + int m = 0; + uint64_t best = UINT64_MAX; + for (int i = 0; i < naw; i++) { + uint64_t d = aw[i].hi < lo ? lo - aw[i].hi : aw[i].lo > hi ? aw[i].lo - hi : 0; + if (d < best) best = d, m = i; + } + if (lo < aw[m].lo) aw[m].lo = lo; + if (hi > aw[m].hi) aw[m].hi = hi; + if (fence > aw[m].fence) aw[m].fence = fence; + aw_joined++; + } else { + aw[naw++] = (AWrite){lo, hi, fence}; + if (naw > aw_peak) aw_peak = naw; + } + plat_mutex_unlock(&aw_lock); +} + +// aw_lock: the newest value in (done, below) that writes into [lo, hi), 0: none +static uint64_t aw_writer(uint64_t lo, uint64_t hi, uint64_t done, uint64_t below) { + uint64_t w = 0; + for (int i = 0; i < naw; i++) + if (aw[i].fence > done && aw[i].fence < below && aw[i].fence > w && aw[i].lo < hi && aw[i].hi > lo) w = aw[i].fence; + return w; +} + +static void ah_time(uint64_t *n, uint64_t *ns, uint64_t *mx, uint64_t dt) { + plat_mutex_lock(&aw_lock); + (*n)++, *ns += dt; + if (mx && dt > *mx) *mx = dt; + plat_mutex_unlock(&aw_lock); +} + +// The in-place form's vk_mem_shadow_read: the rule of the shadow form (below), with the writer's own value waited for +// instead of the guard, which the writer may be far behind of. Only values up to the guard: a write of a value the game +// was not told is done is the game reading ahead of its own wait. +static void ahead_read(uint64_t addr, uint64_t size); + void vk_mem_shadow_read(uint64_t addr, uint64_t size) { + if (vk_ahead_g.reads && size) ahead_read(addr, size); if (vk_ahead_on) vk_ahead_invalidate(addr, size); if (!shadow_on || !size || !nvn_sync_ahead_guard()) return; plat_mutex_lock(&shadow_lock); @@ -1404,6 +1477,44 @@ void vk_mem_shadow_read(uint64_t addr, uint64_t size) { plat_mutex_unlock(&shadow_lock); } +static void ahead_read(uint64_t addr, uint64_t size) { + uint64_t g = nvn_sync_ahead_guard(); + if (!g) return; + uint64_t done = vk_sync_fence_done(); + plat_mutex_lock(&shadow_lock); // sread: the shadow form's table, unused without shadow storage + uint64_t now = vk_sync_issued(); + SRead *e = &sread[0]; + for (int i = 0; i < SREAD_N; i++) { + if (sread[i].addr == addr && sread[i].size == size) { e = &sread[i]; break; } + if (sread[i].at < e->at) e = &sread[i]; + } + int known = e->addr == addr && e->size == size && e->at + SREAD_AGE >= now; + plat_mutex_lock(&aw_lock); + aw_prune(done); + uint64_t w = aw_writer(addr, addr + size, done, UINT64_MAX), wait = !known && w && w <= g ? w : 0; + plat_mutex_unlock(&aw_lock); + if (known || !w || wait) *e = (SRead){addr, size, now}; + plat_mutex_unlock(&shadow_lock); + if (!wait) return; + uint64_t t0 = qpc(); + { + NVN_BLOCKING(); + vk_sync_fence_wait(wait, UINT64_MAX); + } + ah_time(&ah_rd_waits, &ah_rd_ns, &ah_rd_max, qpc() - t0); +} + +void vk_mem_ahead_log(void) { + plat_mutex_lock(&aw_lock); + rt_log("mem: sync-ahead in place: the game waited for a GPU write it reads %llu times, %.1f ms in all, longest " + "%.2f ms; mirror syncs staged on the CPU %.1f MiB in %llu prologues, %.1f ms, waited for an earlier " + "write-back %llu times, %.1f ms, longest %.2f ms; synced in place %.1f MiB; pending writes at most %d, %llu " + "joined", (unsigned long long)ah_rd_waits, qpc_ms(ah_rd_ns), qpc_ms(ah_rd_max), ah_stage_bytes / 1048576.0, + (unsigned long long)ah_stage_n, qpc_ms(ah_stage_ns), (unsigned long long)ah_ov_waits, qpc_ms(ah_ov_ns), + qpc_ms(ah_ov_max), ah_inplace_bytes / 1048576.0, aw_peak, (unsigned long long)aw_joined); + plat_mutex_unlock(&aw_lock); +} + // The run total of the waits above, once at exit, while frame-ahead is in effect: without TOTK_TRACE=perf the shadow // log lines come only at frames 600 and 1800. void vk_mem_shadow_read_log(void) { @@ -1450,6 +1561,10 @@ int vk_mem_shadow_deferred(void) { return ndq; } // have written those shadow bytes since. void vk_mem_gpu_pending(uint64_t addr, uint64_t size) { if (vk_ahead_on) vk_ahead_gpu_write(addr, size, AHW_WRITE, 0); + if (vk_ahead_g.lag && size) { + uint64_t f = vk_exec_recording_fence(); + aw_add(addr, addr + size, f ? f : vk_sync_issued() + 1); + } if (!shadow_on || !size) return; uint64_t lo = addr, hi = addr + size; plat_mutex_lock_shared(&pool_lock); @@ -2429,17 +2544,111 @@ int vk_mem_mirror_pending(void) { return n; } +#define STG_BLOCK (64ull << 20) // staging pool blocks (vk_staging) +typedef struct { VkBuffer b; VmaAllocation a; } StgFree; +static void free_shadow_stg(void *arg) { + StgFree *f = arg; + vmaDestroyBuffer(vk_ctx()->vma, f->b, f->a); + free(f); +} + +// TOTK_SYNC_AHEAD=1 in place (vk_ahead_g.msync): the sealed sync ranges a[0..n) are copied out of guest memory into +// staging now, on the replay thread, and the GPU copies the staging into the mirrors: the frame wait returns only after +// the replay submitted the value (nvn_api_core.c gfx_replay_wait_value), so the GPU gets this frame's bytes even when +// it runs the copy after the game wrote the next frame's. A range an earlier value still writes back into (aw) would +// lose those GPU words, which a copy on the GPU timeline would have read after them: that value is waited for first +// (mirror_lock released meanwhile; the sealed entries stay, as only the prologue takes them). Staging buffers of at +// most STG_BLOCK / 2, freed when fence completes. Replay thread, mirror_lock held. +static void stage_ranges(VkCommandBuffer cb, MRange *a, int n, uint64_t fence) { + uint64_t done = vk_sync_fence_done(), w = 0; + plat_mutex_lock(&aw_lock); + aw_prune(done); + for (int i = 0; i < n && naw; i++) { + uint64_t lo = a[i].r->import_lo + a[i].off, x = aw_writer(lo, lo + a[i].size, done, fence); + if (x > w) w = x; + } + plat_mutex_unlock(&aw_lock); + if (w) { + plat_mutex_unlock(&mirror_lock); + uint64_t t0 = qpc(); + vk_sync_fence_wait(w, UINT64_MAX); + ah_time(&ah_ov_waits, &ah_ov_ns, &ah_ov_max, qpc() - t0); + plat_mutex_lock(&mirror_lock); + a = pend; // the list may have grown meanwhile + } + uint64_t t0 = qpc(), bytes = 0; + qsort(a, (size_t)n, sizeof *a, range_cmp); + const uint64_t lim = STG_BLOCK / 2; + int i = 0; + uint64_t at_i = 0; // bytes of a[i] already staged + while (i < n) { + uint64_t total = 0, jo = at_i; + for (int j = i; j < n && total < lim;) { + uint64_t lim_j = a[j].r->import_hi - a[j].r->import_lo, sz = a[j].off >= lim_j ? 0 + : a[j].off + a[j].size > lim_j ? lim_j - a[j].off : a[j].size; + uint64_t left = sz - jo, take = left < lim - total ? left : lim - total; + total += take; + if (take == left) j++, jo = 0; + else jo += take; + } + if (!total) break; + VkBuffer sb; + VmaAllocation sa; + uint8_t *p = vk_staging(total, &sb, &sa); + int k = 0; + PoolRec *kr = NULL; + for (uint64_t at = 0; at < total;) { + PoolRec *r = a[i].r; + uint64_t lim_i = r->import_hi - r->import_lo, sz = a[i].off >= lim_i ? 0 + : a[i].off + a[i].size > lim_i ? lim_i - a[i].off : a[i].size; + uint64_t left = sz - at_i, take = left < total - at ? left : total - at; + if (take) { + if (kr != r && k) vkCmdCopyBuffer(cb, sb, kr->mbuf, (uint32_t)k, mregions), k = 0; + kr = r; + memcpy(p + at, (const void *)(uintptr_t)(r->import_lo + a[i].off + at_i), take); + if (k == cregions) grow((void **)&mregions, &cregions, sizeof *mregions); + mregions[k++] = (VkBufferCopy){at, a[i].off + at_i, take}; + } + at += take, at_i += take; + if (at_i == sz) i++, at_i = 0; + } + if (k) vkCmdCopyBuffer(cb, sb, kr->mbuf, (uint32_t)k, mregions); + vmaFlushAllocation(vk_ctx()->vma, sa, 0, VK_WHOLE_SIZE); + StgFree *f = malloc(sizeof *f); + if (!f) rt_fatal("vk: out of memory"); + f->b = sb, f->a = sa; + vk_completion_add(fence, free_shadow_stg, f); + bytes += total; + mrep.upload_copies += (uint64_t)k; + } + mrep.upload_bytes += bytes; + uint64_t dt = qpc() - t0; + plat_mutex_lock(&aw_lock); + ah_stage_bytes += bytes, ah_stage_n++, ah_stage_ns += dt; + plat_mutex_unlock(&aw_lock); +} + void vk_mem_mirror_prologue(VkCommandBuffer cb, uint64_t fence) { plat_mutex_lock(&mirror_lock); if (!pend_seal && snl_copied == snl_seal) { plat_mutex_unlock(&mirror_lock); return; } - if (vk_ahead_on) // the copies read guest memory when the GPU runs them + int staged = vk_ahead_g.msync && pend_seal; + if (vk_ahead_on && !staged) // the copies read guest memory when the GPU runs them for (int i = 0; i < pend_seal; i++) vk_ahead_read(pend[i].r->import_lo + pend[i].off, pend[i].size, AH_MSYNC, fence); mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); - copy_ranges(cb, pend, pend_seal, 1, &mrep.upload_bytes, &mrep.upload_copies); + if (staged) stage_ranges(cb, pend, pend_seal, fence); + else { + uint64_t b0 = mrep.upload_bytes; + copy_ranges(cb, pend, pend_seal, 1, &mrep.upload_bytes, &mrep.upload_copies); + if (vk_ahead_g.lag) { + plat_mutex_lock(&aw_lock); + ah_inplace_bytes += mrep.upload_bytes - b0; + plat_mutex_unlock(&aw_lock); + } + } if (snl_copied < snl_seal) { // the snapshots read what the uploads above wrote if (pend_seal) mbarrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); @@ -2550,6 +2759,9 @@ int vk_mem_mirror_end(VkCommandBuffer cb, uint64_t fence) { mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_STORAGE_READ_BIT); int m = wb_merge_record(cb); + if (m && vk_ahead_g.lag) + for (int i = 0; i < nsnl; i++) + aw_add(snl[i].r->import_lo + snl[i].off, snl[i].r->import_lo + snl[i].off + snl[i].size, fence); if (m && vk_ahead_on) // the merge writes guest memory when the GPU runs it for (int i = 0; i < nsnl; i++) vk_ahead_read(snl[i].r->import_lo + snl[i].off, snl[i].size, AH_WBM, fence); if (!m) { // no range list: copy them plainly @@ -2571,6 +2783,9 @@ int vk_mem_mirror_end(VkCommandBuffer cb, uint64_t fence) { mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT); for (int i = 0; i < nwbl; i++) wbl[i].r->wb_sub = msub; + if (vk_ahead_g.lag) + for (int i = 0; i < nwbl; i++) + aw_add(wbl[i].r->import_lo + wbl[i].off, wbl[i].r->import_lo + wbl[i].off + wbl[i].size, fence); if (vk_ahead_on) for (int i = 0; i < nwbl; i++) vk_ahead_read(wbl[i].r->import_lo + wbl[i].off, wbl[i].size, AH_WB, fence); mrep.wb_ranges += (uint64_t)nwbl; @@ -2843,7 +3058,6 @@ void vk_upload_end(void) { plat_rmutex_unlock(&cmd_lock); } // the headroom thread (vk_headroom.c) or a burst of uploads held every staging allocation of the render thread: one // took 47-50 ms, for a few bytes, in the first frame after a save load. Larger buffers than a block go to the default // pool. -#define STG_BLOCK (64ull << 20) #define STG_MIN_BLOCKS 2 static VmaPool stg_pool; static int stg_pool_tried; @@ -2911,12 +3125,6 @@ void *vk_staging(VkDeviceSize size, VkBuffer *buf, VmaAllocation *alloc) { return info.pMappedData; } -typedef struct { VkBuffer b; VmaAllocation a; } StgFree; -static void free_shadow_stg(void *arg) { - StgFree *f = arg; - vmaDestroyBuffer(vk_ctx()->vma, f->b, f->a); - free(f); -} // Records the deferred shadow copies (vk_mem_shadow_sync queued the ranges): the guest bytes go into staging buffers // now, the GPU copies them into the pools' shadows. The barrier before waits for every earlier command, writes and diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index b7761fac..9bc6300d 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -1035,9 +1035,11 @@ static void upload_cpu(Tex *e) { #define DESWZ_CHUNK (32ull << 20) // Uploads by path since the last perf summary (TOTK_TRACE=perf): 0 GPU, else why the CPU took it. -enum { UP_GPU, UP_MODE, UP_LINEAR, UP_RGB, UP_ASTC_CPU, UP_BPB, UP_NOT_LIVE, UP_GEOM, UP_LIMITS, UP_SCRATCH, UP_WHY }; +enum { UP_GPU, UP_MODE, UP_LINEAR, UP_RGB, UP_ASTC_CPU, UP_BPB, UP_NOT_LIVE, UP_GEOM, UP_LIMITS, UP_SCRATCH, UP_AHEAD, + UP_WHY }; static const char *const up_why_name[UP_WHY] = {"gpu", "cpu mode", "linear", "rgb", "astc on cpu", "bytes per block", - "not an imported pool", "layout", "device limits", "scratch"}; + "not an imported pool", "layout", "device limits", "scratch", + "frame ahead"}; static uint64_t up_n[UP_WHY], up_bytes[UP_WHY], up_ns[UP_WHY]; static int up_why; // upload_gpu's reason for returning 0 (texture lock held) @@ -1049,6 +1051,10 @@ static int upload_gpu(Tex *e) { int mode = vk_deswz_mode(); up_why = mode == VK_DESWZ_CPU ? UP_MODE : nvn_tex_is_linear(d) ? UP_LINEAR : e->f.expand_rgb ? UP_RGB : UP_GPU; if (up_why) return 0; + // TOTK_SYNC_AHEAD=1 in place (vk_ahead.h): the batch reads the pool when the GPU runs it, which can be after the + // game was told the frame is done and writes the storage again. A texture the CPU rewrote after its last upload is + // one the game writes while the game runs: deswizzled on the CPU now, from the bytes of this frame. + if (vk_ahead_g.deswz && e->uploaded) return up_why = UP_AHEAD, 0; int astc = e->f.astc_bw != 0, agpu = astc && vk_astc_gpu(); if (astc && (!agpu || vk_astc_mode() == VK_ASTC_CHECK)) return up_why = UP_ASTC_CPU, 0; // or ASTC's check mode int bw, bh, bpb = nvn_fmt_bpb(d->format, &bw, &bh); @@ -2489,6 +2495,41 @@ uint64_t vk_texture_dirty_fence(uint64_t addr, uint64_t size) { return f; } +// TOTK_SYNC_AHEAD=1 in place: the small targets written back lately, with the frame of the write-back. A target not in +// the table within ONEOFF_FRAMES is a one-off read (a photo, a pick), which the frame wait waits for up to the value +// that rendered it (vk_texture_oneshot); a target read back every frame or in turns gets the frame before. Adapted from +// Savage Wind Recomp runtime/nvn/vulkan/exec.c readback_schedule (https://github.com/Revivedx/SavageWindRecomp @ +// bf6146c14c37fee8781a601e5a544b93dff6d75e, MIT, Copyright (c) 2026 Revivedx and the Savage Wind Recomp contributors, +// Copyright (c) 2026 Peiikai). tex_lock. +#define ONEOFF_N 64 +#define ONEOFF_FRAMES 16 +static struct { uint64_t addr, frame; } oneoff_last[ONEOFF_N]; + +static int oneoff_streaming(uint64_t addr, uint64_t fr, int note) { + int k = 0; + while (k < ONEOFF_N - 1 && oneoff_last[k].addr && oneoff_last[k].addr != addr) k++; + int streaming = oneoff_last[k].addr == addr && oneoff_last[k].frame + ONEOFF_FRAMES >= fr; + if (note) oneoff_last[k].addr = addr, oneoff_last[k].frame = fr; + return streaming; +} + +uint64_t vk_texture_oneshot(uint64_t lo, uint64_t hi) { + if (!__atomic_load_n(&small_pending, __ATOMIC_ACQUIRE)) return 0; // nothing dirty left for a later value + uint64_t fr = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED), v = 0; + lock(); + wb_cfg_init(); + plat_mutex_lock_shared(&owned_lock); + for (int i = 0; i < nowned; i++) { + const Tex *e = owned[i]; + if (e->gpu_dirty && e->gpu_fence > lo && e->gpu_fence <= hi && e->gpu_fence > v && wb_wanted(e) && + !oneoff_streaming(e->pub.addr, fr, 0)) + v = e->gpu_fence; + } + plat_mutex_unlock_shared(&owned_lock); + unlock(); + return v; +} + // The image is the newest content only while nothing wrote the storage since the GPU render: TotK finalizes grass maps, // reuses their storage for new data and memcpys that data on, and writing the old image back first handed the copy // a stale map or stale resource bytes (wrong textures in the distance). Every hit is checked before any is written @@ -2511,6 +2552,10 @@ void vk_texture_write_back(uint64_t addr, uint64_t size) { } write_back_n(hit, k); for (int i = 0; i < k; i++) perf_count(PERF_CNT_WB); + if (vk_ahead_g.oneshot) { + uint64_t fr = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); + for (int i = 0; i < k; i++) oneoff_streaming(hit[i]->pub.addr, fr, 1); + } for (int i = k; i < n; i++) perf_count(PERF_CNT_WB_SKIP); free(hit); unlock(); @@ -2585,6 +2630,10 @@ void vk_texture_write_back_small(uint64_t fence) { } write_back_n(hit, k); for (int i = 0; i < k; i++) perf_count(PERF_CNT_WB); + if (vk_ahead_g.oneshot) { + uint64_t fr = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); + for (int i = 0; i < k; i++) oneoff_streaming(hit[i]->pub.addr, fr, 1); + } if (vk_ahead_on) for (int i = 0; i < k; i++) vk_ahead_small(hit[i]->pub.addr, hit[i]->gpu_fence); for (int i = k; i < n; i++) perf_count(PERF_CNT_WB_SKIP); diff --git a/tests/sync_ahead_check.py b/tests/sync_ahead_check.py index 192e6b56..d616dd92 100644 --- a/tests/sync_ahead_check.py +++ b/tests/sync_ahead_check.py @@ -1,4 +1,5 @@ -"""Exercise the production sync-ahead wait selection with a deterministic back end (TOTK_SYNC_AHEAD 0, 1, check). +"""Exercise the production sync-ahead wait selection with a deterministic back end (TOTK_SYNC_AHEAD 0, 1, check; shadow +storage on and off, the in-place form's wait up to a one-off value). python tests/sync_ahead_check.py [--cc path/to/clang] """ @@ -40,7 +41,13 @@ static const char *cfg_value(const char *name) { assert(!strcmp(name, "TOTK_SYNC_AHEAD")); return enabled == 2 ? "check" : enabled ? "1" : "0"; } -static int entry_calls, lag_calls, done_calls, done_ok; +static int entry_calls, lag_calls, done_calls, done_ok, oneshot_calls; +static int vk_ahead_on; // the detector: check mode, or the switch on with TOTK_SYNC_AHEAD_CHECK=1 (det) +static uint64_t oneshot, oneshot_lo, oneshot_hi; // the back end's one-off value (vk_ahead_oneshot: in (lo, hi] or 0) +static uint64_t vk_ahead_oneshot(uint64_t lo, uint64_t hi) { + oneshot_calls++, oneshot_lo = lo, oneshot_hi = hi; + return oneshot > lo && oneshot <= hi ? oneshot : 0; +} static uint64_t lag_asked, lag_value; static int lag_behind; static void vk_ahead_frame_entry(void) { assert(!lag_calls && !done_calls); entry_calls++; } @@ -70,8 +77,9 @@ PRODUCTION static uint32_t seed = 0x19ed634b; static uint32_t random_word(void) { seed ^= seed << 13; seed ^= seed >> 17; return seed ^= seed << 5; } int main(int argc, char **argv) { - assert(argc == 4); + assert(argc == 5); rt_game_version = atoi(argv[1]); enabled = atoi(argv[2]); shadow = atoi(argv[3]); + int det = atoi(argv[4]); int supported = rt_game_version == 110 || rt_game_version == 121 || rt_game_version == 143; uint64_t site = rt_game_version == 110 ? 0x7000b39aa4ull : rt_game_version == 121 ? 0x7000b23dfcull : rt_game_version == 143 ? 0x70002db99cull : 0; @@ -79,11 +87,12 @@ int main(int argc, char **argv) { qstamp_on = rt_game_version == 143 && enabled; last = 20; nvn_sync_ahead_present(); last = 40; nvn_sync_ahead_present(); - int on = enabled == 1 && shadow && supported, check = enabled == 2 && supported; + int on = enabled == 1 && supported, check = enabled == 2 && supported; // shadow off: the in-place form + vk_ahead_on = check || (on && det); assert(pa_fence[0] == (on || check ? 40 : 0)); assert(pa_fence[1] == (on || check ? 20 : 0)); assert(qstamp_on == (rt_game_version == 143 && on)); // a refused switch and check leave the game's check - int relaxed_cases = 0, polls = 0, other_waits = 0; + int relaxed_cases = 0, polls = 0, other_waits = 0, oneshot_cases = 0; for (int i = 0; i < 100000; i++) { sync_t sync = {.fenced = random_word() % 2, .fence = random_word() % 65}; uint64_t timeout = i % 3 == 0 ? 0 : i % 3 == 1 ? 1 : UINT64_MAX; @@ -91,25 +100,36 @@ int main(int argc, char **argv) { completed = random_word() % 65; uint64_t initial_done = completed, target = sync.fence; succeed = random_word() % 2; + oneshot = random_word() % 4 ? 0 : random_word() % 65; int lagcase = sync.fenced && timeout && lr == site && completed < target && target > 20 && target <= 40; - relaxed = on && lagcase; - int checked = check && sync.fenced && timeout && lr == site; + // the late wait waits for 20, or for a one-off value in (20, target] the back end names + uint64_t late = lagcase && oneshot > 20 && oneshot <= target ? oneshot : 20; + relaxed = on && lagcase && late < target; + int oneshot_asked = on && lagcase; + oneshot_cases += oneshot_asked && late > 20; + int checked = vk_ahead_on && sync.fenced && timeout && lr == site; int checkcase = checked && target > 20 && target <= 40; // check: as if the GPU were behind relaxed_cases += relaxed; polls += timeout == 0; other_waits += lr != site; - if (relaxed) target = 20; - replay_calls = shadow_calls = sync_calls = wait_calls = 0; + if (relaxed) target = late; + replay_calls = shadow_calls = sync_calls = wait_calls = oneshot_calls = 0; entry_calls = lag_calls = done_calls = 0; synced = waited = waited_timeout = replay_value = UINT64_MAX; pa_guard = 0; Cpu cpu = {0}; cpu.x[0] = (uintptr_t)&sync; cpu.x[1] = timeout; cpu.x[30] = lr; nvnSyncWait(&cpu); assert(replay_calls == relaxed && shadow_calls == relaxed); + assert(oneshot_calls == oneshot_asked); + if (oneshot_asked) assert(oneshot_lo == 20 && oneshot_hi == sync.fence); assert(entry_calls == checked && lag_calls == checked && done_calls == checked); - if (checked) { // exact wait, the late one's value told to the check + if (checked && check) { // exact wait, the late one's value told to the check assert(lag_asked == sync.fence && lag_value == (checkcase ? 20 : sync.fence)); assert(lag_behind == (initial_done < sync.fence)); assert(done_ok == (initial_done >= target || succeed)); } + if (checked && on) { // the late wait itself: the value it waited for told to the detector + assert(lag_asked == sync.fence && lag_value == (relaxed ? late : sync.fence)); + assert(done_ok == (initial_done >= target || succeed)); + } assert(pa_guard == (relaxed ? sync.fence : 0)); if (relaxed) assert(replay_value == sync.fence); if (!sync.fenced) { @@ -125,16 +145,18 @@ int main(int argc, char **argv) { } } assert(polls > 0 && other_waits > 0); - assert((relaxed_cases > 0) == on); + assert((relaxed_cases > 0) == on && (oneshot_cases > 0) == on); // Raising the guard for an older request must not lower a guard already published by another waiter. if (on) { sync_t sync = {.fenced = 1, .fence = 35}; - completed = 25; pa_guard = 39; relaxed = 1; + completed = 25; pa_guard = 39; relaxed = 1; oneshot = 0; + replay_calls = shadow_calls = sync_calls = wait_calls = entry_calls = lag_calls = done_calls = 0; Cpu cpu = {0}; cpu.x[0] = (uintptr_t)&sync; cpu.x[1] = UINT64_MAX; cpu.x[30] = site; nvnSyncWait(&cpu); assert(pa_guard == 39); } - printf("sync-ahead version %d enabled %d shadow %d: 100000 wait selections passed (%d relaxed)\n", rt_game_version, enabled, shadow, relaxed_cases); + printf("sync-ahead version %d enabled %d shadow %d detector %d: 100000 wait selections passed (%d relaxed, %d up " + "to a one-off value)\n", rt_game_version, enabled, shadow, det, relaxed_cases, oneshot_cases); } ''' @@ -155,7 +177,8 @@ def main(): for version in (110, 121, 143, 999): for enabled in (0, 1, 2): for shadow in (0, 1): - subprocess.run([str(exe), str(version), str(enabled), str(shadow)], check=True) + for det in (0, 1): + subprocess.run([str(exe), str(version), str(enabled), str(shadow), str(det)], check=True) if __name__ == "__main__": From 54b4e092cda430480ab97946760a5eca9093a79f Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 14:43:45 +0300 Subject: [PATCH 6/7] The in-place frame ahead merges every mirror write-back by words, is refused with import memory, and the probe checks its pending-write list Without shadow storage a plain GPU copy back into a mirrored range lands after the game, told the value is done a frame early, may have written other words of that range for the next frame; with TOTK_SYNC_AHEAD_WBMERGE (1 by default) the copies' and updates' write-backs go through the word merge as a shader's do. With memory = import the GPU reads vertex, index and storage data in place, and a check run on the islands saw the game rewrite about 10 KiB of them a frame after the wait, so the switch is refused there unless shadow storage is on. --probe-vkmem checks the pending-write list (newest writer below a bound, pruning, joining when full). The one-off readback rule copied from Savage Wind gets its THIRD_PARTY_NOTICES.md row. Co-Authored-By: Claude Opus 5.5 --- THIRD_PARTY_NOTICES.md | 1 + docs/architecture.md | 3 ++- runtime/nvn_api.h | 1 + runtime/nvn_api_core.c | 8 +++++++ runtime/tex_probe.c | 5 +++++ runtime/vk.h | 1 + runtime/vk_ahead.c | 12 +++++------ runtime/vk_ahead.h | 3 ++- runtime/vk_memory.c | 45 +++++++++++++++++++++++++++++++++++++-- tests/sync_ahead_check.py | 16 +++++++++----- 10 files changed, 79 insertions(+), 16 deletions(-) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 9aa0b12a..c6a8174b 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -33,6 +33,7 @@ Code copied or adapted from another project is listed here with its source, the | AMD FidelityFX SDK 1.1.4 subset (public host and shader headers, frame interpolation, optical flow and FSR 3 upscaler components, the Vulkan backend `ffx_vk.cpp` and its GLSL shaders, shared helpers), with four marked Savage Wind patches (64-byte aligned backend scratch arrays, doubled context size where `wchar_t` is 4 bytes, all six descriptor pool sizes, the upscaler's Catmull-Rom history reprojection) and one of ours (the upscaler's luma history declared `rgba16f`) | https://github.com/GPUOpen-LibrariesAndSDKs/FidelityFX-SDK/tree/v1.1.4, Copyright (C) 2024 Advanced Micro Devices, Inc., via https://github.com/Revivedx/SavageWindRecomp @ `bf6146c14c37fee8781a601e5a544b93dff6d75e` (`third_party/FidelityFX/`) | MIT | `runtime/third_party/FidelityFX/` (compiled into `totk.exe`; licence text in `runtime/third_party/FidelityFX/LICENSE.txt`, shipped as `licenses/FidelityFX-SDK-LICENSE.txt`) | | SavageWindRecomp FidelityFX build support, frame generation and FSR 3 upscaler set-up: the SDK-to-clang compat header, the glslang SPIR-V shader generator with its resource reflection and its FSR 3 upscaler permutations, the blob lookup and the backend and context creation of `runtime/nvn/vulkan/framegen.cpp`, the upscaler context with its shared images and the dispatch parameters of `runtime/nvn/vulkan/fsr3.cpp`, the pair's present spacing of `runtime/nvn/vulkan/device.c` `present_wait_turn` | https://github.com/Revivedx/SavageWindRecomp @ `bf6146c14c37fee8781a601e5a544b93dff6d75e` (`third_party/FidelityFX/ffx_compat.h`, `tools/ffx_shaders.py`, `runtime/nvn/vulkan/framegen.cpp`, `runtime/nvn/vulkan/fsr3.cpp`, `runtime/nvn/vulkan/device.c`) | MIT | `runtime/third_party/FidelityFX/ffx_compat.h`, `runtime/third_party/FidelityFX/spirv/ffx_shaders_vk.h` and `ffx_fsr3upscaler_vk.h` (output of `tools/ffx_shaders.py`), `tools/ffx_shaders.py`, `runtime/vk_fg.cpp`, `runtime/vk_fsr3.cpp`, `runtime/vk_backend.c` (`present_pair`) | | SavageWindRecomp settings menu: two-pane option rows (row look, left / right stepping, hold speed-up, description pane), the menu style and tab bar | https://github.com/Revivedx/SavageWindRecomp @ `575ad88e02f9a89a9a885d90039d48af33c982f4` (`runtime/ui/option_rows.mm`, `runtime/ui/menu.mm`), Copyright (c) 2026 Revivedx and the Savage Wind Recomp contributors, Copyright (c) 2026 Peiikai | MIT | `runtime/ui/option_rows.cpp`, `runtime/ui/option_rows.h`, `runtime/ui/menu.cpp` | +| SavageWindRecomp GPU overlap one-off readback rule: a target not read back within the last 16 frames (64-entry table) makes the frame wait wait up to the value that rendered it | https://github.com/Revivedx/SavageWindRecomp @ `bf6146c14c37fee8781a601e5a544b93dff6d75e` (`runtime/nvn/vulkan/exec.c` readback_schedule, renderer_wait_seq), Copyright (c) 2026 Revivedx and the Savage Wind Recomp contributors, Copyright (c) 2026 Peiikai | MIT | `runtime/vk_texture.c` (oneoff_streaming, vk_texture_oneshot) | | SavageWindRecomp descriptor sets allocated in growing batches per layout | https://github.com/Revivedx/SavageWindRecomp @ `740b18025fe11ba03c117dbb56668db304b5043e` (`runtime/nvn/vulkan/exec.c` vk_alloc_set), Copyright (c) 2026 Revivedx and the Savage Wind Recomp contributors, Copyright (c) 2026 Peiikai | MIT | `runtime/vk_exec_draw.c` (alloc_set_batched) | | SavageWindRecomp performance overlay: the frame ring and its one-second summary, the NVML calls, the sampling loop, the full panel's layout | https://github.com/Revivedx/SavageWindRecomp @ `575ad88e02f9a89a9a885d90039d48af33c982f4` (`runtime/perfmon.c`, `runtime/perfmon.h`, `perf_overlay_draw` in `runtime/ui/menu.mm`), Copyright (c) 2026 Revivedx and the Savage Wind Recomp contributors, Copyright (c) 2026 Peiikai | MIT | `runtime/perfmon.c`, `runtime/perfmon.h`, `runtime/ui/menu.cpp` | | Roboto Medium font (Dear ImGui's `misc/fonts/Roboto-Medium.ttf`, unmodified) | https://fonts.google.com/specimen/Roboto, via https://github.com/peiikai/LASRecomp @ `da35b83a9611d046a9a681aae668d7085954d37f` (`third_party/imgui/misc/fonts/`), Copyright 2011 Google Inc. | Apache-2.0 | `runtime/ui/fonts/Roboto-Medium.ttf` (licence text in `runtime/ui/fonts/LICENSE-Roboto.txt`), embedded by `runtime/ui/menu.cpp` | diff --git a/docs/architecture.md b/docs/architecture.md index a7ffb229..b9c545db 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -90,7 +90,8 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. - Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. -- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.1.0, 1.2.1 and 1.4.3). GPU writes into pool storage come back with their fence, so a buffer the game reads back every frame after that wait holds the result of the frame before; a buffer it has not read lately holds none, and its `nvnMemoryPoolInvalidateMappedRange` waits for the write (`vk_mem_shadow_read`): TotK reads single pixels through a buffer per request, and in the Depths zeros in one of them put Link in gloom. The exit line `mem: the game waited for a GPU write it reads` counts these waits and their time. On 1.4.3 the patch set `qstamp` (`recomp/patches-v143.txt`) lets the game take its depth query results one frame older; it follows the switch and is off when the switch is refused. The fence write-back of small render targets (`vk_texture.c`, `write_back_n`) does not wait for the GPU with `TOTK_WB_EAGER=1` (`wb_eager`, the default, but only while frame-ahead is in effect: with `sync_ahead` off or refused the setting does nothing and the write-back reads the image when it runs): `vk_sync_submit` has `vk_texture_eager_record` copy the targets that value rendered into staging buffers inside the same command buffer, so once the value has passed the write-back only copies from staging into the target's storage. Only a target that was written back within the last 64 timeline values is copied (the first write-back of a target takes the blocking path), and the staging of a target past that window, no longer wanted or released goes through the deferred free (`vk_defer_buffer_until`). A snapshot is used only when its value is the one that last rendered the target, nothing recorded a write to the image since (`wr_fence`) and the GPU has finished it; otherwise the write-back reads the image as before, and the exit line `vk: wb_eager:` counts the reasons. With frame-ahead in effect the exit line `vk: fence write-backs waited for the GPU` counts the write-backs that waited for a copy and the time they waited, for every value of the setting. `check` also reads the old way and compares. +- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. Two forms guard that. With shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows) a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.1.0, 1.2.1 and 1.4.3). GPU writes into pool storage come back with their fence, so a buffer the game reads back every frame after that wait holds the result of the frame before; a buffer it has not read lately holds none, and its `nvnMemoryPoolInvalidateMappedRange` waits for the write (`vk_mem_shadow_read`): TotK reads single pixels through a buffer per request, and in the Depths zeros in one of them put Link in gloom. The exit line `mem: the game waited for a GPU write it reads` counts these waits and their time. On 1.4.3 the patch set `qstamp` (`recomp/patches-v143.txt`) lets the game take its depth query results one frame older; it follows the switch and is off when the switch is refused. The fence write-back of small render targets (`vk_texture.c`, `write_back_n`) does not wait for the GPU with `TOTK_WB_EAGER=1` (`wb_eager`, the default, but only while frame-ahead is in effect: with `sync_ahead` off or refused the setting does nothing and the write-back reads the image when it runs): `vk_sync_submit` has `vk_texture_eager_record` copy the targets that value rendered into staging buffers inside the same command buffer, so once the value has passed the write-back only copies from staging into the target's storage. Only a target that was written back within the last 64 timeline values is copied (the first write-back of a target takes the blocking path), and the staging of a target past that window, no longer wanted or released goes through the deferred free (`vk_defer_buffer_until`). A snapshot is used only when its value is the one that last rendered the target, nothing recorded a write to the image since (`wr_fence`) and the GPU has finished it; otherwise the write-back reads the image as before, and the exit line `vk: wb_eager:` counts the reasons. With frame-ahead in effect the exit line `vk: fence write-backs waited for the GPU` counts the write-backs that waited for a copy and the time they waited, for every value of the setting. `check` also reads the old way and compares. +- Frame ahead in place. Without shadow storage the GPU reads the game's own memory, as in Savage Wind's overlap, and `vk_ahead_init` (`runtime/vk_ahead.c`) turns on guards for what the GPU reads or writes there after the recording, each with a switch `TOTK_SYNC_AHEAD_` (1 by default unless noted) for A/B runs. `MSYNC`: the VRAM mirrors' sync copies take the synced pages from a staging copy the replay thread makes at submit (`vk_memory.c` `stage_ranges`), not from guest memory when the GPU runs them; the frame wait returns only after the replay submitted the asked value, so those are that frame's bytes. A synced range that an earlier value still writes back into waits for that value first, or the copy would drop its GPU words. `WBMERGE`: every GPU write into a mirror goes back through the word merge (`wb_merge_record`), so a write-back that lands after the game wrote the range for the next frame puts back only the words the GPU changed. `DESWZ`: a texture the CPU rewrote after its last upload is deswizzled on the CPU (`upload_gpu`, reason `frame ahead`). `READS`: GPU writes into CPU-visible memory are listed with their value until it completes (bounded, `aw_add`), and an invalidate of a range not read lately waits for the writer's own value (`ahead_read`). `ONESHOT`: the frame wait waits up to the value that rendered a dirty small target not written back within the last 16 frames (Savage Wind's one-off rule, `vk_texture_oneshot`), so a photo or a pick reads its own frame, while targets read every frame or in turns get the frame before. `QUERIES` (0 by default): the same for samples-passed reports; TotK reads those two updates after it asks (1.2.1 `f_70010c3298`). With import memory (`TOTK_MEMORY=import`, the integrated-GPU mode) the GPU reads vertex, index and storage data in place, which no guard covers (a check run saw the game rewrite about 10 KiB of them a frame after the wait, particles mostly), so the switch is refused there without shadow storage. The exit lines `mem: sync-ahead in place` and `sync-ahead: frame waits made exact` give the guards' waits and copies. `TOTK_SYNC_AHEAD_CHECK=1` runs the detector below with the late wait itself. - Frame-ahead check. `TOTK_SYNC_AHEAD=check` keeps the frame wait exact and measures what a one-frame-late wait without shadow storage would race (`runtime/vk_ahead.c`). The replay notes, with each submission's value, every access of guest memory the GPU makes after the recording: in-place bindings of imported pools without a VRAM mirror (`VK_AHEAD_READ`), the mirror's sync copies (`vk_mem_mirror_prologue`), GPU deswizzle sources (`vk_texture.c` `upload_gpu`) and the mirror write-backs (`vk_mem_mirror_end`). At the frame wait for value A, whose late form would have waited for L, the notes of values in (L, A] form the frame's read set: merged per class, gap-joined into at most 512 write-watch queries that stamp the writes made so far into the shared page generations (`vk_mem_ahead_observe`). At the next frame-wait entry the queries run again, and every page of the set stamped since is a CPU write the late wait would have raced; memory without a write watch is compared by page hashes. GPU writes into CPU-visible memory and reports are listed with their value: an invalidate of a range written by a value in (L, A] of the last frame wait, and the fence write-back of a small target rendered by one, count as stale readbacks. One line `sync-ahead check:` per second of host time and a run summary with the largest offenders by pool at exit. `--probe-vkmem` checks the detector on a watched pool (`vk_ahead_selftest`); `tests/sync_ahead_check.py` checks that the wait stays exact. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame diff --git a/runtime/nvn_api.h b/runtime/nvn_api.h index 82dce1dd..7c1cca0c 100644 --- a/runtime/nvn_api.h +++ b/runtime/nvn_api.h @@ -51,6 +51,7 @@ void nvn_sync_ahead_present(void); uint64_t nvn_sync_ahead_guard(void); int nvn_sync_ahead_on(void); // 1 when the frame-ahead wait is in effect: TOTK_SYNC_AHEAD=1 and a wait site known for this version int vk_mem_shadow_on(void); // vk_memory.c: 1 with shadow storage on (valid once the Vulkan back end is up) +int vk_mem_mirror_on(void); // vk_memory.c: 1 when pools used as buffers get VRAM mirrors (memory = mirror or auto on a discrete GPU) void vk_mem_shadow_fence(void); // vk_memory.c: returns once every shadow sync that read the old guard is done // vk_memory.c: the game is about to read [addr, addr+size) of pool storage; waits for a GPU write into it that the game // was told is done, unless the range still holds the result of the write before diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index afa2da6c..26b354e4 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -509,6 +509,14 @@ static int sync_mode(void) { if (mode < 0) { const char *v = cfg_value("TOTK_SYNC_AHEAD"); int on = strcmp(v, "0") != 0 && strcmp(v, "check") != 0, check = !strcmp(v, "check"); + // In place (no shadow storage) the guards cover the VRAM mirrors' copies; with memory = import the GPU reads + // vertex, index and storage data straight from the game's memory, and a check run (TOTK_MEMORY=import, + // islands) saw the game rewrite ~10 KiB of them a frame after the wait, particles mostly. + if (on && !vk_mem_shadow_on() && !vk_mem_mirror_on()) { + rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage or the VRAM mirrors (memory = import reads the game's " + "vertex and storage data in place); the game waits for the GPU as before"); + on = 0; + } if ((on || check) && !frame_wait_lr()) { rt_log("nvn: TOTK_SYNC_AHEAD: no frame wait site is known for this game version; the game waits for the " "GPU as before"); diff --git a/runtime/tex_probe.c b/runtime/tex_probe.c index 06396205..a7fcfa04 100644 --- a/runtime/tex_probe.c +++ b/runtime/tex_probe.c @@ -376,6 +376,11 @@ static int test_import(pool_t *pool) { GCHECK(!ah, "sync-ahead check self test failed at check %d", ah); printf("probe-vkmem: sync-ahead check: a write after the frame wait into the read set is flagged, one before it is not\n"); g_gpu_checks++; + int aw = vk_mem_ahead_selftest(); + GCHECK(!aw, "sync-ahead pending-write list self test failed at check %d", aw); + printf("probe-vkmem: sync-ahead in place: the pending-write list names the newest writer below a bound, prunes at " + "completion and joins when full\n"); + g_gpu_checks++; return 0; } diff --git a/runtime/vk.h b/runtime/vk.h index 02c5a766..fb79fe6d 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -283,6 +283,7 @@ uint64_t vk_mem_ahead_written(uint64_t lo, uint64_t hi, uint32_t since, void (*f void *arg); int vk_mem_pool_info(uint64_t addr, uint64_t *lo, uint64_t *size, uint32_t *flags, int *mirrored); // 0: no pool void vk_mem_ahead_log(void); // TOTK_SYNC_AHEAD=1 in place: the guards' waits and copies over the run, at exit +int vk_mem_ahead_selftest(void); // --probe-vkmem: the in-place form's pending-write list; 0 passed void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size); // A shader's storage-buffer write: goes back through the write-back merge (vk_memory.c), which keeps the words the GPU // did not change. diff --git a/runtime/vk_ahead.c b/runtime/vk_ahead.c index ed69e732..d3121f1b 100644 --- a/runtime/vk_ahead.c +++ b/runtime/vk_ahead.c @@ -107,6 +107,7 @@ void vk_ahead_init(void) { VkAheadGuards *g = &vk_ahead_g; g->lag = 1; g->msync = rt_knob("SYNC_AHEAD_MSYNC", "1"); + g->wbmerge = rt_knob("SYNC_AHEAD_WBMERGE", "1"); g->deswz = rt_knob("SYNC_AHEAD_DESWZ", "1"); g->reads = rt_knob("SYNC_AHEAD_READS", "1"); g->oneshot = rt_knob("SYNC_AHEAD_ONESHOT", "1"); @@ -114,13 +115,10 @@ void vk_ahead_init(void) { // goes 1 -> 2 or 3 at one update, 2 -> 3 at the next, and the report is read in state 3), by then the frame // wait one frame late has covered that frame. Savage Wind (BOTW reads them right after the wait) has it on Mali. g->queries = rt_knob("SYNC_AHEAD_QUERIES", "0"); - rt_log("sync-ahead: in place (no shadow storage), memory %s; guards: mirror syncs staged %d, deswizzle of " - "rewritten textures on the CPU %d, invalidates wait for the writer %d, one-off small targets %d, " - "samples-passed reports %d", vk_mirror_on ? "mirror" : "import", g->msync, g->deswz, g->reads, g->oneshot, - g->queries); - if (!vk_mirror_on) - rt_log("sync-ahead: warning: import memory: vertex, index and storage data are read in place on the GPU, " - "no guard covers them"); + rt_log("sync-ahead: in place (no shadow storage), memory %s; guards: mirror syncs staged %d, write-backs " + "merged %d, deswizzle of rewritten textures on the CPU %d, invalidates wait for the writer %d, one-off " + "small targets %d, samples-passed reports %d", vk_mirror_on ? "mirror" : "import", g->msync, g->wbmerge, + g->deswz, g->reads, g->oneshot, g->queries); } vk_ahead_on = check || (vk_ahead_g.lag && rt_knob("SYNC_AHEAD_CHECK", "0")); if (!vk_ahead_on) return; diff --git a/runtime/vk_ahead.h b/runtime/vk_ahead.h index 2c02dac0..cc48528a 100644 --- a/runtime/vk_ahead.h +++ b/runtime/vk_ahead.h @@ -16,11 +16,12 @@ enum { AHW_WRITE, AHW_COUNTER, AHW_SAMPLES, AHW_EVENT, AHW_N }; extern int vk_ahead_on; // The in-place form's switches, read once by vk_ahead_init; every guard is 0 without lag. A/B: TOTK_SYNC_AHEAD_. // msync: the mirror syncs copy the synced pages into staging on the CPU at submit (vk_memory.c vk_mem_mirror_prologue); +// wbmerge: every GPU write into a mirror goes back through the word merge, not only a shader's (vk_memory.c gpu_wrote); // deswz: a texture re-uploaded after the CPU rewrote it takes the CPU deswizzle (vk_texture.c upload_gpu); reads: an // invalidate of a range not read lately waits for the pending GPU write into it (vk_memory.c vk_mem_shadow_read); // oneshot: a one-off small target is waited for, up to its value (vk_ahead_oneshot); queries (default 0): so is a // samples-passed report. -typedef struct { int lag, msync, deswz, reads, oneshot, queries; } VkAheadGuards; +typedef struct { int lag, msync, wbmerge, deswz, reads, oneshot, queries; } VkAheadGuards; extern VkAheadGuards vk_ahead_g; void vk_ahead_init(void); // vk_memory.c mirror_mode, once at device start, shadow mode known: reads TOTK_SYNC_AHEAD // The newest value in (lo, hi] whose GPU result the game reads right after the frame wait (0: none): the in-place diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 109fef9d..7f5056c7 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -1504,6 +1504,43 @@ static void ahead_read(uint64_t addr, uint64_t size) { ah_time(&ah_rd_waits, &ah_rd_ns, &ah_rd_max, qpc() - t0); } +// --probe-vkmem: the pending-write list with values no submission has: the newest overlapping writer below a bound, +// pruning at completion, and a full list joining a write into its nearest entry, which still covers it. The list is +// restored after. 0: passed, else the failed check. +int vk_mem_ahead_selftest(void) { + static AWrite saved[AW_N]; + plat_mutex_lock(&aw_lock); + int n0 = naw, peak0 = aw_peak, bad = 0; + uint64_t joined0 = aw_joined, f = vk_sync_issued() + 1000000, a = 0x10000000; + memcpy(saved, aw, sizeof aw); + naw = 0; + plat_mutex_unlock(&aw_lock); + aw_add(a, a + 0x100, f); + aw_add(a + 0x80, a + 0x200, f + 1); + aw_add(a, a + 0x100, f); // the same range again: no new entry + plat_mutex_lock(&aw_lock); + if (naw != 2) bad = 1; + else if (aw_writer(a, a + 0x40, 0, UINT64_MAX) != f) bad = 2; + else if (aw_writer(a + 0xc0, a + 0xd0, 0, UINT64_MAX) != f + 1) bad = 3; + else if (aw_writer(a + 0xc0, a + 0xd0, 0, f + 1) != f) bad = 4; // below the value being recorded + else if (aw_writer(a + 0x200, a + 0x300, 0, UINT64_MAX)) bad = 5; // touching, not overlapping + if (!bad) { + aw_prune(f); + if (naw != 1 || aw_writer(a, a + 0x40, f, UINT64_MAX)) bad = 6; + } + int fill = AW_N - naw; // the entry check 6 left stays: fill up to exactly AW_N, so only the next add joins + plat_mutex_unlock(&aw_lock); + for (int i = 0; !bad && i < fill; i++) aw_add(a + 0x1000 * (uint64_t)(i + 1), a + 0x1000 * (uint64_t)(i + 1) + 0x10, f + 2); + uint64_t lo = a + 0x1000 * (AW_N + 8); + if (!bad) aw_add(lo, lo + 0x10, f + 3); // full: joined with the nearest + plat_mutex_lock(&aw_lock); + if (!bad && (naw != AW_N || aw_joined != joined0 + 1 || aw_writer(lo, lo + 0x10, 0, UINT64_MAX) != f + 3)) bad = 7; + memcpy(aw, saved, sizeof aw); + naw = n0, aw_peak = peak0, aw_joined = joined0; + plat_mutex_unlock(&aw_lock); + return bad; +} + void vk_mem_ahead_log(void) { plat_mutex_lock(&aw_lock); rt_log("mem: sync-ahead in place: the game waited for a GPU write it reads %llu times, %.1f ms in all, longest " @@ -1527,6 +1564,7 @@ void vk_mem_shadow_read_log(void) { } int vk_mem_shadow_on(void) { return shadow_on; } +int vk_mem_mirror_on(void) { return vk_mirror_on; } void vk_mem_shadow_sync(uint64_t fence) { if (!shadow_on) return; @@ -2382,8 +2420,11 @@ static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) { if (r && size) gpu_writes(r, sp->offset, size); if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size) return; plat_mutex_lock(&mirror_lock); - // A binding to the end of its pool (no size) can span hundreds of MiB: those stay copies. - if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) { + // A binding to the end of its pool (no size) can span hundreds of MiB: those stay copies. TOTK_SYNC_AHEAD=1 in place + // (vk_ahead_g.wbmerge): the copies' and updates' writes merge too. A plain copy back lands when the GPU runs it, after + // the game, told the value is done a frame early, may have written other words of the range for the next frame, + // and would put the older words back; the merge writes only the words this value changed. + if ((shader || vk_ahead_g.wbmerge) && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) { plat_mutex_unlock(&mirror_lock); return; } diff --git a/tests/sync_ahead_check.py b/tests/sync_ahead_check.py index d616dd92..19bbe447 100644 --- a/tests/sync_ahead_check.py +++ b/tests/sync_ahead_check.py @@ -56,7 +56,9 @@ static void vk_ahead_frame_lag(uint64_t asked, uint64_t lagged, int behind) { lag_calls++, lag_asked = asked, lag_value = lagged, lag_behind = behind; } static void vk_ahead_frame_done(int ok) { assert(lag_calls == 1); done_calls++, done_ok = ok; } +static int mirror; static int vk_mem_shadow_on(void) { return shadow; } +static int vk_mem_mirror_on(void) { return mirror; } static uint64_t nvn_fence_last(void) { return last; } static void rt_log(const char *format, ...) { (void)format; } // The patch table: set qstamp exists on 1.4.3 only and starts as TOTK_SYNC_AHEAD says (runtime/patches.c). @@ -77,9 +79,10 @@ PRODUCTION static uint32_t seed = 0x19ed634b; static uint32_t random_word(void) { seed ^= seed << 13; seed ^= seed >> 17; return seed ^= seed << 5; } int main(int argc, char **argv) { - assert(argc == 5); + assert(argc == 6); rt_game_version = atoi(argv[1]); enabled = atoi(argv[2]); shadow = atoi(argv[3]); int det = atoi(argv[4]); + mirror = atoi(argv[5]); int supported = rt_game_version == 110 || rt_game_version == 121 || rt_game_version == 143; uint64_t site = rt_game_version == 110 ? 0x7000b39aa4ull : rt_game_version == 121 ? 0x7000b23dfcull : rt_game_version == 143 ? 0x70002db99cull : 0; @@ -87,7 +90,8 @@ int main(int argc, char **argv) { qstamp_on = rt_game_version == 143 && enabled; last = 20; nvn_sync_ahead_present(); last = 40; nvn_sync_ahead_present(); - int on = enabled == 1 && supported, check = enabled == 2 && supported; // shadow off: the in-place form + // shadow off: the in-place form, which needs the VRAM mirrors + int on = enabled == 1 && supported && (shadow || mirror), check = enabled == 2 && supported; vk_ahead_on = check || (on && det); assert(pa_fence[0] == (on || check ? 40 : 0)); assert(pa_fence[1] == (on || check ? 20 : 0)); @@ -155,8 +159,8 @@ int main(int argc, char **argv) { nvnSyncWait(&cpu); assert(pa_guard == 39); } - printf("sync-ahead version %d enabled %d shadow %d detector %d: 100000 wait selections passed (%d relaxed, %d up " - "to a one-off value)\n", rt_game_version, enabled, shadow, det, relaxed_cases, oneshot_cases); + printf("sync-ahead version %d enabled %d shadow %d detector %d mirror %d: 100000 wait selections passed (%d relaxed, %d up " + "to a one-off value)\n", rt_game_version, enabled, shadow, det, mirror, relaxed_cases, oneshot_cases); } ''' @@ -178,7 +182,9 @@ def main(): for enabled in (0, 1, 2): for shadow in (0, 1): for det in (0, 1): - subprocess.run([str(exe), str(version), str(enabled), str(shadow), str(det)], check=True) + for mirror in (0, 1): + subprocess.run([str(exe), str(version), str(enabled), str(shadow), str(det), str(mirror)], + check=True) if __name__ == "__main__": From 9422939cb72ccf6a2bf5f88a69f0c6c8694ccb29 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 21 Nov 2026 14:48:26 +0300 Subject: [PATCH 7/7] totk.ini.example: sync_ahead no longer needs shadow storage on a discrete GPU; without it the GPU reads the game's memory in place behind guards, and only memory = import still refuses the switch Co-Authored-By: Claude Opus 5.5 --- totk.ini.example | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/totk.ini.example b/totk.ini.example index a1581d3b..c367228a 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -70,11 +70,14 @@ # game waits once for the GPU to finish the frame it presented last; with 1 that wait covers the frame before it # instead. On Great Sky Island (Linux, RX 9070 XT) with shadow_defer = 1 this took about 73 to 100 fps. Costs one more # frame of input delay, and frame times vary more from frame to frame than with 0. GPU results the game reads back every frame after that wait, for example its depth queries, can be a frame old; a result it reads only once is waited for. -# Works on 1.1.0, 1.2.1 and 1.4.3. Needs shadow storage (shadow below), which is on by default on Linux and -# Wine and needs shadow = 1 on native Windows; without it the setting is ignored and the log says so. +# Works on 1.1.0, 1.2.1 and 1.4.3. With shadow storage (shadow below; on by default on Linux and Wine) the GPU +# reads copies of the game's memory; without it, on a discrete GPU (memory = mirror or auto), the GPU reads the +# game's memory in place and guards copy or wait for what the game rewrites while the GPU still reads it (no shadow +# copies, about 1 ms a frame less on a weak CPU). With memory = import and no shadow storage the setting is ignored +# and the log says so. # sync_ahead = 0 # Shadow storage, 0 or 1: each CPU-visible memory pool of the game gets a host-visible copy that the GPU reads, in -# place of importing the game's memory. Default: on under Linux and Wine, off on native Windows. sync_ahead needs it. +# place of importing the game's memory. Default: on under Linux and Wine, off on native Windows. # Costs about 0.2 ms a frame and doubles the memory of those pools (about 1.1 GB; docs/building.md). Read at start. # shadow = 0 # With sync_ahead = 1: when the game changes memory the GPU still reads, queue the copy into the GPU's shadow of that