diff --git a/docs/architecture.md b/docs/architecture.md index 26b0a14d..17758b1f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,7 +85,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`). - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. -- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS and `fxaa = 0` act on the frame draw by draw and keep the replay sequential. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. +- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/runtime/vk_dlaa.c b/runtime/vk_dlaa.c index 18bd2ebf..94d391de 100644 --- a/runtime/vk_dlaa.c +++ b/runtime/vk_dlaa.c @@ -243,6 +243,8 @@ static struct { VkExecTarget color, depth; // the armed pair, valid in the frame it was armed int armed, done, gbuf; // this frame: armed on the light target, DLAA ran, the G-buffer was drawn int ended, up; // this frame: the scene is drawn (jitter over), DLSS wrote the scaled output + uint64_t end_fence; // a chunked replay: the submission and op the scan ended the scene at (ended_at) + uint32_t end_pos; struct Size { // a DLSS feature and its work images per scene size (dynamic resolution flips it) uint32_t w, h, ow, oh; // render size, output size VkImage out, mv; @@ -777,10 +779,9 @@ static struct Snap *snap_for(VkCommandBuffer cb, uint32_t w, uint32_t h, VkForma } // At arming: the light target as it is before the transparents, outside any pass. -static void mask_snapshot(VkExec *ex) { +static void mask_snapshot(VkExec *ex, const VkExecTarget *t) { R.cur = NULL; if (mask_init() <= 0) return; - const VkExecTarget *t = &D.color; struct Snap *s = snap_for(ex->cb, t->width, t->height, t->format); if (!s) return; vk_exec_barrier(ex); @@ -829,14 +830,33 @@ static float halton(uint32_t i, uint32_t b) { return r; } +// The scene is drawn at the op ex records. A chunked replay's scan ends it ahead of the chunks that still record the +// scene's draws, so a chunk compares its op with the scan's (the chunks after it are taken once the scan is past their +// ops); a later submission of the frame is past it. +static int ended_at(const VkExec *ex) { + if (!__atomic_load_n(&D.ended, __ATOMIC_ACQUIRE)) return 0; + return !ex->chunked || ex->fence != D.end_fence || ex->pos >= D.end_pos; +} + void vk_dlaa_viewport(VkExec *ex, VkViewport *v) { - if (!D.jit_ok || D.ended || !ex->depth.image || ex->depth.image != D.jdepth) return; + if (!D.jit_ok || !ex->depth.image || ex->depth.image != D.jdepth || ended_at(ex)) return; v->x += D.jx, v->y += D.jy; - D.ux = D.jx, D.uy = D.jy, D.jit_used = 1; + D.ux = D.jx, D.uy = D.jy; + __atomic_store_n(&D.jit_used, 1, __ATOMIC_RELAXED); +} + +// The scene camera: c4 of the last scene draw before the scene ends. A chunked replay's scan notes it in replay order +// (vk_dlaa_scan_draw): chunks recording side by side would leave the block of whichever drew last. +static void note_camera(const VkExec *ex) { + const p_bind_buf *b = &ex->ubo[0][1]; // constant bank c4: the replay keeps user banks from c3 on + const VkExecTarget *z = &ex->depth; + if (b->size == 0x940 && b->addr && z->image && z->width >= DLAA_SCENE_MIN_W && z->format == VK_FORMAT_D32_SFLOAT && + (!D.jdepth || z->image == D.jdepth)) + D.cam_addr = b->addr; } void vk_dlaa_draw(VkExec *ex) { - if (!dl_active() || D.ended) return; + if (!dl_active() || ended_at(ex)) return; static int survey = -1; // TOTK_DLAA_SURVEY=1 (test): what is drawn into the armed light target if (survey < 0) survey = rt_knob("DLAA_SURVEY", "0"); static uint64_t dfn[16]; // scene draws writing depth, by depth test (reversed depth: GREATER/GEQUAL) @@ -867,11 +887,11 @@ void vk_dlaa_draw(VkExec *ex) { } } } - const p_bind_buf *b = &ex->ubo[0][1]; // constant bank c4: the replay keeps user banks from c3 on - const VkExecTarget *z = &ex->depth; - if (b->size == 0x940 && b->addr && z->image && z->width >= DLAA_SCENE_MIN_W && z->format == VK_FORMAT_D32_SFLOAT && - (!D.jdepth || z->image == D.jdepth)) - D.cam_addr = b->addr; + if (!ex->chunked) note_camera(ex); +} + +void vk_dlaa_scan_draw(const VkExec *ex) { + if (dl_active() && !D.ended) note_camera(ex); } // The jitter moves on right after DLAA ran, not at the present: the replay can record the next frame's scene before @@ -1062,9 +1082,9 @@ static int scene_color(VkFormat f) { return f == VK_FORMAT_B10G11R11_UFLOAT_PACK32 || f == VK_FORMAT_R16G16B16A16_SFLOAT; } -void vk_dlaa_pass(VkExec *ex) { +int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { const VkExecTarget *c = &ex->color[0], *z = &ex->depth; - if (!dl_active() || D.done) return; + if (!dl_active() || D.done) return 0; // The G-buffer: several colour targets with a D32F depth of at least DLAA_SCENE_MIN_W wide. Only the light target drawn after it // arms the frame; at night the game draws into the light target alone with the scene depth before the G-buffer too, // and DLAA then ran at a small pass ahead of the whole scene, which reached the screen jittered. @@ -1075,7 +1095,7 @@ void vk_dlaa_pass(VkExec *ex) { if (c->image && c->format == VK_FORMAT_R8G8B8A8_UNORM && c->width == z->width && c->height == z->height && c->samples <= 1 && !c->is3d && (!D.ids_ok || c->width >= D.ids.width)) D.ids = *c, D.ids_ok = 1; - return; + return 0; } // the scene depth is D32F; a D16 target of the scene size also serves one composite pass before the transparents. // A smaller pair never takes over an armed one: the camera rune draws a half-size RGBA16F target with a half-size @@ -1087,26 +1107,56 @@ void vk_dlaa_pass(VkExec *ex) { c->samples <= 1 && !c->is3d) { int first = !D.armed; D.color = *c, D.depth = *z, D.armed = 1; - if (first) mask_snapshot(ex); // the lit scene, before the transparents (reactive mask) - return; + if (!first) return 0; + *j = (VkExecJob){.kind = VK_JOB_DLSS_MASK, .scene = *c}; // the lit scene, before the transparents (reactive mask) + return 1; } - if (!D.armed) return; + if (!D.armed) return 0; if (!D.ended) { - if (!ex->pass_w || ex->pass_w * 4 > D.color.width) return; - D.ended = 1; + if (!ex->pass_w || ex->pass_w * 4 > D.color.width) return 0; + D.end_fence = ex->fence, D.end_pos = ex->pos; + __atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE); unsigned o[2], s[2]; res_sizes(o, s); - if (D.color.width >= o[0]) D.done = 1, evaluate(ex, NULL); // DLAA in place; else wait for the scaling pass - return; + if (D.color.width < o[0]) return 0; // wait for the scaling pass + D.done = 1; + *j = (VkExecJob){.kind = VK_JOB_DLSS}; // DLAA in place + return 1; } // the game's scaling pass: one colour target of the output size, no depth unsigned o[2], s[2]; res_sizes(o, s); if (ex->pass_ncolor == 1 && c->image && !z->image && scene_color(c->format) && c->width > D.color.width && c->width == o[0] && c->height == o[1] && c->samples <= 1 && !c->is3d) { - D.done = D.up = 1; - evaluate(ex, c); + D.done = 1; + *j = (VkExecJob){.kind = VK_JOB_DLSS, .scene = *c}; + return 1; } + return 0; +} + +// The scan resolved its targets without making entries (vk_texture_revive); a record made on a chunk takes their images +// now, as the scan's pointers may be stale by then. +static void target_now(VkExecTarget *t) { + const VkTexImage *i = t->tex ? vk_texture_get(t->tex) : NULL; + if (i) t->img = i, t->image = i->image; +} + +void vk_dlaa_exec(VkExec *ex, const VkExecJob *j) { + VkExecTarget t = j->scene; + if (ex->chunked) { + target_now(&t), target_now(&D.color), target_now(&D.depth); + if (D.ids_ok) target_now(&D.ids); + } + if (j->kind == VK_JOB_DLSS_MASK) return mask_snapshot(ex, &t); + if (!t.tex) return evaluate(ex, NULL); + D.up = 1; + evaluate(ex, &t); +} + +void vk_dlaa_pass(VkExec *ex) { + VkExecJob j; + if (!ex->chunked && vk_dlaa_plan(ex, &j)) vk_dlaa_exec(ex, &j); } // TOTK_DLSS, read at each present (the menu switches it live): the scene size of the mode (resolution.c) and the @@ -1178,24 +1228,34 @@ void vk_dlaa_frame(void) { static const uint64_t FXAA_FS[] = {0xf450003ebeb3dc52ull, 0x0a12f4bc2a9a4fbeull, 0x5c43ae18948d52cbull, 0xaad42238801d31d8ull}; +// Chunks of one submission look shaders up side by side: an entry's hash is stored before its shader, and entries are +// added under the lock. static uint64_t spirv_hash(const GfxShader *s) { static struct { const GfxShader *s; uint64_t h; } tab[8192]; + static PlatMutex lk = PLAT_MUTEX_INIT; uintptr_t k = ((uintptr_t)s >> 4) * 0x9E3779B97F4A7C15ull; for (uint32_t i = 0, j = (uint32_t)(k >> 51); i < 8192; i++, j = (j + 1) & 8191) { - if (tab[j].s == s) return tab[j].h; - if (!tab[j].s) { + const GfxShader *e = __atomic_load_n(&tab[j].s, __ATOMIC_ACQUIRE); + if (e == s) return tab[j].h; + if (e) continue; + plat_mutex_lock(&lk); + e = tab[j].s; + if (!e) { uint64_t words = 0, h = 0xcbf29ce484222325ull; const uint8_t *b = (const uint8_t *)gfx_shader_spirv(s, &words); for (uint64_t n = 0; b && n < words * 4; n++) h = (h ^ b[n]) * 0x100000001b3ull; - tab[j].s = s, tab[j].h = h; + tab[j].h = h; + __atomic_store_n(&tab[j].s, s, __ATOMIC_RELEASE); + plat_mutex_unlock(&lk); return h; } + plat_mutex_unlock(&lk); + if (e == s) return tab[j].h; } return 0; // table full: no skip } int vk_dlaa_skip_draw(VkExec *ex, const GfxShader *fs) { - if (ex->chunked) return 0; // a chunked replay runs only with fxaa = 1 and without DLSS (vk_exec.c chunked_on) static uint64_t frame_read = ~0ull; static int fxaa = 1; // fxaa = 0 reports once whether it found the game's FXAA draw: the first skip, or a minute of frames (3600) with diff --git a/runtime/vk_dlaa.h b/runtime/vk_dlaa.h index 82b98113..cce8877d 100644 --- a/runtime/vk_dlaa.h +++ b/runtime/vk_dlaa.h @@ -18,11 +18,17 @@ int vk_dlaa_available(void); // scene's light target or, at the first quarter-width pass after it, runs DLAA on that target. Each present ends the // frame (vk_dlaa_frame). typedef struct VkExec VkExec; +typedef struct VkExecJob VkExecJob; void vk_dlaa_pass(VkExec *ex); +// The same decision for a chunked replay's scan, in replay order: 1 with the record due before this pass (j), which +// vk_dlaa_exec makes on the chunk that holds the op (vk_exec.c run_job). +int vk_dlaa_plan(const VkExec *ex, VkExecJob *j); +void vk_dlaa_exec(VkExec *ex, const VkExecJob *j); void vk_dlaa_frame(void); // vk_exec_draw.c, each draw before its fixed state is recorded: notes the scene camera block (vk_dlaa_draw), and moves // the viewport of the scene's draws by the frame's jitter (vk_dlaa_viewport, when the viewport is recorded). void vk_dlaa_draw(VkExec *ex); +void vk_dlaa_scan_draw(const VkExec *ex); // a chunked replay's scan, at each draw: the camera note of vk_dlaa_draw void vk_dlaa_viewport(VkExec *ex, VkViewport *v); // vk_mvsurvey.c (TOTK_MV_SURVEY): notes the uniform blocks of each scene draw, beside vk_dlaa_draw; each present // compares them with the previous frames' and logs a summary now and then. diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index 8bf87fb7..5c6c88a8 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -315,7 +315,7 @@ static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImag return 1; } -enum { HOLD_TEX, HOLD_WRITES, HOLD_N }; +enum { HOLD_TEX, HOLD_DLSS, HOLD_WRITES, HOLD_N }; static void scan_hold(int why); // scan: the state scan's resolution, which creates nothing and leaves out a texture without an image yet. @@ -933,21 +933,29 @@ static int state_op(VkExec *ex, uint32_t op, const void *a) { return 1; } -// A contact shadow march the scan planned for this point (vk_contact_plan): recorded before the op that triggers it, the -// draw's pass begun after it as in prepare_draw. The marches share vk_contact.c's pipeline and its first use, so -// they go in the scan's order; chunks are taken in order, so the chunk of an earlier record is being recorded or done. +// A record the scan planned for this point (vk_contact_plan, vk_dlaa_plan): recorded before the op that triggers it, the +// draw's pass begun after it as in prepare_draw. The records share their module's pipelines and state, so they go in +// the scan's order; chunks are taken in order, so the chunk of an earlier record is being recorded or done. DLSS's +// evaluation reads what the scene's draws noted on the chunks before it (the camera block, the jitter used, the +// per-object motion list), so it waits until they are recorded; the chunks after it wait for it (a hold, scan_job). static uint32_t job_turn; +static void chunks_wait(int k); static void run_job(VkExec *ex) { uint32_t turn = ex->job_i; VkExecJob j = ex->jobs[ex->job_i++]; + if (j.kind == VK_JOB_DLSS) chunks_wait(j.chunk); while (__atomic_load_n(&job_turn, __ATOMIC_ACQUIRE) != turn) plat_yield(); vk_exec_pass_end(ex); - // The scan resolved the scene and its depth without creating entries (vk_texture_revive): their images now. - const VkTexImage *si = vk_texture_get(j.scene.tex), *di = vk_texture_get(j.depth.tex); - if (si && di) { - j.scene.img = si, j.scene.image = si->image, j.depth.img = di, j.depth.image = di->image; - vk_contact_exec(ex, &j); + if (j.kind != VK_JOB_CONTACT) { + vk_dlaa_exec(ex, &j); + } else { + // The scan resolved the scene and its depth without creating entries (vk_texture_revive): their images now. + const VkTexImage *si = vk_texture_get(j.scene.tex), *di = vk_texture_get(j.depth.tex); + if (si && di) { + j.scene.img = si, j.scene.image = si->image, j.depth.img = di, j.depth.image = di->image; + vk_contact_exec(ex, &j); + } } __atomic_store_n(&job_turn, turn + 1, __ATOMIC_RELEASE); ex->dirty = VK_EXEC_DIRTY_ALL; @@ -1010,10 +1018,11 @@ static int walk(VkExec *ex, const op_hdr *h, int depth, const op_hdr *stop) { // that finishes early takes another: in the town at 4K on an RTX 5070 the replay goes from 8.2 to 4.5 ms // and the frame rate from 71 to 79; more threads add little, since the frame then waits for the GPU. A sequential scan on // the submitting thread applies only what is queue state: the fixed-function and bind ops, render targets, render -// enable and conditions, samples-passed counting and the contact shadows' marches (vk_contact_plan); it looks up no +// enable and conditions, samples-passed counting, the contact shadows' marches (vk_contact_plan) and DLSS's decisions +// (vk_dlaa_plan); it looks up no // shader, pipeline or descriptor and makes no Vulkan call. Where a chunk starts it keeps a copy of that state // (VK_EXEC_STATE_BEGIN..END) and how far the scan's lists got: the submission's inline uniform updates (ubo_ptr), -// RENDER_COND results, planned contact shadow marches and the ops that drop uniform copies (vk_usnap_drop_at, +// RENDER_COND results, planned records (contact shadows, DLSS) and the ops that drop uniform copies (vk_usnap_drop_at, // scan_writes), which a chunk takes // from the lists instead of from earlier chunks. A chunk starts with every image in GENERAL and ends with a full // barrier, as a part of vk_sync_split does, so its recording depends only on the copy and the lists. @@ -1396,10 +1405,20 @@ static void chunk_start(const VkExec *s, int hi, const op_hdr *h) { chunks_submit(0); } -static void scan_job(const VkExecJob *j, const op_hdr *h) { +// The chunks before chunk k are recorded. +static void chunks_wait(int k) { + plat_mutex_lock(&RP.m); + for (int i = 0; i < k; i++) + while (!__atomic_load_n(&SC.ch[i].done, __ATOMIC_ACQUIRE)) plat_cond_wait(&RP.done, &RP.m); + plat_mutex_unlock(&RP.m); +} + +static void scan_job(VkExecJob *j, const op_hdr *h) { static int full; + j->chunk = SC.nch - 1; if (SC.njobs < MAX_JOBS) SC.jobs[SC.njobs] = *j, SC.jobs[SC.njobs++].op = h; - else if (!full++) rt_log("vk_exec: more than %d contact shadow marches in a submission: the others are left out", MAX_JOBS); + else if (!full++) rt_log("vk_exec: more than %d planned records in a submission: the others are left out", MAX_JOBS); + if (j->kind == VK_JOB_DLSS) scan_hold(HOLD_DLSS); // the chunks after it read DLSS's output (vk_dlaa_swap_img) } // What vk_contact_pass_begin (pass) and vk_contact_flush do in the sequential replay, in replay order. A pass of the @@ -1409,7 +1428,15 @@ static void scan_contact(const VkExec *s, int pass, const op_hdr *h) { if (vk_contact_plan(s, pass, &j)) scan_job(&j, h); } -// A draw as prepare_draw sees it: its pass begins, and with it a contact shadow march may be due. +// What vk_exec_pass_begin asks of contact shadows and DLSS. DLSS's plan, too, gives the same answer for a pass asked +// again: it moves on only at the passes that arm, end or scale the scene. +static void scan_pass(const VkExec *s, const op_hdr *h) { + scan_contact(s, 1, h); + VkExecJob j; + if (vk_dlaa_plan(s, &j)) scan_job(&j, h); +} + +// A draw as prepare_draw sees it: its pass begins, and with it a contact shadow march or a DLSS record may be due. static void scan_draw(VkExec *s, uint32_t op, const void *a, const op_hdr *h) { if (s->render_disabled) return; if (op == OP_DRAW && ((const p_draw *)a)->count <= 0) return; @@ -1421,7 +1448,8 @@ static void scan_draw(VkExec *s, uint32_t op, const void *a, const op_hdr *h) { return; if (s->pass_noatt) s->rt_resolved = 0; // sized by this draw's viewport resolve_rt_as(s, 1); - scan_contact(s, 1, h); + scan_pass(s, h); + vk_dlaa_scan_draw(s); } static void scan(VkExec *s, int hi, const op_hdr *h, int depth) { @@ -1457,7 +1485,7 @@ static void scan(VkExec *s, int hi, const op_hdr *h, int depth) { break; case OP_CLEAR_COLOR: case OP_CLEAR_DS: SC.open |= !s->render_disabled; - if (!s->render_disabled) resolve_rt_as(s, 1), scan_contact(s, 1, h); + if (!s->render_disabled) resolve_rt_as(s, 1), scan_pass(s, h); break; case OP_DRAW: case OP_DRAW_IDX: case OP_DRAW_IDX_IND: SC.in_chunk++, SC.open = 1; @@ -1578,14 +1606,8 @@ static int chunked_on(void) { if (SC.workers) rt_log("vk: replay in up to %d chunks per submission of at least %d draws each, %d recording thread%s%s", SC.chunks, SC.min_draws, SC.workers, SC.workers > 1 ? "s" : "", SC.check ? ", state checked" : ""); - if (SC.workers && vk_dlaa_wanted()) - rt_log("vk: DLSS (dlaa = 1) keeps the replay on one thread, so replay_workers has no effect"); - else if (SC.workers && !cfg_bool("TOTK_FXAA")) - rt_log("vk: fxaa = 0 keeps the replay on one thread, so replay_workers has no effect while it is set"); } - // DLSS (the dlaa setting) and the fxaa switch act on what the frame drew so far, draw by draw and in replay order, - // which only the sequential replay gives them. - return SC.workers > 0 && SC.chunks > 1 && !vk_dump_armed && !vk_dlaa_wanted() && cfg_bool("TOTK_FXAA"); + return SC.workers > 0 && SC.chunks > 1 && !vk_dump_armed; } int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t fence) { @@ -1644,8 +1666,9 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t (unsigned long long)SC.chunked, (unsigned long long)SC.subs, (double)SC.nchunks / SC.chunked, SC.prep_ns / k, SC.scan_ns / k, SC.start0_ns / k, SC.first_ns / k, SC.last_ns / k, SC.rec_ns / k, SC.rec_max_ns / 1e6, SC.mirror_ns / k, SC.usnap_ns / k, SC.ready_ns / k); - rt_log("perf: chunked replay: chunks held per submission: %.2f for a new texture, %.2f for unknown storage " - "writes", SC.held[HOLD_TEX] / (double)SC.chunked, SC.held[HOLD_WRITES] / (double)SC.chunked); + rt_log("perf: chunked replay: chunks held per submission: %.2f for a new texture, %.2f for DLSS, %.2f for " + "unknown storage writes", SC.held[HOLD_TEX] / (double)SC.chunked, SC.held[HOLD_DLSS] / (double)SC.chunked, + SC.held[HOLD_WRITES] / (double)SC.chunked); memset(SC.held, 0, sizeof SC.held); char b[768]; int o = 0; diff --git a/runtime/vk_exec.h b/runtime/vk_exec.h index 1f224fcc..fe799392 100644 --- a/runtime/vk_exec.h +++ b/runtime/vk_exec.h @@ -48,11 +48,16 @@ typedef struct { // transitions } VkExecFrame; -// A contact shadow march (vk_contact.c) the scan of a chunked replay planned before an op: vk_contact_plan decides it in -// replay order, vk_contact_exec records it on the chunk that holds the op. +// A record the scan of a chunked replay planned before an op: a contact shadow march (vk_contact.c: vk_contact_plan +// decides it in replay order, vk_contact_exec records it), or DLSS's copy of the lit scene or its evaluation +// (vk_dlaa.c: vk_dlaa_plan, vk_dlaa_exec). The chunk that holds the op records it. +enum { VK_JOB_CONTACT, VK_JOB_DLSS_MASK, VK_JOB_DLSS }; typedef struct VkExecJob { const void *op; // vk_exec.c: the op it precedes in a scanned submission - VkExecTarget scene, depth; // the scene colour and the linear depth + int kind; // VK_JOB_* + int chunk; // vk_exec.c: the chunk that holds the op + VkExecTarget scene, depth; // contact: the scene colour and the linear depth; DLSS: the light target to copy + // (mask) or the scaling pass's target (evaluation; no tex: in place) float pc[16]; // the march's push constants } VkExecJob; @@ -155,7 +160,7 @@ typedef struct VkExec { int chunked; // records a chunk: what depends on earlier chunks comes from the scan const VkExecUpd *upd; // the submission's inline uniform updates const uint8_t *cond; // its RENDER_COND results (1 renders) - const VkExecJob *jobs; // its contact shadow marches + const VkExecJob *jobs; // its planned records (contact shadows, DLSS) uint32_t njobs; uint32_t upd_n, cond_i, job_i; // updates, results and records before the op being recorded uint32_t drop_n; // ops before it that may write uniform ranges (vk_usnap_drop_at) diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index cbf92811..51a2e829 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -936,7 +936,11 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G // *present: the program has this stage. Returns NULL when it has not or the translation failed. pending (graphics // stages, vk_pipeline_async_on): a shader not translated yet is queued and NULL returned with *pending = 1. await: a // shader queued earlier is waited for (gfx_shader_await) instead. -static int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion) +static _Thread_local int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion) +// Chunks of one submission note their scene draws for DLSS's motion one at a time: the list, its memos and the +// last-frame block tables (vk_mvprev.c, vk_mvsurvey.c) are shared; vk_mvobj_execute reads the list once the chunks +// before DLSS's evaluation are recorded (vk_exec.c run_job). +static PlatMutex mo_lock = PLAT_MUTEX_INIT; static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pending, int await) { VkExecDraw *D = ex->draw; @@ -1121,8 +1125,9 @@ static void limits_init(void) { // Per-object motion (section at the end): while it records a draw again, these replace the uniform bindings of the // vertex stage's slots 18-20 (last frame's camera, bones and place) and the fragment stage's slot 3 (its parameters). -static int mo_ovr_on; -static VkDescriptorBufferInfo mo_ovr[2][32]; // [vertex, fragment][slot]; buffer 0: not replaced +// Per thread: the other chunks' draws go on meanwhile and keep their own bindings. +static _Thread_local int mo_ovr_on; +static _Thread_local VkDescriptorBufferInfo mo_ovr[2][32]; // [vertex, fragment][slot]; buffer 0: not replaced static VkDescriptorBufferInfo ubo_info(VkExec *ex, int st, int slot) { if (mo_ovr_on && (st == NVN_STAGE_VS || st == NVN_STAGE_FS) && slot >= 0 && slot < 32 && mo_ovr[st][slot].buffer) @@ -2561,8 +2566,11 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (ex->mark && ex->fr.draws == ex->mark_draws0) ex->mark->first[0] = vs, ex->mark->first[1] = fs; } vk_dlaa_draw(ex); + int mo = ex->chunked && vk_dlaa_available(); + if (mo) plat_mutex_lock(&mo_lock); vk_mvsurvey_draw(ex); vk_mvprev_draw(ex); + if (mo) plat_mutex_unlock(&mo_lock); record_fixed(ex); repeat_stats(ex, info); bind_resources(ex, info); @@ -2883,10 +2891,11 @@ static void mvobj_draw(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSiz if (MO.failed || !vk_mvprev_on() || !D->B.st_valid) return; static int timed = -1; if (timed < 0) timed = rt_knob("MVOBJ_TIMES", "0"); // TOTK_MVOBJ_TIMES=1: the summary's times (the clock reads cost) - if (!timed) return mvobj_draw_(ex, d, ib, ib_off, 0); - uint64_t t0 = kern_now_ns(); + if (ex->chunked) plat_mutex_lock(&mo_lock); + uint64_t t0 = timed ? kern_now_ns() : 0; mvobj_draw_(ex, d, ib, ib_off, t0); - MO.ns_check += kern_now_ns() - t0; + if (timed) MO.ns_check += kern_now_ns() - t0; + if (ex->chunked) plat_mutex_unlock(&mo_lock); } // TOTK_MVOBJ_BANKS=1 (test): for scene draws, the words of each other uniform bank (c3, c6, c8-c16) that changed diff --git a/totk.ini.example b/totk.ini.example index c26ae91b..f786f451 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -40,8 +40,8 @@ # replay_async = 0 # Record each large submission in chunks on this many threads (0 to 16; 0 = one thread, the default). The chunks go to # the GPU in order. In the town at 4K on an RTX 5070, 4 cut the replay from 8.2 to 4.5 ms a frame (71 to 79 fps); more -# threads add little. Needs 8 or more logical processors to help. Contact shadows run in chunks too; DLSS (dlaa = 1) and -# fxaa = 0 keep the replay on one thread. Read at start. +# threads add little. Needs 8 or more logical processors to help. Contact shadows and DLSS (dlaa = 1) run in chunks too. +# Read at start. # replay_workers = 0 # EXPERIMENTAL, THE GAME CRASHES WITH 1. 1 lifts the virtual core gate: guest threads run without the console's limit # of one thread per core, and the game keeps per-core data without locks. With 1, 1.2.1 and 1.4.3 crash or hang,