diff --git a/docs/architecture.md b/docs/architecture.md index 0c123786..26b0a14d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,6 +85,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`). - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. +- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS and `fxaa = 0` act on the frame draw by draw and keep the replay sequential. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/docs/settings.md b/docs/settings.md index 3fee78a3..eebf62e0 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -182,6 +182,9 @@ Many rows below are developer tools that only act together with another setting | `audio_off` | Any value mixes audio without opening an output device. | The game runs in silence and reports two channels; same as `audio = none`. | Caution | | `one_per_core` (default `0`) | `1` keeps the process on the first logical processor of each physical core; `ab` switches it with the `ab` blocks. | Busy threads stop sharing a core with a sibling thread, which can help or hurt on the CPU; it only applies when the CPU has hyperthreads. | Caution | | `reexec` | Counter of address-space retries that the runtime sets itself when it starts the process again. | The counter is cleared when the window is recreated; a value of 20 or more stops the start with a fatal error. | Risky | +| `replay_chunks` (default `0`) | Most chunks a submission is cut into when `replay_workers` is set; `0` uses twice the number of recording threads. | Fewer chunks leave threads idle at the end of a submission, more add scan and submit work per chunk; no effect while `replay_workers` is `0`. | Caution | +| `replay_chunk_draws` (default `128`) | Least draws a chunk of the chunked replay holds before it may end at a pass. | Small values make many short chunks, large ones leave big submissions on few threads; no effect while `replay_workers` is `0`. | Caution | +| `replay_state_check` (default `0`) | `1` compares the replay state after the last chunk of a chunked submission with the state of the scan and logs differences. | Adds a comparison per submission and log lines on a mismatch; no effect while `replay_workers` is `0`. | Caution | | `restart_wait` | Set by the runtime when it restarts itself; names the old process by id and creation time so the new process waits for it to exit. | Meant for the runtime's own use; the code does not describe a manual value. | Unknown | | `spin` (default `1`) | `0` makes a guest thread that polls memory keep its virtual core. | Polling threads can starve the thread whose item they wait for, so frames can stall. | Caution | | `spin_handoff_us` (default `50`) | Microseconds a polling wait lasts before it gives its virtual core to a waiting thread. | A shorter value hands cores over sooner; a longer one keeps them longer. | Caution | diff --git a/runtime/cfg.c b/runtime/cfg.c index e105e916..852a9eb2 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -53,7 +53,7 @@ static const char *const known[] = { "ONE_PER_CORE", "OPERATION_MODE", "OUTPUT_RES", "PAD_LAYOUT", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_SCENE", "PCORE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", "PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_BENCH", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES", "PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_ASYNC", "REC_END", "REC_SPIN_US", "REEXEC", - "RENDER_SCALE", "REPLAY_ASYNC", + "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHUNKS", "REPLAY_CHUNK_DRAWS", "REPLAY_STATE_CHECK", "REPLAY_WORKERS", "REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", "SHADER_MEMO", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", @@ -95,7 +95,7 @@ static const struct { const char *key, *section; } sections[] = { {"BIND_RIGHT", "input"}, {"BIND_DOWN", "input"}, {"BIND_LSTICK_LEFT", "input"}, {"BIND_LSTICK_UP", "input"}, {"BIND_LSTICK_RIGHT", "input"}, {"BIND_LSTICK_DOWN", "input"}, {"BIND_RSTICK_LEFT", "input"}, {"BIND_RSTICK_UP", "input"}, {"BIND_RSTICK_RIGHT", "input"}, {"BIND_RSTICK_DOWN", "input"}, {"BIND_SHAKE", "input"}, - {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"FREE_CORES", "performance"}, + {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"FREE_CORES", "performance"}, {"TIMER_1MS", "performance"}, {"PIPE_WAIT", "performance"}, {"PIPE_SKIP", "performance"}, {"LIB_PREBUILD", "performance"}, {"LIB_PREBUILD_THREADS", "performance"}, {"PIPE_PREBUILD", "performance"}, {"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"}, @@ -163,6 +163,8 @@ static const CfgMeta metas[] = { {"DISCORD_IMAGE", "totk_512x512", NULL, LIVE}, // Advanced: scheduling {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, + {"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_CHUNKS", "0", "0..64", RESTART}, + {"REPLAY_CHUNK_DRAWS", "128", "1..100000", RESTART}, {"REPLAY_STATE_CHECK", "0", "0|1", RESTART}, {"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART}, // shaders and pipelines {"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "fullscreen", "strict|any|fullscreen", RESTART}, diff --git a/runtime/ui/menu.cpp b/runtime/ui/menu.cpp index aa845156..1c6d3efc 100644 --- a/runtime/ui/menu.cpp +++ b/runtime/ui/menu.cpp @@ -51,6 +51,7 @@ extern "C" { void rt_log(const char *fmt, ...) __attribute__((format(printf, 1, 2))); void vk_queue_lock(void); void vk_queue_unlock(void); +int vk_exec_replay_workers(void); void res_sizes(unsigned out[2], unsigned scene[2]); double frame_step_limit_hz(void); extern const int rt_game_version; @@ -905,6 +906,23 @@ static void row_threads(const char *key, UiTextId label, UiTextId desc) { opt_choice_list(key, label, vals, names, N, desc); } +// The chunked replay's recording threads: Off, then the counts; the note under the value is what this run uses, which +// a change replaces only after a restart. +static void row_replay_workers(void) { + static const char *const vals[] = {"0", "2", "3", "4", "6", "8", "12", "16"}; + enum { N = sizeof vals / sizeof *vals }; + const char *names[N]; + names[0] = menu_text(UI_MENU_OFF); + for (int i = 1; i < N; i++) names[i] = vals[i]; + static char now[64]; + char n[16]; + int w = vk_exec_replay_workers(); + snprintf(n, sizeof n, "%d", w); + if (w >= 0) snprintf(now, sizeof now, menu_text(UI_MENU_REPLAY_WORKERS_NOW), w ? n : menu_text(UI_MENU_OFF)); + opt_choice_list("TOTK_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, vals, names, N, UI_MENU_REPLAY_WORKERS_DESC, -1, + w >= 0 ? now : nullptr); +} + // A free-text key shown as Off (unset) / On (value set): On writes on_value, any set value reads as On. static void row_set_or_not(const char *key, UiTextId label, UiTextId desc, const char *on_value) { const char *vals[] = {"", on_value}, *names[] = {menu_text(UI_MENU_OFF), menu_text(UI_MENU_ON)}; @@ -954,6 +972,7 @@ static void tab_advanced(float scale) { static const UiTextId preempt[] = {UI_MENU_PREEMPT_OFF, UI_MENU_PREEMPT_ON, UI_MENU_PREEMPT_INHERIT}; // 0|1|2 opt_choice("TOTK_ASYNC_PREEMPT", UI_MENU_ASYNC_PREEMPT, preempt, 3, UI_MENU_ASYNC_PREEMPT_DESC); opt_toggle("TOTK_REPLAY_ASYNC", UI_MENU_REPLAY_ASYNC, UI_MENU_REPLAY_ASYNC_DESC); + row_replay_workers(); opt_toggle("TOTK_TIMER_1MS", UI_MENU_TIMER_1MS, UI_MENU_TIMER_1MS_DESC); opt_group(UI_MENU_GROUP_SHADERS); diff --git a/runtime/ui_text.c b/runtime/ui_text.c index d9769360..fd74a07f 100644 --- a/runtime/ui_text.c +++ b/runtime/ui_text.c @@ -386,6 +386,9 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_MENU_PREEMPT_OFF] = "Off", [UI_MENU_PREEMPT_ON] = "On", [UI_MENU_PREEMPT_INHERIT] = "Inheritance only", [UI_MENU_REPLAY_ASYNC] = "GPU frame prep on its own thread", [UI_MENU_REPLAY_ASYNC_DESC] = "Prepares frames for the GPU on a separate thread instead of the game thread. Adds up to one frame of latency.", + [UI_MENU_REPLAY_WORKERS] = "GPU frame prep threads", + [UI_MENU_REPLAY_WORKERS_DESC] = "Records large frames for the GPU on several threads at once. Helps when the CPU limits the frame rate; needs 8 or more logical processors. Off records on one thread.", + [UI_MENU_REPLAY_WORKERS_NOW] = "now: %s", [UI_MENU_TIMER_1MS] = "1 ms Windows timer", [UI_MENU_TIMER_1MS_DESC] = "Raises the system timer resolution to 1 ms for steadier frame pacing.", [UI_MENU_PIPE_WAIT] = "Waiting draw handling", @@ -862,6 +865,9 @@ static const char *const text_ru[UI_TEXT_COUNT] = { [UI_MENU_PREEMPT_OFF] = "Выкл.", [UI_MENU_PREEMPT_ON] = "Вкл.", [UI_MENU_PREEMPT_INHERIT] = "Только наследование", [UI_MENU_REPLAY_ASYNC] = "Подготовка кадра для GPU в отдельном потоке", [UI_MENU_REPLAY_ASYNC_DESC] = "Кадр для GPU готовится в отдельном потоке, а не в потоке игры. Добавляет до одного кадра задержки.", + [UI_MENU_REPLAY_WORKERS] = "Потоки подготовки кадра для GPU", + [UI_MENU_REPLAY_WORKERS_DESC] = "Большие кадры для GPU записываются сразу в нескольких потоках. Помогает, когда частоту кадров ограничивает процессор; нужно 8 логических процессоров или больше. «Выкл.» — запись в одном потоке.", + [UI_MENU_REPLAY_WORKERS_NOW] = "сейчас: %s", [UI_MENU_TIMER_1MS] = "Таймер Windows 1 мс", [UI_MENU_TIMER_1MS_DESC] = "Повышает точность системного таймера до 1 мс для ровного темпа кадров.", [UI_MENU_PIPE_WAIT] = "Обработка ожидающих отрисовок", @@ -1235,6 +1241,9 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = { [UI_MENU_PREEMPT_OFF] = "Desligado", [UI_MENU_PREEMPT_ON] = "Ligado", [UI_MENU_PREEMPT_INHERIT] = "Só herança", [UI_MENU_REPLAY_ASYNC] = "Preparo de quadro da GPU em thread própria", [UI_MENU_REPLAY_ASYNC_DESC] = "Prepara os quadros para a GPU numa thread separada em vez da thread do jogo. Adiciona até um quadro de latência.", + [UI_MENU_REPLAY_WORKERS] = "Threads de preparo de quadro da GPU", + [UI_MENU_REPLAY_WORKERS_DESC] = "Grava quadros grandes para a GPU em várias threads ao mesmo tempo. Ajuda quando a CPU limita a taxa de quadros; precisa de 8 ou mais processadores lógicos. Desligado grava numa thread só.", + [UI_MENU_REPLAY_WORKERS_NOW] = "agora: %s", [UI_MENU_TIMER_1MS] = "Timer do Windows de 1 ms", [UI_MENU_TIMER_1MS_DESC] = "Aumenta a resolução do timer do sistema para 1 ms, para um ritmo de quadros mais estável.", [UI_MENU_PIPE_WAIT] = "Tratamento de desenhos em espera", @@ -1842,6 +1851,10 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = { [UI_MENU_REPLAY_ASYNC] = "GPU 帧准备独立线程", [UI_MENU_REPLAY_ASYNC_DESC] = "在单独的线程而不是游戏线程上为 GPU 准备帧。" "最多增加一帧延迟。", + [UI_MENU_REPLAY_WORKERS] = "GPU 帧准备线程数", + [UI_MENU_REPLAY_WORKERS_DESC] = "同时在多个线程上为 GPU 录制大帧。CPU 限制帧率时有帮助;需要 8 个或更多逻辑处理器。" + "关:在单个线程上录制。", + [UI_MENU_REPLAY_WORKERS_NOW] = "当前:%s", [UI_MENU_TIMER_1MS] = "1 ms Windows 计时器", [UI_MENU_TIMER_1MS_DESC] = "将系统计时器精度提高到 1 ms,使帧间隔更平稳。", [UI_MENU_PIPE_WAIT] = "等待中绘制的处理", diff --git a/runtime/ui_text.h b/runtime/ui_text.h index 59e277b4..4ed825cc 100644 --- a/runtime/ui_text.h +++ b/runtime/ui_text.h @@ -175,6 +175,7 @@ typedef enum { UI_MENU_ASYNC_PREEMPT, UI_MENU_ASYNC_PREEMPT_DESC, UI_MENU_PREEMPT_OFF, UI_MENU_PREEMPT_ON, UI_MENU_PREEMPT_INHERIT, UI_MENU_REPLAY_ASYNC, UI_MENU_REPLAY_ASYNC_DESC, + UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, UI_MENU_REPLAY_WORKERS_NOW, UI_MENU_TIMER_1MS, UI_MENU_TIMER_1MS_DESC, UI_MENU_PIPE_WAIT, UI_MENU_PIPE_WAIT_DESC, UI_MENU_PIPE_WAIT_DEFAULT, UI_MENU_PIPE_WAIT_DRAW, UI_MENU_PIPE_SKIP, UI_MENU_PIPE_SKIP_DESC, UI_MENU_PIPE_SKIP_STRICT, UI_MENU_PIPE_SKIP_ANY, diff --git a/runtime/vk.h b/runtime/vk.h index 91a67370..a9c88f12 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -226,6 +226,10 @@ enum { VK_MCLS_NONE, VK_MCLS_HOST_NOACC, VK_MCLS_HOST_UNC, VK_MCLS_HOST_CACHED, enum { VK_MUSE_VB, VK_MUSE_IB, VK_MUSE_UBO, VK_MUSE_SSBO, VK_MUSE_IND, VK_MUSE_COPY_SRC, VK_MUSE_GPU_WRITE, VK_MUSE_CPU_READ, VK_MUSE_N }; void vk_mem_count(int cls, int use, uint64_t bytes); +// A thread recording a chunk of a submission (vk_exec.c) counts into its own acc from vk_mem_count_local(acc) on, until +// vk_mem_count_local(NULL); vk_mem_count_flush adds acc into the census and clears it. +void vk_mem_count_local(int64_t (*acc)[VK_MUSE_N]); +void vk_mem_count_flush(int64_t (*acc)[VK_MUSE_N]); // TOTK_TRACE=perf dirty census (vk_memory.c): the replay notes each imported pool it binds as a vertex, index, uniform, // storage or indirect buffer; at the start of every submit the pages the CPU wrote in such pools since the previous // submit are counted from the shared page generations and logged as `mem: dirty` every 60 frames. Nothing when tracing @@ -258,10 +262,14 @@ void vk_mem_gpu_wrote_shader(const VkMemSpan *sp, uint64_t size); // it records, ubo bindings take the copy (0: not copied, bind memory), GPU writes of the replay drop the copies they touch. void vk_usnap_begin(const uint64_t *handles, int n, uint64_t fence); void vk_usnap_end(uint64_t fence); -int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t align, VkDescriptorBufferInfo *out); +// at: the copies dropped at or before it are not found (vk_usnap_drop_at); UINT32_MAX outside a chunked replay. +int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t at, uint64_t align, VkDescriptorBufferInfo *out); const uint8_t *vk_usnap_map(void); // a host copy of the current arena, at the offsets vk_usnap_find gives const uint8_t *vk_usnap_bytes(uint64_t addr, uint64_t n); // the copy of [addr, addr+n) as of the submit, NULL: none void vk_usnap_drop(uint64_t addr, uint64_t n); // the replay makes the GPU write [addr, addr+n) of guest memory +// The state scan of a chunked replay (vk_exec.c): the op that writes [addr, addr+n) is the at-th that may; the chunks +// pass at with the ops they record (VkExec.drop_n) and find the copies only before it. +void vk_usnap_drop_at(uint64_t addr, uint64_t n, uint32_t at); uint64_t vk_usnap_gen(void); // changes whenever a binding may resolve differently void vk_usnap_shutdown(void); // vk_exec_draw_shutdown: frees the arenas // vk_mvprev.c: last frame's copy of the scene's object (c7) and bone (c5) blocks, for per-object motion vectors. @@ -426,6 +434,7 @@ typedef struct { void vk_hdr_present_source(const VkTexImage *win, VkImage *image, VkFormat *format); void vk_hdr_image_gone(VkImage image); const VkTexImage *vk_texture_get(const tex_t *t); // created and made current; NULL for BUFFER or unmapped formats +const VkTexImage *vk_texture_revive(const tex_t *t, uint32_t pos); // the entry vk_texture_get would return if it exists, made live at op pos // A texture slot; a slot whose texture was finalized follows the live texture at the same storage (hle_nvn.c). const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view); // The replay looks up many slots per draw: it takes the texture lock once around them (vk_texture_lock) and uses the @@ -433,6 +442,12 @@ const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view); void vk_texture_lock(void); void vk_texture_unlock(void); const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view); +// 1 when slot id's entry had its freshness check in this submission, so vk_texture_by_id_locked would only look it up. +// No lock: a context of a chunked replay asks before taking the texture lock. +int vk_texture_slot_checked(int id); +// 1 when the state scan revived a texture at the slot's storage in this submission: the slot's answer depends on the op +// a chunk records, so no memo holds it (vk_exec_draw.c); a chunk sees every revival up to its last op. +int vk_texture_slot_moves(int id); VkSampler vk_sampler_by_id_locked(int id); // The slot's sampler with nearest filtering and no anisotropy: a depth texture read without a compare lands on one texel // on the Switch although the game binds a linear sampler, and MacNativeGames samples it the same way. VK_NULL_HANDLE @@ -495,9 +510,22 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t // batch) without a signal, and returns a new begun one for the rest of the submission. The submission's timeline value // is signalled by its last part, whose signal covers every earlier part in submission order. VkCommandBuffer vk_sync_split(void); +// Inside vk_exec_record: submits cb, a part recorded and ended on another thread, after the upload batch and the +// mirror copies queued so far, without a signal (the submitting thread, in the order of the submission). +void vk_sync_part(VkCommandBuffer cb); +// GPU time of a part recorded on another thread (vk_sync.c "GPU time"): the start timestamp after vkBeginCommandBuffer, +// the pair or -1; the end before vkEndCommandBuffer. +int vk_sync_ts_open(VkCommandBuffer cb); +void vk_sync_ts_close(VkCommandBuffer cb, int pair, uint64_t fence); // Runs fn(arg) on the completion thread once fence is done on the GPU (event writes, counter reports, sync state). // Callbacks run in the order they were added, after those of earlier values; a value already done runs at once there. void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg); +// Callbacks of a part recorded on another thread: between vk_completion_defer(d) and vk_completion_defer(NULL) the +// calling thread's vk_completion_add calls collect in d, and vk_completion_flush adds them, in their order, when the +// part takes its place in the submission (vk_exec.c record_chunks). +typedef struct { void *items; int n, cap; } VkDeferred; +void vk_completion_defer(VkDeferred *d); +void vk_completion_flush(VkDeferred *d); // gfx_vulkan_backend (vk_backend.c): presents go to a present thread and run after the submissions made before them. void rdoc_init(void); // runtime/rdoc.c: TOTK_RDOC debug capture; before vkCreateInstance int rdoc_active(void); diff --git a/runtime/vk_contact.c b/runtime/vk_contact.c index 74eae718..15c73045 100644 --- a/runtime/vk_contact.c +++ b/runtime/vk_contact.c @@ -7,9 +7,13 @@ // passes behind nearby geometry: the small shadows at feet, under stones and in grass that the shadow maps are too // coarse for. Transparent surfaces, the sky and post-processing come after, untouched. // -// Inputs from the game, read from the fragment stage's uniform block c4 bound in the composite pass (guest memory, -// so it may be one frame newer than the replay): the projection scales c4[7].x and c4[8].y, the far plane c4[14].y, -// and the first sun cascade's depth row c4[44].xyz, whose negated direction points from a view-space position toward +// vk_contact_plan decides that in replay order and vk_contact_exec records it: a chunked replay's scan +// plans the march (vk_exec.c scan_contact) and the chunk that holds its op records it. +// +// Inputs from the game, read where the march is planned from the fragment stage's uniform block c4 bound in the +// composite pass (guest memory, so it may be one frame newer than the replay): the projection scales c4[7].x and +// c4[8].y, the far plane c4[14].y, and the first sun cascade's depth row c4[44].xyz, whose negated direction points from +// a view-space position toward // the sun (the G-buffer shaders compare dot(c4[44].xyz, pos) + c4[44].w with the cascade map). The scene colour images // get STORAGE usage when the setting is on (vk_texture.c create_tex). #include "vk.h" @@ -23,22 +27,22 @@ #include "vk_exec.h" static struct { - int tried, ok, strength, debug, logged; + int read, tried, ok, strength, debug, logged; PFN_vkCmdPushDescriptorSetKHR push; VkDescriptorSetLayout dsl; VkPipelineLayout layout; VkPipeline pipe; VkSampler sampler; - const VkTexImage *scene, *depth; // the last G-buffer's scene colour and linear depth - uint32_t w, h; + VkExecTarget scene, depth; // the last G-buffer's scene colour and linear depth int pending; // the composite into scene ran; the march waits for the next pass uint64_t c4; } C; int vk_contact_strength(void) { - if (!C.tried) { + if (!C.read) { // once: the scan asks at every draw int s = cfg_int("TOTK_CONTACT_SHADOWS"); C.strength = s < 0 ? 0 : s > 100 ? 100 : s; + C.read = 1; } return C.strength; } @@ -90,16 +94,16 @@ static int init(VkCtx *c) { } // A G-buffer pass: a B10G11R11 scene colour, an R32F linear depth and a D32F depth, all one size, at least 640 wide. -static int is_gbuffer(const VkExec *ex, const VkTexImage **scene, const VkTexImage **depth) { +static int is_gbuffer(const VkExec *ex, const VkExecTarget **scene, const VkExecTarget **depth) { *scene = *depth = NULL; if (ex->pass_noatt || ex->pass_w < 640 || ex->depth.format != VK_FORMAT_D32_SFLOAT || ex->depth.level) return 0; for (uint32_t i = 0; i < ex->pass_ncolor && i < 8; i++) { const VkExecTarget *t = &ex->color[i]; if (!t->image || t->level || t->width != ex->pass_w || t->height != ex->pass_h) continue; - if (t->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32 && !*scene) *scene = t->img; - if (t->format == VK_FORMAT_R32_SFLOAT && !*depth) *depth = t->img; + if (t->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32 && !*scene) *scene = t; + if (t->format == VK_FORMAT_R32_SFLOAT && !*depth) *depth = t; } - return *scene && *depth && ((*scene)->usage & VK_IMAGE_USAGE_STORAGE_BIT); + return *scene && *depth && (*scene)->img && ((*scene)->img->usage & VK_IMAGE_USAGE_STORAGE_BIT); } static void mem_barrier(VkCommandBuffer cb) { @@ -112,28 +116,62 @@ static void mem_barrier(VkCommandBuffer cb) { vkCmdPipelineBarrier2(cb, &d); } -void vk_contact_flush(VkExec *ex) { - if (!C.pending) return; +// The lighting composite: the remembered scene colour as the only target, with a D16 depth of its size. +static int is_composite(const VkExec *ex) { + return C.scene.image && ex->pass_ncolor >= 1 && ex->color[0].image == C.scene.image && !ex->color[0].level && + ex->depth.format == VK_FORMAT_D16_UNORM && ex->pass_w == C.scene.width && ex->pass_h == C.scene.height; +} + +int vk_contact_plan(const VkExec *ex, int pass, VkExecJob *j) { + if (!vk_contact_strength()) return 0; + if (pass) { + const VkExecTarget *scene, *depth; + // The camera block: c4 of the fragment stage, 0x940 bytes. The composite binds it only after its pass begins, so + // the newest one seen at any pass start serves. + const p_bind_buf *b = &ex->ubo[1][1]; + if (b->addr && b->size == 0x940) C.c4 = b->addr; + if (is_gbuffer(ex, &scene, &depth)) { + C.pending = 0; + C.scene = *scene, C.depth = *depth; + return 0; + } + if (is_composite(ex)) { + C.pending = C.c4 != 0; + return 0; + } + } + if (!C.pending) return 0; C.pending = 0; - if (ex->in_pass) return; const float *u = (const float *)(uintptr_t)C.c4; float lx = -u[44 * 4 + 0], ly = -u[44 * 4 + 1], lz = -u[44 * 4 + 2], ll = sqrtf(lx * lx + ly * ly + lz * lz); float px = u[7 * 4 + 0], py = u[8 * 4 + 1], zfar = u[14 * 4 + 1]; - if (!(ll > 1e-6f) || !(px > 0) || !(py > 0) || !(zfar > 1)) return; - VkImageView in = vk_exec_target_view(C.depth, VK_FORMAT_R32_SFLOAT, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT, + if (!(ll > 1e-6f) || !(px > 0) || !(py > 0) || !(zfar > 1)) return 0; + float w = (float)C.scene.width, h = (float)C.scene.height; + *j = (VkExecJob){.scene = C.scene, .depth = C.depth, + .pc = {px, py, zfar, (float)C.strength / 100.0f, + lx / ll, ly / ll, lz / ll, 0.6f, + w, h, 1.0f / w, 1.0f / h, + 0, 0, 0, 0}}; + return 1; +} + +void vk_contact_exec(VkExec *ex, const VkExecJob *j) { + if (!C.tried) init(vk_ctx()); + if (!C.ok) return; + VkImageView in = vk_exec_target_view(j->depth.img, VK_FORMAT_R32_SFLOAT, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1, 0); - VkImageView out = vk_exec_target_view(C.scene, VK_FORMAT_B10G11R11_UFLOAT_PACK32, VK_IMAGE_VIEW_TYPE_2D, + VkImageView out = vk_exec_target_view(j->scene.img, VK_FORMAT_B10G11R11_UFLOAT_PACK32, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1, 0); if (!in || !out) return; + vk_exec_pass_end(ex); VkCommandBuffer cb = ex->cb; - float pcv[16] = {px, py, zfar, (float)C.strength / 100.0f, - lx / ll, ly / ll, lz / ll, 0.6f, - (float)C.w, (float)C.h, 1.0f / (float)C.w, 1.0f / (float)C.h, - (float)C.debug, 0, 0, 0}; + float pcv[16]; + memcpy(pcv, j->pc, sizeof pcv); + pcv[12] = (float)C.debug; if (!C.logged) { C.logged = 1; - rt_log("vk: contact shadows on a %ux%u scene, sun toward %.3f %.3f %.3f (view space), far %g", C.w, C.h, - pcv[4], pcv[5], pcv[6], zfar); + rt_log("vk: contact shadows on a %ux%u scene, sun toward %.3f %.3f %.3f (view space), far %g", j->scene.width, j->scene.height, + pcv[4], pcv[5], pcv[6], pcv[2]); } mem_barrier(cb); // the compose passes' writes, before the march reads and writes them VkDescriptorImageInfo ii = {C.sampler, in, VK_IMAGE_LAYOUT_GENERAL}, oi = {VK_NULL_HANDLE, out, VK_IMAGE_LAYOUT_GENERAL}; @@ -145,36 +183,21 @@ void vk_contact_flush(VkExec *ex) { vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, C.pipe); rec_push(C.push, cb, VK_PIPELINE_BIND_POINT_COMPUTE, C.layout, 0, 2, w); vkCmdPushConstants(cb, C.layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof pcv, pcv); - vkCmdDispatch(cb, (C.w + 7) / 8, (C.h + 7) / 8, 1); + vkCmdDispatch(cb, (j->scene.width + 7) / 8, (j->scene.height + 7) / 8, 1); mem_barrier(cb); // and what follows sees the darkened colour ex->recorded += 6; - vk_exec_draw_new_cb(); // the game's bound compute pipeline and descriptors were replaced: bind them again -} - -// The lighting composite: the remembered scene colour as the only target, with a D16 depth of its size. -static int is_composite(const VkExec *ex) { - return C.scene && ex->pass_ncolor >= 1 && ex->color[0].img == C.scene && !ex->color[0].level && - ex->depth.format == VK_FORMAT_D16_UNORM && ex->pass_w == C.w && ex->pass_h == C.h; + vk_exec_draw_new_cb(ex); // the game's bound compute pipeline and descriptors were replaced: bind them again } +// The sequential replay's hooks; a chunk leaves the marches to the scan. void vk_contact_pass_begin(VkExec *ex) { - if (!C.tried && !init(vk_ctx())) return; - if (!C.ok) return; - const VkTexImage *scene, *depth; - // The camera block: c4 of the fragment stage, 0x940 bytes. The composite binds it only after its pass begins, so - // the newest one seen at any pass start serves. - const p_bind_buf *b = &ex->ubo[1][1]; - if (b->addr && b->size == 0x940) C.c4 = b->addr; - if (is_gbuffer(ex, &scene, &depth)) { - C.pending = 0; - C.scene = scene, C.depth = depth, C.w = ex->pass_w, C.h = ex->pass_h; - return; - } - if (is_composite(ex)) { - C.pending = C.c4 != 0; - return; - } - vk_contact_flush(ex); + VkExecJob j; + if (!ex->chunked && vk_contact_plan(ex, 1, &j)) vk_contact_exec(ex, &j); +} + +void vk_contact_flush(VkExec *ex) { + VkExecJob j; + if (!ex->chunked && C.pending && vk_contact_plan(ex, 0, &j)) vk_contact_exec(ex, &j); } void vk_contact_release(void) { diff --git a/runtime/vk_crumbs.c b/runtime/vk_crumbs.c index 3d34e024..a5a1e84d 100644 --- a/runtime/vk_crumbs.c +++ b/runtime/vk_crumbs.c @@ -91,7 +91,7 @@ void vk_crumbs_report(void) { // records changed pipeline state also notes the shaders of a pass's first draw. A full barrier already precedes every // pass and dispatch (vk_exec.c vk_exec_pass_begin, vk_exec_draw.c op_dispatch), so the end-of-pipe write adds no wait. static VkPassMark ring[VK_PASS_MARKS]; -static uint32_t seq; // marks made; only the replay thread writes it +static uint32_t seq; // marks made (atomic: the recording threads of a chunked replay) // TOTK_PASS_CRUMBS: unset, buffer marks where VK_AMD_buffer_marker exists; 1, also NVIDIA checkpoints; 0, neither. int vk_pass_marks_want_nv(void) { @@ -124,8 +124,8 @@ void vk_pass_mark_cmd(VkCommandBuffer cb, uint32_t id, int end) { VkPassMark *vk_pass_mark_begin(VkCommandBuffer cb, int kind, uint64_t fence) { if (!vk_pass_marks_on || !cb) return NULL; - uint32_t id = ++seq; - if (!id) id = ++seq; // 0 means no mark + uint32_t id = __atomic_add_fetch(&seq, 1, __ATOMIC_RELAXED); // the recording threads of a chunked replay + if (!id) id = __atomic_add_fetch(&seq, 1, __ATOMIC_RELAXED); // 0 means no mark VkPassMark *m = &ring[id & (VK_PASS_MARKS - 1)]; m->id = id, m->kind = (uint8_t)kind, m->fence = fence, m->draws = UINT32_MAX; rec_pass_mark(cb, id, 0); diff --git a/runtime/vk_dlaa.c b/runtime/vk_dlaa.c index 1fe39bde..18bd2ebf 100644 --- a/runtime/vk_dlaa.c +++ b/runtime/vk_dlaa.c @@ -557,7 +557,7 @@ static int mv_pass(VkExec *ex, struct Size *e, VkImageView zv, float jx, float j VkCommandBuffer cb = ex->cb; uint32_t w = e->w, h = e->h; VkDescriptorBufferInfo cam; - if (!mv_init() || !D.cam_addr || !vk_usnap_find(D.cam_addr, CAM_BLOCK, 4, &cam)) { + if (!mv_init() || !D.cam_addr || !vk_usnap_find(D.cam_addr, CAM_BLOCK, UINT32_MAX, 4, &cam)) { D.mv_none++; D.cam_prev_ok = 0; e->chars_ok = 0; @@ -1045,7 +1045,7 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { full_barrier(cb); if (dst != t) vk_dlaa_swap_img = t->img, vk_dlaa_swap_view = e->out_view; // the scaling draw reads DLSS's picture ex->recorded += 4; - vk_exec_draw_new_cb(); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again + vk_exec_draw_new_cb(ex); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again D.jdepth = z->image; next_jitter(w, ow); if (++D.evals == 1 || D.evals == 300 || D.evals % 3600 == 0) // 300: one early line with the frames counted @@ -1195,6 +1195,7 @@ static uint64_t spirv_hash(const GfxShader *s) { } int vk_dlaa_skip_draw(VkExec *ex, const GfxShader *fs) { + if (ex->chunked) return 0; // a chunked replay runs only with fxaa = 1 and without DLSS (vk_exec.c chunked_on) static uint64_t frame_read = ~0ull; static int fxaa = 1; // fxaa = 0 reports once whether it found the game's FXAA draw: the first skip, or a minute of frames (3600) with diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index 6408d2be..8bf87fb7 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -26,8 +26,8 @@ #include "platform/platform.h" #include "tex.h" #include "vk.h" -#include "vk_exec.h" #include "vk_dlaa.h" +#include "vk_exec.h" #define WEAK __attribute__((weak)) #define CALL_DEPTH 8 @@ -43,23 +43,37 @@ typedef struct { static VkExec X; static int x_ready; -static _Thread_local int x_here; // this thread is inside vk_exec_record (vk_exec_inline_begin) +static _Thread_local VkExec *x_cur; // the context this thread records with (vk_exec_inline_begin), NULL outside // ---- TOTK_TRACE=gfx frame counters VkExecFrame vk_exec_frame; static struct { const uint8_t *mem; uint32_t w, h, fmt, n; } fw[64]; // textures written this frame static int nfw; +static PlatMutex fw_lock = PLAT_MUTEX_INIT; // the recording threads of a chunked replay -uint64_t vk_exec_recording_fence(void) { return x_here ? X.fence : 0; } +uint64_t vk_exec_recording_fence(void) { return x_cur ? x_cur->fence : 0; } +uint32_t vk_exec_replay_pos(void) { return x_cur && x_cur->chunked ? x_cur->pos : UINT32_MAX; } + + +// A context's counters go into vk_exec_frame (the submitting thread, after the recording). +static void frame_add(VkExec *ex) { + uint64_t *d = (uint64_t *)&vk_exec_frame, *s = (uint64_t *)&ex->fr; + for (size_t i = 0; i < sizeof ex->fr / sizeof *s; i++) d[i] += s[i]; + memset(&ex->fr, 0, sizeof ex->fr); +} void vk_exec_frame_written(const tex_t *t) { if (!(rt_trace & RT_TRACE_GFX) || !t) return; - for (int i = 0; i < nfw; i++) - if (fw[i].mem == t->mem) { fw[i].n++; return; } - if (nfw == 64) return; - fw[nfw].mem = t->mem, fw[nfw].w = t->d.width, fw[nfw].h = t->d.height, fw[nfw].fmt = (uint32_t)t->d.format; - fw[nfw++].n = 1; + plat_mutex_lock(&fw_lock); + int i = 0; + while (i < nfw && fw[i].mem != t->mem) i++; + if (i < nfw) fw[i].n++; + else if (nfw < 64) { + fw[nfw].mem = t->mem, fw[nfw].w = t->d.width, fw[nfw].h = t->d.height, fw[nfw].fmt = (uint32_t)t->d.format; + fw[nfw++].n = 1; + } + plat_mutex_unlock(&fw_lock); } void vk_exec_frame_log(const tex_t *p, uint64_t frame) { @@ -120,6 +134,7 @@ static void state_reset(VkExec *ex) { ex->depth_range[1] = 1.0f; vk_pipe_state_defaults(&ex->pipe); ex->barrier_at = -1; + ex->oq_open = -1; } // ---- render targets @@ -265,10 +280,11 @@ static void bc_copy_back(VkExec *ex, const VkExecTarget *t) { // A draw into a 3D BC image without block views through an uncompressed view of its blocks: the target is the // image's stand-in (bc_standin). 1 when o is filled (target or left out), 0 for any other target. -static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImage *img, VkExecTarget *o) { +static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImage *img, VkExecTarget *o, int peek) { int bw, bh; if (img->type != VK_IMAGE_TYPE_3D || img->block_views || !nvn_fmt_bpb(t->d.format, &bw, &bh) || bw != 4 || bh != 4) return 0; + if (peek) return 1; // the scan makes no stand-in: left out VkFormat fmt; BcStandIn s; uint32_t level = has_v && v->set_levels ? v->base_level : 0; @@ -299,12 +315,20 @@ static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImag return 1; } -static void resolve_target(const tex_t *t, const view_t *v, int has_v, int depth_slot, VkExecTarget *o) { +enum { HOLD_TEX, HOLD_WRITES, HOLD_N }; +static void scan_hold(int why); + +// scan: the state scan's resolution, which creates nothing and leaves out a texture without an image yet. +static void resolve_target(const tex_t *t, const view_t *v, int has_v, int depth_slot, VkExecTarget *o, const VkExec *scan) { memset(o, 0, sizeof *o); o->tex = t; if (!t) return; - const VkTexImage *img = vk_texture_get(t); - if (img && !depth_slot && bc_target(t, v, has_v, img, o)) return; // a 3D BC image: its stand-in + const VkTexImage *img = scan ? vk_texture_revive(t, scan->pos) : vk_texture_get(t); + if (scan && !img) { + if (t->mem) scan_hold(HOLD_TEX); + return; + } + if (img && !depth_slot && bc_target(t, v, has_v, img, o, scan != NULL)) return; // a 3D BC image: its stand-in VkImageUsageFlags need = depth_slot ? VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT : VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT; int is3d = img && img->type == VK_IMAGE_TYPE_3D && !depth_slot; // slices drawn through 2D views (2D_ARRAY_COMPATIBLE) if (!img || !(img->usage & need) || (img->type != VK_IMAGE_TYPE_2D && !is3d)) { @@ -382,7 +406,7 @@ void vk_exec_noatt_dims(const VkExec *ex, uint32_t *w, uint32_t *h) { } // Resolves ex->rt to this command buffer's images and derives the pass parameters and the pipeline's target formats. -static void resolve_rt(VkExec *ex) { +static void resolve_rt_as(VkExec *ex, int peek) { if (ex->rt_resolved) return; ex->rt_resolved = 1; const p_set_rt *r = &ex->rt; @@ -390,7 +414,7 @@ static void resolve_rt(VkExec *ex) { uint32_t w = UINT32_MAX, h = UINT32_MAX, layers = UINT32_MAX, samples = 0, scale = 1; for (int i = 0; i < 8; i++) { VkExecTarget *t = &ex->color[i]; - resolve_target(i < n ? r->color[i] : NULL, &r->cview[i], i < n && r->has_cview[i], 0, t); + resolve_target(i < n ? r->color[i] : NULL, &r->cview[i], i < n && r->has_cview[i], 0, t, peek ? ex : NULL); ex->pass_color[i] = t->image ? t->format : VK_FORMAT_UNDEFINED; ex->pipe.color_fmt[i] = t->image ? t->nvn_format : 0; if (!t->image) continue; @@ -401,7 +425,7 @@ static void resolve_rt(VkExec *ex) { if (t->scale > scale) scale = t->scale; } VkExecTarget *d = &ex->depth; - resolve_target(r->depth, &r->dview, r->has_dview, 1, d); + resolve_target(r->depth, &r->dview, r->has_dview, 1, d, peek ? ex : NULL); ex->pass_depth = d->image && (d->aspect & VK_IMAGE_ASPECT_DEPTH_BIT) ? d->format : VK_FORMAT_UNDEFINED; ex->pass_stencil = d->image && (d->aspect & VK_IMAGE_ASPECT_STENCIL_BIT) ? d->format : VK_FORMAT_UNDEFINED; ex->pipe.depth_fmt = d->image ? d->nvn_format : 0; @@ -417,7 +441,7 @@ static void resolve_rt(VkExec *ex) { vk_exec_noatt_dims(ex, &w, &h); layers = 1; static int logged; - if (logged < 8 && ex->has_vp && (ex->vp[0][0] != 0 || ex->vp[0][1] != 0 || ex->vp[0][2] != w || ex->vp[0][3] != h)) + if (!peek && logged < 8 && ex->has_vp && (ex->vp[0][0] != 0 || ex->vp[0][1] != 0 || ex->vp[0][2] != w || ex->vp[0][3] != h)) logged++, rt_log("vk_exec: pass without attachments %ux%u for viewport %g,%g %gx%g", w, h, ex->vp[0][0], ex->vp[0][1], ex->vp[0][2], ex->vp[0][3]); } @@ -428,6 +452,8 @@ static void resolve_rt(VkExec *ex) { ex->pipe.samples = (uint8_t)ex->pass_samples; } +static void resolve_rt(VkExec *ex) { resolve_rt_as(ex, 0); } + // The attachment view of a target. A registered texture's view lives as long as its image (vk_exec_target_view); any // other image's view is made for this command buffer and freed at its fence. static VkImageView view_of(VkExec *ex, const VkExecTarget *t) { @@ -474,6 +500,7 @@ static void free_views(void *arg) { // completion thread, once the submission // collision makes a draw wait rather than skip. typedef struct { VkImage image; uint32_t level, layer; uint64_t last, prev; } TargetFrames; static TargetFrames tfr[1024]; +static PlatMutex tfr_lock = PLAT_MUTEX_INIT; // the recording threads of a chunked replay static TargetFrames *tfr_slot(const VkExecTarget *t) { uint64_t h = ((uint64_t)(uintptr_t)t->image ^ (uint64_t)t->level << 56 ^ (uint64_t)t->base_layer << 40) * @@ -487,13 +514,18 @@ static int tfr_match(const TargetFrames *e, const VkExecTarget *t) { static void tfr_note(const VkExecTarget *t, uint64_t stamp) { TargetFrames *e = tfr_slot(t); + plat_mutex_lock(&tfr_lock); if (!tfr_match(e, t)) *e = (TargetFrames){t->image, t->level, t->base_layer, 0, 0}; if (e->last != stamp) e->prev = e->last, e->last = stamp; + plat_mutex_unlock(&tfr_lock); } static int tfr_recent(const VkExecTarget *t, uint64_t stamp) { const TargetFrames *e = tfr_slot(t); - return tfr_match(e, t) && (e->last == stamp ? e->prev : e->last) == stamp - 1; + plat_mutex_lock(&tfr_lock); + int r = tfr_match(e, t) && (e->last == stamp ? e->prev : e->last) == stamp - 1; + plat_mutex_unlock(&tfr_lock); + return r; } // No frame before the current one rendered the target (a slot another subresource took reads the same). @@ -575,7 +607,7 @@ static void barrier_to(VkExec *ex, const VkImage *want, const VkImageAspectFlags for (int i = 0; i < nwant; i++) ex->fbl_img[i] = want[i], ex->fbl_aspect[i] = wasp[i]; ex->nfbl = nwant; if (ex->barrier_at == ex->recorded && !n) return; - vk_exec_frame.fb_layouts += n; + ex->fr.fb_layouts += n; VkMemoryBarrier2 mb = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, @@ -639,8 +671,8 @@ void vk_exec_pass_begin(VkExec *ex) { m->depth = ex->pass_depth ? ex->pass_depth : ex->pass_stencil; // The shaders bound now are the first draw's when it repeats the pipeline state; when it does not, // vk_exec_draw.c prepare_draw replaces them as it records the new state. - vk_exec_draw_shaders(&m->first[0], &m->first[1]); - ex->mark_draws0 = vk_exec_frame.draws; + vk_exec_draw_shaders(ex, &m->first[0], &m->first[1]); + ex->mark_draws0 = ex->fr.draws; } VkRenderingInfo ri = {.sType = VK_STRUCTURE_TYPE_RENDERING_INFO, .renderArea = {{0, 0}, {ex->pass_w, ex->pass_h}}, .layerCount = ex->pass_layers, .colorAttachmentCount = ex->pass_ncolor, @@ -648,7 +680,7 @@ void vk_exec_pass_begin(VkExec *ex) { .pDepthAttachment = ex->pass_depth ? &da : NULL, .pStencilAttachment = ex->pass_stencil ? &sa : NULL}; vkCmdBeginRendering(ex->cb, &ri); - vk_exec_frame.passes++; + ex->fr.passes++; ex->recorded++; ex->in_pass = 1; } @@ -660,10 +692,10 @@ void vk_exec_pass_end(VkExec *ex) { ex->recorded++; ex->in_pass = 0; if (ex->mark) { - // vk_exec_frame_log resets the frame counters at a present (TOTK_TRACE=gfx): then only the draws since it count - uint64_t d = vk_exec_frame.draws, d0 = ex->mark_draws0; + // frame_add empties the context's counters at the end of a submission: then only the draws since it count + uint64_t d = ex->fr.draws, d0 = ex->mark_draws0; ex->mark->draws = (uint32_t)(d >= d0 ? d - d0 : d); - if (ex->mark->draws) vk_exec_draw_shaders(&ex->mark->last[0], &ex->mark->last[1]); + if (ex->mark->draws) vk_exec_draw_shaders(ex, &ex->mark->last[0], &ex->mark->last[1]); vk_pass_mark_end(ex->cb, ex->mark); ex->mark = NULL; } @@ -678,17 +710,17 @@ void vk_exec_pass_end(VkExec *ex) { } VkCommandBuffer vk_exec_inline_begin(int *reopen) { - VkExec *ex = &X; + VkExec *ex = x_cur; *reopen = 0; - if (!x_here || !ex->cb) return VK_NULL_HANDLE; + if (!ex || !ex->cb) return VK_NULL_HANDLE; *reopen = ex->in_pass; vk_exec_barrier(ex); // ends the pass; feedback-layout targets go back to GENERAL return ex->cb; } void vk_exec_inline_end(int reopen) { - X.recorded++; - if (reopen) vk_exec_pass_begin(&X); + x_cur->recorded++; + if (reopen) vk_exec_pass_begin(x_cur); } typedef struct { VkBuffer b; VmaAllocation a; } InlineBuf; @@ -705,7 +737,7 @@ VkBuffer vk_exec_inline_buffer(VkDeviceSize size) { .sharingMode = VK_SHARING_MODE_EXCLUSIVE}; VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; if (!t || vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &t->b, &t->a, NULL) != VK_SUCCESS) { free(t); return VK_NULL_HANDLE; } - vk_completion_add(X.fence, free_inline_buf, t); + vk_completion_add(x_cur->fence, free_inline_buf, t); return t->b; } @@ -746,7 +778,7 @@ static void clear_attachment(VkExec *ex, const VkClearAttachment *a) { } static void clear_color(VkExec *ex, const p_clear_color *c) { - vk_exec_frame.clears++; + ex->fr.clears++; if (ex->render_disabled) return; resolve_rt(ex); int i = c->index; @@ -824,8 +856,8 @@ static void set_rt(VkExec *ex, const p_set_rt *r) { // A part also spans at least TOTK_SPLIT_US (1000) of replay time: a split costs about 40 us of the render thread // (vkQueueSubmit2, a command buffer, the mirror prologue), and a fight frame of 2,000 draws made 15 parts on the 7900X; // a millisecond of recording is a fraction of a millisecond of GPU work, so the GPU still starts early. -static int split_n = -1, split_draws, part_start; -static uint64_t split_ns, part_t0; +static int split_n = -1; +static uint64_t split_ns; static uint64_t n_parts, n_subs; static void maybe_split(VkExec *ex) { @@ -837,103 +869,728 @@ static void maybe_split(VkExec *ex) { rt_log("vk: replay command buffer split every %d draws between passes, parts of at least %d us%s", split_n, us > 0 ? us : 0, split_n ? "" : " (off)"); } - if (!split_n || ex->in_pass || split_draws < split_n) return; - if (split_ns && kern_now_ns() - part_t0 < split_ns) return; // split_draws stays: the next draw asks again - split_draws = 0; - if (ex->recorded == part_start) return; + if (!split_n || ex->in_pass || ex->split_draws < split_n) return; + if (split_ns && kern_now_ns() - ex->part_t0 < split_ns) return; // split_draws stays: the next draw asks again + ex->split_draws = 0; + if (ex->recorded == ex->part_start) return; ex->cb = vk_sync_split(); - part_start = ex->recorded; - part_t0 = kern_now_ns(); + ex->part_start = ex->recorded; + ex->part_t0 = kern_now_ns(); ex->dirty = VK_EXEC_DIRTY_ALL; - vk_exec_draw_new_cb(); + vk_exec_draw_new_cb(ex); n_parts++; } -static void walk(VkExec *ex, const op_hdr *h, int depth) { +// The fixed-function state ops: writes into ex alone, applied by the replay and the state scan. 0: not one of them. +static int state_op(VkExec *ex, uint32_t op, const void *a) { + switch (op) { + case OP_VIEWPORTS: { + const p_viewports *v = a; + for (int i = 0; i < v->n && v->first + i < 16; i++) + if (v->first + i >= 0) memcpy(ex->vp[v->first + i], v->v[i], sizeof ex->vp[0]); + ex->has_vp = 1; + ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; + break; + } + case OP_SCISSOR: + ex->scissor = *(const p_scissor *)a; + ex->has_scissor = 1; + ex->dirty |= VK_EXEC_DIRTY_SCISSOR; + break; + case OP_DEPTH_RANGE: { + const p_depth_range *r = a; + ex->depth_range[0] = r->n, ex->depth_range[1] = r->f; + ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; + break; + } + case OP_STENCIL: { + const p_stencil *s = a; + for (int f = 0; f < 2; f++) { + if (!(s->face & (1 << f))) continue; + if (s->which == 0) ex->st_vmask[f] = (uint32_t)s->value_mask; + else if (s->which == 1) ex->st_wmask[f] = (uint32_t)s->mask; + else ex->st_ref[f] = (uint32_t)s->ref; + } + ex->dirty |= VK_EXEC_DIRTY_STENCIL; + break; + } + case OP_POLY_OFFSET: + ex->poly_offset = *(const p_poly_offset *)a; + ex->dirty |= VK_EXEC_DIRTY_DEPTH_BIAS; + break; + case OP_BLEND_COLOR: + memcpy(ex->blend_color, ((const p_blend_color *)a)->rgba, sizeof ex->blend_color); + ex->dirty |= VK_EXEC_DIRTY_BLEND_CONST; + break; + case OP_DEPTH_CLAMP: ex->pipe.depth_clamp = ((const p_flag *)a)->value != 0; break; + case OP_RASTER_DISCARD: ex->pipe.raster_discard = ((const p_flag *)a)->value != 0; break; + case OP_PRIM_RESTART: ex->prim_restart = *(const p_prim_restart *)a; break; + case OP_COPY_STRIDE: ex->copy_stride = ((const p_copy_stride *)a)->stride; break; + case OP_SET_TEX_POOL: ex->tex_pool = ((const p_set_pool *)a)->pool, ex->tex_serial++; break; + case OP_SET_SAMP_POOL: ex->samp_pool = ((const p_set_pool *)a)->pool; break; + default: return 0; + } + return 1; +} + +// A contact shadow march the scan planned for this point (vk_contact_plan): recorded before the op that triggers it, the +// draw's pass begun after it as in prepare_draw. The marches share vk_contact.c's pipeline and its first use, so +// they go in the scan's order; chunks are taken in order, so the chunk of an earlier record is being recorded or done. +static uint32_t job_turn; + +static void run_job(VkExec *ex) { + uint32_t turn = ex->job_i; + VkExecJob j = ex->jobs[ex->job_i++]; + while (__atomic_load_n(&job_turn, __ATOMIC_ACQUIRE) != turn) plat_yield(); + vk_exec_pass_end(ex); + // The scan resolved the scene and its depth without creating entries (vk_texture_revive): their images now. + const VkTexImage *si = vk_texture_get(j.scene.tex), *di = vk_texture_get(j.depth.tex); + if (si && di) { + j.scene.img = si, j.scene.image = si->image, j.depth.img = di, j.depth.image = di->image; + vk_contact_exec(ex, &j); + } + __atomic_store_n(&job_turn, turn + 1, __ATOMIC_RELEASE); + ex->dirty = VK_EXEC_DIRTY_ALL; + vk_exec_draw_new_cb(ex); +} + +// Replays from h to OP_END; 1 when it stopped at stop instead, an op at depth 0 (a chunk's end, record_chunks). +// The ops that may make the GPU write a uniform range the submission's snapshot copied (vk_usnap_drop_at). +static int drop_op(uint32_t op) { return op == OP_UPDATE_UB || op == OP_COPY_B2B || op == OP_COPY_T2B || op == OP_CLEAR_BUF; } + +static int walk(VkExec *ex, const op_hdr *h, int depth, const op_hdr *stop) { for (;;) { if (h->size < sizeof *h || (h->size & 7) || h->op >= OP_COUNT_) rt_fatal("vk: corrupt command record at %p: op %u size %u", (const void *)h, h->op, h->size); + if (h == stop) return 1; + while (ex->job_i < ex->njobs && h == ex->jobs[ex->job_i].op) run_job(ex); const void *a = nvn_op_payload(h); + if (h->op == OP_END) return 0; + if (ex->chunked) { + ex->drop_n += (uint32_t)drop_op(h->op), ex->pos++; + if (ex->wpos_i < ex->wpos_end && ex->wpos[ex->wpos_i] == ex->pos) ex->drop_n++, ex->wpos_i++; + } switch (h->op) { - case OP_END: return; case OP_BEGIN: break; case OP_JUMP: h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; continue; case OP_CALL: if (depth >= CALL_DEPTH) rt_fatal("vk: command buffer calls nested deeper than %d", CALL_DEPTH); - walk(ex, nvn_op_first(((const p_jump *)a)->target), depth + 1); + walk(ex, nvn_op_first(((const p_jump *)a)->target), depth + 1, NULL); break; case OP_SET_RT: set_rt(ex, a); break; case OP_CLEAR_COLOR: clear_color(ex, a); break; case OP_CLEAR_DS: clear_ds(ex, a); break; - case OP_VIEWPORTS: { - const p_viewports *v = a; - for (int i = 0; i < v->n && v->first + i < 16; i++) - if (v->first + i >= 0) memcpy(ex->vp[v->first + i], v->v[i], sizeof ex->vp[0]); - ex->has_vp = 1; - ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; - break; - } - case OP_SCISSOR: - ex->scissor = *(const p_scissor *)a; - ex->has_scissor = 1; - ex->dirty |= VK_EXEC_DIRTY_SCISSOR; - break; - case OP_DEPTH_RANGE: { - const p_depth_range *r = a; - ex->depth_range[0] = r->n, ex->depth_range[1] = r->f; - ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; - break; - } - case OP_STENCIL: { - const p_stencil *s = a; - for (int f = 0; f < 2; f++) { - if (!(s->face & (1 << f))) continue; - if (s->which == 0) ex->st_vmask[f] = (uint32_t)s->value_mask; - else if (s->which == 1) ex->st_wmask[f] = (uint32_t)s->mask; - else ex->st_ref[f] = (uint32_t)s->ref; - } - ex->dirty |= VK_EXEC_DIRTY_STENCIL; - break; - } - case OP_POLY_OFFSET: - ex->poly_offset = *(const p_poly_offset *)a; - ex->dirty |= VK_EXEC_DIRTY_DEPTH_BIAS; - break; - case OP_BLEND_COLOR: - memcpy(ex->blend_color, ((const p_blend_color *)a)->rgba, sizeof ex->blend_color); - ex->dirty |= VK_EXEC_DIRTY_BLEND_CONST; - break; - case OP_DEPTH_CLAMP: ex->pipe.depth_clamp = ((const p_flag *)a)->value != 0; break; - case OP_RASTER_DISCARD: ex->pipe.raster_discard = ((const p_flag *)a)->value != 0; break; - case OP_PRIM_RESTART: ex->prim_restart = *(const p_prim_restart *)a; break; - case OP_COPY_STRIDE: ex->copy_stride = ((const p_copy_stride *)a)->stride; break; - case OP_SET_TEX_POOL: ex->tex_pool = ((const p_set_pool *)a)->pool, ex->tex_serial++; break; - case OP_SET_SAMP_POOL: ex->samp_pool = ((const p_set_pool *)a)->pool; break; case OP_BARRIER: vk_exec_barrier(ex); break; case OP_DRAW: case OP_DRAW_IDX: case OP_DISPATCH: case OP_UPDATE_UB: case OP_DRAW_IDX_IND: case OP_DISPATCH_IND: - if (h->op == OP_DRAW || h->op == OP_DRAW_IDX || h->op == OP_DRAW_IDX_IND) vk_exec_frame.draws_seen++, split_draws++; - if (h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND) vk_exec_frame.dispatches++, split_draws++; + if (h->op == OP_DRAW || h->op == OP_DRAW_IDX || h->op == OP_DRAW_IDX_IND) ex->fr.draws_seen++, ex->split_draws++; + if (h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND) ex->fr.dispatches++, ex->split_draws++; case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: case OP_BIND_UBO: case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: vk_exec_op_draw(ex, h->op, a); break; case OP_COPY_B2B: case OP_COPY_B2T: case OP_COPY_T2B: case OP_COPY_T2T: case OP_CLEAR_TEX: - if (h->op != OP_CLEAR_TEX) vk_exec_frame.copies++; + if (h->op != OP_CLEAR_TEX) ex->fr.copies++; if (vk_dump_armed) vk_dump_copy(ex, h->op, a); // fall through case OP_SIGNAL_EVENT: case OP_REPORT_COUNTER: case OP_RESET_COUNTER: case OP_RENDER_ENABLE: case OP_RENDER_COND: case OP_CLEAR_BUF: vk_exec_op_xfer(ex, h->op, a); break; - default: rt_fatal("vk: op %s has no replay handler", nvn_op_name(h->op)); + default: + if (!state_op(ex, h->op, a)) rt_fatal("vk: op %s has no replay handler", nvn_op_name(h->op)); } - if (split_draws >= split_n) maybe_split(ex); + if (ex->split_draws >= split_n && !ex->chunked) maybe_split(ex); h = nvn_op_next(h); } } +// ---- chunked recording (TOTK_REPLAY_WORKERS) +// A submission is recorded in chunks that start between passes, by TOTK_REPLAY_WORKERS threads (0: the sequential +// replay, the default) and twice as many chunks (TOTK_REPLAY_CHUNKS), so a thread +// that finishes early takes another: in the town at 4K on an RTX 5070 the replay goes from 8.2 to 4.5 ms +// and the frame rate from 71 to 79; more threads add little, since the frame then waits for the GPU. A sequential scan on +// the submitting thread applies only what is queue state: the fixed-function and bind ops, render targets, render +// enable and conditions, samples-passed counting and the contact shadows' marches (vk_contact_plan); it looks up no +// shader, pipeline or descriptor and makes no Vulkan call. Where a chunk starts it keeps a copy of that state +// (VK_EXEC_STATE_BEGIN..END) and how far the scan's lists got: the submission's inline uniform updates (ubo_ptr), +// RENDER_COND results, planned contact shadow marches and the ops that drop uniform copies (vk_usnap_drop_at, +// scan_writes), which a chunk takes +// from the lists instead of from earlier chunks. A chunk starts with every image in GENERAL and ends with a full +// barrier, as a part of vk_sync_split does, so its recording depends only on the copy and the lists. +// A chunk can be taken as soon as the scan has placed the next one and the threads finished what the chunks share, which +// they do beside the scan (prep_run); they record into command buffers of their own, each with a context of its own +// (memos, descriptor pools, transient memory), and make their Vulkan calls themselves. The submitting thread records none while there are such threads: it submits the chunks in order as they +// are done, between the scan's steps and after it (vk_sync_part: the uploads and mirror copies queued so far go first), +// with their completion callbacks, so the callbacks keep the order of the sequential replay. The first chunk is a +// quarter of the others: the GPU starts sooner. A chunk ends where the replay ends a pass once it has its share of +// draws, or inside a pass at half as many more (TotK's G-buffer and shadow passes have 1,000 to 2,000 draws): the next +// chunk begins the pass again on the loaded attachments. +// The scan resolves render targets without creating images and revives finalized ones at their op (vk_texture_revive); +// a chunk, which the scan is ahead of, asks for a texture slot as of its own op (VkExec.pos, vk_exec_replay_pos). +// A texture that has no entry yet is made by the chunk that reaches it, and only that chunk has its position: its +// content from a live texture at the storage (vk_texture.c alias_copy) goes at that op. A later chunk sampling it first +// would follow another texture at the storage or copy over what the earlier chunk renders, so the chunks after the +// first use wait until that chunk is recorded (SC.holds). TotK makes a few such targets a frame: thumbnails, and depth +// targets whose size follows the camera; a wrong one darkened the whole scene for some frames. +// TOTK_REPLAY_STATE_CHECK=1 compares the state after the last chunk with the scan's. +#define MAX_JOBS 64 +#define MAX_RECS 16 + +typedef struct { + int hi; // handle index and record of the first op + const op_hdr *op; + uint32_t upd_n, cond_i, job_i, job_end, drop_n, pos; // the scan's lists, drop and op positions before it; its records + uint32_t wpos_i, wpos_end; // its writing draws and dispatches in SC.wpos + VkExec st; // queue state at its start + VkCommandBuffer cb; // recorded, ended, submitted in its turn + VkDeferred defer; // its completion callbacks + VkExec *by; // the context that recorded it + uint64_t ns, start; // its recording time, and when it began after SC.t0 + int draws; // draws and dispatches in it + volatile int32_t done; +} Chunk; + +typedef struct { + VkExec ex; // the thread's recording context + VkCommandPool pool; + struct { VkCommandBuffer cb; uint64_t fence; } *cbs; // reusable once the timeline reached fence + int ncbs; + int64_t traffic[VK_MCLS_N][VK_MUSE_N]; // vk_mem_count_local + PlatThread *t; +} Rec; + +typedef struct { uint64_t addr, size; uint32_t at; } ScanDrop; + +static struct { + int workers, chunks, min_draws, check; + VkExec s; // the scan's state + Chunk *ch; // chunks + 1 entries, never moved while threads record + int nch; + VkExecUpd *upd; // sized before the scan (count_ops): threads read them while it appends + uint32_t nupd, cupd; + uint8_t *cond; + uint32_t ncond, ccond; + VkExecJob jobs[MAX_JOBS]; + uint32_t njobs; + uint32_t drop_n; // scan: ops so far that may write a uniform range + uint32_t *wpos; // scan: the positions of the draws and dispatches among them, sized like upd + uint32_t nwpos, cwpos; + ScanDrop *dr; // scan: its uniform drops, applied once the snapshot exists (chunks_publish) + uint32_t ndr, cdr, dr_done; + int pub; // chunks published + uint64_t holds; // chunks that make a new texture or record writes the scan does not know: the + // ones after them wait until they are done + int per, open, in_chunk; // scan: draws for this chunk, a pass of the replay may be open, draws in it + int sub; // chunks submitted + uint64_t t0; // the submission's start + // TOTK_TRACE=perf, per chunked submission: the scan, the first and the last chunk submitted after t0, the chunks' + // recording summed and the longest + uint64_t subs, chunked, nchunks, scan_ns, first_ns, last_ns, rec_ns, rec_max_ns, bad, prep_ns, start0_ns; + uint64_t rec_t0; // vk_exec_record's start + uint64_t held[HOLD_N]; // TOTK_TRACE=perf: chunks held, by scan_hold's reason + uint64_t mirror_ns, usnap_ns, ready_ns; // vk_mem_mirror_begin, vk_usnap_begin, both done after t0 +} SC = {.workers = -1}; + +static Rec recs[MAX_RECS + 1]; // [0]: the submitting thread's, used without recording threads +static int nrecs, nthreads; +static struct { + PlatMutex m; + PlatCond go, done; // more chunks can be taken or a new submission; a chunk is done + uint32_t gen; // bumped for every chunked submission + // gen << 32 | scan done << 31 | chunks that can be taken << 16 | next to take + volatile uint64_t take; + const uint64_t *handles; + int n, prio; + uint64_t fence; + int prep_next, prep_left; // prep_run: the next job, the jobs not done +} RP; + +static void *grow_n(void *p, uint32_t *cap, size_t elem) { + *cap = *cap ? *cap * 2 : 64; + p = realloc(p, elem * *cap); + if (!p) rt_fatal("vk: out of memory"); + return p; +} + +static void copy_state(VkExec *dst, const VkExec *src) { + memcpy((char *)dst + VK_EXEC_STATE_BEGIN, (const char *)src + VK_EXEC_STATE_BEGIN, VK_EXEC_STATE_END - VK_EXEC_STATE_BEGIN); +} + +static int draw_op(uint32_t op) { return op == OP_DRAW || op == OP_DRAW_IDX || op == OP_DRAW_IDX_IND; } + +static void check_op(const op_hdr *h) { + if (h->size < sizeof *h || (h->size & 7) || h->op >= OP_COUNT_) + rt_fatal("vk: corrupt command record at %p: op %u size %u", (const void *)h, h->op, h->size); +} + +// Draws and dispatches, inline uniform updates and conditions of a command stream. +typedef struct { uint32_t draws, upd, cond; } OpCounts; + +static void count_ops(const op_hdr *h, int depth, OpCounts *c) { + for (;;) { + check_op(h); + const void *a = nvn_op_payload(h); + if (h->op == OP_END) return; + if (h->op == OP_JUMP) { + h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; + continue; + } + if (h->op == OP_CALL && depth < CALL_DEPTH) count_ops(nvn_op_first(((const p_jump *)a)->target), depth + 1, c); + c->draws += draw_op(h->op) || h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND; + c->upd += h->op == OP_UPDATE_UB; + c->cond += h->op == OP_RENDER_COND; + h = nvn_op_next(h); + } +} + +// The take word: chunks [0, avail) can be taken, final once the scan placed them all. +static void take_publish(int avail, int final) { + plat_mutex_lock(&RP.m); + uint64_t w = __atomic_load_n(&RP.take, __ATOMIC_RELAXED); + while (!__atomic_compare_exchange_n(&RP.take, &w, (w & ~(0xffffull << 16 | 1ull << 31)) | (uint64_t)avail << 16 | (uint64_t)final << 31, + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) {} + plat_cond_broadcast(&RP.go); + plat_mutex_unlock(&RP.m); +} + +// What every chunk needs before it records, independent of each other and of the scan, so they run beside it on the +// recording threads and the submission's chunks start sooner: the mirrors' write-watch queries and the uniform snapshot. +static void prep_run(void) { + for (;;) { + int j = __atomic_fetch_add(&RP.prep_next, 1, __ATOMIC_ACQ_REL); + if (j >= 2) return; + uint64_t t0 = kern_now_ns(); + if (j == 0) vk_mem_mirror_begin(); + else vk_usnap_begin(RP.handles, RP.n, RP.fence); + uint64_t t1 = kern_now_ns(); + if (j == 0) SC.mirror_ns += t1 - t0; + else SC.usnap_ns += t1 - t0; + if (__atomic_load_n(&RP.prep_left, __ATOMIC_RELAXED) == 1) SC.ready_ns += t1 - SC.t0; + __atomic_sub_fetch(&RP.prep_left, 1, __ATOMIC_RELEASE); + } +} + +// Chunks [0, avail) less those behind a hold that is not done. +static int chunks_allowed(int avail) { + for (uint64_t m = SC.holds; m; m &= m - 1) { + int k = __builtin_ctzll(m); + if (k + 1 >= avail) break; + if (!__atomic_load_n(&SC.ch[k].done, __ATOMIC_ACQUIRE)) return k + 1; + } + return avail; +} + +// The chunks placed so far can be taken once prep_run is done, with the uniform drops the scan found up to them. +static void chunks_publish(int final) { + int all = final ? SC.nch : SC.nch - 1, avail = chunks_allowed(all); + final = final && avail == all; // the threads go to sleep on the final word: only with nothing held back + if ((avail == SC.pub && !final) || __atomic_load_n(&RP.prep_left, __ATOMIC_ACQUIRE)) return; + for (; SC.dr_done < SC.ndr; SC.dr_done++) vk_usnap_drop_at(SC.dr[SC.dr_done].addr, SC.dr[SC.dr_done].size, SC.dr[SC.dr_done].at); + SC.pub = avail; + take_publish(avail, final); +} + +// TOTK_TRACE=perf counts the chunks held per reason (SC.held). +static void scan_hold(int why) { + if (SC.nch <= 0 || SC.nch > 64 || SC.holds >> (SC.nch - 1) & 1) return; + SC.holds |= 1ull << (SC.nch - 1); + SC.held[why]++; +} + +static void scan_drop(uint64_t addr, uint64_t size) { + if (SC.ndr == SC.cdr) SC.dr = grow_n(SC.dr, &SC.cdr, sizeof *SC.dr); + SC.dr[SC.ndr++] = (ScanDrop){addr, size, SC.drop_n}; +} + +// A draw or dispatch whose shaders write storage buffers drops the uniform copies they cover from its own op on, as +// the sequential replay does in bind_range. The scan looks up no shader: it takes the ones the contexts translated +// (vk_exec_draw_writes). With one not translated yet the op still counts, the chunk that records it drops what the +// shader writes at its position (bind_range), and the chunks after it wait until it is recorded. Without this a chunk +// recorded before an earlier chunk's write bound the copy taken at the submit call, and on an RX 7800 XT the grass +// near the camera came out dark or unshaded in some runs. +static void scan_writes(VkExec *s, uint32_t op) { + uint64_t w[16][2]; + s->chunked = 1, s->upd = SC.upd, s->upd_n = SC.nupd; // ubo_ptr reads the scan's updates + int n = vk_exec_draw_writes(s, op == OP_DISPATCH || op == OP_DISPATCH_IND, w, 16); + s->chunked = 0; + if (!n) return; + SC.drop_n++; + SC.wpos[SC.nwpos++] = s->pos; + if (n < 0) scan_hold(HOLD_WRITES); + for (int i = 0; i < n; i++) scan_drop(w[i][0], w[i][1]); +} + +// The next chunk of submission gen nobody took: -1 none yet, -2 none left (or a later submission). +static int chunk_take(uint32_t gen) { + uint64_t w = __atomic_load_n(&RP.take, __ATOMIC_ACQUIRE); + for (;;) { + if ((uint32_t)(w >> 32) != gen) return -2; + if ((w & 0xffff) >= (w >> 16 & 0x7fff)) return w >> 31 & 1 ? -2 : -1; + if (__atomic_compare_exchange_n(&RP.take, &w, w + 1, 0, __ATOMIC_ACQ_REL, __ATOMIC_ACQUIRE)) return (int)(w & 0xffff); + } +} + +static void views_release(VkExec *ex) { + if (!ex->nviews) return; + ViewList *l = malloc(sizeof *l + sizeof(VkImageView) * (size_t)ex->nviews); + if (!l) rt_fatal("vk: out of memory"); + l->n = ex->nviews; + for (int i = 0; i < ex->nviews; i++) l->v[i] = ((ViewEnt *)ex->views)[i].view; + vk_completion_add(ex->fence, free_views, l); + ex->nviews = 0; +} + +static VkCommandBuffer rec_cb(Rec *r, uint64_t fence) { + VkCtx *c = vk_ctx(); + if (!r->pool) { + VkCommandPoolCreateInfo pi = {.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO, + .flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT, + .queueFamilyIndex = c->gfx_family}; + if (vkCreateCommandPool(c->device, &pi, NULL, &r->pool) != VK_SUCCESS) rt_fatal("vk: chunk command pool"); + } + uint64_t gpu = vk_sync_fence_done(); + int i = 0; + while (i < r->ncbs && r->cbs[i].fence > gpu) i++; + if (i == r->ncbs) { + r->cbs = realloc(r->cbs, sizeof *r->cbs * (size_t)(r->ncbs + 1)); + if (!r->cbs) rt_fatal("vk: out of memory (command buffers)"); + VkCommandBufferAllocateInfo ai = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO, .commandPool = r->pool, + .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, .commandBufferCount = 1}; + if (vkAllocateCommandBuffers(c->device, &ai, &r->cbs[i].cb) != VK_SUCCESS) rt_fatal("vk: command buffer allocation"); + r->ncbs++; + } + r->cbs[i].fence = fence; + VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vkBeginCommandBuffer(r->cbs[i].cb, &bi); + return r->cbs[i].cb; +} + +// Chunk k on the calling thread with context r: from its state copy up to the next chunk's first op, into a command +// buffer of r's pool, ending with every image in GENERAL. +static void chunk_record(Rec *r, int k) { + uint64_t t0 = kern_now_ns(); + VkExec *ex = &r->ex; + Chunk *c = &SC.ch[k]; + c->start = t0 - SC.t0; + const Chunk *next = k + 1 < __atomic_load_n(&SC.nch, __ATOMIC_ACQUIRE) ? &SC.ch[k + 1] : NULL; + ex->cb = rec_cb(r, RP.fence); + ex->fence = RP.fence, ex->recorded = 0, ex->in_pass = 0, ex->nviews = 0, ex->barrier_at = -1, ex->nfbl = 0; + ex->part_start = 0; + x_cur = ex; + vk_completion_defer(&c->defer); + vk_mem_count_local(r->traffic); + int ts = vk_sync_ts_open(ex->cb); + copy_state(ex, &c->st); + ex->rt_resolved = 0, ex->dirty = VK_EXEC_DIRTY_ALL, ex->tex_serial++, ex->fb_fs = NULL; + vk_exec_draw_new_cb(ex); + ex->chunked = 1, ex->upd = SC.upd, ex->cond = SC.cond, ex->jobs = SC.jobs, ex->njobs = c->job_end; + ex->upd_n = c->upd_n, ex->cond_i = c->cond_i, ex->job_i = c->job_i, ex->drop_n = c->drop_n, ex->pos = c->pos; + ex->wpos = SC.wpos, ex->wpos_i = c->wpos_i, ex->wpos_end = c->wpos_end; + for (int i = c->hi; i < RP.n; i++) + if (walk(ex, i == c->hi ? c->op : nvn_op_first(RP.handles[i]), 0, next && next->hi == i ? next->op : NULL)) break; + vk_exec_barrier(ex); + vk_exec_oq_flush(ex); + ex->chunked = 0, ex->njobs = ex->job_i = 0; + views_release(ex); + vk_sync_ts_close(ex->cb, ts, RP.fence); + vk_completion_defer(NULL); + vk_mem_count_local(NULL); + vk_mem_count_flush(r->traffic); + x_cur = NULL; + vkEndCommandBuffer(ex->cb); + c->cb = ex->cb, c->by = ex, c->ns = kern_now_ns() - t0; + plat_mutex_lock(&RP.m); + __atomic_store_n(&c->done, 1, __ATOMIC_RELEASE); + plat_cond_broadcast(&RP.done); + plat_mutex_unlock(&RP.m); +} + +static void rec_main(void *arg) { + Rec *r = arg; + plat_thread_priority(RP.prio); + uint32_t seen = 0; + for (;;) { + plat_mutex_lock(&RP.m); + while (RP.gen == seen) plat_cond_wait(&RP.go, &RP.m); + seen = RP.gen; + plat_mutex_unlock(&RP.m); + prep_run(); + for (;;) { + uint64_t w = __atomic_load_n(&RP.take, __ATOMIC_ACQUIRE); + int k = chunk_take(seen); + if (k >= 0) chunk_record(r, k); + if (k != -1) { + if (k == -2) break; + continue; + } + plat_mutex_lock(&RP.m); + while (__atomic_load_n(&RP.take, __ATOMIC_ACQUIRE) == w) plat_cond_wait(&RP.go, &RP.m); + plat_mutex_unlock(&RP.m); + } + } +} + +// TOTK_REPLAY_WORKERS threads record; with one, the submitting thread does. +static void recs_start(void) { + if (nrecs) return; + nthreads = SC.workers > 1 ? (SC.workers < MAX_RECS ? SC.workers : MAX_RECS) : 0; + nrecs = nthreads + 1; + RP.prio = plat_thread_priority_get(); + for (int i = 0; i < nrecs; i++) { + state_reset(&recs[i].ex); + if (i && !(recs[i].t = plat_thread_new(rec_main, &recs[i], "vk replay"))) rt_fatal("vk: no replay thread"); + } +} + +// Submits the chunks that are done, in order; wait: until all are. +static void chunks_submit(int wait) { + while (SC.sub < SC.nch) { + Chunk *c = &SC.ch[SC.sub]; + if (wait) chunks_publish(1); // a hold may have ended + if (!__atomic_load_n(&c->done, __ATOMIC_ACQUIRE)) { + if (!wait) return; + int k = nthreads ? -1 : chunk_take(RP.gen); + if (k >= 0) { + chunk_record(&recs[0], k); + continue; + } + plat_mutex_lock(&RP.m); + while (!__atomic_load_n(&c->done, __ATOMIC_ACQUIRE)) plat_cond_wait(&RP.done, &RP.m); + plat_mutex_unlock(&RP.m); + } + vk_completion_flush(&c->defer); + vk_sync_part(c->cb); + uint64_t t = kern_now_ns() - SC.t0; + if (!SC.sub) SC.first_ns += t, SC.start0_ns += c->start; + SC.last_ns += SC.sub == SC.nch - 1 ? t : 0; + SC.rec_ns += c->ns; + if (c->ns > SC.rec_max_ns) SC.rec_max_ns = c->ns; + SC.sub++; + n_parts++; + } +} + +// 1 when the replay ends an open pass at this op, so a chunk starting here splits no pass. +static int scan_ends_pass(const VkExec *s, uint32_t op, const void *a) { + switch (op) { + case OP_SET_RT: return memcmp(&s->rt, a, sizeof s->rt) != 0; + case OP_BARRIER: case OP_DISPATCH: case OP_DISPATCH_IND: case OP_UPDATE_UB: case OP_COPY_B2B: case OP_COPY_B2T: + case OP_COPY_T2B: case OP_COPY_T2T: case OP_CLEAR_TEX: case OP_CLEAR_BUF: return 1; + default: return 0; + } +} + +// Places chunk nch at h; the chunk before it can be taken from now on. +static void chunk_start(const VkExec *s, int hi, const op_hdr *h) { + Chunk *c = &SC.ch[SC.nch]; + c->hi = hi, c->op = h, c->upd_n = SC.nupd, c->cond_i = SC.ncond, c->job_i = SC.njobs, c->drop_n = SC.drop_n, c->pos = s->pos; + c->wpos_i = SC.nwpos; + c->done = 0, c->cb = VK_NULL_HANDLE, c->by = NULL; + copy_state(&c->st, s); + if (SC.nch) SC.ch[SC.nch - 1].job_end = SC.njobs, SC.ch[SC.nch - 1].wpos_end = SC.nwpos, SC.ch[SC.nch - 1].draws = SC.in_chunk; + SC.in_chunk = 0; + __atomic_store_n(&SC.nch, SC.nch + 1, __ATOMIC_RELEASE); + if (SC.nch > 1) chunks_publish(0); + if (SC.nch == 2) SC.per *= 4; // the first chunk was a quarter + chunks_submit(0); +} + +static void scan_job(const VkExecJob *j, const op_hdr *h) { + static int full; + if (SC.njobs < MAX_JOBS) SC.jobs[SC.njobs] = *j, SC.jobs[SC.njobs++].op = h; + else if (!full++) rt_log("vk_exec: more than %d contact shadow marches in a submission: the others are left out", MAX_JOBS); +} + +// What vk_contact_pass_begin (pass) and vk_contact_flush do in the sequential replay, in replay order. A pass of the +// replay may start at any draw or clear here: the plan only looks at the targets, so asking again changes nothing. +static void scan_contact(const VkExec *s, int pass, const op_hdr *h) { + VkExecJob j; + if (vk_contact_plan(s, pass, &j)) scan_job(&j, h); +} + +// A draw as prepare_draw sees it: its pass begins, and with it a contact shadow march may be due. +static void scan_draw(VkExec *s, uint32_t op, const void *a, const op_hdr *h) { + if (s->render_disabled) return; + if (op == OP_DRAW && ((const p_draw *)a)->count <= 0) return; + if (op == OP_DRAW_IDX && (((const p_draw_idx *)a)->itype < 0 || ((const p_draw_idx *)a)->itype > 2 || + ((const p_draw_idx *)a)->count <= 0)) + return; + if (op == OP_DRAW_IDX_IND && ((const p_draw_idx_ind *)a)->itype != NVN_INDEX_TYPE_UNSIGNED_SHORT && + ((const p_draw_idx_ind *)a)->itype != NVN_INDEX_TYPE_UNSIGNED_INT) + return; + if (s->pass_noatt) s->rt_resolved = 0; // sized by this draw's viewport + resolve_rt_as(s, 1); + scan_contact(s, 1, h); +} + +static void scan(VkExec *s, int hi, const op_hdr *h, int depth) { + for (;;) { + check_op(h); + const void *a = nvn_op_payload(h); + if (h->op == OP_END) return; + if (!depth && SC.in_chunk >= SC.per && SC.nch < SC.chunks && + (!SC.open || scan_ends_pass(s, h->op, a) || 2 * SC.in_chunk >= 3 * SC.per)) + chunk_start(s, hi, h); + if (drop_op(h->op)) SC.drop_n++; + s->pos++; + if ((draw_op(h->op) && !s->render_disabled) || h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND) scan_writes(s, h->op); + switch (h->op) { // the ops vk_exec_op_xfer and op_dispatch flush the contact shadows at + case OP_DISPATCH: case OP_DISPATCH_IND: case OP_COPY_B2B: case OP_COPY_B2T: case OP_COPY_T2B: case OP_COPY_T2T: + case OP_CLEAR_TEX: case OP_REPORT_COUNTER: case OP_RESET_COUNTER: case OP_RENDER_COND: case OP_CLEAR_BUF: + scan_contact(s, 0, h); + break; + default: break; + } + switch (h->op) { + case OP_BEGIN: break; + case OP_JUMP: h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; continue; + case OP_CALL: + if (depth < CALL_DEPTH) scan(s, hi, nvn_op_first(((const p_jump *)a)->target), depth + 1); // else walk stops + break; + case OP_SET_RT: + if (memcmp(&s->rt, a, sizeof s->rt)) { + s->rt = *(const p_set_rt *)a, s->rt_resolved = 0, SC.open = 0; + resolve_rt_as(s, 1); // here, not at the first draw: a clear or a later chunk may need them live + chunks_publish(0); + } + break; + case OP_CLEAR_COLOR: case OP_CLEAR_DS: + SC.open |= !s->render_disabled; + if (!s->render_disabled) resolve_rt_as(s, 1), scan_contact(s, 1, h); + break; + case OP_DRAW: case OP_DRAW_IDX: case OP_DRAW_IDX_IND: + SC.in_chunk++, SC.open = 1; + scan_draw(s, h->op, a, h); + break; + case OP_DISPATCH: case OP_DISPATCH_IND: SC.in_chunk++, SC.open = 0; break; + case OP_UPDATE_UB: { // listed where op_update_ub notes it + const p_update_ub *u = a; + VkMemSpan sp; + if (u->size && vk_mem_resolve(u->dst, u->size, &sp)) { + SC.upd[SC.nupd++] = (VkExecUpd){u->dst, u->dst + u->size, u->data}; + scan_drop(u->dst, u->size); + } + SC.open = 0; + break; + } + case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: + case OP_BIND_UBO: case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: + vk_exec_bind(s, h->op, a); + break; + case OP_COPY_B2B: case OP_COPY_T2B: case OP_CLEAR_BUF: { + uint64_t at, size = vk_exec_xfer_written(s, h->op, a, &at); + if (size) scan_drop(at, size); + if (vk_exec_xfer_revive(h->op, a, s->pos)) scan_hold(HOLD_TEX); + SC.open = 0; + break; + } + case OP_BARRIER: case OP_COPY_B2T: case OP_COPY_T2T: case OP_CLEAR_TEX: + if (vk_exec_xfer_revive(h->op, a, s->pos)) scan_hold(HOLD_TEX); + SC.open = 0; + break; + case OP_SIGNAL_EVENT: break; + case OP_RENDER_COND: SC.cond[SC.ncond++] = (uint8_t)vk_exec_xfer_scan(s, h->op, a); break; + case OP_REPORT_COUNTER: case OP_RESET_COUNTER: case OP_RENDER_ENABLE: vk_exec_xfer_scan(s, h->op, a); break; + default: state_op(s, h->op, a); // walk stops on an op without a handler + } + h = nvn_op_next(h); + } +} + +// The queue state of a and b without what resolve_rt derives from rt and the images. +static int state_differs(const VkExec *a, const VkExec *b) { + static VkExec x, y; + copy_state(&x, a), copy_state(&y, b); + VkExec *e[2] = {&x, &y}; + for (int i = 0; i < 2; i++) { + e[i]->rt_resolved = 0; + memset(e[i]->color, 0, sizeof e[i]->color), memset(&e[i]->depth, 0, sizeof e[i]->depth); + e[i]->pass_scale = e[i]->pass_w = e[i]->pass_h = e[i]->pass_layers = 0, e[i]->pass_noatt = 0; + memset(e[i]->pass_color, 0, sizeof e[i]->pass_color); + e[i]->pass_depth = e[i]->pass_stencil = VK_FORMAT_UNDEFINED, e[i]->pass_ncolor = e[i]->pass_samples = 0; + memset(e[i]->pipe.color_fmt, 0, sizeof e[i]->pipe.color_fmt); + e[i]->pipe.depth_fmt = 0, e[i]->pipe.samples = 0; + } + const uint8_t *p = (const uint8_t *)&x, *q = (const uint8_t *)&y; + for (size_t o = VK_EXEC_STATE_BEGIN; o < VK_EXEC_STATE_END; o++) + if (p[o] != q[o]) return (int)o; + return 0; +} + +static void record_chunks(VkExec *ex, const uint64_t *handles, int n, const OpCounts *cnt) { + uint64_t t0 = SC.t0 = kern_now_ns(); + SC.prep_ns += t0 - SC.rec_t0; + recs_start(); + if (!SC.ch && !(SC.ch = calloc((size_t)SC.chunks + 1, sizeof *SC.ch))) rt_fatal("vk: out of memory"); + while (SC.cupd < cnt->upd) SC.upd = grow_n(SC.upd, &SC.cupd, sizeof *SC.upd); + while (SC.ccond < cnt->cond) SC.cond = grow_n(SC.cond, &SC.ccond, sizeof *SC.cond); + while (SC.cwpos < cnt->draws) SC.wpos = grow_n(SC.wpos, &SC.cwpos, sizeof *SC.wpos); + vk_exec_oq_flush(ex); // queries a sequential submission closed count before the chunks' + VkExec *s = &SC.s; + copy_state(s, ex); + s->fence = ex->fence, s->rt_resolved = 0, s->pos = 0; + SC.holds = 0; + SC.nch = SC.sub = SC.pub = 0, SC.nupd = SC.ncond = SC.njobs = SC.drop_n = SC.ndr = SC.dr_done = 0, SC.open = 0; + SC.nwpos = 0; + job_turn = 0; + SC.per = (SC.per + 3) / 4; + plat_mutex_lock(&RP.m); + RP.handles = handles, RP.n = n, RP.fence = ex->fence, RP.prep_next = 0, RP.prep_left = 2; + if (!++RP.gen) RP.gen = 1; + __atomic_store_n(&RP.take, (uint64_t)RP.gen << 32, __ATOMIC_RELEASE); + plat_cond_broadcast(&RP.go); + plat_mutex_unlock(&RP.m); + if (!nthreads) prep_run(); + chunk_start(s, 0, nvn_op_first(handles[0])); + for (int i = 0; i < n; i++) scan(s, i, nvn_op_first(handles[i]), 0); + SC.ch[SC.nch - 1].job_end = SC.njobs, SC.ch[SC.nch - 1].wpos_end = SC.nwpos, SC.ch[SC.nch - 1].draws = SC.in_chunk; + SC.scan_ns += kern_now_ns() - t0; + prep_run(); // threads that wake late leave it to the scan's end + while (__atomic_load_n(&RP.prep_left, __ATOMIC_ACQUIRE)) plat_yield(); + chunks_publish(1); + SC.chunked++, SC.nchunks += (uint64_t)SC.nch; + chunks_submit(1); + x_cur = ex; + for (int i = 0; i < nrecs; i++) frame_add(&recs[i].ex); + int o = SC.check ? state_differs(SC.ch[SC.nch - 1].by, s) : 0; + if (o && SC.bad++ < 16) + rt_log("vk_exec: chunked replay: the state after the last chunk differs from the scan's at byte %d of VkExec", o); + copy_state(ex, s); + ex->rt_resolved = 0; +} + +int vk_exec_replay_workers(void) { return SC.workers; } + +void vk_exec_contexts(void (*fn)(VkExec *ex)) { + fn(&X); + for (int i = 0; i < nrecs; i++) fn(&recs[i].ex); +} + +static int chunked_on(void) { + if (SC.workers < 0) { + SC.workers = cfg_int("TOTK_REPLAY_WORKERS"); + if (SC.workers < 0) SC.workers = 0; + SC.chunks = cfg_int("TOTK_REPLAY_CHUNKS"); + if (SC.chunks <= 0) SC.chunks = 2 * SC.workers; + SC.min_draws = cfg_int("TOTK_REPLAY_CHUNK_DRAWS"); + SC.check = cfg_bool("TOTK_REPLAY_STATE_CHECK"); + if (SC.workers) + rt_log("vk: replay in up to %d chunks per submission of at least %d draws each, %d recording thread%s%s", + SC.chunks, SC.min_draws, SC.workers, SC.workers > 1 ? "s" : "", SC.check ? ", state checked" : ""); + if (SC.workers && vk_dlaa_wanted()) + rt_log("vk: DLSS (dlaa = 1) keeps the replay on one thread, so replay_workers has no effect"); + else if (SC.workers && !cfg_bool("TOTK_FXAA")) + rt_log("vk: fxaa = 0 keeps the replay on one thread, so replay_workers has no effect while it is set"); + } + // DLSS (the dlaa setting) and the fxaa switch act on what the frame drew so far, draw by draw and in replay order, + // which only the sequential replay gives them. + return SC.workers > 0 && SC.chunks > 1 && !vk_dump_armed && !vk_dlaa_wanted() && cfg_bool("TOTK_FXAA"); +} + int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t fence) { VkExec *ex = &X; + SC.rec_t0 = kern_now_ns(); if (!x_ready) state_reset(ex), vk_dump_init(), x_ready = 1; const tex_t *keep[9]; // the render targets carry over to this submission (set_rt compares them by pointer) memcpy(keep, ex->rt.color, sizeof ex->rt.color); @@ -950,33 +1607,56 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t ex->nviews = 0; ex->barrier_at = -1; ex->nfbl = 0; // the previous command buffer ended with a full barrier: every image is in GENERAL - vk_exec_frame.submits++; - split_draws = part_start = 0; - part_t0 = kern_now_ns(); - x_here = 1; - if (n) vk_mem_mirror_begin(), vk_usnap_begin(handles, n, fence); - for (int i = 0; i < n; i++) { + ex->fr.submits++; + ex->split_draws = ex->part_start = 0; + ex->part_t0 = kern_now_ns(); + x_cur = ex; + for (int i = 0; i < n; i++) if (!handles[i]) rt_fatal("vk: command handle %d of a submission is null", i); - walk(ex, nvn_op_first(handles[i]), 0); + OpCounts cnt = {0}; + if (n && chunked_on()) + for (int i = 0; i < n; i++) count_ops(nvn_op_first(handles[i]), 0, &cnt); + SC.subs += n != 0; + if (cnt.draws && cnt.draws >= 2 * (uint32_t)SC.min_draws) { + SC.per = (int)((cnt.draws * 4 + 4u * SC.chunks - 4) / (4u * SC.chunks - 3)); // the first chunk a quarter + if (SC.per < SC.min_draws) SC.per = SC.min_draws; + record_chunks(ex, handles, n, &cnt); + } else { + if (n) vk_mem_mirror_begin(), vk_usnap_begin(handles, n, fence); + for (int i = 0; i < n; i++) walk(ex, nvn_op_first(handles[i]), 0, NULL); } vk_exec_pass_end(ex); - x_here = 0; + x_cur = NULL; if (ex->recorded) vk_exec_barrier(ex); // later submissions see this one's writes if (n) ex->recorded += vk_mem_mirror_end(ex->cb, fence), vk_usnap_end(fence); // VRAM mirror bytes the GPU wrote go back to guest memory - if (ex->nviews) { - ViewList *l = malloc(sizeof *l + sizeof(VkImageView) * (size_t)ex->nviews); - if (!l) rt_fatal("vk: out of memory"); - l->n = ex->nviews; - for (int i = 0; i < ex->nviews; i++) l->v[i] = ((ViewEnt *)ex->views)[i].view; - vk_completion_add(fence, free_views, l); - ex->nviews = 0; - } + views_release(ex); ex->cb = VK_NULL_HANDLE; + frame_add(ex); if (n) n_subs++; if (n && (rt_trace & RT_TRACE_PERF)) vk_exec_ub_perf(); if ((rt_trace & RT_TRACE_PERF) && n_subs >= 240) { rt_log("perf: replay command buffer parts per submission %.2f", (double)(n_subs + n_parts) / (double)n_subs); + if (SC.chunked) { + double k = 1e6 * (double)SC.chunked; + rt_log("perf: chunked replay: %llu of %llu submissions, %.2f chunks each; per submission %.3f ms before the scan, " + "scan %.3f ms, first chunk begun at %.3f ms and submitted at %.3f ms, last at %.3f ms, recording %.3f ms " + "in all, longest chunk %.3f ms; beside the scan: mirrors %.3f ms, uniform snapshot %.3f ms, both done at %.3f ms", + (unsigned long long)SC.chunked, (unsigned long long)SC.subs, (double)SC.nchunks / SC.chunked, + SC.prep_ns / k, SC.scan_ns / k, SC.start0_ns / k, SC.first_ns / k, SC.last_ns / k, SC.rec_ns / k, + SC.rec_max_ns / 1e6, SC.mirror_ns / k, SC.usnap_ns / k, SC.ready_ns / k); + rt_log("perf: chunked replay: chunks held per submission: %.2f for a new texture, %.2f for unknown storage " + "writes", SC.held[HOLD_TEX] / (double)SC.chunked, SC.held[HOLD_WRITES] / (double)SC.chunked); + memset(SC.held, 0, sizeof SC.held); + char b[768]; + int o = 0; + for (int i = 0; i < SC.nch && o < 700; i++) + o += snprintf(b + o, sizeof b - o, " %d:%.2f@%.2f%s", SC.ch[i].draws, SC.ch[i].ns / 1e6, SC.ch[i].start / 1e6, + SC.holds >> i & 1 ? "h" : ""); + rt_log("perf: chunked replay: the last submission's chunks (draws:ms@begun, h held)%s", b); + } n_subs = n_parts = 0; + SC.subs = SC.chunked = SC.nchunks = SC.scan_ns = SC.first_ns = SC.last_ns = SC.rec_ns = SC.rec_max_ns = 0; + SC.prep_ns = SC.start0_ns = SC.mirror_ns = SC.usnap_ns = SC.ready_ns = 0; } - return ex->recorded - part_start; // what the last part holds + return ex->recorded - ex->part_start; // what the last part holds } diff --git a/runtime/vk_exec.h b/runtime/vk_exec.h index e3692d48..1f224fcc 100644 --- a/runtime/vk_exec.h +++ b/runtime/vk_exec.h @@ -2,10 +2,12 @@ // handles of a submission, keeps render targets, passes, clears and fixed-function state; draws, binds and dispatches // (vk_exec_draw.c) and copies, counters, events and conditional rendering (vk_exec_xfer.c) plug in through the two // hooks at the bottom. -// Runs on the submitting thread under vk_sync_submit's lock: no locking inside. +// Runs on the submitting thread under vk_sync_submit's lock; a chunked replay (TOTK_REPLAY_WORKERS, vk_exec.c) records +// on several threads, each with a VkExec of its own, and shares only the locked tables named where they are declared. // The op loop follows MacNativeGames nvn_mtl.m (MIT, Copyright (c) 2026 jhaizhou) and the pass handling LAS Recomp // (MIT, Copyright (c) 2026 Peiikai); vk_exec.c names the sources. #pragma once +#include #include #include "nvn_api.h" @@ -38,8 +40,31 @@ typedef struct { uint8_t scale; // VkTexImage.scale } VkExecTarget; -// The replay state. One instance lives for the run: NVN state is queue state and carries over from one submission to -// the next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record. +// TOTK_TRACE=gfx: replay counters of the current frame, logged and reset at each present (vk_exec_frame_log). +typedef struct { + uint64_t passes, draws_seen, draws, clears, copies, dispatches, submits; + uint64_t fb_draws, fb_barriers, fb_restarts, fb_layouts; // feedback loops: draws sampling a written target of + // their pass, in-pass barriers, pass restarts, layout + // transitions +} VkExecFrame; + +// A contact shadow march (vk_contact.c) the scan of a chunked replay planned before an op: vk_contact_plan decides it in +// replay order, vk_contact_exec records it on the chunk that holds the op. +typedef struct VkExecJob { + const void *op; // vk_exec.c: the op it precedes in a scanned submission + VkExecTarget scene, depth; // the scene colour and the linear depth + float pc[16]; // the march's push constants +} VkExecJob; + +// An OP_UPDATE_UB of the submission: [lo, hi) of guest memory takes data (vk_exec_draw.c ubo_ptr reads them). +typedef struct { uint64_t lo, hi; const uint8_t *data; } VkExecUpd; + +struct VkExecDraw; // vk_exec_draw.c: the memos and command buffer state of one recording context + +// The replay state and one recording context. NVN state is queue state and carries over from one submission to the +// next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record. +// The fields from rt up to views are that queue state (VK_EXEC_STATE_BEGIN/END); the rest belongs to the context that +// records with it. typedef struct VkExec { // ---- per command buffer VkCommandBuffer cb; // being recorded @@ -94,8 +119,14 @@ typedef struct VkExec { int render_disabled; // OP_RENDER_ENABLE false or a failed OP_RENDER_COND: clears and draws skip p_render_cond render_cond; // last OP_RENDER_COND int has_render_cond; + // samples-passed counting (vk_exec_xfer.c): a reset since the last report, a report since the last reset, the game + // reported twice without a reset (every draw counts from then on) + int oq_counting, oq_reported, oq_keep; // ---- internal to vk_exec.c + VkExecFrame fr; // counters of this context, added to vk_exec_frame at the end of a submission + int split_draws, part_start; // draws and dispatches since the last part; recorded at its start + uint64_t part_t0; // replay time at its start void *views; // image views created for this command buffer (freed at its fence) int nviews, cap_views; int barrier_at; // recorded right after the last full barrier (-1: none yet) @@ -112,8 +143,30 @@ typedef struct VkExec { const void *fb_fs; // feedback_scan's memo: fragment shader, texture bind serial, pass serial uint32_t fb_serial, fb_pass, fb_sampled; uint32_t tex_serial, pass_serial; // bumped by every texture or image bind and pass begin + + // ---- vk_exec_xfer.c: the occlusion query of the open pass and the queries closed since the last report or reset + int oq_open; // slot of the active query, -1 none + uint32_t *oq_seg, oq_nseg, oq_capseg; + int oq_lost; // the segment missed a draw (ring full) + + struct VkExecDraw *draw; // vk_exec_draw.c, made on the context's first draw op + + // ---- a chunk of a scanned submission (vk_exec.c): the scan's answers, taken in op order + int chunked; // records a chunk: what depends on earlier chunks comes from the scan + const VkExecUpd *upd; // the submission's inline uniform updates + const uint8_t *cond; // its RENDER_COND results (1 renders) + const VkExecJob *jobs; // its contact shadow marches + uint32_t njobs; + uint32_t upd_n, cond_i, job_i; // updates, results and records before the op being recorded + uint32_t drop_n; // ops before it that may write uniform ranges (vk_usnap_drop_at) + const uint32_t *wpos; // the positions of the draws and dispatches among those ops (vk_exec_draw_writes) + uint32_t wpos_i, wpos_end; // the next of them, and the end of the chunk's + uint32_t pos; // ops of the submission up to and with it (the scan's too) } VkExec; +#define VK_EXEC_STATE_BEGIN offsetof(VkExec, rt) +#define VK_EXEC_STATE_END offsetof(VkExec, fr) + // Begins dynamic rendering with the current targets unless a pass is open (a full memory barrier precedes it). Call // before a draw or an attachment clear. void vk_exec_pass_begin(VkExec *ex); @@ -136,9 +189,16 @@ VkCommandBuffer vk_exec_inline_begin(int *reopen); void vk_exec_inline_end(int reopen); // vk_exec_draw.c: forget what the command buffer being recorded has bound (pipeline, sets, dynamic state, vertex // buffers): the replay continues in a new one (vk_sync_split). -void vk_exec_draw_new_cb(void); +void vk_exec_draw_new_cb(VkExec *ex); +// The recording threads of the chunked replay as this run read TOTK_REPLAY_WORKERS (0 off), -1 before the first +// submission read it (the settings menu shows it). +int vk_exec_replay_workers(void); +// vk_exec_draw.c, for the state scan of a chunked replay: the storage ranges the shaders of a draw (dispatch 0) or a +// dispatch at state s write, as {address, size} into out; their count, or -1 when there are more than max or a stage's +// shader is not known yet (no context has translated it). Reads uniform blocks through s->upd like a chunk. +int vk_exec_draw_writes(const VkExec *s, int dispatch, uint64_t (*out)[2], int max); // vk_exec_draw.c: the vertex and fragment shader of the last draw recorded into this command buffer (NULL: none yet). -void vk_exec_draw_shaders(const GfxShader **vs, const GfxShader **fs); +void vk_exec_draw_shaders(const VkExec *ex, const GfxShader **vs, const GfxShader **fs); VkBuffer vk_exec_inline_buffer(VkDeviceSize size); // vk_exec.c: the stand-in a 3D BC render target is drawn through goes with its image (vk_exec_image_gone). void vk_exec_bc_gone(VkImage image); @@ -155,16 +215,10 @@ int vk_exec_targets_recent(VkExec *ex, int any); // The current pass's targets with their formats, extents and the frames they were last rendered in (diagnostics). void vk_exec_targets_describe(VkExec *ex, char *buf, size_t cap); -// TOTK_TRACE=gfx: replay counters of the current frame, logged and reset at each present (vk_exec_frame_log). -typedef struct { - uint64_t passes, draws_seen, draws, clears, copies, dispatches, submits; - uint64_t fb_draws, fb_barriers, fb_restarts, fb_layouts; // feedback loops: draws sampling a written target of - // their pass, in-pass barriers, pass restarts, layout - // transitions -} VkExecFrame; extern VkExecFrame vk_exec_frame; void vk_exec_frame_written(const tex_t *t); // vk_texture_gpu_written: t rendered or copied into uint64_t vk_exec_recording_fence(void); // fence of the submission this thread records, else 0 +uint32_t vk_exec_replay_pos(void); // the op this thread records in a chunk, else UINT32_MAX void vk_exec_frame_log(const tex_t *presented, uint64_t frame); // TOTK_DUMP= (vk_dump.c): between presents frame-1 and frame, every pass's targets and every dispatch's storage @@ -188,6 +242,7 @@ void vk_c0_fill(uint8_t bank[VK_C0_BYTES], int stage, const p_bind_buf ssbo[16], // vk_exec_draw.c: OP_DRAW, OP_DRAW_IDX, OP_DISPATCH, OP_BIND_* (program, state, vattr, vstream, vbuf, ubo, ssbo, tex, // samp, img), OP_UPDATE_UB, OP_DRAW_IDX_IND, OP_DISPATCH_IND. payload: nvn_op_payload of the record. void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload); +int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload); // a bind op's writes into ex; 1: VkExecDraw.bind_gen // The aspect a sampled view of img reads (vk_exec_draw.c get_view): stencil for a view in STENCIL depth-stencil mode. VkImageAspectFlags vk_exec_view_aspect(const VkTexImage *img, const view_t *v, int hv); VkImageView vk_exec_target_view(const VkTexImage *img, VkFormat format, VkImageViewType type, uint32_t aspect, @@ -199,10 +254,26 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload); // sample DLSS's output vk_dlaa_swap_view in its place, so the game's scaling draw scales DLSS's picture. extern const VkTexImage *vk_dlaa_swap_img; extern VkImageView vk_dlaa_swap_view; -// vk_contact.c: a pass about to begin (its targets resolved, no pass open), and a dispatch or copy about to be recorded +// The state scan's part of the transfer ops: render enable, conditional rendering (returns the condition: 1 renders), +// samples-passed counting state and the pending reports conditions read. Returns 1 for every other op. +// The state scan of a chunked replay (vk_exec.c): the textures op uses go live in replay order (vk_texture_revive); pos: +// the op's place in the submission. 1 when one of them has no entry yet. +int vk_exec_xfer_revive(uint32_t op, const void *payload, uint32_t pos); +int vk_exec_xfer_scan(VkExec *ex, uint32_t op, const void *payload); +// The guest memory a COPY_B2B, COPY_T2B or CLEAR_BUF op makes the GPU write, as far as the scan can tell without the +// texture's image: [*addr, *addr + return value). +uint64_t vk_exec_xfer_written(const VkExec *ex, uint32_t op, const void *payload, uint64_t *addr); +// vk_contact.c: a pass about to begin (its targets resolved, no pass open), and a dispatch or copy about to be recorded. +// vk_contact_plan is what they do in replay order (pass: the first; else the second), for the scan; 1: a march is due (j). void vk_contact_pass_begin(VkExec *ex); void vk_contact_flush(VkExec *ex); +int vk_contact_plan(const VkExec *ex, int pass, VkExecJob *j); +void vk_contact_exec(VkExec *ex, const VkExecJob *j); // vk_exec_xfer.c, samples-passed counting: a draw about to be recorded in the open pass (begins an occlusion query // when counting and none is active), and the open pass about to end (ends the active query). void vk_exec_oq_draw(VkExec *ex); void vk_exec_oq_pass_end(VkExec *ex); +void vk_exec_oq_forget(VkExec *ex); +void vk_exec_oq_flush(VkExec *ex); // the end of a chunk: its closed queries counted as a segment without a report // vk_exec_xfer_shutdown: the context's queries belonged to the old pool +// vk_exec.c: fn on every recording context (teardown, the device idle). +void vk_exec_contexts(void (*fn)(VkExec *ex)); diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index f86fd1ae..cbf92811 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -8,7 +8,8 @@ // Per command buffer the replay keeps transient memory (support buffers, copied uniform ranges, generated indices), // descriptor pools, image views and the storage-buffer ranges the GPU may write; all of it is freed, and the ranges are // reported with vk_gpu_written, from one completion callback once the submission's fence is done (the write watch -// does not see GPU writes). Nothing here locks: it runs on the submitting thread under vk_sync_submit's lock. +// does not see GPU writes). The memos and per-command-buffer state belong to one recording context (VkExecDraw); the +// tables the contexts of a chunked replay share (stand-ins, depth twins, views) take their own locks. // // Provenance: the op semantics (bind ops, program and slot conventions, draw and dispatch flow, vertex streams with a // zeroed buffer for unbound ones) follow MacNativeGames games/totk/third_party/d2r_gfx/gfx/nvn/nvn_mtl.m:7409-7460, @@ -81,9 +82,6 @@ typedef struct { int nwritten, cwritten; } CbRes; -static CbRes *cur; -static uint64_t cur_fence; - // Descriptor pools and full-size transient chunks go back to these lists when their command buffer completes, instead // of being destroyed and created again for the next one: each creation is a driver memory allocation (on Windows an // allocation, a map and a paging-fence wait on the CPU), about ten of them per frame inside the replay. @@ -95,7 +93,7 @@ static Chunk spare_chunk[SPARE_CHUNKS]; static int nspare_pool, nspare_chunk; // State of the command buffer being recorded that the Vulkan command buffer itself holds. -static struct { +typedef struct { VkPipeline pipe[2]; // bound per bind point: 0 graphics, 1 compute VkPipeState st; // the state vk_pipeline_set_dynamic last recorded int st_valid; @@ -126,20 +124,19 @@ static struct { VkDescriptorSetLayout ds_last_layout[2]; int ds_pushed[2]; // set number holds the push of DS.psig under ds_layout int res_valid; // the graphics sets bound hold what the inputs in RS give (bind_resources) -} B; +} CbState; -// Contents of B.ds_last: the set's packed descriptors (VkDescSlot), which with its layout determine every write. A draw +// Contents of CbState.ds_last: the set's packed descriptors (VkDescSlot), which with its layout determine every write. A draw // whose allocated set would hold the same reuses the set instead of allocating and writing one (a per-draw // cost). psig: the same for the last push per set number; a push equal to it under the same layout is not recorded // again (the descriptors stay bound across pipeline binds of that layout). #define DS_SIG_BYTES (MAX_DESC * sizeof(VkDescSlot)) -static struct { uint32_t n[2], pn[2]; uint8_t sig[2][DS_SIG_BYTES], psig[2][DS_SIG_BYTES]; } DS; +typedef struct { uint32_t n[2], pn[2]; uint8_t sig[2][DS_SIG_BYTES], psig[2][DS_SIG_BYTES]; } DescSigs; // TOTK_BIND_STATS counters (bind_stats). -static struct { - int on; +typedef struct { uint64_t draws, bind[2], desc[2], push, push_skip, set_new, set_reuse, ac_miss, same_prog, same_all, bind_skip; -} BS = {-1}; +} BindStats; // TOTK_DYN_DELTA (prepare_draw): the mode and, for the perf line, the set_dynamic calls, the commands they recorded // (a full call counts 1) and the calls that recorded none. @@ -150,8 +147,7 @@ static struct { int mode; uint64_t calls, cmds, empty; } DD = {-1}; // c0 storage table, otherwise one bound range or handle per descriptor. #define RS_BYTES (MAX_BINDINGS * sizeof(((VkExec *)0)->ssbo[0]) + MAX_DESC * sizeof(p_bind_buf)) typedef struct { int cls, use; uint64_t bytes; } RsCount; -static struct { - int on; // TOTK_BIND_SKIP: 1 skip (default), 2 check, 0 off +typedef struct { uint32_t hn; // words of h the last signature filled; 0: none uint64_t h[24]; // generations, feedback images, support inputs, base instance const VkPipeInfo *info; // the pipeline whose RsList ranges body holds, as of bind op generation gen @@ -164,7 +160,7 @@ static struct { int cmds; // check mode: pushes, binds and set writes the build recorded uint32_t ckn; // check mode: the last build's packed descriptors (bytes in ck) uint8_t ck[DS_SIG_BYTES]; -} RS = {-1}; +} RsState; // UpdateUniformBuffer overlays of the submission being replayed (op_update_ub, TOTK_UB_OVERLAY). Here because // bind_sig reads them (ov_at). @@ -205,12 +201,111 @@ static uint64_t ov_at(const VkExec *ex, uint64_t addr, uint64_t n) { typedef struct { uint32_t off, len; } RsRange; typedef struct { int st; const GfxShaderRes *res; } RsPtr; typedef struct { uint32_t n, bytes; uint8_t sup, c0, ok, np; RsPtr p[RS_PTRS]; RsRange r[]; } RsList; -static nvn_hmap rs_lists; // VkPipeInfo pointer -> RsList, until shutdown -static const VkPipeInfo *rs_last; // the last lookup -static const RsList *rs_last_l; +// (program, stage, answers) -> shader, 4-way set-associative, most recently used first. A miss costs a SHA-256 of the +// whole shader code (gfx_shader.c req_init), slow on CPUs without SHA instructions: a field frame looks up ~3000 and +// the 2048-slot direct-mapped table this replaced missed 3% of them on conflicts. +#define SHADER_SETS 2048 +#define SHADER_WAYS 4 +typedef struct { uint64_t serial; int stage; int used; GfxShaderAnswers a; const GfxShader *s; } ShaderEnt; -static uint32_t c0_dim[3], c0_binst; // the dispatch size and base instance of the draw or dispatch being bound -static uint64_t bind_gen = 1; // bumped by every bind op that writes ex->prog, ubo, ssbo, tex or img +// Which shader of a program serves each NVN stage, and its code length, per program serial (new at every +// nvnProgramSetShaders): the scan reads two words of every stage's control block, cold guest cache lines, and runs +// several times a draw. Direct-mapped; a collision only scans again. +#define STAGEM_N 512 +typedef struct { uint64_t serial; int8_t idx[6]; uint32_t len[6]; } StageMemo; + +// Texture and image types of a stage's first 32 slots, each kept with the handle and the generation and slot token it +// was looked up under: a slot is looked up again, a hash probe, only when the handle changed or its slot may resolve +// otherwise (slot_still). +typedef struct { uint64_t tex[32], img[32], ttok[32], itok[32]; uint32_t tgen[32], igen[32]; uint8_t tt[32], it[32]; } AcSlots; + +// The slots of (program, stage) whose types its translation reads (gfx_shader_slot_mask), learned from its first +// shader; ~0u while unknown. The types of other slots select the same shader, so answers_for leaves them zero instead +// of looking up all 64 handles. Direct-mapped by serial and stage; a collision only relearns. +#define SLOTM_N 4096 +typedef struct { uint64_t serial; int st; uint32_t m; } SlotMask; + +// The inline uniform updates of the submission being replayed (upd_note). +#define UPD_N 4096 +typedef struct { + uint64_t fence; // the submission the entries belong to + uint32_t n; // updates noted in it + VkExecUpd u[UPD_N]; +} UpdRing; + +// The image and the levels [l0, l1) and layers [a0, a1) a texture's view covers. +typedef struct { VkImage image; uint32_t l0, l1, a0, a1; } SubRange; + +// The subresource a texture handle samples (feedback_scan). +#define FM_N 8192 +typedef struct { uint64_t h, tok; uint32_t gen; SubRange r; } FbMemo; + +// Sampled-texture descriptors per (handle, resource type, separate) (tex_info_lookup). +#define TD_N 8192 +typedef struct { + uint64_t h, tok; + uint32_t gen, epoch, sgen; + uint16_t type; + uint8_t separate, valid; + VkImage image; + VkDescriptorImageInfo out; +} TexMemo; + +// The previous draw's shaders and what they were looked up from (prepare_draw). +typedef struct { + const prog_t *prog; + uint64_t serial; + uint32_t gen, tex_serial; + uint64_t tex[2][32], img[2][32]; + VkPipeState pipe; +} MemoKey; +typedef struct { int valid, has_fs; MemoKey k; const GfxShader *vs, *fs; } ShaderMemo; + +// The previous draw's pipeline and the state it was found for (prepare_draw). +typedef struct { int valid; VkPipeState s; const VkPipeInfo *info; } PipeMemo; +// The pipelines found before, by state: vk_pipeline_graphics_async takes the pipeline cache's lock, which the contexts +// of a chunked replay would take on every draw whose state differs from the draw before. Direct-mapped. +#define PC_N 1024 +typedef struct { const VkPipeInfo *info; VkPipeState s; } PipeCache; + +// The memos and command buffer state of one recording context (VkExec.draw). Nothing in it is shared: the context's +// thread reads and writes it without a lock. +typedef struct VkExecDraw { + CbRes *cur; // resources of the submission being recorded + uint64_t cur_fence; + CbState B; + DescSigs DS; + BindStats BS; + RsState RS; + nvn_hmap rs_lists; // VkPipeInfo pointer -> RsList, until shutdown + const VkPipeInfo *rs_last; // the last lookup + const RsList *rs_last_l; + uint32_t c0_dim[3], c0_binst; // the dispatch size and base instance of the draw or dispatch being bound + uint64_t bind_gen; // bumped by every bind op that writes ex->prog, ubo, ssbo, tex or img + int twin_dispatch; // op_dispatch*: the resources being bound are a dispatch's + uint64_t wait_frame, wait_left_ns; // prepare_draw: full-screen pipeline waits left in this frame + ShaderEnt shader_tab[SHADER_SETS][SHADER_WAYS]; + StageMemo stagem[STAGEM_N]; + AcSlots AC[6]; + SlotMask slotm[SLOTM_N]; + FbMemo FM[FM_N]; + TexMemo TD[TD_N]; + ShaderMemo SM; + PipeMemo PM; + PipeCache PC[PC_N]; + UpdRing UPD; + VkWriteDescriptorSet w[MAX_BINDINGS]; // bind_resources' scratch + VkDescSlot d[MAX_DESC]; + VkBufferView bv[MAX_DESC]; + RsRange rs_tmp[MAX_DESC]; // rs_list's +} VkExecDraw; + +static VkExecDraw *draw_new(VkExec *ex) { + VkExecDraw *D = calloc(1, sizeof *D); + if (!D) rt_fatal("vk: out of memory (replay context)"); + D->bind_gen = 1; + return ex->draw = D; +} static void cb_done(void *arg) { // completion thread CbRes *r = arg; @@ -234,11 +329,17 @@ static void cb_done(void *arg) { // completion thread free(r->written), free(r->pools), free(r->chunks), free(r); } -void vk_exec_draw_new_cb(void) { memset(&B, 0, sizeof B); } +// Also after the bound state was replaced (a chunk's start): bind_sig compares the bound slots again. +void vk_exec_draw_new_cb(VkExec *ex) { + if (!ex->draw) return; + memset(&ex->draw->B, 0, sizeof ex->draw->B); + ex->draw->bind_gen++; +} -void vk_exec_draw_shaders(const GfxShader **vs, const GfxShader **fs) { - *vs = B.st_valid ? B.st.vs : NULL; - *fs = B.st_valid ? B.st.fs : NULL; +void vk_exec_draw_shaders(const VkExec *ex, const GfxShader **vs, const GfxShader **fs) { + int valid = ex->draw && ex->draw->B.st_valid; + *vs = valid ? ex->draw->B.st.vs : NULL; + *fs = valid ? ex->draw->B.st.fs : NULL; } // Marks a dispatch about to be recorded (vk.h vk_pass_mark_begin); the caller ends it after the dispatch. @@ -252,13 +353,14 @@ static VkPassMark *dispatch_mark(VkExec *ex, const GfxShader *cs, uint32_t x, ui } static CbRes *cb_res(VkExec *ex) { - if (cur && cur_fence == ex->fence) return cur; - cur = calloc(1, sizeof *cur); - if (!cur) rt_fatal("vk: out of memory"); - cur_fence = ex->fence; - memset(&B, 0, sizeof B); - vk_completion_add(ex->fence, cb_done, cur); - return cur; + VkExecDraw *D = ex->draw; + if (D->cur && D->cur_fence == ex->fence) return D->cur; + D->cur = calloc(1, sizeof *D->cur); + if (!D->cur) rt_fatal("vk: out of memory"); + D->cur_fence = ex->fence; + memset(&D->B, 0, sizeof D->B); + vk_completion_add(ex->fence, cb_done, D->cur); + return D->cur; } static uint8_t *scratch(VkExec *ex, uint64_t size, VkBuffer *buf, VkDeviceSize *off) { @@ -390,22 +492,39 @@ static const VkFormat tex_format[40] = { VK_FORMAT_A2B10G10R10_UNORM_PACK32, VK_FORMAT_A2B10G10R10_UINT_PACK32, VK_FORMAT_B10G11R11_UFLOAT_PACK32}; static struct { int used, di; VkFormat fmt; VkImage image; VmaAllocation alloc; VkImageView view, sview; } SD[48]; +static PlatMutex dummy_lock = PLAT_MUTEX_INIT; // the contexts make stand-ins on first use + static VkImageView storage_dummy(int di, int code) { VkFormat fmt = code > 0 && code < 40 ? tex_format[code] : VK_FORMAT_R8G8B8A8_UNORM; if (fmt == VK_FORMAT_R8G8B8A8_UNORM || dinfo[di].s != VK_SAMPLE_COUNT_1_BIT) return Z.img[di].sview; + VkImageView v = Z.img[di].sview; + plat_mutex_lock(&dummy_lock); for (int i = 0; i < 48; i++) { if (!SD[i].used) { SD[i].used = 1, SD[i].di = di, SD[i].fmt = fmt; make_dummy(di, fmt, &SD[i].image, &SD[i].alloc, &SD[i].view, &SD[i].sview); - return SD[i].sview; + v = SD[i].sview; + break; + } + if (SD[i].di == di && SD[i].fmt == fmt) { + v = SD[i].sview; + break; } - if (SD[i].di == di && SD[i].fmt == fmt) return SD[i].sview; } - return Z.img[di].sview; + plat_mutex_unlock(&dummy_lock); + return v; } +static void dummies_make(void); + static void dummies_init(void) { - if (Z.ready) return; + if (__atomic_load_n(&Z.ready, __ATOMIC_ACQUIRE)) return; + plat_mutex_lock(&dummy_lock); + if (!Z.ready) dummies_make(); + plat_mutex_unlock(&dummy_lock); +} + +static void dummies_make(void) { VkCtx *c = vk_ctx(); VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = ZERO_BYTES, .usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | @@ -429,11 +548,18 @@ static void dummies_init(void) { for (int i = 0; i < DV_COUNT; i++) make_dummy(i, VK_FORMAT_R8G8B8A8_UNORM, &Z.img[i].image, &Z.img[i].alloc, &Z.img[i].view, &Z.img[i].sview); - Z.ready = 1; + __atomic_store_n(&Z.ready, 1, __ATOMIC_RELEASE); } static void twins_release(VkCtx *c); +static void rs_forget(VkExec *ex) { // the pipeline infos the lists are keyed by go with the device + if (!ex->draw) return; + nvn_hm_clear(&ex->draw->rs_lists, free); + ex->draw->rs_last = NULL, ex->draw->rs_last_l = NULL; + memset(ex->draw->PC, 0, sizeof ex->draw->PC), ex->draw->PM.valid = 0; +} + // Releases the stand-ins and the depth twins. The device must be idle; the game path leaves them to process exit like // the pipelines. void vk_exec_draw_shutdown(void) { @@ -451,8 +577,7 @@ void vk_exec_draw_shutdown(void) { vmaDestroyImage(c->vma, SD[i].image, SD[i].alloc); } memset(SD, 0, sizeof SD); - nvn_hm_clear(&rs_lists, free); - rs_last = NULL, rs_last_l = NULL; + vk_exec_contexts(rs_forget); for (int i = 0; i < nspare_pool; i++) vkDestroyDescriptorPool(c->device, spare_pool[i], NULL); for (int i = 0; i < nspare_chunk; i++) vmaDestroyBuffer(c->vma, spare_chunk[i].buf, spare_chunk[i].alloc); nspare_pool = nspare_chunk = 0; @@ -651,13 +776,6 @@ void vk_exec_image_gone(VkImage image) { // ---- program to shaders -// (program, stage, answers) -> shader, 4-way set-associative, most recently used first. A miss costs a SHA-256 of the -// whole shader code (gfx_shader.c req_init), slow on CPUs without SHA instructions: a field frame looks up ~3000 and -// the 2048-slot direct-mapped table this replaced missed 3% of them on conflicts. -#define SHADER_SETS 2048 -#define SHADER_WAYS 4 -typedef struct { uint64_t serial; int stage; int used; GfxShaderAnswers a; const GfxShader *s; } ShaderEnt; -static ShaderEnt shader_tab[SHADER_SETS][SHADER_WAYS]; static int find_stage_scan(const prog_t *p, int nvn_stage, const uint8_t **ctl, const uint8_t **code, uint32_t *len) { for (uint32_t i = 0; i < p->nstage && i < 6; i++) { @@ -673,15 +791,10 @@ static int find_stage_scan(const prog_t *p, int nvn_stage, const uint8_t **ctl, return 0; } -// Which shader of a program serves each NVN stage, and its code length, per program serial (new at every -// nvnProgramSetShaders): the scan reads two words of every stage's control block, cold guest cache lines, and runs -// several times a draw. Direct-mapped; a collision only scans again. -#define STAGEM_N 512 -static struct { uint64_t serial; int8_t idx[6]; uint32_t len[6]; } stagem[STAGEM_N]; -static int find_stage(const prog_t *p, int nvn_stage, const uint8_t **ctl, const uint8_t **code, uint32_t *len) { +static int find_stage(VkExecDraw *D, const prog_t *p, int nvn_stage, const uint8_t **ctl, const uint8_t **code, uint32_t *len) { if (!p->serial || (unsigned)nvn_stage >= 6) return find_stage_scan(p, nvn_stage, ctl, code, len); - typeof(stagem[0]) *m = &stagem[p->serial * 0x9E3779B97F4A7C15ull >> 55]; + typeof(D->stagem[0]) *m = &D->stagem[p->serial * 0x9E3779B97F4A7C15ull >> 55]; if (m->serial != p->serial) { m->serial = p->serial; memset(m->idx, -1, sizeof m->idx); @@ -699,14 +812,13 @@ static int find_stage(const prog_t *p, int nvn_stage, const uint8_t **ctl, const return 1; } -// Texture and image types of a stage's first 32 slots, each kept with the handle and the generation and slot token it -// was looked up under: a slot is looked up again, a hash probe, only when the handle changed or its slot may resolve -// otherwise (slot_still). -static struct { uint64_t tex[32], img[32], ttok[32], itok[32]; uint32_t tgen[32], igen[32]; uint8_t tt[32], it[32]; } AC[6]; // vk_texture_slot_token of the slot a texture or image handle names; an empty handle always resolves to nothing. static uint64_t token_of(uint64_t h) { return h ? vk_texture_slot_token((int)(h & 0xfffff)) : 1; } +// A chunk asks for this handle at every op: what it resolves to depends on the op (vk_texture_slot_moves). +static int slot_moves(const VkExec *ex, uint64_t h) { return ex->chunked && h && vk_texture_slot_moves((int)(h & 0xfffff)); } + // Whether what handle h resolved to under the global generation *egen and its slot token etok (both read before that // lookup) still holds at global generation gen: nothing at all changed since, or not its slot (the token, read only when // the global generation moved, about every 16 draws). A token hit moves *egen to gen. @@ -719,22 +831,65 @@ static int slot_still(uint32_t *egen, uint64_t etok, uint64_t h, uint32_t gen) { } static int shader_memo_on(void); -// The slots of (program, stage) whose types its translation reads (gfx_shader_slot_mask), learned from its first -// shader; ~0u while unknown. The types of other slots select the same shader, so answers_for leaves them zero instead -// of looking up all 64 handles. Direct-mapped by serial and stage; a collision only relearns. -#define SLOTM_N 4096 -static struct { uint64_t serial; int st; uint32_t m; } slotm[SLOTM_N]; static uint32_t slotm_idx(uint64_t serial, int st) { return (uint32_t)((serial * 8 + (uint64_t)st) * 0x9E3779B97F4A7C15ull >> 52); } -static uint32_t slot_mask_of(uint64_t serial, int st) { - const typeof(slotm[0]) *e = &slotm[slotm_idx(serial, st)]; +static uint32_t slot_mask_of(VkExecDraw *D, uint64_t serial, int st) { + const typeof(D->slotm[0]) *e = &D->slotm[slotm_idx(serial, st)]; return serial && e->serial == serial && e->st == st ? e->m : ~0u; } -static void slot_mask_learn(uint64_t serial, int st, const GfxShader *s) { - if (!serial || slot_mask_of(serial, st) != ~0u) return; - slotm[slotm_idx(serial, st)] = (typeof(slotm[0])){serial, st, gfx_shader_slot_mask(s)}; +static void slot_mask_learn(VkExecDraw *D, uint64_t serial, int st, const GfxShader *s) { + if (!serial || slot_mask_of(D, serial, st) != ~0u) return; + D->slotm[slotm_idx(serial, st)] = (typeof(D->slotm[0])){serial, st, gfx_shader_slot_mask(s)}; +} + +// The shader of each (program, stage) a context translated, for the state scan of a chunked replay +// (vk_exec_draw_writes): which storage buffers a stage writes is a property of its code, the same in every translation. +// s NULL: the program has no such stage, or it failed to translate. Shared by the contexts and read by the scan without +// a lock: a writer takes wt_lock and clears the key while it changes the entry, a reader checks the key before and +// after reading the shader. A program not found holds the chunks after its draw (vk_exec.c scan_writes), so the table +// is large and set-associative: a field frame in 1.2.1 draws with programs whose serials reach some 25,000 within a +// minute, and the 4096 direct-mapped entries this replaced missed in nearly every chunk, which then waited for the one +// before it. A full set gives up its oldest program (the lowest serial). +#define WT_WAYS 8 +#define WT_SETS (1 << 14) +static struct { uint64_t key; const GfxShader *s; } WT[WT_SETS][WT_WAYS]; +static PlatMutex wt_lock = PLAT_MUTEX_INIT; + +static uint64_t wt_key(uint64_t serial, int st) { return serial << 4 | (uint64_t)st << 1 | 1; } + +static typeof(WT[0][0]) *wt_set(uint64_t k) { return WT[(k * 0x9E3779B97F4A7C15ull) >> (64 - 14)]; } + +static int wt_way(const typeof(WT[0][0]) *set, uint64_t k) { + for (int w = 0; w < WT_WAYS; w++) + if (__atomic_load_n(&set[w].key, __ATOMIC_ACQUIRE) == k) return w; + return -1; +} + +static void wt_learn(uint64_t serial, int st, const GfxShader *s) { + uint64_t k = wt_key(serial, st); + typeof(WT[0][0]) *set = wt_set(k); + if (!serial || wt_way(set, k) >= 0) return; + plat_mutex_lock(&wt_lock); + if (wt_way(set, k) < 0) { + int w = 0; + for (int i = 1; i < WT_WAYS; i++) + if (set[i].key < set[w].key) w = i; // an empty way (key 0) first, else the oldest + __atomic_store_n(&set[w].key, 0, __ATOMIC_RELEASE); + __atomic_store_n(&set[w].s, s, __ATOMIC_RELEASE); + __atomic_store_n(&set[w].key, k, __ATOMIC_RELEASE); + } + plat_mutex_unlock(&wt_lock); +} + +static int wt_find(uint64_t serial, int st, const GfxShader **s) { + uint64_t k = wt_key(serial, st); + const typeof(WT[0][0]) *set = wt_set(k); + int w = serial ? wt_way(set, k) : -1; + if (w < 0) return 0; + *s = __atomic_load_n(&set[w].s, __ATOMIC_ACQUIRE); + return __atomic_load_n(&set[w].key, __ATOMIC_ACQUIRE) == k; } static uint8_t type_of_handle(uint64_t h) { // texture lock held @@ -746,6 +901,7 @@ static uint8_t type_of_handle(uint64_t h) { // texture lock held // The answers of stage st for the slots in mask; the types of other slots stay zero. static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, GfxShaderAnswers *a) { + VkExecDraw *D = ex->draw; memset(a, 0, sizeof *a); if (st == NVN_STAGE_VS) vk_pipeline_answers(&ex->pipe, a, NULL); else if (st == NVN_STAGE_FS) vk_pipeline_answers(&ex->pipe, NULL, a); @@ -755,7 +911,7 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G memcpy(&l, ctl + 0x748 + 4 * k, 4); a->local[k] = (uint16_t)(l ? l : 1); } - typeof(AC[0]) *c = &AC[st]; + typeof(D->AC[0]) *c = &D->AC[st]; // The generation and tokens are read before the lookups (one that creates an entry changes them) and unlocked: a // registration racing with them is ordered after this draw, and a slot stored under older ones is only looked up // again. @@ -764,10 +920,10 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G for (uint32_t m = mask; m; m &= m - 1) { int slot = __builtin_ctz(m); uint64_t h = ex->tex[st][slot], hi = ex->img[st][slot]; - int tmiss = c->tex[slot] != h || !c->ttok[slot] || !slot_still(&c->tgen[slot], c->ttok[slot], h, gen); - int imiss = c->img[slot] != hi || !c->itok[slot] || !slot_still(&c->igen[slot], c->itok[slot], hi, gen); + int tmiss = c->tex[slot] != h || !c->ttok[slot] || slot_moves(ex, h) || !slot_still(&c->tgen[slot], c->ttok[slot], h, gen); + int imiss = c->img[slot] != hi || !c->itok[slot] || slot_moves(ex, hi) || !slot_still(&c->igen[slot], c->itok[slot], hi, gen); uint64_t tt = tmiss ? token_of(h) : 0, it = imiss ? token_of(hi) : 0; - if ((tmiss || imiss || check) && !locked) vk_texture_lock(), locked = 1, BS.ac_miss++; + if ((tmiss || imiss || check) && !locked) vk_texture_lock(), locked = 1, D->BS.ac_miss++; if (tmiss) c->tt[slot] = type_of_handle(h), c->tex[slot] = h, c->ttok[slot] = tt, c->tgen[slot] = gen; if (imiss) c->it[slot] = type_of_handle(hi), c->img[slot] = hi, c->itok[slot] = it, c->igen[slot] = gen; if (check && (type_of_handle(h) != c->tt[slot] || type_of_handle(hi) != c->it[slot])) // TOTK_SHADER_MEMO=check @@ -783,20 +939,25 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G static int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion) static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pending, int await) { + VkExecDraw *D = ex->draw; const uint8_t *ctl, *code; uint32_t len; *present = 0; if (pending) *pending = 0; - if (!ex->prog || !find_stage(ex->prog, st, &ctl, &code, &len)) return NULL; + if (!ex->prog) return NULL; + if (!find_stage(ex->draw, ex->prog, st, &ctl, &code, &len)) { + wt_learn(ex->prog->serial, st, NULL); + return NULL; + } *present = 1; GfxShaderAnswers a; uint64_t serial = ex->prog->serial; - answers_for(ex, st, ctl, slot_mask_of(serial, st), &a); + answers_for(ex, st, ctl, slot_mask_of(D, serial, st), &a); a.mvprev = (uint8_t)(sf_mvprev && st == NVN_STAGE_VS); ShaderEnt *set = NULL; if (serial) { uint64_t h = (serial * 0x9E3779B97F4A7C15ull ^ (uint64_t)st) ^ nvn_hm_hash(&a, sizeof a); // a is zero-padded - set = shader_tab[(h ^ h >> 29) & (SHADER_SETS - 1)]; + set = D->shader_tab[(h ^ h >> 29) & (SHADER_SETS - 1)]; for (int w = 0; w < SHADER_WAYS && set[w].used; w++) { ShaderEnt *e = &set[w]; if (e->serial != serial || e->stage != st || memcmp(&e->a, &a, sizeof a)) continue; @@ -805,6 +966,7 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin memmove(&set[1], &set[0], w * sizeof *set); set[0] = hit; } + wt_learn(serial, st, set[0].s); // again after WT gave the entry up return set[0].s; } } @@ -813,7 +975,8 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin : pending && (vk_pipeline_async_on() || a.mvprev) ? gfx_shader_get_async(gs, ctl, code, len, &a, pending) : gfx_shader_get(gs, ctl, code, len, NULL, &a); if (pending && *pending) return NULL; - if (s) slot_mask_learn(serial, st, s); + if (s) slot_mask_learn(D, serial, st, s); + wt_learn(serial, st, s); if (!s) warn_once(W_SHADER, "a stage %d shader of program %llu failed to translate: draws with it are skipped", st, (unsigned long long)serial); else if (set) { // in front, the least recently used way drops out @@ -834,16 +997,16 @@ static VkDescriptorBufferInfo zero_info(uint64_t max) { } // vk_mem_count, also kept per (class, use) while a graphics build runs (RS). -static void count_use(int cls, int use, uint64_t n) { +static void count_use(VkExecDraw *D, int cls, int use, uint64_t n) { vk_mem_count(cls, use, n); - if (!RS.rec) return; - for (int i = 0; i < RS.ncnt; i++) - if (RS.cnt[i].cls == cls && RS.cnt[i].use == use) { - RS.cnt[i].bytes += n; + if (!D->RS.rec) return; + for (int i = 0; i < D->RS.ncnt; i++) + if (D->RS.cnt[i].cls == cls && D->RS.cnt[i].use == use) { + D->RS.cnt[i].bytes += n; return; } - if (RS.ncnt < (int)(sizeof RS.cnt / sizeof RS.cnt[0])) RS.cnt[RS.ncnt++] = (RsCount){cls, use, n}; - else RS.cnt_over = 1; + if (D->RS.ncnt < (int)(sizeof D->RS.cnt / sizeof D->RS.cnt[0])) D->RS.cnt[D->RS.ncnt++] = (RsCount){cls, use, n}; + else D->RS.cnt_over = 1; } // A buffer descriptor for [addr, addr+size) of guest memory: its backing (a pool's VRAM mirror when it has one) when @@ -851,28 +1014,31 @@ static void count_use(int cls, int use, uint64_t n) { // written: the shader stores into it (a mirror's range goes back to guest memory at the end of the submission). static int bind_range(VkExec *ex, uint64_t addr, uint64_t size, uint64_t align, uint64_t max, int use, int written, VkDescriptorBufferInfo *out) { + VkExecDraw *D = ex->draw; VkMemSpan sp; if (!addr || !vk_mem_resolve(addr, 1, &sp)) return 0; uint64_t n = size && size < sp.avail ? size : sp.avail; if (n > max) n = max; if (!n) return 0; if (!written && (use == VK_MUSE_UBO || use == VK_MUSE_SSBO) && ub_bind(ex, addr, n, align, out)) { - count_use(VK_MCLS_SCRATCH, use, n); + count_use(ex->draw, VK_MCLS_SCRATCH, use, n); return 1; } if (sp.offset % align == 0) { VK_MEM_USE(sp); if (written) ub_written(ex, addr, n); *out = (VkDescriptorBufferInfo){sp.buffer, sp.offset, n}; - if (written) vk_mem_gpu_wrote_shader(&sp, n), vk_usnap_drop(addr, n), vk_exec_ub_foreign(ex, addr, n); - count_use(sp.cls, use, n); + if (written) + vk_mem_gpu_wrote_shader(&sp, n), vk_usnap_drop_at(addr, n, ex->chunked ? ex->drop_n : 0), + vk_exec_ub_foreign(ex, addr, n); + count_use(ex->draw, sp.cls, use, n); VK_MEM_BOUND(sp); return 1; } if (sp.alloc) return 0; VkBuffer b; VkDeviceSize o; - RS.volatile_ = 1; + D->RS.volatile_ = 1; memcpy(scratch(ex, n, &b, &o), (const void *)(uintptr_t)addr, n); vk_mem_count(sp.cls, VK_MUSE_CPU_READ, n); vk_mem_count(VK_MCLS_SCRATCH, use, n); @@ -921,25 +1087,26 @@ static int c0_on(void) { // TOTK_FIX_C0=0: c0 reads zeros instead of the drive // c0 from replay scratch; a bank equal to the stage's previous one reuses its range, so it costs no copy. static VkDescriptorBufferInfo c0_info(VkExec *ex, int st) { + VkExecDraw *D = ex->draw; int k = st == NVN_STAGE_VS ? 0 : st == NVN_STAGE_FS ? 1 : 2; // vk_c0_fill reads the storage slots, and the uniform slots and dispatch size for compute or the base instance for // graphics: equal inputs give the same bank, so neither the fill nor the 2 KiB compare is needed. - typeof(B.c0_in[0]) in; + typeof(D->B.c0_in[0]) in; memset(&in, 0, sizeof in); memcpy(in.ssbo, ex->ssbo[st], sizeof in.ssbo); - if (st == NVN_STAGE_CS) memcpy(in.ubo, ex->ubo[st], sizeof in.ubo), memcpy(in.dim, c0_dim, sizeof in.dim); - else in.binst = c0_binst; - if (B.c0_valid[k] && !memcmp(&in, &B.c0_in[k], sizeof in)) - return (VkDescriptorBufferInfo){B.c0_buf[k], B.c0_off[k], VK_C0_BYTES}; - B.c0_in[k] = in; + if (st == NVN_STAGE_CS) memcpy(in.ubo, ex->ubo[st], sizeof in.ubo), memcpy(in.dim, D->c0_dim, sizeof in.dim); + else in.binst = D->c0_binst; + if (D->B.c0_valid[k] && !memcmp(&in, &D->B.c0_in[k], sizeof in)) + return (VkDescriptorBufferInfo){D->B.c0_buf[k], D->B.c0_off[k], VK_C0_BYTES}; + D->B.c0_in[k] = in; uint8_t tmp[VK_C0_BYTES]; - vk_c0_fill(tmp, st, ex->ssbo[st], ex->ubo[st], c0_dim, c0_binst); - if (!B.c0_valid[k] || memcmp(tmp, B.c0[k], VK_C0_BYTES)) { - memcpy(scratch(ex, VK_C0_BYTES, &B.c0_buf[k], &B.c0_off[k]), tmp, VK_C0_BYTES); - memcpy(B.c0[k], tmp, VK_C0_BYTES); - B.c0_valid[k] = 1; + vk_c0_fill(tmp, st, ex->ssbo[st], ex->ubo[st], D->c0_dim, D->c0_binst); + if (!D->B.c0_valid[k] || memcmp(tmp, D->B.c0[k], VK_C0_BYTES)) { + memcpy(scratch(ex, VK_C0_BYTES, &D->B.c0_buf[k], &D->B.c0_off[k]), tmp, VK_C0_BYTES); + memcpy(D->B.c0[k], tmp, VK_C0_BYTES); + D->B.c0_valid[k] = 1; } - return (VkDescriptorBufferInfo){B.c0_buf[k], B.c0_off[k], VK_C0_BYTES}; + return (VkDescriptorBufferInfo){D->B.c0_buf[k], D->B.c0_off[k], VK_C0_BYTES}; } // The buffer range limits, read once: the device's properties are a cold cache line on every binding of every draw. @@ -967,7 +1134,7 @@ static VkDescriptorBufferInfo ubo_info(VkExec *ex, int st, int slot) { if (slot == 1 && ex->prog) { // compiler constants stored after the shader code const uint8_t *ctl, *code; uint32_t len, c1; - if (find_stage(ex->prog, st, &ctl, &code, &len)) { + if (find_stage(ex->draw, ex->prog, st, &ctl, &code, &len)) { memcpy(&c1, ctl + 0x700, 4); if (c1 < len && bind_range(ex, (uint64_t)(uintptr_t)code + c1, len - c1, align, max, VK_MUSE_UBO, 0, &bi)) return bi; @@ -975,11 +1142,12 @@ static VkDescriptorBufferInfo ubo_info(VkExec *ex, int st, int slot) { } else if (slot >= 3 && slot - 3 < 18) { const p_bind_buf *b = &ex->ubo[st][slot - 3]; if (b->size && ub_bind(ex, b->addr, b->size < max ? b->size : max, align, &bi)) { // updated in this pass - count_use(VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); + count_use(ex->draw, VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); return bi; } - if (b->size && vk_usnap_find(b->addr, b->size < max ? b->size : max, align, &bi)) { // as of the submit call - count_use(VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); + if (b->size && vk_usnap_find(b->addr, b->size < max ? b->size : max, ex->chunked ? ex->drop_n : UINT32_MAX, align, + &bi)) { // as of the submit call + count_use(ex->draw, VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); return bi; } if (bind_range(ex, b->addr, b->size, align, max, VK_MUSE_UBO, 0, &bi)) return bi; @@ -1016,23 +1184,27 @@ static int ubo_ptr_res(const GfxShaderRes *r) { // into the buffer's GPU copy; guest memory gets them only from the write-back when the submission ends. The terrain // rewrites its pointer block this way before every draw, all at one address, so ubo_ptr reads the latest update that // covers the pointer, else memory. Payloads stay in the recorded command buffer for the whole replay. -#define UPD_N 4096 -static struct { - uint64_t fence; // the submission the entries belong to - uint32_t n; // updates noted in it - struct { uint64_t lo, hi; const uint8_t *data; } u[UPD_N]; -} UPD; - -static void upd_note(const VkExec *ex, const p_update_ub *u) { - if (UPD.fence != ex->fence) UPD.fence = ex->fence, UPD.n = 0; - UPD.u[UPD.n++ % UPD_N] = (typeof(UPD.u[0])){u->dst, u->dst + u->size, u->data}; +static void upd_note(VkExec *ex, const p_update_ub *u) { + VkExecDraw *D = ex->draw; + if (ex->chunked) { // the scan listed it + ex->upd_n++; + return; + } + if (D->UPD.fence != ex->fence) D->UPD.fence = ex->fence, D->UPD.n = 0; + D->UPD.u[D->UPD.n++ % UPD_N] = (VkExecUpd){u->dst, u->dst + u->size, u->data}; } // 1 when an update of this submission supplied [a, a+n). static int upd_read(const VkExec *ex, uint64_t a, void *out, size_t n) { - if (UPD.fence != ex->fence) return 0; - for (uint32_t i = UPD.n, end = UPD.n > UPD_N ? UPD.n - UPD_N : 0; i-- > end;) { - const typeof(UPD.u[0]) *e = &UPD.u[i % UPD_N]; + VkExecDraw *D = ex->draw; + const VkExecUpd *u = ex->upd; // a chunk: the scan's list of the submission, of which upd_n precede this op + uint32_t cnt = ex->upd_n, mask = UINT32_MAX; + if (!ex->chunked) { + if (D->UPD.fence != ex->fence) return 0; + u = D->UPD.u, cnt = D->UPD.n, mask = UPD_N - 1; + } + for (uint32_t i = cnt, end = cnt > UPD_N ? cnt - UPD_N : 0; i-- > end;) { + const VkExecUpd *e = &u[i & mask]; if (a >= e->hi || a + n <= e->lo) continue; if (a < e->lo || a + n > e->hi) return 0; // partly covered: memory memcpy(out, e->data + (a - e->lo), n); @@ -1043,9 +1215,10 @@ static int upd_read(const VkExec *ex, uint64_t a, void *out, size_t n) { // Applies the updates of this submission that touch [a, a+n) onto out, oldest first (ub_bind's merged copies). static void upd_apply(const VkExec *ex, uint64_t a, uint8_t *out, size_t n) { - if (UPD.fence != ex->fence) return; - for (uint32_t i = UPD.n > UPD_N ? UPD.n - UPD_N : 0; i < UPD.n; i++) { - const typeof(UPD.u[0]) *e = &UPD.u[i % UPD_N]; + const UpdRing *U = &ex->draw->UPD; + if (U->fence != ex->fence) return; + for (uint32_t i = U->n > UPD_N ? U->n - UPD_N : 0; i < U->n; i++) { + const VkExecUpd *e = &U->u[i % UPD_N]; uint64_t lo = e->lo > a ? e->lo : a, hi = e->hi < a + n ? e->hi : a + n; if (lo < hi) memcpy(out + (lo - a), e->data + (lo - e->lo), hi - lo); } @@ -1062,18 +1235,29 @@ static uint64_t ubo_ptr(const VkExec *ex, int st, const GfxShaderRes *r) { return a; } +// The binding a storage buffer resource names at this point of the replay, NULL when it names none. *ubo: a compute +// shader's uniform binding; up holds the binding of an address read from a uniform block (size 0: up to the end of +// the backing). +static const p_bind_buf *ssbo_bound(const VkExec *ex, int st, const GfxShaderRes *r, p_bind_buf *up, int *ubo) { + int slot = ssbo_slot(r, st, ubo); + if (slot >= 0 && slot < 16) return *ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot]; + if (!ubo_ptr_res(r)) return NULL; + up->addr = ubo_ptr(ex, st, r); + return up; +} + +// vk_caps.c: shaders translated for sbuf_align address the buffer from its address rounded down to it. +static uint64_t sbuf_lead(uint64_t addr) { return vk_caps.sbuf_align ? addr & (vk_caps.sbuf_align - 1) : 0; } + static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes *r, int written) { if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init(); uint64_t max = LIM.sbo_max, align = LIM.sbo_align; VkDescriptorBufferInfo bi; - int ubo, slot = ssbo_slot(r, st, &ubo); + int ubo; p_bind_buf up = {0}; - const p_bind_buf *b = NULL; - if (slot >= 0 && slot < 16) b = ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot]; - else if (ubo_ptr_res(r)) up.addr = ubo_ptr(ex, st, r), b = &up; // size 0: up to the end of the backing + const p_bind_buf *b = ssbo_bound(ex, st, r, &up, &ubo); if (b) { - // vk_caps.c: shaders translated for sbuf_align address the buffer from its address rounded down to it. - uint64_t lead = vk_caps.sbuf_align ? b->addr & (vk_caps.sbuf_align - 1) : 0; + uint64_t lead = sbuf_lead(b->addr); if (bind_range(ex, b->addr - lead, b->size ? b->size + lead : 0, align, max, ubo ? VK_MUSE_UBO : VK_MUSE_SSBO, written, &bi)) { if (written && bi.buffer != VK_NULL_HANDLE) note_written(ex, b->addr - lead, bi.range); @@ -1088,6 +1272,36 @@ static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes * return zero_info(max); } +int vk_exec_draw_writes(const VkExec *s, int dispatch, uint64_t (*out)[2], int max) { + if (!s->prog) return 0; + static const int gfx[] = {NVN_STAGE_VS, NVN_STAGE_FS}, cs[] = {NVN_STAGE_CS}; + const int *st = dispatch ? cs : gfx, nst = dispatch ? 1 : 2; + const GfxShader *sh[2]; + for (int i = 0; i < nst; i++) + if (!wt_find(s->prog->serial, st[i], &sh[i])) return -1; + if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init(); + int n = 0; + for (int i = 0; i < nst; i++) { + const GfxShaderRes *r; + int nr = sh[i] ? gfx_shader_resources(sh[i], GFX_RES_SSBO, &r) : 0; + for (int j = 0; j < nr; j++) { + int ubo; + p_bind_buf up = {0}; + const p_bind_buf *b = (r[j].flags & 1) ? ssbo_bound(s, st[i], &r[j], &up, &ubo) : NULL; + VkMemSpan sp; + if (!b || !b->addr) continue; + uint64_t lead = sbuf_lead(b->addr), addr = b->addr - lead, size = b->size ? b->size + lead : 0; + if (!vk_mem_resolve(addr, 1, &sp)) continue; + uint64_t len = size && size < sp.avail ? size : sp.avail; // as bind_range binds it + if (len > LIM.sbo_max) len = LIM.sbo_max; + if (!len) continue; + if (n == max) return -1; + out[n][0] = addr, out[n][1] = len, n++; + } + } + return n; +} + // ---- feedback loops (maxwell.md 1.9, vk.h VkCtx.feedback) // TotK's G-buffer and terrain shaders sample targets of the pass that renders them, at their own pixel, after earlier // draws of the pass wrote them (albedo, normals, R32F linear depth). Maxwell reads see those writes; in Vulkan the read @@ -1105,8 +1319,6 @@ static int in_fbl(const VkExec *ex, VkImage img) { return 0; } -// The image and the levels [l0, l1) and layers [a0, a1) a texture's view covers. -typedef struct { VkImage image; uint32_t l0, l1, a0, a1; } SubRange; static SubRange sub_of(const VkTexImage *img, const view_t *v, int hv) { SubRange r = {img->image, 0, img->levels, 0, img->layers}; @@ -1142,9 +1354,6 @@ static int td_mode(void) { // The subresource a texture handle samples, per handle, kept while its slot resolves the same (slot_still): a draw's // texture bindings change the whole-scan memo below, but the handles they bind were mostly looked up before, in this // pass or an earlier one. Direct-mapped. -#define FM_N 8192 -static struct { uint64_t h, tok; uint32_t gen; SubRange r; } FM[FM_N]; - static SubRange sampled_sub(uint64_t h) { // texture lock held view_t v; int hv; @@ -1155,6 +1364,7 @@ static SubRange sampled_sub(uint64_t h) { // texture lock held // Attachments of the open pass the fragment shader fs samples; kept while the shader, the texture bindings and the // pass stay the same. static uint32_t feedback_scan(VkExec *ex, const GfxShader *fs) { + VkExecDraw *D = ex->draw; if (ex->fb_fs == fs && ex->fb_serial == ex->tex_serial && ex->fb_pass == ex->pass_serial) return ex->fb_sampled; const GfxShaderRes *res; int n = gfx_shader_resources(fs, GFX_RES_TEX, &res); @@ -1165,8 +1375,8 @@ static uint32_t feedback_scan(VkExec *ex, const GfxShader *fs) { int slot = res[r].slot + k; uint64_t h = slot >= 0 && slot < 128 ? ex->tex[NVN_STAGE_FS][slot] : 0; if (!h) continue; - typeof(FM[0]) *f = &FM[(h * 0x9E3779B97F4A7C15ull) >> 51]; - if (memo && f->h == h && slot_still(&f->gen, f->tok, h, gen)) { + typeof(D->FM[0]) *f = &D->FM[(h * 0x9E3779B97F4A7C15ull) >> 51]; + if (memo && f->h == h && !slot_moves(ex, h) && slot_still(&f->gen, f->tok, h, gen)) { if (memo == 2) { if (!locked) vk_texture_lock(), locked = 1; SubRange c = sampled_sub(h); @@ -1195,7 +1405,7 @@ static uint32_t feedback_prepare(VkExec *ex, const GfxShader *fs) { if (mode == VK_FEEDBACK_OFF || !fs || !ex->in_pass) return 0; uint32_t m = feedback_scan(ex, fs); uint32_t d = m & ex->fb_dirty; - if (d) vk_exec_frame.fb_draws++; + if (d) ex->fr.fb_draws++; if (d & ~(mode == VK_FEEDBACK_LAYOUT ? ex->fbl_mask : 0)) { if (mode == VK_FEEDBACK_LAYOUT) for (uint32_t i = 0; i < 9; i++) { @@ -1204,7 +1414,7 @@ static uint32_t feedback_prepare(VkExec *ex, const GfxShader *fs) { } vk_exec_pass_end(ex); vk_exec_pass_begin(ex); // learned targets now in the feedback layout; the full barrier cleared fb_dirty - vk_exec_frame.fb_restarts++; + ex->fr.fb_restarts++; } return mode == VK_FEEDBACK_LAYOUT ? m & ex->fbl_mask : 0; } @@ -1222,7 +1432,7 @@ static void feedback_barrier(VkExec *ex) { vkCmdPipelineBarrier2(ex->cb, &di); ex->recorded++; ex->fb_dirty = 0; - vk_exec_frame.fb_barriers++; + ex->fr.fb_barriers++; } // Marks the attachments the draw with state s writes: colour targets with a nonzero effective write mask, depth with @@ -1243,29 +1453,29 @@ static void tex_info_full(VkExec *ex, const GfxShaderRes *r, uint64_t h, int sto // for its storage check; a re-upload writes the same image, so the descriptor stays. Draws bind the same textures // thousands of times a frame (15,000 lookups a frame in the monsters fight); a hit skips the slot lookup, the view key // with its hash probe and the sampler lookup. Storage images, which note their binding per submission, are -// not kept. Direct-mapped; texture lock held. TOTK_TEX_MEMO=0 turns it off; =check computes every hit again and stops +// not kept. Direct-mapped; the texture lock only for the lookups (tex_lock). TOTK_TEX_MEMO=0 turns it off; =check computes every hit again and stops // on a difference. -#define TD_N 8192 -static struct { - uint64_t h, tok; - uint32_t gen, epoch, sgen; - uint16_t type; - uint8_t separate, valid; - VkImage image; - VkDescriptorImageInfo out; -} TD[TD_N]; +// The texture lock, taken on the first lookup that needs it (*locked): a memo hit needs none, and the contexts of a +// chunked replay take it from several threads. +static void tex_lock(int *locked) { + if (!*locked) vk_texture_lock(), *locked = 1; +} static void tex_info_lookup(VkExec *ex, const GfxShaderRes *r, uint64_t h, int storage, int separate, - VkDescriptorImageInfo *out) { + VkDescriptorImageInfo *out, int *locked) { + VkExecDraw *D = ex->draw; int td_on = td_mode(); VkImage image; - if (!h || storage || !td_on) return tex_info_full(ex, r, h, storage, separate, out, &image); + if (!h || storage || !td_on || slot_moves(ex, h)) return tex_lock(locked), tex_info_full(ex, r, h, storage, separate, out, &image); uint32_t gen = vk_texture_slots_gen(), ep = vk_texture_epoch(), sg = vk_sampler_gen(); // before any lookup - typeof(TD[0]) *m = &TD[(h * 0x9E3779B97F4A7C15ull ^ (uint64_t)r->type << 7 ^ (uint64_t)separate) >> 51]; + typeof(D->TD[0]) *m = &D->TD[(h * 0x9E3779B97F4A7C15ull ^ (uint64_t)r->type << 7 ^ (uint64_t)separate) >> 51]; if (m->valid && m->h == h && m->type == r->type && m->separate == separate && m->sgen == sg && slot_still(&m->gen, m->tok, h, gen)) { if (m->epoch != ep) { // the slot's first binding in this submission: only its storage check is due - vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL); + if (!vk_texture_slot_checked((int)(h & 0xfffff))) { // another context may have made it + tex_lock(locked); + vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL); + } m->epoch = ep; } *out = m->out; @@ -1273,6 +1483,7 @@ static void tex_info_lookup(VkExec *ex, const GfxShaderRes *r, uint64_t h, int s : VK_IMAGE_LAYOUT_GENERAL; if (td_on == 2) { VkDescriptorImageInfo full; + tex_lock(locked); tex_info_full(ex, r, h, storage, separate, &full, &image); if (full.sampler != out->sampler || full.imageView != out->imageView || full.imageLayout != out->imageLayout) rt_fatal("vk_exec: texture descriptor memo of handle %#llx type %#x differs from a lookup", @@ -1281,6 +1492,7 @@ static void tex_info_lookup(VkExec *ex, const GfxShaderRes *r, uint64_t h, int s return; } uint64_t tok = token_of(h); // before the lookup + tex_lock(locked); tex_info_full(ex, r, h, storage, separate, out, &image); *m = (typeof(*m)){.h = h, .tok = tok, .gen = gen, .epoch = ep, .sgen = sg, .type = r->type, .separate = (uint8_t)separate, .valid = 1, .image = image, .out = *out}; @@ -1327,9 +1539,9 @@ static void tex_info_full(VkExec *ex, const GfxShaderRes *r, uint64_t h, int sto // (https://github.com/Revivedx/SavageWindRecomp, MIT); the code is ours. TOTK_DEPTH_TWIN=0: off. #define TWINS 8 typedef struct { VkImage src, img; VmaAllocation mem; VkImageView view[2]; uint32_t w, h, layers; VkFormat fmt; } Twin; +static PlatMutex twin_lock = PLAT_MUTEX_INIT; // the table and the copy buffer: contexts recording chunks share them static Twin twins[TWINS]; static unsigned twin_next; -static int twin_dispatch; // op_dispatch*: the resources being bound are a dispatch's static VkBuffer twin_buf; static VmaAllocation twin_buf_mem; static VkDeviceSize twin_buf_cap; @@ -1478,17 +1690,24 @@ static Twin *twin_update(VkExec *ex, const VkTexImage *img) { // The image descriptor of slot h; in a dispatch, a compare-sampled colour image the driver cannot compare is swapped // for its depth twin; in a pass larger than it, the scene's light target, once DLSS scaled it this frame, for DLSS's // output (vk_dlaa.c). -static void tex_info(VkExec *ex, const GfxShaderRes *r, uint64_t h, int storage, int separate, VkDescriptorImageInfo *out) { - tex_info_lookup(ex, r, h, storage, separate, out); - if (h && !storage && vk_dlaa_swap_img && ex->in_pass && ex->pass_w > vk_dlaa_swap_img->width + 1 && - vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL) == vk_dlaa_swap_img) - out->imageView = vk_dlaa_swap_view, out->imageLayout = VK_IMAGE_LAYOUT_GENERAL; - if (!twin_dispatch || storage || separate || !h || !(r->type & 0x400)) return; +static void tex_info(VkExec *ex, const GfxShaderRes *r, uint64_t h, int storage, int separate, VkDescriptorImageInfo *out, + int *locked) { + VkExecDraw *D = ex->draw; + tex_info_lookup(ex, r, h, storage, separate, out, locked); + if (h && !storage && vk_dlaa_swap_img && ex->in_pass && ex->pass_w > vk_dlaa_swap_img->width + 1) { + tex_lock(locked); + if (vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL) == vk_dlaa_swap_img) + out->imageView = vk_dlaa_swap_view, out->imageLayout = VK_IMAGE_LAYOUT_GENERAL; + } + if (!D->twin_dispatch || storage || separate || !h || !(r->type & 0x400)) return; VkImageViewType vt = res_view_type(r->type); if (vt != VK_IMAGE_VIEW_TYPE_2D && vt != VK_IMAGE_VIEW_TYPE_2D_ARRAY) return; + tex_lock(locked); const VkTexImage *img = vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL); + plat_mutex_lock(&twin_lock); Twin *t = img ? twin_update(ex, img) : NULL; VkImageView v = t ? t->view[vt == VK_IMAGE_VIEW_TYPE_2D_ARRAY] : VK_NULL_HANDLE; + plat_mutex_unlock(&twin_lock); if (v) out->imageView = v, out->imageLayout = VK_IMAGE_LAYOUT_GENERAL; } @@ -1589,33 +1808,35 @@ static void support_fill(uint8_t *b, float w, float h, float point_size) { // and textures (shaders and pipeline would be the same) and those that also have its buffers and viewport (all // descriptors would be the same). BS is declared beside DS. static int bind_stats_on(void) { - if (BS.on < 0) { + static int on = -1; + if (on < 0) { const char *e = cfg_get("TOTK_BIND_STATS"); - BS.on = e && *e && strcmp(e, "0"); + on = e && *e && strcmp(e, "0"); } - return BS.on; + return on; } -static void bind_stats(const VkPipeInfo *info) { +static void bind_stats(VkExecDraw *D, const VkPipeInfo *info) { if (!bind_stats_on() || info->bind_point != VK_PIPELINE_BIND_POINT_GRAPHICS) return; for (uint32_t i = 0; i < info->n_bindings; i++) { const VkPipeBinding *b = &info->bindings[i]; int fs = info->n_sets > 1 ? b->set : !(b->stages & VK_SHADER_STAGE_VERTEX_BIT); - BS.bind[fs]++, BS.desc[fs] += b->count ? b->count : 1; + D->BS.bind[fs]++, D->BS.desc[fs] += b->count ? b->count : 1; } - if (++BS.draws % 20000 == 0) { - double d = (double)BS.draws; + if (++D->BS.draws % 20000 == 0) { + double d = (double)D->BS.draws; rt_log("vk_exec: per draw over %llu draws: vertex %.2f bindings %.2f descriptors, fragment %.2f bindings %.2f descriptors", - (unsigned long long)BS.draws, BS.bind[0] / d, BS.desc[0] / d, BS.bind[1] / d, BS.desc[1] / d); + (unsigned long long)D->BS.draws, D->BS.bind[0] / d, D->BS.desc[0] / d, D->BS.bind[1] / d, D->BS.desc[1] / d); rt_log("vk_exec: per draw: pushes %.3f skipped %.3f, sets written %.3f reused %.3f, texture types %.3f; " "same program and state as the draw before %.3f, also same buffers %.3f; descriptor builds skipped %.3f", - BS.push / d, BS.push_skip / d, BS.set_new / d, BS.set_reuse / d, BS.ac_miss / d, BS.same_prog / d, - BS.same_all / d, BS.bind_skip / d); + D->BS.push / d, D->BS.push_skip / d, D->BS.set_new / d, D->BS.set_reuse / d, D->BS.ac_miss / d, D->BS.same_prog / d, + D->BS.same_all / d, D->BS.bind_skip / d); } } // The bound state a draw reads, compared with the previous draw's (TOTK_BIND_STATS only). static void repeat_stats(const VkExec *ex, const VkPipeInfo *info) { + VkExecDraw *D = ex->draw; static struct { const prog_t *prog; VkPipeState pipe; @@ -1635,8 +1856,8 @@ static void repeat_stats(const VkExec *ex, const VkPipeInfo *info) { memcpy(cur.vp, ex->vp[0], sizeof cur.vp), cur.point_size = ex->point_size; size_t prog_part = offsetof(typeof(cur), info); if (!memcmp(&cur, &last, prog_part)) { - BS.same_prog++; - if (!memcmp((const char *)&cur + prog_part, (const char *)&last + prog_part, sizeof cur - prog_part)) BS.same_all++; + D->BS.same_prog++; + if (!memcmp((const char *)&cur + prog_part, (const char *)&last + prog_part, sizeof cur - prog_part)) D->BS.same_all++; } last = cur; } @@ -1655,6 +1876,7 @@ static int push_skip_on(void) { // TOTK_PUSH_SKIP=0: every graphics draw pushe // bind_resources and its helpers read beyond the pipeline info's fixed bindings and Z's stand-ins: // - the info (bindings, layouts), the program and its serial (the compiler-constant range of UBO slot 1); // - vk_texture_slots_gen, vk_texture_epoch (a slot's image, view and once-per-submit storage check), vk_sampler_gen; +// - in a chunk, the pass: a slot can resolve otherwise from one op to the next (vk_texture_slot_moves), at a SET_RT or a copy; // - vk_mem_gen (buffer and offset of a span, mirror creation and per-submission sync); // - the images in the feedback layout (the sampled layout); // - per binding: the support inputs, c0's storage table and base instance, the bound uniform and storage ranges, the @@ -1668,13 +1890,14 @@ static int push_skip_on(void) { // TOTK_PUSH_SKIP=0: every graphics draw pushe // the last build; vk_exec_draw_new_cb's memset clears it, and every build clears it until it ends. // TOTK_BIND_SKIP=0 builds every draw; =check builds every draw and stops when a draw with equal inputs gets other // writes, census counts or descriptor commands. -static int bind_skip_on(void) { - if (RS.on < 0) { +static int bind_skip_on(void) { // TOTK_BIND_SKIP: 1 skip (default), 2 check, 0 off + static int on = -1; + if (on < 0) { const char *e = cfg_value("TOTK_BIND_SKIP"); - RS.on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; - if (RS.on != 1) rt_log("vk_exec: descriptor build skip %s", RS.on ? "check" : "off"); + on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; + if (on != 1) rt_log("vk_exec: descriptor build skip %s", on ? "check" : "off"); } - return RS.on; + return on; } static void rs_add(RsRange *r, uint32_t *n, const VkExec *ex, const void *at, size_t len) { @@ -1684,11 +1907,12 @@ static void rs_add(RsRange *r, uint32_t *n, const VkExec *ex, const void *at, si } static const RsList *rs_list(const VkExec *ex, const VkPipeInfo *info) { - if (info == rs_last) return rs_last_l; - if (!rs_lists.klen) nvn_hm_init(&rs_lists, sizeof info); - RsList *l = nvn_hm_get(&rs_lists, &info); + VkExecDraw *D = ex->draw; + if (info == D->rs_last) return D->rs_last_l; + if (!D->rs_lists.klen) nvn_hm_init(&D->rs_lists, sizeof info); + RsList *l = nvn_hm_get(&D->rs_lists, &info); if (!l) { - static RsRange tmp[MAX_DESC]; + RsRange *tmp = ex->draw->rs_tmp; RsPtr ptr[RS_PTRS]; uint32_t n = 0, nd = 0, np = 0; int sup = 0, c0 = 0, ok = 1; @@ -1733,9 +1957,9 @@ static const RsList *rs_list(const VkExec *ex, const VkPipeInfo *info) { l->np = (uint8_t)np; memcpy(l->p, ptr, sizeof *ptr * np); memcpy(l->r, tmp, sizeof *tmp * n); - nvn_hm_put(&rs_lists, &info, l); + nvn_hm_put(&D->rs_lists, &info, l); } - rs_last = info, rs_last_l = l; + D->rs_last = info, D->rs_last_l = l; return l; } @@ -1743,9 +1967,10 @@ static const RsList *rs_list(const VkExec *ex, const VkPipeInfo *info) { // are compared only when a bind op ran since they were stored or the pipeline changed: otherwise they are the same. // Check mode compares them anyway and stops when the bind op generation said they could not differ but they do. static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { + VkExecDraw *D = ex->draw; const RsList *l = rs_list(ex, info); - if (!l->ok) return RS.hn = 0, RS.info = NULL, 0; - uint64_t *h = RS.h, diff = 0; + if (!l->ok) return D->RS.hn = 0, D->RS.info = NULL, 0; + uint64_t *h = D->RS.h, diff = 0; uint32_t n = 0; #define PUT(v) do { uint64_t v_ = (uint64_t)(v); diff |= h[n] ^ v_; h[n++] = v_; } while (0) PUT((uintptr_t)info); @@ -1753,7 +1978,8 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { PUT(ex->prog->serial); PUT((uint64_t)vk_texture_slots_gen() << 32 | vk_texture_epoch()); PUT((uint64_t)vk_sampler_gen() << 32 | (uint32_t)ex->nfbl); - PUT(vk_mem_gen() + vk_usnap_gen()); // both only grow + PUT(ex->chunked ? ex->pass_serial : 0); + PUT(vk_mem_gen() + vk_usnap_gen() + ex->drop_n); // all only grow in a submission; drop_n: copies a chunk passed for (int i = 0; i < ex->nfbl && i < 9; i++) PUT((uintptr_t)ex->fbl_img[i]); if (l->sup) { float in[3]; @@ -1764,7 +1990,7 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { PUT(a); PUT(ps); } - if (l->c0) PUT(c0_binst); + if (l->c0) PUT(D->c0_binst); for (int i = 0; i < l->np; i++) PUT(ubo_ptr(ex, l->p[i].st, l->p[i].res)); // The uniform overlays the draw's uniform and storage bindings read (op_update_ub): a draw of an updated block binds // other bytes after each update, every other draw is left to skip @@ -1791,12 +2017,12 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { PUT(0); } #undef PUT - diff |= n != RS.hn; - RS.hn = n; - int fast = RS.info == info && RS.gen == bind_gen; - if (fast && RS.on != 2) return !diff; + diff |= n != D->RS.hn; + D->RS.hn = n; + int fast = D->RS.info == info && D->RS.gen == D->bind_gen; + if (fast && bind_skip_on() != 2) return !diff; const uint8_t *e = (const uint8_t *)ex; - uint8_t *d = RS.body; + uint8_t *d = D->RS.body; int bd = 0; for (uint32_t i = 0; i < l->n; i++) { const RsRange *r = &l->r[i]; @@ -1805,7 +2031,7 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { d += r->len; } if (fast && bd) rt_fatal("vk_exec: descriptor build skip check: bound slots changed without a bind op"); - RS.info = info, RS.gen = bind_gen; + D->RS.info = info, D->RS.gen = D->bind_gen; return !diff && !bd; } @@ -1829,24 +2055,25 @@ static void write_set(const VkPipeInfo *info, uint32_t s, VkDescriptorSet set, c // The descriptors go into one packed array d, a VkDescSlot per descriptor in binding order, which a set's update // template reads directly (VkPipeInfo.tpl); write structs are filled only when a set has no template. static void bind_resources(VkExec *ex, const VkPipeInfo *info) { - static VkWriteDescriptorSet w[MAX_BINDINGS]; - static VkDescSlot d[MAX_DESC]; - static VkBufferView bv[MAX_DESC]; + VkExecDraw *D = ex->draw; + VkWriteDescriptorSet *w = ex->draw->w; + VkDescSlot *d = ex->draw->d; + VkBufferView *bv = ex->draw->bv; if (info->n_bindings > MAX_BINDINGS) rt_fatal("vk: %u descriptor bindings exceed the replay's %d", info->n_bindings, MAX_BINDINGS); - bind_stats(info); + bind_stats(ex->draw, info); int gfx = info->bind_point == VK_PIPELINE_BIND_POINT_GRAPHICS; - int skip = gfx ? bind_skip_on() : 0, same = 0, was_valid = B.res_valid; + int skip = gfx ? bind_skip_on() : 0, same = 0, was_valid = D->B.res_valid; if (skip) { same = bind_sig(ex, info); if (same && was_valid && skip == 1) { - for (int i = 0; i < RS.ncnt; i++) vk_mem_count(RS.cnt[i].cls, RS.cnt[i].use, RS.cnt[i].bytes); - BS.bind_skip++; + for (int i = 0; i < D->RS.ncnt; i++) vk_mem_count(D->RS.cnt[i].cls, D->RS.cnt[i].use, D->RS.cnt[i].bytes); + D->BS.bind_skip++; ex->recorded++; return; } - B.res_valid = 0; // until the build below ends - if (skip == 2) memcpy(RS.pcnt, RS.cnt, sizeof RS.cnt), RS.pncnt = RS.ncnt; - RS.rec = 1, RS.ncnt = 0, RS.cnt_over = 0, RS.volatile_ = 0, RS.cmds = 0; + D->B.res_valid = 0; // until the build below ends + if (skip == 2) memcpy(D->RS.pcnt, D->RS.cnt, sizeof D->RS.cnt), D->RS.pncnt = D->RS.ncnt; + D->RS.rec = 1, D->RS.ncnt = 0, D->RS.cnt_over = 0, D->RS.volatile_ = 0, D->RS.cmds = 0; } uint32_t n_sets = info->n_sets ? info->n_sets : 1; uint32_t lo[2] = {0, 0}, hi[2] = {0, 0}, d0[2] = {0, 0}; // bindings [lo, hi) and first descriptor per set @@ -1869,19 +2096,19 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { if (b->kind == VK_PIPE_SUPPORT) { float in[3]; support_in(ex, in); - if (!B.sup_valid || memcmp(in, B.sup_in, sizeof in)) { // support_fill reads only these + if (!D->B.sup_valid || memcmp(in, D->B.sup_in, sizeof in)) { // support_fill reads only these uint8_t tmp[SUPPORT_BYTES]; support_fill(tmp, in[0], in[1], in[2]); - memcpy(B.sup_in, in, sizeof in); - if (!B.sup_valid || memcmp(tmp, B.support, SUPPORT_BYTES)) { // a new buffer only when the contents changed + memcpy(D->B.sup_in, in, sizeof in); + if (!D->B.sup_valid || memcmp(tmp, D->B.support, SUPPORT_BYTES)) { // a new buffer only when the contents changed VkBuffer sb; VkDeviceSize so; memcpy(scratch(ex, SUPPORT_BYTES, &sb, &so), tmp, SUPPORT_BYTES); - memcpy(B.support, tmp, SUPPORT_BYTES); - B.sup_buf = sb, B.sup_off = so, B.sup_valid = 1; + memcpy(D->B.support, tmp, SUPPORT_BYTES); + D->B.sup_buf = sb, D->B.sup_off = so, D->B.sup_valid = 1; } } - o->buffer = (VkDescriptorBufferInfo){B.sup_buf, B.sup_off, SUPPORT_BYTES}; + o->buffer = (VkDescriptorBufferInfo){D->B.sup_buf, D->B.sup_off, SUPPORT_BYTES}; } else if (b->kind == GFX_RES_UBO) { for (uint32_t k = 0; k < cnt; k++) o[k].buffer = ubo_info(ex, st, r->slot + (int)k); } else if (b->kind == GFX_RES_SSBO) { @@ -1897,16 +2124,16 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { } } else { kind = 1; - vk_texture_lock(); // tex_info looks up with the texture lock held + int locked = 0; // tex_info takes the texture lock for the lookups that need it for (uint32_t k = 0; k < cnt; k++) { int slot = r->slot + (int)k; uint64_t h = 0; if (img) h = slot >= 0 && slot < 32 ? ex->img[st][slot] : 0; else h = slot >= 0 && slot < 128 ? ex->tex[st][slot] : 0; - tex_info(ex, r, h, img, r->separate, &o[k].image); + tex_info(ex, r, h, img, r->separate, &o[k].image, &locked); o[k].w[2] = (uint32_t)o[k].image.imageLayout; // zero the tail padding } - vk_texture_unlock(); + if (locked) vk_texture_unlock(); } } else { rt_fatal("vk: binding %u has an unknown kind %d", (unsigned)b->binding, (int)b->kind); @@ -1926,10 +2153,10 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { // layout lacks (false VUID-vkCmdDrawIndexed-None-08114, then an access violation inside the layer). A bind at set 1 // disturbs set 0 by its canonical id, which drops the layer's push set: the push after it gets a new one. int set1_first = 0; - if (gfx && info->layout != B.ds_layout) { // another layout: sets bound before may be disturbed - B.ds_layout = info->layout; - B.ds_bound[0] = B.ds_bound[1] = VK_NULL_HANDLE; - B.ds_pushed[0] = B.ds_pushed[1] = 0; + if (gfx && info->layout != D->B.ds_layout) { // another layout: sets bound before may be disturbed + D->B.ds_layout = info->layout; + D->B.ds_bound[0] = D->B.ds_bound[1] = VK_NULL_HANDLE; + D->B.ds_pushed[0] = D->B.ds_pushed[1] = 0; set1_first = n_sets == 2 && vk_ctx()->validation; } for (uint32_t k = 0; k < n_sets; k++) { @@ -1940,24 +2167,24 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { if (set1_first) { // bound only to drop the layer's push set (comment above) VkDescriptorSet set = alloc_set(ex, info->set_layout1); vkCmdBindDescriptorSets(ex->cb, info->bind_point, info->layout, 1, 1, &set, 0, NULL); - B.ds_bound[1] = set, B.ds_pushed[1] = 0; - RS.cmds++; + D->B.ds_bound[1] = set, D->B.ds_pushed[1] = 0; + D->RS.cmds++; } continue; } if (s ? info->push1 : info->push) { uint32_t n = gfx && push_skip_on() ? dbytes : 0; - if (n && B.ds_pushed[s] && DS.pn[s] == n && !memcmp(DS.psig[s], ds, n)) { - BS.push_skip++; + if (n && D->B.ds_pushed[s] && D->DS.pn[s] == n && !memcmp(D->DS.psig[s], ds, n)) { + D->BS.push_skip++; continue; } if (info->tpl[s]) rec_push_tpl(vk_pipeline_push_tpl_fn(), ex->cb, info->tpl[s], info->layout, s, ds, dbytes); else rec_push(vk_pipeline_push_fn(), ex->cb, info->bind_point, info->layout, s, i1 - i0, w + i0); - BS.push++; - RS.cmds++; + D->BS.push++; + D->RS.cmds++; if (gfx) { - B.ds_bound[s] = VK_NULL_HANDLE, B.ds_pushed[s] = n != 0; - if (n) DS.pn[s] = n, memcpy(DS.psig[s], ds, n); + D->B.ds_bound[s] = VK_NULL_HANDLE, D->B.ds_pushed[s] = n != 0; + if (n) D->DS.pn[s] = n, memcpy(D->DS.psig[s], ds, n); } } else if (!gfx || s > 1) { VkDescriptorSet set = alloc_set(ex, s ? info->set_layout1 : info->set_layout); @@ -1965,44 +2192,44 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { vkCmdBindDescriptorSets(ex->cb, info->bind_point, info->layout, s, 1, &set, 0, NULL); } else { VkDescriptorSetLayout sl = s ? info->set_layout1 : info->set_layout; - VkDescriptorSet set = B.ds_last[s]; - if (!set || B.ds_last_layout[s] != sl || DS.n[s] != dbytes || memcmp(DS.sig[s], ds, dbytes)) { + VkDescriptorSet set = D->B.ds_last[s]; + if (!set || D->B.ds_last_layout[s] != sl || D->DS.n[s] != dbytes || memcmp(D->DS.sig[s], ds, dbytes)) { set = alloc_set(ex, sl); write_set(info, s, set, ds, w, i0, i1); - B.ds_last[s] = set, B.ds_last_layout[s] = sl, DS.n[s] = dbytes; - memcpy(DS.sig[s], ds, dbytes); - BS.set_new++; - RS.cmds++; + D->B.ds_last[s] = set, D->B.ds_last_layout[s] = sl, D->DS.n[s] = dbytes; + memcpy(D->DS.sig[s], ds, dbytes); + D->BS.set_new++; + D->RS.cmds++; } else { - BS.set_reuse++; + D->BS.set_reuse++; } - if (B.ds_bound[s] != set) { + if (D->B.ds_bound[s] != set) { vkCmdBindDescriptorSets(ex->cb, info->bind_point, info->layout, s, 1, &set, 0, NULL); - B.ds_bound[s] = set, B.ds_pushed[s] = 0; - RS.cmds++; + D->B.ds_bound[s] = set, D->B.ds_pushed[s] = 0; + D->RS.cmds++; } } } if (skip) { - RS.rec = 0; + D->RS.rec = 0; // An equal next draw records nothing when every push set holds its push (TOTK_PUSH_SKIP on) and nothing made // the build unrepeatable; allocated sets are always bound to the one written last. - int valid = !RS.volatile_ && !RS.cnt_over && RS.hn; + int valid = !D->RS.volatile_ && !D->RS.cnt_over && D->RS.hn; for (uint32_t s = 0; s < n_sets; s++) - if (hi[s] > lo[s] && (s ? info->push1 : info->push) && !B.ds_pushed[s]) valid = 0; + if (hi[s] > lo[s] && (s ? info->push1 : info->push) && !D->B.ds_pushed[s]) valid = 0; if (skip == 2) { // with the pipeline info equal (bind_sig), the packed descriptors determine every write uint32_t n = (uint32_t)(nd * sizeof *d); - int wr = n != RS.ckn || memcmp(RS.ck, d, n); - int cn = RS.ncnt != RS.pncnt || memcmp(RS.cnt, RS.pcnt, sizeof RS.cnt[0] * (size_t)RS.ncnt); - if (same && was_valid && (wr || cn || RS.cmds)) + int wr = n != D->RS.ckn || memcmp(D->RS.ck, d, n); + int cn = D->RS.ncnt != D->RS.pncnt || memcmp(D->RS.cnt, D->RS.pcnt, sizeof D->RS.cnt[0] * (size_t)D->RS.ncnt); + if (same && was_valid && (wr || cn || D->RS.cmds)) rt_fatal("vk_exec: descriptor build skip check: a draw of program %llu with the inputs of the previous one " "gets %s", (unsigned long long)ex->prog->serial, - RS.cmds ? "descriptor commands" : wr ? "other writes" : "other census counts"); - if (same && was_valid) BS.bind_skip++; // the draws a skip would have spared, compared instead - RS.ckn = n; - memcpy(RS.ck, d, n); + D->RS.cmds ? "descriptor commands" : wr ? "other writes" : "other census counts"); + if (same && was_valid) D->BS.bind_skip++; // the draws a skip would have spared, compared instead + D->RS.ckn = n; + memcpy(D->RS.ck, d, n); } - B.res_valid = valid; + D->B.res_valid = valid; } ex->recorded++; } @@ -2010,12 +2237,13 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { // ---- draws static void bind_pipeline(VkExec *ex, VkPipelineBindPoint bp, VkPipeline p) { + VkExecDraw *D = ex->draw; int k = bp == VK_PIPELINE_BIND_POINT_COMPUTE; - if (B.pipe[k] == p) return; + if (D->B.pipe[k] == p) return; vkCmdBindPipeline(ex->cb, bp, p); ex->recorded++; - B.pipe[k] = p; - if (!k) B.st_valid = 0; + D->B.pipe[k] = p; + if (!k) D->B.st_valid = 0; } static VkViewport draw_viewport(VkExec *ex) { @@ -2071,27 +2299,28 @@ static int vb_used_on(void) { } static void bind_vertex_buffers(VkExec *ex) { + VkExecDraw *D = ex->draw; int dyn = vk_pipeline_vertex_input_dynamic(); - // Only the streams the draw's vertex input reads (B.vmask, from vk_pipeline_set_dynamic): the game keeps about ten + // Only the streams the draw's vertex input reads (D->B.vmask, from vk_pipeline_set_dynamic): the game keeps about ten // bound and a draw reads a few. The streams it does not read keep what is bound. Giving them the zero buffer made // nearly every draw rebind all 17 slots, since draws in a row read different streams; the NVIDIA driver's cost // for those binds took more than the resolves saved (issue #974: RTX 5060 Laptop, 16 % fps back with vb_used=0). // A buffer left bound was bound earlier in this command buffer, so the deferred frees keep it alive until the // submission is done. - uint32_t used = vb_used_on() && B.st_valid ? B.vmask : VK_PIPE_ALL_BINDINGS; + uint32_t used = vb_used_on() && D->B.st_valid ? D->B.vmask : VK_PIPE_ALL_BINDINGS; uint32_t chg = 0; - if (!B.vb_valid) { // the command buffer's first bind sets all 17 slots, the zero buffer where nothing resolves + if (!D->B.vb_valid) { // the command buffer's first bind sets all 17 slots, the zero buffer where nothing resolves for (int b = 0; b < 17; b++) - B.vb[b] = Z.zbuf, B.vo[b] = 0, B.vsz[b] = ZERO_BYTES, B.vstr[b] = b < 16 ? ex->pipe.vstream[b].stride : 0, - B.vcls[b] = 0, B.vpool[b] = NULL; - B.vres = 0, chg = VK_PIPE_ALL_BINDINGS; + D->B.vb[b] = Z.zbuf, D->B.vo[b] = 0, D->B.vsz[b] = ZERO_BYTES, D->B.vstr[b] = b < 16 ? ex->pipe.vstream[b].stride : 0, + D->B.vcls[b] = 0, D->B.vpool[b] = NULL; + D->B.vres = 0, chg = VK_PIPE_ALL_BINDINGS; } // A stream whose range is the one its bound buffer was resolved from, with no pool or mirror change since // (vk_mem_gen, read before any resolve), resolves to that buffer again: most draws change one stream or none. A // change of vk_mem_gen voids the resolutions of every slot, read by this draw or not. uint64_t gen = vk_mem_gen(); - if (B.vgen != gen) B.vres = 0; - B.vgen = gen; + if (D->B.vgen != gen) D->B.vres = 0; + D->B.vgen = gen; for (int b = 0; b < 17; b++) { if (!(used >> b & 1)) continue; VkBuffer nb = Z.zbuf; @@ -2100,8 +2329,8 @@ static void bind_vertex_buffers(VkExec *ex) { void *npool = NULL; if (b != VK_PIPE_ZERO_BINDING) { const NVNbufferRange *r = &ex->vbuf[b]; - if ((B.vres >> b & 1) && B.vin[b].address == r->address && B.vin[b].size == r->size) { - nb = B.vb[b], no = B.vo[b], nsz = B.vsz[b], ncls = B.vcls[b], npool = B.vpool[b]; + if ((D->B.vres >> b & 1) && D->B.vin[b].address == r->address && D->B.vin[b].size == r->size) { + nb = D->B.vb[b], no = D->B.vo[b], nsz = D->B.vsz[b], ncls = D->B.vcls[b], npool = D->B.vpool[b]; } else { VkMemSpan sp; if (r->address && vk_mem_resolve(r->address, 1, &sp)) { @@ -2109,23 +2338,23 @@ static void bind_vertex_buffers(VkExec *ex) { nb = sp.buffer, no = sp.offset, nsz = r->size && r->size < sp.avail ? r->size : sp.avail; ncls = sp.cls, npool = sp.pool; } - B.vin[b] = *r, B.vres |= 1u << b; + D->B.vin[b] = *r, D->B.vres |= 1u << b; } } - if (nb != B.vb[b] || no != B.vo[b] || nsz != B.vsz[b] || (!dyn && nstr != B.vstr[b])) chg |= 1u << b; - B.vb[b] = nb, B.vo[b] = no, B.vsz[b] = nsz, B.vstr[b] = nstr, B.vcls[b] = ncls, B.vpool[b] = npool; + if (nb != D->B.vb[b] || no != D->B.vo[b] || nsz != D->B.vsz[b] || (!dyn && nstr != D->B.vstr[b])) chg |= 1u << b; + D->B.vb[b] = nb, D->B.vo[b] = no, D->B.vsz[b] = nsz, D->B.vstr[b] = nstr, D->B.vcls[b] = ncls, D->B.vpool[b] = npool; } if (!chg) return; // One bind over the changed slots; the slots between them get what they hold again. int lo = __builtin_ctz(chg), hi = 31 - __builtin_clz(chg); - vkCmdBindVertexBuffers2(ex->cb, (uint32_t)lo, (uint32_t)(hi - lo + 1), &B.vb[lo], &B.vo[lo], &B.vsz[lo], - dyn ? NULL : &B.vstr[lo]); + vkCmdBindVertexBuffers2(ex->cb, (uint32_t)lo, (uint32_t)(hi - lo + 1), &D->B.vb[lo], &D->B.vo[lo], &D->B.vsz[lo], + dyn ? NULL : &D->B.vstr[lo]); ex->recorded++; for (int b = lo; b <= hi; b++) { - if (B.vcls[b]) vk_mem_count(B.vcls[b], VK_MUSE_VB, B.vsz[b]); - if ((rt_trace & RT_TRACE_PERF) && B.vpool[b]) vk_mem_bound_note(B.vpool[b]); + if (D->B.vcls[b]) vk_mem_count(D->B.vcls[b], VK_MUSE_VB, D->B.vsz[b]); + if ((rt_trace & RT_TRACE_PERF) && D->B.vpool[b]) vk_mem_bound_note(D->B.vpool[b]); } - B.vb_valid = 1; + D->B.vb_valid = 1; } // TOTK_PIPE_SKIP: which draws are skipped, rather than waited for, while their shaders or pipeline build. @@ -2160,14 +2389,6 @@ static int pipe_skip_mode(void) { // texture type in answers_for, and stops on a difference. // tex_serial (bumped by every texture and image bind) equal to the key's means the handles were not rebound since // they were compared, so the 1 KiB compare is skipped; a rebind to the same handles still matches by the compare. -typedef struct { - const prog_t *prog; - uint64_t serial; - uint32_t gen, tex_serial; - uint64_t tex[2][32], img[2][32]; - VkPipeState pipe; -} MemoKey; -static struct { int on, valid, has_fs; MemoKey k; const GfxShader *vs, *fs; } SM = {-1}; static void memo_key(const VkExec *ex, uint32_t gen, MemoKey *k) { k->prog = ex->prog, k->serial = ex->prog->serial, k->gen = gen, k->tex_serial = ex->tex_serial; @@ -2188,22 +2409,23 @@ static int memo_match(const VkExec *ex, uint32_t gen, MemoKey *k) { } static int shader_memo_on(void) { - if (SM.on < 0) { + static int on = -1; + if (on < 0) { const char *e = cfg_value("TOTK_SHADER_MEMO"); - SM.on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; + on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; } - return SM.on; + return on; } -static struct { int on, valid; VkPipeState s; const VkPipeInfo *info; } PM = {-1}; - static int pipe_memo_on(void) { - if (PM.on < 0) PM.on = cfg_bool("TOTK_PIPE_MEMO"); - return PM.on; + static int on = -1; + if (on < 0) on = cfg_bool("TOTK_PIPE_MEMO"); + return on; } // Pass, shaders, pipeline, fixed state and resources for a draw. 0: the draw is skipped. static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { + VkExecDraw *D = ex->draw; if (ex->render_disabled) return 0; cb_res(ex); dummies_init(); @@ -2220,9 +2442,10 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (!ex->prog) return warn_once(W_PROGRAM, "a draw without a bound program is skipped"), 0; int has_vs, has_fs, vs_wait = 0, fs_wait = 0, memo = shader_memo_on(); const GfxShader *vs, *fs; - uint32_t gen = memo ? vk_texture_slots_gen() : 0; // before shader_for, whose lookups may bump it - if (memo && SM.valid && memo_match(ex, gen, &SM.k)) { - vs = SM.vs, fs = SM.fs, has_vs = 1, has_fs = SM.has_fs; + // Before shader_for, whose lookups may bump it. In a chunk per pass too, as in bind_sig. + uint32_t gen = memo ? vk_texture_slots_gen() + (ex->chunked ? ex->pass_serial : 0) : 0; + if (memo && D->SM.valid && memo_match(ex, gen, &D->SM.k)) { + vs = D->SM.vs, fs = D->SM.fs, has_vs = 1, has_fs = D->SM.has_fs; if (memo == 2) { int hv, hf, wv, wf; const GfxShader *v2 = shader_for(ex, NVN_STAGE_VS, &hv, &wv, 0), *f2 = shader_for(ex, NVN_STAGE_FS, &hf, &wf, 0); @@ -2230,11 +2453,11 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { rt_fatal("vk_exec: shader memo of program %llu differs from the lookup", (unsigned long long)ex->prog->serial); } } else { - SM.valid = 0; + D->SM.valid = 0; vs = shader_for(ex, NVN_STAGE_VS, &has_vs, &vs_wait, 0); fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &fs_wait, 0); if (memo && vs && !vs_wait && !fs_wait && (!has_fs || fs)) - memo_key(ex, gen, &SM.k), SM.vs = vs, SM.fs = fs, SM.has_fs = has_fs, SM.valid = 1; + memo_key(ex, gen, &D->SM.k), D->SM.vs = vs, D->SM.fs = fs, D->SM.has_fs = has_fs, D->SM.valid = 1; } if (!has_vs) return warn_once(W_NO_VS, "a draw with a program that has no vertex stage is skipped"), 0; // A draw into a target the previous frame did not render (or into none) waits for a pending shader or pipeline: @@ -2263,16 +2486,14 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { // its pipeline: skipping it can leave a black tile. The waits share 50 ms per presented frame // (TOTK_PIPE_SKIP=fullscreen skips them instead). A one-shot draw waits without that budget. Everything else is // skipped while its pipeline is built. - static uint64_t wait_frame; - static uint64_t wait_left_ns; uint32_t wait_ms = 0; uint64_t t0 = 0; if (oneshot > 0) { wait_ms = ONESHOT_WAIT_MS; } else if (skip < 2 && count <= 6 && (uint64_t)sc.extent.width * sc.extent.height * 2 >= (uint64_t)ex->pass_w * ex->pass_h) { uint64_t f = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); - if (f != wait_frame) wait_frame = f, wait_left_ns = 50000000; - wait_ms = (uint32_t)(wait_left_ns / 1000000); + if (f != D->wait_frame) D->wait_frame = f, D->wait_left_ns = 50000000; + wait_ms = (uint32_t)(D->wait_left_ns / 1000000); if (wait_ms) t0 = kern_now_ns(); } // The previous draw's pipeline while its state repeats (PM, TOTK_PIPE_MEMO=0 turns it off): the cache key is made @@ -2281,15 +2502,19 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { VkPipeline p; const char *why = NULL; uint64_t w0 = 0; - if (pipe_memo_on() && PM.valid && !memcmp(&PM.s, &s, sizeof s)) { - info = PM.info, pending = 0; + PipeCache *pc = NULL; + if (pipe_memo_on() && !(D->PM.valid && !memcmp(&D->PM.s, &s, sizeof s))) + pc = &D->PC[nvn_hm_hash(&s, sizeof s) & (PC_N - 1)]; + if (pipe_memo_on() && (!pc || (pc->info && !memcmp(&pc->s, &s, sizeof s)))) { + info = pc ? pc->info : D->PM.info, pending = 0; p = __atomic_load_n(&info->pipeline, __ATOMIC_ACQUIRE); + if (pc) D->PM.valid = 1, D->PM.s = s, D->PM.info = info; } else { w0 = t0 ? t0 : wait_ms ? kern_now_ns() : 0; // the clock is read only for draws that may wait p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms); if (t0) { uint64_t spent = kern_now_ns() - t0; - wait_left_ns = spent < wait_left_ns ? wait_left_ns - spent : 0; + D->wait_left_ns = spent < D->wait_left_ns ? D->wait_left_ns - spent : 0; } why = oneshot > 0 ? "one-shot, shader" : wait_ms ? "full-screen" : NULL; if (pending && oneshot < 0 && !vk_exec_targets_recent(ex, skip >= 1)) { @@ -2298,8 +2523,9 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (!w0) w0 = kern_now_ns(); p = vk_pipeline_graphics_async(&s, &info, &pending, ONESHOT_WAIT_MS); } - PM.valid = !pending && p && info; - if (PM.valid) PM.s = s, PM.info = info; + D->PM.valid = !pending && p && info; + if (D->PM.valid) D->PM.s = s, D->PM.info = info; + if (D->PM.valid && pc) pc->s = s, pc->info = info; } uint64_t waited = w0 ? kern_now_ns() - w0 : 0; static int wait_logs; @@ -2314,23 +2540,25 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (pending) return 0; // a background worker is building it: the draw is skipped until it is ready if (!p || !info) return warn_once(W_PIPELINE, "a graphics pipeline could not be created: draws with it are skipped"), 0; bind_pipeline(ex, VK_PIPELINE_BIND_POINT_GRAPHICS, p); - if (!B.st_valid || memcmp(&B.st, &s, sizeof s)) { + if (!D->B.st_valid || memcmp(&D->B.st, &s, sizeof s)) { // TOTK_DYN_DELTA: only the dynamic states whose values changed since the last call in this command buffer. A // pipeline bind clears st_valid but not dyn_valid: every pipeline here declares the same dynamic states, which // the bind leaves as they were (the viewport, scissor and stencil values of record_fixed rely on that too). + // dyn_valid is per context and cleared with D->B for every new command buffer (vk_exec_draw_new_cb). if (DD.mode < 0) DD.mode = rt_knob("DYN_DELTA", "1"); uint32_t cmds = 1; if (RT_KNOB_ON(DD.mode)) { - B.vmask = vk_pipeline_set_dynamic_delta(ex->cb, &s, B.dyn_valid, &cmds); - B.dyn_valid = 1; + D->B.vmask = vk_pipeline_set_dynamic_delta(ex->cb, &s, D->B.dyn_valid, &cmds); + D->B.dyn_valid = 1; } else { - B.vmask = vk_pipeline_set_dynamic(ex->cb, &s); - B.dyn_valid = 0; // the delta's record is stale from here on + D->B.vmask = vk_pipeline_set_dynamic(ex->cb, &s); + D->B.dyn_valid = 0; // the delta's record is stale from here on } - DD.calls++, DD.cmds += cmds, DD.empty += !cmds; + __atomic_fetch_add(&DD.calls, 1, __ATOMIC_RELAXED), __atomic_fetch_add(&DD.cmds, cmds, __ATOMIC_RELAXED); + if (!cmds) __atomic_fetch_add(&DD.empty, 1, __ATOMIC_RELAXED); ex->recorded += cmds != 0; - B.st = s, B.st_valid = 1; - if (ex->mark && vk_exec_frame.draws == ex->mark_draws0) ex->mark->first[0] = vs, ex->mark->first[1] = fs; + D->B.st = s, D->B.st_valid = 1; + if (ex->mark && ex->fr.draws == ex->mark_draws0) ex->mark->first[0] = vs, ex->mark->first[1] = fs; } vk_dlaa_draw(ex); vk_mvsurvey_draw(ex); @@ -2523,12 +2751,12 @@ static int mo_desc(VkExec *ex, const VkPipeInfo *info, MoDraw *q) { } else if (b->kind == GFX_RES_SSBO && !(r->flags & 1)) { // read only: instance data of foliage and such for (uint32_t k = 0; k < cnt; k++) o[k].buffer = ssbo_info(ex, st, r, 0); } else if (b->kind == GFX_RES_TEX && b->type != VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER) { - vk_texture_lock(); + int locked = 0; // tex_info takes the texture lock for the lookups that need it for (uint32_t k = 0; k < cnt; k++) { int slot = r->slot + (int)k; - tex_info(ex, r, slot >= 0 && slot < 128 ? ex->tex[st][slot] : 0, 0, r->separate, &o[k].image); + tex_info(ex, r, slot >= 0 && slot < 128 ? ex->tex[st][slot] : 0, 0, r->separate, &o[k].image, &locked); } - vk_texture_unlock(); + if (locked) vk_texture_unlock(); } else { return 0; } @@ -2651,7 +2879,8 @@ static void mo_emit(VkCommandBuffer cb) { static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSize ib_off, uint64_t t0); static void mvobj_draw(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSize ib_off) { - if (MO.failed || !vk_mvprev_on() || !B.st_valid) return; + VkExecDraw *D = ex->draw; + if (MO.failed || !vk_mvprev_on() || !D->B.st_valid) return; static int timed = -1; if (timed < 0) timed = rt_knob("MVOBJ_TIMES", "0"); // TOTK_MVOBJ_TIMES=1: the summary's times (the clock reads cost) if (!timed) return mvobj_draw_(ex, d, ib, ib_off, 0); @@ -2663,15 +2892,16 @@ static void mvobj_draw(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSiz // TOTK_MVOBJ_BANKS=1 (test): for scene draws, the words of each other uniform bank (c3, c6, c8-c16) that changed // since the same shader's draw last frame, logged at frame 3000: where the time that moves wind and water lives static void mvobj_banks(VkExec *ex) { + VkExecDraw *D = ex->draw; static int on = -1; if (on < 0) on = rt_knob("MVOBJ_BANKS", "0"); - if (!on || !B.st_valid) return; + if (!on || !D->B.st_valid) return; uint64_t frame = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); if (frame < 2999 || frame > 3000) return; static struct { const GfxShader *vs; int bank; uint64_t frame; float w[64]; int n; } tab[4096]; static int logged; const GfxShaderRes *ur; - int nu = gfx_shader_resources(B.st.vs, GFX_RES_UBO, &ur); + int nu = gfx_shader_resources(D->B.st.vs, GFX_RES_UBO, &ur); for (int i = 0; i < nu; i++) { int bank = ur[i].slot; if (bank < 3 || bank == 4 || bank == 5 || bank == 7 || bank > 16) continue; @@ -2679,16 +2909,16 @@ static void mvobj_banks(VkExec *ex) { uint32_t n = u->size < 256 ? (uint32_t)u->size / 4 : 64; const float *now = n ? (const float *)vk_usnap_bytes(u->addr, n * 4) : NULL; if (!now) continue; - uint32_t k = (uint32_t)(((uintptr_t)B.st.vs ^ (uint64_t)bank * 0x9E3779B97F4A7C15ull) * 0xff51afd7ed558ccdull >> 52); + uint32_t k = (uint32_t)(((uintptr_t)D->B.st.vs ^ (uint64_t)bank * 0x9E3779B97F4A7C15ull) * 0xff51afd7ed558ccdull >> 52); if (frame == 2999) { - tab[k].vs = B.st.vs, tab[k].bank = bank, tab[k].frame = frame, tab[k].n = (int)n; + tab[k].vs = D->B.st.vs, tab[k].bank = bank, tab[k].frame = frame, tab[k].n = (int)n; memcpy(tab[k].w, now, n * 4); - } else if (tab[k].vs == B.st.vs && tab[k].bank == bank && tab[k].frame == 2999 && logged < 80) { + } else if (tab[k].vs == D->B.st.vs && tab[k].bank == bank && tab[k].frame == 2999 && logged < 80) { char line[700]; int o = 0, nd = 0; for (int j = 0; j < tab[k].n && j < (int)n && o < 600; j++) if (memcmp(&tab[k].w[j], &now[j], 4)) nd++, o += snprintf(line + o, sizeof line - o, " [%d] %g>%g", j, tab[k].w[j], now[j]); - if (nd) logged++, rt_log("mvbanks: vs %p c%d size %#llx: %d words%s", (const void *)B.st.vs, bank, + if (nd) logged++, rt_log("mvbanks: vs %p c%d size %#llx: %d words%s", (const void *)D->B.st.vs, bank, (unsigned long long)u->size, nd, line); tab[k].frame = 0; } @@ -2743,7 +2973,7 @@ static int mo_prev_block(int index, const p_bind_buf *u, VkDescriptorBufferInfo uint64_t c1 = __builtin_ia32_rdtsc(); MO.cyc[0] += c1 - c0, MO.ncyc++, MO.big += n; if (index >= 0 && index < 14) MO.lk[index][found ? how : MVPREV_NONE]++; - if (!found || !vk_usnap_find(pa, n, LIM.ubo_align, out)) return 0; + if (!found || !vk_usnap_find(pa, n, UINT32_MAX, LIM.ubo_align, out)) return 0; if (how == MVPREV_SELF) return 1; // bound at the same address last frame: not written again const uint8_t *map = vk_usnap_map(), *cur = vk_usnap_bytes(u->addr, n); uint64_t c2 = __builtin_ia32_rdtsc(); @@ -2781,16 +3011,17 @@ static int mo_prev_cached(int index, const p_bind_buf *u, VkDescriptorBufferInfo } static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSize ib_off, uint64_t t0) { + VkExecDraw *D = ex->draw; mvobj_banks(ex); const VkExecTarget *z = &ex->depth; - if (!z->image || z->width < DLAA_SCENE_MIN_W || z->format != VK_FORMAT_D32_SFLOAT || B.st.samples > 1 || B.st.raster_discard) + if (!z->image || z->width < DLAA_SCENE_MIN_W || z->format != VK_FORMAT_D32_SFLOAT || D->B.st.samples > 1 || D->B.st.raster_discard) return; // solid geometry only, which tests and writes the scene depth: effects, particles and screen quads write none, and // drawn again they would put their motion on the surfaces behind them (DLSS takes those from the surface) - if (!B.st.ds.dt || !B.st.ds.dw) return; + if (!D->B.st.ds.dt || !D->B.st.ds.dw) return; // a vertex shader that reads a user bank besides the camera's (c4) may move on its own const GfxShaderRes *ur; - int nu = gfx_shader_resources(B.st.vs, GFX_RES_UBO, &ur), reads = 0; + int nu = gfx_shader_resources(D->B.st.vs, GFX_RES_UBO, &ur), reads = 0; for (int i = 0; i < nu; i++) reads |= ur[i].slot >= 3 && ur[i].slot <= 16 && ur[i].slot != 4; if (!reads) return; MO.cand++; @@ -2802,8 +3033,8 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi // the variant: the translation of the same code and answers, kept per shader once found; it names the banks the // position depends on (0: none, the shader is skipped from then on) static struct { const GfxShader *of, *vs; uint32_t banks; } vmemo[1024]; - uint32_t vi = (uint32_t)((uintptr_t)B.st.vs * 0x9E3779B97F4A7C15ull >> 54); - if (vmemo[vi].of != B.st.vs) { + uint32_t vi = (uint32_t)((uintptr_t)D->B.st.vs * 0x9E3779B97F4A7C15ull >> 54); + if (vmemo[vi].of != D->B.st.vs) { int has, pending = 0; sf_mvprev = 1; const GfxShader *v = shader_for(ex, NVN_STAGE_VS, &has, &pending, 0); @@ -2813,9 +3044,9 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi return; } GfxShaderIo io[32]; - uint32_t extra = gfx_shader_outputs(v, io) & ~gfx_shader_outputs(B.st.vs, io); + uint32_t extra = gfx_shader_outputs(v, io) & ~gfx_shader_outputs(D->B.st.vs, io); uint32_t banks = extra ? gfx_shader_mvprev_banks(v) & 0x1fff8u : 0; - vmemo[vi].of = B.st.vs, vmemo[vi].vs = v, vmemo[vi].banks = banks; + vmemo[vi].of = D->B.st.vs, vmemo[vi].vs = v, vmemo[vi].banks = banks; vk_mvprev_want((banks & ~(1u << 4)) >> 3); // binding index = bank - 3 } const GfxShader *vs = vmemo[vi].vs; @@ -2856,7 +3087,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi exact = e && *e ? strtol(e, NULL, 16) : -1; } if (exact >= 0 && banks != (uint32_t)exact) return; - mo_mask_note(banks, B.st.vs); + mo_mask_note(banks, D->B.st.vs); // last frame's camera and blocks; held still when none of the position's blocks changed (most of the scenery) VkDescriptorBufferInfo cam, prevb[17]; float jit[2]; @@ -2895,7 +3126,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi } if (t0) MO.ns_part[1] += kern_now_ns() - tp, MO.n_part[1]++, tp = kern_now_ns(); GfxShaderIo io[32]; - uint32_t extra = gfx_shader_outputs(vs, io) & ~gfx_shader_outputs(B.st.vs, io); + uint32_t extra = gfx_shader_outputs(vs, io) & ~gfx_shader_outputs(D->B.st.vs, io); const GfxShaderRes *r; if (gfx_shader_resources(vs, GFX_RES_IMG, &r)) { MO.storage++; @@ -2906,7 +3137,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi MO.no_fs++; return; } - VkPipeState ms = B.st; + VkPipeState ms = D->B.st; ms.vs = vs, ms.fs = fs, ms.feedback = 0; memset(ms.color_fmt, 0, sizeof ms.color_fmt); ms.color_fmt[0] = NVN_FORMAT_RG16F; @@ -2918,7 +3149,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi // scene depth passes, anything nearer hides it (TOTK_MVOBJ_EQUAL=1: EQUAL, for comparison) static int eq = -1; if (eq < 0) eq = rt_knob("MVOBJ_EQUAL", "0"); - int df = B.st.ds.df; + int df = D->B.st.ds.df; df = eq ? NVN_DEPTH_FUNC_EQUAL : df == NVN_DEPTH_FUNC_LESS ? NVN_DEPTH_FUNC_LEQUAL : df == NVN_DEPTH_FUNC_GREATER ? NVN_DEPTH_FUNC_GEQUAL : df == NVN_DEPTH_FUNC_LEQUAL || df == NVN_DEPTH_FUNC_GEQUAL ? df : NVN_DEPTH_FUNC_EQUAL; @@ -3094,7 +3325,7 @@ int vk_mvobj_execute(VkExec *ex, uint64_t cam, VkImageView mv, VkImageView depth mo_emit(cb); vkCmdEndRendering(cb); // the draws left their pipeline, descriptors, buffers and dynamic state bound: the next game draw sets everything - vk_exec_draw_new_cb(); + vk_exec_draw_new_cb(ex); ex->dirty = VK_EXEC_DIRTY_ALL; ex->recorded += MQ.n; MQ.n = MQ.nst = MQ.nvb = 0, MQ.ndesc = 0, MQ.frame = 0; @@ -3112,9 +3343,10 @@ static void bind_index(VkExec *ex, const void *data, uint64_t bytes, VkIndexType } static void op_draw(VkExec *ex, const p_draw *d) { - c0_binst = (uint32_t)d->base_instance; + VkExecDraw *D = ex->draw; + D->c0_binst = (uint32_t)d->base_instance; if (d->count <= 0 || !prepare_draw(ex, d->prim, d->count, 0)) return; - vk_exec_frame.draws++; + ex->fr.draws++; uint32_t inst = d->instances > 0 ? (uint32_t)d->instances : 1, first = (uint32_t)d->first, n = (uint32_t)d->count; if (d->prim == NVN_DRAW_PRIMITIVE_QUADS && n >= 4) { // quads become two triangles each uint32_t q = n / 4; @@ -3161,12 +3393,13 @@ static int restart_for(const VkExec *ex, int prim, int itype) { } static void op_draw_idx(VkExec *ex, const p_draw_idx *d) { - c0_binst = (uint32_t)d->base_instance; + VkExecDraw *D = ex->draw; + D->c0_binst = (uint32_t)d->base_instance; static const uint32_t isize[3] = {1, 2, 4}; if (d->itype < 0 || d->itype > 2) return warn_once(W_INDEX, "index type %d is unknown: draw skipped", d->itype); int restart = restart_for(ex, d->prim, d->itype); if (d->count <= 0 || !prepare_draw(ex, d->prim, d->count, restart)) return; - vk_exec_frame.draws++; + ex->fr.draws++; uint64_t bytes = (uint64_t)d->count * isize[d->itype]; VkMemSpan sp; if (!vk_mem_resolve(d->ibuf, bytes, &sp)) return warn_once(W_INDEX, "index buffer %#llx is not backed: draws with it are skipped", (unsigned long long)d->ibuf); @@ -3201,7 +3434,8 @@ static void op_draw_idx(VkExec *ex, const p_draw_idx *d) { // dispatch), as nxnative nvn_mtl.m does. The base instance stays out of c0 (0x644): c0 cannot be patched by a transfer // inside a pass; gl_BaseInstance of the Vulkan draw is the argument's. Byte indices need the count to be widened: skipped. static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { - c0_binst = 0; + VkExecDraw *D = ex->draw; + D->c0_binst = 0; if (d->itype != NVN_INDEX_TYPE_UNSIGNED_SHORT && d->itype != NVN_INDEX_TYPE_UNSIGNED_INT) return warn_once(W_INDEX, "indirect draw with index type %d: skipped", d->itype); VkMemSpan ib, ip; @@ -3210,7 +3444,7 @@ static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { if (!vk_mem_resolve(d->ibuf, 1, &ib) || ib.offset % (d->itype == NVN_INDEX_TYPE_UNSIGNED_SHORT ? 2 : 4)) return warn_once(W_INDEX, "index buffer %#llx is not backed or not aligned: indirect draws skipped", (unsigned long long)d->ibuf); if (!prepare_draw(ex, d->prim, 0x7fffffff, restart_for(ex, d->prim, d->itype))) return; - vk_exec_frame.draws++; + ex->fr.draws++; VK_MEM_USE(ip); VK_MEM_USE(ib); vkCmdBindIndexBuffer(ex->cb, ib.buffer, ib.offset, d->itype == NVN_INDEX_TYPE_UNSIGNED_SHORT ? VK_INDEX_TYPE_UINT16 : VK_INDEX_TYPE_UINT32); @@ -3224,6 +3458,7 @@ static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { // ---- dispatch and uniform buffer updates static void op_dispatch(VkExec *ex, const p_dispatch *d) { + VkExecDraw *D = ex->draw; if (d->x <= 0 || d->y <= 0 || d->z <= 0) return; if (!ex->prog) return warn_once(W_PROGRAM, "a dispatch without a bound program is skipped"); vk_contact_flush(ex); @@ -3237,10 +3472,10 @@ static void op_dispatch(VkExec *ex, const p_dispatch *d) { if (!p || !info) return warn_once(W_PIPELINE, "a compute pipeline could not be created: dispatches with it are skipped"); vk_exec_barrier(ex); // ends the pass; earlier writes are visible to the shader bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p); - c0_dim[0] = (uint32_t)d->x, c0_dim[1] = (uint32_t)d->y, c0_dim[2] = (uint32_t)d->z; - twin_dispatch = 1; + D->c0_dim[0] = (uint32_t)d->x, D->c0_dim[1] = (uint32_t)d->y, D->c0_dim[2] = (uint32_t)d->z; + D->twin_dispatch = 1; bind_resources(ex, info); - twin_dispatch = 0; + D->twin_dispatch = 0; VkPassMark *m = dispatch_mark(ex, cs, (uint32_t)d->x, (uint32_t)d->y, (uint32_t)d->z); vkCmdDispatch(ex->cb, (uint32_t)d->x, (uint32_t)d->y, (uint32_t)d->z); vk_pass_mark_end(ex->cb, m); @@ -3251,6 +3486,7 @@ static void op_dispatch(VkExec *ex, const p_dispatch *d) { // DispatchComputeIndirect: vkCmdDispatchIndirect from the pool buffer. The shader reads the group counts from c0 0x200, // so they are copied there on the GPU, into a c0 range no other dispatch uses, before the dispatch. static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { + VkExecDraw *D = ex->draw; if (!ex->prog) return warn_once(W_PROGRAM, "a dispatch without a bound program is skipped"); VkMemSpan ip; if (!vk_mem_resolve(d->ind, sizeof(VkDispatchIndirectCommand), &ip) || ip.offset % 4) @@ -3266,17 +3502,17 @@ static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { vk_exec_barrier(ex); VK_MEM_USE(ip); bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p); - c0_dim[0] = c0_dim[1] = c0_dim[2] = 0; - B.c0_valid[2] = 0; // a fresh c0 range: the copy below must not change one an earlier dispatch reads - twin_dispatch = 1; + D->c0_dim[0] = D->c0_dim[1] = D->c0_dim[2] = 0; + D->B.c0_valid[2] = 0; // a fresh c0 range: the copy below must not change one an earlier dispatch reads + D->twin_dispatch = 1; bind_resources(ex, info); - twin_dispatch = 0; - if (B.c0_valid[2]) { // the shader binds c0 - VkBufferCopy cp = {ip.offset, B.c0_off[2] + 0x200, sizeof(VkDispatchIndirectCommand)}; - vkCmdCopyBuffer(ex->cb, ip.buffer, B.c0_buf[2], 1, &cp); + D->twin_dispatch = 0; + if (D->B.c0_valid[2]) { // the shader binds c0 + VkBufferCopy cp = {ip.offset, D->B.c0_off[2] + 0x200, sizeof(VkDispatchIndirectCommand)}; + vkCmdCopyBuffer(ex->cb, ip.buffer, D->B.c0_buf[2], 1, &cp); ex->recorded++; vk_exec_barrier(ex); - B.c0_valid[2] = 0; // the range no longer holds B.c0[2] + D->B.c0_valid[2] = 0; // the range no longer holds B.c0[2] } VkPassMark *m = dispatch_mark(ex, cs, 0, 0, 0); vkCmdDispatchIndirect(ex->cb, ip.buffer, ip.offset); @@ -3306,6 +3542,9 @@ static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { // - The descriptor build skip (bind_sig) hashes the overlay each uniform and storage binding of the draw reads // (ov_at), so a draw of an updated block binds the newest bytes and the other draws still skip. The lists are per submission and keep one entry per range (ov_add), and a binding outside their span is // answered in one comparison; when one is full, updates take the old path. +// - A chunked replay (vk_exec.c) takes the old path for every update: OV is one list in replay order, which the chunks +// recording on several threads at once neither share nor could fill in order. Its fence never matches OV.fence, so +// ub_bind, ub_written, ov_at and vk_exec_ub_flush leave its chunks alone. static int ov_on(void) { if (OV.mode < 0) OV.mode = rt_knob("UB_OVERLAY", "1"); @@ -3380,7 +3619,7 @@ static void ov_retire(uint64_t lo, uint64_t hi) { // A GPU write of another kind into [addr, addr+n) in this submission (vk_exec_xfer.c note_gpu_write, storage bindings). void vk_exec_ub_foreign(VkExec *ex, uint64_t addr, uint64_t n) { - if (OV.mode <= 0 || !n) return; + if (OV.mode <= 0 || !n || ex->chunked) return; ov_sub(ex); if (OV.nf == OV_FOREIGN) OV.f_over = 1; else OV.f[OV.nf].lo = addr, OV.f[OV.nf].hi = addr + n, OV.nf++; @@ -3469,7 +3708,7 @@ static int ub_bind(VkExec *ex, uint64_t addr, uint64_t n, uint64_t align, VkDesc VkBuffer b; VkDeviceSize o; uint8_t *dst = scratch(ex, n, &b, &o); - RS.volatile_ = 1; + ex->draw->RS.volatile_ = 1; const uint8_t *base = vk_usnap_bytes(addr, n); VkMemSpan sp; if (base) memcpy(dst, base, n); @@ -3504,7 +3743,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { cb_res(ex); VkBuffer b; VkDeviceSize o; - if (ov_on()) { + if (!ex->chunked && ov_on()) { ov_sub(ex); OV.c_update++; int here = ex->in_pass || ov_out_on(); @@ -3532,7 +3771,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { VkBufferCopy cp = {o, sp.offset, u->size}; vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp); vk_mem_gpu_wrote(&sp, u->size); - vk_usnap_drop(u->dst, u->size); + if (!ex->chunked) vk_usnap_drop(u->dst, u->size); // a chunk: the scan dropped it in its place ex->recorded++; vk_mem_count(sp.cls, VK_MUSE_GPU_WRITE, u->size); note_written(ex, u->dst, u->size); @@ -3549,12 +3788,15 @@ static void bind_handles(uint64_t *dst, int cap, const p_bind_handles *b) { memcpy(&dst[b->first], b->h, sizeof(uint64_t) * (size_t)b->n); } -void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { +// The bind ops: writes into ex alone, so the state scan (vk_exec.c) applies them too. 1 when the op wrote ex->prog, +// ubo, ssbo, tex or img (VkExecDraw.bind_gen). +int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload) { + int gen = 0; switch (op) { case OP_BIND_PROGRAM: { const p_bind_program *b = payload; ex->prog = b->prog, ex->prog_stages = b->stages; - bind_gen++; + gen = 1; break; } case OP_BIND_STATE: { @@ -3597,20 +3839,20 @@ void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { case OP_BIND_UBO: { const p_bind_buf *b = payload; if (b->stage >= 0 && b->stage < 6 && b->index >= 0 && b->index < 18) ex->ubo[b->stage][b->index] = *b; - bind_gen++; + gen = 1; break; } case OP_BIND_SSBO: { const p_bind_buf *b = payload; if (b->stage >= 0 && b->stage < 6 && b->index >= 0 && b->index < 16) ex->ssbo[b->stage][b->index] = *b; - bind_gen++; + gen = 1; break; } case OP_BIND_TEX: { const p_bind_handles *b = payload; bind_handles(b->stage >= 0 && b->stage < 6 ? ex->tex[b->stage] : NULL, 128, b); ex->tex_serial++; - bind_gen++; + gen = 1; break; } case OP_BIND_SAMP: { @@ -3622,9 +3864,21 @@ void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { const p_bind_handles *b = payload; bind_handles(b->stage >= 0 && b->stage < 6 ? ex->img[b->stage] : NULL, 32, b); ex->tex_serial++; // the shader memo's handle check (feedback_scan reads only textures: a rescan at most) - bind_gen++; + gen = 1; break; } + default: rt_fatal("vk: op %s is not a bind op", nvn_op_name(op)); + } + return gen; +} + +void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { + VkExecDraw *D = ex->draw ? ex->draw : draw_new(ex); + switch (op) { + case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: + case OP_BIND_UBO: case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: + if (vk_exec_bind(ex, op, payload)) D->bind_gen++; + break; case OP_UPDATE_UB: op_update_ub(ex, payload); break; case OP_DRAW: op_draw(ex, payload); break; case OP_DRAW_IDX: op_draw_idx(ex, payload); break; diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index 9cca6405..d3b35b19 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -72,7 +72,7 @@ static void guest_written_cb(void *arg) { // The GPU wrote [addr, addr+size) of CPU-visible memory: tell the texture cache once the fence is reached. static void note_gpu_write(VkExec *ex, uint64_t addr, uint64_t size) { - vk_usnap_drop(addr, size); + if (!ex->chunked) vk_usnap_drop(addr, size); // a chunk: the scan dropped it in its place vk_exec_ub_foreign(ex, addr, size); if (nvn_gpu_only(addr)) return; Range *r = malloc(sizeof *r); @@ -129,13 +129,6 @@ static int oq_state; // 0 not set up, 1 counting available, static VkQueryControlFlags oq_flags; static volatile uint8_t oq_busy[OQ_N]; // from begin until the completion thread read and reset the slot static uint32_t oq_next; // replay: next ring slot -static int oq_counting; // replay: draws are counted (a reset since the last report, or oq_keep) -static int oq_reported; // replay: a samples-passed report since the last reset -static int oq_keep; // replay: the game reported twice without a reset: count every draw -static int oq_open = -1; // replay: the slot whose query is active in the open pass -static uint32_t *oq_seg; // replay: slots closed since the last report or reset -static uint32_t oq_nseg, oq_capseg; -static int oq_lost; // replay: the segment missed a draw (ring full) static uint64_t oq_count; // completion thread: samples since the last reset typedef struct { uint64_t addr; int reset, lost; uint32_t n; uint32_t q[]; } OqSegment; @@ -166,39 +159,49 @@ static int oq_ready(void) { } static void oq_close(VkExec *ex) { - if (oq_open < 0) return; - vkCmdEndQuery(ex->cb, oq_pool, (uint32_t)oq_open); + if (ex->oq_open < 0) return; + vkCmdEndQuery(ex->cb, oq_pool, (uint32_t)ex->oq_open); ex->recorded++; - if (oq_nseg == oq_capseg) { - oq_capseg = oq_capseg ? oq_capseg * 2 : 64; - oq_seg = realloc(oq_seg, sizeof *oq_seg * oq_capseg); - if (!oq_seg) rt_fatal("vk: out of memory"); + if (ex->oq_nseg == ex->oq_capseg) { + ex->oq_capseg = ex->oq_capseg ? ex->oq_capseg * 2 : 64; + ex->oq_seg = realloc(ex->oq_seg, sizeof *ex->oq_seg * ex->oq_capseg); + if (!ex->oq_seg) rt_fatal("vk: out of memory"); } - oq_seg[oq_nseg++] = (uint32_t)oq_open; - oq_open = -1; + ex->oq_seg[ex->oq_nseg++] = (uint32_t)ex->oq_open; + ex->oq_open = -1; } void vk_exec_oq_draw(VkExec *ex) { - if (!oq_counting || oq_open >= 0) return; - uint32_t q = oq_next % OQ_N; - if (__atomic_load_n(&oq_busy[q], __ATOMIC_ACQUIRE)) { + if (!ex->oq_counting || ex->oq_open >= 0) return; + uint32_t q = __atomic_fetch_add(&oq_next, 1, __ATOMIC_RELAXED) % OQ_N; // threads recording chunks take slots too + if (__atomic_exchange_n(&oq_busy[q], 1, __ATOMIC_ACQUIRE)) { warn_once(W_QUERY, "occlusion query ring full: draws left uncounted"); - oq_lost = 1; + ex->oq_lost = 1; return; } - oq_busy[q] = 1; - oq_next++; vkCmdBeginQuery(ex->cb, oq_pool, q, oq_flags); ex->recorded++; - oq_open = (int)q; + ex->oq_open = (int)q; } void vk_exec_oq_pass_end(VkExec *ex) { oq_close(ex); } +static void oq_segment(VkExec *ex, uint64_t addr, int reset); + +// The queries a chunk closed: the completion thread adds them to the count before the next chunk's (vk_exec.c). +void vk_exec_oq_flush(VkExec *ex) { + if (ex->oq_nseg || ex->oq_lost) oq_segment(ex, 0, 0); +} + +void vk_exec_oq_forget(VkExec *ex) { + ex->oq_counting = ex->oq_reported = ex->oq_keep = 0, ex->oq_open = -1; + ex->oq_nseg = 0, ex->oq_lost = 0; +} + void vk_exec_xfer_shutdown(void) { if (oq_state > 0) vkDestroyQueryPool(vk_ctx()->device, oq_pool, NULL); - oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_counting = oq_reported = oq_keep = 0, oq_open = -1; - oq_nseg = 0, oq_lost = 0, oq_count = 0; + oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_count = 0; + vk_exec_contexts(vk_exec_oq_forget); memset((void *)oq_busy, 0, sizeof oq_busy); } @@ -240,30 +243,34 @@ static void oq_cb(void *arg) { // Hands the segment to the completion thread with a report (addr; 0: no CPU-visible memory) or a reset. static void oq_segment(VkExec *ex, uint64_t addr, int reset) { oq_close(ex); - OqSegment *s = malloc(sizeof *s + sizeof *s->q * oq_nseg); + OqSegment *s = malloc(sizeof *s + sizeof *s->q * ex->oq_nseg); if (!s) rt_fatal("vk: out of memory"); - s->addr = addr, s->reset = reset, s->lost = oq_lost, s->n = oq_nseg; - memcpy(s->q, oq_seg, sizeof *s->q * oq_nseg); - oq_nseg = 0, oq_lost = 0; + s->addr = addr, s->reset = reset, s->lost = ex->oq_lost, s->n = ex->oq_nseg; + memcpy(s->q, ex->oq_seg, sizeof *s->q * ex->oq_nseg); + ex->oq_nseg = 0, ex->oq_lost = 0; vk_completion_add(ex->fence, oq_cb, s); } static void reset_counter(VkExec *ex, const p_reset_counter *p) { if (p->type != NVN_COUNTER_TYPE_SAMPLES_PASSED || !oq_ready()) return; oq_segment(ex, 0, 1); - oq_counting = 1, oq_reported = 0; + ex->oq_counting = 1, ex->oq_reported = 0; } // A samples-passed report: the segment goes with it (addr 0: no CPU-visible memory) and counting stops until the next // reset, unless reports came without resets before. +static void oq_report_state(VkExec *ex) { + if (ex->oq_reported && !ex->oq_keep) { + ex->oq_keep = 1; + LOG_ONCE("vk: samples passed reported twice without a reset: counting every draw from now on"); + } + ex->oq_reported = 1; + if (!ex->oq_keep) ex->oq_counting = 0; +} + static void oq_report(VkExec *ex, uint64_t addr) { oq_segment(ex, addr, 0); - if (oq_reported && !oq_keep) { - oq_keep = 1; - rt_log("vk: samples passed reported twice without a reset: counting every draw from now on"); - } - oq_reported = 1; - if (!oq_keep) oq_counting = 0; + oq_report_state(ex); } static void signal_event(VkExec *ex, const p_signal_event *p) { @@ -290,7 +297,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) { *a = p->addr; vk_completion_add(ex->fence, counter_cb, a); } - pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; + if (!ex->chunked) pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; // else the scan's } // ---- conditional rendering @@ -612,8 +619,77 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload) { case OP_RENDER_COND: ex->render_cond = *(const p_render_cond *)payload; ex->has_render_cond = 1; - ex->render_disabled = !cond_enabled(payload); + ex->render_disabled = !(ex->chunked ? ex->cond[ex->cond_i++] : cond_enabled(payload)); break; default: rt_fatal("vk: op %s is not a transfer", nvn_op_name(op)); } } + +static int revive_new(const tex_t *t, uint32_t pos) { return t && t->mem && !vk_texture_revive(t, pos); } + +int vk_exec_xfer_revive(uint32_t op, const void *payload, uint32_t pos) { + if (op == OP_COPY_T2T) { + int n = revive_new(((const p_copy_t2t *)payload)->src, pos); + return revive_new(((const p_copy_t2t *)payload)->dst, pos) | n; + } + if (op == OP_COPY_B2T || op == OP_COPY_T2B) return revive_new(((const p_copy_b2t *)payload)->tex, pos); + if (op == OP_CLEAR_TEX) return revive_new(((const p_clear_tex *)payload)->tex, pos); + return 0; +} + +uint64_t vk_exec_xfer_written(const VkExec *ex, uint32_t op, const void *payload, uint64_t *addr) { + VkMemSpan m; + if (op == OP_COPY_B2B) { + const p_copy_b2b *p = payload; + *addr = p->dst; + return vk_mem_resolve(p->src, 1, &m) && vk_mem_resolve(p->dst, 1, &m) ? p->size : 0; + } + if (op == OP_CLEAR_BUF) { + const p_clear_buf *p = payload; + *addr = p->addr; + return (p->addr | p->size) & 3 || !vk_mem_resolve(p->addr, 1, &m) ? 0 : p->size; + } + // COPY_T2B: the buffer side of copy_buffer_image from the texture descriptor (sub_resolve, buffer_side) + const p_copy_b2t *p = payload; + const tex_t *t = p->tex; + int bw, bh, bpb = t ? nvn_fmt_bpb(t->d.format, &bw, &bh) : 0; + *addr = p->src; + if (!bpb) return 0; + uint32_t level = p->has_view && p->view.set_levels ? p->view.base_level : 0; + uint32_t lw = max1((uint32_t)t->d.width >> level), lh = max1((uint32_t)t->d.height >> level); + const NVNcopyRegion *r = &p->r; + uint32_t w = r->width > 0 ? (uint32_t)r->width : r->xoffset < (int32_t)lw ? lw - (uint32_t)r->xoffset : 0; + uint32_t h = r->height > 0 ? (uint32_t)r->height : r->yoffset < (int32_t)lh ? lh - (uint32_t)r->yoffset : 0; + uint32_t nz = r->depth > 0 ? (uint32_t)r->depth : 1; + uint32_t wb = (w + (uint32_t)bw - 1) / (uint32_t)bw, hb = (h + (uint32_t)bh - 1) / (uint32_t)bh; + if (!wb || !hb) return 0; + uint64_t row = ex->copy_stride > 0 ? (uint64_t)ex->copy_stride : (uint64_t)wb * (uint64_t)bpb; + return row * hb * (nz - 1) + row * (hb - 1) + (uint64_t)wb * (uint64_t)bpb; +} + +int vk_exec_xfer_scan(VkExec *ex, uint32_t op, const void *payload) { + switch (op) { + case OP_RENDER_ENABLE: ex->render_disabled = !((const p_flag *)payload)->value; break; + case OP_RENDER_COND: { + int on = cond_enabled(payload); + ex->render_cond = *(const p_render_cond *)payload; + ex->has_render_cond = 1; + ex->render_disabled = !on; + return on; + } + case OP_RESET_COUNTER: + if (((const p_reset_counter *)payload)->type == NVN_COUNTER_TYPE_SAMPLES_PASSED && oq_ready()) + ex->oq_counting = 1, ex->oq_reported = 0; + break; + case OP_REPORT_COUNTER: { + const p_report_counter *p = payload; + int samples = p->type == NVN_COUNTER_TYPE_SAMPLES_PASSED && oq_ready(); + VkMemSpan m; + if (!nvn_gpu_only(p->addr) && !vk_mem_resolve(p->addr, sizeof(NVNcounterData), &m)) break; // the recording stops + if (samples) oq_report_state(ex); + if (!nvn_gpu_only(p->addr)) pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; + break; + } + } + return 1; +} diff --git a/runtime/vk_hdr.c b/runtime/vk_hdr.c index 806ffd1b..bf968357 100644 --- a/runtime/vk_hdr.c +++ b/runtime/vk_hdr.c @@ -243,7 +243,9 @@ void vk_hdr_shader_swap(uint32_t **spirv, uint64_t *words) { int vk_hdr_track; // vk_exec_draw.c calls vk_hdr_draw: set at the first present with TOTK_HDR=1 static volatile int32_t g_active; // the swapchain is HDR10 (gfx_present.c vk_hdr_set_active) static PlatMutex rec_lock = PLAT_MUTEX_INIT; -static struct { VkImage win, src; VkFormat fmt; } recs[4]; // per presented window image: what the last draw copied +// Per presented window image: what the last draw in replay order copied, and where that draw is. Chunks of a replay +// record at once, so a draw of an earlier chunk can get here after one of a later chunk. +static struct { VkImage win, src; VkFormat fmt; uint64_t fence; uint32_t pos; } recs[4]; static int nrecs; void vk_hdr_set_active(int on) { plat_xchg32(&g_active, on); } @@ -277,7 +279,8 @@ void vk_hdr_draw(VkExec *ex, const VkPipeInfo *info) { } plat_mutex_lock(&rec_lock); for (int i = 0; i < nrecs; i++) - if (recs[i].win == t->image) recs[i].src = src, recs[i].fmt = fmt; + if (recs[i].win == t->image && (ex->fence > recs[i].fence || (ex->fence == recs[i].fence && ex->pos >= recs[i].pos))) + recs[i].src = src, recs[i].fmt = fmt, recs[i].fence = ex->fence, recs[i].pos = ex->pos; plat_mutex_unlock(&rec_lock); } diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 3e30682f..5c06c1fb 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -490,12 +490,24 @@ int vk_mem_class_of(uint64_t addr) { // ---------------------------------------------------------------- census static volatile int64_t traffic[VK_MCLS_N][VK_MUSE_N]; +static _Thread_local int64_t (*count_to)[VK_MUSE_N]; + // Every caller is the replay (vk_exec*, under vk_sync_submit's lock), about ten adds per draw: one writer at a time, so -// a plain add, not a locked one. An add racing the census's reset may be lost. +// a plain add, not a locked one. An add racing the census's reset may be lost. Threads recording chunks in parallel +// count into their own arrays (vk_mem_count_local), added with atomics once per chunk. void vk_mem_count(int cls, int use, uint64_t bytes) { - if ((unsigned)cls < VK_MCLS_N && (unsigned)use < VK_MUSE_N) - __atomic_store_n(&traffic[cls][use], __atomic_load_n(&traffic[cls][use], __ATOMIC_RELAXED) + (int64_t)bytes, - __ATOMIC_RELAXED); + if ((unsigned)cls >= VK_MCLS_N || (unsigned)use >= VK_MUSE_N) return; + if (count_to) count_to[cls][use] += (int64_t)bytes; + else __atomic_store_n(&traffic[cls][use], __atomic_load_n(&traffic[cls][use], __ATOMIC_RELAXED) + (int64_t)bytes, + __ATOMIC_RELAXED); +} + +void vk_mem_count_local(int64_t (*acc)[VK_MUSE_N]) { count_to = acc; } + +void vk_mem_count_flush(int64_t (*acc)[VK_MUSE_N]) { + for (int c = 0; c < VK_MCLS_N; c++) + for (int u = 0; u < VK_MUSE_N; u++) + if (acc[c][u]) __atomic_fetch_add(&traffic[c][u], acc[c][u], __ATOMIC_RELAXED), acc[c][u] = 0; } static void flag_names(uint32_t f, char *buf, size_t n) { @@ -1212,6 +1224,10 @@ static VkBufferCopy *mregions; // replay: vkCmdCopyBuffer regions static int cregions; static PlatMutex flush_lock = PLAT_MUTEX_INIT; static int flush_cap = FLUSH_RANGES; // ranges a pool keeps; 1 with TOTK_FLUSH_SPAN=1 +// The first uses of a submission's mirrors, the copies they queue, the GPU writes noted and their snapshots (pend, wbl, snl, mrep): threads +// recording chunks of one submission take it. A mirror synced in this submission is read without it (vk_mem_use): its +// acc_sub is stored last. Before pool_lock, gen_lock and flush_lock. +static PlatMutex mirror_lock = PLAT_MUTEX_INIT; typedef struct { uint64_t subs, syncs, upload_bytes, upload_copies, first_n, first_bytes, wb_bytes, wb_copies, wb_ranges; @@ -1336,16 +1352,21 @@ static void refuse(PoolRec *r, int why, const char *what) { static void mirror_create(PoolRec *r) { VkCtx *c = vk_ctx(); uint64_t n = r->import_hi - r->import_lo; - r->mstate = MS_REFUSED; __atomic_add_fetch(&mirror_gen, 1, __ATOMIC_RELEASE); - if (r->kind != POOL_IMPORT || !r->buf || !n) return; + if (r->kind != POOL_IMPORT || !r->buf || !n) { + __atomic_store_n(&r->mstate, MS_REFUSED, __ATOMIC_RELEASE); + return; + } plat_mutex_lock_shared(&pool_lock); int i = pidx_upper(&pool_index, r->lo) - 1; while (i >= 0 && pool_index.e[i].buf != r) i--; int alone = i >= 0 && (i == 0 || pool_index.maxhi[i - 1] <= pool_index.e[i].lo) && (i + 1 == pool_index.n || pool_index.e[i + 1].lo >= pool_index.e[i].hi); plat_mutex_unlock_shared(&pool_lock); - if (i < 0) return; // finalized meanwhile + if (i < 0) { // finalized meanwhile + __atomic_store_n(&r->mstate, MS_REFUSED, __ATOMIC_RELEASE); + return; + } if (!alone) return refuse(r, MR_OVERLAP, "it overlaps another pool"); uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT; if (r1 >= SNAP_REGIONS) return refuse(r, MR_UNTRACKED, "outside the write-watched address space"); @@ -1383,9 +1404,9 @@ static void mirror_create(PoolRec *r) { mlist[at] = r; nm++; plat_mutex_unlock(&pool_lock); - r->acc_sub = msub; - r->mstate = MS_ON; range_add(&pend, &npend, &cpend, r, 0, n); + __atomic_store_n(&r->acc_sub, msub, __ATOMIC_RELEASE); + __atomic_store_n(&r->mstate, MS_ON, __ATOMIC_RELEASE); mrep.first_n++, mrep.first_bytes += n; plat_inc64(&m_live_n); plat_add64(&m_live_bytes, (int64_t)n); @@ -1426,7 +1447,6 @@ static void pool_retire(void *arg) { // Replay thread, the first access of this submission: queues the copies of the pages written since the last sync. static void mirror_sync(PoolRec *r) { uint64_t t0 = qpc(); - r->acc_sub = msub; uint64_t lo = r->import_lo, hi = r->import_hi; uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT; int excl = 0; @@ -1475,21 +1495,40 @@ static void mirror_sync(PoolRec *r) { } mrep.syncs++; mrep.sync_t += qpc() - t0; + __atomic_store_n(&r->acc_sub, msub, __ATOMIC_RELEASE); +} + +// 1 when r is settled for this submission without mirror_lock: refused, or mirrored and synced (*on). +static int mirror_settled(PoolRec *r, int *on) { + int st = __atomic_load_n(&r->mstate, __ATOMIC_ACQUIRE); + *on = st == MS_ON; + return st == MS_REFUSED || (st == MS_ON && __atomic_load_n(&r->acc_sub, __ATOMIC_ACQUIRE) == msub); } void vk_mem_use(VkMemSpan *sp) { PoolRec *r = sp->pool; - if (r->mstate == MS_NONE) mirror_create(r); - if (r->mstate != MS_ON) return; - if (r->acc_sub != msub) mirror_sync(r); - sp->buffer = r->mbuf; + int on; + if (!mirror_settled(r, &on)) { + plat_mutex_lock(&mirror_lock); + if (r->mstate == MS_NONE) mirror_create(r); + on = r->mstate == MS_ON; + if (on && r->acc_sub != msub) mirror_sync(r); + plat_mutex_unlock(&mirror_lock); + } + if (on) sp->buffer = r->mbuf; } void vk_mem_pick(VkMemSpan *sp) { PoolRec *r = sp->pool; - if (!r || r->mstate != MS_ON) return; - if (r->acc_sub != msub) mirror_sync(r); - sp->buffer = r->mbuf; + int on; + if (!r) return; + if (!mirror_settled(r, &on)) { + plat_mutex_lock(&mirror_lock); + on = r->mstate == MS_ON; + if (on && r->acc_sub != msub) mirror_sync(r); + plat_mutex_unlock(&mirror_lock); + } + if (on) sp->buffer = r->mbuf; } int vk_mem_mirror_probe(uint64_t addr) { @@ -1753,10 +1792,14 @@ static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) { PoolRec *r = sp->pool; if (r && size) gpu_writes(r, sp->offset, size); if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size) return; + plat_mutex_lock(&mirror_lock); // A binding to the end of its pool (no size) can span hundreds of MiB: those stay copies. - if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) + if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) { + plat_mutex_unlock(&mirror_lock); return; + } range_add(&wbl, &nwbl, &cwbl, r, sp->offset, size); + plat_mutex_unlock(&mirror_lock); } void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size) { gpu_wrote(sp, size, 0); } void vk_mem_gpu_wrote_shader(const VkMemSpan *sp, uint64_t size) { gpu_wrote(sp, size, 1); } @@ -1888,10 +1931,19 @@ void vk_mem_mirror_begin(void) { mrep.presnap_t += qpc() - t0; } -int vk_mem_mirror_pending(void) { return npend || snl_copied < nsnl; } +int vk_mem_mirror_pending(void) { + plat_mutex_lock(&mirror_lock); + int n = npend || snl_copied < nsnl; + plat_mutex_unlock(&mirror_lock); + return n; +} void vk_mem_mirror_prologue(VkCommandBuffer cb) { - if (!npend && snl_copied == nsnl) return; + plat_mutex_lock(&mirror_lock); + if (!npend && snl_copied == nsnl) { + plat_mutex_unlock(&mirror_lock); + return; + } mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); copy_ranges(cb, pend, npend, 1, &mrep.upload_bytes, &mrep.upload_copies); @@ -1904,6 +1956,7 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb) { VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); npend = 0; mrep.prologues++; + plat_mutex_unlock(&mirror_lock); } // ---- verify mode diff --git a/runtime/vk_mvsurvey.c b/runtime/vk_mvsurvey.c index 0e782c21..293ddcb8 100644 --- a/runtime/vk_mvsurvey.c +++ b/runtime/vk_mvsurvey.c @@ -157,10 +157,10 @@ void vk_mvsurvey_draw(VkExec *ex) { k->size = (uint32_t)u->size; uint32_t want = u->size < SNAP ? (uint32_t)u->size : SNAP; VkDescriptorBufferInfo bi; - const uint8_t *src = vk_usnap_find(u->addr, want, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; + const uint8_t *src = vk_usnap_find(u->addr, want, UINT32_MAX, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; k->have = src ? want : 0; if (src) memcpy(k->bytes, src, want); - const uint8_t *all = vk_usnap_find(u->addr, u->size, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; + const uint8_t *all = vk_usnap_find(u->addr, u->size, UINT32_MAX, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; k->mats = all ? lead_mats((const float *)all, (uint32_t)(u->size / 4)) : 0; if (all) { Stat *s = &S.st[b]; diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index 57142b21..21aa2de9 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -3223,10 +3223,11 @@ static void dyn_vals(const VkPipeState *s, DynVals *o) { } } -// What the replay's command buffer holds, as vk_pipeline_set_dynamic_delta last recorded it (the replay thread only): -// the values, the inputs of the vertex input and the vertex bindings it reads. Invariant while the caller passes -// valid 1: cb holds DC.v for every dynamic state of P.dyn, colour attachments [0, DC.v.n) included. -static struct { +// What the replay's command buffer holds, as vk_pipeline_set_dynamic_delta last recorded it: the values, the inputs of +// the vertex input and the vertex bindings it reads. Invariant while the caller passes valid 1: cb holds DC.v for every +// dynamic state of P.dyn, colour attachments [0, DC.v.n) included. Per thread: the threads of a chunked replay +// (vk_exec.c) each record their own command buffers, and a new one starts with valid 0. +static _Thread_local struct { DynVals v; const GfxShader *vs; uint32_t vattr[16]; diff --git a/runtime/vk_rec.c b/runtime/vk_rec.c index ddbe24a6..5f80db80 100644 --- a/runtime/vk_rec.c +++ b/runtime/vk_rec.c @@ -454,6 +454,8 @@ void rec_check_log(void) { } #define OWNED(cb) (rec_live && (cb) == rec_cb) + +int rec_owns(VkCommandBuffer cb) { return OWNED(cb); } // A call the ring takes no packet for: everything queued first, then the call here. #define FALLBACK() (rec_drain(), RS_.fallbacks++) diff --git a/runtime/vk_rec.h b/runtime/vk_rec.h index 28c737af..37e5b5d5 100644 --- a/runtime/vk_rec.h +++ b/runtime/vk_rec.h @@ -33,6 +33,7 @@ void rec_drain(void); // waits until the recorder made every cal // vk_queue_lock, before it takes the lock (not on the recorder, not when the thread holds the lock already): waits // until the recorder submitted every submission queued so far. void rec_queue_wait(void); +int rec_owns(VkCommandBuffer cb); // the recorder makes cb's calls: a direct call into cb follows rec_drain void rec_stats_log(void); // TOTK_TRACE=perf: packets, bytes, drains and waits since the last call // TOTK_REC_CHECK=1 (debug, recorder on): vk_memory.c reports each object it defers (destroy 0, stamped tl) or destroys // (destroy 1) with the value the GPU reached (gpu); a packet of the part being recorded that still names it is logged diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index 3b90c026..0493dcfc 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -56,7 +56,7 @@ static VkQueryPool ts_pool; static int ts_state; // 0 not set up, 1 on, -1 off static double ts_period; // ns per tick static uint64_t ts_mask; // valid bits of a timestamp -static uint32_t ts_next; // submitting thread (sub_lock): next pair +static uint32_t ts_next; // next pair (atomic: parts recorded on other threads take pairs too) static volatile uint8_t ts_busy[TS_PAIRS]; // written until the completion thread read the pair static int ts_cur = -1; // the pair of the part being recorded, -1 untimed static uint64_t ts_last_end; // completion thread: end of the latest part read (ticks) @@ -175,6 +175,12 @@ static void ts_read_cb(void *arg) { // completion thread, in submission order free(t); } +static int ts_take(void) { + uint32_t p = __atomic_fetch_add(&ts_next, 1, __ATOMIC_RELAXED) % TS_PAIRS; + if (__atomic_exchange_n(&ts_busy[p], 1, __ATOMIC_ACQUIRE)) { plat_inc64(&ts_missed); return -1; } + return (int)p; +} + // At the start of a command buffer (sub_lock held): the pair it times, -1 untimed. static int ts_open(VkCommandBuffer cb) { if (ts_state != 1 || !perf_gpu_timing()) { @@ -182,10 +188,9 @@ static int ts_open(VkCommandBuffer cb) { ts_setup(vk_ctx()); // the overlay turned full: timestamps from now on if (ts_state != 1) return -1; } - uint32_t p = ts_next % TS_PAIRS; - if (__atomic_load_n(&ts_busy[p], __ATOMIC_ACQUIRE)) { plat_inc64(&ts_missed); return -1; } - ts_next++; - ts_busy[p] = 1; + int t = ts_take(); + if (t < 0) return -1; + uint32_t p = (uint32_t)t; vkCmdResetQueryPool(cb, ts_pool, 2 * p, 2); vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, ts_pool, 2 * p); return (int)p; @@ -203,6 +208,17 @@ static void ts_close(VkCommandBuffer cb, int pair, uint64_t fence) { vk_completion_add(fence, ts_read_cb, t); } +int vk_sync_ts_open(VkCommandBuffer cb) { + if (__atomic_load_n(&ts_state, __ATOMIC_ACQUIRE) != 1 || !perf_gpu_timing()) return -1; + int p = ts_take(); + if (p < 0) return -1; + vkCmdResetQueryPool(cb, ts_pool, 2 * (uint32_t)p, 2); + vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, ts_pool, 2 * (uint32_t)p); + return p; +} + +void vk_sync_ts_close(VkCommandBuffer cb, int pair, uint64_t fence) { ts_close(cb, pair, fence); } + // The pair of the part being recorded. static void ts_begin(VkCommandBuffer cb) { ts_cur = ts_open(cb); } @@ -349,26 +365,56 @@ static CmdSlot *take_slot(VkCtx *c, uint64_t fence) { // submitted right before the part that needs them and timed like a part; VK_NULL_HANDLE when there are none. After the // part was recorded, sub_lock held; may move slots[]. // First brings the shadow storage up to date with the CPU's writes, which needs no command buffer. -static VkCommandBuffer prologue(VkCtx *c, uint64_t fence) { +// direct: the calling thread records, ends and submits it itself while the part being replayed stays the recorder's +// (vk_sync_part, which the caller made the recorder idle for). +static VkCommandBuffer prologue(VkCtx *c, uint64_t fence, int direct) { vk_mem_shadow_sync(); // shadow storage: the CPU's writes go in before the part is submitted if (!vk_mem_mirror_pending()) return VK_NULL_HANDLE; int i = claim_slot(c, fence); // before reading slots: the claim may move it VkCommandBuffer cb = slots[i].cb; - rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts + if (direct) { + VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vkBeginCommandBuffer(cb, &bi); + } else { + rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts + } int pair = ts_open(cb); vk_mem_mirror_prologue(cb); ts_close(cb, pair, fence); - if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); + if (direct || !rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); return cb; } +// A chunk of a chunked replay (vk_exec.c chunks_submit), while the submission's own part is open on the recorder. The +// recorder begins, records, ends and submits the parts it owns (vk_rec.c rec_end_cb), so the prologue stays off it: +// the recorder is made idle first, as the part pool allows one thread at a time, and the prologue is recorded, ended +// and submitted here without taking the recorder's part away. +void vk_sync_part(VkCommandBuffer cb) { + VkCtx *c = vk_ctx(); + vk_upload_flush(0); + rec_drain(); + VkCommandBuffer pro = prologue(c, cur_fence, 1); + VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, + {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}}; + VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1, + .pCommandBufferInfos = pro ? cbi : cbi + 1}; + vk_queue_lock(); + VkResult r = vkQueueSubmit2(c->gfx_queue, 1, &si, VK_NULL_HANDLE); + vk_queue_unlock(); + if (r != VK_SUCCESS) { + vk_device_fault_report(r); + rt_fatal("vk: vkQueueSubmit2 of a part of value %llu failed (%d)", (unsigned long long)cur_fence, (int)r); + } +} + VkCommandBuffer vk_sync_split(void) { VkCtx *c = vk_ctx(); VkCommandBuffer cb = slots[cur_slot].cb; ts_end(cb, cur_fence); if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); // every command of the part is in cb, ended by its recorder vk_upload_flush(0); // the uploads this part asked for precede it in queue order - VkCommandBuffer pro = prologue(c, cur_fence); + VkCommandBuffer pro = prologue(c, cur_fence, 0); VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}}; VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1, @@ -419,7 +465,7 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) { perf_replay(t0); // After the recording: the uploads it asked for (and every earlier one) precede it in queue order. vk_upload_flush(0); - VkCommandBuffer pro = prologue(c, fence); + VkCommandBuffer pro = prologue(c, fence, 0); s = &slots[cur_slot]; VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = s->cb}}; @@ -446,8 +492,11 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) { return fence; } -void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) { - plat_mutex_lock(&st_lock); +static _Thread_local VkDeferred *deferred; + +void vk_completion_defer(VkDeferred *d) { deferred = d; } + +static void queue_put(uint64_t fence, void (*fn)(void *), void *arg) { // st_lock held if (qhead + qn == qcap) { if (qhead) { memmove(queue, queue + qhead, sizeof *queue * (size_t)qn); @@ -463,9 +512,34 @@ void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) { // The completion thread sleeps only while nothing is issued beyond done: a callback for a later value (the replay's, // for the submission it records: counter reports, freed resources) becomes due through the submit, which wakes it. if (fence <= done) plat_cond_broadcast(&cv_work); +} + +void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) { + VkDeferred *d = deferred; + if (d) { + if (d->n == d->cap) { + d->cap = d->cap ? d->cap * 2 : 64; + d->items = realloc(d->items, sizeof(Pending) * (size_t)d->cap); + if (!d->items) rt_fatal("vk: out of memory (completion queue)"); + } + ((Pending *)d->items)[d->n++] = (Pending){fence, fn, arg}; + return; + } + plat_mutex_lock(&st_lock); + queue_put(fence, fn, arg); plat_mutex_unlock(&st_lock); } +void vk_completion_flush(VkDeferred *d) { + plat_mutex_lock(&st_lock); + for (int i = 0; i < d->n; i++) { + const Pending *p = &((Pending *)d->items)[i]; + queue_put(p->fence, p->fn, p->arg); + } + plat_mutex_unlock(&st_lock); + d->n = 0; +} + static int fence_wait_impl(uint64_t value, uint64_t timeout_ns); int vk_sync_fence_wait(uint64_t value, uint64_t timeout_ns) { diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index d461f561..99ecb49b 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -46,6 +46,7 @@ typedef struct Tex { uint32_t wgen; // page generation of the last upload check uint32_t checked; // submit_epoch of the last check from a draw binding int uploaded, released; + uint32_t rev_epoch, rev_pos; // the state scan made it live in that submission at that op (vk_texture_revive) uint32_t n_up; // uploads of this entry (vk_texture_debug) int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked struct Tex *next_at; // other entries at the same storage address @@ -122,10 +123,12 @@ static uint64_t n_checks; // draw-binding storage checks (make_fresh_snap) si // which emptied every cache keyed on the global generations. #define AGEN_N 4096 static uint32_t agen[AGEN_N]; +static uint32_t arev[AGEN_N]; // per storage address (hashed): the submit_epoch of the state scan's last revival there // Per slot id, for vk_texture_slot_token without the lock: the registration serial (bits 16-47; 0: never registered) // and the address generation index (bits 0-15), one word so a reader gets a consistent pair. Fixed-size (slot ids are // 20 bits), unlike tslots, which moves when it grows. static uint64_t *sinfo; +static uint32_t *sepoch; // per slot: the submit_epoch its entry was checked in (vk_texture_slot_checked), 0 none static uint32_t reg_serial; static uint32_t agen_idx(uint64_t addr) { return (uint32_t)(addr * 0x9E3779B97F4A7C15ull >> 52); } @@ -1288,6 +1291,13 @@ static int alias_copy(Tex *e, const Tex *src) { return 1; } +static void entry_live(Tex *e) { + if (!e->released) return; + takeback_n[e->pub.gpu_owned != 0][age_bucket(coarse_ms - e->rel_ms)]++; + e->use_seq = ++use_counter; + e->released = 0, nreleased--, rel_unlink(e), tex_gen_bump(e->key.addr); +} + static Tex *entry_of(const tex_t *t) { TexKey k; make_key(t, &k); @@ -1336,11 +1346,7 @@ static Tex *entry_of(const tex_t *t) { nvn_hm_put(&by_key, &k, e); tex_gen_bump(k.addr); } - if (e->released) { - takeback_n[e->pub.gpu_owned != 0][age_bucket(coarse_ms - e->rel_ms)]++; - e->use_seq = ++use_counter; - e->released = 0, nreleased--, rel_unlink(e), tex_gen_bump(k.addr); - } + entry_live(e); e->t = *t; return e; } @@ -1366,6 +1372,25 @@ const VkTexImage *vk_texture_get(const tex_t *t) { return e ? &e->pub : NULL; } +// The replay's state scan (vk_exec.c): the entry vk_texture_get would return, neither created nor made current; the +// recording does that. It is revived here, in replay order, and remembers the op it happened at: a chunk may bind a +// finalized target's slot before the chunk that renders it reached its SET_RT, or long after the scan went past a later +// target at the same storage, and slot_resolve has to answer as the replay would at the chunk's own op (live_at). +const VkTexImage *vk_texture_revive(const tex_t *t, uint32_t pos) { + if (!t || !t->mem) return NULL; + TexKey k; + make_key(t, &k); + lock(); + Tex *e = nvn_hm_get(&by_key, &k); + if (e && e->released) { + e->rev_epoch = submit_epoch, e->rev_pos = pos; + __atomic_store_n(&arev[agen_idx(k.addr)], submit_epoch, __ATOMIC_RELEASE); + entry_live(e); + } + unlock(); + return e ? &e->pub : NULL; +} + // Render targets the CPU reads without NVN's invalidate (vk_texture_write_back_small); small_pending: one may be dirty. // wb_limit: targets up to this size (bytes) always go back when the CPU waits for a fence, TOTK_WB_FENCE_LIMIT, read // once. A larger one goes back unless the GPU wrote it in each of the last WB_RUN frames (wb_wanted). wb_trace: @@ -1468,7 +1493,9 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) { } if (!sinfo) { uint64_t *p = calloc(1 << 20, sizeof *p); - if (!p) rt_fatal("vk: out of memory for texture slot tokens"); + uint32_t *q = calloc(1 << 20, sizeof *q); + if (!p || !q) rt_fatal("vk: out of memory for texture slot tokens"); + __atomic_store_n(&sepoch, q, __ATOMIC_RELEASE); __atomic_store_n(&sinfo, p, __ATOMIC_RELEASE); } TexSlot *s = &tslots[id]; @@ -1484,15 +1511,23 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) { unlock(); } -// The entry slot s resolves to, and the view and has_view a caller gets with it (the slot's, with the descriptor's -// swizzle when the entry is another descriptor at the same storage). NULL when the slot names no texture. -static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) { +// Whether e is live at op pos of a chunked replay: the state scan is ahead of the chunks, and an entry it revived at a +// later op is still released for this one. Two 4K targets at one address, RGBA8 and R8 with memory of their own, were +// swapped whenever the scan got past the second's SET_RT before a chunk sampling the storage got its turn, and large +// parts of the scene flickered. UINT32_MAX: the state now. +static int live_at(const Tex *e, uint32_t pos) { + return !e->released && !(e->rev_epoch == submit_epoch && e->rev_pos > pos); +} + +// The entry slot s resolves to at op pos, and the view and has_view a caller gets with it (the slot's, with the +// descriptor's swizzle when the entry is another descriptor at the same storage). NULL when the slot names no texture. +static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view, uint32_t pos) { *view = s->v; *has_view = s->has_view; TexKey k; make_key(&s->t, &k); Tex *e = nvn_hm_get(&by_key, &k); - if (!e || e->released) { // MNG hle_nvn.c: a finalized slot follows the live texture at its storage + if (!e || !live_at(e, pos)) { // MNG hle_nvn.c: a finalized slot follows the live texture at its storage // Prefer the live entry that differs from the slot descriptor only in its swizzle: transient render targets // share storage, and the newest one at the address is usually another size. The map screen samples its // 2250x1080 map target through an (R,G,B,1) descriptor and followed a 960x60 target at the same address. @@ -1505,7 +1540,7 @@ static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) { Tex *own = e, *any = NULL; e = NULL; for (Tex *p = nvn_hm_get(&by_addr, &k.addr); p; p = p->next_at) { - if (p->released || linear_format(p->key.d.format) != linear_format(k.d.format)) continue; + if (!live_at(p, pos) || linear_format(p->key.d.format) != linear_format(k.d.format)) continue; if (!any) any = p; if (same_but_swizzle(&p->key, &k)) { e = p; break; } } @@ -1553,14 +1588,28 @@ const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view) { const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view) { if (id < 0 || id >= ntslots || !tslots[id].used) return NULL; TexSlot *s = &tslots[id]; - uint32_t g = agen[sinfo[id] & 0xffff]; + uint32_t idx = sinfo[id] & 0xffff, g = agen[idx]; + uint32_t pos = arev[idx] == submit_epoch ? vk_exec_replay_pos() : UINT32_MAX; + if (pos != UINT32_MAX) { // what the storage holds depends on the op: chunks at other ops share no answer in s + view_t v; + int hv; + Tex *f = slot_resolve(s, &v, &hv, pos); + if (f && (!f->uploaded || f->checked != submit_epoch)) { + make_fresh_snap(f); + f->checked = submit_epoch; + } + if (view) *view = has_view ? v : s->v; + if (has_view) *has_view = view ? hv : s->has_view; + return f ? &f->pub : NULL; + } if (!s->ce || s->cgen != g) { // resolved again: an entry at its address was created, dropped or released, or the view_t v; // slot registered (s->ce cleared); an empty result is not kept: image creation may int hv; // succeed later - s->ce = slot_resolve(s, &v, &hv); // entry_of may create an entry and bump g: g read before it + s->ce = slot_resolve(s, &v, &hv, UINT32_MAX); // entry_of may create an entry and bump g: g read before it s->cview = v, s->chas_view = (uint8_t)hv; s->cgen = g; s->cepoch = 0; + __atomic_store_n(&sepoch[id], 0, __ATOMIC_RELAXED); } Tex *e = s->ce; // Bindings of one submit share one check, answered from the submit's write-watch snapshot (guest writes during a @@ -1572,6 +1621,7 @@ const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view) { e->checked = submit_epoch; } s->cepoch = submit_epoch; + __atomic_store_n(&sepoch[id], submit_epoch, __ATOMIC_RELEASE); } if (view) *view = has_view ? s->cview : s->v; if (has_view) *has_view = view ? s->chas_view : s->has_view; @@ -1593,6 +1643,19 @@ int vk_texture_slot_swz(int id, uint8_t swz[4]) { uint32_t vk_texture_epoch(void) { return __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); } +int vk_texture_slot_moves(int id) { + const uint64_t *si = __atomic_load_n(&sinfo, __ATOMIC_ACQUIRE); + if (!si || id < 0 || id >= (1 << 20)) return 0; + uint64_t v = __atomic_load_n(&si[id], __ATOMIC_RELAXED); + return __atomic_load_n(&arev[v & 0xffff], __ATOMIC_ACQUIRE) == __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); +} + +int vk_texture_slot_checked(int id) { + const uint32_t *e = __atomic_load_n(&sepoch, __ATOMIC_ACQUIRE); + return e && id >= 0 && id < (1 << 20) && + __atomic_load_n(&e[id], __ATOMIC_ACQUIRE) == __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); +} + void vk_texture_new_submit(void) { lock(); if (!++submit_epoch) submit_epoch = 1; @@ -2419,6 +2482,6 @@ void vk_texture_shutdown(void) { free(tslots), free(sslots); tslots = NULL, sslots = NULL, ntslots = nsslots = 0, nreleased = 0; rel_head = rel_tail = NULL; - if (sinfo) memset(sinfo, 0, sizeof *sinfo << 20); // no slot is registered: tokens 0 + if (sinfo) memset(sinfo, 0, sizeof *sinfo << 20), memset(sepoch, 0, sizeof *sepoch << 20); // no slot is registered: tokens 0 unlock(); } diff --git a/runtime/vk_usnap.c b/runtime/vk_usnap.c index 4e9744b9..660aa724 100644 --- a/runtime/vk_usnap.c +++ b/runtime/vk_usnap.c @@ -11,11 +11,13 @@ // that submission's write-back, the mirror holds them until then. // - A GPU write within the submission (UpdateUniformBuffer, a storage buffer a shader writes, a buffer copy or fill, // an image-to-buffer copy: vk_usnap_drop) drops the copies it touches from then on, so later draws read the result. +// A chunked replay (vk_exec.c) records the ops out of order: its scan drops at the position of the op +// (vk_usnap_drop_at, scan_writes for the shaders' stores), and a chunk finds a copy only at a position before that. // - The arena goes back to a free list once the submission's fence is done (completion thread). // The design follows MacNativeGames' TOTK_UBO_SNAPSHOT (games/totk/third_party/d2r_gfx/gfx/nvn/nvn_mtl.m usnap_take, // usnap_drop; MIT, Copyright (c) 2026 jhaizhou); the code is ours. // The copy is taken where vk_exec_record starts: at the submit call when the replay runs on the submitting thread, later -// with TOTK_REPLAY_ASYNC. TOTK_UBO_SNAPSHOT=0 binds live memory as before; =ab alternates (TOTK_AB). +// with TOTK_REPLAY_ASYNC; a chunked replay takes it on a recording thread while its scan runs. TOTK_UBO_SNAPSHOT=0 binds live memory as before; =ab alternates (TOTK_AB). #include #include @@ -42,7 +44,7 @@ typedef struct Arena { } Arena; typedef struct { uint64_t lo, hi; } Rng; -typedef struct { uint64_t lo, hi, off; int dropped; } Reg; +typedef struct { uint64_t lo, hi, off, drop_at; } Reg; // drop_at: dropped from this position on; UINT64_MAX not typedef struct { uint64_t lo, hi, fence; } Wrote; static struct { @@ -62,6 +64,7 @@ static struct { static Arena *free_arenas; static PlatMutex free_lock = PLAT_MUTEX_INIT; +static PlatMutex drop_lock = PLAT_MUTEX_INIT; // threads recording chunks of one submission drop concurrently static void *grow(void *p, int *cap, size_t elem) { int n = *cap ? *cap * 2 : 1024; @@ -279,7 +282,7 @@ void vk_usnap_begin(const uint64_t *handles, int n, uint64_t fence) { continue; } if (U.nreg == U.creg) U.reg = grow(U.reg, &U.creg, sizeof *U.reg); - U.reg[U.nreg++] = (Reg){lo, hi, total, 0}; + U.reg[U.nreg++] = (Reg){lo, hi, total, UINT64_MAX}; total += (hi - lo + ALIGN - 1) & ~(uint64_t)(ALIGN - 1); } if (!U.nreg) return; @@ -313,29 +316,38 @@ static int first_above(uint64_t a) { return lo; } -int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t align, VkDescriptorBufferInfo *out) { +int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t at, uint64_t align, VkDescriptorBufferInfo *out) { if (!U.on || !n) return 0; int i = first_above(addr); if (i >= U.nreg) return 0; const Reg *r = &U.reg[i]; - if (r->dropped || addr < r->lo || addr + n > r->hi) return 0; + if (__atomic_load_n(&r->drop_at, __ATOMIC_RELAXED) <= at || addr < r->lo || addr + n > r->hi) return 0; uint64_t off = r->off + (addr - r->lo); if (off % align) return 0; *out = (VkDescriptorBufferInfo){U.arena->buf, off, n}; - U.hits++; + __atomic_fetch_add(&U.hits, 1, __ATOMIC_RELAXED); return 1; } -void vk_usnap_drop(uint64_t addr, uint64_t n) { +static void drop(uint64_t addr, uint64_t n, uint64_t at) { if (!U.mode || !n) return; + plat_mutex_lock(&drop_lock); if (U.nwr == U.cwr) U.wr = grow(U.wr, &U.cwr, sizeof *U.wr); U.wr[U.nwr++] = (Wrote){addr, addr + n, U.fence}; - if (!U.on) return; - for (int i = first_above(addr); i < U.nreg && U.reg[i].lo < addr + n; i++) - if (!U.reg[i].dropped) U.reg[i].dropped = 1, U.drops++, U.gen++; + if (U.on) + for (int i = first_above(addr); i < U.nreg && U.reg[i].lo < addr + n; i++) + if (at < U.reg[i].drop_at) { + if (U.reg[i].drop_at == UINT64_MAX) U.drops++; + __atomic_store_n(&U.reg[i].drop_at, at, __ATOMIC_RELAXED); + __atomic_add_fetch(&U.gen, 1, __ATOMIC_RELAXED); + } + plat_mutex_unlock(&drop_lock); } -uint64_t vk_usnap_gen(void) { return U.gen; } +void vk_usnap_drop(uint64_t addr, uint64_t n) { drop(addr, n, 0); } +void vk_usnap_drop_at(uint64_t addr, uint64_t n, uint32_t at) { drop(addr, n, at); } + +uint64_t vk_usnap_gen(void) { return __atomic_load_n(&U.gen, __ATOMIC_RELAXED); } // the host copy where there is one (DLSS), else the device copy static uint8_t *host_copy(void) { return U.arena->cpu ? U.arena->cpu : U.arena->map; } @@ -344,7 +356,7 @@ const uint8_t *vk_usnap_map(void) { return U.arena ? host_copy() : NULL; } const uint8_t *vk_usnap_bytes(uint64_t addr, uint64_t n) { VkDescriptorBufferInfo bi; - return vk_usnap_find(addr, n, 1, &bi) ? host_copy() + bi.offset : NULL; + return vk_usnap_find(addr, n, UINT32_MAX, 1, &bi) ? host_copy() + bi.offset : NULL; } // The arenas back on the free list (vk_exec_draw_shutdown: the device is idle, every completion ran). diff --git a/totk.ini.example b/totk.ini.example index 33e7dc4a..c26ae91b 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -38,6 +38,11 @@ # Replay NVN submissions on their own thread, at most one frame behind the game, 0 or 1. Default: off. Helps where the # game waits for the GPU; adds up to a frame of latency. # replay_async = 0 +# Record each large submission in chunks on this many threads (0 to 16; 0 = one thread, the default). The chunks go to +# the GPU in order. In the town at 4K on an RTX 5070, 4 cut the replay from 8.2 to 4.5 ms a frame (71 to 79 fps); more +# threads add little. Needs 8 or more logical processors to help. Contact shadows run in chunks too; DLSS (dlaa = 1) and +# fxaa = 0 keep the replay on one thread. Read at start. +# replay_workers = 0 # EXPERIMENTAL, THE GAME CRASHES WITH 1. 1 lifts the virtual core gate: guest threads run without the console's limit # of one thread per core, and the game keeps per-core data without locks. With 1, 1.2.1 and 1.4.3 crash or hang, # usually at start (#212). Keep 0 to play.