Merge task/chunked: record large submissions in chunks on several threads (#628, #794), off by default

replay_workers stays 0. On the 7900X 4 threads cut the replay by a quarter (town 6.7 -> 5.0 ms, fight2 5.3 -> 3.9 ms)
with no fps change: the game's own threads bound the frame there. Weak CPUs may gain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-11-08 20:24:08 +03:00
commit 5e56bfc7f3
25 files changed
+2245 -665

No files matched your search

+1
View File
@@ -85,6 +85,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app
- Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`).
- Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`).
- Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on.
- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's.
- Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread).
- Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame
that reads them, while the replay still records it. At the start of a submission's replay every uniform range it
+4
View File
@@ -182,6 +182,10 @@ Many rows below are developer tools that only act together with another setting
| `audio_off` | Any value mixes audio without opening an output device. | The game runs in silence and reports two channels; same as `audio = none`. | Caution |
| `one_per_core` (default `0`) | `1` keeps the process on the first logical processor of each physical core; `ab` switches it with the `ab` blocks. | Busy threads stop sharing a core with a sibling thread, which can help or hurt on the CPU; it only applies when the CPU has hyperthreads. | Caution |
| `reexec` | Counter of address-space retries that the runtime sets itself when it starts the process again. | The counter is cleared when the window is recreated; a value of 20 or more stops the start with a fatal error. | Risky |
| `replay_chunks` (default `0`) | Most chunks a submission is cut into when `replay_workers` is set; `0` uses twice the number of recording threads. | Fewer chunks leave threads idle at the end of a submission, more add scan and submit work per chunk; no effect while `replay_workers` is `0`. | Caution |
| `replay_chunk_draws` (default `128`) | Least draws a chunk of the chunked replay holds before it may end at a pass. | Small values make many short chunks, large ones leave big submissions on few threads; no effect while `replay_workers` is `0`. | Caution |
| `replay_state_check` (default `0`) | `1` compares the replay state after the last chunk of a chunked submission with the state of the scan and logs differences. | Adds a comparison per submission and log lines on a mismatch; no effect while `replay_workers` is `0`. | Caution |
| `replay_skew_us` (default `0`) | Debug: the even chunks of a chunked submission wait this many microseconds before they record, so the chunk after each one records first. | Shows a recording that depends on which thread gets to a chunk first in every submission; slows the replay; no effect with fewer than 2 `replay_workers`. | Caution |
| `restart_wait` | Set by the runtime when it restarts itself; names the old process by id and creation time so the new process waits for it to exit. | Meant for the runtime's own use; the code does not describe a manual value. | Unknown |
| `spin` (default `1`) | `0` makes a guest thread that polls memory keep its virtual core. | Polling threads can starve the thread whose item they wait for, so frames can stall. | Caution |
| `spin_handoff_us` (default `50`) | Microseconds a polling wait lasts before it gives its virtual core to a waiting thread. | A shorter value hands cores over sooner; a longer one keeps them longer. | Caution |
+5 -2
View File
@@ -53,7 +53,7 @@ static const char *const known[] = {
"ONE_PER_CORE", "OPERATION_MODE", "OUTPUT_RES", "PAD_LAYOUT", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_SCENE", "PCORE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT",
"PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_BENCH", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES",
"PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_ASYNC", "REC_END", "REC_SPIN_US", "REEXEC",
"RENDER_SCALE", "REPLAY_ASYNC",
"RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHUNKS", "REPLAY_CHUNK_DRAWS", "REPLAY_SKEW_US", "REPLAY_STATE_CHECK", "REPLAY_WORKERS",
"REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES",
"ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE",
"SHADER_MEMO", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR",
@@ -95,7 +95,7 @@ static const struct { const char *key, *section; } sections[] = {
{"BIND_RIGHT", "input"}, {"BIND_DOWN", "input"}, {"BIND_LSTICK_LEFT", "input"}, {"BIND_LSTICK_UP", "input"},
{"BIND_LSTICK_RIGHT", "input"}, {"BIND_LSTICK_DOWN", "input"}, {"BIND_RSTICK_LEFT", "input"},
{"BIND_RSTICK_UP", "input"}, {"BIND_RSTICK_RIGHT", "input"}, {"BIND_RSTICK_DOWN", "input"}, {"BIND_SHAKE", "input"},
{"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"FREE_CORES", "performance"},
{"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"FREE_CORES", "performance"},
{"TIMER_1MS", "performance"}, {"PIPE_WAIT", "performance"}, {"PIPE_SKIP", "performance"},
{"LIB_PREBUILD", "performance"}, {"LIB_PREBUILD_THREADS", "performance"}, {"PIPE_PREBUILD", "performance"},
{"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"},
@@ -163,6 +163,9 @@ static const CfgMeta metas[] = {
{"DISCORD_IMAGE", "totk_512x512", NULL, LIVE},
// Advanced: scheduling
{"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART},
{"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_CHUNKS", "0", "0..64", RESTART},
{"REPLAY_CHUNK_DRAWS", "128", "1..100000", RESTART}, {"REPLAY_STATE_CHECK", "0", "0|1", RESTART},
{"REPLAY_SKEW_US", "0", "0..100000", RESTART},
{"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART},
// shaders and pipelines
{"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "fullscreen", "strict|any|fullscreen", RESTART},
+19
View File
@@ -51,6 +51,7 @@ extern "C" {
void rt_log(const char *fmt, ...) __attribute__((format(printf, 1, 2)));
void vk_queue_lock(void);
void vk_queue_unlock(void);
int vk_exec_replay_workers(void);
void res_sizes(unsigned out[2], unsigned scene[2]);
double frame_step_limit_hz(void);
extern const int rt_game_version;
@@ -905,6 +906,23 @@ static void row_threads(const char *key, UiTextId label, UiTextId desc) {
opt_choice_list(key, label, vals, names, N, desc);
}
// The chunked replay's recording threads: Off, then the counts; the note under the value is what this run uses, which
// a change replaces only after a restart.
static void row_replay_workers(void) {
static const char *const vals[] = {"0", "2", "3", "4", "6", "8", "12", "16"};
enum { N = sizeof vals / sizeof *vals };
const char *names[N];
names[0] = menu_text(UI_MENU_OFF);
for (int i = 1; i < N; i++) names[i] = vals[i];
static char now[64];
char n[16];
int w = vk_exec_replay_workers();
snprintf(n, sizeof n, "%d", w);
if (w >= 0) snprintf(now, sizeof now, menu_text(UI_MENU_REPLAY_WORKERS_NOW), w ? n : menu_text(UI_MENU_OFF));
opt_choice_list("TOTK_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, vals, names, N, UI_MENU_REPLAY_WORKERS_DESC, -1,
w >= 0 ? now : nullptr);
}
// A free-text key shown as Off (unset) / On (value set): On writes on_value, any set value reads as On.
static void row_set_or_not(const char *key, UiTextId label, UiTextId desc, const char *on_value) {
const char *vals[] = {"", on_value}, *names[] = {menu_text(UI_MENU_OFF), menu_text(UI_MENU_ON)};
@@ -954,6 +972,7 @@ static void tab_advanced(float scale) {
static const UiTextId preempt[] = {UI_MENU_PREEMPT_OFF, UI_MENU_PREEMPT_ON, UI_MENU_PREEMPT_INHERIT}; // 0|1|2
opt_choice("TOTK_ASYNC_PREEMPT", UI_MENU_ASYNC_PREEMPT, preempt, 3, UI_MENU_ASYNC_PREEMPT_DESC);
opt_toggle("TOTK_REPLAY_ASYNC", UI_MENU_REPLAY_ASYNC, UI_MENU_REPLAY_ASYNC_DESC);
row_replay_workers();
opt_toggle("TOTK_TIMER_1MS", UI_MENU_TIMER_1MS, UI_MENU_TIMER_1MS_DESC);
opt_group(UI_MENU_GROUP_SHADERS);
+13
View File
@@ -386,6 +386,9 @@ static const char *const text_en[UI_TEXT_COUNT] = {
[UI_MENU_PREEMPT_OFF] = "Off", [UI_MENU_PREEMPT_ON] = "On", [UI_MENU_PREEMPT_INHERIT] = "Inheritance only",
[UI_MENU_REPLAY_ASYNC] = "GPU frame prep on its own thread",
[UI_MENU_REPLAY_ASYNC_DESC] = "Prepares frames for the GPU on a separate thread instead of the game thread. Adds up to one frame of latency.",
[UI_MENU_REPLAY_WORKERS] = "GPU frame prep threads",
[UI_MENU_REPLAY_WORKERS_DESC] = "Records large frames for the GPU on several threads at once. Helps when the CPU limits the frame rate; needs 8 or more logical processors. Off records on one thread.",
[UI_MENU_REPLAY_WORKERS_NOW] = "now: %s",
[UI_MENU_TIMER_1MS] = "1 ms Windows timer",
[UI_MENU_TIMER_1MS_DESC] = "Raises the system timer resolution to 1 ms for steadier frame pacing.",
[UI_MENU_PIPE_WAIT] = "Waiting draw handling",
@@ -862,6 +865,9 @@ static const char *const text_ru[UI_TEXT_COUNT] = {
[UI_MENU_PREEMPT_OFF] = "Выкл.", [UI_MENU_PREEMPT_ON] = "Вкл.", [UI_MENU_PREEMPT_INHERIT] = "Только наследование",
[UI_MENU_REPLAY_ASYNC] = "Подготовка кадра для GPU в отдельном потоке",
[UI_MENU_REPLAY_ASYNC_DESC] = "Кадр для GPU готовится в отдельном потоке, а не в потоке игры. Добавляет до одного кадра задержки.",
[UI_MENU_REPLAY_WORKERS] = "Потоки подготовки кадра для GPU",
[UI_MENU_REPLAY_WORKERS_DESC] = "Большие кадры для GPU записываются сразу в нескольких потоках. Помогает, когда частоту кадров ограничивает процессор; нужно 8 логических процессоров или больше. «Выкл.» — запись в одном потоке.",
[UI_MENU_REPLAY_WORKERS_NOW] = "сейчас: %s",
[UI_MENU_TIMER_1MS] = "Таймер Windows 1 мс",
[UI_MENU_TIMER_1MS_DESC] = "Повышает точность системного таймера до 1 мс для ровного темпа кадров.",
[UI_MENU_PIPE_WAIT] = "Обработка ожидающих отрисовок",
@@ -1235,6 +1241,9 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = {
[UI_MENU_PREEMPT_OFF] = "Desligado", [UI_MENU_PREEMPT_ON] = "Ligado", [UI_MENU_PREEMPT_INHERIT] = "Só herança",
[UI_MENU_REPLAY_ASYNC] = "Preparo de quadro da GPU em thread própria",
[UI_MENU_REPLAY_ASYNC_DESC] = "Prepara os quadros para a GPU numa thread separada em vez da thread do jogo. Adiciona até um quadro de latência.",
[UI_MENU_REPLAY_WORKERS] = "Threads de preparo de quadro da GPU",
[UI_MENU_REPLAY_WORKERS_DESC] = "Grava quadros grandes para a GPU em várias threads ao mesmo tempo. Ajuda quando a CPU limita a taxa de quadros; precisa de 8 ou mais processadores lógicos. Desligado grava numa thread só.",
[UI_MENU_REPLAY_WORKERS_NOW] = "agora: %s",
[UI_MENU_TIMER_1MS] = "Timer do Windows de 1 ms",
[UI_MENU_TIMER_1MS_DESC] = "Aumenta a resolução do timer do sistema para 1 ms, para um ritmo de quadros mais estável.",
[UI_MENU_PIPE_WAIT] = "Tratamento de desenhos em espera",
@@ -1842,6 +1851,10 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = {
[UI_MENU_REPLAY_ASYNC] = "GPU 帧准备独立线程",
[UI_MENU_REPLAY_ASYNC_DESC] = "在单独的线程而不是游戏线程上为 GPU 准备帧。"
"最多增加一帧延迟。",
[UI_MENU_REPLAY_WORKERS] = "GPU 帧准备线程数",
[UI_MENU_REPLAY_WORKERS_DESC] = "同时在多个线程上为 GPU 录制大帧。CPU 限制帧率时有帮助;需要 8 个或更多逻辑处理器。"
"关:在单个线程上录制。",
[UI_MENU_REPLAY_WORKERS_NOW] = "当前:%s",
[UI_MENU_TIMER_1MS] = "1 ms Windows 计时器",
[UI_MENU_TIMER_1MS_DESC] = "将系统计时器精度提高到 1 ms,使帧间隔更平稳。",
[UI_MENU_PIPE_WAIT] = "等待中绘制的处理",
+1
View File
@@ -175,6 +175,7 @@ typedef enum {
UI_MENU_ASYNC_PREEMPT, UI_MENU_ASYNC_PREEMPT_DESC, UI_MENU_PREEMPT_OFF, UI_MENU_PREEMPT_ON,
UI_MENU_PREEMPT_INHERIT,
UI_MENU_REPLAY_ASYNC, UI_MENU_REPLAY_ASYNC_DESC,
UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, UI_MENU_REPLAY_WORKERS_NOW,
UI_MENU_TIMER_1MS, UI_MENU_TIMER_1MS_DESC,
UI_MENU_PIPE_WAIT, UI_MENU_PIPE_WAIT_DESC, UI_MENU_PIPE_WAIT_DEFAULT, UI_MENU_PIPE_WAIT_DRAW,
UI_MENU_PIPE_SKIP, UI_MENU_PIPE_SKIP_DESC, UI_MENU_PIPE_SKIP_STRICT, UI_MENU_PIPE_SKIP_ANY,
+29 -1
View File
@@ -226,6 +226,10 @@ enum { VK_MCLS_NONE, VK_MCLS_HOST_NOACC, VK_MCLS_HOST_UNC, VK_MCLS_HOST_CACHED,
enum { VK_MUSE_VB, VK_MUSE_IB, VK_MUSE_UBO, VK_MUSE_SSBO, VK_MUSE_IND, VK_MUSE_COPY_SRC, VK_MUSE_GPU_WRITE,
VK_MUSE_CPU_READ, VK_MUSE_N };
void vk_mem_count(int cls, int use, uint64_t bytes);
// A thread recording a chunk of a submission (vk_exec.c) counts into its own acc from vk_mem_count_local(acc) on, until
// vk_mem_count_local(NULL); vk_mem_count_flush adds acc into the census and clears it.
void vk_mem_count_local(int64_t (*acc)[VK_MUSE_N]);
void vk_mem_count_flush(int64_t (*acc)[VK_MUSE_N]);
// TOTK_TRACE=perf dirty census (vk_memory.c): the replay notes each imported pool it binds as a vertex, index, uniform,
// storage or indirect buffer; at the start of every submit the pages the CPU wrote in such pools since the previous
// submit are counted from the shared page generations and logged as `mem: dirty` every 60 frames. Nothing when tracing
@@ -258,10 +262,14 @@ void vk_mem_gpu_wrote_shader(const VkMemSpan *sp, uint64_t size);
// it records, ubo bindings take the copy (0: not copied, bind memory), GPU writes of the replay drop the copies they touch.
void vk_usnap_begin(const uint64_t *handles, int n, uint64_t fence);
void vk_usnap_end(uint64_t fence);
int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t align, VkDescriptorBufferInfo *out);
// at: the copies dropped at or before it are not found (vk_usnap_drop_at); UINT32_MAX outside a chunked replay.
int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t at, uint64_t align, VkDescriptorBufferInfo *out);
const uint8_t *vk_usnap_map(void); // a host copy of the current arena, at the offsets vk_usnap_find gives
const uint8_t *vk_usnap_bytes(uint64_t addr, uint64_t n); // the copy of [addr, addr+n) as of the submit, NULL: none
void vk_usnap_drop(uint64_t addr, uint64_t n); // the replay makes the GPU write [addr, addr+n) of guest memory
// The state scan of a chunked replay (vk_exec.c): the op that writes [addr, addr+n) is the at-th that may; the chunks
// pass at with the ops they record (VkExec.drop_n) and find the copies only before it.
void vk_usnap_drop_at(uint64_t addr, uint64_t n, uint32_t at);
uint64_t vk_usnap_gen(void); // changes whenever a binding may resolve differently
void vk_usnap_shutdown(void); // vk_exec_draw_shutdown: frees the arenas
// vk_mvprev.c: last frame's copy of the scene's object (c7) and bone (c5) blocks, for per-object motion vectors.
@@ -426,6 +434,7 @@ typedef struct {
void vk_hdr_present_source(const VkTexImage *win, VkImage *image, VkFormat *format);
void vk_hdr_image_gone(VkImage image);
const VkTexImage *vk_texture_get(const tex_t *t); // created and made current; NULL for BUFFER or unmapped formats
const VkTexImage *vk_texture_revive(const tex_t *t, uint32_t pos); // the entry vk_texture_get would return if it exists, made live at op pos
// A texture slot; a slot whose texture was finalized follows the live texture at the same storage (hle_nvn.c).
const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view);
// The replay looks up many slots per draw: it takes the texture lock once around them (vk_texture_lock) and uses the
@@ -433,6 +442,12 @@ const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view);
void vk_texture_lock(void);
void vk_texture_unlock(void);
const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view);
// 1 when slot id's entry had its freshness check in this submission, so vk_texture_by_id_locked would only look it up.
// No lock: a context of a chunked replay asks before taking the texture lock.
int vk_texture_slot_checked(int id);
// 1 when the state scan revived a texture at the slot's storage in this submission: the slot's answer depends on the op
// a chunk records, so no memo holds it (vk_exec_draw.c); a chunk sees every revival up to its last op.
int vk_texture_slot_moves(int id);
VkSampler vk_sampler_by_id_locked(int id);
// The slot's sampler with nearest filtering and no anisotropy: a depth texture read without a compare lands on one texel
// on the Switch although the game binds a linear sampler, and MacNativeGames samples it the same way. VK_NULL_HANDLE
@@ -495,9 +510,22 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t
// batch) without a signal, and returns a new begun one for the rest of the submission. The submission's timeline value
// is signalled by its last part, whose signal covers every earlier part in submission order.
VkCommandBuffer vk_sync_split(void);
// Inside vk_exec_record: submits cb, a part recorded and ended on another thread, after the upload batch and the
// mirror copies queued so far, without a signal (the submitting thread, in the order of the submission).
void vk_sync_part(VkCommandBuffer cb);
// GPU time of a part recorded on another thread (vk_sync.c "GPU time"): the start timestamp after vkBeginCommandBuffer,
// the pair or -1; the end before vkEndCommandBuffer.
int vk_sync_ts_open(VkCommandBuffer cb);
void vk_sync_ts_close(VkCommandBuffer cb, int pair, uint64_t fence);
// Runs fn(arg) on the completion thread once fence is done on the GPU (event writes, counter reports, sync state).
// Callbacks run in the order they were added, after those of earlier values; a value already done runs at once there.
void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg);
// Callbacks of a part recorded on another thread: between vk_completion_defer(d) and vk_completion_defer(NULL) the
// calling thread's vk_completion_add calls collect in d, and vk_completion_flush adds them, in their order, when the
// part takes its place in the submission (vk_exec.c record_chunks).
typedef struct { void *items; int n, cap; } VkDeferred;
void vk_completion_defer(VkDeferred *d);
void vk_completion_flush(VkDeferred *d);
// gfx_vulkan_backend (vk_backend.c): presents go to a present thread and run after the submissions made before them.
void rdoc_init(void); // runtime/rdoc.c: TOTK_RDOC debug capture; before vkCreateInstance
int rdoc_active(void);
+71 -48
View File
@@ -7,9 +7,13 @@
// passes behind nearby geometry: the small shadows at feet, under stones and in grass that the shadow maps are too
// coarse for. Transparent surfaces, the sky and post-processing come after, untouched.
//
// Inputs from the game, read from the fragment stage's uniform block c4 bound in the composite pass (guest memory,
// so it may be one frame newer than the replay): the projection scales c4[7].x and c4[8].y, the far plane c4[14].y,
// and the first sun cascade's depth row c4[44].xyz, whose negated direction points from a view-space position toward
// vk_contact_plan decides that in replay order and vk_contact_exec records it: a chunked replay's scan
// plans the march (vk_exec.c scan_contact) and the chunk that holds its op records it.
//
// Inputs from the game, read where the march is planned from the fragment stage's uniform block c4 bound in the
// composite pass (guest memory, so it may be one frame newer than the replay): the projection scales c4[7].x and
// c4[8].y, the far plane c4[14].y, and the first sun cascade's depth row c4[44].xyz, whose negated direction points from
// a view-space position toward
// the sun (the G-buffer shaders compare dot(c4[44].xyz, pos) + c4[44].w with the cascade map). The scene colour images
// get STORAGE usage when the setting is on (vk_texture.c create_tex).
#include "vk.h"
@@ -23,22 +27,22 @@
#include "vk_exec.h"
static struct {
int tried, ok, strength, debug, logged;
int read, tried, ok, strength, debug, logged;
PFN_vkCmdPushDescriptorSetKHR push;
VkDescriptorSetLayout dsl;
VkPipelineLayout layout;
VkPipeline pipe;
VkSampler sampler;
const VkTexImage *scene, *depth; // the last G-buffer's scene colour and linear depth
uint32_t w, h;
VkExecTarget scene, depth; // the last G-buffer's scene colour and linear depth
int pending; // the composite into scene ran; the march waits for the next pass
uint64_t c4;
} C;
int vk_contact_strength(void) {
if (!C.tried) {
if (!C.read) { // once: the scan asks at every draw
int s = cfg_int("TOTK_CONTACT_SHADOWS");
C.strength = s < 0 ? 0 : s > 100 ? 100 : s;
C.read = 1;
}
return C.strength;
}
@@ -90,16 +94,16 @@ static int init(VkCtx *c) {
}
// A G-buffer pass: a B10G11R11 scene colour, an R32F linear depth and a D32F depth, all one size, at least 640 wide.
static int is_gbuffer(const VkExec *ex, const VkTexImage **scene, const VkTexImage **depth) {
static int is_gbuffer(const VkExec *ex, const VkExecTarget **scene, const VkExecTarget **depth) {
*scene = *depth = NULL;
if (ex->pass_noatt || ex->pass_w < 640 || ex->depth.format != VK_FORMAT_D32_SFLOAT || ex->depth.level) return 0;
for (uint32_t i = 0; i < ex->pass_ncolor && i < 8; i++) {
const VkExecTarget *t = &ex->color[i];
if (!t->image || t->level || t->width != ex->pass_w || t->height != ex->pass_h) continue;
if (t->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32 && !*scene) *scene = t->img;
if (t->format == VK_FORMAT_R32_SFLOAT && !*depth) *depth = t->img;
if (t->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32 && !*scene) *scene = t;
if (t->format == VK_FORMAT_R32_SFLOAT && !*depth) *depth = t;
}
return *scene && *depth && ((*scene)->usage & VK_IMAGE_USAGE_STORAGE_BIT);
return *scene && *depth && (*scene)->img && ((*scene)->img->usage & VK_IMAGE_USAGE_STORAGE_BIT);
}
static void mem_barrier(VkCommandBuffer cb) {
@@ -112,28 +116,62 @@ static void mem_barrier(VkCommandBuffer cb) {
vkCmdPipelineBarrier2(cb, &d);
}
void vk_contact_flush(VkExec *ex) {
if (!C.pending) return;
// The lighting composite: the remembered scene colour as the only target, with a D16 depth of its size.
static int is_composite(const VkExec *ex) {
return C.scene.image && ex->pass_ncolor >= 1 && ex->color[0].image == C.scene.image && !ex->color[0].level &&
ex->depth.format == VK_FORMAT_D16_UNORM && ex->pass_w == C.scene.width && ex->pass_h == C.scene.height;
}
int vk_contact_plan(const VkExec *ex, int pass, VkExecJob *j) {
if (!vk_contact_strength()) return 0;
if (pass) {
const VkExecTarget *scene, *depth;
// The camera block: c4 of the fragment stage, 0x940 bytes. The composite binds it only after its pass begins, so
// the newest one seen at any pass start serves.
const p_bind_buf *b = &ex->ubo[1][1];
if (b->addr && b->size == 0x940) C.c4 = b->addr;
if (is_gbuffer(ex, &scene, &depth)) {
C.pending = 0;
C.scene = *scene, C.depth = *depth;
return 0;
}
if (is_composite(ex)) {
C.pending = C.c4 != 0;
return 0;
}
}
if (!C.pending) return 0;
C.pending = 0;
if (ex->in_pass) return;
const float *u = (const float *)(uintptr_t)C.c4;
float lx = -u[44 * 4 + 0], ly = -u[44 * 4 + 1], lz = -u[44 * 4 + 2], ll = sqrtf(lx * lx + ly * ly + lz * lz);
float px = u[7 * 4 + 0], py = u[8 * 4 + 1], zfar = u[14 * 4 + 1];
if (!(ll > 1e-6f) || !(px > 0) || !(py > 0) || !(zfar > 1)) return;
VkImageView in = vk_exec_target_view(C.depth, VK_FORMAT_R32_SFLOAT, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT,
if (!(ll > 1e-6f) || !(px > 0) || !(py > 0) || !(zfar > 1)) return 0;
float w = (float)C.scene.width, h = (float)C.scene.height;
*j = (VkExecJob){.scene = C.scene, .depth = C.depth,
.pc = {px, py, zfar, (float)C.strength / 100.0f,
lx / ll, ly / ll, lz / ll, 0.6f,
w, h, 1.0f / w, 1.0f / h,
0, 0, 0, 0}};
return 1;
}
void vk_contact_exec(VkExec *ex, const VkExecJob *j) {
if (!C.tried) init(vk_ctx());
if (!C.ok) return;
VkImageView in = vk_exec_target_view(j->depth.img, VK_FORMAT_R32_SFLOAT, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT,
0, 0, 1, 0);
VkImageView out = vk_exec_target_view(C.scene, VK_FORMAT_B10G11R11_UFLOAT_PACK32, VK_IMAGE_VIEW_TYPE_2D,
VkImageView out = vk_exec_target_view(j->scene.img, VK_FORMAT_B10G11R11_UFLOAT_PACK32, VK_IMAGE_VIEW_TYPE_2D,
VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1, 0);
if (!in || !out) return;
vk_exec_pass_end(ex);
VkCommandBuffer cb = ex->cb;
float pcv[16] = {px, py, zfar, (float)C.strength / 100.0f,
lx / ll, ly / ll, lz / ll, 0.6f,
(float)C.w, (float)C.h, 1.0f / (float)C.w, 1.0f / (float)C.h,
(float)C.debug, 0, 0, 0};
float pcv[16];
memcpy(pcv, j->pc, sizeof pcv);
pcv[12] = (float)C.debug;
if (!C.logged) {
C.logged = 1;
rt_log("vk: contact shadows on a %ux%u scene, sun toward %.3f %.3f %.3f (view space), far %g", C.w, C.h,
pcv[4], pcv[5], pcv[6], zfar);
rt_log("vk: contact shadows on a %ux%u scene, sun toward %.3f %.3f %.3f (view space), far %g", j->scene.width, j->scene.height,
pcv[4], pcv[5], pcv[6], pcv[2]);
}
mem_barrier(cb); // the compose passes' writes, before the march reads and writes them
VkDescriptorImageInfo ii = {C.sampler, in, VK_IMAGE_LAYOUT_GENERAL}, oi = {VK_NULL_HANDLE, out, VK_IMAGE_LAYOUT_GENERAL};
@@ -145,36 +183,21 @@ void vk_contact_flush(VkExec *ex) {
vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, C.pipe);
rec_push(C.push, cb, VK_PIPELINE_BIND_POINT_COMPUTE, C.layout, 0, 2, w);
vkCmdPushConstants(cb, C.layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof pcv, pcv);
vkCmdDispatch(cb, (C.w + 7) / 8, (C.h + 7) / 8, 1);
vkCmdDispatch(cb, (j->scene.width + 7) / 8, (j->scene.height + 7) / 8, 1);
mem_barrier(cb); // and what follows sees the darkened colour
ex->recorded += 6;
vk_exec_draw_new_cb(); // the game's bound compute pipeline and descriptors were replaced: bind them again
}
// The lighting composite: the remembered scene colour as the only target, with a D16 depth of its size.
static int is_composite(const VkExec *ex) {
return C.scene && ex->pass_ncolor >= 1 && ex->color[0].img == C.scene && !ex->color[0].level &&
ex->depth.format == VK_FORMAT_D16_UNORM && ex->pass_w == C.w && ex->pass_h == C.h;
vk_exec_draw_new_cb(ex); // the game's bound compute pipeline and descriptors were replaced: bind them again
}
// The sequential replay's hooks; a chunk leaves the marches to the scan.
void vk_contact_pass_begin(VkExec *ex) {
if (!C.tried && !init(vk_ctx())) return;
if (!C.ok) return;
const VkTexImage *scene, *depth;
// The camera block: c4 of the fragment stage, 0x940 bytes. The composite binds it only after its pass begins, so
// the newest one seen at any pass start serves.
const p_bind_buf *b = &ex->ubo[1][1];
if (b->addr && b->size == 0x940) C.c4 = b->addr;
if (is_gbuffer(ex, &scene, &depth)) {
C.pending = 0;
C.scene = scene, C.depth = depth, C.w = ex->pass_w, C.h = ex->pass_h;
return;
}
if (is_composite(ex)) {
C.pending = C.c4 != 0;
return;
}
vk_contact_flush(ex);
VkExecJob j;
if (!ex->chunked && vk_contact_plan(ex, 1, &j)) vk_contact_exec(ex, &j);
}
void vk_contact_flush(VkExec *ex) {
VkExecJob j;
if (!ex->chunked && C.pending && vk_contact_plan(ex, 0, &j)) vk_contact_exec(ex, &j);
}
void vk_contact_release(void) {
+3 -3
View File
@@ -91,7 +91,7 @@ void vk_crumbs_report(void) {
// records changed pipeline state also notes the shaders of a pass's first draw. A full barrier already precedes every
// pass and dispatch (vk_exec.c vk_exec_pass_begin, vk_exec_draw.c op_dispatch), so the end-of-pipe write adds no wait.
static VkPassMark ring[VK_PASS_MARKS];
static uint32_t seq; // marks made; only the replay thread writes it
static uint32_t seq; // marks made (atomic: the recording threads of a chunked replay)
// TOTK_PASS_CRUMBS: unset, buffer marks where VK_AMD_buffer_marker exists; 1, also NVIDIA checkpoints; 0, neither.
int vk_pass_marks_want_nv(void) {
@@ -124,8 +124,8 @@ void vk_pass_mark_cmd(VkCommandBuffer cb, uint32_t id, int end) {
VkPassMark *vk_pass_mark_begin(VkCommandBuffer cb, int kind, uint64_t fence) {
if (!vk_pass_marks_on || !cb) return NULL;
uint32_t id = ++seq;
if (!id) id = ++seq; // 0 means no mark
uint32_t id = __atomic_add_fetch(&seq, 1, __ATOMIC_RELAXED); // the recording threads of a chunked replay
if (!id) id = __atomic_add_fetch(&seq, 1, __ATOMIC_RELAXED); // 0 means no mark
VkPassMark *m = &ring[id & (VK_PASS_MARKS - 1)];
m->id = id, m->kind = (uint8_t)kind, m->fence = fence, m->draws = UINT32_MAX;
rec_pass_mark(cb, id, 0);
+88 -27
View File
@@ -243,6 +243,8 @@ static struct {
VkExecTarget color, depth; // the armed pair, valid in the frame it was armed
int armed, done, gbuf; // this frame: armed on the light target, DLAA ran, the G-buffer was drawn
int ended, up; // this frame: the scene is drawn (jitter over), DLSS wrote the scaled output
uint64_t end_fence; // a chunked replay: the submission and op the scan ended the scene at (ended_at)
uint32_t end_pos;
struct Size { // a DLSS feature and its work images per scene size (dynamic resolution flips it)
uint32_t w, h, ow, oh; // render size, output size
VkImage out, mv;
@@ -557,7 +559,7 @@ static int mv_pass(VkExec *ex, struct Size *e, VkImageView zv, float jx, float j
VkCommandBuffer cb = ex->cb;
uint32_t w = e->w, h = e->h;
VkDescriptorBufferInfo cam;
if (!mv_init() || !D.cam_addr || !vk_usnap_find(D.cam_addr, CAM_BLOCK, 4, &cam)) {
if (!mv_init() || !D.cam_addr || !vk_usnap_find(D.cam_addr, CAM_BLOCK, UINT32_MAX, 4, &cam)) {
D.mv_none++;
D.cam_prev_ok = 0;
e->chars_ok = 0;
@@ -777,10 +779,9 @@ static struct Snap *snap_for(VkCommandBuffer cb, uint32_t w, uint32_t h, VkForma
}
// At arming: the light target as it is before the transparents, outside any pass.
static void mask_snapshot(VkExec *ex) {
static void mask_snapshot(VkExec *ex, const VkExecTarget *t) {
R.cur = NULL;
if (mask_init() <= 0) return;
const VkExecTarget *t = &D.color;
struct Snap *s = snap_for(ex->cb, t->width, t->height, t->format);
if (!s) return;
vk_exec_barrier(ex);
@@ -829,14 +830,33 @@ static float halton(uint32_t i, uint32_t b) {
return r;
}
// The scene is drawn at the op ex records. A chunked replay's scan ends it ahead of the chunks that still record the
// scene's draws, so a chunk compares its op with the scan's (the chunks after it are taken once the scan is past their
// ops); a later submission of the frame is past it.
static int ended_at(const VkExec *ex) {
if (!__atomic_load_n(&D.ended, __ATOMIC_ACQUIRE)) return 0;
return !ex->chunked || ex->fence != D.end_fence || ex->pos >= D.end_pos;
}
void vk_dlaa_viewport(VkExec *ex, VkViewport *v) {
if (!D.jit_ok || D.ended || !ex->depth.image || ex->depth.image != D.jdepth) return;
if (!D.jit_ok || !ex->depth.image || ex->depth.image != D.jdepth || ended_at(ex)) return;
v->x += D.jx, v->y += D.jy;
D.ux = D.jx, D.uy = D.jy, D.jit_used = 1;
D.ux = D.jx, D.uy = D.jy;
__atomic_store_n(&D.jit_used, 1, __ATOMIC_RELAXED);
}
// The scene camera: c4 of the last scene draw before the scene ends. A chunked replay's scan notes it in replay order
// (vk_dlaa_scan_draw): chunks recording side by side would leave the block of whichever drew last.
static void note_camera(const VkExec *ex) {
const p_bind_buf *b = &ex->ubo[0][1]; // constant bank c4: the replay keeps user banks from c3 on
const VkExecTarget *z = &ex->depth;
if (b->size == 0x940 && b->addr && z->image && z->width >= DLAA_SCENE_MIN_W && z->format == VK_FORMAT_D32_SFLOAT &&
(!D.jdepth || z->image == D.jdepth))
D.cam_addr = b->addr;
}
void vk_dlaa_draw(VkExec *ex) {
if (!dl_active() || D.ended) return;
if (!dl_active() || ended_at(ex)) return;
static int survey = -1; // TOTK_DLAA_SURVEY=1 (test): what is drawn into the armed light target
if (survey < 0) survey = rt_knob("DLAA_SURVEY", "0");
static uint64_t dfn[16]; // scene draws writing depth, by depth test (reversed depth: GREATER/GEQUAL)
@@ -867,11 +887,11 @@ void vk_dlaa_draw(VkExec *ex) {
}
}
}
const p_bind_buf *b = &ex->ubo[0][1]; // constant bank c4: the replay keeps user banks from c3 on
const VkExecTarget *z = &ex->depth;
if (b->size == 0x940 && b->addr && z->image && z->width >= DLAA_SCENE_MIN_W && z->format == VK_FORMAT_D32_SFLOAT &&
(!D.jdepth || z->image == D.jdepth))
D.cam_addr = b->addr;
if (!ex->chunked) note_camera(ex);
}
void vk_dlaa_scan_draw(const VkExec *ex) {
if (dl_active() && !D.ended) note_camera(ex);
}
// The jitter moves on right after DLAA ran, not at the present: the replay can record the next frame's scene before
@@ -1045,7 +1065,7 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) {
full_barrier(cb);
if (dst != t) vk_dlaa_swap_img = t->img, vk_dlaa_swap_view = e->out_view; // the scaling draw reads DLSS's picture
ex->recorded += 4;
vk_exec_draw_new_cb(); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again
vk_exec_draw_new_cb(ex); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again
D.jdepth = z->image;
next_jitter(w, ow);
if (++D.evals == 1 || D.evals == 300 || D.evals % 3600 == 0) // 300: one early line with the frames counted
@@ -1062,9 +1082,9 @@ static int scene_color(VkFormat f) {
return f == VK_FORMAT_B10G11R11_UFLOAT_PACK32 || f == VK_FORMAT_R16G16B16A16_SFLOAT;
}
void vk_dlaa_pass(VkExec *ex) {
int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) {
const VkExecTarget *c = &ex->color[0], *z = &ex->depth;
if (!dl_active() || D.done) return;
if (!dl_active() || D.done) return 0;
// The G-buffer: several colour targets with a D32F depth of at least DLAA_SCENE_MIN_W wide. Only the light target drawn after it
// arms the frame; at night the game draws into the light target alone with the scene depth before the G-buffer too,
// and DLAA then ran at a small pass ahead of the whole scene, which reached the screen jittered.
@@ -1075,7 +1095,7 @@ void vk_dlaa_pass(VkExec *ex) {
if (c->image && c->format == VK_FORMAT_R8G8B8A8_UNORM && c->width == z->width && c->height == z->height &&
c->samples <= 1 && !c->is3d && (!D.ids_ok || c->width >= D.ids.width))
D.ids = *c, D.ids_ok = 1;
return;
return 0;
}
// the scene depth is D32F; a D16 target of the scene size also serves one composite pass before the transparents.
// A smaller pair never takes over an armed one: the camera rune draws a half-size RGBA16F target with a half-size
@@ -1087,26 +1107,56 @@ void vk_dlaa_pass(VkExec *ex) {
c->samples <= 1 && !c->is3d) {
int first = !D.armed;
D.color = *c, D.depth = *z, D.armed = 1;
if (first) mask_snapshot(ex); // the lit scene, before the transparents (reactive mask)
return;
if (!first) return 0;
*j = (VkExecJob){.kind = VK_JOB_DLSS_MASK, .scene = *c}; // the lit scene, before the transparents (reactive mask)
return 1;
}
if (!D.armed) return;
if (!D.armed) return 0;
if (!D.ended) {
if (!ex->pass_w || ex->pass_w * 4 > D.color.width) return;
D.ended = 1;
if (!ex->pass_w || ex->pass_w * 4 > D.color.width) return 0;
D.end_fence = ex->fence, D.end_pos = ex->pos;
__atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE);
unsigned o[2], s[2];
res_sizes(o, s);
if (D.color.width >= o[0]) D.done = 1, evaluate(ex, NULL); // DLAA in place; else wait for the scaling pass
return;
if (D.color.width < o[0]) return 0; // wait for the scaling pass
D.done = 1;
*j = (VkExecJob){.kind = VK_JOB_DLSS}; // DLAA in place
return 1;
}
// the game's scaling pass: one colour target of the output size, no depth
unsigned o[2], s[2];
res_sizes(o, s);
if (ex->pass_ncolor == 1 && c->image && !z->image && scene_color(c->format) && c->width > D.color.width &&
c->width == o[0] && c->height == o[1] && c->samples <= 1 && !c->is3d) {
D.done = D.up = 1;
evaluate(ex, c);
D.done = 1;
*j = (VkExecJob){.kind = VK_JOB_DLSS, .scene = *c};
return 1;
}
return 0;
}
// The scan resolved its targets without making entries (vk_texture_revive); a record made on a chunk takes their images
// now, as the scan's pointers may be stale by then.
static void target_now(VkExecTarget *t) {
const VkTexImage *i = t->tex ? vk_texture_get(t->tex) : NULL;
if (i) t->img = i, t->image = i->image;
}
void vk_dlaa_exec(VkExec *ex, const VkExecJob *j) {
VkExecTarget t = j->scene;
if (ex->chunked) {
target_now(&t), target_now(&D.color), target_now(&D.depth);
if (D.ids_ok) target_now(&D.ids);
}
if (j->kind == VK_JOB_DLSS_MASK) return mask_snapshot(ex, &t);
if (!t.tex) return evaluate(ex, NULL);
D.up = 1;
evaluate(ex, &t);
}
void vk_dlaa_pass(VkExec *ex) {
VkExecJob j;
if (!ex->chunked && vk_dlaa_plan(ex, &j)) vk_dlaa_exec(ex, &j);
}
// TOTK_DLSS, read at each present (the menu switches it live): the scene size of the mode (resolution.c) and the
@@ -1178,18 +1228,29 @@ void vk_dlaa_frame(void) {
static const uint64_t FXAA_FS[] = {0xf450003ebeb3dc52ull, 0x0a12f4bc2a9a4fbeull, 0x5c43ae18948d52cbull,
0xaad42238801d31d8ull};
// Chunks of one submission look shaders up side by side: an entry's hash is stored before its shader, and entries are
// added under the lock.
static uint64_t spirv_hash(const GfxShader *s) {
static struct { const GfxShader *s; uint64_t h; } tab[8192];
static PlatMutex lk = PLAT_MUTEX_INIT;
uintptr_t k = ((uintptr_t)s >> 4) * 0x9E3779B97F4A7C15ull;
for (uint32_t i = 0, j = (uint32_t)(k >> 51); i < 8192; i++, j = (j + 1) & 8191) {
if (tab[j].s == s) return tab[j].h;
if (!tab[j].s) {
const GfxShader *e = __atomic_load_n(&tab[j].s, __ATOMIC_ACQUIRE);
if (e == s) return tab[j].h;
if (e) continue;
plat_mutex_lock(&lk);
e = tab[j].s;
if (!e) {
uint64_t words = 0, h = 0xcbf29ce484222325ull;
const uint8_t *b = (const uint8_t *)gfx_shader_spirv(s, &words);
for (uint64_t n = 0; b && n < words * 4; n++) h = (h ^ b[n]) * 0x100000001b3ull;
tab[j].s = s, tab[j].h = h;
tab[j].h = h;
__atomic_store_n(&tab[j].s, s, __ATOMIC_RELEASE);
plat_mutex_unlock(&lk);
return h;
}
plat_mutex_unlock(&lk);
if (e == s) return tab[j].h;
}
return 0; // table full: no skip
}
+6
View File
@@ -18,11 +18,17 @@ int vk_dlaa_available(void);
// scene's light target or, at the first quarter-width pass after it, runs DLAA on that target. Each present ends the
// frame (vk_dlaa_frame).
typedef struct VkExec VkExec;
typedef struct VkExecJob VkExecJob;
void vk_dlaa_pass(VkExec *ex);
// The same decision for a chunked replay's scan, in replay order: 1 with the record due before this pass (j), which
// vk_dlaa_exec makes on the chunk that holds the op (vk_exec.c run_job).
int vk_dlaa_plan(const VkExec *ex, VkExecJob *j);
void vk_dlaa_exec(VkExec *ex, const VkExecJob *j);
void vk_dlaa_frame(void);
// vk_exec_draw.c, each draw before its fixed state is recorded: notes the scene camera block (vk_dlaa_draw), and moves
// the viewport of the scene's draws by the frame's jitter (vk_dlaa_viewport, when the viewport is recorded).
void vk_dlaa_draw(VkExec *ex);
void vk_dlaa_scan_draw(const VkExec *ex); // a chunked replay's scan, at each draw: the camera note of vk_dlaa_draw
void vk_dlaa_viewport(VkExec *ex, VkViewport *v);
// vk_mvsurvey.c (TOTK_MV_SURVEY): notes the uniform blocks of each scene draw, beside vk_dlaa_draw; each present
// compares them with the previous frames' and logs a summary now and then.
+842 -108
View File
File diff suppressed because it is too large. Load diff
+94 -13
View File
@@ -2,10 +2,12 @@
// handles of a submission, keeps render targets, passes, clears and fixed-function state; draws, binds and dispatches
// (vk_exec_draw.c) and copies, counters, events and conditional rendering (vk_exec_xfer.c) plug in through the two
// hooks at the bottom.
// Runs on the submitting thread under vk_sync_submit's lock: no locking inside.
// Runs on the submitting thread under vk_sync_submit's lock; a chunked replay (TOTK_REPLAY_WORKERS, vk_exec.c) records
// on several threads, each with a VkExec of its own, and shares only the locked tables named where they are declared.
// The op loop follows MacNativeGames nvn_mtl.m (MIT, Copyright (c) 2026 jhaizhou) and the pass handling LAS Recomp
// (MIT, Copyright (c) 2026 Peiikai); vk_exec.c names the sources.
#pragma once
#include <stddef.h>
#include <stdint.h>
#include "nvn_api.h"
@@ -38,8 +40,36 @@ typedef struct {
uint8_t scale; // VkTexImage.scale
} VkExecTarget;
// The replay state. One instance lives for the run: NVN state is queue state and carries over from one submission to
// the next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record.
// TOTK_TRACE=gfx: replay counters of the current frame, logged and reset at each present (vk_exec_frame_log).
typedef struct {
uint64_t passes, draws_seen, draws, clears, copies, dispatches, submits;
uint64_t fb_draws, fb_barriers, fb_restarts, fb_layouts; // feedback loops: draws sampling a written target of
// their pass, in-pass barriers, pass restarts, layout
// transitions
} VkExecFrame;
// A record the scan of a chunked replay planned before an op: a contact shadow march (vk_contact.c: vk_contact_plan
// decides it in replay order, vk_contact_exec records it), or DLSS's copy of the lit scene or its evaluation
// (vk_dlaa.c: vk_dlaa_plan, vk_dlaa_exec). The chunk that holds the op records it.
enum { VK_JOB_CONTACT, VK_JOB_DLSS_MASK, VK_JOB_DLSS };
typedef struct VkExecJob {
const void *op; // vk_exec.c: the op it precedes in a scanned submission
int kind; // VK_JOB_*
int chunk; // vk_exec.c: the chunk that holds the op
VkExecTarget scene, depth; // contact: the scene colour and the linear depth; DLSS: the light target to copy
// (mask) or the scaling pass's target (evaluation; no tex: in place)
float pc[16]; // the march's push constants
} VkExecJob;
// An OP_UPDATE_UB of the submission: [lo, hi) of guest memory takes data (vk_exec_draw.c ubo_ptr reads them).
typedef struct { uint64_t lo, hi; const uint8_t *data; } VkExecUpd;
struct VkExecDraw; // vk_exec_draw.c: the memos and command buffer state of one recording context
// The replay state and one recording context. NVN state is queue state and carries over from one submission to the
// next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record.
// The fields from rt up to views are that queue state (VK_EXEC_STATE_BEGIN/END); the rest belongs to the context that
// records with it.
typedef struct VkExec {
// ---- per command buffer
VkCommandBuffer cb; // being recorded
@@ -94,8 +124,14 @@ typedef struct VkExec {
int render_disabled; // OP_RENDER_ENABLE false or a failed OP_RENDER_COND: clears and draws skip
p_render_cond render_cond; // last OP_RENDER_COND
int has_render_cond;
// samples-passed counting (vk_exec_xfer.c): a reset since the last report, a report since the last reset, the game
// reported twice without a reset (every draw counts from then on)
int oq_counting, oq_reported, oq_keep;
// ---- internal to vk_exec.c
VkExecFrame fr; // counters of this context, added to vk_exec_frame at the end of a submission
int split_draws, part_start; // draws and dispatches since the last part; recorded at its start
uint64_t part_t0; // replay time at its start
void *views; // image views created for this command buffer (freed at its fence)
int nviews, cap_views;
int barrier_at; // recorded right after the last full barrier (-1: none yet)
@@ -112,8 +148,30 @@ typedef struct VkExec {
const void *fb_fs; // feedback_scan's memo: fragment shader, texture bind serial, pass serial
uint32_t fb_serial, fb_pass, fb_sampled;
uint32_t tex_serial, pass_serial; // bumped by every texture or image bind and pass begin
// ---- vk_exec_xfer.c: the occlusion query of the open pass and the queries closed since the last report or reset
int oq_open; // slot of the active query, -1 none
uint32_t *oq_seg, oq_nseg, oq_capseg;
int oq_lost; // the segment missed a draw (ring full)
struct VkExecDraw *draw; // vk_exec_draw.c, made on the context's first draw op
// ---- a chunk of a scanned submission (vk_exec.c): the scan's answers, taken in op order
int chunked; // records a chunk: what depends on earlier chunks comes from the scan
const VkExecUpd *upd; // the submission's inline uniform updates
const uint8_t *cond; // its RENDER_COND results (1 renders)
const VkExecJob *jobs; // its planned records (contact shadows, DLSS)
uint32_t njobs;
uint32_t upd_n, cond_i, job_i; // updates, results and records before the op being recorded
uint32_t drop_n; // ops before it that may write uniform ranges (vk_usnap_drop_at)
const uint32_t *wpos; // the positions of the draws and dispatches among those ops (vk_exec_draw_writes)
uint32_t wpos_i, wpos_end; // the next of them, and the end of the chunk's
uint32_t pos; // ops of the submission up to and with it (the scan's too)
} VkExec;
#define VK_EXEC_STATE_BEGIN offsetof(VkExec, rt)
#define VK_EXEC_STATE_END offsetof(VkExec, fr)
// Begins dynamic rendering with the current targets unless a pass is open (a full memory barrier precedes it). Call
// before a draw or an attachment clear.
void vk_exec_pass_begin(VkExec *ex);
@@ -136,9 +194,16 @@ VkCommandBuffer vk_exec_inline_begin(int *reopen);
void vk_exec_inline_end(int reopen);
// vk_exec_draw.c: forget what the command buffer being recorded has bound (pipeline, sets, dynamic state, vertex
// buffers): the replay continues in a new one (vk_sync_split).
void vk_exec_draw_new_cb(void);
void vk_exec_draw_new_cb(VkExec *ex);
// The recording threads of the chunked replay as this run read TOTK_REPLAY_WORKERS (0 off), -1 before the first
// submission read it (the settings menu shows it).
int vk_exec_replay_workers(void);
// vk_exec_draw.c, for the state scan of a chunked replay: the storage ranges the shaders of a draw (dispatch 0) or a
// dispatch at state s write, as {address, size} into out; their count, or -1 when there are more than max or a stage's
// shader is not known yet (no context has translated it). Reads uniform blocks through s->upd like a chunk.
int vk_exec_draw_writes(const VkExec *s, int dispatch, uint64_t (*out)[2], int max);
// vk_exec_draw.c: the vertex and fragment shader of the last draw recorded into this command buffer (NULL: none yet).
void vk_exec_draw_shaders(const GfxShader **vs, const GfxShader **fs);
void vk_exec_draw_shaders(const VkExec *ex, const GfxShader **vs, const GfxShader **fs);
VkBuffer vk_exec_inline_buffer(VkDeviceSize size);
// vk_exec.c: the stand-in a 3D BC render target is drawn through goes with its image (vk_exec_image_gone).
void vk_exec_bc_gone(VkImage image);
@@ -155,16 +220,15 @@ int vk_exec_targets_recent(VkExec *ex, int any);
// The current pass's targets with their formats, extents and the frames they were last rendered in (diagnostics).
void vk_exec_targets_describe(VkExec *ex, char *buf, size_t cap);
// TOTK_TRACE=gfx: replay counters of the current frame, logged and reset at each present (vk_exec_frame_log).
typedef struct {
uint64_t passes, draws_seen, draws, clears, copies, dispatches, submits;
uint64_t fb_draws, fb_barriers, fb_restarts, fb_layouts; // feedback loops: draws sampling a written target of
// their pass, in-pass barriers, pass restarts, layout
// transitions
} VkExecFrame;
extern VkExecFrame vk_exec_frame;
void vk_exec_frame_written(const tex_t *t); // vk_texture_gpu_written: t rendered or copied into
uint64_t vk_exec_recording_fence(void); // fence of the submission this thread records, else 0
uint32_t vk_exec_replay_pos(void); // the op this thread records in a chunk, else UINT32_MAX
// A chunk that is about to make a texture entry whose content depends on the textures at its storage (vk_texture.c
// entry_of): whether a chunk before it is still being recorded, and the wait until none is. The caller holds no lock
// but the texture lock, which it releases for the wait.
int vk_exec_chunk_behind(void);
void vk_exec_chunk_settle(void);
void vk_exec_frame_log(const tex_t *presented, uint64_t frame);
// TOTK_DUMP=<frame> (vk_dump.c): between presents frame-1 and frame, every pass's targets and every dispatch's storage
@@ -188,6 +252,7 @@ void vk_c0_fill(uint8_t bank[VK_C0_BYTES], int stage, const p_bind_buf ssbo[16],
// vk_exec_draw.c: OP_DRAW, OP_DRAW_IDX, OP_DISPATCH, OP_BIND_* (program, state, vattr, vstream, vbuf, ubo, ssbo, tex,
// samp, img), OP_UPDATE_UB, OP_DRAW_IDX_IND, OP_DISPATCH_IND. payload: nvn_op_payload of the record.
void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload);
int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload); // a bind op's writes into ex; 1: VkExecDraw.bind_gen
// The aspect a sampled view of img reads (vk_exec_draw.c get_view): stencil for a view in STENCIL depth-stencil mode.
VkImageAspectFlags vk_exec_view_aspect(const VkTexImage *img, const view_t *v, int hv);
VkImageView vk_exec_target_view(const VkTexImage *img, VkFormat format, VkImageViewType type, uint32_t aspect,
@@ -199,10 +264,26 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload);
// sample DLSS's output vk_dlaa_swap_view in its place, so the game's scaling draw scales DLSS's picture.
extern const VkTexImage *vk_dlaa_swap_img;
extern VkImageView vk_dlaa_swap_view;
// vk_contact.c: a pass about to begin (its targets resolved, no pass open), and a dispatch or copy about to be recorded
// The state scan's part of the transfer ops: render enable, conditional rendering (returns the condition: 1 renders),
// samples-passed counting state and the pending reports conditions read. Returns 1 for every other op.
// The state scan of a chunked replay (vk_exec.c): the textures op uses go live in replay order (vk_texture_revive); pos:
// the op's place in the submission. 1 when one of them has no entry yet.
int vk_exec_xfer_revive(uint32_t op, const void *payload, uint32_t pos);
int vk_exec_xfer_scan(VkExec *ex, uint32_t op, const void *payload);
// The guest memory a COPY_B2B, COPY_T2B or CLEAR_BUF op makes the GPU write, as far as the scan can tell without the
// texture's image: [*addr, *addr + return value).
uint64_t vk_exec_xfer_written(const VkExec *ex, uint32_t op, const void *payload, uint64_t *addr);
// vk_contact.c: a pass about to begin (its targets resolved, no pass open), and a dispatch or copy about to be recorded.
// vk_contact_plan is what they do in replay order (pass: the first; else the second), for the scan; 1: a march is due (j).
void vk_contact_pass_begin(VkExec *ex);
void vk_contact_flush(VkExec *ex);
int vk_contact_plan(const VkExec *ex, int pass, VkExecJob *j);
void vk_contact_exec(VkExec *ex, const VkExecJob *j);
// vk_exec_xfer.c, samples-passed counting: a draw about to be recorded in the open pass (begins an occlusion query
// when counting and none is active), and the open pass about to end (ends the active query).
void vk_exec_oq_draw(VkExec *ex);
void vk_exec_oq_pass_end(VkExec *ex);
void vk_exec_oq_forget(VkExec *ex);
void vk_exec_oq_flush(VkExec *ex); // the end of a chunk: its closed queries counted as a segment without a report // vk_exec_xfer_shutdown: the context's queries belonged to the old pool
// vk_exec.c: fn on every recording context (teardown, the device idle).
void vk_exec_contexts(void (*fn)(VkExec *ex));
+615 -352
View File
File diff suppressed because it is too large. Load diff
+114 -38
View File
@@ -72,7 +72,7 @@ static void guest_written_cb(void *arg) {
// The GPU wrote [addr, addr+size) of CPU-visible memory: tell the texture cache once the fence is reached.
static void note_gpu_write(VkExec *ex, uint64_t addr, uint64_t size) {
vk_usnap_drop(addr, size);
if (!ex->chunked) vk_usnap_drop(addr, size); // a chunk: the scan dropped it in its place
vk_exec_ub_foreign(ex, addr, size);
if (nvn_gpu_only(addr)) return;
Range *r = malloc(sizeof *r);
@@ -129,13 +129,6 @@ static int oq_state; // 0 not set up, 1 counting available,
static VkQueryControlFlags oq_flags;
static volatile uint8_t oq_busy[OQ_N]; // from begin until the completion thread read and reset the slot
static uint32_t oq_next; // replay: next ring slot
static int oq_counting; // replay: draws are counted (a reset since the last report, or oq_keep)
static int oq_reported; // replay: a samples-passed report since the last reset
static int oq_keep; // replay: the game reported twice without a reset: count every draw
static int oq_open = -1; // replay: the slot whose query is active in the open pass
static uint32_t *oq_seg; // replay: slots closed since the last report or reset
static uint32_t oq_nseg, oq_capseg;
static int oq_lost; // replay: the segment missed a draw (ring full)
static uint64_t oq_count; // completion thread: samples since the last reset
typedef struct { uint64_t addr; int reset, lost; uint32_t n; uint32_t q[]; } OqSegment;
@@ -166,39 +159,49 @@ static int oq_ready(void) {
}
static void oq_close(VkExec *ex) {
if (oq_open < 0) return;
vkCmdEndQuery(ex->cb, oq_pool, (uint32_t)oq_open);
if (ex->oq_open < 0) return;
vkCmdEndQuery(ex->cb, oq_pool, (uint32_t)ex->oq_open);
ex->recorded++;
if (oq_nseg == oq_capseg) {
oq_capseg = oq_capseg ? oq_capseg * 2 : 64;
oq_seg = realloc(oq_seg, sizeof *oq_seg * oq_capseg);
if (!oq_seg) rt_fatal("vk: out of memory");
if (ex->oq_nseg == ex->oq_capseg) {
ex->oq_capseg = ex->oq_capseg ? ex->oq_capseg * 2 : 64;
ex->oq_seg = realloc(ex->oq_seg, sizeof *ex->oq_seg * ex->oq_capseg);
if (!ex->oq_seg) rt_fatal("vk: out of memory");
}
oq_seg[oq_nseg++] = (uint32_t)oq_open;
oq_open = -1;
ex->oq_seg[ex->oq_nseg++] = (uint32_t)ex->oq_open;
ex->oq_open = -1;
}
void vk_exec_oq_draw(VkExec *ex) {
if (!oq_counting || oq_open >= 0) return;
uint32_t q = oq_next % OQ_N;
if (__atomic_load_n(&oq_busy[q], __ATOMIC_ACQUIRE)) {
if (!ex->oq_counting || ex->oq_open >= 0) return;
uint32_t q = __atomic_fetch_add(&oq_next, 1, __ATOMIC_RELAXED) % OQ_N; // threads recording chunks take slots too
if (__atomic_exchange_n(&oq_busy[q], 1, __ATOMIC_ACQUIRE)) {
warn_once(W_QUERY, "occlusion query ring full: draws left uncounted");
oq_lost = 1;
ex->oq_lost = 1;
return;
}
oq_busy[q] = 1;
oq_next++;
vkCmdBeginQuery(ex->cb, oq_pool, q, oq_flags);
ex->recorded++;
oq_open = (int)q;
ex->oq_open = (int)q;
}
void vk_exec_oq_pass_end(VkExec *ex) { oq_close(ex); }
static void oq_segment(VkExec *ex, uint64_t addr, int reset);
// The queries a chunk closed: the completion thread adds them to the count before the next chunk's (vk_exec.c).
void vk_exec_oq_flush(VkExec *ex) {
if (ex->oq_nseg || ex->oq_lost) oq_segment(ex, 0, 0);
}
void vk_exec_oq_forget(VkExec *ex) {
ex->oq_counting = ex->oq_reported = ex->oq_keep = 0, ex->oq_open = -1;
ex->oq_nseg = 0, ex->oq_lost = 0;
}
void vk_exec_xfer_shutdown(void) {
if (oq_state > 0) vkDestroyQueryPool(vk_ctx()->device, oq_pool, NULL);
oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_counting = oq_reported = oq_keep = 0, oq_open = -1;
oq_nseg = 0, oq_lost = 0, oq_count = 0;
oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_count = 0;
vk_exec_contexts(vk_exec_oq_forget);
memset((void *)oq_busy, 0, sizeof oq_busy);
}
@@ -240,30 +243,34 @@ static void oq_cb(void *arg) {
// Hands the segment to the completion thread with a report (addr; 0: no CPU-visible memory) or a reset.
static void oq_segment(VkExec *ex, uint64_t addr, int reset) {
oq_close(ex);
OqSegment *s = malloc(sizeof *s + sizeof *s->q * oq_nseg);
OqSegment *s = malloc(sizeof *s + sizeof *s->q * ex->oq_nseg);
if (!s) rt_fatal("vk: out of memory");
s->addr = addr, s->reset = reset, s->lost = oq_lost, s->n = oq_nseg;
memcpy(s->q, oq_seg, sizeof *s->q * oq_nseg);
oq_nseg = 0, oq_lost = 0;
s->addr = addr, s->reset = reset, s->lost = ex->oq_lost, s->n = ex->oq_nseg;
memcpy(s->q, ex->oq_seg, sizeof *s->q * ex->oq_nseg);
ex->oq_nseg = 0, ex->oq_lost = 0;
vk_completion_add(ex->fence, oq_cb, s);
}
static void reset_counter(VkExec *ex, const p_reset_counter *p) {
if (p->type != NVN_COUNTER_TYPE_SAMPLES_PASSED || !oq_ready()) return;
oq_segment(ex, 0, 1);
oq_counting = 1, oq_reported = 0;
ex->oq_counting = 1, ex->oq_reported = 0;
}
// A samples-passed report: the segment goes with it (addr 0: no CPU-visible memory) and counting stops until the next
// reset, unless reports came without resets before.
static void oq_report_state(VkExec *ex) {
if (ex->oq_reported && !ex->oq_keep) {
ex->oq_keep = 1;
LOG_ONCE("vk: samples passed reported twice without a reset: counting every draw from now on");
}
ex->oq_reported = 1;
if (!ex->oq_keep) ex->oq_counting = 0;
}
static void oq_report(VkExec *ex, uint64_t addr) {
oq_segment(ex, addr, 0);
if (oq_reported && !oq_keep) {
oq_keep = 1;
rt_log("vk: samples passed reported twice without a reset: counting every draw from now on");
}
oq_reported = 1;
if (!oq_keep) oq_counting = 0;
oq_report_state(ex);
}
static void signal_event(VkExec *ex, const p_signal_event *p) {
@@ -290,7 +297,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) {
*a = p->addr;
vk_completion_add(ex->fence, counter_cb, a);
}
pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples};
if (!ex->chunked) pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; // else the scan's
}
// ---- conditional rendering
@@ -612,8 +619,77 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload) {
case OP_RENDER_COND:
ex->render_cond = *(const p_render_cond *)payload;
ex->has_render_cond = 1;
ex->render_disabled = !cond_enabled(payload);
ex->render_disabled = !(ex->chunked ? ex->cond[ex->cond_i++] : cond_enabled(payload));
break;
default: rt_fatal("vk: op %s is not a transfer", nvn_op_name(op));
}
}
static int revive_new(const tex_t *t, uint32_t pos) { return t && t->mem && !vk_texture_revive(t, pos); }
int vk_exec_xfer_revive(uint32_t op, const void *payload, uint32_t pos) {
if (op == OP_COPY_T2T) {
int n = revive_new(((const p_copy_t2t *)payload)->src, pos);
return revive_new(((const p_copy_t2t *)payload)->dst, pos) | n;
}
if (op == OP_COPY_B2T || op == OP_COPY_T2B) return revive_new(((const p_copy_b2t *)payload)->tex, pos);
if (op == OP_CLEAR_TEX) return revive_new(((const p_clear_tex *)payload)->tex, pos);
return 0;
}
uint64_t vk_exec_xfer_written(const VkExec *ex, uint32_t op, const void *payload, uint64_t *addr) {
VkMemSpan m;
if (op == OP_COPY_B2B) {
const p_copy_b2b *p = payload;
*addr = p->dst;
return vk_mem_resolve(p->src, 1, &m) && vk_mem_resolve(p->dst, 1, &m) ? p->size : 0;
}
if (op == OP_CLEAR_BUF) {
const p_clear_buf *p = payload;
*addr = p->addr;
return (p->addr | p->size) & 3 || !vk_mem_resolve(p->addr, 1, &m) ? 0 : p->size;
}
// COPY_T2B: the buffer side of copy_buffer_image from the texture descriptor (sub_resolve, buffer_side)
const p_copy_b2t *p = payload;
const tex_t *t = p->tex;
int bw, bh, bpb = t ? nvn_fmt_bpb(t->d.format, &bw, &bh) : 0;
*addr = p->src;
if (!bpb) return 0;
uint32_t level = p->has_view && p->view.set_levels ? p->view.base_level : 0;
uint32_t lw = max1((uint32_t)t->d.width >> level), lh = max1((uint32_t)t->d.height >> level);
const NVNcopyRegion *r = &p->r;
uint32_t w = r->width > 0 ? (uint32_t)r->width : r->xoffset < (int32_t)lw ? lw - (uint32_t)r->xoffset : 0;
uint32_t h = r->height > 0 ? (uint32_t)r->height : r->yoffset < (int32_t)lh ? lh - (uint32_t)r->yoffset : 0;
uint32_t nz = r->depth > 0 ? (uint32_t)r->depth : 1;
uint32_t wb = (w + (uint32_t)bw - 1) / (uint32_t)bw, hb = (h + (uint32_t)bh - 1) / (uint32_t)bh;
if (!wb || !hb) return 0;
uint64_t row = ex->copy_stride > 0 ? (uint64_t)ex->copy_stride : (uint64_t)wb * (uint64_t)bpb;
return row * hb * (nz - 1) + row * (hb - 1) + (uint64_t)wb * (uint64_t)bpb;
}
int vk_exec_xfer_scan(VkExec *ex, uint32_t op, const void *payload) {
switch (op) {
case OP_RENDER_ENABLE: ex->render_disabled = !((const p_flag *)payload)->value; break;
case OP_RENDER_COND: {
int on = cond_enabled(payload);
ex->render_cond = *(const p_render_cond *)payload;
ex->has_render_cond = 1;
ex->render_disabled = !on;
return on;
}
case OP_RESET_COUNTER:
if (((const p_reset_counter *)payload)->type == NVN_COUNTER_TYPE_SAMPLES_PASSED && oq_ready())
ex->oq_counting = 1, ex->oq_reported = 0;
break;
case OP_REPORT_COUNTER: {
const p_report_counter *p = payload;
int samples = p->type == NVN_COUNTER_TYPE_SAMPLES_PASSED && oq_ready();
VkMemSpan m;
if (!nvn_gpu_only(p->addr) && !vk_mem_resolve(p->addr, sizeof(NVNcounterData), &m)) break; // the recording stops
if (samples) oq_report_state(ex);
if (!nvn_gpu_only(p->addr)) pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples};
break;
}
}
return 1;
}
+5 -2
View File
@@ -243,7 +243,9 @@ void vk_hdr_shader_swap(uint32_t **spirv, uint64_t *words) {
int vk_hdr_track; // vk_exec_draw.c calls vk_hdr_draw: set at the first present with TOTK_HDR=1
static volatile int32_t g_active; // the swapchain is HDR10 (gfx_present.c vk_hdr_set_active)
static PlatMutex rec_lock = PLAT_MUTEX_INIT;
static struct { VkImage win, src; VkFormat fmt; } recs[4]; // per presented window image: what the last draw copied
// Per presented window image: what the last draw in replay order copied, and where that draw is. Chunks of a replay
// record at once, so a draw of an earlier chunk can get here after one of a later chunk.
static struct { VkImage win, src; VkFormat fmt; uint64_t fence; uint32_t pos; } recs[4];
static int nrecs;
void vk_hdr_set_active(int on) { plat_xchg32(&g_active, on); }
@@ -277,7 +279,8 @@ void vk_hdr_draw(VkExec *ex, const VkPipeInfo *info) {
}
plat_mutex_lock(&rec_lock);
for (int i = 0; i < nrecs; i++)
if (recs[i].win == t->image) recs[i].src = src, recs[i].fmt = fmt;
if (recs[i].win == t->image && (ex->fence > recs[i].fence || (ex->fence == recs[i].fence && ex->pos >= recs[i].pos)))
recs[i].src = src, recs[i].fmt = fmt, recs[i].fence = ex->fence, recs[i].pos = ex->pos;
plat_mutex_unlock(&rec_lock);
}
+73 -20
View File
@@ -490,12 +490,24 @@ int vk_mem_class_of(uint64_t addr) {
// ---------------------------------------------------------------- census
static volatile int64_t traffic[VK_MCLS_N][VK_MUSE_N];
static _Thread_local int64_t (*count_to)[VK_MUSE_N];
// Every caller is the replay (vk_exec*, under vk_sync_submit's lock), about ten adds per draw: one writer at a time, so
// a plain add, not a locked one. An add racing the census's reset may be lost.
// a plain add, not a locked one. An add racing the census's reset may be lost. Threads recording chunks in parallel
// count into their own arrays (vk_mem_count_local), added with atomics once per chunk.
void vk_mem_count(int cls, int use, uint64_t bytes) {
if ((unsigned)cls < VK_MCLS_N && (unsigned)use < VK_MUSE_N)
__atomic_store_n(&traffic[cls][use], __atomic_load_n(&traffic[cls][use], __ATOMIC_RELAXED) + (int64_t)bytes,
__ATOMIC_RELAXED);
if ((unsigned)cls >= VK_MCLS_N || (unsigned)use >= VK_MUSE_N) return;
if (count_to) count_to[cls][use] += (int64_t)bytes;
else __atomic_store_n(&traffic[cls][use], __atomic_load_n(&traffic[cls][use], __ATOMIC_RELAXED) + (int64_t)bytes,
__ATOMIC_RELAXED);
}
void vk_mem_count_local(int64_t (*acc)[VK_MUSE_N]) { count_to = acc; }
void vk_mem_count_flush(int64_t (*acc)[VK_MUSE_N]) {
for (int c = 0; c < VK_MCLS_N; c++)
for (int u = 0; u < VK_MUSE_N; u++)
if (acc[c][u]) __atomic_fetch_add(&traffic[c][u], acc[c][u], __ATOMIC_RELAXED), acc[c][u] = 0;
}
static void flag_names(uint32_t f, char *buf, size_t n) {
@@ -1212,6 +1224,10 @@ static VkBufferCopy *mregions; // replay: vkCmdCopyBuffer regions
static int cregions;
static PlatMutex flush_lock = PLAT_MUTEX_INIT;
static int flush_cap = FLUSH_RANGES; // ranges a pool keeps; 1 with TOTK_FLUSH_SPAN=1
// The first uses of a submission's mirrors, the copies they queue, the GPU writes noted and their snapshots (pend, wbl, snl, mrep): threads
// recording chunks of one submission take it. A mirror synced in this submission is read without it (vk_mem_use): its
// acc_sub is stored last. Before pool_lock, gen_lock and flush_lock.
static PlatMutex mirror_lock = PLAT_MUTEX_INIT;
typedef struct {
uint64_t subs, syncs, upload_bytes, upload_copies, first_n, first_bytes, wb_bytes, wb_copies, wb_ranges;
@@ -1336,16 +1352,21 @@ static void refuse(PoolRec *r, int why, const char *what) {
static void mirror_create(PoolRec *r) {
VkCtx *c = vk_ctx();
uint64_t n = r->import_hi - r->import_lo;
r->mstate = MS_REFUSED;
__atomic_add_fetch(&mirror_gen, 1, __ATOMIC_RELEASE);
if (r->kind != POOL_IMPORT || !r->buf || !n) return;
if (r->kind != POOL_IMPORT || !r->buf || !n) {
__atomic_store_n(&r->mstate, MS_REFUSED, __ATOMIC_RELEASE);
return;
}
plat_mutex_lock_shared(&pool_lock);
int i = pidx_upper(&pool_index, r->lo) - 1;
while (i >= 0 && pool_index.e[i].buf != r) i--;
int alone = i >= 0 && (i == 0 || pool_index.maxhi[i - 1] <= pool_index.e[i].lo) &&
(i + 1 == pool_index.n || pool_index.e[i + 1].lo >= pool_index.e[i].hi);
plat_mutex_unlock_shared(&pool_lock);
if (i < 0) return; // finalized meanwhile
if (i < 0) { // finalized meanwhile
__atomic_store_n(&r->mstate, MS_REFUSED, __ATOMIC_RELEASE);
return;
}
if (!alone) return refuse(r, MR_OVERLAP, "it overlaps another pool");
uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT;
if (r1 >= SNAP_REGIONS) return refuse(r, MR_UNTRACKED, "outside the write-watched address space");
@@ -1383,9 +1404,9 @@ static void mirror_create(PoolRec *r) {
mlist[at] = r;
nm++;
plat_mutex_unlock(&pool_lock);
r->acc_sub = msub;
r->mstate = MS_ON;
range_add(&pend, &npend, &cpend, r, 0, n);
__atomic_store_n(&r->acc_sub, msub, __ATOMIC_RELEASE);
__atomic_store_n(&r->mstate, MS_ON, __ATOMIC_RELEASE);
mrep.first_n++, mrep.first_bytes += n;
plat_inc64(&m_live_n);
plat_add64(&m_live_bytes, (int64_t)n);
@@ -1426,7 +1447,6 @@ static void pool_retire(void *arg) {
// Replay thread, the first access of this submission: queues the copies of the pages written since the last sync.
static void mirror_sync(PoolRec *r) {
uint64_t t0 = qpc();
r->acc_sub = msub;
uint64_t lo = r->import_lo, hi = r->import_hi;
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
int excl = 0;
@@ -1475,21 +1495,40 @@ static void mirror_sync(PoolRec *r) {
}
mrep.syncs++;
mrep.sync_t += qpc() - t0;
__atomic_store_n(&r->acc_sub, msub, __ATOMIC_RELEASE);
}
// 1 when r is settled for this submission without mirror_lock: refused, or mirrored and synced (*on).
static int mirror_settled(PoolRec *r, int *on) {
int st = __atomic_load_n(&r->mstate, __ATOMIC_ACQUIRE);
*on = st == MS_ON;
return st == MS_REFUSED || (st == MS_ON && __atomic_load_n(&r->acc_sub, __ATOMIC_ACQUIRE) == msub);
}
void vk_mem_use(VkMemSpan *sp) {
PoolRec *r = sp->pool;
if (r->mstate == MS_NONE) mirror_create(r);
if (r->mstate != MS_ON) return;
if (r->acc_sub != msub) mirror_sync(r);
sp->buffer = r->mbuf;
int on;
if (!mirror_settled(r, &on)) {
plat_mutex_lock(&mirror_lock);
if (r->mstate == MS_NONE) mirror_create(r);
on = r->mstate == MS_ON;
if (on && r->acc_sub != msub) mirror_sync(r);
plat_mutex_unlock(&mirror_lock);
}
if (on) sp->buffer = r->mbuf;
}
void vk_mem_pick(VkMemSpan *sp) {
PoolRec *r = sp->pool;
if (!r || r->mstate != MS_ON) return;
if (r->acc_sub != msub) mirror_sync(r);
sp->buffer = r->mbuf;
int on;
if (!r) return;
if (!mirror_settled(r, &on)) {
plat_mutex_lock(&mirror_lock);
on = r->mstate == MS_ON;
if (on && r->acc_sub != msub) mirror_sync(r);
plat_mutex_unlock(&mirror_lock);
}
if (on) sp->buffer = r->mbuf;
}
int vk_mem_mirror_probe(uint64_t addr) {
@@ -1753,10 +1792,14 @@ static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) {
PoolRec *r = sp->pool;
if (r && size) gpu_writes(r, sp->offset, size);
if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size) return;
plat_mutex_lock(&mirror_lock);
// A binding to the end of its pool (no size) can span hundreds of MiB: those stay copies.
if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size))
if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) {
plat_mutex_unlock(&mirror_lock);
return;
}
range_add(&wbl, &nwbl, &cwbl, r, sp->offset, size);
plat_mutex_unlock(&mirror_lock);
}
void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size) { gpu_wrote(sp, size, 0); }
void vk_mem_gpu_wrote_shader(const VkMemSpan *sp, uint64_t size) { gpu_wrote(sp, size, 1); }
@@ -1888,10 +1931,19 @@ void vk_mem_mirror_begin(void) {
mrep.presnap_t += qpc() - t0;
}
int vk_mem_mirror_pending(void) { return npend || snl_copied < nsnl; }
int vk_mem_mirror_pending(void) {
plat_mutex_lock(&mirror_lock);
int n = npend || snl_copied < nsnl;
plat_mutex_unlock(&mirror_lock);
return n;
}
void vk_mem_mirror_prologue(VkCommandBuffer cb) {
if (!npend && snl_copied == nsnl) return;
plat_mutex_lock(&mirror_lock);
if (!npend && snl_copied == nsnl) {
plat_mutex_unlock(&mirror_lock);
return;
}
mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT,
VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT);
copy_ranges(cb, pend, npend, 1, &mrep.upload_bytes, &mrep.upload_copies);
@@ -1904,6 +1956,7 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb) {
VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT);
npend = 0;
mrep.prologues++;
plat_mutex_unlock(&mirror_lock);
}
// ---- verify mode
+2 -2
View File
@@ -157,10 +157,10 @@ void vk_mvsurvey_draw(VkExec *ex) {
k->size = (uint32_t)u->size;
uint32_t want = u->size < SNAP ? (uint32_t)u->size : SNAP;
VkDescriptorBufferInfo bi;
const uint8_t *src = vk_usnap_find(u->addr, want, 1, &bi) ? vk_usnap_map() + bi.offset : NULL;
const uint8_t *src = vk_usnap_find(u->addr, want, UINT32_MAX, 1, &bi) ? vk_usnap_map() + bi.offset : NULL;
k->have = src ? want : 0;
if (src) memcpy(k->bytes, src, want);
const uint8_t *all = vk_usnap_find(u->addr, u->size, 1, &bi) ? vk_usnap_map() + bi.offset : NULL;
const uint8_t *all = vk_usnap_find(u->addr, u->size, UINT32_MAX, 1, &bi) ? vk_usnap_map() + bi.offset : NULL;
k->mats = all ? lead_mats((const float *)all, (uint32_t)(u->size / 4)) : 0;
if (all) {
Stat *s = &S.st[b];
+5 -4
View File
@@ -3223,10 +3223,11 @@ static void dyn_vals(const VkPipeState *s, DynVals *o) {
}
}
// What the replay's command buffer holds, as vk_pipeline_set_dynamic_delta last recorded it (the replay thread only):
// the values, the inputs of the vertex input and the vertex bindings it reads. Invariant while the caller passes
// valid 1: cb holds DC.v for every dynamic state of P.dyn, colour attachments [0, DC.v.n) included.
static struct {
// What the replay's command buffer holds, as vk_pipeline_set_dynamic_delta last recorded it: the values, the inputs of
// the vertex input and the vertex bindings it reads. Invariant while the caller passes valid 1: cb holds DC.v for every
// dynamic state of P.dyn, colour attachments [0, DC.v.n) included. Per thread: the threads of a chunked replay
// (vk_exec.c) each record their own command buffers, and a new one starts with valid 0.
static _Thread_local struct {
DynVals v;
const GfxShader *vs;
uint32_t vattr[16];
+2
View File
@@ -454,6 +454,8 @@ void rec_check_log(void) {
}
#define OWNED(cb) (rec_live && (cb) == rec_cb)
int rec_owns(VkCommandBuffer cb) { return OWNED(cb); }
// A call the ring takes no packet for: everything queued first, then the call here.
#define FALLBACK() (rec_drain(), RS_.fallbacks++)
+1
View File
@@ -33,6 +33,7 @@ void rec_drain(void); // waits until the recorder made every cal
// vk_queue_lock, before it takes the lock (not on the recorder, not when the thread holds the lock already): waits
// until the recorder submitted every submission queued so far.
void rec_queue_wait(void);
int rec_owns(VkCommandBuffer cb); // the recorder makes cb's calls: a direct call into cb follows rec_drain
void rec_stats_log(void); // TOTK_TRACE=perf: packets, bytes, drains and waits since the last call
// TOTK_REC_CHECK=1 (debug, recorder on): vk_memory.c reports each object it defers (destroy 0, stamped tl) or destroys
// (destroy 1) with the value the GPU reached (gpu); a packet of the part being recorded that still names it is logged
+86 -12
View File
@@ -56,7 +56,7 @@ static VkQueryPool ts_pool;
static int ts_state; // 0 not set up, 1 on, -1 off
static double ts_period; // ns per tick
static uint64_t ts_mask; // valid bits of a timestamp
static uint32_t ts_next; // submitting thread (sub_lock): next pair
static uint32_t ts_next; // next pair (atomic: parts recorded on other threads take pairs too)
static volatile uint8_t ts_busy[TS_PAIRS]; // written until the completion thread read the pair
static int ts_cur = -1; // the pair of the part being recorded, -1 untimed
static uint64_t ts_last_end; // completion thread: end of the latest part read (ticks)
@@ -175,6 +175,12 @@ static void ts_read_cb(void *arg) { // completion thread, in submission order
free(t);
}
static int ts_take(void) {
uint32_t p = __atomic_fetch_add(&ts_next, 1, __ATOMIC_RELAXED) % TS_PAIRS;
if (__atomic_exchange_n(&ts_busy[p], 1, __ATOMIC_ACQUIRE)) { plat_inc64(&ts_missed); return -1; }
return (int)p;
}
// At the start of a command buffer (sub_lock held): the pair it times, -1 untimed.
static int ts_open(VkCommandBuffer cb) {
if (ts_state != 1 || !perf_gpu_timing()) {
@@ -182,10 +188,9 @@ static int ts_open(VkCommandBuffer cb) {
ts_setup(vk_ctx()); // the overlay turned full: timestamps from now on
if (ts_state != 1) return -1;
}
uint32_t p = ts_next % TS_PAIRS;
if (__atomic_load_n(&ts_busy[p], __ATOMIC_ACQUIRE)) { plat_inc64(&ts_missed); return -1; }
ts_next++;
ts_busy[p] = 1;
int t = ts_take();
if (t < 0) return -1;
uint32_t p = (uint32_t)t;
vkCmdResetQueryPool(cb, ts_pool, 2 * p, 2);
vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, ts_pool, 2 * p);
return (int)p;
@@ -203,6 +208,17 @@ static void ts_close(VkCommandBuffer cb, int pair, uint64_t fence) {
vk_completion_add(fence, ts_read_cb, t);
}
int vk_sync_ts_open(VkCommandBuffer cb) {
if (__atomic_load_n(&ts_state, __ATOMIC_ACQUIRE) != 1 || !perf_gpu_timing()) return -1;
int p = ts_take();
if (p < 0) return -1;
vkCmdResetQueryPool(cb, ts_pool, 2 * (uint32_t)p, 2);
vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, ts_pool, 2 * (uint32_t)p);
return p;
}
void vk_sync_ts_close(VkCommandBuffer cb, int pair, uint64_t fence) { ts_close(cb, pair, fence); }
// The pair of the part being recorded.
static void ts_begin(VkCommandBuffer cb) { ts_cur = ts_open(cb); }
@@ -349,26 +365,56 @@ static CmdSlot *take_slot(VkCtx *c, uint64_t fence) {
// submitted right before the part that needs them and timed like a part; VK_NULL_HANDLE when there are none. After the
// part was recorded, sub_lock held; may move slots[].
// First brings the shadow storage up to date with the CPU's writes, which needs no command buffer.
static VkCommandBuffer prologue(VkCtx *c, uint64_t fence) {
// direct: the calling thread records, ends and submits it itself while the part being replayed stays the recorder's
// (vk_sync_part, which the caller made the recorder idle for).
static VkCommandBuffer prologue(VkCtx *c, uint64_t fence, int direct) {
vk_mem_shadow_sync(); // shadow storage: the CPU's writes go in before the part is submitted
if (!vk_mem_mirror_pending()) return VK_NULL_HANDLE;
int i = claim_slot(c, fence); // before reading slots: the claim may move it
VkCommandBuffer cb = slots[i].cb;
rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts
if (direct) {
VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};
vkBeginCommandBuffer(cb, &bi);
} else {
rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts
}
int pair = ts_open(cb);
vk_mem_mirror_prologue(cb);
ts_close(cb, pair, fence);
if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb);
if (direct || !rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb);
return cb;
}
// A chunk of a chunked replay (vk_exec.c chunks_submit), while the submission's own part is open on the recorder. The
// recorder begins, records, ends and submits the parts it owns (vk_rec.c rec_end_cb), so the prologue stays off it:
// the recorder is made idle first, as the part pool allows one thread at a time, and the prologue is recorded, ended
// and submitted here without taking the recorder's part away.
void vk_sync_part(VkCommandBuffer cb) {
VkCtx *c = vk_ctx();
vk_upload_flush(0);
rec_drain();
VkCommandBuffer pro = prologue(c, cur_fence, 1);
VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro},
{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}};
VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1,
.pCommandBufferInfos = pro ? cbi : cbi + 1};
vk_queue_lock();
VkResult r = vkQueueSubmit2(c->gfx_queue, 1, &si, VK_NULL_HANDLE);
vk_queue_unlock();
if (r != VK_SUCCESS) {
vk_device_fault_report(r);
rt_fatal("vk: vkQueueSubmit2 of a part of value %llu failed (%d)", (unsigned long long)cur_fence, (int)r);
}
}
VkCommandBuffer vk_sync_split(void) {
VkCtx *c = vk_ctx();
VkCommandBuffer cb = slots[cur_slot].cb;
ts_end(cb, cur_fence);
if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); // every command of the part is in cb, ended by its recorder
vk_upload_flush(0); // the uploads this part asked for precede it in queue order
VkCommandBuffer pro = prologue(c, cur_fence);
VkCommandBuffer pro = prologue(c, cur_fence, 0);
VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro},
{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}};
VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1,
@@ -419,7 +465,7 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) {
perf_replay(t0);
// After the recording: the uploads it asked for (and every earlier one) precede it in queue order.
vk_upload_flush(0);
VkCommandBuffer pro = prologue(c, fence);
VkCommandBuffer pro = prologue(c, fence, 0);
s = &slots[cur_slot];
VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro},
{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = s->cb}};
@@ -446,8 +492,11 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) {
return fence;
}
void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) {
plat_mutex_lock(&st_lock);
static _Thread_local VkDeferred *deferred;
void vk_completion_defer(VkDeferred *d) { deferred = d; }
static void queue_put(uint64_t fence, void (*fn)(void *), void *arg) { // st_lock held
if (qhead + qn == qcap) {
if (qhead) {
memmove(queue, queue + qhead, sizeof *queue * (size_t)qn);
@@ -463,9 +512,34 @@ void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) {
// The completion thread sleeps only while nothing is issued beyond done: a callback for a later value (the replay's,
// for the submission it records: counter reports, freed resources) becomes due through the submit, which wakes it.
if (fence <= done) plat_cond_broadcast(&cv_work);
}
void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) {
VkDeferred *d = deferred;
if (d) {
if (d->n == d->cap) {
d->cap = d->cap ? d->cap * 2 : 64;
d->items = realloc(d->items, sizeof(Pending) * (size_t)d->cap);
if (!d->items) rt_fatal("vk: out of memory (completion queue)");
}
((Pending *)d->items)[d->n++] = (Pending){fence, fn, arg};
return;
}
plat_mutex_lock(&st_lock);
queue_put(fence, fn, arg);
plat_mutex_unlock(&st_lock);
}
void vk_completion_flush(VkDeferred *d) {
plat_mutex_lock(&st_lock);
for (int i = 0; i < d->n; i++) {
const Pending *p = &((Pending *)d->items)[i];
queue_put(p->fence, p->fn, p->arg);
}
plat_mutex_unlock(&st_lock);
d->n = 0;
}
static int fence_wait_impl(uint64_t value, uint64_t timeout_ns);
int vk_sync_fence_wait(uint64_t value, uint64_t timeout_ns) {
+137 -21
View File
@@ -46,6 +46,8 @@ typedef struct Tex {
uint32_t wgen; // page generation of the last upload check
uint32_t checked; // submit_epoch of the last check from a draw binding
int uploaded, released;
uint32_t rev_epoch, rev_pos; // the state scan made it live in that submission at that op (vk_texture_revive)
uint32_t own_epoch, own_pos; // a chunk made it GPU-owned in that submission at that op (vk_texture_gpu_written)
uint32_t n_up; // uploads of this entry (vk_texture_debug)
int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked
struct Tex *next_at; // other entries at the same storage address
@@ -122,10 +124,12 @@ static uint64_t n_checks; // draw-binding storage checks (make_fresh_snap) si
// which emptied every cache keyed on the global generations.
#define AGEN_N 4096
static uint32_t agen[AGEN_N];
static uint32_t arev[AGEN_N]; // per storage address (hashed): the submit_epoch of the state scan's last revival there
// Per slot id, for vk_texture_slot_token without the lock: the registration serial (bits 16-47; 0: never registered)
// and the address generation index (bits 0-15), one word so a reader gets a consistent pair. Fixed-size (slot ids are
// 20 bits), unlike tslots, which moves when it grows.
static uint64_t *sinfo;
static uint32_t *sepoch; // per slot: the submit_epoch its entry was checked in (vk_texture_slot_checked), 0 none
static uint32_t reg_serial;
static uint32_t agen_idx(uint64_t addr) { return (uint32_t)(addr * 0x9E3779B97F4A7C15ull >> 52); }
@@ -143,11 +147,17 @@ static void tex_init(void *p) {
__atomic_store_n(&tex_ready, 1, __ATOMIC_RELEASE);
return;
}
static _Thread_local int tex_depth; // how often this thread holds tex_lock (entry_of releases it for a wait)
static _Thread_local uint32_t tex_settles; // entry_of waits of this thread (vk_texture_by_id_locked)
static void lock(void) {
if (!__atomic_load_n(&tex_ready, __ATOMIC_ACQUIRE)) plat_once(&tex_once, tex_init, NULL);
plat_rmutex_lock(&tex_lock);
tex_depth++;
}
static void unlock(void) {
tex_depth--;
plat_rmutex_unlock(&tex_lock);
}
static void unlock(void) { plat_rmutex_unlock(&tex_lock); }
static uint32_t umax(uint32_t a, uint32_t b) { return a > b ? a : b; }
static uint32_t mip(uint32_t v, unsigned lv) { return umax(v >> lv, 1); }
@@ -1231,10 +1241,20 @@ static int alias_compatible(const Tex *e, const Tex *src) {
return eb && eb == sb && ew == sw && eh == sh;
}
// src holds content the storage bytes do not: the GPU wrote it and, for a mirrored texture, it was not written back
// and the CPU has not rewritten the storage since (the new entry's upload then reads the newer bytes).
static int alias_has_gpu_content(const Tex *src) {
if (!src->pub.gpu_owned) return 0;
// Whether e is live at op pos of a chunked replay: the state scan is ahead of the chunks, and an entry it revived at a
// later op is still released for this one. Two 4K targets at one address, RGBA8 and R8 with memory of their own, were
// swapped whenever the scan got past the second's SET_RT before a chunk sampling the storage got its turn, and large
// parts of the scene flickered. UINT32_MAX: the state now.
static int live_at(const Tex *e, uint32_t pos) {
return !e->released && !(e->rev_epoch == submit_epoch && e->rev_pos > pos);
}
// src holds content the storage bytes do not, at op pos of a chunked replay (UINT32_MAX: the state now): the GPU wrote
// it and, for a mirrored texture, it was not written back and the CPU has not rewritten the storage since (the new
// entry's upload then reads the newer bytes). A chunk at a later op that rendered src first does not count: in replay
// order that render comes after the new entry's first use.
static int alias_has_gpu_content(const Tex *src, uint32_t pos) {
if (!src->pub.gpu_owned || (src->own_epoch == submit_epoch && src->own_pos > pos)) return 0;
if (!src->pub.mirrored) return 1;
uint32_t now;
return src->gpu_dirty && !storage_rewritten(src, src->gpu_gen, &now);
@@ -1288,10 +1308,45 @@ static int alias_copy(Tex *e, const Tex *src) {
return 1;
}
static void entry_live(Tex *e) {
if (!e->released) return;
takeback_n[e->pub.gpu_owned != 0][age_bucket(coarse_ms - e->rel_ms)]++;
e->use_seq = ++use_counter;
e->released = 0, nreleased--, rel_unlink(e), tex_gen_bump(e->key.addr);
}
// A live colour entry at e's storage of e's size, as of op pos: the new entry may take its content (alias_copy).
static int alias_candidate(const TexKey *k, const tex_t *t, uint32_t pos) {
for (const Tex *p = nvn_hm_get(&by_addr, &k->addr); p; p = p->next_at)
if (live_at(p, pos) && !p->f.depth && !p->f.stencil && p->t.L.size == t->L.size) return 1;
return 0;
}
static Tex *entry_of(const tex_t *t) {
TexKey k;
make_key(t, &k);
Tex *e = nvn_hm_get(&by_key, &k);
uint32_t pos = vk_exec_replay_pos();
// A chunk of a chunked replay makes the entry as of its own op: whether it takes the content of a texture at its
// storage (alias_copy) depends on what the chunks before it render there, and those may still be recording on
// other threads. Taken from a render they had not recorded yet, the new image got the storage's old bytes instead:
// on an RX 7800 XT the grass near Link was missing in 3 of 29 launches with 4 threads (a map texture drawn once
// per tile, kept from then on). With a live colour texture of its size at the storage, the chunk waits until the
// chunks before it are recorded, the texture lock released for the wait (as often as this thread holds it).
if (!e && pos != UINT32_MAX && tex_alias != 0 && alias_candidate(&k, t, pos) && vk_exec_chunk_behind()) {
tex_t own = *t; // t may point into tslots, which a registration moves while the lock is released
t = &own;
int d = tex_depth;
for (int i = 0; i < d; i++) unlock();
vk_exec_chunk_settle();
for (int i = 0; i < d; i++) lock();
tex_settles++;
if ((e = nvn_hm_get(&by_key, &k))) { // another chunk made it meanwhile
entry_live(e);
e->t = *t;
return e;
}
}
if (!e) {
if (tex_alias < 0) {
tex_alias = cfg_bool("TOTK_TEX_ALIAS");
@@ -1309,8 +1364,9 @@ static Tex *entry_of(const tex_t *t) {
nx = p->next_at;
if (p->released) { destroy_tex(p); continue; }
alias = 1;
if (!live_at(p, pos)) continue; // the scan revives it at a later op: released at this chunk's
if (tex_alias && (!src || (same_but_swizzle(&p->key, &k) && !same_but_swizzle(&src->key, &k))) &&
alias_compatible(e, p) && alias_has_gpu_content(p))
alias_compatible(e, p) && alias_has_gpu_content(p, pos))
src = p;
}
if (alias) perf_count(PERF_CNT_TEX_ALIAS);
@@ -1336,11 +1392,7 @@ static Tex *entry_of(const tex_t *t) {
nvn_hm_put(&by_key, &k, e);
tex_gen_bump(k.addr);
}
if (e->released) {
takeback_n[e->pub.gpu_owned != 0][age_bucket(coarse_ms - e->rel_ms)]++;
e->use_seq = ++use_counter;
e->released = 0, nreleased--, rel_unlink(e), tex_gen_bump(k.addr);
}
entry_live(e);
e->t = *t;
return e;
}
@@ -1366,6 +1418,25 @@ const VkTexImage *vk_texture_get(const tex_t *t) {
return e ? &e->pub : NULL;
}
// The replay's state scan (vk_exec.c): the entry vk_texture_get would return, neither created nor made current; the
// recording does that. It is revived here, in replay order, and remembers the op it happened at: a chunk may bind a
// finalized target's slot before the chunk that renders it reached its SET_RT, or long after the scan went past a later
// target at the same storage, and slot_resolve has to answer as the replay would at the chunk's own op (live_at).
const VkTexImage *vk_texture_revive(const tex_t *t, uint32_t pos) {
if (!t || !t->mem) return NULL;
TexKey k;
make_key(t, &k);
lock();
Tex *e = nvn_hm_get(&by_key, &k);
if (e && e->released) {
e->rev_epoch = submit_epoch, e->rev_pos = pos;
__atomic_store_n(&arev[agen_idx(k.addr)], submit_epoch, __ATOMIC_RELEASE);
entry_live(e);
}
unlock();
return e ? &e->pub : NULL;
}
// Render targets the CPU reads without NVN's invalidate (vk_texture_write_back_small); small_pending: one may be dirty.
// wb_limit: targets up to this size (bytes) always go back when the CPU waits for a fence, TOTK_WB_FENCE_LIMIT, read
// once. A larger one goes back unless the GPU wrote it in each of the last WB_RUN frames (wb_wanted). wb_trace:
@@ -1398,6 +1469,10 @@ void vk_texture_gpu_written(const tex_t *t) {
Tex *e = entry_of(t);
if (e) {
make_fresh(e); // a partial render keeps what the storage held before
if (!e->pub.gpu_owned) { // entry_of's alias choice of a chunk at an earlier op leaves this render out
uint32_t pos = vk_exec_replay_pos();
e->own_epoch = pos == UINT32_MAX ? 0 : submit_epoch, e->own_pos = pos;
}
e->pub.gpu_owned = 1;
uint64_t f = vk_exec_recording_fence();
if (f > e->wr_fence) e->wr_fence = f;
@@ -1468,7 +1543,9 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) {
}
if (!sinfo) {
uint64_t *p = calloc(1 << 20, sizeof *p);
if (!p) rt_fatal("vk: out of memory for texture slot tokens");
uint32_t *q = calloc(1 << 20, sizeof *q);
if (!p || !q) rt_fatal("vk: out of memory for texture slot tokens");
__atomic_store_n(&sepoch, q, __ATOMIC_RELEASE);
__atomic_store_n(&sinfo, p, __ATOMIC_RELEASE);
}
TexSlot *s = &tslots[id];
@@ -1484,15 +1561,15 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) {
unlock();
}
// The entry slot s resolves to, and the view and has_view a caller gets with it (the slot's, with the descriptor's
// swizzle when the entry is another descriptor at the same storage). NULL when the slot names no texture.
static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) {
// The entry slot s resolves to at op pos, and the view and has_view a caller gets with it (the slot's, with the
// descriptor's swizzle when the entry is another descriptor at the same storage). NULL when the slot names no texture.
static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view, uint32_t pos) {
*view = s->v;
*has_view = s->has_view;
TexKey k;
make_key(&s->t, &k);
Tex *e = nvn_hm_get(&by_key, &k);
if (!e || e->released) { // MNG hle_nvn.c: a finalized slot follows the live texture at its storage
if (!e || !live_at(e, pos)) { // MNG hle_nvn.c: a finalized slot follows the live texture at its storage
// Prefer the live entry that differs from the slot descriptor only in its swizzle: transient render targets
// share storage, and the newest one at the address is usually another size. The map screen samples its
// 2250x1080 map target through an (R,G,B,1) descriptor and followed a 960x60 target at the same address.
@@ -1505,13 +1582,14 @@ static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) {
Tex *own = e, *any = NULL;
e = NULL;
for (Tex *p = nvn_hm_get(&by_addr, &k.addr); p; p = p->next_at) {
if (p->released || linear_format(p->key.d.format) != linear_format(k.d.format)) continue;
if (!live_at(p, pos) || linear_format(p->key.d.format) != linear_format(k.d.format)) continue;
if (!any) any = p;
if (same_but_swizzle(&p->key, &k)) { e = p; break; }
}
if (!e) e = any ? any : own;
}
if (!e) e = s->t.mem ? entry_of(&s->t) : NULL;
TexSlot copy; // entry_of may release the lock (a chunk's wait), and a registration then moves tslots
if (!e && s->t.mem) copy = *s, s = &copy, e = entry_of(&s->t);
// A followed texture is another descriptor at the same storage: its image holds the content, but sampling takes the
// slot descriptor's swizzle. The UI renders text into RGBA8 and samples it through a (A,A,A,A) descriptor; with the
// render target's RGBA swizzle the text blur saw R = 1 everywhere and drew solid boxes behind tips and labels.
@@ -1553,14 +1631,38 @@ const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view) {
const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view) {
if (id < 0 || id >= ntslots || !tslots[id].used) return NULL;
TexSlot *s = &tslots[id];
uint32_t g = agen[sinfo[id] & 0xffff];
uint32_t idx = sinfo[id] & 0xffff, g = agen[idx];
uint32_t pos = arev[idx] == submit_epoch ? vk_exec_replay_pos() : UINT32_MAX;
if (pos != UINT32_MAX) { // what the storage holds depends on the op: chunks at other ops share no answer in s
view_t v;
int hv;
Tex *f = slot_resolve(s, &v, &hv, pos);
s = &tslots[id]; // slot_resolve may have released the lock
if (f && (!f->uploaded || f->checked != submit_epoch)) {
make_fresh_snap(f);
f->checked = submit_epoch;
}
if (view) *view = has_view ? v : s->v;
if (has_view) *has_view = view ? hv : s->has_view;
return f ? &f->pub : NULL;
}
if (!s->ce || s->cgen != g) { // resolved again: an entry at its address was created, dropped or released, or the
view_t v; // slot registered (s->ce cleared); an empty result is not kept: image creation may
int hv; // succeed later
s->ce = slot_resolve(s, &v, &hv); // entry_of may create an entry and bump g: g read before it
uint32_t settles = tex_settles;
Tex *r = slot_resolve(s, &v, &hv, UINT32_MAX); // entry_of may create an entry and bump g: g read before it
s = &tslots[id];
if (settles != tex_settles) { // the lock was released for a wait: the slot may be another texture by now
if (r && (!r->uploaded || r->checked != submit_epoch)) make_fresh_snap(r), r->checked = submit_epoch;
if (view) *view = has_view ? v : s->v;
if (has_view) *has_view = view ? hv : s->has_view;
return r ? &r->pub : NULL;
}
s->ce = r;
s->cview = v, s->chas_view = (uint8_t)hv;
s->cgen = g;
s->cepoch = 0;
__atomic_store_n(&sepoch[id], 0, __ATOMIC_RELAXED);
}
Tex *e = s->ce;
// Bindings of one submit share one check, answered from the submit's write-watch snapshot (guest writes during a
@@ -1572,6 +1674,7 @@ const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view) {
e->checked = submit_epoch;
}
s->cepoch = submit_epoch;
__atomic_store_n(&sepoch[id], submit_epoch, __ATOMIC_RELEASE);
}
if (view) *view = has_view ? s->cview : s->v;
if (has_view) *has_view = view ? s->chas_view : s->has_view;
@@ -1593,6 +1696,19 @@ int vk_texture_slot_swz(int id, uint8_t swz[4]) {
uint32_t vk_texture_epoch(void) { return __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); }
int vk_texture_slot_moves(int id) {
const uint64_t *si = __atomic_load_n(&sinfo, __ATOMIC_ACQUIRE);
if (!si || id < 0 || id >= (1 << 20)) return 0;
uint64_t v = __atomic_load_n(&si[id], __ATOMIC_RELAXED);
return __atomic_load_n(&arev[v & 0xffff], __ATOMIC_ACQUIRE) == __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED);
}
int vk_texture_slot_checked(int id) {
const uint32_t *e = __atomic_load_n(&sepoch, __ATOMIC_ACQUIRE);
return e && id >= 0 && id < (1 << 20) &&
__atomic_load_n(&e[id], __ATOMIC_ACQUIRE) == __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED);
}
void vk_texture_new_submit(void) {
lock();
if (!++submit_epoch) submit_epoch = 1;
@@ -2419,6 +2535,6 @@ void vk_texture_shutdown(void) {
free(tslots), free(sslots);
tslots = NULL, sslots = NULL, ntslots = nsslots = 0, nreleased = 0;
rel_head = rel_tail = NULL;
if (sinfo) memset(sinfo, 0, sizeof *sinfo << 20); // no slot is registered: tokens 0
if (sinfo) memset(sinfo, 0, sizeof *sinfo << 20), memset(sepoch, 0, sizeof *sepoch << 20); // no slot is registered: tokens 0
unlock();
}
+24 -12
View File
@@ -11,11 +11,13 @@
// that submission's write-back, the mirror holds them until then.
// - A GPU write within the submission (UpdateUniformBuffer, a storage buffer a shader writes, a buffer copy or fill,
// an image-to-buffer copy: vk_usnap_drop) drops the copies it touches from then on, so later draws read the result.
// A chunked replay (vk_exec.c) records the ops out of order: its scan drops at the position of the op
// (vk_usnap_drop_at, scan_writes for the shaders' stores), and a chunk finds a copy only at a position before that.
// - The arena goes back to a free list once the submission's fence is done (completion thread).
// The design follows MacNativeGames' TOTK_UBO_SNAPSHOT (games/totk/third_party/d2r_gfx/gfx/nvn/nvn_mtl.m usnap_take,
// usnap_drop; MIT, Copyright (c) 2026 jhaizhou); the code is ours.
// The copy is taken where vk_exec_record starts: at the submit call when the replay runs on the submitting thread, later
// with TOTK_REPLAY_ASYNC. TOTK_UBO_SNAPSHOT=0 binds live memory as before; =ab alternates (TOTK_AB).
// with TOTK_REPLAY_ASYNC; a chunked replay takes it on a recording thread while its scan runs. TOTK_UBO_SNAPSHOT=0 binds live memory as before; =ab alternates (TOTK_AB).
#include <stdlib.h>
#include <string.h>
@@ -42,7 +44,7 @@ typedef struct Arena {
} Arena;
typedef struct { uint64_t lo, hi; } Rng;
typedef struct { uint64_t lo, hi, off; int dropped; } Reg;
typedef struct { uint64_t lo, hi, off, drop_at; } Reg; // drop_at: dropped from this position on; UINT64_MAX not
typedef struct { uint64_t lo, hi, fence; } Wrote;
static struct {
@@ -62,6 +64,7 @@ static struct {
static Arena *free_arenas;
static PlatMutex free_lock = PLAT_MUTEX_INIT;
static PlatMutex drop_lock = PLAT_MUTEX_INIT; // threads recording chunks of one submission drop concurrently
static void *grow(void *p, int *cap, size_t elem) {
int n = *cap ? *cap * 2 : 1024;
@@ -279,7 +282,7 @@ void vk_usnap_begin(const uint64_t *handles, int n, uint64_t fence) {
continue;
}
if (U.nreg == U.creg) U.reg = grow(U.reg, &U.creg, sizeof *U.reg);
U.reg[U.nreg++] = (Reg){lo, hi, total, 0};
U.reg[U.nreg++] = (Reg){lo, hi, total, UINT64_MAX};
total += (hi - lo + ALIGN - 1) & ~(uint64_t)(ALIGN - 1);
}
if (!U.nreg) return;
@@ -313,29 +316,38 @@ static int first_above(uint64_t a) {
return lo;
}
int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t align, VkDescriptorBufferInfo *out) {
int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t at, uint64_t align, VkDescriptorBufferInfo *out) {
if (!U.on || !n) return 0;
int i = first_above(addr);
if (i >= U.nreg) return 0;
const Reg *r = &U.reg[i];
if (r->dropped || addr < r->lo || addr + n > r->hi) return 0;
if (__atomic_load_n(&r->drop_at, __ATOMIC_RELAXED) <= at || addr < r->lo || addr + n > r->hi) return 0;
uint64_t off = r->off + (addr - r->lo);
if (off % align) return 0;
*out = (VkDescriptorBufferInfo){U.arena->buf, off, n};
U.hits++;
__atomic_fetch_add(&U.hits, 1, __ATOMIC_RELAXED);
return 1;
}
void vk_usnap_drop(uint64_t addr, uint64_t n) {
static void drop(uint64_t addr, uint64_t n, uint64_t at) {
if (!U.mode || !n) return;
plat_mutex_lock(&drop_lock);
if (U.nwr == U.cwr) U.wr = grow(U.wr, &U.cwr, sizeof *U.wr);
U.wr[U.nwr++] = (Wrote){addr, addr + n, U.fence};
if (!U.on) return;
for (int i = first_above(addr); i < U.nreg && U.reg[i].lo < addr + n; i++)
if (!U.reg[i].dropped) U.reg[i].dropped = 1, U.drops++, U.gen++;
if (U.on)
for (int i = first_above(addr); i < U.nreg && U.reg[i].lo < addr + n; i++)
if (at < U.reg[i].drop_at) {
if (U.reg[i].drop_at == UINT64_MAX) U.drops++;
__atomic_store_n(&U.reg[i].drop_at, at, __ATOMIC_RELAXED);
__atomic_add_fetch(&U.gen, 1, __ATOMIC_RELAXED);
}
plat_mutex_unlock(&drop_lock);
}
uint64_t vk_usnap_gen(void) { return U.gen; }
void vk_usnap_drop(uint64_t addr, uint64_t n) { drop(addr, n, 0); }
void vk_usnap_drop_at(uint64_t addr, uint64_t n, uint32_t at) { drop(addr, n, at); }
uint64_t vk_usnap_gen(void) { return __atomic_load_n(&U.gen, __ATOMIC_RELAXED); }
// the host copy where there is one (DLSS), else the device copy
static uint8_t *host_copy(void) { return U.arena->cpu ? U.arena->cpu : U.arena->map; }
@@ -344,7 +356,7 @@ const uint8_t *vk_usnap_map(void) { return U.arena ? host_copy() : NULL; }
const uint8_t *vk_usnap_bytes(uint64_t addr, uint64_t n) {
VkDescriptorBufferInfo bi;
return vk_usnap_find(addr, n, 1, &bi) ? host_copy() + bi.offset : NULL;
return vk_usnap_find(addr, n, UINT32_MAX, 1, &bi) ? host_copy() + bi.offset : NULL;
}
// The arenas back on the free list (vk_exec_draw_shutdown: the device is idle, every completion ran).
+5
View File
@@ -38,6 +38,11 @@
# Replay NVN submissions on their own thread, at most one frame behind the game, 0 or 1. Default: off. Helps where the
# game waits for the GPU; adds up to a frame of latency.
# replay_async = 0
# Record each large submission in chunks on this many threads (0 to 16; 0 = one thread, the default). The chunks go to
# the GPU in order. In the town at 4K on an RTX 5070, 4 cut the replay from 8.2 to 4.5 ms a frame (71 to 79 fps); more
# threads add little. Needs 8 or more logical processors to help. Contact shadows and DLSS (dlaa = 1) run in chunks too.
# Read at start.
# replay_workers = 0
# EXPERIMENTAL, THE GAME CRASHES WITH 1. 1 lifts the virtual core gate: guest threads run without the console's limit
# of one thread per core, and the game keeps per-core data without locks. With 1, 1.2.1 and 1.4.3 crash or hang,
# usually at start (#212). Keep 0 to play.