diff --git a/docs/architecture.md b/docs/architecture.md index 56ac5987..c5931732 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -90,7 +90,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. - Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. -- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `TOTK_SHADOW=1` on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.2.1 and 1.4.3); results the game reads back from the GPU after it can be a frame old. +- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.2.1 and 1.4.3); results the game reads back from the GPU after it can be a frame old. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/docs/building.md b/docs/building.md index 0a307eb1..8f2ab0dd 100644 --- a/docs/building.md +++ b/docs/building.md @@ -200,7 +200,7 @@ re-protecting imported memory invalidates the driver's userptr buffer on every s amdgpu), and importing pins the pages, which drops their write protection. Each CPU-visible pool therefore gets a host-visible copy (shadow storage, `runtime/vk_memory.c`): the pages the CPU wrote go in before each submit, and the GPU's writes come back at its fence. In play that costs about 0.2 ms a frame and doubles the memory of those pools (about -1.1 GB). `TOTK_SHADOW=1` turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as +1.1 GB). `shadow = 1` in `totk.ini` (or `TOTK_SHADOW=1`) turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as well: the shadow copies are chosen for the measured cost above, not for a missing extension. The Deck's GPU (Van Gogh) has no native ASTC, so its textures go through the compute decoder (`runtime/vk_astc.c`, diff --git a/runtime/cfg.c b/runtime/cfg.c index 9488381b..96465f60 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -61,7 +61,7 @@ static const char *const known[] = { "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHUNKS", "REPLAY_CHUNK_DRAWS", "REPLAY_SKEW_US", "REPLAY_STATE_CHECK", "REPLAY_WORKERS", "REPLAY_CHECK", "REPORT_GUARD", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", - "SHADER_MEMO", "SHADOW_DEFER", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHADOW_SNAP", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", + "SHADER_MEMO", "SHADOW", "SHADOW_DEFER", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHADOW_SNAP", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "STICK_SENS", "SWKBD_ANSWER", "SYNC_AHEAD", "SWKBD_TEST", "TEST_OOM", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_BLOCK_MIB", "TEX_HEAP_EXTRA_MB", "TEX_MEMO", "TEX_RELEASED_MIB", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UB_OVERLAY", "UB_OVERLAY_OUT", "UPSCALER", "VB_USED", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64", diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 9007902a..560c64f6 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -494,7 +494,7 @@ static int sync_ahead(void) { // that. The Vulkan back end sets the mode while it starts, before the game's first present or wait; the null // back end never does. if (on && !vk_mem_shadow_on()) { - rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage (TOTK_SHADOW=1 on native Windows); " + rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage (shadow = 1 in totk.ini or TOTK_SHADOW=1 on native Windows); " "the game waits for the GPU as before"); on = 0; } diff --git a/runtime/vk_dlaa.c b/runtime/vk_dlaa.c index 18643d82..6f6d42fc 100644 --- a/runtime/vk_dlaa.c +++ b/runtime/vk_dlaa.c @@ -942,6 +942,18 @@ static int perf_quality(uint32_t w, uint32_t ow) { return k >= 0.66 ? 2 : k >= 0.57 ? 1 : k >= 0.49 ? 0 : 3; } +// A target of the scene's shape (resolution.c's scene size, within 5 %; the dynamic resolution tiers keep it). In the +// inventory the game renders the items' icons as the slots fill in, each through a G-buffer and a light target of its +// own with a D32F depth, square (432x432 or 426x426 at a 3840x2160 scene) and with no wider G-buffer before it in the +// frame; DLSS armed on that, scaled it to the output and wrote it over the menu, so a weapon filled the screen on the +// frames an icon was drawn. +static int scene_shaped(const VkExecTarget *c) { + unsigned o[2], s[2]; + res_sizes(o, s); + uint64_t a = (uint64_t)c->width * s[1], b = (uint64_t)c->height * s[0]; + return a * 20 >= b * 19 && b * 20 >= a * 19; +} + // dst: the target DLSS's picture goes to, NULL for the light target itself (DLAA in place). static void evaluate(VkExec *ex, const VkExecTarget *dst) { const VkExecTarget *t = &D.color, *z = &D.depth; @@ -1081,10 +1093,40 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { (unsigned long long)D.why[2], (unsigned long long)R.made, (unsigned long long)R.missed); } +// More than half the scene size resolution.c set, on each side (the dynamic resolution tiers go down to 80 %). Around +// the quick weapon menu the only G-buffer before the scene's end is at times the half-size one (1920x1080 at a +// 3840x2160 scene), and the half-size RGBA16F effects target with its D32F depth armed; DLSS scaled the effects to the +// output in place of the scene, which showed dark and blurred for that frame. +static int scene_large(const VkExecTarget *c) { + unsigned o[2], s[2]; + res_sizes(o, s); + return c->width * 2 > s[0] && c->height * 2 > s[1]; +} + static int scene_color(VkFormat f) { return f == VK_FORMAT_B10G11R11_UFLOAT_PACK32 || f == VK_FORMAT_R16G16B16A16_SFLOAT; } +// The bloom chain's first pass: at most a quarter of the scene's width, into level 0 of a scene colour target. At some +// views (seen in the yard of Lookout Landing) the game draws a full-screen pass without depth into the light target, +// then a half-size scene colour target and its mips, and only then the light target with the scene depth. Level 1 of +// those mips is a quarter of the scene's width; taken for the bloom it ended the scene before anything was armed, DLSS +// skipped those frames, and the picture flickered between anti-aliased frames and plain ones as the view changed. +// The pass also has the scene's shape (resolution.c's scene size, within 5 %) and no depth. In rain by a waterfall the +// game drew two transparents into the light target with the scene depth before its sky lookups (25x25 and 256x256 +// RGBA16F) and its quarter-size particles (960x540 RGBA16F with a depth of their own), which then ended the armed +// scene: DLSS ran before the mist and most transparents, which were drawn over its picture unprocessed, on runs of +// frames, and the mist in the distance darkened or brightened for a frame wherever a frame in between came out whole. +static int bloom_start(const VkExec *ex, uint32_t scene_w) { + const VkExecTarget *c = &ex->color[0]; + if (!ex->pass_w || ex->pass_w * 4 > scene_w || !c->image || !scene_color(c->format) || c->level || ex->depth.image) + return 0; + unsigned o[2], s[2]; + res_sizes(o, s); + uint64_t a = (uint64_t)ex->pass_w * s[1], b = (uint64_t)ex->pass_h * s[0]; + return a * 20 >= b * 19 && b * 20 >= a * 19; +} + int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { const VkExecTarget *c = &ex->color[0], *z = &ex->depth; if (!dl_active() || D.done) return 0; @@ -1116,8 +1158,7 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { if (D.gbuf && !D.armed && !D.ended && !z->image && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && c->width >= D.gbuf_w && !c->level) D.nodepth = c->image; - if (D.nodepth && !D.armed && !D.ended && ex->pass_w && ex->pass_w * 4 <= D.gbuf_w && c->image && - scene_color(c->format)) { + if (D.nodepth && !D.armed && !D.ended && bloom_start(ex, D.gbuf_w)) { D.end_fence = ex->fence, D.end_pos = ex->pos; __atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE); } @@ -1130,9 +1171,9 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { // first pair to arm; DLSS then scaled the effects into the light target, which filled the lens with garbage in the // DLAA mode. if (D.gbuf && !D.ended && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && z->image && z->format == VK_FORMAT_D32_SFLOAT && - c->width == z->width && c->height == z->height && c->width >= DLAA_SCENE_MIN_W && c->width >= D.gbuf_w && + c->width == z->width && c->height == z->height && c->width >= DLAA_SCENE_MIN_W && c->width >= D.gbuf_w && scene_shaped(c) && (!D.armed || (c->width >= D.color.width && c->height >= D.color.height)) && - c->samples <= 1 && !c->is3d) { + c->samples <= 1 && !c->is3d && scene_large(c)) { int first = !D.armed; D.color = *c, D.depth = *z, D.armed = 1, D.nodepth = VK_NULL_HANDLE; if (!first) return 0; @@ -1141,11 +1182,11 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { } if (!D.armed) return 0; if (!D.ended) { - // The bloom chain's first pass ends the scene: a pass at most a quarter of the scene's width into a scene - // colour target. With the camera rune or the scope open the game draws a 160-wide depth-only pass and 160x90 - // RG16F and RGBA8 passes right after the lighting, then more geometry and the transparents; DLSS ended the - // scene at those, the next G-buffer pass disarmed it and the scaling pass found nothing to evaluate. - if (!ex->pass_w || ex->pass_w * 4 > D.color.width || !c->image || !scene_color(c->format)) return 0; + // The bloom chain's first pass ends the scene (bloom_start). With the camera rune or the scope open the game + // draws a 160-wide depth-only pass and 160x90 RG16F and RGBA8 passes right after the lighting, then more + // geometry and the transparents; DLSS ended the scene at those, the next G-buffer pass disarmed it and the + // scaling pass found nothing to evaluate. + if (!bloom_start(ex, D.color.width)) return 0; D.end_fence = ex->fence, D.end_pos = ex->pos; __atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE); unsigned o[2], s[2]; diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 6c3504f0..9b959184 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -991,7 +991,7 @@ static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) { // - GPU -> guest: the replay notes each GPU write into CPU-visible memory at record time (vk_mem_gpu_pending) and // reports it at the fence (vk_gpu_written), which copies those bytes back into guest memory. Until then the CPU // copies skip them: a page the CPU and the GPU both write keeps the GPU's bytes. -// TOTK_SHADOW=1/0 forces the mode on or off; the default is on for Linux and Wine. +// TOTK_SHADOW=1/0 (shadow in totk.ini) forces the mode on or off; the default is on for Linux and Wine. typedef struct { uint64_t lo, hi; } SRange; static PlatMutex shadow_lock = PLAT_MUTEX_INIT; // after pool_lock; before gen_lock static PoolRec **stouch; // shadow_lock: pools resolved since the last sync diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index 68b5a9b5..5bd74ecf 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -321,6 +321,24 @@ static void init_image(Tex *e, VkCommandBuffer cb) { } } +// The first transition of a new image, in the upload batch, which runs ahead of the submission being recorded. A chunk +// of a chunked replay records it into its own command buffer at its op instead when the image is bound into pool +// memory: the chunks before it may be on the GPU already, and a clear of the new image (its depth) there would land on +// what they rendered into the same storage for the chunks after them to read. The inventory and the quick weapon menu +// render the items' icons into new targets at the storage the frame's other targets share, and a black strip showed +// right of the quick menu for a frame where its new depth targets were cleared. +static void init_new(Tex *e) { + int reopen = 0; + VkCommandBuffer cb = e->bound_to && vk_exec_replay_pos() != UINT32_MAX ? vk_exec_inline_begin(&reopen) : VK_NULL_HANDLE; + if (cb) { + init_image(e, cb); + vk_exec_inline_end(reopen); + return; + } + init_image(e, vk_upload_begin()); + vk_upload_end(); +} + // TOTK_SHADOW_SCALE (1 to 4): the game's shadow maps are made this many times larger on each side. TotK renders its // sun cascades into a 960x960 DEPTH16 2D array of 4 layers and its static shadows into 1440x1440 and 720x720 arrays of 2, // whatever the output resolution, and shrinks the cascades under load (872, 848, 800, 480 and so on). Those are made @@ -505,10 +523,7 @@ static Tex *create_tex(const tex_t *t, const TexKey *k, int defer_init) { p->fb_usage = (uint8_t)fb_usage; p->scale = (uint8_t)scale; - if (!defer_init) { - init_image(e, vk_upload_begin()); - vk_upload_end(); - } + if (!defer_init) init_new(e); return e; } @@ -1435,8 +1450,7 @@ static Tex *entry_of(const tex_t *t) { e->pub.gpu_owned = 1; } } else { - init_image(e, vk_upload_begin()); - vk_upload_end(); + init_new(e); if (alias) { if (e->pub.mirrored) n_alias_upload++; else n_alias_uninit++; diff --git a/totk.ini.example b/totk.ini.example index 86c87046..1ad08738 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -57,9 +57,13 @@ # game waits once for the GPU to finish the frame it presented last; with 1 that wait covers the frame before it # instead. On Great Sky Island (Linux, RX 9070 XT) with shadow_defer = 1 this took about 73 to 100 fps. Costs one more # frame of input delay, and frame times vary more from frame to frame than with 0. GPU results the game reads back after that wait, for example occlusion queries, can be a frame old. -# Works on 1.2.1 and 1.4.3; 1.1.0 ignores it. Needs shadow storage, which is on by default on Linux and Wine and needs -# the environment variable TOTK_SHADOW=1 on native Windows; without it the setting is ignored and the log says so. +# Works on 1.2.1 and 1.4.3; 1.1.0 ignores it. Needs shadow storage (shadow below), which is on by default on Linux and +# Wine and needs shadow = 1 on native Windows; without it the setting is ignored and the log says so. # sync_ahead = 0 +# Shadow storage, 0 or 1: each CPU-visible memory pool of the game gets a host-visible copy that the GPU reads, in +# place of importing the game's memory. Default: on under Linux and Wine, off on native Windows. sync_ahead needs it. +# Costs about 0.2 ms a frame and doubles the memory of those pools (about 1.1 GB; docs/building.md). Read at start. +# shadow = 0 # With sync_ahead = 1: when the game changes memory the GPU still reads, queue the copy into the GPU's shadow of that # memory for the GPU to run in order, 0 or 1. Default: on. With 0 the replay thread waits for the GPU to finish # reading instead, which costs part of the gain.