mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-19 04:15:57 +00:00
With sync_ahead an exit line gives the run total of the waits at a pool invalidate, as the shadow log lines stop at frame 1800 without TOTK_TRACE=perf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bc579c62e8
commit
234a10f413
5 files changed
+20
-3
No files matched your search
@@ -44,7 +44,7 @@ GPU semantics behind the graphics rows: [maxwell.md](../maxwell.md). The method
|
||||
| frame counter wanders 50-60 | two 60 Hz clocks (vsync event and present acquire) drifting apart | whether both follow one display tick |
|
||||
| a 30 Hz thread runs at 27 Hz, slow cutscenes | host sleep overshoot: Windows waitable timers wake 0.3-0.5 ms late | `TOTK_TRACE=sched` sleep asked versus took |
|
||||
| 20 fps after a scheduler change | a yield back-off that starts too early | yield handling against the console's |
|
||||
| in the Depths Link takes gloom damage and wears the gloom haze while standing on clean ground, some runs only; only with `sync_ahead = 1` | the game read a GPU result of the frame it was told is done before the GPU wrote it: its one-pixel readbacks go through a buffer per request, which then holds zeros or an answer from the ring's last round (fixed: `vk_memory.c` `vk_mem_shadow_read` waits at the invalidate) | `sync_ahead = 0`; the shadow log line's "the game waited for a GPU write it reads" count |
|
||||
| in the Depths Link takes gloom damage and wears the gloom haze while standing on clean ground, some runs only; only with `sync_ahead = 1` | the game read a GPU result of the frame it was told is done before the GPU wrote it: its one-pixel readbacks go through a buffer per request, which then holds zeros or an answer from the ring's last round (fixed: `vk_memory.c` `vk_mem_shadow_read` waits at the invalidate) | `sync_ahead = 0`; the exit line `mem: the game waited for a GPU write it reads` |
|
||||
| in shrines the lamps' reflections on the floor go out for single frames, every other frame at some camera angles; only with `sync_ahead = 1`, more with DLAA (the GPU runs further behind) | the game reads its depth query results two frames later and refuses them unless their frame number is exactly the counter - 2; with sync_ahead it is the counter - 3 on frames where the GPU is behind (fixed: patch set `qstamp` in `recomp/patches-v143.txt` also takes the counter - 3) | `sync_ahead = 0`; a `--capture` run with `TOTK_SYNC_AHEAD` on, counting one-frame drops in floor regions |
|
||||
| climbing or movement wrong only above some fps | game code that assumes a fixed step count per frame | compare at 30, 60 and the failing rate |
|
||||
|
||||
|
||||
@@ -90,7 +90,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app
|
||||
- Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`).
|
||||
- Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on.
|
||||
- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's.
|
||||
- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.1.0, 1.2.1 and 1.4.3). GPU writes into pool storage come back with their fence, so a buffer the game reads back every frame after that wait holds the result of the frame before; a buffer it has not read lately holds none, and its `nvnMemoryPoolInvalidateMappedRange` waits for the write (`vk_mem_shadow_read`): TotK reads single pixels through a buffer per request, and in the Depths zeros in one of them put Link in gloom. The fence write-back of small render targets (`vk_texture.c`, `write_back_n`) does not wait for the GPU with `TOTK_WB_EAGER=1` (`wb_eager`, the default, but only while frame-ahead is in effect: with `sync_ahead` off or refused the setting does nothing and the write-back reads the image when it runs): `vk_sync_submit` has `vk_texture_eager_record` copy the targets that value rendered into staging buffers inside the same command buffer, so once the value has passed the write-back only copies from staging into the target's storage. Only a target that was written back within the last 64 timeline values is copied (the first write-back of a target takes the blocking path), and the staging of a target past that window, no longer wanted or released goes through the deferred free (`vk_defer_buffer_until`). A snapshot is used only when its value is the one that last rendered the target, nothing recorded a write to the image since (`wr_fence`) and the GPU has finished it; otherwise the write-back reads the image as before, and the exit line `vk: wb_eager:` counts the reasons. With frame-ahead in effect the exit line `vk: fence write-backs waited for the GPU` counts the write-backs that waited for a copy and the time they waited, for every value of the setting. `check` also reads the old way and compares.
|
||||
- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.1.0, 1.2.1 and 1.4.3). GPU writes into pool storage come back with their fence, so a buffer the game reads back every frame after that wait holds the result of the frame before; a buffer it has not read lately holds none, and its `nvnMemoryPoolInvalidateMappedRange` waits for the write (`vk_mem_shadow_read`): TotK reads single pixels through a buffer per request, and in the Depths zeros in one of them put Link in gloom. The exit line `mem: the game waited for a GPU write it reads` counts these waits and their time. On 1.4.3 the patch set `qstamp` (`recomp/patches-v143.txt`) lets the game take its depth query results one frame older; it follows the switch and is off when the switch is refused. The fence write-back of small render targets (`vk_texture.c`, `write_back_n`) does not wait for the GPU with `TOTK_WB_EAGER=1` (`wb_eager`, the default, but only while frame-ahead is in effect: with `sync_ahead` off or refused the setting does nothing and the write-back reads the image when it runs): `vk_sync_submit` has `vk_texture_eager_record` copy the targets that value rendered into staging buffers inside the same command buffer, so once the value has passed the write-back only copies from staging into the target's storage. Only a target that was written back within the last 64 timeline values is copied (the first write-back of a target takes the blocking path), and the staging of a target past that window, no longer wanted or released goes through the deferred free (`vk_defer_buffer_until`). A snapshot is used only when its value is the one that last rendered the target, nothing recorded a write to the image since (`wr_fence`) and the GPU has finished it; otherwise the write-back reads the image as before, and the exit line `vk: wb_eager:` counts the reasons. With frame-ahead in effect the exit line `vk: fence write-backs waited for the GPU` counts the write-backs that waited for a copy and the time they waited, for every value of the setting. `check` also reads the old way and compares.
|
||||
- Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread).
|
||||
- Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame
|
||||
that reads them, while the replay still records it. At the start of a submission's replay every uniform range it
|
||||
|
||||
@@ -304,6 +304,7 @@ void vk_mem_shadow_sync(uint64_t fence); // fence: the value of the part about
|
||||
// vk_mem_mirror_prologue's copies, which read the shadow.
|
||||
int vk_mem_shadow_deferred(void);
|
||||
void vk_mem_shadow_prologue(VkCommandBuffer cb, uint64_t fence);
|
||||
void vk_mem_shadow_read_log(void); // at exit, with TOTK_SYNC_AHEAD: the run total of vk_mem_shadow_read's waits
|
||||
void vk_mem_gpu_pending(uint64_t addr, uint64_t size);
|
||||
void vk_gpu_written(uint64_t addr, uint64_t size);
|
||||
int vk_mem_flush_add_probe(void); // --probe-vkexec: the flush range list; 0 or the number of the failed check
|
||||
|
||||
@@ -339,6 +339,7 @@ void gfx_exit_log(void) {
|
||||
static int32_t done;
|
||||
rec_check_log();
|
||||
vk_texture_eager_log();
|
||||
vk_mem_shadow_read_log();
|
||||
if (gfx_replay_async() && r_started && !plat_xchg32(&done, 1)) replay_log(game_frames, 1);
|
||||
}
|
||||
|
||||
|
||||
+16
-1
@@ -1319,6 +1319,7 @@ void vk_mem_shadow_fence(void) {
|
||||
typedef struct { uint64_t addr, size, at; } SRead;
|
||||
static SRead sread[SREAD_N]; // shadow_lock: the ranges the game invalidated last, with the value issued then
|
||||
static uint64_t s_read_waits, s_read_ticks; // shadow_lock: since the last log
|
||||
static uint64_t r_read_waits, r_read_ticks, r_read_max; // shadow_lock: over the run (vk_mem_shadow_read_log)
|
||||
|
||||
void vk_mem_shadow_read(uint64_t addr, uint64_t size) {
|
||||
if (!shadow_on || !size || !nvn_sync_ahead_guard()) return;
|
||||
@@ -1342,8 +1343,22 @@ void vk_mem_shadow_read(uint64_t addr, uint64_t size) {
|
||||
NVN_BLOCKING();
|
||||
vk_sync_fence_wait(wait, UINT64_MAX);
|
||||
}
|
||||
uint64_t dt = qpc() - t0;
|
||||
plat_mutex_lock(&shadow_lock);
|
||||
s_read_waits++, s_read_ticks += qpc() - t0;
|
||||
s_read_waits++, s_read_ticks += dt;
|
||||
r_read_waits++, r_read_ticks += dt;
|
||||
if (dt > r_read_max) r_read_max = dt;
|
||||
plat_mutex_unlock(&shadow_lock);
|
||||
}
|
||||
|
||||
// The run total of the waits above, once at exit, while frame-ahead is in effect: without TOTK_TRACE=perf the shadow
|
||||
// log lines come only at frames 600 and 1800.
|
||||
void vk_mem_shadow_read_log(void) {
|
||||
static int32_t done;
|
||||
if (!shadow_on || !nvn_sync_ahead_on() || plat_xchg32(&done, 1)) return;
|
||||
plat_mutex_lock(&shadow_lock);
|
||||
rt_log("mem: the game waited for a GPU write it reads %llu times, %.1f ms in all, longest %.2f ms",
|
||||
(unsigned long long)r_read_waits, qpc_ms(r_read_ticks), qpc_ms(r_read_max));
|
||||
plat_mutex_unlock(&shadow_lock);
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user