diff --git a/docs/architecture.md b/docs/architecture.md index 0c123786..17758b1f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,6 +85,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`). - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on. +- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's. - Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread). - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it diff --git a/docs/controls.md b/docs/controls.md index 9c14b771..0fd4d567 100644 --- a/docs/controls.md +++ b/docs/controls.md @@ -70,10 +70,10 @@ The Switch prompts stay, with a note under *Button prompts* and a `RomFS overlay | Left stick, its press | W A S D (or the keys of the directions meant), Ctrl | | Right stick, its press | the mouse, with an arrow for each direction meant, C | -- The glyphs keep the width of the game's own, so the right-aligned guides do not run into their text. A key is a rounded square with its label cut out in the game's own text fonts; a mouse button is a mouse with that button filled and the others hollow. +- The glyphs keep the width of the game's own, so the right-aligned guides do not run into their text. A key is a rounded square with its label cut out in the game's own text fonts (Shift: an arrow over its name); a mouse button is a mouse with that button filled and the others hollow. - The button guide in play draws every button from images of the Common archive (`Nt_KeyTexA/B/X/Y/L/ZL_00`, `Nt_KeyTexLSt_00` and `_01`, the stick tilted and pressed): all of them are painted with the same icons. The letters of the action prompt's diamond are one character of the Normal font each: E, an arrow for Shift, a bar for Space, L and R for the mouse buttons. - Like the Xbox prompts they follow the defaults and not the bindings. `--binds ` draws the first key of each `bind_` line of that file instead, for a folder made by hand. -- At a game resolution of another shape than 16:9 the game loads `romfs-patches-pc-x` in place of `romfs-patches-x` when it is there: the same prompts with that shape's UI layout, `make_romfs_patches.py --prompts pc --aspect x --out \romfs-patches-pc-x`. The setup does not write those, so without one the Switch prompts stay (*off at a game resolution not 16:9*). +- At a game resolution of another shape than 16:9 the game loads `romfs-patches-pc-x` in place of `romfs-patches-x`: the same prompts with that shape's UI layout, `make_romfs_patches.py --prompts pc --aspect x --reuse \romfs-patches-pc --out \romfs-patches-pc-x`. The setup writes one for each of the fifteen sizes it lays the UI out for, so on a display of one of those shapes the game resolution follows the display with these prompts too. At another size, without that folder, the Switch prompts stay (*off at a game resolution not 16:9*). ## Window keys diff --git a/docs/installer.md b/docs/installer.md index 46d8c5c9..5a8806ad 100644 --- a/docs/installer.md +++ b/docs/installer.md @@ -74,8 +74,11 @@ loads the saves 1.4.3 wrote. writes the translations it makes in play as loose files; running the tool again with the game closed folds those in. The patches are written four times: `romfs-patches`, `romfs-patches-xbox` and `romfs-patches-xbox-label` (`--prompts`: Xbox button prompts for `prompts = xbox`, the first for `pad_layout = nintendo`, the second for `xbox` - and `auto`), and `romfs-patches-pc` (keyboard and mouse prompts for `prompts = pc`). The three prompt folders are - optional: when one cannot be made, the setup warns and finishes. + and `auto`), and `romfs-patches-pc` (keyboard and mouse prompts for `prompts = pc`). The PC prompts are then laid + out once more for each game resolution of another shape, `romfs-patches-pc-x` (`--prompts pc --aspect`). + Those differ from `romfs-patches-pc` in the UI archive only, and `--reuse` hard-links the other files from it, so + the fifteen folders take seconds and a few MiB each. The prompt folders are optional: when one cannot be made, the + setup warns and finishes. When the graphics driver crashes while building the shaders, `prepare_shaders.py` runs that part again without the variants it was building ([building.md](building.md), step 6), and the setup finishes with a warning that names how many were left out. diff --git a/docs/settings.md b/docs/settings.md index bb2427f3..7d581507 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -77,6 +77,7 @@ Many rows below are developer tools that only act together with another setting | `probe_movie_bench` | `1` makes the movie probe also time the resampler that fits a movie to a game resolution of another shape, and print the time per frame. | Only used by the movie decoder probe. | Safe | | `swkbd_answer` | ``, `` or ``: answers the on-screen keyboard (for example a horse name) without opening a window. | The game gets that answer every time it asks for text. | Caution | | `swkbd_test` | `ok` or `cancel` opens the keyboard window and presses OK or Cancel on the first timer tick. | The keyboard window closes by itself. | Caution | +| `test_oom` | `:[,...]` fails the first n allocations at a site: `commit` (guest memory), `staging` (upload buffers), `image` (texture images), `chunk` (draw data buffers). | Each forced failure takes the out-of-memory path: a memory release, a wait of 10 ms per retry and log lines; a large n stalls up to 30 s and then ends the game. | Caution | | `test_pad` | `,,,` attaches a virtual gamepad whose sticks rest at these raw SDL axis values (-32768 to 32767). | The game sees a controller with that drift or deflection; a malformed value is logged and ignored. | Caution | | `test_prepare_shaders` | `1` translates every recorded shader on all usable CPUs during start. | Start takes longer and fills the shader cache folder; the CPU is fully used while it runs. | Caution | @@ -124,6 +125,7 @@ Many rows below are developer tools that only act together with another setting | `dyn_delta` (default `1`) | Records only the pipeline dynamic states (cull mode, depth and stencil tests, vertex input, blend enables, equations and write masks, ...) whose values changed since the last draw in the command buffer, also across pipeline changes; `0` records the full set at every change of pipeline state, as before; `ab` alternates with the `ab` blocks. | Fewer commands per pipeline change on the render and recorder threads; a driver that loses dynamic state across pipeline binds would draw with stale state. `TOTK_TRACE=perf` prints the calls and commands per submission. | Caution | | `feedback` | `0` samples a target that an earlier draw of the same pass wrote with no barrier; `restart` ends and restarts the pass behind a full barrier; otherwise the feedback-loop layout is used when the GPU supports it. | `0` brings back the old behaviour that the barrier fixed, so the picture can break in passes that read a target they draw into; `restart` costs extra passes. | Risky | | `flush_span` (default `0`) | `1` copies one span from the lowest to the highest flushed address of a pool, as an older version did. | Copies far more data to the GPU in crowded towns: an estimated 30-50 ms of GPU time per large copy over PCIe 3.0 x8. | Caution | +| `shadow_snap` (default `1`) | Shadow storage (Linux, Wine): `1` syncs the pools from per-submission write-watch snapshots, each 2 MiB region queried at most once per submission and shared with the texture checks; `0` queries each pool at every part of a submission, as before. | `1` cut the replay thread's write tracking from about 4.4 to 1.3 ms a frame on a Steam Deck; `0` is for comparison runs. Under Wine/Proton the default is `0` until it has been measured there; `1` turns it on. No effect where shadow storage is off (Windows by default). | Caution | | `freeze_idle` (default `60`) | Number of submissions a 2 MiB region stays unwritten before it is made read-only; `0` turns the freeze off. | A larger value keeps more regions writable and costs more write tracking; a write to a frozen region costs about 60 microseconds. | Caution | | `gpl_draw_build` (default `1`) | `0` leaves the missing shader libraries of a waiting draw to its background job instead of building them in parallel at once. | Draws whose pipeline is not ready are skipped meanwhile, until the job finishes. | Caution | | `lib_prebuild_keep` (default `0`) | Number of recorded graphics shader library variants that stay in memory after the boot prebuild. | Each kept library costs 140-460 KB of memory; the driver cache already makes a rebuild cheap. | Caution | @@ -180,11 +182,19 @@ Many rows below are developer tools that only act together with another setting | --- | --- | --- | --- | | `app_pool` | Size in bytes of the guest application memory pool; accepts decimal or `0x` hex. | The default depends on the resolution settings; `0` stops the start with a message, and too small a pool leaves the game without memory. | Risky | | `audio_off` | Any value mixes audio without opening an output device. | The game runs in silence and reports two channels; same as `audio = none`. | Caution | +| `mem_reserve_mib` (default `1536`) | MiB of system memory kept free for Windows, other programs and the game's own heap when the renderer sizes what it keeps in device memory, where device memory takes system memory (integrated GPUs, and discrete GPUs whose driver charges it to the process). | A larger value keeps fewer released textures and spare blocks when the system runs short; a smaller one lets the game use more before it holds back. | Caution | | `one_per_core` (default `0`) | `1` keeps the process on the first logical processor of each physical core; `ab` switches it with the `ab` blocks. | Busy threads stop sharing a core with a sibling thread, which can help or hurt on the CPU; it only applies when the CPU has hyperthreads. | Caution | | `reexec` | Counter of address-space retries that the runtime sets itself when it starts the process again. | The counter is cleared when the window is recreated; a value of 20 or more stops the start with a fatal error. | Risky | +| `replay_chunks` (default `0`) | Most chunks a submission is cut into when `replay_workers` is set; `0` uses twice the number of recording threads. | Fewer chunks leave threads idle at the end of a submission, more add scan and submit work per chunk; no effect while `replay_workers` is `0`. | Caution | +| `replay_chunk_draws` (default `128`) | Least draws a chunk of the chunked replay holds before it may end at a pass. | Small values make many short chunks, large ones leave big submissions on few threads; no effect while `replay_workers` is `0`. | Caution | +| `replay_state_check` (default `0`) | `1` compares the replay state after the last chunk of a chunked submission with the state of the scan and logs differences. | Adds a comparison per submission and log lines on a mismatch; no effect while `replay_workers` is `0`. | Caution | +| `replay_skew_us` (default `0`) | Debug: the even chunks of a chunked submission wait this many microseconds before they record, so the chunk after each one records first. | Shows a recording that depends on which thread gets to a chunk first in every submission; slows the replay; no effect with fewer than 2 `replay_workers`. | Caution | | `restart_wait` | Set by the runtime when it restarts itself; names the old process by id and creation time so the new process waits for it to exit. | Meant for the runtime's own use; the code does not describe a manual value. | Unknown | | `spin` (default `1`) | `0` makes a guest thread that polls memory keep its virtual core. | Polling threads can starve the thread whose item they wait for, so frames can stall. | Caution | | `spin_handoff_us` (default `50`) | Microseconds a polling wait lasts before it gives its virtual core to a waiting thread. | A shorter value hands cores over sooner; a longer one keeps them longer. | Caution | +| `tex_block_mib` (default `64`) | Block size in MiB of the pool that holds the game's textures; images of a quarter block or more get memory of their own. `0` puts textures in the default pools with the buffers. | Larger blocks hold more memory that no texture uses; smaller ones mean more allocations from the driver. | Caution | +| `tex_released_mib` (default `256`) | MiB of textures the game released that are kept for reuse; older ones beyond it are dropped. `0` keeps up to 2048 of them, whatever their size. | A larger value keeps more device memory and process commit; a smaller one makes the game create more textures again. | Caution | +| `vma_block_mib` (default `64`) | Block size in MiB that the Vulkan memory allocator takes from the driver for buffers and other images on heaps over 1 GiB; larger allocations than half a block get memory of their own. `0` uses the allocator's 256. | Larger blocks hold more memory that nothing uses; smaller ones mean more allocations from the driver. | Caution | ## Generated-code patches diff --git a/installer/steps.py b/installer/steps.py index 4db8a2b5..ba439a5f 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -8,7 +8,7 @@ with the bundled clang (hostcheck.c). Install folder: exefs/, romfs/ (romfs.bin, index.json), profile/, shared-fonts/ (Chinese and Korean text), totk.exe and its DLLs, nvngx_dlss.dll (with --dlss), romfs-index.bin, romfs-patches/, romfs-patches-xbox/, -romfs-patches-xbox-label/, romfs-patches-pc/, totk_xlate.exe, the +romfs-patches-xbox-label/, romfs-patches-pc/, romfs-patches-pc-x/, totk_xlate.exe, the shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json. Temp folder (/-): check/ (ExeFS of both inputs, the host check), unpacked/ (the NSP of an NSZ or XCZ input until the extraction is done), gen/ (C), build/ (objects), logs/. The cleanup keeps gen/, build/ and @@ -1097,7 +1097,7 @@ class Prepare(Step): # input_prompts_folder). Both are written on every install, so the menu choices work without another run of # the setup. Optional: when one fails, the game keeps the Switch prompts with that layout and the menu says the # files are missing, so the setup warns once and goes on. romfs-patches-pc likewise, with keyboard and mouse - # prompts for prompts = pc (16:9 only: another shape takes make_romfs_patches.py --prompts pc --aspect). + # prompts for prompts = pc. failed = None for style in ('xbox', 'xbox-label', 'pc'): out = dest / f'romfs-patches-{style}' @@ -1108,6 +1108,24 @@ class Prepare(Step): if code: remove(out) failed = failed or (code, log) + # The PC prompts once more for each shape of ASPECT_SIZES, romfs-patches-pc-x with that shape's UI layout + # (--prompts pc --aspect), which runtime/romfs.c overlay_init loads in place of romfs-patches-x: without + # it a player with prompts = pc on a screen that is not 16:9 chose between the prompts and the screen's shape. + # Only the Common archive differs from romfs-patches-pc; --reuse links the fonts, messages and size table from + # there instead of compressing them again, so the fifteen folders take seconds and a few MiB each. Optional + # like romfs-patches-pc, and skipped without it. + pc = dest / 'romfs-patches-pc' + for size in ASPECT_SIZES: + out = dest / f'romfs-patches-pc-{size}' + remove(out) + if not (pc / 'manifest.tsv').is_file(): + continue + code, log = run_tool(ctx, f'prepare-pc-{size}', layout.tools / 'make_romfs_patches.py', + ['--version', version_tag(game_version(ctx)), '--romfs', dest / 'romfs', '--out', out, + '--prompts', 'pc', '--aspect', size, '--reuse', pc]) + if code: + remove(out) + failed = failed or (code, log) if failed: rep.warning('warn.xbox_prompts_failed', code=failed[0], log=str(failed[1])) diff --git a/runtime/cfg.c b/runtime/cfg.c index 6136f26d..8bf596a9 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -45,22 +45,22 @@ static const char *const known[] = { "FLIGHT_SPIKE_MS", "FLIGHT_THUMBS", "FLUSH_SPAN", "FOV", "FOV_PATCH", "FPS60", "FPS_LIMIT", "FRAME_QUEUE", "FREEZE_IDLE", "FREE_CORES", "FULLSCREEN", "FXAA", "GPL", "GPL_DRAW_BUILD", "GPU", "GPU_INDEX", "GPU_NAME", "HDR", "HDR_GAME_NITS", "HDR_HUE", - "HDR_PEAK_NITS", "HDR_UI_NITS", "HIGHFPS_FIX", "INPUT", "INPUT_RECORD", "INPUT_SCRIPT", "LANG", "LIB_PREBUILD", + "HDR_PEAK_NITS", "HDR_UI_NITS", "HIGHFPS_FIX", "INPUT", "INPUT_RECORD", "INPUT_SCRIPT", "KSPIN_US", "LANG", "LIB_PREBUILD", "LIB_PREBUILD_KEEP", "LIB_PREBUILD_THREADS", "LOCALE", "LOD_GUARD", "LSTICK_DEADZONE", "MEMORY", "MENU_LANG", "MENU_TEST", - "MENU_CURSOR", "MENU_CURSOR_LOG", "MENU_TEST_DIR", "MENU_TEST_SET", "MINIMAL_LAYOUT", "MIRROR_GAP", "MODS_DIR", "MODS_DISABLED", "MODS_ORDER", "MODS_REBUILD", "MOTION", "MOUSE_AIM_SCALE", "MOUSE_MODE", "MOUSE_DIRECT_PATCH", "MOUSE_SENS", "MOVIE_ASPECT", "MOVIE_HOST", "MOVIE_PACE", + "MEM_RESERVE_MIB", "MENU_CURSOR", "MENU_CURSOR_LOG", "MENU_TEST_DIR", "MENU_TEST_SET", "MINIMAL_LAYOUT", "MIRROR_GAP", "MODS_DIR", "MODS_DISABLED", "MODS_ORDER", "MODS_REBUILD", "MOTION", "MOUSE_AIM_SCALE", "MOUSE_MODE", "MOUSE_DIRECT_PATCH", "MOUSE_SENS", "MOVIE_ASPECT", "MOVIE_HOST", "MOVIE_PACE", "MVOBJ_ALL", "MVOBJ_BANKS", "MVOBJ_EQUAL", "MVOBJ_EXACT", "MVOBJ_EXEC", "MVOBJ_NOC4", "MVOBJ_ONLY", "MVOBJ_TIMES", "MVPREV_CHECK", "MVPREV_GROW", "MVPREV_SKIP", "MVPREV_TRUST", "MV_CHECK", "MV_SURVEY", - "ONE_PER_CORE", "OPERATION_MODE", "OUTPUT_RES", "PAD_LAYOUT", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_SCENE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", + "ONE_PER_CORE", "OPERATION_MODE", "OUTPUT_RES", "PAD_LAYOUT", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_SCENE", "PCORE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", "PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_BENCH", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES", - "PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_END", "REEXEC", - "RENDER_SCALE", "REPLAY_ASYNC", + "PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_ASYNC", "REC_END", "REC_SPIN_US", "REEXEC", + "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHUNKS", "REPLAY_CHUNK_DRAWS", "REPLAY_SKEW_US", "REPLAY_STATE_CHECK", "REPLAY_WORKERS", "REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", - "SHADER_MEMO", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", + "SHADER_MEMO", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHADOW_SNAP", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "STICK_SENS", "SWKBD_ANSWER", - "SWKBD_TEST", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_HEAP_EXTRA_MB", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UB_OVERLAY", "UB_OVERLAY_OUT", "UPSCALER", "VB_USED", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64", - "VK_PRESENT", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VRAM_LIMIT_MIB", "VS_OUTS", + "SWKBD_TEST", "TEST_OOM", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_BLOCK_MIB", "TEX_HEAP_EXTRA_MB", "TEX_MEMO", "TEX_RELEASED_MIB", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UB_OVERLAY", "UB_OVERLAY_OUT", "UPSCALER", "VB_USED", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64", + "VK_PRESENT", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VMA_BLOCK_MIB", "VOLUME", "VRAM_HEADROOM", "VRAM_LIMIT_MIB", "VS_OUTS", "WALK_KEY", "WALK_SCALE", "WB_EXACT", "WB_FENCE_LIMIT", "WB_MERGE", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_KEEP", "XLATE_PROCS", "ZSTD_CHECK", "ZSTD_NATIVE", }; @@ -95,11 +95,11 @@ static const struct { const char *key, *section; } sections[] = { {"BIND_RIGHT", "input"}, {"BIND_DOWN", "input"}, {"BIND_LSTICK_LEFT", "input"}, {"BIND_LSTICK_UP", "input"}, {"BIND_LSTICK_RIGHT", "input"}, {"BIND_LSTICK_DOWN", "input"}, {"BIND_RSTICK_LEFT", "input"}, {"BIND_RSTICK_UP", "input"}, {"BIND_RSTICK_RIGHT", "input"}, {"BIND_RSTICK_DOWN", "input"}, {"BIND_SHAKE", "input"}, - {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"FREE_CORES", "performance"}, + {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"FREE_CORES", "performance"}, {"TIMER_1MS", "performance"}, {"PIPE_WAIT", "performance"}, {"PIPE_SKIP", "performance"}, {"LIB_PREBUILD", "performance"}, {"LIB_PREBUILD_THREADS", "performance"}, {"PIPE_PREBUILD", "performance"}, {"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"}, - {"TEX_ALIAS", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"}, + {"TEX_ALIAS", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"TEX_RELEASED_MIB", "performance"}, {"TEX_BLOCK_MIB", "performance"}, {"VMA_BLOCK_MIB", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"}, {"BIND_SKIP", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"TEX_MEMO", "performance"}, {"VB_USED", "performance"}, {"DESC_TEMPLATE", "performance"}, {"PUSH_SKIP", "performance"}, {"SPLIT_DRAWS", "performance"}, {"SPLIT_US", "performance"}, {"VK_VALIDATION", "performance"}, {"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"}, @@ -164,6 +164,9 @@ static const CfgMeta metas[] = { {"DISCORD_IMAGE", "totk_512x512", NULL, LIVE}, // Advanced: scheduling {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, + {"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_CHUNKS", "0", "0..64", RESTART}, + {"REPLAY_CHUNK_DRAWS", "128", "1..100000", RESTART}, {"REPLAY_STATE_CHECK", "0", "0|1", RESTART}, + {"REPLAY_SKEW_US", "0", "0..100000", RESTART}, {"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART}, // shaders and pipelines {"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "fullscreen", "strict|any|fullscreen", RESTART}, @@ -171,7 +174,8 @@ static const CfgMeta metas[] = { {"PIPE_PREBUILD", "1", "0|1", RESTART}, {"PREWARM_XLATE", "", "1..64", RESTART}, {"GPL", "1", "0|1", RESTART}, // textures and memory {"ASTC", "auto", "auto|gpu|native|cpu", RESTART}, {"MEMORY", "auto", "auto|import|mirror|verify", RESTART}, - {"TEX_ALIAS", "1", "0|1", RESTART}, {"TEX_VRAM_PRESSURE", "90", "0..100", RESTART}, {"ASTC_BC", "", "0|1", RESTART}, {"ZSTD_NATIVE", "1", "0|1", RESTART}, {"ZSTD_CHECK", "0", "0|1", RESTART}, + {"TEX_ALIAS", "1", "0|1", RESTART}, {"TEX_VRAM_PRESSURE", "90", "0..100", RESTART}, + {"TEX_RELEASED_MIB", "256", "0..65536", RESTART}, {"TEX_BLOCK_MIB", "64", "0..1024", RESTART}, {"VMA_BLOCK_MIB", "64", "0..1024", RESTART}, {"ASTC_BC", "", "0|1", RESTART}, {"ZSTD_NATIVE", "1", "0|1", RESTART}, {"ZSTD_CHECK", "0", "0|1", RESTART}, // render fixes {"LOD_GUARD", "1", "0|1", RESTART}, {"FIX_C0", "1", "0|1", RESTART}, {"FIX_ZERO", "1", "0|1", RESTART}, {"BIND_SKIP", "1", "0|1", RESTART}, {"SHADER_MEMO", "1", "0|1", RESTART}, {"PIPE_MEMO", "1", "0|1", RESTART}, {"TEX_MEMO", "1", "0|1", RESTART}, @@ -338,8 +342,10 @@ static const char *raw_get(const char *name) { // picture between black bars. The sizes are the ones the installer lays the HUD out for (installer/steps.py // ASPECT_SIZES, ui/menu.cpp row_output_res): of the display's shape within 1%, the smallest at least 1080 high that fits // the display, else the largest that fits. A 16:9 display, an unknown one or one of another shape keeps 1920x1080. -// Xbox or keyboard prompts keep 1920x1080 too: the setup builds their folders for 16:9 only (romfs.c overlay_init), so -// a default of another shape would trade the prompts the player chose for the Switch ones. +// Xbox prompts keep 1920x1080 too: the setup builds their folders for 16:9 only (romfs.c overlay_init), so a default of +// another shape would trade the prompts the player chose for the Switch ones. Keyboard prompts have a folder for each +// of these sizes (romfs-patches-pc-x, installer/steps.py Prepare): they keep 1920x1080 only when the one for +// the display's size is missing, as in a source build without make_romfs_patches.py --prompts pc --aspect. static const unsigned SHAPED[][2] = { {2560, 1080}, {3440, 1440}, {3840, 1600}, {5120, 2160}, // 21:9 {3840, 1080}, {5120, 1440}, // 32:9 @@ -347,10 +353,14 @@ static const unsigned SHAPED[][2] = { {1920, 1280}, {2160, 1440}, {2256, 1504}, {2880, 1920}, {3000, 2000}, // 3:2 }; static char screen_res[16]; // set once by cfg_init, "" for none +static int screen_pc; // romfs-patches-pc- has a manifest (set once by cfg_init) -static int switch_prompts(void) { +// Whether the prompts in effect have their folder at screen_res. The same picks as input_map.c input_prompts_pick: an +// unknown or empty value is the Nintendo prompts, an unset one Xbox while the pad layout is "xbox_prompts". +static int shape_keeps_prompts(void) { const char *p = raw_get(PREFIX "PROMPTS"), *l = raw_get(PREFIX "PAD_LAYOUT"); - return (!p || !strcasecmp(p, "nintendo")) && !(l && !strcasecmp(l, "xbox_prompts")); + if (p && *p) return !strcasecmp(p, "pc") ? screen_pc : strcasecmp(p, "xbox") != 0; + return !(l && !strcasecmp(l, "xbox_prompts")); } static void screen_default(void) { @@ -368,7 +378,8 @@ static void screen_default(void) { if (best < 0) best = fit; if (best < 0) return; snprintf(screen_res, sizeof screen_res, "%ux%u", SHAPED[best][0], SHAPED[best][1]); - if (!raw_get(PREFIX "OUTPUT_RES") && switch_prompts()) + screen_pc = romfs_shaped_pc_ready(SHAPED[best][0], SHAPED[best][1]); + if (!raw_get(PREFIX "OUTPUT_RES") && shape_keeps_prompts()) rt_log("cfg: the display is %dx%d: output_res defaults to %s, its shape (totk.ini output_res sets another)", sw, sh, screen_res); } @@ -377,7 +388,8 @@ const char *cfg_get(const char *name) { const char *v = raw_get(name); if (v || strncmp(name, PREFIX, PREFIX_LEN)) return v; if (!profile_of()) - return screen_res[0] && !strcasecmp(name + PREFIX_LEN, "OUTPUT_RES") && switch_prompts() ? screen_res : NULL; + return screen_res[0] && !strcasecmp(name + PREFIX_LEN, "OUTPUT_RES") && shape_keeps_prompts() ? screen_res + : NULL; for (size_t i = 0; i < sizeof DECK / sizeof *DECK; i++) if (!strcasecmp(DECK[i].key, name + PREFIX_LEN)) return DECK[i].value; return NULL; diff --git a/runtime/gfx_present.c b/runtime/gfx_present.c index 0f7a29da..f936b721 100644 --- a/runtime/gfx_present.c +++ b/runtime/gfx_present.c @@ -33,9 +33,20 @@ #define VSYNC_HZ 60 // ---- queue lock +// The recorder queues the parts it records and submits them later (vk_rec.c rec_submit): the first lock of a thread +// waits until the parts queued before it are submitted, so whatever it puts on the queue comes after them, as when +// the submitting thread submitted them itself. Not inside a held lock: the recorder takes this lock to submit. static PlatRMutex g_ql; -void vk_queue_lock(void) { plat_rmutex_lock(&g_ql); } -void vk_queue_unlock(void) { plat_rmutex_unlock(&g_ql); } +static _Thread_local int ql_depth; +void vk_queue_lock(void) { + if (!ql_depth) rec_queue_wait(); + plat_rmutex_lock(&g_ql); + ql_depth++; +} +void vk_queue_unlock(void) { + ql_depth--; + plat_rmutex_unlock(&g_ql); +} // ---- window state shared with the back end static VkSurfaceKHR g_surface; diff --git a/runtime/host.h b/runtime/host.h index 06c17b11..f0ed8835 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -36,6 +36,10 @@ void perf_end(int kind, uint64_t t0); // counts render-thread time onl void perf_replay(uint64_t t0); // replay (NVN -> Vulkan recording) time on any thread, waits inside it included void perf_replay_thread(void); // the calling thread replays for the render thread: perf_end counts its waits void perf_main_thread(void); // the calling thread is the guest main thread (TOTK_PROFILE_MAIN) +// The calling thread is one every frame goes through: on a hybrid CPU it prefers the P-cores (main.c, TOTK_PCORE); +// the first thread of each role logs it. +enum { PCORE_MAIN, PCORE_RENDER, PCORE_REPLAY, PCORE_RECORDER, PCORE_PRESENT, PCORE_N }; +void perf_core_thread(int role); // Events per 60-frame window: graphics pipelines finished by the background workers, draws skipped while pending, // first texture uploads and re-uploads after CPU writes, draws skipped while a shader translates in the background, // draws that waited for their pending pipeline and got it, draws into a target the previous frame did not render that @@ -171,6 +175,23 @@ int mem_guest_readable(uint64_t a, uint64_t n); // [a, a+n) can be re // module or the committed heap answered from the layout, // anything else by plat_mem_readable (an OS query) +// ---- short out-of-memory spells (oom.c) +enum { RT_OOM_COMMIT, RT_OOM_STAGING, RT_OOM_IMAGE, RT_OOM_CHUNK, RT_OOM_SITES }; +extern volatile int32_t rt_oom_req; // set by rt_oom_request; vk_texture_new_submit takes it +void rt_oom_request(void); // asks the Vulkan back end to give memory back at its next + // submit (released textures, headroom blocks); any thread, + // any lock held +int rt_oom_wait(const char *what, uint64_t bytes, int (*retry)(void *arg), void *arg); + // after a failed allocation: requests a release, then calls + // retry every 10 ms for up to 30 s; 1 once retry returned 1, + // 0 after the wait (the caller ends the game). Logs the + // first 8 with the commit numbers +void *rt_realloc_wait(void *old, size_t n, const char *what); // realloc, NULL only after rt_oom_wait gave up +int rt_oom_forced(int site); // 1: fail this attempt at site (TOTK_TEST_OOM=:, + // sites commit, staging, image, chunk; or rt_oom_force) +void rt_oom_force(int site, int n); // probes: fail the next n attempts at site +int rt_oom_forced_left(int site); // forced failures still to come at site + // ---- modules (loader.c) typedef struct { const char *name; uint64_t base, size, text_end, data_off; // text_end, data_off: offsets from base @@ -280,6 +301,7 @@ int res_is_16x9(unsigned w, unsigned h); // within 1% of 16:9 int res_wants_wide_overlay(const unsigned out[2]); // not 16:9 on a game version with the camera aspect patch int romfs_shaped_ready(unsigned w, unsigned h); // romfs-patches-x has a manifest (romfs.c; works // before fs_init) +int romfs_shaped_pc_ready(unsigned w, unsigned h); // the same for romfs-patches-pc-x void res_wanted(unsigned out[2]); // the output size TOTK_OUTPUT_RES asks for void res_fov_live(void); // TOTK_FOV to the camera (set fov); on a settings change void res_dynres_live(void); // TOTK_DYNAMIC_RES to the game (set fixres); on a settings change diff --git a/runtime/kernel.c b/runtime/kernel.c index e8fb48b4..4b5114c1 100644 --- a/runtime/kernel.c +++ b/runtime/kernel.c @@ -246,6 +246,44 @@ static void wake_sync_waiters(void) { } } +// ---- a short poll before a sleep +// Most guest waits and waits for a virtual core end within microseconds. The thread polls for TOTK_KSPIN_US (default +// 20) with G released before it sleeps on its condition variable: a wake in that time costs no OS wake-up (4-12 us +// on a desktop Ryzen, more from a deep C-state) and the waker's signal finds nobody asleep. Off with fewer than 6 logical +// processors, where the poll would take a CPU that a running thread needs; TOTK_KSPIN_US=0 turns it off. Savage Wind +// Recomp polls the same way before it parks (runtime/ksync.c cond_wait). TOTK_KSPIN_US=ab polls 20 us in the +// B blocks of TOTK_AB only. +static int64_t g_kspin_ns = -1; +static int g_kspin_ab; + +static uint64_t kspin_ns(void) { + int64_t v = __atomic_load_n(&g_kspin_ns, __ATOMIC_RELAXED); + if (v < 0) { + const char *e = cfg_get("TOTK_KSPIN_US"); + g_kspin_ab = e && !strcmp(e, "ab"); + long long us = e && *e && !g_kspin_ab ? atoll(e) : 20; + uint32_t cpus = plat_cpu_usable(); + v = us <= 0 || cpus < 6 ? 0 : us * 1000; + __atomic_store_n(&g_kspin_ns, v, __ATOMIC_RELAXED); + rt_log("kernel: waits poll %lld us before they sleep%s (TOTK_KSPIN_US, %u logical processors)", + (long long)(v / 1000), g_kspin_ab ? " in the B blocks" : "", cpus); + } + return v && (!g_kspin_ab || rt_ab()) ? (uint64_t)v : 0; +} + +// Holding G: releases it and polls *p until it reaches at_least or ns pass, then takes G again. 1 if it was reached. +static int kspin(const int *p, int at_least, uint64_t ns) { + uint64_t end = kern_now_ns() + ns; + int ok = 0; + unlock(); + for (unsigned i = 1; !(ok = __atomic_load_n(p, __ATOMIC_ACQUIRE) >= at_least); i++) { + plat_pause(); + if (!(i & 31) && kern_now_ns() >= end) break; + } + lock(); + return ok; +} + // SleepConditionVariableSRW counts milliseconds: round up (late is legal, early is not) and re-check the deadline // against the performance counter, because the wait may also end early. static uint32_t block_inner(RtThread *t, int kind, int64_t ns) { @@ -262,6 +300,8 @@ static uint32_t block_inner(RtThread *t, int kind, int64_t ns) { uint64_t now = kern_now_ns(); int forever = ns < 0 || (uint64_t)ns > UINT64_MAX - now; uint64_t deadline = forever ? 0 : now + (uint64_t)ns; + uint64_t poll = kspin_ns(); + if (poll) kspin(&t->woke, 1, forever || poll < (uint64_t)ns ? poll : (uint64_t)ns); while (!t->woke) { if (forever) { plat_cond_wait(&t->cv, &G); @@ -991,6 +1031,9 @@ static void core_take(RtThread *t) { while (__atomic_load_n(&t->cur_core, __ATOMIC_RELAXED) < 0 && kern_now_ns() < end) plat_pause(); lock(); if (t->cur_core >= 0) g_ap_spin_ok++; + } else if (t->cur_core < 0) { // a holder may block or yield any moment (kspin_ns) + uint64_t poll = kspin_ns(); + if (poll) kspin(&t->cur_core, 0, poll); } int sampled = 0; while (t->cur_core < 0) { @@ -1518,6 +1561,7 @@ uint32_t kern_main_thread(Cpu **cpu) { t->cpu->sp = t->stack_top; t->tid = plat_thread_id(); perf_main_thread(); + perf_core_thread(PCORE_MAIN); PlatIntr *self = ap_open_self(); lock(); t->hthr = self; diff --git a/runtime/kernel_probe.c b/runtime/kernel_probe.c index 191a1c28..83f11101 100644 --- a/runtime/kernel_probe.c +++ b/runtime/kernel_probe.c @@ -682,6 +682,25 @@ static int cmp_u64(const void *a, const void *b) { return x < y ? -1 : x > y; } +// A refused commit waits and retries (memory.c commit, oom.c rt_oom_wait): three forced refusals, then the page maps. +static void test_oom_commit(void) { + const RtGuestLayout *g = mem_layout(); + uint64_t a = g->alias.base + g->alias.size - 0x10000; + rt_oom_force(RT_OOM_COMMIT, 3); + uint64_t t0 = plat_now_ms(); + uint32_t r = mem_map_physical(a, 0x1000); + uint64_t ms = plat_now_ms() - t0; + CHECK(r == 0 && rt_oom_forced_left(RT_OOM_COMMIT) == 0 && ms >= 20, + "MapPhysicalMemory after 3 refused commits: result %#x, %d refusals left, %llu ms", r, + rt_oom_forced_left(RT_OOM_COMMIT), (unsigned long long)ms); + rt_oom_force(RT_OOM_COMMIT, 0); + if (r == 0) { + *(volatile uint64_t *)(uintptr_t)a = 1; + CHECK(mem_unmap_physical(a, 0x1000) == 0, "UnmapPhysicalMemory after the refused commits"); + } + printf("oom: commit waited %llu ms through 3 refusals\n", (unsigned long long)ms); +} + static void test_platform(void) { PlatThread *t = plat_thread_new(plat_worker, (void *)(intptr_t)1, "probe-kernel plat"); CHECK(t, "plat_thread_new"); @@ -719,6 +738,49 @@ static void test_platform(void) { CHECK(over[N / 2] < 500000, "plat_sleep_ns(1 ms) median overshoot %.3f ms (want < 0.5)", over[N / 2] / 1e6); } +// ---- crash report (main.c crash_handler): a child faults on a guest thread with known values in rbx and x19; the +// report must hold the host registers, the faulting instruction's bytes and the memory both values point at. +#define CRASH_CHILD_ENV "TOTK_PROBE_KERNEL_CRASH_CHILD" +static uint64_t crash_host_words[6] = {0, 0, 0x5ca1ab1e0c0de001ull, 0, 0, 0}; +static uint64_t crash_guest_words[6] = {0, 0, 0x5ca1ab1e0c0de002ull, 0, 0, 0}; + +static void crash_child_main(Cpu *c) { + c->x[19] = (uint64_t)(uintptr_t)&crash_guest_words[2]; + c->sp -= 0x100; // a frame on the guest stack, as a crashing function has + __asm__ volatile("movq %0, %%rbx\n\tmovq 0x10, %%rax" : : "r"(&crash_host_words[2]) : "rax", "rbx", "memory"); +} + +// The report line that holds needle, NULL when none does +static const char *report_line(const char *out, const char *needle, char *line, size_t cap) { + const char *p = strstr(out, needle); + if (!p) return NULL; + while (p > out && p[-1] != '\n') p--; + size_t n = strcspn(p, "\r\n"); + snprintf(line, cap, "%.*s", (int)(n < cap ? n : cap - 1), p); + return line; +} + +static void test_crash_report(void) { + static char out[65536]; + char line[512], want[64]; + int code = plat_run_self(CRASH_CHILD_ENV, "1", out, sizeof out, 30000); + CHECK(code == 71 && strstr(out, "FATAL: access violation") && strstr(out, "reading address 0x10"), + "crash child: exit code %d, no access violation at 0x10 reported", code); + snprintf(want, sizeof want, "rbx %016llx", (unsigned long long)(uintptr_t)&crash_host_words[2]); + CHECK(strstr(out, want), "crash report: no host register line with %s", want); + CHECK(report_line(out, "host code at rip-", line, sizeof line) && strstr(line, "| 48 8b 04 25 10 00 00 00"), + "crash report: the faulting instruction's bytes are missing"); + const char *h = report_line(out, "*5ca1ab1e0c0de001", line, sizeof line); + CHECK(h && strstr(h, "rbx"), "crash report: no memory line for rbx (%s)", h ? h : "none"); + const char *g = report_line(out, "*5ca1ab1e0c0de002", line, sizeof line); + CHECK(g && strstr(g, "x19"), "crash report: no memory line for x19 (%s)", g ? g : "none"); + CHECK(strstr(out, "guest stack sp+0x00:"), "crash report: no guest stack words"); + CHECK(report_line(out, " memory commit: this process ", line, sizeof line) && !strstr(line, "process -1 MB"), + "crash report: no memory commit line"); + printf("crash report: exit code %d, host registers, code bytes, memory at rbx and x19, guest stack, commit %s\n", code, + failures ? "checked with failures" : "present"); +} + int probe_kernel(void) { a_lock_waiter = hostfn_register("probe-kernel lock waiter", lock_waiter); a_cv_waiter = hostfn_register("probe-kernel condvar waiter", cv_waiter); @@ -742,7 +804,15 @@ int probe_kernel(void) { printf("probe-kernel: FAILED (no PLT stub for %s in main)\n", TICK_SYMBOL); return 1; } + const char *child = getenv(CRASH_CHILD_ENV); + if (child && *child) { + kern_run_on_guest_main(crash_child_main); + printf("crash child: returned without a fault\n"); + return 1; + } test_platform(); + test_oom_commit(); + test_crash_report(); kern_run_on_guest_main(probe_main); RtStats s = rt_stats(); printf("stats: %llu SVCs, %llu threads, %llu host calls\n", (unsigned long long)s.svcs, diff --git a/runtime/main.c b/runtime/main.c index e3428719..a0e05c80 100644 --- a/runtime/main.c +++ b/runtime/main.c @@ -21,7 +21,8 @@ // // Also here: the runtime services the generated code calls besides rt_dispatch and rt_svc (rt_brk, rt_udf, // rt_ticks, the function profiler hook), logging, SHA-256, and the crash reporter (a vectored exception handler that -// names the lifted function, the guest registers and the guest frame-pointer chain). +// names the lifted function, the host and guest registers, the machine code around the fault, the guest frame-pointer +// chain and the memory the registers point at). #include #include #include @@ -188,8 +189,8 @@ static int by_host(const void *a, const void *b) { } // The lifted function that contains a host code address, by the nearest function start below it; 0 outside the lifted -// code. -static uint64_t lifted_function(uintptr_t rip) { +// code. *host, when given, gets the function's host start. +static uint64_t lifted_function_at(uintptr_t rip, uintptr_t *host) { if (!fn_rev) { static RtModule mods[MAX_MODULES]; int n = rt_linked_modules(mods, MAX_MODULES); @@ -211,11 +212,78 @@ static uint64_t lifted_function(uintptr_t rip) { if (fn_rev[mid].host <= rip) lo = mid + 1; else hi = mid; } + if (lo && host) *host = fn_rev[lo - 1].host; return lo ? fn_rev[lo - 1].guest : 0; } +static uint64_t lifted_function(uintptr_t rip) { return lifted_function_at(rip, NULL); } + static int readable(uint64_t a, uint64_t n) { return plat_mem_readable(a, n); } +// The machine code around the faulting instruction: 32 bytes before rip, then 32 from rip. The user's totk.exe is built +// on their machine, so its offsets need not match ours; these bytes and the offset into the lifted function find the +// same instruction in a local build of that version, and with it the guest instruction and the operands' registers. +static void crash_code_bytes(uintptr_t rip) { + uint8_t b[64]; + int before = 32; + if (!plat_mem_read(rip - 32, b, sizeof b)) { + before = 0; // rip near the start of a mapping: the bytes from rip only + if (!plat_mem_read(rip, b + 32, 32)) return; + } + rt_logf(" host code at rip-%d:", before); + for (int i = 32 - before; i < 64; i++) rt_logf("%s%02x", i == 32 ? " | " : " ", b[i]); + rt_log_raw("\n", 1); +} + +// 64 bytes around each register value that points at readable memory, as 8-byte words from 16 bytes below the value, +// the word holding the value marked with *. Host and guest registers that land on the same window share a line. A +// guest pointer is a host address here (the guest address space is mapped one to one), so both kinds are read alike. +// The guest registers come from the Cpu, which the lifted code writes back only at calls and returns, so the host +// registers carry the faulting function's live values. +typedef struct { uint64_t start; char names[56]; uint64_t w[8]; } CrashWin; + +static void crash_win_add(CrashWin *wins, int *n, int cap, const char *name, uint64_t v) { + if (v < 0x10000 || v > 0x00007fffffffffffull) return; // small integers and non-canonical values + uint64_t start = (v & ~7ull) - 16; + for (int i = 0; i < *n; i++) + if (wins[i].start == start) { + size_t len = strlen(wins[i].names); + if (len + strlen(name) + 2 < sizeof wins[i].names) + snprintf(wins[i].names + len, sizeof wins[i].names - len, " %s", name); + return; + } + if (*n == cap || !plat_mem_read(start, wins[*n].w, sizeof wins[*n].w)) return; + wins[*n].start = start; + snprintf(wins[*n].names, sizeof wins[*n].names, "%s", name); + (*n)++; +} + +static void crash_memory(const PlatCrash *x, const Cpu *c) { + static CrashWin wins[64]; // the crash handler runs once (busy flag), so static storage needs no allocation + static const char *const gpr_names[16] = PLAT_GPR_NAMES; + int n = 0; + char name[8]; + if (x->has_regs) + for (int i = 0; i < 16; i++) crash_win_add(wins, &n, 64, gpr_names[i], x->gpr[i]); + if (c) { + for (int i = 0; i < 31; i++) snprintf(name, sizeof name, "x%d", i), crash_win_add(wins, &n, 64, name, c->x[i]); + crash_win_add(wins, &n, 64, "sp", c->sp); + } + if (!n) return; + rt_logf(" memory at the register values (8-byte words from value-16, * the value's word):\n"); + for (int i = 0; i < n; i++) { + rt_logf(" %016llx %-12s", (unsigned long long)wins[i].start, wins[i].names); + for (int k = 0; k < 8; k++) rt_logf(" %s%016llx", k == 2 ? "*" : "", (unsigned long long)wins[i].w[k]); + rt_log_raw("\n", 1); + } + // The guest stack above sp: the crashing function's spills and locals, and the callers' saved registers + uint64_t s[16]; + if (c && plat_mem_read(c->sp, s, sizeof s)) + for (int i = 0; i < 16; i += 4) + rt_logf(" guest stack sp+0x%02x: %016llx %016llx %016llx %016llx\n", i * 8, (unsigned long long)s[i], + (unsigned long long)s[i + 1], (unsigned long long)s[i + 2], (unsigned long long)s[i + 3]); +} + static void crash_handler(const PlatCrash *x) { static volatile int32_t busy; if (plat_xchg32(&busy, 1)) plat_exit(71); @@ -230,8 +298,22 @@ static void crash_handler(const PlatCrash *x) { rt_log_raw("\n", 1); if (x->cxx_type) rt_logf(" C++ type %s%s%s\n", x->cxx_type, x->cxx_what ? ", what(): " : "", x->cxx_what ? x->cxx_what : ""); - uint64_t f = x->module ? 0 : lifted_function(rip); - if (f) rt_logf(" in lifted function f_%llx = %s\n", (unsigned long long)f, ldr_describe(f, d, sizeof d)); + // A driver that runs out of memory faults or throws (std::bad_alloc) instead of returning an error: the commit + // tells that apart from a driver bug, as rt_fatal's line does for our own failed allocations. + double proc, used, limit; + plat_commit_info(&proc, &used, &limit); + rt_logf(" memory commit: this process %.0f MB, the system %.0f of %.0f MB\n", proc, used, limit); + uintptr_t fstart = 0; + uint64_t f = x->module ? 0 : lifted_function_at(rip, &fstart); + if (f) + rt_logf(" in lifted function f_%llx = %s, host +%#llx into it\n", (unsigned long long)f, + ldr_describe(f, d, sizeof d), (unsigned long long)(rip - fstart)); + if (x->has_regs) { + static const char *const gpr_names[16] = PLAT_GPR_NAMES; + for (int i = 0; i < 16; i++) + rt_logf(" %-3s %016llx%s", gpr_names[i], (unsigned long long)x->gpr[i], i % 4 == 3 ? "\n" : ""); + crash_code_bytes(rip); + } // abort and invalid parameter: where the host code called it; a fault in another module: the driver code and the // host calls that led into it uintptr_t exe = plat_exe_base(); @@ -266,6 +348,7 @@ static void crash_handler(const PlatCrash *x) { } else { rt_logf(" not on a guest thread\n"); } + crash_memory(x, c); kern_free_cores_note(); fflush(stderr); plat_exit(71); @@ -532,6 +615,20 @@ int main(int argc, char **argv) { // Timed guest waits (kernel.c block_inner) sleep in whole milliseconds; at the default 15.6 ms tick a 1 ms wait // costs a full tick. TOTK_TIMER_1MS=0 keeps the default for comparison. if (cfg_bool("TOTK_TIMER_1MS")) plat_timer_1ms(); + int power = plat_power_full(); + if (power >= 0) + rt_log("power: %s", power ? "OS power throttling off (EcoQoS, timer resolution in the background)" + : "OS power throttling: the opt-out was refused"); + // Hybrid CPUs: the threads every frame goes through prefer the P-cores (perf_core_thread). TOTK_PCORE=0: off. + if (rt_knob("PCORE", "1")) { + char why[160]; + int n = plat_perf_core_setup(why, sizeof why); + if (n) rt_log("power: %s; the game's main and render threads and the vk replay, recorder and present threads " + "prefer the %d P-core ones (TOTK_PCORE=0 turns it off)", why, n); + else rt_log("power: %s: no P-core preference", why); + } else { + rt_log("power: TOTK_PCORE=0: no P-core preference"); + } cfg = (RtConfig){.exefs = rt_default_exefs, .romfs = rt_default_romfs, .profile = rt_default_profile, .gpu = cfg_get("TOTK_GPU")}; portable_paths(&cfg); // command-line paths below still win diff --git a/runtime/memory.c b/runtime/memory.c index 019c0fab..bfe11290 100644 --- a/runtime/memory.c +++ b/runtime/memory.c @@ -175,12 +175,29 @@ static int thaw_fault(uint64_t a) { } // Mapping changes thaw the regions they touch and run under frz_lock, so no freeze sees them half done. -static void commit(uint64_t a, uint64_t n, const char *what) { +typedef struct { uint64_t a, n; } CommitTry; +static int commit_try(void *arg) { + const CommitTry *t = arg; + if (rt_oom_forced(RT_OOM_COMMIT)) return 0; plat_mutex_lock(&frz_lock); - thaw_range_locked(a, n, 3); - int ok = !n || plat_mem_commit(a, n); + thaw_range_locked(t->a, t->n, 3); + int ok = !t->n || plat_mem_commit(t->a, t->n); plat_mutex_unlock(&frz_lock); - if (!ok) rt_fatal("memory: commit %s %#llx+%#llx failed", what, (unsigned long long)a, (unsigned long long)n); + return ok; +} + +// A refused commit waits for memory (rt_oom_wait): the system's commit ran out for a moment (a page file growing, the +// driver's device memory charged to this process), and the Vulkan back end gives memory back at its next submit. The +// wait holds the caller's lock (lock or klock: only memory SVCs and kernel-region allocations wait behind it) but not +// frz_lock, which write faults on frozen regions need. Every caller waits, the boot's ones too: there a refusal has the +// same cause, and the wait only delays an end the game would reach anyway. +static void commit(uint64_t a, uint64_t n, const char *what) { + CommitTry t = {a, n}; + if (commit_try(&t)) return; + char w[64]; + snprintf(w, sizeof w, "memory: commit of %s at %#llx", what, (unsigned long long)a); + if (!rt_oom_wait(w, n, commit_try, &t)) + rt_fatal("memory: commit %s %#llx+%#llx failed", what, (unsigned long long)a, (unsigned long long)n); } static void decommit(uint64_t a, uint64_t n, const char *what) { diff --git a/runtime/nvn_api_cmd.c b/runtime/nvn_api_cmd.c index 874002f7..8141c61b 100644 --- a/runtime/nvn_api_cmd.c +++ b/runtime/nvn_api_cmd.c @@ -148,6 +148,7 @@ static void *realloc_wait(void *old, size_t n, const char *what) { if (p) return p; pool_drain(); big_drain(); + rt_oom_request(); // the Vulkan back end gives memory back at its next submit (oom.c) static uint32_t reports; int report = __atomic_fetch_add(&reports, 1, __ATOMIC_RELAXED) < 8; if (report) { diff --git a/runtime/oom.c b/runtime/oom.c new file mode 100644 index 00000000..77f891c0 --- /dev/null +++ b/runtime/oom.c @@ -0,0 +1,106 @@ +// Short out-of-memory spells: a failed allocation waits for memory instead of ending the game. +// +// Windows refuses commit beyond the current limit while it grows a system-managed page file, and on some drivers device +// memory counts as process commit (vk_memory.c vk_mem_host_probe), so for a moment every allocation of the process can +// fail and the same request succeeds a little later. The failure sites (memory.c commit, vk_memory.c vk_staging, +// vk_texture.c create_tex, vk_exec_draw.c transient chunks, nvn_api_cmd.c realloc_wait) ask the Vulkan back end to give +// memory back (rt_oom_request: released textures and headroom blocks, at its next submit, where its locks are free) and +// retry for up to OOM_WAIT_MS before they end the game. +// +// TOTK_TEST_OOM=:[,:...] (sites commit, staging, image, chunk) fails the first n attempts at a site, so +// the wait can be tested without running out of memory; the probes arm it with rt_oom_force. +#include +#include + +#include "cfg.h" +#include "host.h" +#include "platform/platform.h" + +#define OOM_WAIT_MS 30000u +#define OOM_STEP_MS 10u +#define OOM_REPORTS 8 +#define OOM_REQUEST_EVERY 100 // retries between two release requests: a second, since a release only frees what is old enough + +volatile int32_t rt_oom_req; + +static const char *const site_name[RT_OOM_SITES] = {"commit", "staging", "image", "chunk"}; +static volatile int32_t force_left[RT_OOM_SITES]; +static volatile int32_t forced_parsed; +static PlatOnce force_once = PLAT_ONCE_INIT; + +static void force_parse(void *arg) { + (void)arg; + const char *s = cfg_get("TOTK_TEST_OOM"); + while (s && *s) { + const char *colon = strchr(s, ':'), *end = strchr(s, ','); + if (!end) end = s + strlen(s); + if (colon && colon < end) { + for (int i = 0; i < RT_OOM_SITES; i++) + if ((size_t)(colon - s) == strlen(site_name[i]) && !strncmp(s, site_name[i], (size_t)(colon - s))) { + int n = atoi(colon + 1); + __atomic_store_n(&force_left[i], n > 0 ? n : 0, __ATOMIC_RELAXED); + rt_log("oom: TOTK_TEST_OOM fails the first %d %s attempts", n, site_name[i]); + } + } + s = *end ? end + 1 : end; + } + __atomic_store_n(&forced_parsed, 1, __ATOMIC_RELEASE); +} + +int rt_oom_forced(int site) { + if (!__atomic_load_n(&forced_parsed, __ATOMIC_ACQUIRE)) plat_once(&force_once, force_parse, NULL); + if (__atomic_load_n(&force_left[site], __ATOMIC_RELAXED) <= 0) return 0; + return plat_dec32(&force_left[site]) >= 0; +} + +void rt_oom_force(int site, int n) { + if (!__atomic_load_n(&forced_parsed, __ATOMIC_ACQUIRE)) plat_once(&force_once, force_parse, NULL); + __atomic_store_n(&force_left[site], n, __ATOMIC_RELAXED); +} + +int rt_oom_forced_left(int site) { + int n = __atomic_load_n(&force_left[site], __ATOMIC_RELAXED); + return n > 0 ? n : 0; +} + +void rt_oom_request(void) { __atomic_store_n(&rt_oom_req, 1, __ATOMIC_RELEASE); } + +int rt_oom_wait(const char *what, uint64_t bytes, int (*retry)(void *arg), void *arg) { + rt_oom_request(); + static volatile int32_t reports; + int report = plat_inc32(&reports) <= OOM_REPORTS; + double proc, used, limit; + if (report) { + plat_commit_info(&proc, &used, &limit); + rt_log("%s of %llu bytes failed (memory commit: this process %.0f MB, the system %.0f of %.0f MB); waiting up " + "to %u s for memory", what, (unsigned long long)bytes, proc, used, limit, OOM_WAIT_MS / 1000); + } + uint64_t t0 = plat_now_ms(); + for (int n = 1; plat_now_ms() - t0 < OOM_WAIT_MS; n++) { + plat_sleep_ms(OOM_STEP_MS); + if (retry(arg)) { + if (report) + rt_log("%s of %llu bytes succeeded after %llu ms", what, (unsigned long long)bytes, + (unsigned long long)(plat_now_ms() - t0)); + return 1; + } + if (n % OOM_REQUEST_EVERY == 0) rt_oom_request(); + } + plat_commit_info(&proc, &used, &limit); + rt_log("%s of %llu bytes still failed after %u s (memory commit: this process %.0f MB, the system %.0f of %.0f MB)", + what, (unsigned long long)bytes, OOM_WAIT_MS / 1000, proc, used, limit); + return 0; +} + +typedef struct { void *old, *p; size_t n; } Realloc; +static int realloc_try(void *arg) { + Realloc *r = arg; + return (r->p = realloc(r->old, r->n)) != NULL; +} + +void *rt_realloc_wait(void *old, size_t n, const char *what) { + void *p = realloc(old, n); + if (p || !n) return p; + Realloc r = {old, NULL, n}; + return rt_oom_wait(what, n, realloc_try, &r) ? r.p : NULL; +} diff --git a/runtime/perf.c b/runtime/perf.c index ef04ede5..cae0ff4d 100644 --- a/runtime/perf.c +++ b/runtime/perf.c @@ -62,6 +62,15 @@ uint64_t perf_begin(void) { return kern_now_ns(); } void perf_replay_thread(void) { prof_replay_tid = plat_thread_id(); } void perf_main_thread(void) { prof_main_tid = plat_thread_id(); } +void perf_core_thread(int role) { + static const char *const names[] = {"game main", "game render", "vk replay", "vk recorder", "vk present"}; + static int32_t said; + int r = plat_thread_perf_core(); + if (r < 0 || role < 0 || role >= PCORE_N) return; + if (__atomic_fetch_or(&said, 1 << role, __ATOMIC_RELAXED) & (1 << role)) return; + rt_log("power: the %s thread %s the P-cores", names[role], r ? "prefers" : "could not select"); +} + void perf_end(int kind, uint64_t t0) { if (!t0) return; uint32_t tid = plat_thread_id(); diff --git a/runtime/platform/linux.c b/runtime/platform/linux.c index 158601cb..24a0f0b1 100644 --- a/runtime/platform/linux.c +++ b/runtime/platform/linux.c @@ -481,6 +481,14 @@ uint64_t plat_counter_hz(void) { return 1000000000u; } // inherit it from the thread that creates them. void plat_timer_1ms(void) { prctl(PR_SET_TIMERSLACK, 1000ul, 0, 0, 0); } +int plat_power_full(void) { return -1; } + +int plat_perf_core_setup(char *why, size_t n) { + snprintf(why, n, "not done on Linux"); + return 0; +} +int plat_thread_perf_core(void) { return -1; } + // clock_nanosleep wakes within the timer slack (50 us by default) of its deadline; the last PLAT_SPIN_NS are spent // yielding so the deadline is met closely, as on Windows. #define PLAT_SPIN_NS 100000u @@ -949,6 +957,10 @@ static void fault_handler(int sig, siginfo_t *si, void *uc) { } if (sig == SIGSEGV || sig == SIGBUS) c.access = (err & 16) ? 8 : (err & 2) ? 1 : 0, c.addr = (uint64_t)(uintptr_t)si->si_addr; + static const int gr[16] = {REG_RAX, REG_RCX, REG_RDX, REG_RBX, REG_RSP, REG_RBP, REG_RSI, REG_RDI, + REG_R8, REG_R9, REG_R10, REG_R11, REG_R12, REG_R13, REG_R14, REG_R15}; + for (int i = 0; i < 16; i++) c.gpr[i] = (uint64_t)u->uc_mcontext.gregs[gr[i]]; + c.has_regs = 1; crash_fn(&c); } signal(sig, SIG_DFL); // returns into the faulting instruction, which faults again with the default action @@ -1661,6 +1673,17 @@ void plat_commit_info(double *proc_mb, double *used_mb, double *limit_mb) { if (private_bytes) *proc_mb = (double)(private_bytes >> 20); } +void plat_mem_room(uint64_t *commit_room, uint64_t *phys_room) { + *commit_room = *phys_room = UINT64_MAX; + char line[128]; + unsigned long long k; + FILE *f = fopen("/proc/meminfo", "r"); + if (!f) return; + while (fgets(line, sizeof line, f)) + if (sscanf(line, "MemAvailable: %llu kB", &k) == 1) *phys_room = (uint64_t)k << 10; + fclose(f); +} + // ---- child processes struct PlatChild { diff --git a/runtime/platform/platform.h b/runtime/platform/platform.h index 97911d27..1f67f661 100644 --- a/runtime/platform/platform.h +++ b/runtime/platform/platform.h @@ -99,6 +99,12 @@ uint64_t plat_now_ns(void); // monotonic, high resolution, arbitra uint64_t plat_counter(void); // the OS's raw monotonic counter (QPC on Windows), plat_counter_hz per second uint64_t plat_counter_hz(void); void plat_timer_1ms(void); // asks the OS for timed waits accurate to 1 ms or better (call before threads) +int plat_power_full(void); // opts the process out of OS power throttling: 1 done, 0 refused, -1 no such thing +// Hybrid CPUs: finds the performance-core logical processors the process may use, once, before any thread asks; returns +// their count (0: none to prefer, why says which case). plat_thread_perf_core: the calling thread prefers them; 1 done, +// 0 refused, -1 nothing selected. +int plat_perf_core_setup(char *why, size_t n); +int plat_thread_perf_core(void); // t * num / den for a fixed num / den without a 128-bit division per call (__udivti3 took 5% of the game's running // samples through plat_now_ns and rt_ticks): t * q + (t * f >> 64) with q = num / den and f = (num % den) * 2^64 / den. // Monotonic in t and at most one below the exact quotient while t * (num % den) / den fits 64 bits. @@ -243,7 +249,11 @@ typedef struct { const char *module; // the module holding rip: NULL for this executable, "" for none (a jump to address 0) const char *cxx_type; // a C++ exception from MSVC-built code: the thrown type's decorated name (.?AV...@@) const char *cxx_what; // and what() when the type derives from std::exception; NULL when unknown + int has_regs; // gpr holds the faulting thread's registers (a fault; not abort or invalid parameter) + uint64_t gpr[16]; // rax rcx rdx rbx rsp rbp rsi rdi r8..r15, in the x86 encoding order } PlatCrash; +#define PLAT_GPR_NAMES {"rax", "rcx", "rdx", "rbx", "rsp", "rbp", "rsi", "rdi", \ + "r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15"} void plat_crash_hook(void (*fn)(const PlatCrash *c)); int plat_mem_read(uint64_t addr, void *out, size_t n); // copies n bytes, 0 instead of faulting on unmapped memory int plat_screen_size(int *w, int *h); // the primary display's current mode in pixels; 0 unknown @@ -423,6 +433,9 @@ void plat_mem_info(float *used_mb, float *total_mb, float *proc_mb); // Commit charge in MB: this process's private bytes, the system's commit in use and its limit (RAM plus page files). // An allocation fails when the system's reaches the limit, however much physical memory is free. -1 when unknown. void plat_commit_info(double *proc_mb, double *used_mb, double *limit_mb); +// What the system can still give, in bytes: commit (Windows: limit minus charge; Linux, which overcommits: UINT64_MAX) +// and physical memory available without paging. UINT64_MAX when unknown. +void plat_mem_room(uint64_t *commit_room, uint64_t *phys_room); // This process's memory in bytes: resident (the working set) and private (Windows: committed private bytes; Linux: the // resident anonymous pages). 0 when unknown. void plat_process_memory(uint64_t *resident, uint64_t *private_bytes); diff --git a/runtime/platform/win32.c b/runtime/platform/win32.c index 5dda4538..6a291b81 100644 --- a/runtime/platform/win32.c +++ b/runtime/platform/win32.c @@ -18,6 +18,13 @@ #include "platform.h" +// Hybrid laptops: the drivers read these exports of the executable when it starts and run it on the dedicated GPU. +// NVIDIA Optimus can otherwise hide its GPU from a program it put on the integrated one, whatever the Windows and NVIDIA +// settings say; AMD switchable graphics read the second one. TOTK_GPU_INDEX still picks another GPU. Savage Wind Recomp +// exports the same two (runtime/platform/windows.c). +__declspec(dllexport) DWORD NvOptimusEnablement = 1; +__declspec(dllexport) int AmdPowerXpressRequestHighPerformance = 1; + _Static_assert(sizeof(PlatMutex) == sizeof(SRWLOCK), "PlatMutex holds an SRWLOCK"); _Static_assert(sizeof(PlatCond) == sizeof(CONDITION_VARIABLE), "PlatCond holds a CONDITION_VARIABLE"); @@ -289,6 +296,98 @@ uint32_t plat_cpu_usable(void) { return plat_cpu_count(); } +// ---- performance cores of a hybrid CPU +// An Intel CPU with P-cores and E-cores (a Core 5 210H: 4 P-cores with two threads each, 4 E-cores) runs a thread on an +// E-core at about 60% of a P-core's speed, and the frame waits for its slowest critical thread. The few threads every +// frame goes through ask for the P-cores (plat_thread_perf_core): their selected CPU sets are the logical processors +// above the lowest efficiency class that the process affinity allows. A thread's CPU sets bind it like an affinity +// mask, except that processors the system reserves for itself are left out instead of failing. Nothing is selected +// on a CPU with one efficiency class, or when the affinity already leaves only P-cores or only E-cores. LAS Recomp +// does the same for its game, render and vsync threads (runtime/platform/windows.c plat_thread_perf_core). +// Nothing is selected either below PC_MIN_CORES physical P-cores (an i5-1235U or Core 3 100U has 2): the five bound +// threads (perf_core_thread) and a spinning recorder would crowd two cores and their hyperthreads while the E-cores +// stay idle, which can be slower than letting the scheduler place them. +#define PC_MIN_CORES 4 +static ULONG *pc_ids; // the CPU set ids of the allowed P-core logical processors; NULL: none selected +static ULONG pc_n; + +int plat_perf_core_setup(char *why, size_t n) { + ULONG len = 0; + GetSystemCpuSetInformation(NULL, 0, &len, GetCurrentProcess(), 0); + uint8_t *buf = len ? malloc(len) : NULL; + if (!buf || !GetSystemCpuSetInformation((PSYSTEM_CPU_SET_INFORMATION)buf, len, &len, GetCurrentProcess(), 0)) { + free(buf); + snprintf(why, n, "no CPU set information (error %lu)", GetLastError()); + return 0; + } + // The process affinity mask covers its first processor group only: it filters on a single-group system. + DWORD_PTR proc = 0, sys = 0; + int filter = GetActiveProcessorGroupCount() == 1 && GetProcessAffinityMask(GetCurrentProcess(), &proc, &sys) && proc; + int lo = 255, hi = -1, all = 0; + for (ULONG off = 0; off < len;) { + SYSTEM_CPU_SET_INFORMATION *i = (SYSTEM_CPU_SET_INFORMATION *)(buf + off); + if (!i->Size) break; + if (i->Type == CpuSetInformation) { + int ec = i->CpuSet.EfficiencyClass; + lo = ec < lo ? ec : lo, hi = ec > hi ? ec : hi, all++; + } + off += i->Size; + } + if (!all || hi <= lo) { + free(buf); + snprintf(why, n, "not a hybrid CPU (%d logical processors, one efficiency class)", all); + return 0; + } + ULONG *ids = calloc((size_t)all, sizeof *ids); + uint16_t *cores = calloc((size_t)all, sizeof *cores); // group << 8 | core index of each allowed physical P-core + int np = 0, ne = 0, ok_p = 0, ok_e = 0, nc = 0; + for (ULONG off = 0; ids && cores && off < len;) { + SYSTEM_CPU_SET_INFORMATION *i = (SYSTEM_CPU_SET_INFORMATION *)(buf + off); + if (!i->Size) break; + if (i->Type == CpuSetInformation) { + int perf = i->CpuSet.EfficiencyClass > lo; + int allowed = !filter || (i->CpuSet.Group == 0 && i->CpuSet.LogicalProcessorIndex < 64 && + (proc >> i->CpuSet.LogicalProcessorIndex) & 1); + perf ? np++ : ne++; + if (allowed) perf ? ok_p++ : ok_e++; + if (allowed && perf) { + ids[pc_n++] = i->CpuSet.Id; + uint16_t key = (uint16_t)(i->CpuSet.Group << 8 | i->CpuSet.CoreIndex); + int k = 0; + while (k < nc && cores[k] != key) k++; + if (k == nc) cores[nc++] = key; + } + } + off += i->Size; + } + int counted = ids && cores; + free(buf); + free(cores); + if (!counted || !ok_p || !ok_e) { + free(ids); + pc_n = 0; + snprintf(why, n, "hybrid CPU, %d P-core + %d E-core logical processors; the affinity allows %d + %d, so it decides", + np, ne, ok_p, ok_e); + return 0; + } + if (nc < PC_MIN_CORES) { + free(ids); + pc_n = 0; + snprintf(why, n, "hybrid CPU, %d P-core + %d E-core logical processors on %d P-cores, fewer than %d", np, ne, nc, + PC_MIN_CORES); + return 0; + } + pc_ids = ids; + snprintf(why, n, "hybrid CPU, %d P-core + %d E-core logical processors%s", np, ne, + ok_p != np || ok_e != ne ? " (affinity-limited)" : ""); + return (int)pc_n; +} + +int plat_thread_perf_core(void) { + if (!pc_ids) return -1; + return SetThreadSelectedCpuSets(GetCurrentThread(), pc_ids, pc_n) ? 1 : 0; +} + // PLAT_PRIO_* are the THREAD_PRIORITY_* values. void plat_thread_priority(int prio) { SetThreadPriority(GetCurrentThread(), prio); } void plat_thread_set_priority(PlatThread *t, int prio) { SetThreadPriority(t->h, prio); } @@ -319,6 +418,17 @@ uint64_t plat_counter_hz(void) { return qpc_freq(); } // The default timer tick is 15.6 ms; a 1 ms wait would cost a full tick. void plat_timer_1ms(void) { timeBeginPeriod(1); } +// Windows 11 runs the threads of a process it deems in the background at EcoQoS (on the E-cores of a hybrid Intel CPU, +// at a low clock) and ignores its timer resolution while its window is hidden. The game is always latency-bound, so it +// opts out of both. Windows before 10 1709 does not know the class: 0. +int plat_power_full(void) { + PROCESS_POWER_THROTTLING_STATE s = { + .Version = PROCESS_POWER_THROTTLING_CURRENT_VERSION, + .ControlMask = PROCESS_POWER_THROTTLING_EXECUTION_SPEED | PROCESS_POWER_THROTTLING_IGNORE_TIMER_RESOLUTION, + .StateMask = 0}; + return SetProcessInformation(GetCurrentProcess(), ProcessPowerThrottling, &s, sizeof s) ? 1 : 0; +} + static PlatScale qpc_ns; // set before main, so no thread reads it half written __attribute__((constructor)) static void qpc_ns_init(void) { qpc_ns = plat_scale(1000000000u, qpc_freq()); } @@ -626,8 +736,21 @@ void plat_mem_fault_hook(int (*fn)(uint64_t addr)) { static void (*crash_fn)(const PlatCrash *c); +// The faulting thread's general registers, for the report to tie the faulting instruction to its operands. +static void regs_of(const CONTEXT *x, PlatCrash *c) { + const DWORD64 r[16] = {x->Rax, x->Rcx, x->Rdx, x->Rbx, x->Rsp, x->Rbp, x->Rsi, x->Rdi, + x->R8, x->R9, x->R10, x->R11, x->R12, x->R13, x->R14, x->R15}; + for (int i = 0; i < 16; i++) c->gpr[i] = r[i]; + c->has_regs = 1; +} + +static LONG WINAPI crash_filter(EXCEPTION_POINTERS *ep); + static LONG CALLBACK crash_veh(EXCEPTION_POINTERS *ep) { DWORD code = ep->ExceptionRecord->ExceptionCode; + // Heap corruption is raised by the heap manager, which then ends the process itself: it never reaches the + // unhandled-exception filter, so it is reported from here, wherever the heap was called from. + if (code == 0xc0000374u) return crash_filter(ep); // STATUS_HEAP_CORRUPTION const char *what = code == EXCEPTION_ACCESS_VIOLATION ? "access violation" : code == EXCEPTION_ILLEGAL_INSTRUCTION ? "illegal instruction" : code == EXCEPTION_INT_DIVIDE_BY_ZERO ? "integer division by zero" @@ -643,6 +766,7 @@ static LONG CALLBACK crash_veh(EXCEPTION_POINTERS *ep) { PlatCrash c = {what, (uint32_t)code, rip, rip - (uintptr_t)mi.lpBaseOfDll, -1, 0}; if (code == EXCEPTION_ACCESS_VIOLATION && ep->ExceptionRecord->NumberParameters >= 2) c.access = (int)ep->ExceptionRecord->ExceptionInformation[0], c.addr = ep->ExceptionRecord->ExceptionInformation[1]; + regs_of(ep->ContextRecord, &c); crash_fn(&c); return EXCEPTION_CONTINUE_SEARCH; } @@ -780,6 +904,7 @@ static LONG WINAPI crash_filter(EXCEPTION_POINTERS *ep) { (uint32_t)code, rip, 0, -1, 0}; if (code == EXCEPTION_ACCESS_VIOLATION && er->NumberParameters >= 2) c.access = (int)er->ExceptionInformation[0], c.addr = er->ExceptionInformation[1]; + regs_of(ep->ContextRecord, &c); void *base = NULL; RtlPcToFileHeader((void *)rip, &base); uintptr_t exe = (uintptr_t)GetModuleHandleW(NULL); @@ -1659,6 +1784,12 @@ void plat_commit_info(double *proc_mb, double *used_mb, double *limit_mb) { *proc_mb = (double)(pmc.PrivateUsage >> 20); } +void plat_mem_room(uint64_t *commit_room, uint64_t *phys_room) { + *commit_room = *phys_room = UINT64_MAX; + MEMORYSTATUSEX ms = {.dwLength = sizeof ms}; + if (GlobalMemoryStatusEx(&ms)) *commit_room = ms.ullAvailPageFile, *phys_room = ms.ullAvailPhys; +} + void plat_process_memory(uint64_t *resident, uint64_t *private_bytes) { PROCESS_MEMORY_COUNTERS_EX pm = {.cb = sizeof pm}; *resident = *private_bytes = 0; diff --git a/runtime/romfs.c b/runtime/romfs.c index 81665098..52c80481 100644 --- a/runtime/romfs.c +++ b/runtime/romfs.c @@ -1080,19 +1080,26 @@ static void index_path(char *out, size_t cap) { strcpy(out + k, "romfs-index.bin"); } -// 1 when the game resolution w x h has the UI layout of its shape: romfs-patches-x next to romfs-index.bin with a -// manifest, the folder overlay_init looks for. Works before fs_init: index_dir is not set that early (res_init and -// resolution.c decide run first), so the index path is worked out again here. -int romfs_shaped_ready(unsigned w, unsigned h) { +// 1 when -x has a manifest next to romfs-index.bin. Works before fs_init: index_dir is not set that early +// (cfg_init, res_init and resolution.c decide run first), so the index path is worked out again here. +static int shaped_folder_ready(const char *prefix, unsigned w, unsigned h) { char dir[2100], path[2200]; index_path(dir, sizeof dir); char *sep = plat_last_sep(dir); if (sep) *sep = 0; else strcpy(dir, "."); - return snprintf(path, sizeof path, "%s/romfs-patches-%ux%u/manifest.tsv", dir, w, h) < (int)sizeof path && + return snprintf(path, sizeof path, "%s/%s-%ux%u/manifest.tsv", dir, prefix, w, h) < (int)sizeof path && plat_file_exists(path); } +// 1 when the game resolution w x h has the UI layout of its shape: romfs-patches-x, the folder overlay_init +// looks for. +int romfs_shaped_ready(unsigned w, unsigned h) { return shaped_folder_ready("romfs-patches", w, h); } + +// 1 when it has the PC prompts with that layout too: romfs-patches-pc-x, which overlay_init loads in its place +// with prompts = pc. +int romfs_shaped_pc_ready(unsigned w, unsigned h) { return shaped_folder_ready("romfs-patches-pc", w, h); } + static void load_index(void) { char path[2100]; index_path(path, sizeof path); diff --git a/runtime/simd_fp.h b/runtime/simd_fp.h index 5593bb5b..1e7721ed 100644 --- a/runtime/simd_fp.h +++ b/runtime/simd_fp.h @@ -205,19 +205,21 @@ SN T FN(slow)(Cpu *c, int op, T a, T b, T d) { // it to fcmp one, a single ucomiss + jne for scalars and one vcmpps for vectors). #define FOK(r) ((__builtin_elementwise_abs(r) < FN(fv)(MINN)) | (__builtin_elementwise_abs(r) > FN(fv)(MINN))) SN T FN(redo)(Cpu *c, int op, T a, T b, T d, T r) { - if (c->fpcr & FPCR_FZ) return FN(slow)(c, op, a, b, d); + if (c->fpcr & (FPCR_FZ | FPCR_RM)) return FN(slow)(c, op, a, b, d); return FBAD(op, r) ? FN(fix)(c, op, a, b, d) : r; } -// Scalar FP op (constant op folds the selection). RMode = nearest runs on the host, with or without FZ. +// Scalar FP op (constant op folds the selection). RMode = nearest runs on the host, with or without FZ. A directed +// RMode is rare, so it shares the result check's branch: the host result, computed in round-to-nearest, is then +// thrown away and redo recomputes in the FPCR mode. Min and max do not round, so RMode does not matter to them. SI T FN(op)(Cpu *c, int op, T a, T b, T d) { - if (simd_directed(c)) return FN(slow)(c, op, a, b, d); if (FMINMAX(op)) { if (c->fpcr & FPCR_FZ) a = FN(flush)(a), b = FN(flush)(b); return FN(minmax)(c, op, a, b); } T r = FCALC(op, a, b, d, K, FMA_S); - if (__builtin_expect(op == FOP_RSQRTS ? FREDO(op, r) : !FOK(r), 0)) r = FN(redo)(c, op, a, b, d, r); + if (__builtin_expect((op == FOP_RSQRTS ? FREDO(op, r) : !FOK(r)) | simd_directed(c), 0)) + r = FN(redo)(c, op, a, b, d, r); return r; } @@ -237,9 +239,9 @@ SI VT FN(vflush)(VT a) { return (VT)FN(vsel)(den, u & SGN, u); } -// Vector FP op: native lanes; any lane FREDO flags (or FPCR.RMode) reruns all lanes through the scalar path. +// Vector FP op: native lanes; any lane FREDO flags (or a directed FPCR.RMode, checked in the same branch) reruns all +// lanes through the scalar path. Min and max ignore RMode, as in FN(op). SI VT FN(vop)(Cpu *c, int op, VT a, VT b, VT d) { - if (simd_directed(c)) return FN(vslow)(c, op, a, b, d); VT r; if (FMINMAX(op)) { if (c->fpcr & FPCR_FZ) a = FN(vflush)(a), b = FN(vflush)(b); @@ -252,7 +254,8 @@ SI VT FN(vop)(Cpu *c, int op, VT a, VT b, VT d) { return r; } r = FCALC(op, a, b, d, VK, FMA_V); - if (__builtin_expect(op == FOP_RSQRTS ? __builtin_reduce_or(FREDO(op, r)) != 0 : __builtin_reduce_and(FOK(r)) == 0, 0)) + if (__builtin_expect((op == FOP_RSQRTS ? __builtin_reduce_or(FREDO(op, r)) != 0 : __builtin_reduce_and(FOK(r)) == 0) + | simd_directed(c), 0)) return FN(vslow)(c, op, a, b, d); return r; } @@ -260,13 +263,15 @@ SI VT FN(vop)(Cpu *c, int op, VT a, VT b, VT d) { SI VT FN(vfz)(Cpu *c, VT a) { return c->fpcr & FPCR_FZ ? FN(vflush)(a) : a; } // ---- compares +// FPCR.FZ needs no flush here: rt_fpcr_sync (runtime/cpu.h) keeps MXCSR.DAZ equal to FPCR.FZ, and x86 compares +// under DAZ read a denormal operand as a zero of its sign, as ARM compares do under FZ. SI uint64_t FN(cmp)(Cpu *c, T a, T b) { // FCMP NZCV - a = FN(fz)(c, a), b = FN(fz)(c, b); + (void)c; return a == b ? 0x60000000ull : a < b ? 0x80000000ull : a > b ? 0x20000000ull : 0x30000000ull; } // FCMEQ/FCMGE/FCMGT/FACGE/FACGT (op 0..4) SI VTU FN(vcmp)(Cpu *c, int op, VT a, VT b) { - a = FN(vfz)(c, a), b = FN(vfz)(c, b); + (void)c; if (op >= 3) a = __builtin_elementwise_abs(a), b = __builtin_elementwise_abs(b); switch (op) { case 0: return (VTU)(a == b); diff --git a/runtime/ui/amiibo_page.cpp b/runtime/ui/amiibo_page.cpp index 2f2ed996..80597c43 100644 --- a/runtime/ui/amiibo_page.cpp +++ b/runtime/ui/amiibo_page.cpp @@ -36,6 +36,8 @@ static int g_selected_entry = -1; // The last action: a catalog id and the file or folder it names, formatted when drawn (the menu language can change) static UiTextId g_status_id = UI_MENU_AMIIBO_READY; static std::string g_status_arg; +// Flag to request font rebuild (set when amiibo directory is refreshed with new files) +static bool g_font_rebuild_requested = false; static void set_status(UiTextId id, const std::string &arg = std::string()) { g_status_id = id; @@ -124,6 +126,9 @@ static void refresh_dir(void) { if (a.is_dir != b.is_dir) return a.is_dir; return a.name < b.name; }); + + // Request font rebuild to include any new filenames + amiibo_request_font_rebuild(); } void tab_amiibo(float scale) { @@ -303,3 +308,44 @@ void tab_amiibo(float scale) { ImGui::EndChild(); } + +// Collect amiibo filenames for font glyph building (called from menu.cpp fonts_up) +void amiibo_collect_filenames(void (*callback)(void *, const char *), void *ctx) { + std::error_code ec; + fs::path dir_path(g_current_dir); + + if (!fs::exists(dir_path, ec) || !fs::is_directory(dir_path, ec)) { + return; + } + + for (const auto &entry : fs::directory_iterator(dir_path, fs::directory_options::skip_permission_denied, ec)) { + if (ec) break; + std::error_code status_ec; + bool is_dir = entry.is_directory(status_ec); + if (status_ec || is_dir) continue; + + std::string ext = entry.path().extension().string(); + if (!iequals(ext, ".bin") && !iequals(ext, ".nfc")) { + continue; + } + + std::string filename = entry.path().filename().string(); + if (!filename.empty()) { + callback(ctx, filename.c_str()); + } + } +} + +// Request font atlas rebuild on next frame +void amiibo_request_font_rebuild(void) { + g_font_rebuild_requested = true; +} + +// Check and clear font rebuild request flag +extern "C" bool amiibo_check_font_rebuild(void) { + if (g_font_rebuild_requested) { + g_font_rebuild_requested = false; + return true; + } + return false; +} diff --git a/runtime/ui/amiibo_page.h b/runtime/ui/amiibo_page.h index 167ee48c..5b2e8711 100644 --- a/runtime/ui/amiibo_page.h +++ b/runtime/ui/amiibo_page.h @@ -2,3 +2,9 @@ #pragma once void tab_amiibo(float scale); + +// Collect amiibo filenames for font glyph building (called from menu.cpp fonts_up) +void amiibo_collect_filenames(void (*callback)(void *, const char *), void *ctx); + +// Request font atlas rebuild on next frame (called when amiibo directory is refreshed) +void amiibo_request_font_rebuild(void); diff --git a/runtime/ui/menu.cpp b/runtime/ui/menu.cpp index 6f4497a0..a4d8ec56 100644 --- a/runtime/ui/menu.cpp +++ b/runtime/ui/menu.cpp @@ -51,6 +51,7 @@ extern "C" { void rt_log(const char *fmt, ...) __attribute__((format(printf, 1, 2))); void vk_queue_lock(void); void vk_queue_unlock(void); +int vk_exec_replay_workers(void); void res_sizes(unsigned out[2], unsigned scene[2]); double frame_step_limit_hz(void); extern const int rt_game_version; @@ -58,6 +59,7 @@ extern const char rt_port_version[]; void cheats_frame(uint64_t frame); // cheats.c bool tab_cheats(float scale); // ui/cheats_tab.cpp; true: restart into the rebuild int playtime_get(uint64_t *total_s, uint64_t *session_s); // playtime.c +bool amiibo_check_font_rebuild(void); // ui/amiibo_page.cpp; check and clear font rebuild request } // Roboto Medium (Apache-2.0, ui/fonts/LICENSE-Roboto.txt) with Latin and Cyrillic glyphs. @@ -353,6 +355,9 @@ static int fonts_up(float scale) { if (const CheatInfo *ci = cheats_info(k); ci && ci->name) b.AddText(ci->name); for (int k = 0; k < mods_count(); k++) if (const ModInfo *mi = mods_at(k); mi) b.AddText(mi->name); + // Add amiibo filenames so they display correctly in the menu + amiibo_collect_filenames(add_row_glyphs, &b); + b.BuildRanges(&ranges[i]); ImFontConfig mc; mc.MergeMode = true; @@ -792,8 +797,8 @@ static const char *prompts_note(bool *restart) { const char *layout = cfg_value("TOTK_PAD_LAYOUT"), *used = romfs_prompts(); const char *want = input_prompts_folder(input_prompts_pick(cfg_get("TOTK_PROMPTS"), layout), layout); if (want == used || (want && used && !strcmp(want, used))) return nullptr; - // A game resolution of another shape needs a folder of its own for the PC prompts (romfs-patches-pc-x), which - // the setup does not write: the shape is the reason then, not the setup. + // A game resolution of another shape needs a folder of its own for the PC prompts (romfs-patches-pc-x). The + // setup writes one for each size the Screen tab offers; at any other size the shape is the reason, not the setup. int off = romfs_prompts_off(want); if (want && !romfs_prompts_ready(want) && off != ROMFS_PROMPTS_SHAPE) return menu_text(UI_MENU_PROMPTS_MISSING); switch (off) { @@ -906,6 +911,23 @@ static void row_threads(const char *key, UiTextId label, UiTextId desc) { opt_choice_list(key, label, vals, names, N, desc); } +// The chunked replay's recording threads: Off, then the counts; the note under the value is what this run uses, which +// a change replaces only after a restart. +static void row_replay_workers(void) { + static const char *const vals[] = {"0", "2", "3", "4", "6", "8", "12", "16"}; + enum { N = sizeof vals / sizeof *vals }; + const char *names[N]; + names[0] = menu_text(UI_MENU_OFF); + for (int i = 1; i < N; i++) names[i] = vals[i]; + static char now[64]; + char n[16]; + int w = vk_exec_replay_workers(); + snprintf(n, sizeof n, "%d", w); + if (w >= 0) snprintf(now, sizeof now, menu_text(UI_MENU_REPLAY_WORKERS_NOW), w ? n : menu_text(UI_MENU_OFF)); + opt_choice_list("TOTK_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, vals, names, N, UI_MENU_REPLAY_WORKERS_DESC, -1, + w >= 0 ? now : nullptr); +} + // A free-text key shown as Off (unset) / On (value set): On writes on_value, any set value reads as On. static void row_set_or_not(const char *key, UiTextId label, UiTextId desc, const char *on_value) { const char *vals[] = {"", on_value}, *names[] = {menu_text(UI_MENU_OFF), menu_text(UI_MENU_ON)}; @@ -955,6 +977,7 @@ static void tab_advanced(float scale) { static const UiTextId preempt[] = {UI_MENU_PREEMPT_OFF, UI_MENU_PREEMPT_ON, UI_MENU_PREEMPT_INHERIT}; // 0|1|2 opt_choice("TOTK_ASYNC_PREEMPT", UI_MENU_ASYNC_PREEMPT, preempt, 3, UI_MENU_ASYNC_PREEMPT_DESC); opt_toggle("TOTK_REPLAY_ASYNC", UI_MENU_REPLAY_ASYNC, UI_MENU_REPLAY_ASYNC_DESC); + row_replay_workers(); opt_toggle("TOTK_TIMER_1MS", UI_MENU_TIMER_1MS, UI_MENU_TIMER_1MS_DESC); opt_group(UI_MENU_GROUP_SHADERS); @@ -1537,7 +1560,9 @@ extern "C" int menu_overlay_frame(uint64_t frame, const MenuTarget *t, const cha uint64_t t0 = plat_now_ns(); if (!g_ctx_ok) make_context(); float scale = menu_scale(t->height); - if (!vk_up(t) || (g_font_scale != scale && !fonts_up(scale))) { + // Check if amiibo page requested font rebuild (e.g., after refreshing directory with new files) + bool amiibo_rebuild = amiibo_check_font_rebuild(); + if (!vk_up(t) || (g_font_scale != scale && !fonts_up(scale)) || (amiibo_rebuild && !fonts_up(scale))) { rt_log("menu: Dear ImGui Vulkan back end set-up failed; the menu stays closed"); g_failed.store(1); set_open(0); diff --git a/runtime/ui_text.c b/runtime/ui_text.c index 461f3eb9..bafaf7f4 100644 --- a/runtime/ui_text.c +++ b/runtime/ui_text.c @@ -388,6 +388,9 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_MENU_PREEMPT_OFF] = "Off", [UI_MENU_PREEMPT_ON] = "On", [UI_MENU_PREEMPT_INHERIT] = "Inheritance only", [UI_MENU_REPLAY_ASYNC] = "GPU frame prep on its own thread", [UI_MENU_REPLAY_ASYNC_DESC] = "Prepares frames for the GPU on a separate thread instead of the game thread. Adds up to one frame of latency.", + [UI_MENU_REPLAY_WORKERS] = "GPU frame prep threads", + [UI_MENU_REPLAY_WORKERS_DESC] = "Records large frames for the GPU on several threads at once. Helps when the CPU limits the frame rate; needs 8 or more logical processors. Off records on one thread.", + [UI_MENU_REPLAY_WORKERS_NOW] = "now: %s", [UI_MENU_TIMER_1MS] = "1 ms Windows timer", [UI_MENU_TIMER_1MS_DESC] = "Raises the system timer resolution to 1 ms for steadier frame pacing.", [UI_MENU_PIPE_WAIT] = "Waiting draw handling", @@ -866,6 +869,9 @@ static const char *const text_ru[UI_TEXT_COUNT] = { [UI_MENU_PREEMPT_OFF] = "Выкл.", [UI_MENU_PREEMPT_ON] = "Вкл.", [UI_MENU_PREEMPT_INHERIT] = "Только наследование", [UI_MENU_REPLAY_ASYNC] = "Подготовка кадра для GPU в отдельном потоке", [UI_MENU_REPLAY_ASYNC_DESC] = "Кадр для GPU готовится в отдельном потоке, а не в потоке игры. Добавляет до одного кадра задержки.", + [UI_MENU_REPLAY_WORKERS] = "Потоки подготовки кадра для GPU", + [UI_MENU_REPLAY_WORKERS_DESC] = "Большие кадры для GPU записываются сразу в нескольких потоках. Помогает, когда частоту кадров ограничивает процессор; нужно 8 логических процессоров или больше. «Выкл.» — запись в одном потоке.", + [UI_MENU_REPLAY_WORKERS_NOW] = "сейчас: %s", [UI_MENU_TIMER_1MS] = "Таймер Windows 1 мс", [UI_MENU_TIMER_1MS_DESC] = "Повышает точность системного таймера до 1 мс для ровного темпа кадров.", [UI_MENU_PIPE_WAIT] = "Обработка ожидающих отрисовок", @@ -1241,6 +1247,9 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = { [UI_MENU_PREEMPT_OFF] = "Desligado", [UI_MENU_PREEMPT_ON] = "Ligado", [UI_MENU_PREEMPT_INHERIT] = "Só herança", [UI_MENU_REPLAY_ASYNC] = "Preparo de quadro da GPU em thread própria", [UI_MENU_REPLAY_ASYNC_DESC] = "Prepara os quadros para a GPU numa thread separada em vez da thread do jogo. Adiciona até um quadro de latência.", + [UI_MENU_REPLAY_WORKERS] = "Threads de preparo de quadro da GPU", + [UI_MENU_REPLAY_WORKERS_DESC] = "Grava quadros grandes para a GPU em várias threads ao mesmo tempo. Ajuda quando a CPU limita a taxa de quadros; precisa de 8 ou mais processadores lógicos. Desligado grava numa thread só.", + [UI_MENU_REPLAY_WORKERS_NOW] = "agora: %s", [UI_MENU_TIMER_1MS] = "Timer do Windows de 1 ms", [UI_MENU_TIMER_1MS_DESC] = "Aumenta a resolução do timer do sistema para 1 ms, para um ritmo de quadros mais estável.", [UI_MENU_PIPE_WAIT] = "Tratamento de desenhos em espera", @@ -1850,6 +1859,10 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = { [UI_MENU_REPLAY_ASYNC] = "GPU 帧准备独立线程", [UI_MENU_REPLAY_ASYNC_DESC] = "在单独的线程而不是游戏线程上为 GPU 准备帧。" "最多增加一帧延迟。", + [UI_MENU_REPLAY_WORKERS] = "GPU 帧准备线程数", + [UI_MENU_REPLAY_WORKERS_DESC] = "同时在多个线程上为 GPU 录制大帧。CPU 限制帧率时有帮助;需要 8 个或更多逻辑处理器。" + "关:在单个线程上录制。", + [UI_MENU_REPLAY_WORKERS_NOW] = "当前:%s", [UI_MENU_TIMER_1MS] = "1 ms Windows 计时器", [UI_MENU_TIMER_1MS_DESC] = "将系统计时器精度提高到 1 ms,使帧间隔更平稳。", [UI_MENU_PIPE_WAIT] = "等待中绘制的处理", diff --git a/runtime/ui_text.h b/runtime/ui_text.h index bb8e2658..101eb702 100644 --- a/runtime/ui_text.h +++ b/runtime/ui_text.h @@ -176,6 +176,7 @@ typedef enum { UI_MENU_ASYNC_PREEMPT, UI_MENU_ASYNC_PREEMPT_DESC, UI_MENU_PREEMPT_OFF, UI_MENU_PREEMPT_ON, UI_MENU_PREEMPT_INHERIT, UI_MENU_REPLAY_ASYNC, UI_MENU_REPLAY_ASYNC_DESC, + UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, UI_MENU_REPLAY_WORKERS_NOW, UI_MENU_TIMER_1MS, UI_MENU_TIMER_1MS_DESC, UI_MENU_PIPE_WAIT, UI_MENU_PIPE_WAIT_DESC, UI_MENU_PIPE_WAIT_DEFAULT, UI_MENU_PIPE_WAIT_DRAW, UI_MENU_PIPE_SKIP, UI_MENU_PIPE_SKIP_DESC, UI_MENU_PIPE_SKIP_STRICT, UI_MENU_PIPE_SKIP_ANY, diff --git a/runtime/vk.h b/runtime/vk.h index 91a67370..ee50f2cd 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -44,6 +44,7 @@ typedef struct VkCtx { // VK_FEEDBACK_RESTART (the pass ends and restarts behind the full barrier). int feedback; // VK_FEEDBACK_* uint32_t min_imported_host_alignment; // VK_EXT_external_memory_host + VkDeviceSize vma_block; // VMA's block size for heaps over 1 GiB (TOTK_VMA_BLOCK_MIB, vk_device.c) int device_index; // index in vkEnumeratePhysicalDevices } VkCtx; enum { VK_FEEDBACK_OFF, VK_FEEDBACK_RESTART, VK_FEEDBACK_LAYOUT }; @@ -226,6 +227,10 @@ enum { VK_MCLS_NONE, VK_MCLS_HOST_NOACC, VK_MCLS_HOST_UNC, VK_MCLS_HOST_CACHED, enum { VK_MUSE_VB, VK_MUSE_IB, VK_MUSE_UBO, VK_MUSE_SSBO, VK_MUSE_IND, VK_MUSE_COPY_SRC, VK_MUSE_GPU_WRITE, VK_MUSE_CPU_READ, VK_MUSE_N }; void vk_mem_count(int cls, int use, uint64_t bytes); +// A thread recording a chunk of a submission (vk_exec.c) counts into its own acc from vk_mem_count_local(acc) on, until +// vk_mem_count_local(NULL); vk_mem_count_flush adds acc into the census and clears it. +void vk_mem_count_local(int64_t (*acc)[VK_MUSE_N]); +void vk_mem_count_flush(int64_t (*acc)[VK_MUSE_N]); // TOTK_TRACE=perf dirty census (vk_memory.c): the replay notes each imported pool it binds as a vertex, index, uniform, // storage or indirect buffer; at the start of every submit the pages the CPU wrote in such pools since the previous // submit are counted from the shared page generations and logged as `mem: dirty` every 60 frames. Nothing when tracing @@ -258,10 +263,14 @@ void vk_mem_gpu_wrote_shader(const VkMemSpan *sp, uint64_t size); // it records, ubo bindings take the copy (0: not copied, bind memory), GPU writes of the replay drop the copies they touch. void vk_usnap_begin(const uint64_t *handles, int n, uint64_t fence); void vk_usnap_end(uint64_t fence); -int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t align, VkDescriptorBufferInfo *out); +// at: the copies dropped at or before it are not found (vk_usnap_drop_at); UINT32_MAX outside a chunked replay. +int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t at, uint64_t align, VkDescriptorBufferInfo *out); const uint8_t *vk_usnap_map(void); // a host copy of the current arena, at the offsets vk_usnap_find gives const uint8_t *vk_usnap_bytes(uint64_t addr, uint64_t n); // the copy of [addr, addr+n) as of the submit, NULL: none void vk_usnap_drop(uint64_t addr, uint64_t n); // the replay makes the GPU write [addr, addr+n) of guest memory +// The state scan of a chunked replay (vk_exec.c): the op that writes [addr, addr+n) is the at-th that may; the chunks +// pass at with the ops they record (VkExec.drop_n) and find the copies only before it. +void vk_usnap_drop_at(uint64_t addr, uint64_t n, uint32_t at); uint64_t vk_usnap_gen(void); // changes whenever a binding may resolve differently void vk_usnap_shutdown(void); // vk_exec_draw_shutdown: frees the arenas // vk_mvprev.c: last frame's copy of the scene's object (c7) and bone (c5) blocks, for per-object motion vectors. @@ -292,13 +301,24 @@ int vk_mem_class_of(uint64_t addr); // VK_MCLS_* of the pool at ad void vk_mem_census(uint64_t frame); // at each present: pool census and traffic, logged at frame 1800 and, // with TOTK_TRACE=perf, every 1800 frames // VRAM in use and its budget, summed over the DEVICE_LOCAL heaps. exact 1: VK_EXT_memory_budget (the whole process), -// 0: this allocator's own allocations against the heap sizes. 0 when unknown. +// 0: this allocator's own allocations against the heap sizes. 0 when unknown. Where device memory takes host memory +// (vk_mem_host_room) the budget is at most the use plus that room. int vk_mem_vram(uint64_t *used, uint64_t *budget, int *exact); +// Bytes the system can still give device memory beyond a reserve (TOTK_MEM_RESERVE_MIB) where device memory takes host +// memory: a UMA GPU (commit and physical memory), or a discrete GPU whose device-local memory took process commit in +// vk_mem_host_probe (commit). UINT64_MAX elsewhere. +uint64_t vk_mem_host_room(void); +void vk_mem_host_probe(void); // vk_device.c, after the device: measures that for a discrete GPU, once +int vk_mem_uma(void); // 1: the device class is uma (vk_mem_class_log), its memory is RAM // VRAM headroom (vk_headroom.c): the VMA callbacks that see new device memory, and the thread that allocates the next // block of each memory type in use ahead of the render thread. const VmaDeviceMemoryCallbacks *vk_headroom_callbacks(void); void vk_headroom_start(void); void vk_headroom_stop(void); // before vmaDestroyAllocator +// The texture pool (vk_memory.c vk_texture_image) gets its next block ahead too; pool NULL frees the anchor in it. +void vk_headroom_pool(VmaPool pool, uint32_t type, VkDeviceSize block); +void vk_headroom_release(void); // after a failed allocation: frees the spare blocks and keeps none for a while; + // any thread that holds no VMA lock (takes only the headroom lock) int vk_mem_resolve(uint64_t addr, uint64_t size, VkMemSpan *out); // 1 when all of [addr, addr+size) is backed // Changes whenever vk_mem_resolve plus VK_MEM_USE may give another span for an address: a pool added, finalized or // mapped, a mirror created, refused or forgotten, a submission started. @@ -334,6 +354,8 @@ void vk_defer_buffer(VkBuffer buf, VmaAllocation alloc); void vk_defer_free(void *p); // Device-local scratch for the upload batch (storage, transfer src and dst), from a pool of its own (vk_memory.c). VkResult vk_scratch_buffer(VkDeviceSize size, VkBuffer *buf, VmaAllocation *alloc); +// A game texture's image with memory of its own: the texture pool, or a dedicated allocation (vk_memory.c). +VkResult vk_texture_image(const VkImageCreateInfo *ci, VkImage *img, VmaAllocation *alloc); // Staging and scratch bytes the open upload batch holds until it completes. Large texture loads submit the batch early // (vk_texture.c UPLOAD_HELD_MAX) so the GPU starts on it and its buffers come back before the pools need new blocks. VkDeviceSize vk_upload_held(void); @@ -387,6 +409,10 @@ int vk_mem_span_live(const VkMemSpan *sp); void vk_mem_note_fence(uint64_t timeline_value); void vk_mem_fault_log(void); // vk_device_fault_report: the deferred-free state, lock-free void vk_mem_collect(void); // frees what has completed +// After a failed allocation, on a thread that may hold the texture lock and the upload lock (cmd_lock) but no VMA +// lock: frees the headroom blocks and the completed upload batches and deferred objects, and requests the release +// of released textures at the next submit (rt_oom_request). +void vk_oom_release(void); typedef struct { uint64_t imported_pools, physical_pools, virtual_pools, import_failures, upload_batches; } VkMemStats; void vk_mem_stats(VkMemStats *s); void vk_mem_shutdown(void); // before vk_shutdown: waits idle, frees every memory and texture object @@ -426,6 +452,7 @@ typedef struct { void vk_hdr_present_source(const VkTexImage *win, VkImage *image, VkFormat *format); void vk_hdr_image_gone(VkImage image); const VkTexImage *vk_texture_get(const tex_t *t); // created and made current; NULL for BUFFER or unmapped formats +const VkTexImage *vk_texture_revive(const tex_t *t, uint32_t pos); // the entry vk_texture_get would return if it exists, made live at op pos // A texture slot; a slot whose texture was finalized follows the live texture at the same storage (hle_nvn.c). const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view); // The replay looks up many slots per draw: it takes the texture lock once around them (vk_texture_lock) and uses the @@ -433,6 +460,12 @@ const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view); void vk_texture_lock(void); void vk_texture_unlock(void); const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view); +// 1 when slot id's entry had its freshness check in this submission, so vk_texture_by_id_locked would only look it up. +// No lock: a context of a chunked replay asks before taking the texture lock. +int vk_texture_slot_checked(int id); +// 1 when the state scan revived a texture at the slot's storage in this submission: the slot's answer depends on the op +// a chunk records, so no memo holds it (vk_exec_draw.c); a chunk sees every revival up to its last op. +int vk_texture_slot_moves(int id); VkSampler vk_sampler_by_id_locked(int id); // The slot's sampler with nearest filtering and no anisotropy: a depth texture read without a compare lands on one texel // on the Switch although the game binds a linear sampler, and MacNativeGames samples it the same way. VK_NULL_HANDLE @@ -495,9 +528,22 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t // batch) without a signal, and returns a new begun one for the rest of the submission. The submission's timeline value // is signalled by its last part, whose signal covers every earlier part in submission order. VkCommandBuffer vk_sync_split(void); +// Inside vk_exec_record: submits cb, a part recorded and ended on another thread, after the upload batch and the +// mirror copies queued so far, without a signal (the submitting thread, in the order of the submission). +void vk_sync_part(VkCommandBuffer cb); +// GPU time of a part recorded on another thread (vk_sync.c "GPU time"): the start timestamp after vkBeginCommandBuffer, +// the pair or -1; the end before vkEndCommandBuffer. +int vk_sync_ts_open(VkCommandBuffer cb); +void vk_sync_ts_close(VkCommandBuffer cb, int pair, uint64_t fence); // Runs fn(arg) on the completion thread once fence is done on the GPU (event writes, counter reports, sync state). // Callbacks run in the order they were added, after those of earlier values; a value already done runs at once there. void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg); +// Callbacks of a part recorded on another thread: between vk_completion_defer(d) and vk_completion_defer(NULL) the +// calling thread's vk_completion_add calls collect in d, and vk_completion_flush adds them, in their order, when the +// part takes its place in the submission (vk_exec.c record_chunks). +typedef struct { void *items; int n, cap; } VkDeferred; +void vk_completion_defer(VkDeferred *d); +void vk_completion_flush(VkDeferred *d); // gfx_vulkan_backend (vk_backend.c): presents go to a present thread and run after the submissions made before them. void rdoc_init(void); // runtime/rdoc.c: TOTK_RDOC debug capture; before vkCreateInstance int rdoc_active(void); diff --git a/runtime/vk_astc.c b/runtime/vk_astc.c index ccfa4235..ff0b6028 100644 --- a/runtime/vk_astc.c +++ b/runtime/vk_astc.c @@ -300,17 +300,19 @@ static int create_pipeline(VkCtx *c) { // TOTK_ASTC_BC=1: the BC3 encoder, when the device samples BC3 (every desktop GPU does); 0 leaves ASTC as RGBA8. // Unset: on when the largest device-local heap is under 5 GiB. There RGBA8 ASTC, four times the size of BC3, fills // video memory (a GTX 1650 4 GiB went from 10-23 to 3-12 ms of GPU time a frame, issue #87); BC3 loses some precision -// (normal maps), so GPUs with more memory keep RGBA8. +// (normal maps), so GPUs with more memory keep RGBA8. A UMA GPU's heap is the machine's RAM however large it reads +// (10.6 GiB on an ROG Ally X, 14.9 GiB on a Radeon 880M), so there BC3 is on too: on the RX 9070, BC3 took 737 MiB off +// VMA's device-local allocations after a minute in the field, at the same GPU time. static int create_bc_pipeline(VkCtx *c) { const char *e = cfg_get("TOTK_ASTC_BC"); VkDeviceSize vram = 0; for (uint32_t h = 0; h < c->mem_props.memoryHeapCount && h < VK_MAX_MEMORY_HEAPS; h++) if (c->mem_props.memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT && c->mem_props.memoryHeaps[h].size > vram) vram = c->mem_props.memoryHeaps[h].size; - int on = e && *e ? *e != '0' : vram < (5ull << 30); + int on = e && *e ? *e != '0' : vram < (5ull << 30) || vk_mem_uma(); if (!(e && *e)) - rt_log("vk: TOTK_ASTC_BC unset: BC3 %s (largest device-local heap %llu MiB)", on ? "on" : "off", - (unsigned long long)(vram >> 20)); + rt_log("vk: TOTK_ASTC_BC unset: BC3 %s (largest device-local heap %llu MiB%s)", on ? "on" : "off", + (unsigned long long)(vram >> 20), vk_mem_uma() ? ", uma: system memory" : ""); if (!on) return 0; VkFormatProperties p; vkGetPhysicalDeviceFormatProperties(c->phys, VK_FORMAT_BC3_SRGB_BLOCK, &p); diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index 5749de6b..5e7535e7 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -49,6 +49,7 @@ static void wait_until(uint64_t at) { static void present_main(void *arg) { (void)arg; + perf_core_thread(PCORE_PRESENT); uint64_t last = 0, last_queued = 0; plat_mutex_lock(&pl); for (;;) { @@ -201,6 +202,7 @@ static uint64_t submit_sync(const uint64_t *handles, int n) { static void replay_main(void *arg) { (void)arg; perf_replay_thread(); + perf_core_thread(PCORE_REPLAY); plat_mutex_lock(&rl); for (;;) { while (!rq_n) plat_cond_wait(&rcv, &rl); @@ -326,6 +328,8 @@ void gfx_exit_log(void) { } static void vk_be_present(const tex_t *t, const int crop[4], uint64_t at_ns) { + static _Thread_local int pcore; // the game's render thread: the one that presents + if (!pcore) pcore = 1, perf_core_thread(PCORE_RENDER); vk_sync_start(); if (!gfx_replay_async()) { uint64_t frame = __atomic_add_fetch(&gfx_counters.frames, 1, __ATOMIC_RELAXED); diff --git a/runtime/vk_contact.c b/runtime/vk_contact.c index 74eae718..15c73045 100644 --- a/runtime/vk_contact.c +++ b/runtime/vk_contact.c @@ -7,9 +7,13 @@ // passes behind nearby geometry: the small shadows at feet, under stones and in grass that the shadow maps are too // coarse for. Transparent surfaces, the sky and post-processing come after, untouched. // -// Inputs from the game, read from the fragment stage's uniform block c4 bound in the composite pass (guest memory, -// so it may be one frame newer than the replay): the projection scales c4[7].x and c4[8].y, the far plane c4[14].y, -// and the first sun cascade's depth row c4[44].xyz, whose negated direction points from a view-space position toward +// vk_contact_plan decides that in replay order and vk_contact_exec records it: a chunked replay's scan +// plans the march (vk_exec.c scan_contact) and the chunk that holds its op records it. +// +// Inputs from the game, read where the march is planned from the fragment stage's uniform block c4 bound in the +// composite pass (guest memory, so it may be one frame newer than the replay): the projection scales c4[7].x and +// c4[8].y, the far plane c4[14].y, and the first sun cascade's depth row c4[44].xyz, whose negated direction points from +// a view-space position toward // the sun (the G-buffer shaders compare dot(c4[44].xyz, pos) + c4[44].w with the cascade map). The scene colour images // get STORAGE usage when the setting is on (vk_texture.c create_tex). #include "vk.h" @@ -23,22 +27,22 @@ #include "vk_exec.h" static struct { - int tried, ok, strength, debug, logged; + int read, tried, ok, strength, debug, logged; PFN_vkCmdPushDescriptorSetKHR push; VkDescriptorSetLayout dsl; VkPipelineLayout layout; VkPipeline pipe; VkSampler sampler; - const VkTexImage *scene, *depth; // the last G-buffer's scene colour and linear depth - uint32_t w, h; + VkExecTarget scene, depth; // the last G-buffer's scene colour and linear depth int pending; // the composite into scene ran; the march waits for the next pass uint64_t c4; } C; int vk_contact_strength(void) { - if (!C.tried) { + if (!C.read) { // once: the scan asks at every draw int s = cfg_int("TOTK_CONTACT_SHADOWS"); C.strength = s < 0 ? 0 : s > 100 ? 100 : s; + C.read = 1; } return C.strength; } @@ -90,16 +94,16 @@ static int init(VkCtx *c) { } // A G-buffer pass: a B10G11R11 scene colour, an R32F linear depth and a D32F depth, all one size, at least 640 wide. -static int is_gbuffer(const VkExec *ex, const VkTexImage **scene, const VkTexImage **depth) { +static int is_gbuffer(const VkExec *ex, const VkExecTarget **scene, const VkExecTarget **depth) { *scene = *depth = NULL; if (ex->pass_noatt || ex->pass_w < 640 || ex->depth.format != VK_FORMAT_D32_SFLOAT || ex->depth.level) return 0; for (uint32_t i = 0; i < ex->pass_ncolor && i < 8; i++) { const VkExecTarget *t = &ex->color[i]; if (!t->image || t->level || t->width != ex->pass_w || t->height != ex->pass_h) continue; - if (t->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32 && !*scene) *scene = t->img; - if (t->format == VK_FORMAT_R32_SFLOAT && !*depth) *depth = t->img; + if (t->format == VK_FORMAT_B10G11R11_UFLOAT_PACK32 && !*scene) *scene = t; + if (t->format == VK_FORMAT_R32_SFLOAT && !*depth) *depth = t; } - return *scene && *depth && ((*scene)->usage & VK_IMAGE_USAGE_STORAGE_BIT); + return *scene && *depth && (*scene)->img && ((*scene)->img->usage & VK_IMAGE_USAGE_STORAGE_BIT); } static void mem_barrier(VkCommandBuffer cb) { @@ -112,28 +116,62 @@ static void mem_barrier(VkCommandBuffer cb) { vkCmdPipelineBarrier2(cb, &d); } -void vk_contact_flush(VkExec *ex) { - if (!C.pending) return; +// The lighting composite: the remembered scene colour as the only target, with a D16 depth of its size. +static int is_composite(const VkExec *ex) { + return C.scene.image && ex->pass_ncolor >= 1 && ex->color[0].image == C.scene.image && !ex->color[0].level && + ex->depth.format == VK_FORMAT_D16_UNORM && ex->pass_w == C.scene.width && ex->pass_h == C.scene.height; +} + +int vk_contact_plan(const VkExec *ex, int pass, VkExecJob *j) { + if (!vk_contact_strength()) return 0; + if (pass) { + const VkExecTarget *scene, *depth; + // The camera block: c4 of the fragment stage, 0x940 bytes. The composite binds it only after its pass begins, so + // the newest one seen at any pass start serves. + const p_bind_buf *b = &ex->ubo[1][1]; + if (b->addr && b->size == 0x940) C.c4 = b->addr; + if (is_gbuffer(ex, &scene, &depth)) { + C.pending = 0; + C.scene = *scene, C.depth = *depth; + return 0; + } + if (is_composite(ex)) { + C.pending = C.c4 != 0; + return 0; + } + } + if (!C.pending) return 0; C.pending = 0; - if (ex->in_pass) return; const float *u = (const float *)(uintptr_t)C.c4; float lx = -u[44 * 4 + 0], ly = -u[44 * 4 + 1], lz = -u[44 * 4 + 2], ll = sqrtf(lx * lx + ly * ly + lz * lz); float px = u[7 * 4 + 0], py = u[8 * 4 + 1], zfar = u[14 * 4 + 1]; - if (!(ll > 1e-6f) || !(px > 0) || !(py > 0) || !(zfar > 1)) return; - VkImageView in = vk_exec_target_view(C.depth, VK_FORMAT_R32_SFLOAT, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT, + if (!(ll > 1e-6f) || !(px > 0) || !(py > 0) || !(zfar > 1)) return 0; + float w = (float)C.scene.width, h = (float)C.scene.height; + *j = (VkExecJob){.scene = C.scene, .depth = C.depth, + .pc = {px, py, zfar, (float)C.strength / 100.0f, + lx / ll, ly / ll, lz / ll, 0.6f, + w, h, 1.0f / w, 1.0f / h, + 0, 0, 0, 0}}; + return 1; +} + +void vk_contact_exec(VkExec *ex, const VkExecJob *j) { + if (!C.tried) init(vk_ctx()); + if (!C.ok) return; + VkImageView in = vk_exec_target_view(j->depth.img, VK_FORMAT_R32_SFLOAT, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1, 0); - VkImageView out = vk_exec_target_view(C.scene, VK_FORMAT_B10G11R11_UFLOAT_PACK32, VK_IMAGE_VIEW_TYPE_2D, + VkImageView out = vk_exec_target_view(j->scene.img, VK_FORMAT_B10G11R11_UFLOAT_PACK32, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1, 0); if (!in || !out) return; + vk_exec_pass_end(ex); VkCommandBuffer cb = ex->cb; - float pcv[16] = {px, py, zfar, (float)C.strength / 100.0f, - lx / ll, ly / ll, lz / ll, 0.6f, - (float)C.w, (float)C.h, 1.0f / (float)C.w, 1.0f / (float)C.h, - (float)C.debug, 0, 0, 0}; + float pcv[16]; + memcpy(pcv, j->pc, sizeof pcv); + pcv[12] = (float)C.debug; if (!C.logged) { C.logged = 1; - rt_log("vk: contact shadows on a %ux%u scene, sun toward %.3f %.3f %.3f (view space), far %g", C.w, C.h, - pcv[4], pcv[5], pcv[6], zfar); + rt_log("vk: contact shadows on a %ux%u scene, sun toward %.3f %.3f %.3f (view space), far %g", j->scene.width, j->scene.height, + pcv[4], pcv[5], pcv[6], pcv[2]); } mem_barrier(cb); // the compose passes' writes, before the march reads and writes them VkDescriptorImageInfo ii = {C.sampler, in, VK_IMAGE_LAYOUT_GENERAL}, oi = {VK_NULL_HANDLE, out, VK_IMAGE_LAYOUT_GENERAL}; @@ -145,36 +183,21 @@ void vk_contact_flush(VkExec *ex) { vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, C.pipe); rec_push(C.push, cb, VK_PIPELINE_BIND_POINT_COMPUTE, C.layout, 0, 2, w); vkCmdPushConstants(cb, C.layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof pcv, pcv); - vkCmdDispatch(cb, (C.w + 7) / 8, (C.h + 7) / 8, 1); + vkCmdDispatch(cb, (j->scene.width + 7) / 8, (j->scene.height + 7) / 8, 1); mem_barrier(cb); // and what follows sees the darkened colour ex->recorded += 6; - vk_exec_draw_new_cb(); // the game's bound compute pipeline and descriptors were replaced: bind them again -} - -// The lighting composite: the remembered scene colour as the only target, with a D16 depth of its size. -static int is_composite(const VkExec *ex) { - return C.scene && ex->pass_ncolor >= 1 && ex->color[0].img == C.scene && !ex->color[0].level && - ex->depth.format == VK_FORMAT_D16_UNORM && ex->pass_w == C.w && ex->pass_h == C.h; + vk_exec_draw_new_cb(ex); // the game's bound compute pipeline and descriptors were replaced: bind them again } +// The sequential replay's hooks; a chunk leaves the marches to the scan. void vk_contact_pass_begin(VkExec *ex) { - if (!C.tried && !init(vk_ctx())) return; - if (!C.ok) return; - const VkTexImage *scene, *depth; - // The camera block: c4 of the fragment stage, 0x940 bytes. The composite binds it only after its pass begins, so - // the newest one seen at any pass start serves. - const p_bind_buf *b = &ex->ubo[1][1]; - if (b->addr && b->size == 0x940) C.c4 = b->addr; - if (is_gbuffer(ex, &scene, &depth)) { - C.pending = 0; - C.scene = scene, C.depth = depth, C.w = ex->pass_w, C.h = ex->pass_h; - return; - } - if (is_composite(ex)) { - C.pending = C.c4 != 0; - return; - } - vk_contact_flush(ex); + VkExecJob j; + if (!ex->chunked && vk_contact_plan(ex, 1, &j)) vk_contact_exec(ex, &j); +} + +void vk_contact_flush(VkExec *ex) { + VkExecJob j; + if (!ex->chunked && C.pending && vk_contact_plan(ex, 0, &j)) vk_contact_exec(ex, &j); } void vk_contact_release(void) { diff --git a/runtime/vk_crumbs.c b/runtime/vk_crumbs.c index 3d34e024..a5a1e84d 100644 --- a/runtime/vk_crumbs.c +++ b/runtime/vk_crumbs.c @@ -91,7 +91,7 @@ void vk_crumbs_report(void) { // records changed pipeline state also notes the shaders of a pass's first draw. A full barrier already precedes every // pass and dispatch (vk_exec.c vk_exec_pass_begin, vk_exec_draw.c op_dispatch), so the end-of-pipe write adds no wait. static VkPassMark ring[VK_PASS_MARKS]; -static uint32_t seq; // marks made; only the replay thread writes it +static uint32_t seq; // marks made (atomic: the recording threads of a chunked replay) // TOTK_PASS_CRUMBS: unset, buffer marks where VK_AMD_buffer_marker exists; 1, also NVIDIA checkpoints; 0, neither. int vk_pass_marks_want_nv(void) { @@ -124,8 +124,8 @@ void vk_pass_mark_cmd(VkCommandBuffer cb, uint32_t id, int end) { VkPassMark *vk_pass_mark_begin(VkCommandBuffer cb, int kind, uint64_t fence) { if (!vk_pass_marks_on || !cb) return NULL; - uint32_t id = ++seq; - if (!id) id = ++seq; // 0 means no mark + uint32_t id = __atomic_add_fetch(&seq, 1, __ATOMIC_RELAXED); // the recording threads of a chunked replay + if (!id) id = __atomic_add_fetch(&seq, 1, __ATOMIC_RELAXED); // 0 means no mark VkPassMark *m = &ring[id & (VK_PASS_MARKS - 1)]; m->id = id, m->kind = (uint8_t)kind, m->fence = fence, m->draws = UINT32_MAX; rec_pass_mark(cb, id, 0); diff --git a/runtime/vk_device.c b/runtime/vk_device.c index 31fbc1a4..760a3a46 100644 --- a/runtime/vk_device.c +++ b/runtime/vk_device.c @@ -429,8 +429,14 @@ static int create_device(void) { g_vk.min_imported_host_alignment = (uint32_t)emh.minImportedHostPointerAlignment; vk_caps_init(&g_vk); + // Blocks of 64 MiB instead of VMA's 256 (TOTK_VMA_BLOCK_MIB, 0 = VMA's). Since the textures moved to their own pool + // (vk_memory.c vk_texture_image), the default device-local blocks hold mirrors and other buffers: on the RX 9070 + // after 10 minutes in the field one 256 MiB block held 53 MiB of them, and the headroom thread kept half a block of + // each device-local type ahead (128 MiB each). Larger buffers than half a block get memory of their own. + g_vk.vma_block = (VkDeviceSize)cfg_int("TOTK_VMA_BLOCK_MIB") << 20; + if (!g_vk.vma_block) g_vk.vma_block = 256ull << 20; // VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE VmaAllocatorCreateInfo ai = {.physicalDevice = g_vk.phys, .device = g_vk.device, .instance = g_vk.instance, - .vulkanApiVersion = VK_API_VERSION_1_3, + .vulkanApiVersion = VK_API_VERSION_1_3, .preferredLargeHeapBlockSize = g_vk.vma_block, .flags = (g_vk.has_buffer_device_address ? VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT : 0) | (g_vk.has_memory_budget ? VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT : 0), .pDeviceMemoryCallbacks = vk_headroom_callbacks()}; @@ -475,7 +481,8 @@ int vk_init(void) { plat_mutex_lock(&g_lock); if (!g_ready) { memset(&g_vk, 0, sizeof g_vk); - if (create_instance() && pick_device() && create_device()) g_ready = 1, vk_upload_pools_init(), vk_headroom_start(), vk_dlaa_init(); + if (create_instance() && pick_device() && create_device()) + g_ready = 1, vk_mem_host_probe(), vk_upload_pools_init(), vk_headroom_start(), vk_dlaa_init(); else destroy_all(); } int ok = g_ready; diff --git a/runtime/vk_dlaa.c b/runtime/vk_dlaa.c index 1fe39bde..194c23b9 100644 --- a/runtime/vk_dlaa.c +++ b/runtime/vk_dlaa.c @@ -242,7 +242,11 @@ int vk_dlaa_available(void) { return vk_dlaa_wanted() && N.available; } static struct { VkExecTarget color, depth; // the armed pair, valid in the frame it was armed int armed, done, gbuf; // this frame: armed on the light target, DLAA ran, the G-buffer was drawn + uint32_t gbuf_w; // the widest G-buffer's width this frame + VkImage nodepth; // this frame: a G-buffer-size target drawn without depth while nothing was armed int ended, up; // this frame: the scene is drawn (jitter over), DLSS wrote the scaled output + uint64_t end_fence; // a chunked replay: the submission and op the scan ended the scene at (ended_at) + uint32_t end_pos; struct Size { // a DLSS feature and its work images per scene size (dynamic resolution flips it) uint32_t w, h, ow, oh; // render size, output size VkImage out, mv; @@ -557,7 +561,7 @@ static int mv_pass(VkExec *ex, struct Size *e, VkImageView zv, float jx, float j VkCommandBuffer cb = ex->cb; uint32_t w = e->w, h = e->h; VkDescriptorBufferInfo cam; - if (!mv_init() || !D.cam_addr || !vk_usnap_find(D.cam_addr, CAM_BLOCK, 4, &cam)) { + if (!mv_init() || !D.cam_addr || !vk_usnap_find(D.cam_addr, CAM_BLOCK, UINT32_MAX, 4, &cam)) { D.mv_none++; D.cam_prev_ok = 0; e->chars_ok = 0; @@ -777,10 +781,9 @@ static struct Snap *snap_for(VkCommandBuffer cb, uint32_t w, uint32_t h, VkForma } // At arming: the light target as it is before the transparents, outside any pass. -static void mask_snapshot(VkExec *ex) { +static void mask_snapshot(VkExec *ex, const VkExecTarget *t) { R.cur = NULL; if (mask_init() <= 0) return; - const VkExecTarget *t = &D.color; struct Snap *s = snap_for(ex->cb, t->width, t->height, t->format); if (!s) return; vk_exec_barrier(ex); @@ -829,14 +832,33 @@ static float halton(uint32_t i, uint32_t b) { return r; } +// The scene is drawn at the op ex records. A chunked replay's scan ends it ahead of the chunks that still record the +// scene's draws, so a chunk compares its op with the scan's (the chunks after it are taken once the scan is past their +// ops); a later submission of the frame is past it. +static int ended_at(const VkExec *ex) { + if (!__atomic_load_n(&D.ended, __ATOMIC_ACQUIRE)) return 0; + return !ex->chunked || ex->fence != D.end_fence || ex->pos >= D.end_pos; +} + void vk_dlaa_viewport(VkExec *ex, VkViewport *v) { - if (!D.jit_ok || D.ended || !ex->depth.image || ex->depth.image != D.jdepth) return; + if (!D.jit_ok || !ex->depth.image || ex->depth.image != D.jdepth || ended_at(ex)) return; v->x += D.jx, v->y += D.jy; - D.ux = D.jx, D.uy = D.jy, D.jit_used = 1; + D.ux = D.jx, D.uy = D.jy; + __atomic_store_n(&D.jit_used, 1, __ATOMIC_RELAXED); +} + +// The scene camera: c4 of the last scene draw before the scene ends. A chunked replay's scan notes it in replay order +// (vk_dlaa_scan_draw): chunks recording side by side would leave the block of whichever drew last. +static void note_camera(const VkExec *ex) { + const p_bind_buf *b = &ex->ubo[0][1]; // constant bank c4: the replay keeps user banks from c3 on + const VkExecTarget *z = &ex->depth; + if (b->size == 0x940 && b->addr && z->image && z->width >= DLAA_SCENE_MIN_W && z->format == VK_FORMAT_D32_SFLOAT && + (!D.jdepth || z->image == D.jdepth)) + D.cam_addr = b->addr; } void vk_dlaa_draw(VkExec *ex) { - if (!dl_active() || D.ended) return; + if (!dl_active() || ended_at(ex)) return; static int survey = -1; // TOTK_DLAA_SURVEY=1 (test): what is drawn into the armed light target if (survey < 0) survey = rt_knob("DLAA_SURVEY", "0"); static uint64_t dfn[16]; // scene draws writing depth, by depth test (reversed depth: GREATER/GEQUAL) @@ -867,11 +889,11 @@ void vk_dlaa_draw(VkExec *ex) { } } } - const p_bind_buf *b = &ex->ubo[0][1]; // constant bank c4: the replay keeps user banks from c3 on - const VkExecTarget *z = &ex->depth; - if (b->size == 0x940 && b->addr && z->image && z->width >= DLAA_SCENE_MIN_W && z->format == VK_FORMAT_D32_SFLOAT && - (!D.jdepth || z->image == D.jdepth)) - D.cam_addr = b->addr; + if (!ex->chunked) note_camera(ex); +} + +void vk_dlaa_scan_draw(const VkExec *ex) { + if (dl_active() && !D.ended) note_camera(ex); } // The jitter moves on right after DLAA ran, not at the present: the replay can record the next frame's scene before @@ -1045,7 +1067,7 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { full_barrier(cb); if (dst != t) vk_dlaa_swap_img = t->img, vk_dlaa_swap_view = e->out_view; // the scaling draw reads DLSS's picture ex->recorded += 4; - vk_exec_draw_new_cb(); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again + vk_exec_draw_new_cb(ex); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again D.jdepth = z->image; next_jitter(w, ow); if (++D.evals == 1 || D.evals == 300 || D.evals % 3600 == 0) // 300: one early line with the frames counted @@ -1062,51 +1084,110 @@ static int scene_color(VkFormat f) { return f == VK_FORMAT_B10G11R11_UFLOAT_PACK32 || f == VK_FORMAT_R16G16B16A16_SFLOAT; } -void vk_dlaa_pass(VkExec *ex) { +int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { const VkExecTarget *c = &ex->color[0], *z = &ex->depth; - if (!dl_active() || D.done) return; + if (!dl_active() || D.done) return 0; // The G-buffer: several colour targets with a D32F depth of at least DLAA_SCENE_MIN_W wide. Only the light target drawn after it // arms the frame; at night the game draws into the light target alone with the scene depth before the G-buffer too, // and DLAA then ran at a small pass ahead of the whole scene, which reached the screen jittered. if (ex->pass_ncolor >= 3 && z->image && z->format == VK_FORMAT_D32_SFLOAT && z->width >= DLAA_SCENE_MIN_W) { + D.gbuf_w = D.gbuf && D.gbuf_w > z->width ? D.gbuf_w : z->width; D.gbuf = 1, D.armed = 0; // target 0 is the object-class buffer (RGBA8, the scene size): the characters' bias mask reads its green. A // smaller pass of the same shape (640x360, seen while loading) does not replace the scene's in the same frame. if (c->image && c->format == VK_FORMAT_R8G8B8A8_UNORM && c->width == z->width && c->height == z->height && c->samples <= 1 && !c->is3d && (!D.ids_ok || c->width >= D.ids.width)) D.ids = *c, D.ids_ok = 1; - return; + return 0; + } + // A pass into one scene colour target of the G-buffer's size or larger without depth while nothing is armed ends + // the scene once the bloom chain follows with still nothing armed: in the inventory (1.4.3) the preview's light + // target is drawn that way before the bloom, and with the scene depth only after it, while the menu is composed; + // DLAA ran on the finished menu picture with the previous frame's history, which left stray dark pixels around Link. + // Arming cancels the end, on that target too: at some camera angles the field's effects composite (a full-screen + // draw without depth into the light target) comes before the first pass with the scene depth, and with that target + // barred from arming DLSS stopped for as long as the view held (in the inventory the bloom chain always ended the + // scene first, in both modes, menu opening included). While the scope zooms in, a second G-buffer pass disarms the + // frame, the scene is copied into an output-sized target without depth, and only then is the light target drawn + // again with the scene depth. + // The G-buffer's width is the widest of the frame: the game also draws a half-size G-buffer, after which a + // half-size pass without depth ended the scene at some camera angles. + if (D.gbuf && !D.armed && !D.ended && !z->image && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && + c->width >= D.gbuf_w && !c->level) + D.nodepth = c->image; + if (D.nodepth && !D.armed && !D.ended && ex->pass_w && ex->pass_w * 4 <= D.gbuf_w && c->image && + scene_color(c->format)) { + D.end_fence = ex->fence, D.end_pos = ex->pos; + __atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE); } // the scene depth is D32F; a D16 target of the scene size also serves one composite pass before the transparents. // A smaller pair never takes over an armed one: the camera rune draws a half-size RGBA16F target with a half-size // D32F depth after the lighting (852x480 under a 1704x960 scene), and DLSS ran on that and wrote its picture into - // the scene's own light target, which left every object black. - if (D.gbuf && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && z->image && z->format == VK_FORMAT_D32_SFLOAT && - c->width == z->width && c->height == z->height && c->width >= DLAA_SCENE_MIN_W && + // the scene's own light target, which left every object black. Nor does a pair narrower than the frame's widest + // G-buffer arm at all: the game draws its particles into a half-size RGBA16F target with a half-size D32F depth, and + // with the scope open, where the light target is drawn with a D16 depth and never arms, that effects pass was the + // first pair to arm; DLSS then scaled the effects into the light target, which filled the lens with garbage in the + // DLAA mode. + if (D.gbuf && !D.ended && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && z->image && z->format == VK_FORMAT_D32_SFLOAT && + c->width == z->width && c->height == z->height && c->width >= DLAA_SCENE_MIN_W && c->width >= D.gbuf_w && (!D.armed || (c->width >= D.color.width && c->height >= D.color.height)) && c->samples <= 1 && !c->is3d) { int first = !D.armed; - D.color = *c, D.depth = *z, D.armed = 1; - if (first) mask_snapshot(ex); // the lit scene, before the transparents (reactive mask) - return; + D.color = *c, D.depth = *z, D.armed = 1, D.nodepth = VK_NULL_HANDLE; + if (!first) return 0; + *j = (VkExecJob){.kind = VK_JOB_DLSS_MASK, .scene = *c}; // the lit scene, before the transparents (reactive mask) + return 1; } - if (!D.armed) return; + if (!D.armed) return 0; if (!D.ended) { - if (!ex->pass_w || ex->pass_w * 4 > D.color.width) return; - D.ended = 1; + // The bloom chain's first pass ends the scene: a pass at most a quarter of the scene's width into a scene + // colour target. With the camera rune or the scope open the game draws a 160-wide depth-only pass and 160x90 + // RG16F and RGBA8 passes right after the lighting, then more geometry and the transparents; DLSS ended the + // scene at those, the next G-buffer pass disarmed it and the scaling pass found nothing to evaluate. + if (!ex->pass_w || ex->pass_w * 4 > D.color.width || !c->image || !scene_color(c->format)) return 0; + D.end_fence = ex->fence, D.end_pos = ex->pos; + __atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE); unsigned o[2], s[2]; res_sizes(o, s); - if (D.color.width >= o[0]) D.done = 1, evaluate(ex, NULL); // DLAA in place; else wait for the scaling pass - return; + if (D.color.width < o[0]) return 0; // wait for the scaling pass + D.done = 1; + *j = (VkExecJob){.kind = VK_JOB_DLSS}; // DLAA in place + return 1; } // the game's scaling pass: one colour target of the output size, no depth unsigned o[2], s[2]; res_sizes(o, s); if (ex->pass_ncolor == 1 && c->image && !z->image && scene_color(c->format) && c->width > D.color.width && c->width == o[0] && c->height == o[1] && c->samples <= 1 && !c->is3d) { - D.done = D.up = 1; - evaluate(ex, c); + D.done = 1; + *j = (VkExecJob){.kind = VK_JOB_DLSS, .scene = *c}; + return 1; } + return 0; +} + +// The scan resolved its targets without making entries (vk_texture_revive); a record made on a chunk takes their images +// now, as the scan's pointers may be stale by then. +static void target_now(VkExecTarget *t) { + const VkTexImage *i = t->tex ? vk_texture_get(t->tex) : NULL; + if (i) t->img = i, t->image = i->image; +} + +void vk_dlaa_exec(VkExec *ex, const VkExecJob *j) { + VkExecTarget t = j->scene; + if (ex->chunked) { + target_now(&t), target_now(&D.color), target_now(&D.depth); + if (D.ids_ok) target_now(&D.ids); + } + if (j->kind == VK_JOB_DLSS_MASK) return mask_snapshot(ex, &t); + if (!t.tex) return evaluate(ex, NULL); + D.up = 1; + evaluate(ex, &t); +} + +void vk_dlaa_pass(VkExec *ex) { + VkExecJob j; + if (!ex->chunked && vk_dlaa_plan(ex, &j)) vk_dlaa_exec(ex, &j); } // TOTK_DLSS, read at each present (the menu switches it live): the scene size of the mode (resolution.c) and the @@ -1149,6 +1230,7 @@ void vk_dlaa_frame(void) { D.on = on; // either way the next DLAA frame starts a new history from an unjittered frame D.reset = 1, D.cam_prev_ok = 0, D.jit_ok = D.jit_used = D.armed = D.done = D.gbuf = D.ended = D.up = 0; + D.nodepth = VK_NULL_HANDLE; D.hold = 0; return; } @@ -1157,7 +1239,7 @@ void vk_dlaa_frame(void) { if (D.jit_used && !D.done) D.hold = 60, D.holds++; // jittered but not resolved: that frame shook else if (D.hold) D.hold--; if (!D.done) D.jit_ok = 0; - D.armed = D.done = D.jit_used = D.gbuf = D.ended = D.up = D.ids_ok = 0; + D.armed = D.done = D.jit_used = D.gbuf = D.ended = D.up = D.ids_ok = 0, D.nodepth = VK_NULL_HANDLE; D.cam_addr = 0; } @@ -1178,18 +1260,29 @@ void vk_dlaa_frame(void) { static const uint64_t FXAA_FS[] = {0xf450003ebeb3dc52ull, 0x0a12f4bc2a9a4fbeull, 0x5c43ae18948d52cbull, 0xaad42238801d31d8ull}; +// Chunks of one submission look shaders up side by side: an entry's hash is stored before its shader, and entries are +// added under the lock. static uint64_t spirv_hash(const GfxShader *s) { static struct { const GfxShader *s; uint64_t h; } tab[8192]; + static PlatMutex lk = PLAT_MUTEX_INIT; uintptr_t k = ((uintptr_t)s >> 4) * 0x9E3779B97F4A7C15ull; for (uint32_t i = 0, j = (uint32_t)(k >> 51); i < 8192; i++, j = (j + 1) & 8191) { - if (tab[j].s == s) return tab[j].h; - if (!tab[j].s) { + const GfxShader *e = __atomic_load_n(&tab[j].s, __ATOMIC_ACQUIRE); + if (e == s) return tab[j].h; + if (e) continue; + plat_mutex_lock(&lk); + e = tab[j].s; + if (!e) { uint64_t words = 0, h = 0xcbf29ce484222325ull; const uint8_t *b = (const uint8_t *)gfx_shader_spirv(s, &words); for (uint64_t n = 0; b && n < words * 4; n++) h = (h ^ b[n]) * 0x100000001b3ull; - tab[j].s = s, tab[j].h = h; + tab[j].h = h; + __atomic_store_n(&tab[j].s, s, __ATOMIC_RELEASE); + plat_mutex_unlock(&lk); return h; } + plat_mutex_unlock(&lk); + if (e == s) return tab[j].h; } return 0; // table full: no skip } diff --git a/runtime/vk_dlaa.h b/runtime/vk_dlaa.h index 82b98113..cce8877d 100644 --- a/runtime/vk_dlaa.h +++ b/runtime/vk_dlaa.h @@ -18,11 +18,17 @@ int vk_dlaa_available(void); // scene's light target or, at the first quarter-width pass after it, runs DLAA on that target. Each present ends the // frame (vk_dlaa_frame). typedef struct VkExec VkExec; +typedef struct VkExecJob VkExecJob; void vk_dlaa_pass(VkExec *ex); +// The same decision for a chunked replay's scan, in replay order: 1 with the record due before this pass (j), which +// vk_dlaa_exec makes on the chunk that holds the op (vk_exec.c run_job). +int vk_dlaa_plan(const VkExec *ex, VkExecJob *j); +void vk_dlaa_exec(VkExec *ex, const VkExecJob *j); void vk_dlaa_frame(void); // vk_exec_draw.c, each draw before its fixed state is recorded: notes the scene camera block (vk_dlaa_draw), and moves // the viewport of the scene's draws by the frame's jitter (vk_dlaa_viewport, when the viewport is recorded). void vk_dlaa_draw(VkExec *ex); +void vk_dlaa_scan_draw(const VkExec *ex); // a chunked replay's scan, at each draw: the camera note of vk_dlaa_draw void vk_dlaa_viewport(VkExec *ex, VkViewport *v); // vk_mvsurvey.c (TOTK_MV_SURVEY): notes the uniform blocks of each scene draw, beside vk_dlaa_draw; each present // compares them with the previous frames' and logs a summary now and then. diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index 6408d2be..a28b0a1b 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -26,8 +26,8 @@ #include "platform/platform.h" #include "tex.h" #include "vk.h" -#include "vk_exec.h" #include "vk_dlaa.h" +#include "vk_exec.h" #define WEAK __attribute__((weak)) #define CALL_DEPTH 8 @@ -43,23 +43,38 @@ typedef struct { static VkExec X; static int x_ready; -static _Thread_local int x_here; // this thread is inside vk_exec_record (vk_exec_inline_begin) +static _Thread_local VkExec *x_cur; // the context this thread records with (vk_exec_inline_begin), NULL outside // ---- TOTK_TRACE=gfx frame counters VkExecFrame vk_exec_frame; static struct { const uint8_t *mem; uint32_t w, h, fmt, n; } fw[64]; // textures written this frame static int nfw; +static PlatMutex fw_lock = PLAT_MUTEX_INIT; // the recording threads of a chunked replay -uint64_t vk_exec_recording_fence(void) { return x_here ? X.fence : 0; } +uint64_t vk_exec_recording_fence(void) { return x_cur ? x_cur->fence : 0; } +uint32_t vk_exec_replay_pos(void) { return x_cur && x_cur->chunked ? x_cur->pos : UINT32_MAX; } +static _Thread_local int chunk_k = -1; // the chunk this thread records (chunk_record), -1 none + + +// A context's counters go into vk_exec_frame (the submitting thread, after the recording). +static void frame_add(VkExec *ex) { + uint64_t *d = (uint64_t *)&vk_exec_frame, *s = (uint64_t *)&ex->fr; + for (size_t i = 0; i < sizeof ex->fr / sizeof *s; i++) d[i] += s[i]; + memset(&ex->fr, 0, sizeof ex->fr); +} void vk_exec_frame_written(const tex_t *t) { if (!(rt_trace & RT_TRACE_GFX) || !t) return; - for (int i = 0; i < nfw; i++) - if (fw[i].mem == t->mem) { fw[i].n++; return; } - if (nfw == 64) return; - fw[nfw].mem = t->mem, fw[nfw].w = t->d.width, fw[nfw].h = t->d.height, fw[nfw].fmt = (uint32_t)t->d.format; - fw[nfw++].n = 1; + plat_mutex_lock(&fw_lock); + int i = 0; + while (i < nfw && fw[i].mem != t->mem) i++; + if (i < nfw) fw[i].n++; + else if (nfw < 64) { + fw[nfw].mem = t->mem, fw[nfw].w = t->d.width, fw[nfw].h = t->d.height, fw[nfw].fmt = (uint32_t)t->d.format; + fw[nfw++].n = 1; + } + plat_mutex_unlock(&fw_lock); } void vk_exec_frame_log(const tex_t *p, uint64_t frame) { @@ -120,6 +135,7 @@ static void state_reset(VkExec *ex) { ex->depth_range[1] = 1.0f; vk_pipe_state_defaults(&ex->pipe); ex->barrier_at = -1; + ex->oq_open = -1; } // ---- render targets @@ -265,10 +281,11 @@ static void bc_copy_back(VkExec *ex, const VkExecTarget *t) { // A draw into a 3D BC image without block views through an uncompressed view of its blocks: the target is the // image's stand-in (bc_standin). 1 when o is filled (target or left out), 0 for any other target. -static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImage *img, VkExecTarget *o) { +static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImage *img, VkExecTarget *o, int peek) { int bw, bh; if (img->type != VK_IMAGE_TYPE_3D || img->block_views || !nvn_fmt_bpb(t->d.format, &bw, &bh) || bw != 4 || bh != 4) return 0; + if (peek) return 1; // the scan makes no stand-in: left out VkFormat fmt; BcStandIn s; uint32_t level = has_v && v->set_levels ? v->base_level : 0; @@ -299,12 +316,20 @@ static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImag return 1; } -static void resolve_target(const tex_t *t, const view_t *v, int has_v, int depth_slot, VkExecTarget *o) { +enum { HOLD_TEX, HOLD_DLSS, HOLD_WRITES, HOLD_N }; +static void scan_hold(int why); + +// scan: the state scan's resolution, which creates nothing and leaves out a texture without an image yet. +static void resolve_target(const tex_t *t, const view_t *v, int has_v, int depth_slot, VkExecTarget *o, const VkExec *scan) { memset(o, 0, sizeof *o); o->tex = t; if (!t) return; - const VkTexImage *img = vk_texture_get(t); - if (img && !depth_slot && bc_target(t, v, has_v, img, o)) return; // a 3D BC image: its stand-in + const VkTexImage *img = scan ? vk_texture_revive(t, scan->pos) : vk_texture_get(t); + if (scan && !img) { + if (t->mem) scan_hold(HOLD_TEX); + return; + } + if (img && !depth_slot && bc_target(t, v, has_v, img, o, scan != NULL)) return; // a 3D BC image: its stand-in VkImageUsageFlags need = depth_slot ? VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT : VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT; int is3d = img && img->type == VK_IMAGE_TYPE_3D && !depth_slot; // slices drawn through 2D views (2D_ARRAY_COMPATIBLE) if (!img || !(img->usage & need) || (img->type != VK_IMAGE_TYPE_2D && !is3d)) { @@ -382,7 +407,7 @@ void vk_exec_noatt_dims(const VkExec *ex, uint32_t *w, uint32_t *h) { } // Resolves ex->rt to this command buffer's images and derives the pass parameters and the pipeline's target formats. -static void resolve_rt(VkExec *ex) { +static void resolve_rt_as(VkExec *ex, int peek) { if (ex->rt_resolved) return; ex->rt_resolved = 1; const p_set_rt *r = &ex->rt; @@ -390,7 +415,7 @@ static void resolve_rt(VkExec *ex) { uint32_t w = UINT32_MAX, h = UINT32_MAX, layers = UINT32_MAX, samples = 0, scale = 1; for (int i = 0; i < 8; i++) { VkExecTarget *t = &ex->color[i]; - resolve_target(i < n ? r->color[i] : NULL, &r->cview[i], i < n && r->has_cview[i], 0, t); + resolve_target(i < n ? r->color[i] : NULL, &r->cview[i], i < n && r->has_cview[i], 0, t, peek ? ex : NULL); ex->pass_color[i] = t->image ? t->format : VK_FORMAT_UNDEFINED; ex->pipe.color_fmt[i] = t->image ? t->nvn_format : 0; if (!t->image) continue; @@ -401,7 +426,7 @@ static void resolve_rt(VkExec *ex) { if (t->scale > scale) scale = t->scale; } VkExecTarget *d = &ex->depth; - resolve_target(r->depth, &r->dview, r->has_dview, 1, d); + resolve_target(r->depth, &r->dview, r->has_dview, 1, d, peek ? ex : NULL); ex->pass_depth = d->image && (d->aspect & VK_IMAGE_ASPECT_DEPTH_BIT) ? d->format : VK_FORMAT_UNDEFINED; ex->pass_stencil = d->image && (d->aspect & VK_IMAGE_ASPECT_STENCIL_BIT) ? d->format : VK_FORMAT_UNDEFINED; ex->pipe.depth_fmt = d->image ? d->nvn_format : 0; @@ -417,7 +442,7 @@ static void resolve_rt(VkExec *ex) { vk_exec_noatt_dims(ex, &w, &h); layers = 1; static int logged; - if (logged < 8 && ex->has_vp && (ex->vp[0][0] != 0 || ex->vp[0][1] != 0 || ex->vp[0][2] != w || ex->vp[0][3] != h)) + if (!peek && logged < 8 && ex->has_vp && (ex->vp[0][0] != 0 || ex->vp[0][1] != 0 || ex->vp[0][2] != w || ex->vp[0][3] != h)) logged++, rt_log("vk_exec: pass without attachments %ux%u for viewport %g,%g %gx%g", w, h, ex->vp[0][0], ex->vp[0][1], ex->vp[0][2], ex->vp[0][3]); } @@ -428,6 +453,8 @@ static void resolve_rt(VkExec *ex) { ex->pipe.samples = (uint8_t)ex->pass_samples; } +static void resolve_rt(VkExec *ex) { resolve_rt_as(ex, 0); } + // The attachment view of a target. A registered texture's view lives as long as its image (vk_exec_target_view); any // other image's view is made for this command buffer and freed at its fence. static VkImageView view_of(VkExec *ex, const VkExecTarget *t) { @@ -474,6 +501,7 @@ static void free_views(void *arg) { // completion thread, once the submission // collision makes a draw wait rather than skip. typedef struct { VkImage image; uint32_t level, layer; uint64_t last, prev; } TargetFrames; static TargetFrames tfr[1024]; +static PlatMutex tfr_lock = PLAT_MUTEX_INIT; // the recording threads of a chunked replay static TargetFrames *tfr_slot(const VkExecTarget *t) { uint64_t h = ((uint64_t)(uintptr_t)t->image ^ (uint64_t)t->level << 56 ^ (uint64_t)t->base_layer << 40) * @@ -487,13 +515,18 @@ static int tfr_match(const TargetFrames *e, const VkExecTarget *t) { static void tfr_note(const VkExecTarget *t, uint64_t stamp) { TargetFrames *e = tfr_slot(t); + plat_mutex_lock(&tfr_lock); if (!tfr_match(e, t)) *e = (TargetFrames){t->image, t->level, t->base_layer, 0, 0}; if (e->last != stamp) e->prev = e->last, e->last = stamp; + plat_mutex_unlock(&tfr_lock); } static int tfr_recent(const VkExecTarget *t, uint64_t stamp) { const TargetFrames *e = tfr_slot(t); - return tfr_match(e, t) && (e->last == stamp ? e->prev : e->last) == stamp - 1; + plat_mutex_lock(&tfr_lock); + int r = tfr_match(e, t) && (e->last == stamp ? e->prev : e->last) == stamp - 1; + plat_mutex_unlock(&tfr_lock); + return r; } // No frame before the current one rendered the target (a slot another subresource took reads the same). @@ -575,7 +608,7 @@ static void barrier_to(VkExec *ex, const VkImage *want, const VkImageAspectFlags for (int i = 0; i < nwant; i++) ex->fbl_img[i] = want[i], ex->fbl_aspect[i] = wasp[i]; ex->nfbl = nwant; if (ex->barrier_at == ex->recorded && !n) return; - vk_exec_frame.fb_layouts += n; + ex->fr.fb_layouts += n; VkMemoryBarrier2 mb = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, .srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT, @@ -639,8 +672,8 @@ void vk_exec_pass_begin(VkExec *ex) { m->depth = ex->pass_depth ? ex->pass_depth : ex->pass_stencil; // The shaders bound now are the first draw's when it repeats the pipeline state; when it does not, // vk_exec_draw.c prepare_draw replaces them as it records the new state. - vk_exec_draw_shaders(&m->first[0], &m->first[1]); - ex->mark_draws0 = vk_exec_frame.draws; + vk_exec_draw_shaders(ex, &m->first[0], &m->first[1]); + ex->mark_draws0 = ex->fr.draws; } VkRenderingInfo ri = {.sType = VK_STRUCTURE_TYPE_RENDERING_INFO, .renderArea = {{0, 0}, {ex->pass_w, ex->pass_h}}, .layerCount = ex->pass_layers, .colorAttachmentCount = ex->pass_ncolor, @@ -648,7 +681,7 @@ void vk_exec_pass_begin(VkExec *ex) { .pDepthAttachment = ex->pass_depth ? &da : NULL, .pStencilAttachment = ex->pass_stencil ? &sa : NULL}; vkCmdBeginRendering(ex->cb, &ri); - vk_exec_frame.passes++; + ex->fr.passes++; ex->recorded++; ex->in_pass = 1; } @@ -660,10 +693,10 @@ void vk_exec_pass_end(VkExec *ex) { ex->recorded++; ex->in_pass = 0; if (ex->mark) { - // vk_exec_frame_log resets the frame counters at a present (TOTK_TRACE=gfx): then only the draws since it count - uint64_t d = vk_exec_frame.draws, d0 = ex->mark_draws0; + // frame_add empties the context's counters at the end of a submission: then only the draws since it count + uint64_t d = ex->fr.draws, d0 = ex->mark_draws0; ex->mark->draws = (uint32_t)(d >= d0 ? d - d0 : d); - if (ex->mark->draws) vk_exec_draw_shaders(&ex->mark->last[0], &ex->mark->last[1]); + if (ex->mark->draws) vk_exec_draw_shaders(ex, &ex->mark->last[0], &ex->mark->last[1]); vk_pass_mark_end(ex->cb, ex->mark); ex->mark = NULL; } @@ -678,17 +711,17 @@ void vk_exec_pass_end(VkExec *ex) { } VkCommandBuffer vk_exec_inline_begin(int *reopen) { - VkExec *ex = &X; + VkExec *ex = x_cur; *reopen = 0; - if (!x_here || !ex->cb) return VK_NULL_HANDLE; + if (!ex || !ex->cb) return VK_NULL_HANDLE; *reopen = ex->in_pass; vk_exec_barrier(ex); // ends the pass; feedback-layout targets go back to GENERAL return ex->cb; } void vk_exec_inline_end(int reopen) { - X.recorded++; - if (reopen) vk_exec_pass_begin(&X); + x_cur->recorded++; + if (reopen) vk_exec_pass_begin(x_cur); } typedef struct { VkBuffer b; VmaAllocation a; } InlineBuf; @@ -705,7 +738,7 @@ VkBuffer vk_exec_inline_buffer(VkDeviceSize size) { .sharingMode = VK_SHARING_MODE_EXCLUSIVE}; VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; if (!t || vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &t->b, &t->a, NULL) != VK_SUCCESS) { free(t); return VK_NULL_HANDLE; } - vk_completion_add(X.fence, free_inline_buf, t); + vk_completion_add(x_cur->fence, free_inline_buf, t); return t->b; } @@ -746,7 +779,7 @@ static void clear_attachment(VkExec *ex, const VkClearAttachment *a) { } static void clear_color(VkExec *ex, const p_clear_color *c) { - vk_exec_frame.clears++; + ex->fr.clears++; if (ex->render_disabled) return; resolve_rt(ex); int i = c->index; @@ -824,8 +857,8 @@ static void set_rt(VkExec *ex, const p_set_rt *r) { // A part also spans at least TOTK_SPLIT_US (1000) of replay time: a split costs about 40 us of the render thread // (vkQueueSubmit2, a command buffer, the mirror prologue), and a fight frame of 2,000 draws made 15 parts on the 7900X; // a millisecond of recording is a fraction of a millisecond of GPU work, so the GPU still starts early. -static int split_n = -1, split_draws, part_start; -static uint64_t split_ns, part_t0; +static int split_n = -1; +static uint64_t split_ns; static uint64_t n_parts, n_subs; static void maybe_split(VkExec *ex) { @@ -837,103 +870,777 @@ static void maybe_split(VkExec *ex) { rt_log("vk: replay command buffer split every %d draws between passes, parts of at least %d us%s", split_n, us > 0 ? us : 0, split_n ? "" : " (off)"); } - if (!split_n || ex->in_pass || split_draws < split_n) return; - if (split_ns && kern_now_ns() - part_t0 < split_ns) return; // split_draws stays: the next draw asks again - split_draws = 0; - if (ex->recorded == part_start) return; + if (!split_n || ex->in_pass || ex->split_draws < split_n) return; + if (split_ns && kern_now_ns() - ex->part_t0 < split_ns) return; // split_draws stays: the next draw asks again + ex->split_draws = 0; + if (ex->recorded == ex->part_start) return; ex->cb = vk_sync_split(); - part_start = ex->recorded; - part_t0 = kern_now_ns(); + ex->part_start = ex->recorded; + ex->part_t0 = kern_now_ns(); ex->dirty = VK_EXEC_DIRTY_ALL; - vk_exec_draw_new_cb(); + vk_exec_draw_new_cb(ex); n_parts++; } -static void walk(VkExec *ex, const op_hdr *h, int depth) { +// The fixed-function state ops: writes into ex alone, applied by the replay and the state scan. 0: not one of them. +static int state_op(VkExec *ex, uint32_t op, const void *a) { + switch (op) { + case OP_VIEWPORTS: { + const p_viewports *v = a; + for (int i = 0; i < v->n && v->first + i < 16; i++) + if (v->first + i >= 0) memcpy(ex->vp[v->first + i], v->v[i], sizeof ex->vp[0]); + ex->has_vp = 1; + ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; + break; + } + case OP_SCISSOR: + ex->scissor = *(const p_scissor *)a; + ex->has_scissor = 1; + ex->dirty |= VK_EXEC_DIRTY_SCISSOR; + break; + case OP_DEPTH_RANGE: { + const p_depth_range *r = a; + ex->depth_range[0] = r->n, ex->depth_range[1] = r->f; + ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; + break; + } + case OP_STENCIL: { + const p_stencil *s = a; + for (int f = 0; f < 2; f++) { + if (!(s->face & (1 << f))) continue; + if (s->which == 0) ex->st_vmask[f] = (uint32_t)s->value_mask; + else if (s->which == 1) ex->st_wmask[f] = (uint32_t)s->mask; + else ex->st_ref[f] = (uint32_t)s->ref; + } + ex->dirty |= VK_EXEC_DIRTY_STENCIL; + break; + } + case OP_POLY_OFFSET: + ex->poly_offset = *(const p_poly_offset *)a; + ex->dirty |= VK_EXEC_DIRTY_DEPTH_BIAS; + break; + case OP_BLEND_COLOR: + memcpy(ex->blend_color, ((const p_blend_color *)a)->rgba, sizeof ex->blend_color); + ex->dirty |= VK_EXEC_DIRTY_BLEND_CONST; + break; + case OP_DEPTH_CLAMP: ex->pipe.depth_clamp = ((const p_flag *)a)->value != 0; break; + case OP_RASTER_DISCARD: ex->pipe.raster_discard = ((const p_flag *)a)->value != 0; break; + case OP_PRIM_RESTART: ex->prim_restart = *(const p_prim_restart *)a; break; + case OP_COPY_STRIDE: ex->copy_stride = ((const p_copy_stride *)a)->stride; break; + case OP_SET_TEX_POOL: ex->tex_pool = ((const p_set_pool *)a)->pool, ex->tex_serial++; break; + case OP_SET_SAMP_POOL: ex->samp_pool = ((const p_set_pool *)a)->pool; break; + default: return 0; + } + return 1; +} + +// A record the scan planned for this point (vk_contact_plan, vk_dlaa_plan): recorded before the op that triggers it, the +// draw's pass begun after it as in prepare_draw. The records share their module's pipelines and state, so they go in +// the scan's order; chunks are taken in order, so the chunk of an earlier record is being recorded or done. DLSS's +// evaluation reads what the scene's draws noted on the chunks before it (the camera block, the jitter used, the +// per-object motion list), so it waits until they are recorded; the chunks after it wait for it (a hold, scan_job). +static uint32_t job_turn; +static void chunks_wait(int k); + +static void run_job(VkExec *ex) { + uint32_t turn = ex->job_i; + VkExecJob j = ex->jobs[ex->job_i++]; + if (j.kind == VK_JOB_DLSS) chunks_wait(j.chunk); + while (__atomic_load_n(&job_turn, __ATOMIC_ACQUIRE) != turn) plat_yield(); + vk_exec_pass_end(ex); + if (j.kind != VK_JOB_CONTACT) { + vk_dlaa_exec(ex, &j); + } else { + // The scan resolved the scene and its depth without creating entries (vk_texture_revive): their images now. + const VkTexImage *si = vk_texture_get(j.scene.tex), *di = vk_texture_get(j.depth.tex); + if (si && di) { + j.scene.img = si, j.scene.image = si->image, j.depth.img = di, j.depth.image = di->image; + vk_contact_exec(ex, &j); + } + } + __atomic_store_n(&job_turn, turn + 1, __ATOMIC_RELEASE); + ex->dirty = VK_EXEC_DIRTY_ALL; + vk_exec_draw_new_cb(ex); +} + +// Replays from h to OP_END; 1 when it stopped at stop instead, an op at depth 0 (a chunk's end, record_chunks). +// The ops that may make the GPU write a uniform range the submission's snapshot copied (vk_usnap_drop_at). +static int drop_op(uint32_t op) { return op == OP_UPDATE_UB || op == OP_COPY_B2B || op == OP_COPY_T2B || op == OP_CLEAR_BUF; } + +static int walk(VkExec *ex, const op_hdr *h, int depth, const op_hdr *stop) { for (;;) { if (h->size < sizeof *h || (h->size & 7) || h->op >= OP_COUNT_) rt_fatal("vk: corrupt command record at %p: op %u size %u", (const void *)h, h->op, h->size); + if (h == stop) return 1; + while (ex->job_i < ex->njobs && h == ex->jobs[ex->job_i].op) run_job(ex); const void *a = nvn_op_payload(h); + if (h->op == OP_END) return 0; + if (ex->chunked) { + ex->drop_n += (uint32_t)drop_op(h->op), ex->pos++; + if (ex->wpos_i < ex->wpos_end && ex->wpos[ex->wpos_i] == ex->pos) ex->drop_n++, ex->wpos_i++; + } switch (h->op) { - case OP_END: return; case OP_BEGIN: break; case OP_JUMP: h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; continue; case OP_CALL: if (depth >= CALL_DEPTH) rt_fatal("vk: command buffer calls nested deeper than %d", CALL_DEPTH); - walk(ex, nvn_op_first(((const p_jump *)a)->target), depth + 1); + walk(ex, nvn_op_first(((const p_jump *)a)->target), depth + 1, NULL); break; case OP_SET_RT: set_rt(ex, a); break; case OP_CLEAR_COLOR: clear_color(ex, a); break; case OP_CLEAR_DS: clear_ds(ex, a); break; - case OP_VIEWPORTS: { - const p_viewports *v = a; - for (int i = 0; i < v->n && v->first + i < 16; i++) - if (v->first + i >= 0) memcpy(ex->vp[v->first + i], v->v[i], sizeof ex->vp[0]); - ex->has_vp = 1; - ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; - break; - } - case OP_SCISSOR: - ex->scissor = *(const p_scissor *)a; - ex->has_scissor = 1; - ex->dirty |= VK_EXEC_DIRTY_SCISSOR; - break; - case OP_DEPTH_RANGE: { - const p_depth_range *r = a; - ex->depth_range[0] = r->n, ex->depth_range[1] = r->f; - ex->dirty |= VK_EXEC_DIRTY_VIEWPORT; - break; - } - case OP_STENCIL: { - const p_stencil *s = a; - for (int f = 0; f < 2; f++) { - if (!(s->face & (1 << f))) continue; - if (s->which == 0) ex->st_vmask[f] = (uint32_t)s->value_mask; - else if (s->which == 1) ex->st_wmask[f] = (uint32_t)s->mask; - else ex->st_ref[f] = (uint32_t)s->ref; - } - ex->dirty |= VK_EXEC_DIRTY_STENCIL; - break; - } - case OP_POLY_OFFSET: - ex->poly_offset = *(const p_poly_offset *)a; - ex->dirty |= VK_EXEC_DIRTY_DEPTH_BIAS; - break; - case OP_BLEND_COLOR: - memcpy(ex->blend_color, ((const p_blend_color *)a)->rgba, sizeof ex->blend_color); - ex->dirty |= VK_EXEC_DIRTY_BLEND_CONST; - break; - case OP_DEPTH_CLAMP: ex->pipe.depth_clamp = ((const p_flag *)a)->value != 0; break; - case OP_RASTER_DISCARD: ex->pipe.raster_discard = ((const p_flag *)a)->value != 0; break; - case OP_PRIM_RESTART: ex->prim_restart = *(const p_prim_restart *)a; break; - case OP_COPY_STRIDE: ex->copy_stride = ((const p_copy_stride *)a)->stride; break; - case OP_SET_TEX_POOL: ex->tex_pool = ((const p_set_pool *)a)->pool, ex->tex_serial++; break; - case OP_SET_SAMP_POOL: ex->samp_pool = ((const p_set_pool *)a)->pool; break; case OP_BARRIER: vk_exec_barrier(ex); break; case OP_DRAW: case OP_DRAW_IDX: case OP_DISPATCH: case OP_UPDATE_UB: case OP_DRAW_IDX_IND: case OP_DISPATCH_IND: - if (h->op == OP_DRAW || h->op == OP_DRAW_IDX || h->op == OP_DRAW_IDX_IND) vk_exec_frame.draws_seen++, split_draws++; - if (h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND) vk_exec_frame.dispatches++, split_draws++; + if (h->op == OP_DRAW || h->op == OP_DRAW_IDX || h->op == OP_DRAW_IDX_IND) ex->fr.draws_seen++, ex->split_draws++; + if (h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND) ex->fr.dispatches++, ex->split_draws++; case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: case OP_BIND_UBO: case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: vk_exec_op_draw(ex, h->op, a); break; case OP_COPY_B2B: case OP_COPY_B2T: case OP_COPY_T2B: case OP_COPY_T2T: case OP_CLEAR_TEX: - if (h->op != OP_CLEAR_TEX) vk_exec_frame.copies++; + if (h->op != OP_CLEAR_TEX) ex->fr.copies++; if (vk_dump_armed) vk_dump_copy(ex, h->op, a); // fall through case OP_SIGNAL_EVENT: case OP_REPORT_COUNTER: case OP_RESET_COUNTER: case OP_RENDER_ENABLE: case OP_RENDER_COND: case OP_CLEAR_BUF: vk_exec_op_xfer(ex, h->op, a); break; - default: rt_fatal("vk: op %s has no replay handler", nvn_op_name(h->op)); + default: + if (!state_op(ex, h->op, a)) rt_fatal("vk: op %s has no replay handler", nvn_op_name(h->op)); } - if (split_draws >= split_n) maybe_split(ex); + if (ex->split_draws >= split_n && !ex->chunked) maybe_split(ex); h = nvn_op_next(h); } } +// ---- chunked recording (TOTK_REPLAY_WORKERS) +// A submission is recorded in chunks that start between passes, by TOTK_REPLAY_WORKERS threads (0: the sequential +// replay, the default) and twice as many chunks (TOTK_REPLAY_CHUNKS), so a thread +// that finishes early takes another: in the town at 4K on an RTX 5070 the replay goes from 8.2 to 4.5 ms +// and the frame rate from 71 to 79; more threads add little, since the frame then waits for the GPU. A sequential scan on +// the submitting thread applies only what is queue state: the fixed-function and bind ops, render targets, render +// enable and conditions, samples-passed counting, the contact shadows' marches (vk_contact_plan) and DLSS's decisions +// (vk_dlaa_plan); it looks up no +// shader, pipeline or descriptor and makes no Vulkan call. Where a chunk starts it keeps a copy of that state +// (VK_EXEC_STATE_BEGIN..END) and how far the scan's lists got: the submission's inline uniform updates (ubo_ptr), +// RENDER_COND results, planned records (contact shadows, DLSS) and the ops that drop uniform copies (vk_usnap_drop_at, +// scan_writes), which a chunk takes +// from the lists instead of from earlier chunks. A chunk starts with every image in GENERAL and ends with a full +// barrier, as a part of vk_sync_split does, so its recording depends only on the copy and the lists. +// A chunk can be taken as soon as the scan has placed the next one and the threads finished what the chunks share, which +// they do beside the scan (prep_run); they record into command buffers of their own, each with a context of its own +// (memos, descriptor pools, transient memory), and make their Vulkan calls themselves. The submitting thread records none while there are such threads: it submits the chunks in order as they +// are done, between the scan's steps and after it (vk_sync_part: the uploads and mirror copies queued so far go first), +// with their completion callbacks, so the callbacks keep the order of the sequential replay. The first chunk is a +// quarter of the others: the GPU starts sooner. A chunk ends where the replay ends a pass once it has its share of +// draws, or inside a pass at half as many more (TotK's G-buffer and shadow passes have 1,000 to 2,000 draws): the next +// chunk begins the pass again on the loaded attachments. +// The scan resolves render targets without creating images and revives finalized ones at their op (vk_texture_revive); +// a chunk, which the scan is ahead of, asks for a texture slot as of its own op (VkExec.pos, vk_exec_replay_pos). +// A texture that has no entry yet is made by the chunk that reaches it, and only that chunk has its position: its +// content from a live texture at the storage (vk_texture.c alias_copy) goes at that op. A later chunk sampling it first +// would follow another texture at the storage or copy over what the earlier chunk renders, so the chunks after the +// first use wait until that chunk is recorded (SC.holds). TotK makes a few such targets a frame: thumbnails, and depth +// targets whose size follows the camera; a wrong one darkened the whole scene for some frames. +// TOTK_REPLAY_STATE_CHECK=1 compares the state after the last chunk with the scan's. +#define MAX_JOBS 64 +#define MAX_RECS 16 + +typedef struct { + int hi; // handle index and record of the first op + const op_hdr *op; + uint32_t upd_n, cond_i, job_i, job_end, drop_n, pos; // the scan's lists, drop and op positions before it; its records + uint32_t wpos_i, wpos_end; // its writing draws and dispatches in SC.wpos + VkExec st; // queue state at its start + VkCommandBuffer cb; // recorded, ended, submitted in its turn + VkDeferred defer; // its completion callbacks + VkExec *by; // the context that recorded it + uint64_t ns, start; // its recording time, and when it began after SC.t0 + int draws; // draws and dispatches in it + volatile int32_t done; +} Chunk; + +typedef struct { + VkExec ex; // the thread's recording context + VkCommandPool pool; + struct { VkCommandBuffer cb; uint64_t fence; } *cbs; // reusable once the timeline reached fence + int ncbs; + int64_t traffic[VK_MCLS_N][VK_MUSE_N]; // vk_mem_count_local + PlatThread *t; +} Rec; + +typedef struct { uint64_t addr, size; uint32_t at; } ScanDrop; + +static struct { + int workers, chunks, min_draws, check; + int skew_us; // TOTK_REPLAY_SKEW_US (chunk_record) + VkExec s; // the scan's state + Chunk *ch; // chunks + 1 entries, never moved while threads record + int nch; + VkExecUpd *upd; // sized before the scan (count_ops): threads read them while it appends + uint32_t nupd, cupd; + uint8_t *cond; + uint32_t ncond, ccond; + VkExecJob jobs[MAX_JOBS]; + uint32_t njobs; + uint32_t drop_n; // scan: ops so far that may write a uniform range + uint32_t *wpos; // scan: the positions of the draws and dispatches among them, sized like upd + uint32_t nwpos, cwpos; + ScanDrop *dr; // scan: its uniform drops, applied once the snapshot exists (chunks_publish) + uint32_t ndr, cdr, dr_done; + int pub; // chunks published + uint64_t holds; // chunks that make a new texture or record writes the scan does not know: the + // ones after them wait until they are done + int per, open, in_chunk; // scan: draws for this chunk, a pass of the replay may be open, draws in it + int sub; // chunks submitted + uint64_t t0; // the submission's start + // TOTK_TRACE=perf, per chunked submission: the scan, the first and the last chunk submitted after t0, the chunks' + // recording summed and the longest + uint64_t subs, chunked, nchunks, scan_ns, first_ns, last_ns, rec_ns, rec_max_ns, bad, prep_ns, start0_ns; + uint64_t rec_t0; // vk_exec_record's start + uint64_t held[HOLD_N]; // TOTK_TRACE=perf: chunks held, by scan_hold's reason + uint64_t settles, settle_ns; // TOTK_TRACE=perf: vk_exec_chunk_settle waits (atomic: recording threads) + uint64_t mirror_ns, usnap_ns, ready_ns; // vk_mem_mirror_begin, vk_usnap_begin, both done after t0 +} SC = {.workers = -1}; + +static Rec recs[MAX_RECS + 1]; // [0]: the submitting thread's, used without recording threads +static int nrecs, nthreads; +static struct { + PlatMutex m; + PlatCond go, done; // more chunks can be taken or a new submission; a chunk is done + uint32_t gen; // bumped for every chunked submission + // gen << 32 | scan done << 31 | chunks that can be taken << 16 | next to take + volatile uint64_t take; + const uint64_t *handles; + int n, prio; + uint64_t fence; + int prep_next, prep_left; // prep_run: the next job, the jobs not done +} RP; + +static void *grow_n(void *p, uint32_t *cap, size_t elem) { + *cap = *cap ? *cap * 2 : 64; + p = realloc(p, elem * *cap); + if (!p) rt_fatal("vk: out of memory"); + return p; +} + +static void copy_state(VkExec *dst, const VkExec *src) { + memcpy((char *)dst + VK_EXEC_STATE_BEGIN, (const char *)src + VK_EXEC_STATE_BEGIN, VK_EXEC_STATE_END - VK_EXEC_STATE_BEGIN); +} + +static int draw_op(uint32_t op) { return op == OP_DRAW || op == OP_DRAW_IDX || op == OP_DRAW_IDX_IND; } + +static void check_op(const op_hdr *h) { + if (h->size < sizeof *h || (h->size & 7) || h->op >= OP_COUNT_) + rt_fatal("vk: corrupt command record at %p: op %u size %u", (const void *)h, h->op, h->size); +} + +// Draws and dispatches, inline uniform updates and conditions of a command stream. +typedef struct { uint32_t draws, upd, cond; } OpCounts; + +static void count_ops(const op_hdr *h, int depth, OpCounts *c) { + for (;;) { + check_op(h); + const void *a = nvn_op_payload(h); + if (h->op == OP_END) return; + if (h->op == OP_JUMP) { + h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; + continue; + } + if (h->op == OP_CALL && depth < CALL_DEPTH) count_ops(nvn_op_first(((const p_jump *)a)->target), depth + 1, c); + c->draws += draw_op(h->op) || h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND; + c->upd += h->op == OP_UPDATE_UB; + c->cond += h->op == OP_RENDER_COND; + h = nvn_op_next(h); + } +} + +// The take word: chunks [0, avail) can be taken, final once the scan placed them all. +static void take_publish(int avail, int final) { + plat_mutex_lock(&RP.m); + uint64_t w = __atomic_load_n(&RP.take, __ATOMIC_RELAXED); + while (!__atomic_compare_exchange_n(&RP.take, &w, (w & ~(0xffffull << 16 | 1ull << 31)) | (uint64_t)avail << 16 | (uint64_t)final << 31, + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) {} + plat_cond_broadcast(&RP.go); + plat_mutex_unlock(&RP.m); +} + +// What every chunk needs before it records, independent of each other and of the scan, so they run beside it on the +// recording threads and the submission's chunks start sooner: the mirrors' write-watch queries and the uniform snapshot. +static void prep_run(void) { + for (;;) { + int j = __atomic_fetch_add(&RP.prep_next, 1, __ATOMIC_ACQ_REL); + if (j >= 2) return; + uint64_t t0 = kern_now_ns(); + if (j == 0) vk_mem_mirror_begin(); + else vk_usnap_begin(RP.handles, RP.n, RP.fence); + uint64_t t1 = kern_now_ns(); + if (j == 0) SC.mirror_ns += t1 - t0; + else SC.usnap_ns += t1 - t0; + if (__atomic_load_n(&RP.prep_left, __ATOMIC_RELAXED) == 1) SC.ready_ns += t1 - SC.t0; + __atomic_sub_fetch(&RP.prep_left, 1, __ATOMIC_RELEASE); + } +} + +// Chunks [0, avail) less those behind a hold that is not done. +static int chunks_allowed(int avail) { + for (uint64_t m = SC.holds; m; m &= m - 1) { + int k = __builtin_ctzll(m); + if (k + 1 >= avail) break; + if (!__atomic_load_n(&SC.ch[k].done, __ATOMIC_ACQUIRE)) return k + 1; + } + return avail; +} + +// The chunks placed so far can be taken once prep_run is done, with the uniform drops the scan found up to them. +static void chunks_publish(int final) { + int all = final ? SC.nch : SC.nch - 1, avail = chunks_allowed(all); + final = final && avail == all; // the threads go to sleep on the final word: only with nothing held back + if ((avail == SC.pub && !final) || __atomic_load_n(&RP.prep_left, __ATOMIC_ACQUIRE)) return; + for (; SC.dr_done < SC.ndr; SC.dr_done++) vk_usnap_drop_at(SC.dr[SC.dr_done].addr, SC.dr[SC.dr_done].size, SC.dr[SC.dr_done].at); + SC.pub = avail; + take_publish(avail, final); +} + +// TOTK_TRACE=perf counts the chunks held per reason (SC.held). +static void scan_hold(int why) { + if (SC.nch <= 0 || SC.nch > 64 || SC.holds >> (SC.nch - 1) & 1) return; + SC.holds |= 1ull << (SC.nch - 1); + SC.held[why]++; +} + +static void scan_drop(uint64_t addr, uint64_t size) { + if (SC.ndr == SC.cdr) SC.dr = grow_n(SC.dr, &SC.cdr, sizeof *SC.dr); + SC.dr[SC.ndr++] = (ScanDrop){addr, size, SC.drop_n}; +} + +// A draw or dispatch whose shaders write storage buffers drops the uniform copies they cover from its own op on, as +// the sequential replay does in bind_range. The scan looks up no shader: it takes the ones the contexts translated +// (vk_exec_draw_writes). With one not translated yet the op still counts, the chunk that records it drops what the +// shader writes at its position (bind_range), and the chunks after it wait until it is recorded. Without this a chunk +// recorded before an earlier chunk's write bound the copy taken at the submit call, and on an RX 7800 XT the grass +// near the camera came out dark or unshaded in some runs. +static void scan_writes(VkExec *s, uint32_t op) { + uint64_t w[16][2]; + s->chunked = 1, s->upd = SC.upd, s->upd_n = SC.nupd; // ubo_ptr reads the scan's updates + int n = vk_exec_draw_writes(s, op == OP_DISPATCH || op == OP_DISPATCH_IND, w, 16); + s->chunked = 0; + if (!n) return; + SC.drop_n++; + SC.wpos[SC.nwpos++] = s->pos; + if (n < 0) scan_hold(HOLD_WRITES); + for (int i = 0; i < n; i++) scan_drop(w[i][0], w[i][1]); +} + +// The next chunk of submission gen nobody took: -1 none yet, -2 none left (or a later submission). +static int chunk_take(uint32_t gen) { + uint64_t w = __atomic_load_n(&RP.take, __ATOMIC_ACQUIRE); + for (;;) { + if ((uint32_t)(w >> 32) != gen) return -2; + if ((w & 0xffff) >= (w >> 16 & 0x7fff)) return w >> 31 & 1 ? -2 : -1; + if (__atomic_compare_exchange_n(&RP.take, &w, w + 1, 0, __ATOMIC_ACQ_REL, __ATOMIC_ACQUIRE)) return (int)(w & 0xffff); + } +} + +static void views_release(VkExec *ex) { + if (!ex->nviews) return; + ViewList *l = malloc(sizeof *l + sizeof(VkImageView) * (size_t)ex->nviews); + if (!l) rt_fatal("vk: out of memory"); + l->n = ex->nviews; + for (int i = 0; i < ex->nviews; i++) l->v[i] = ((ViewEnt *)ex->views)[i].view; + vk_completion_add(ex->fence, free_views, l); + ex->nviews = 0; +} + +static VkCommandBuffer rec_cb(Rec *r, uint64_t fence) { + VkCtx *c = vk_ctx(); + if (!r->pool) { + VkCommandPoolCreateInfo pi = {.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO, + .flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT, + .queueFamilyIndex = c->gfx_family}; + if (vkCreateCommandPool(c->device, &pi, NULL, &r->pool) != VK_SUCCESS) rt_fatal("vk: chunk command pool"); + } + uint64_t gpu = vk_sync_fence_done(); + int i = 0; + while (i < r->ncbs && r->cbs[i].fence > gpu) i++; + if (i == r->ncbs) { + r->cbs = realloc(r->cbs, sizeof *r->cbs * (size_t)(r->ncbs + 1)); + if (!r->cbs) rt_fatal("vk: out of memory (command buffers)"); + VkCommandBufferAllocateInfo ai = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO, .commandPool = r->pool, + .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, .commandBufferCount = 1}; + if (vkAllocateCommandBuffers(c->device, &ai, &r->cbs[i].cb) != VK_SUCCESS) rt_fatal("vk: command buffer allocation"); + r->ncbs++; + } + r->cbs[i].fence = fence; + VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vkBeginCommandBuffer(r->cbs[i].cb, &bi); + return r->cbs[i].cb; +} + +// Chunk k on the calling thread with context r: from its state copy up to the next chunk's first op, into a command +// buffer of r's pool, ending with every image in GENERAL. +static void chunk_record(Rec *r, int k) { + // TOTK_REPLAY_SKEW_US (debug): the even chunks wait this long before they record, so with several threads the + // chunk after each one records first. A recording that depends on the order the threads get to the chunks, rather + // than on replay order, then shows in every submission instead of a few launches in thirty. + if (SC.skew_us > 0 && nthreads && !(k & 1)) plat_sleep_ns((uint64_t)SC.skew_us * 1000); + uint64_t t0 = kern_now_ns(); + VkExec *ex = &r->ex; + Chunk *c = &SC.ch[k]; + c->start = t0 - SC.t0; + const Chunk *next = k + 1 < __atomic_load_n(&SC.nch, __ATOMIC_ACQUIRE) ? &SC.ch[k + 1] : NULL; + ex->cb = rec_cb(r, RP.fence); + ex->fence = RP.fence, ex->recorded = 0, ex->in_pass = 0, ex->nviews = 0, ex->barrier_at = -1, ex->nfbl = 0; + ex->part_start = 0; + x_cur = ex; + chunk_k = k; + vk_completion_defer(&c->defer); + vk_mem_count_local(r->traffic); + int ts = vk_sync_ts_open(ex->cb); + copy_state(ex, &c->st); + ex->rt_resolved = 0, ex->dirty = VK_EXEC_DIRTY_ALL, ex->tex_serial++, ex->fb_fs = NULL; + vk_exec_draw_new_cb(ex); + ex->chunked = 1, ex->upd = SC.upd, ex->cond = SC.cond, ex->jobs = SC.jobs, ex->njobs = c->job_end; + ex->upd_n = c->upd_n, ex->cond_i = c->cond_i, ex->job_i = c->job_i, ex->drop_n = c->drop_n, ex->pos = c->pos; + ex->wpos = SC.wpos, ex->wpos_i = c->wpos_i, ex->wpos_end = c->wpos_end; + for (int i = c->hi; i < RP.n; i++) + if (walk(ex, i == c->hi ? c->op : nvn_op_first(RP.handles[i]), 0, next && next->hi == i ? next->op : NULL)) break; + vk_exec_barrier(ex); + vk_exec_oq_flush(ex); + ex->chunked = 0, ex->njobs = ex->job_i = 0; + views_release(ex); + vk_sync_ts_close(ex->cb, ts, RP.fence); + vk_completion_defer(NULL); + vk_mem_count_local(NULL); + vk_mem_count_flush(r->traffic); + x_cur = NULL; + chunk_k = -1; + vkEndCommandBuffer(ex->cb); + c->cb = ex->cb, c->by = ex, c->ns = kern_now_ns() - t0; + plat_mutex_lock(&RP.m); + __atomic_store_n(&c->done, 1, __ATOMIC_RELEASE); + plat_cond_broadcast(&RP.done); + plat_mutex_unlock(&RP.m); +} + +static void rec_main(void *arg) { + Rec *r = arg; + plat_thread_priority(RP.prio); + uint32_t seen = 0; + for (;;) { + plat_mutex_lock(&RP.m); + while (RP.gen == seen) plat_cond_wait(&RP.go, &RP.m); + seen = RP.gen; + plat_mutex_unlock(&RP.m); + prep_run(); + for (;;) { + uint64_t w = __atomic_load_n(&RP.take, __ATOMIC_ACQUIRE); + int k = chunk_take(seen); + if (k >= 0) chunk_record(r, k); + if (k != -1) { + if (k == -2) break; + continue; + } + plat_mutex_lock(&RP.m); + while (__atomic_load_n(&RP.take, __ATOMIC_ACQUIRE) == w) plat_cond_wait(&RP.go, &RP.m); + plat_mutex_unlock(&RP.m); + } + } +} + +// TOTK_REPLAY_WORKERS threads record; with one, the submitting thread does. +static void recs_start(void) { + if (nrecs) return; + nthreads = SC.workers > 1 ? (SC.workers < MAX_RECS ? SC.workers : MAX_RECS) : 0; + nrecs = nthreads + 1; + RP.prio = plat_thread_priority_get(); + for (int i = 0; i < nrecs; i++) { + state_reset(&recs[i].ex); + if (i && !(recs[i].t = plat_thread_new(rec_main, &recs[i], "vk replay"))) rt_fatal("vk: no replay thread"); + } +} + +// Submits the chunks that are done, in order; wait: until all are. +static void chunks_submit(int wait) { + while (SC.sub < SC.nch) { + Chunk *c = &SC.ch[SC.sub]; + if (wait) chunks_publish(1); // a hold may have ended + if (!__atomic_load_n(&c->done, __ATOMIC_ACQUIRE)) { + if (!wait) return; + int k = nthreads ? -1 : chunk_take(RP.gen); + if (k >= 0) { + chunk_record(&recs[0], k); + continue; + } + plat_mutex_lock(&RP.m); + while (!__atomic_load_n(&c->done, __ATOMIC_ACQUIRE)) plat_cond_wait(&RP.done, &RP.m); + plat_mutex_unlock(&RP.m); + } + vk_completion_flush(&c->defer); + vk_sync_part(c->cb); + uint64_t t = kern_now_ns() - SC.t0; + if (!SC.sub) SC.first_ns += t, SC.start0_ns += c->start; + SC.last_ns += SC.sub == SC.nch - 1 ? t : 0; + SC.rec_ns += c->ns; + if (c->ns > SC.rec_max_ns) SC.rec_max_ns = c->ns; + SC.sub++; + n_parts++; + } +} + +// 1 when the replay ends an open pass at this op, so a chunk starting here splits no pass. +static int scan_ends_pass(const VkExec *s, uint32_t op, const void *a) { + switch (op) { + case OP_SET_RT: return memcmp(&s->rt, a, sizeof s->rt) != 0; + case OP_BARRIER: case OP_DISPATCH: case OP_DISPATCH_IND: case OP_UPDATE_UB: case OP_COPY_B2B: case OP_COPY_B2T: + case OP_COPY_T2B: case OP_COPY_T2T: case OP_CLEAR_TEX: case OP_CLEAR_BUF: return 1; + default: return 0; + } +} + +// Places chunk nch at h; the chunk before it can be taken from now on. +static void chunk_start(const VkExec *s, int hi, const op_hdr *h) { + Chunk *c = &SC.ch[SC.nch]; + c->hi = hi, c->op = h, c->upd_n = SC.nupd, c->cond_i = SC.ncond, c->job_i = SC.njobs, c->drop_n = SC.drop_n, c->pos = s->pos; + c->wpos_i = SC.nwpos; + c->done = 0, c->cb = VK_NULL_HANDLE, c->by = NULL; + copy_state(&c->st, s); + if (SC.nch) SC.ch[SC.nch - 1].job_end = SC.njobs, SC.ch[SC.nch - 1].wpos_end = SC.nwpos, SC.ch[SC.nch - 1].draws = SC.in_chunk; + SC.in_chunk = 0; + __atomic_store_n(&SC.nch, SC.nch + 1, __ATOMIC_RELEASE); + if (SC.nch > 1) chunks_publish(0); + if (SC.nch == 2) SC.per *= 4; // the first chunk was a quarter + chunks_submit(0); +} + +// The chunks before chunk k are recorded. +static void chunks_wait(int k) { + plat_mutex_lock(&RP.m); + for (int i = 0; i < k; i++) + while (!__atomic_load_n(&SC.ch[i].done, __ATOMIC_ACQUIRE)) plat_cond_wait(&RP.done, &RP.m); + plat_mutex_unlock(&RP.m); +} + +int vk_exec_chunk_behind(void) { + for (int i = chunk_k - 1; i >= 0; i--) + if (!__atomic_load_n(&SC.ch[i].done, __ATOMIC_ACQUIRE)) return 1; + return 0; +} + +// The chunks before take no lock this one holds and wait only for chunks before them, so they all get done: a chunk +// is taken only after every chunk before it was (chunk_take), and the DLSS records wait the same way (chunks_wait). +void vk_exec_chunk_settle(void) { + if (chunk_k <= 0) return; + uint64_t t0 = kern_now_ns(); + chunks_wait(chunk_k); + __atomic_add_fetch(&SC.settles, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&SC.settle_ns, kern_now_ns() - t0, __ATOMIC_RELAXED); +} + +static void scan_job(VkExecJob *j, const op_hdr *h) { + static int full; + j->chunk = SC.nch - 1; + if (SC.njobs < MAX_JOBS) SC.jobs[SC.njobs] = *j, SC.jobs[SC.njobs++].op = h; + else if (!full++) rt_log("vk_exec: more than %d planned records in a submission: the others are left out", MAX_JOBS); + if (j->kind == VK_JOB_DLSS) scan_hold(HOLD_DLSS); // the chunks after it read DLSS's output (vk_dlaa_swap_img) +} + +// What vk_contact_pass_begin (pass) and vk_contact_flush do in the sequential replay, in replay order. A pass of the +// replay may start at any draw or clear here: the plan only looks at the targets, so asking again changes nothing. +static void scan_contact(const VkExec *s, int pass, const op_hdr *h) { + VkExecJob j; + if (vk_contact_plan(s, pass, &j)) scan_job(&j, h); +} + +// What vk_exec_pass_begin asks of contact shadows and DLSS. DLSS's plan, too, gives the same answer for a pass asked +// again: it moves on only at the passes that arm, end or scale the scene. +static void scan_pass(const VkExec *s, const op_hdr *h) { + scan_contact(s, 1, h); + VkExecJob j; + if (vk_dlaa_plan(s, &j)) scan_job(&j, h); +} + +// A draw as prepare_draw sees it: its pass begins, and with it a contact shadow march or a DLSS record may be due. +static void scan_draw(VkExec *s, uint32_t op, const void *a, const op_hdr *h) { + if (s->render_disabled) return; + if (op == OP_DRAW && ((const p_draw *)a)->count <= 0) return; + if (op == OP_DRAW_IDX && (((const p_draw_idx *)a)->itype < 0 || ((const p_draw_idx *)a)->itype > 2 || + ((const p_draw_idx *)a)->count <= 0)) + return; + if (op == OP_DRAW_IDX_IND && ((const p_draw_idx_ind *)a)->itype != NVN_INDEX_TYPE_UNSIGNED_SHORT && + ((const p_draw_idx_ind *)a)->itype != NVN_INDEX_TYPE_UNSIGNED_INT) + return; + if (s->pass_noatt) s->rt_resolved = 0; // sized by this draw's viewport + resolve_rt_as(s, 1); + scan_pass(s, h); + vk_dlaa_scan_draw(s); +} + +static void scan(VkExec *s, int hi, const op_hdr *h, int depth) { + for (;;) { + check_op(h); + const void *a = nvn_op_payload(h); + if (h->op == OP_END) return; + if (!depth && SC.in_chunk >= SC.per && SC.nch < SC.chunks && + (!SC.open || scan_ends_pass(s, h->op, a) || 2 * SC.in_chunk >= 3 * SC.per)) + chunk_start(s, hi, h); + if (drop_op(h->op)) SC.drop_n++; + s->pos++; + if ((draw_op(h->op) && !s->render_disabled) || h->op == OP_DISPATCH || h->op == OP_DISPATCH_IND) scan_writes(s, h->op); + switch (h->op) { // the ops vk_exec_op_xfer and op_dispatch flush the contact shadows at + case OP_DISPATCH: case OP_DISPATCH_IND: case OP_COPY_B2B: case OP_COPY_B2T: case OP_COPY_T2B: case OP_COPY_T2T: + case OP_CLEAR_TEX: case OP_REPORT_COUNTER: case OP_RESET_COUNTER: case OP_RENDER_COND: case OP_CLEAR_BUF: + scan_contact(s, 0, h); + break; + default: break; + } + switch (h->op) { + case OP_BEGIN: break; + case OP_JUMP: h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; continue; + case OP_CALL: + if (depth < CALL_DEPTH) scan(s, hi, nvn_op_first(((const p_jump *)a)->target), depth + 1); // else walk stops + break; + case OP_SET_RT: + if (memcmp(&s->rt, a, sizeof s->rt)) { + s->rt = *(const p_set_rt *)a, s->rt_resolved = 0, SC.open = 0; + resolve_rt_as(s, 1); // here, not at the first draw: a clear or a later chunk may need them live + chunks_publish(0); + } + break; + case OP_CLEAR_COLOR: case OP_CLEAR_DS: + SC.open |= !s->render_disabled; + if (!s->render_disabled) resolve_rt_as(s, 1), scan_pass(s, h); + break; + case OP_DRAW: case OP_DRAW_IDX: case OP_DRAW_IDX_IND: + SC.in_chunk++, SC.open = 1; + scan_draw(s, h->op, a, h); + break; + case OP_DISPATCH: case OP_DISPATCH_IND: SC.in_chunk++, SC.open = 0; break; + case OP_UPDATE_UB: { // listed where op_update_ub notes it + const p_update_ub *u = a; + VkMemSpan sp; + if (u->size && vk_mem_resolve(u->dst, u->size, &sp)) { + SC.upd[SC.nupd++] = (VkExecUpd){u->dst, u->dst + u->size, u->data}; + scan_drop(u->dst, u->size); + } + SC.open = 0; + break; + } + case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: + case OP_BIND_UBO: case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: + vk_exec_bind(s, h->op, a); + break; + case OP_COPY_B2B: case OP_COPY_T2B: case OP_CLEAR_BUF: { + uint64_t at, size = vk_exec_xfer_written(s, h->op, a, &at); + if (size) scan_drop(at, size); + if (vk_exec_xfer_revive(h->op, a, s->pos)) scan_hold(HOLD_TEX); + SC.open = 0; + break; + } + case OP_BARRIER: case OP_COPY_B2T: case OP_COPY_T2T: case OP_CLEAR_TEX: + if (vk_exec_xfer_revive(h->op, a, s->pos)) scan_hold(HOLD_TEX); + SC.open = 0; + break; + case OP_SIGNAL_EVENT: break; + case OP_RENDER_COND: SC.cond[SC.ncond++] = (uint8_t)vk_exec_xfer_scan(s, h->op, a); break; + case OP_REPORT_COUNTER: case OP_RESET_COUNTER: case OP_RENDER_ENABLE: vk_exec_xfer_scan(s, h->op, a); break; + default: state_op(s, h->op, a); // walk stops on an op without a handler + } + h = nvn_op_next(h); + } +} + +// The queue state of a and b without what resolve_rt derives from rt and the images. +static int state_differs(const VkExec *a, const VkExec *b) { + static VkExec x, y; + copy_state(&x, a), copy_state(&y, b); + VkExec *e[2] = {&x, &y}; + for (int i = 0; i < 2; i++) { + e[i]->rt_resolved = 0; + memset(e[i]->color, 0, sizeof e[i]->color), memset(&e[i]->depth, 0, sizeof e[i]->depth); + e[i]->pass_scale = e[i]->pass_w = e[i]->pass_h = e[i]->pass_layers = 0, e[i]->pass_noatt = 0; + memset(e[i]->pass_color, 0, sizeof e[i]->pass_color); + e[i]->pass_depth = e[i]->pass_stencil = VK_FORMAT_UNDEFINED, e[i]->pass_ncolor = e[i]->pass_samples = 0; + memset(e[i]->pipe.color_fmt, 0, sizeof e[i]->pipe.color_fmt); + e[i]->pipe.depth_fmt = 0, e[i]->pipe.samples = 0; + } + const uint8_t *p = (const uint8_t *)&x, *q = (const uint8_t *)&y; + for (size_t o = VK_EXEC_STATE_BEGIN; o < VK_EXEC_STATE_END; o++) + if (p[o] != q[o]) return (int)o; + return 0; +} + +static void record_chunks(VkExec *ex, const uint64_t *handles, int n, const OpCounts *cnt) { + uint64_t t0 = SC.t0 = kern_now_ns(); + SC.prep_ns += t0 - SC.rec_t0; + recs_start(); + if (!SC.ch && !(SC.ch = calloc((size_t)SC.chunks + 1, sizeof *SC.ch))) rt_fatal("vk: out of memory"); + while (SC.cupd < cnt->upd) SC.upd = grow_n(SC.upd, &SC.cupd, sizeof *SC.upd); + while (SC.ccond < cnt->cond) SC.cond = grow_n(SC.cond, &SC.ccond, sizeof *SC.cond); + while (SC.cwpos < cnt->draws) SC.wpos = grow_n(SC.wpos, &SC.cwpos, sizeof *SC.wpos); + vk_exec_oq_flush(ex); // queries a sequential submission closed count before the chunks' + VkExec *s = &SC.s; + copy_state(s, ex); + s->fence = ex->fence, s->rt_resolved = 0, s->pos = 0; + SC.holds = 0; + SC.nch = SC.sub = SC.pub = 0, SC.nupd = SC.ncond = SC.njobs = SC.drop_n = SC.ndr = SC.dr_done = 0, SC.open = 0; + SC.nwpos = 0; + job_turn = 0; + SC.per = (SC.per + 3) / 4; + plat_mutex_lock(&RP.m); + RP.handles = handles, RP.n = n, RP.fence = ex->fence, RP.prep_next = 0, RP.prep_left = 2; + if (!++RP.gen) RP.gen = 1; + __atomic_store_n(&RP.take, (uint64_t)RP.gen << 32, __ATOMIC_RELEASE); + plat_cond_broadcast(&RP.go); + plat_mutex_unlock(&RP.m); + if (!nthreads) prep_run(); + chunk_start(s, 0, nvn_op_first(handles[0])); + for (int i = 0; i < n; i++) scan(s, i, nvn_op_first(handles[i]), 0); + SC.ch[SC.nch - 1].job_end = SC.njobs, SC.ch[SC.nch - 1].wpos_end = SC.nwpos, SC.ch[SC.nch - 1].draws = SC.in_chunk; + SC.scan_ns += kern_now_ns() - t0; + prep_run(); // threads that wake late leave it to the scan's end + while (__atomic_load_n(&RP.prep_left, __ATOMIC_ACQUIRE)) plat_yield(); + chunks_publish(1); + SC.chunked++, SC.nchunks += (uint64_t)SC.nch; + chunks_submit(1); + x_cur = ex; + for (int i = 0; i < nrecs; i++) frame_add(&recs[i].ex); + int o = SC.check ? state_differs(SC.ch[SC.nch - 1].by, s) : 0; + if (o && SC.bad++ < 16) + rt_log("vk_exec: chunked replay: the state after the last chunk differs from the scan's at byte %d of VkExec", o); + copy_state(ex, s); + ex->rt_resolved = 0; +} + +int vk_exec_replay_workers(void) { return SC.workers; } + +void vk_exec_contexts(void (*fn)(VkExec *ex)) { + fn(&X); + for (int i = 0; i < nrecs; i++) fn(&recs[i].ex); +} + +static int chunked_on(void) { + if (SC.workers < 0) { + SC.workers = cfg_int("TOTK_REPLAY_WORKERS"); + if (SC.workers < 0) SC.workers = 0; + SC.chunks = cfg_int("TOTK_REPLAY_CHUNKS"); + if (SC.chunks <= 0) SC.chunks = 2 * SC.workers; + SC.min_draws = cfg_int("TOTK_REPLAY_CHUNK_DRAWS"); + SC.check = cfg_bool("TOTK_REPLAY_STATE_CHECK"); + SC.skew_us = cfg_int("TOTK_REPLAY_SKEW_US"); + if (SC.workers && SC.skew_us > 0) + rt_log("vk: TOTK_REPLAY_SKEW_US: the even chunks wait %d us before they record", SC.skew_us); + if (SC.workers) + rt_log("vk: replay in up to %d chunks per submission of at least %d draws each, %d recording thread%s%s", + SC.chunks, SC.min_draws, SC.workers, SC.workers > 1 ? "s" : "", SC.check ? ", state checked" : ""); + } + return SC.workers > 0 && SC.chunks > 1 && !vk_dump_armed; +} + int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t fence) { VkExec *ex = &X; + SC.rec_t0 = kern_now_ns(); if (!x_ready) state_reset(ex), vk_dump_init(), x_ready = 1; const tex_t *keep[9]; // the render targets carry over to this submission (set_rt compares them by pointer) memcpy(keep, ex->rt.color, sizeof ex->rt.color); @@ -950,33 +1657,60 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t ex->nviews = 0; ex->barrier_at = -1; ex->nfbl = 0; // the previous command buffer ended with a full barrier: every image is in GENERAL - vk_exec_frame.submits++; - split_draws = part_start = 0; - part_t0 = kern_now_ns(); - x_here = 1; - if (n) vk_mem_mirror_begin(), vk_usnap_begin(handles, n, fence); - for (int i = 0; i < n; i++) { + ex->fr.submits++; + ex->split_draws = ex->part_start = 0; + ex->part_t0 = kern_now_ns(); + x_cur = ex; + for (int i = 0; i < n; i++) if (!handles[i]) rt_fatal("vk: command handle %d of a submission is null", i); - walk(ex, nvn_op_first(handles[i]), 0); + OpCounts cnt = {0}; + if (n && chunked_on()) + for (int i = 0; i < n; i++) count_ops(nvn_op_first(handles[i]), 0, &cnt); + SC.subs += n != 0; + if (cnt.draws && cnt.draws >= 2 * (uint32_t)SC.min_draws) { + SC.per = (int)((cnt.draws * 4 + 4u * SC.chunks - 4) / (4u * SC.chunks - 3)); // the first chunk a quarter + if (SC.per < SC.min_draws) SC.per = SC.min_draws; + record_chunks(ex, handles, n, &cnt); + } else { + if (n) vk_mem_mirror_begin(), vk_usnap_begin(handles, n, fence); + for (int i = 0; i < n; i++) walk(ex, nvn_op_first(handles[i]), 0, NULL); } vk_exec_pass_end(ex); - x_here = 0; + x_cur = NULL; if (ex->recorded) vk_exec_barrier(ex); // later submissions see this one's writes if (n) ex->recorded += vk_mem_mirror_end(ex->cb, fence), vk_usnap_end(fence); // VRAM mirror bytes the GPU wrote go back to guest memory - if (ex->nviews) { - ViewList *l = malloc(sizeof *l + sizeof(VkImageView) * (size_t)ex->nviews); - if (!l) rt_fatal("vk: out of memory"); - l->n = ex->nviews; - for (int i = 0; i < ex->nviews; i++) l->v[i] = ((ViewEnt *)ex->views)[i].view; - vk_completion_add(fence, free_views, l); - ex->nviews = 0; - } + views_release(ex); ex->cb = VK_NULL_HANDLE; + frame_add(ex); if (n) n_subs++; if (n && (rt_trace & RT_TRACE_PERF)) vk_exec_ub_perf(); if ((rt_trace & RT_TRACE_PERF) && n_subs >= 240) { rt_log("perf: replay command buffer parts per submission %.2f", (double)(n_subs + n_parts) / (double)n_subs); + if (SC.chunked) { + double k = 1e6 * (double)SC.chunked; + rt_log("perf: chunked replay: %llu of %llu submissions, %.2f chunks each; per submission %.3f ms before the scan, " + "scan %.3f ms, first chunk begun at %.3f ms and submitted at %.3f ms, last at %.3f ms, recording %.3f ms " + "in all, longest chunk %.3f ms; beside the scan: mirrors %.3f ms, uniform snapshot %.3f ms, both done at %.3f ms", + (unsigned long long)SC.chunked, (unsigned long long)SC.subs, (double)SC.nchunks / SC.chunked, + SC.prep_ns / k, SC.scan_ns / k, SC.start0_ns / k, SC.first_ns / k, SC.last_ns / k, SC.rec_ns / k, + SC.rec_max_ns / 1e6, SC.mirror_ns / k, SC.usnap_ns / k, SC.ready_ns / k); + rt_log("perf: chunked replay: chunks held per submission: %.2f for a new texture, %.2f for DLSS, %.2f for " + "unknown storage writes", SC.held[HOLD_TEX] / (double)SC.chunked, SC.held[HOLD_DLSS] / (double)SC.chunked, + SC.held[HOLD_WRITES] / (double)SC.chunked); + rt_log("perf: chunked replay: %llu chunks waited for the chunks before them to make a texture (%.3f ms in all)", + (unsigned long long)__atomic_exchange_n(&SC.settles, 0, __ATOMIC_RELAXED), + __atomic_exchange_n(&SC.settle_ns, 0, __ATOMIC_RELAXED) / 1e6); + memset(SC.held, 0, sizeof SC.held); + char b[768]; + int o = 0; + for (int i = 0; i < SC.nch && o < 700; i++) + o += snprintf(b + o, sizeof b - o, " %d:%.2f@%.2f%s", SC.ch[i].draws, SC.ch[i].ns / 1e6, SC.ch[i].start / 1e6, + SC.holds >> i & 1 ? "h" : ""); + rt_log("perf: chunked replay: the last submission's chunks (draws:ms@begun, h held)%s", b); + } n_subs = n_parts = 0; + SC.subs = SC.chunked = SC.nchunks = SC.scan_ns = SC.first_ns = SC.last_ns = SC.rec_ns = SC.rec_max_ns = 0; + SC.prep_ns = SC.start0_ns = SC.mirror_ns = SC.usnap_ns = SC.ready_ns = 0; } - return ex->recorded - part_start; // what the last part holds + return ex->recorded - ex->part_start; // what the last part holds } diff --git a/runtime/vk_exec.h b/runtime/vk_exec.h index e3692d48..37579804 100644 --- a/runtime/vk_exec.h +++ b/runtime/vk_exec.h @@ -2,10 +2,12 @@ // handles of a submission, keeps render targets, passes, clears and fixed-function state; draws, binds and dispatches // (vk_exec_draw.c) and copies, counters, events and conditional rendering (vk_exec_xfer.c) plug in through the two // hooks at the bottom. -// Runs on the submitting thread under vk_sync_submit's lock: no locking inside. +// Runs on the submitting thread under vk_sync_submit's lock; a chunked replay (TOTK_REPLAY_WORKERS, vk_exec.c) records +// on several threads, each with a VkExec of its own, and shares only the locked tables named where they are declared. // The op loop follows MacNativeGames nvn_mtl.m (MIT, Copyright (c) 2026 jhaizhou) and the pass handling LAS Recomp // (MIT, Copyright (c) 2026 Peiikai); vk_exec.c names the sources. #pragma once +#include #include #include "nvn_api.h" @@ -38,8 +40,36 @@ typedef struct { uint8_t scale; // VkTexImage.scale } VkExecTarget; -// The replay state. One instance lives for the run: NVN state is queue state and carries over from one submission to -// the next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record. +// TOTK_TRACE=gfx: replay counters of the current frame, logged and reset at each present (vk_exec_frame_log). +typedef struct { + uint64_t passes, draws_seen, draws, clears, copies, dispatches, submits; + uint64_t fb_draws, fb_barriers, fb_restarts, fb_layouts; // feedback loops: draws sampling a written target of + // their pass, in-pass barriers, pass restarts, layout + // transitions +} VkExecFrame; + +// A record the scan of a chunked replay planned before an op: a contact shadow march (vk_contact.c: vk_contact_plan +// decides it in replay order, vk_contact_exec records it), or DLSS's copy of the lit scene or its evaluation +// (vk_dlaa.c: vk_dlaa_plan, vk_dlaa_exec). The chunk that holds the op records it. +enum { VK_JOB_CONTACT, VK_JOB_DLSS_MASK, VK_JOB_DLSS }; +typedef struct VkExecJob { + const void *op; // vk_exec.c: the op it precedes in a scanned submission + int kind; // VK_JOB_* + int chunk; // vk_exec.c: the chunk that holds the op + VkExecTarget scene, depth; // contact: the scene colour and the linear depth; DLSS: the light target to copy + // (mask) or the scaling pass's target (evaluation; no tex: in place) + float pc[16]; // the march's push constants +} VkExecJob; + +// An OP_UPDATE_UB of the submission: [lo, hi) of guest memory takes data (vk_exec_draw.c ubo_ptr reads them). +typedef struct { uint64_t lo, hi; const uint8_t *data; } VkExecUpd; + +struct VkExecDraw; // vk_exec_draw.c: the memos and command buffer state of one recording context + +// The replay state and one recording context. NVN state is queue state and carries over from one submission to the +// next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record. +// The fields from rt up to views are that queue state (VK_EXEC_STATE_BEGIN/END); the rest belongs to the context that +// records with it. typedef struct VkExec { // ---- per command buffer VkCommandBuffer cb; // being recorded @@ -94,8 +124,14 @@ typedef struct VkExec { int render_disabled; // OP_RENDER_ENABLE false or a failed OP_RENDER_COND: clears and draws skip p_render_cond render_cond; // last OP_RENDER_COND int has_render_cond; + // samples-passed counting (vk_exec_xfer.c): a reset since the last report, a report since the last reset, the game + // reported twice without a reset (every draw counts from then on) + int oq_counting, oq_reported, oq_keep; // ---- internal to vk_exec.c + VkExecFrame fr; // counters of this context, added to vk_exec_frame at the end of a submission + int split_draws, part_start; // draws and dispatches since the last part; recorded at its start + uint64_t part_t0; // replay time at its start void *views; // image views created for this command buffer (freed at its fence) int nviews, cap_views; int barrier_at; // recorded right after the last full barrier (-1: none yet) @@ -112,8 +148,30 @@ typedef struct VkExec { const void *fb_fs; // feedback_scan's memo: fragment shader, texture bind serial, pass serial uint32_t fb_serial, fb_pass, fb_sampled; uint32_t tex_serial, pass_serial; // bumped by every texture or image bind and pass begin + + // ---- vk_exec_xfer.c: the occlusion query of the open pass and the queries closed since the last report or reset + int oq_open; // slot of the active query, -1 none + uint32_t *oq_seg, oq_nseg, oq_capseg; + int oq_lost; // the segment missed a draw (ring full) + + struct VkExecDraw *draw; // vk_exec_draw.c, made on the context's first draw op + + // ---- a chunk of a scanned submission (vk_exec.c): the scan's answers, taken in op order + int chunked; // records a chunk: what depends on earlier chunks comes from the scan + const VkExecUpd *upd; // the submission's inline uniform updates + const uint8_t *cond; // its RENDER_COND results (1 renders) + const VkExecJob *jobs; // its planned records (contact shadows, DLSS) + uint32_t njobs; + uint32_t upd_n, cond_i, job_i; // updates, results and records before the op being recorded + uint32_t drop_n; // ops before it that may write uniform ranges (vk_usnap_drop_at) + const uint32_t *wpos; // the positions of the draws and dispatches among those ops (vk_exec_draw_writes) + uint32_t wpos_i, wpos_end; // the next of them, and the end of the chunk's + uint32_t pos; // ops of the submission up to and with it (the scan's too) } VkExec; +#define VK_EXEC_STATE_BEGIN offsetof(VkExec, rt) +#define VK_EXEC_STATE_END offsetof(VkExec, fr) + // Begins dynamic rendering with the current targets unless a pass is open (a full memory barrier precedes it). Call // before a draw or an attachment clear. void vk_exec_pass_begin(VkExec *ex); @@ -136,9 +194,16 @@ VkCommandBuffer vk_exec_inline_begin(int *reopen); void vk_exec_inline_end(int reopen); // vk_exec_draw.c: forget what the command buffer being recorded has bound (pipeline, sets, dynamic state, vertex // buffers): the replay continues in a new one (vk_sync_split). -void vk_exec_draw_new_cb(void); +void vk_exec_draw_new_cb(VkExec *ex); +// The recording threads of the chunked replay as this run read TOTK_REPLAY_WORKERS (0 off), -1 before the first +// submission read it (the settings menu shows it). +int vk_exec_replay_workers(void); +// vk_exec_draw.c, for the state scan of a chunked replay: the storage ranges the shaders of a draw (dispatch 0) or a +// dispatch at state s write, as {address, size} into out; their count, or -1 when there are more than max or a stage's +// shader is not known yet (no context has translated it). Reads uniform blocks through s->upd like a chunk. +int vk_exec_draw_writes(const VkExec *s, int dispatch, uint64_t (*out)[2], int max); // vk_exec_draw.c: the vertex and fragment shader of the last draw recorded into this command buffer (NULL: none yet). -void vk_exec_draw_shaders(const GfxShader **vs, const GfxShader **fs); +void vk_exec_draw_shaders(const VkExec *ex, const GfxShader **vs, const GfxShader **fs); VkBuffer vk_exec_inline_buffer(VkDeviceSize size); // vk_exec.c: the stand-in a 3D BC render target is drawn through goes with its image (vk_exec_image_gone). void vk_exec_bc_gone(VkImage image); @@ -155,16 +220,15 @@ int vk_exec_targets_recent(VkExec *ex, int any); // The current pass's targets with their formats, extents and the frames they were last rendered in (diagnostics). void vk_exec_targets_describe(VkExec *ex, char *buf, size_t cap); -// TOTK_TRACE=gfx: replay counters of the current frame, logged and reset at each present (vk_exec_frame_log). -typedef struct { - uint64_t passes, draws_seen, draws, clears, copies, dispatches, submits; - uint64_t fb_draws, fb_barriers, fb_restarts, fb_layouts; // feedback loops: draws sampling a written target of - // their pass, in-pass barriers, pass restarts, layout - // transitions -} VkExecFrame; extern VkExecFrame vk_exec_frame; void vk_exec_frame_written(const tex_t *t); // vk_texture_gpu_written: t rendered or copied into uint64_t vk_exec_recording_fence(void); // fence of the submission this thread records, else 0 +uint32_t vk_exec_replay_pos(void); // the op this thread records in a chunk, else UINT32_MAX +// A chunk that is about to make a texture entry whose content depends on the textures at its storage (vk_texture.c +// entry_of): whether a chunk before it is still being recorded, and the wait until none is. The caller holds no lock +// but the texture lock, which it releases for the wait. +int vk_exec_chunk_behind(void); +void vk_exec_chunk_settle(void); void vk_exec_frame_log(const tex_t *presented, uint64_t frame); // TOTK_DUMP= (vk_dump.c): between presents frame-1 and frame, every pass's targets and every dispatch's storage @@ -188,6 +252,7 @@ void vk_c0_fill(uint8_t bank[VK_C0_BYTES], int stage, const p_bind_buf ssbo[16], // vk_exec_draw.c: OP_DRAW, OP_DRAW_IDX, OP_DISPATCH, OP_BIND_* (program, state, vattr, vstream, vbuf, ubo, ssbo, tex, // samp, img), OP_UPDATE_UB, OP_DRAW_IDX_IND, OP_DISPATCH_IND. payload: nvn_op_payload of the record. void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload); +int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload); // a bind op's writes into ex; 1: VkExecDraw.bind_gen // The aspect a sampled view of img reads (vk_exec_draw.c get_view): stencil for a view in STENCIL depth-stencil mode. VkImageAspectFlags vk_exec_view_aspect(const VkTexImage *img, const view_t *v, int hv); VkImageView vk_exec_target_view(const VkTexImage *img, VkFormat format, VkImageViewType type, uint32_t aspect, @@ -199,10 +264,26 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload); // sample DLSS's output vk_dlaa_swap_view in its place, so the game's scaling draw scales DLSS's picture. extern const VkTexImage *vk_dlaa_swap_img; extern VkImageView vk_dlaa_swap_view; -// vk_contact.c: a pass about to begin (its targets resolved, no pass open), and a dispatch or copy about to be recorded +// The state scan's part of the transfer ops: render enable, conditional rendering (returns the condition: 1 renders), +// samples-passed counting state and the pending reports conditions read. Returns 1 for every other op. +// The state scan of a chunked replay (vk_exec.c): the textures op uses go live in replay order (vk_texture_revive); pos: +// the op's place in the submission. 1 when one of them has no entry yet. +int vk_exec_xfer_revive(uint32_t op, const void *payload, uint32_t pos); +int vk_exec_xfer_scan(VkExec *ex, uint32_t op, const void *payload); +// The guest memory a COPY_B2B, COPY_T2B or CLEAR_BUF op makes the GPU write, as far as the scan can tell without the +// texture's image: [*addr, *addr + return value). +uint64_t vk_exec_xfer_written(const VkExec *ex, uint32_t op, const void *payload, uint64_t *addr); +// vk_contact.c: a pass about to begin (its targets resolved, no pass open), and a dispatch or copy about to be recorded. +// vk_contact_plan is what they do in replay order (pass: the first; else the second), for the scan; 1: a march is due (j). void vk_contact_pass_begin(VkExec *ex); void vk_contact_flush(VkExec *ex); +int vk_contact_plan(const VkExec *ex, int pass, VkExecJob *j); +void vk_contact_exec(VkExec *ex, const VkExecJob *j); // vk_exec_xfer.c, samples-passed counting: a draw about to be recorded in the open pass (begins an occlusion query // when counting and none is active), and the open pass about to end (ends the active query). void vk_exec_oq_draw(VkExec *ex); void vk_exec_oq_pass_end(VkExec *ex); +void vk_exec_oq_forget(VkExec *ex); +void vk_exec_oq_flush(VkExec *ex); // the end of a chunk: its closed queries counted as a segment without a report // vk_exec_xfer_shutdown: the context's queries belonged to the old pool +// vk_exec.c: fn on every recording context (teardown, the device idle). +void vk_exec_contexts(void (*fn)(VkExec *ex)); diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index 7bede297..f7ed6d10 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -8,7 +8,8 @@ // Per command buffer the replay keeps transient memory (support buffers, copied uniform ranges, generated indices), // descriptor pools, image views and the storage-buffer ranges the GPU may write; all of it is freed, and the ranges are // reported with vk_gpu_written, from one completion callback once the submission's fence is done (the write watch -// does not see GPU writes). Nothing here locks: it runs on the submitting thread under vk_sync_submit's lock. +// does not see GPU writes). The memos and per-command-buffer state belong to one recording context (VkExecDraw); the +// tables the contexts of a chunked replay share (stand-ins, depth twins, views) take their own locks. // // Provenance: the op semantics (bind ops, program and slot conventions, draw and dispatch flow, vertex streams with a // zeroed buffer for unbound ones) follow MacNativeGames games/totk/third_party/d2r_gfx/gfx/nvn/nvn_mtl.m:7409-7460, @@ -81,9 +82,6 @@ typedef struct { int nwritten, cwritten; } CbRes; -static CbRes *cur; -static uint64_t cur_fence; - // Descriptor pools and full-size transient chunks go back to these lists when their command buffer completes, instead // of being destroyed and created again for the next one: each creation is a driver memory allocation (on Windows an // allocation, a map and a paging-fence wait on the CPU), about ten of them per frame inside the replay. @@ -95,7 +93,7 @@ static Chunk spare_chunk[SPARE_CHUNKS]; static int nspare_pool, nspare_chunk; // State of the command buffer being recorded that the Vulkan command buffer itself holds. -static struct { +typedef struct { VkPipeline pipe[2]; // bound per bind point: 0 graphics, 1 compute VkPipeState st; // the state vk_pipeline_set_dynamic last recorded int st_valid; @@ -126,20 +124,19 @@ static struct { VkDescriptorSetLayout ds_last_layout[2]; int ds_pushed[2]; // set number holds the push of DS.psig under ds_layout int res_valid; // the graphics sets bound hold what the inputs in RS give (bind_resources) -} B; +} CbState; -// Contents of B.ds_last: the set's packed descriptors (VkDescSlot), which with its layout determine every write. A draw +// Contents of CbState.ds_last: the set's packed descriptors (VkDescSlot), which with its layout determine every write. A draw // whose allocated set would hold the same reuses the set instead of allocating and writing one (a per-draw // cost). psig: the same for the last push per set number; a push equal to it under the same layout is not recorded // again (the descriptors stay bound across pipeline binds of that layout). #define DS_SIG_BYTES (MAX_DESC * sizeof(VkDescSlot)) -static struct { uint32_t n[2], pn[2]; uint8_t sig[2][DS_SIG_BYTES], psig[2][DS_SIG_BYTES]; } DS; +typedef struct { uint32_t n[2], pn[2]; uint8_t sig[2][DS_SIG_BYTES], psig[2][DS_SIG_BYTES]; } DescSigs; // TOTK_BIND_STATS counters (bind_stats). -static struct { - int on; +typedef struct { uint64_t draws, bind[2], desc[2], push, push_skip, set_new, set_reuse, ac_miss, same_prog, same_all, bind_skip; -} BS = {-1}; +} BindStats; // TOTK_DYN_DELTA (prepare_draw): the mode and, for the perf line, the set_dynamic calls, the commands they recorded // (a full call counts 1) and the calls that recorded none. @@ -150,8 +147,7 @@ static struct { int mode; uint64_t calls, cmds, empty; } DD = {-1}; // c0 storage table, otherwise one bound range or handle per descriptor. #define RS_BYTES (MAX_BINDINGS * sizeof(((VkExec *)0)->ssbo[0]) + MAX_DESC * sizeof(p_bind_buf)) typedef struct { int cls, use; uint64_t bytes; } RsCount; -static struct { - int on; // TOTK_BIND_SKIP: 1 skip (default), 2 check, 0 off +typedef struct { uint32_t hn; // words of h the last signature filled; 0: none uint64_t h[24]; // generations, feedback images, support inputs, base instance const VkPipeInfo *info; // the pipeline whose RsList ranges body holds, as of bind op generation gen @@ -164,7 +160,7 @@ static struct { int cmds; // check mode: pushes, binds and set writes the build recorded uint32_t ckn; // check mode: the last build's packed descriptors (bytes in ck) uint8_t ck[DS_SIG_BYTES]; -} RS = {-1}; +} RsState; // UpdateUniformBuffer overlays of the submission being replayed (op_update_ub, TOTK_UB_OVERLAY). Here because // bind_sig reads them (ov_at). @@ -205,12 +201,111 @@ static uint64_t ov_at(const VkExec *ex, uint64_t addr, uint64_t n) { typedef struct { uint32_t off, len; } RsRange; typedef struct { int st; const GfxShaderRes *res; } RsPtr; typedef struct { uint32_t n, bytes; uint8_t sup, c0, ok, np; RsPtr p[RS_PTRS]; RsRange r[]; } RsList; -static nvn_hmap rs_lists; // VkPipeInfo pointer -> RsList, until shutdown -static const VkPipeInfo *rs_last; // the last lookup -static const RsList *rs_last_l; +// (program, stage, answers) -> shader, 4-way set-associative, most recently used first. A miss costs a SHA-256 of the +// whole shader code (gfx_shader.c req_init), slow on CPUs without SHA instructions: a field frame looks up ~3000 and +// the 2048-slot direct-mapped table this replaced missed 3% of them on conflicts. +#define SHADER_SETS 2048 +#define SHADER_WAYS 4 +typedef struct { uint64_t serial; int stage; int used; GfxShaderAnswers a; const GfxShader *s; } ShaderEnt; -static uint32_t c0_dim[3], c0_binst; // the dispatch size and base instance of the draw or dispatch being bound -static uint64_t bind_gen = 1; // bumped by every bind op that writes ex->prog, ubo, ssbo, tex or img +// Which shader of a program serves each NVN stage, and its code length, per program serial (new at every +// nvnProgramSetShaders): the scan reads two words of every stage's control block, cold guest cache lines, and runs +// several times a draw. Direct-mapped; a collision only scans again. +#define STAGEM_N 512 +typedef struct { uint64_t serial; int8_t idx[6]; uint32_t len[6]; } StageMemo; + +// Texture and image types of a stage's first 32 slots, each kept with the handle and the generation and slot token it +// was looked up under: a slot is looked up again, a hash probe, only when the handle changed or its slot may resolve +// otherwise (slot_still). +typedef struct { uint64_t tex[32], img[32], ttok[32], itok[32]; uint32_t tgen[32], igen[32]; uint8_t tt[32], it[32]; } AcSlots; + +// The slots of (program, stage) whose types its translation reads (gfx_shader_slot_mask), learned from its first +// shader; ~0u while unknown. The types of other slots select the same shader, so answers_for leaves them zero instead +// of looking up all 64 handles. Direct-mapped by serial and stage; a collision only relearns. +#define SLOTM_N 4096 +typedef struct { uint64_t serial; int st; uint32_t m; } SlotMask; + +// The inline uniform updates of the submission being replayed (upd_note). +#define UPD_N 4096 +typedef struct { + uint64_t fence; // the submission the entries belong to + uint32_t n; // updates noted in it + VkExecUpd u[UPD_N]; +} UpdRing; + +// The image and the levels [l0, l1) and layers [a0, a1) a texture's view covers. +typedef struct { VkImage image; uint32_t l0, l1, a0, a1; } SubRange; + +// The subresource a texture handle samples (feedback_scan). +#define FM_N 8192 +typedef struct { uint64_t h, tok; uint32_t gen; SubRange r; } FbMemo; + +// Sampled-texture descriptors per (handle, resource type, separate) (tex_info_lookup). +#define TD_N 8192 +typedef struct { + uint64_t h, tok; + uint32_t gen, epoch, sgen; + uint16_t type; + uint8_t separate, valid; + VkImage image; + VkDescriptorImageInfo out; +} TexMemo; + +// The previous draw's shaders and what they were looked up from (prepare_draw). +typedef struct { + const prog_t *prog; + uint64_t serial; + uint32_t gen, tex_serial; + uint64_t tex[2][32], img[2][32]; + VkPipeState pipe; +} MemoKey; +typedef struct { int valid, has_fs; MemoKey k; const GfxShader *vs, *fs; } ShaderMemo; + +// The previous draw's pipeline and the state it was found for (prepare_draw). +typedef struct { int valid; VkPipeState s; const VkPipeInfo *info; } PipeMemo; +// The pipelines found before, by state: vk_pipeline_graphics_async takes the pipeline cache's lock, which the contexts +// of a chunked replay would take on every draw whose state differs from the draw before. Direct-mapped. +#define PC_N 1024 +typedef struct { const VkPipeInfo *info; VkPipeState s; } PipeCache; + +// The memos and command buffer state of one recording context (VkExec.draw). Nothing in it is shared: the context's +// thread reads and writes it without a lock. +typedef struct VkExecDraw { + CbRes *cur; // resources of the submission being recorded + uint64_t cur_fence; + CbState B; + DescSigs DS; + BindStats BS; + RsState RS; + nvn_hmap rs_lists; // VkPipeInfo pointer -> RsList, until shutdown + const VkPipeInfo *rs_last; // the last lookup + const RsList *rs_last_l; + uint32_t c0_dim[3], c0_binst; // the dispatch size and base instance of the draw or dispatch being bound + uint64_t bind_gen; // bumped by every bind op that writes ex->prog, ubo, ssbo, tex or img + int twin_dispatch; // op_dispatch*: the resources being bound are a dispatch's + uint64_t wait_frame, wait_left_ns; // prepare_draw: full-screen pipeline waits left in this frame + ShaderEnt shader_tab[SHADER_SETS][SHADER_WAYS]; + StageMemo stagem[STAGEM_N]; + AcSlots AC[6]; + SlotMask slotm[SLOTM_N]; + FbMemo FM[FM_N]; + TexMemo TD[TD_N]; + ShaderMemo SM; + PipeMemo PM; + PipeCache PC[PC_N]; + UpdRing UPD; + VkWriteDescriptorSet w[MAX_BINDINGS]; // bind_resources' scratch + VkDescSlot d[MAX_DESC]; + VkBufferView bv[MAX_DESC]; + RsRange rs_tmp[MAX_DESC]; // rs_list's +} VkExecDraw; + +static VkExecDraw *draw_new(VkExec *ex) { + VkExecDraw *D = calloc(1, sizeof *D); + if (!D) rt_fatal("vk: out of memory (replay context)"); + D->bind_gen = 1; + return ex->draw = D; +} static void cb_done(void *arg) { // completion thread CbRes *r = arg; @@ -234,11 +329,17 @@ static void cb_done(void *arg) { // completion thread free(r->written), free(r->pools), free(r->chunks), free(r); } -void vk_exec_draw_new_cb(void) { memset(&B, 0, sizeof B); } +// Also after the bound state was replaced (a chunk's start): bind_sig compares the bound slots again. +void vk_exec_draw_new_cb(VkExec *ex) { + if (!ex->draw) return; + memset(&ex->draw->B, 0, sizeof ex->draw->B); + ex->draw->bind_gen++; +} -void vk_exec_draw_shaders(const GfxShader **vs, const GfxShader **fs) { - *vs = B.st_valid ? B.st.vs : NULL; - *fs = B.st_valid ? B.st.fs : NULL; +void vk_exec_draw_shaders(const VkExec *ex, const GfxShader **vs, const GfxShader **fs) { + int valid = ex->draw && ex->draw->B.st_valid; + *vs = valid ? ex->draw->B.st.vs : NULL; + *fs = valid ? ex->draw->B.st.fs : NULL; } // Marks a dispatch about to be recorded (vk.h vk_pass_mark_begin); the caller ends it after the dispatch. @@ -252,13 +353,24 @@ static VkPassMark *dispatch_mark(VkExec *ex, const GfxShader *cs, uint32_t x, ui } static CbRes *cb_res(VkExec *ex) { - if (cur && cur_fence == ex->fence) return cur; - cur = calloc(1, sizeof *cur); - if (!cur) rt_fatal("vk: out of memory"); - cur_fence = ex->fence; - memset(&B, 0, sizeof B); - vk_completion_add(ex->fence, cb_done, cur); - return cur; + VkExecDraw *D = ex->draw; + if (D->cur && D->cur_fence == ex->fence) return D->cur; + D->cur = calloc(1, sizeof *D->cur); + if (!D->cur) rt_fatal("vk: out of memory"); + D->cur_fence = ex->fence; + memset(&D->B, 0, sizeof D->B); + vk_completion_add(ex->fence, cb_done, D->cur); + return D->cur; +} + +typedef struct { const VkBufferCreateInfo *bi; const VmaAllocationCreateInfo *ai; Chunk *c; VmaAllocationInfo info; + VkResult r; } ChunkTry; +static int chunk_try(void *arg) { + ChunkTry *t = arg; + vk_mem_collect(); // the fence thread's collect waits while the caller holds cmd_lock + t->r = rt_oom_forced(RT_OOM_CHUNK) ? VK_ERROR_OUT_OF_DEVICE_MEMORY + : vmaCreateBuffer(vk_ctx()->vma, t->bi, t->ai, &t->c->buf, &t->c->alloc, &t->info); + return t->r == VK_SUCCESS; } static uint8_t *scratch(VkExec *ex, uint64_t size, VkBuffer *buf, VkDeviceSize *off) { @@ -294,10 +406,17 @@ static uint8_t *scratch(VkExec *ex, uint64_t size, VkBuffer *buf, VkDeviceSize * VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_HOST, .flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT, .requiredFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT}; - VmaAllocationInfo info; - VkResult res = vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &c->buf, &c->alloc, &info); - if (res != VK_SUCCESS) rt_fatal("vk: transient buffer of %llu bytes failed (%d)", (unsigned long long)n, (int)res); - c->map = info.pMappedData, c->size = n, c->used = 0; + // Out of memory: the recording waits for memory (rt_oom_wait) with what it holds (the texture lock at most, + // which vk_oom_release takes before cmd_lock); new chunks are made only when the spare ones ran out. + ChunkTry ct = {.bi = &bi, .ai = &ai, .c = c}; + ct.r = rt_oom_forced(RT_OOM_CHUNK) ? VK_ERROR_OUT_OF_DEVICE_MEMORY + : vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &c->buf, &c->alloc, &ct.info); + if (ct.r == VK_ERROR_OUT_OF_HOST_MEMORY || ct.r == VK_ERROR_OUT_OF_DEVICE_MEMORY) { + vk_oom_release(); + rt_oom_wait("vk: transient buffer", n, chunk_try, &ct); + } + if (ct.r != VK_SUCCESS) rt_fatal("vk: transient buffer of %llu bytes failed (%d)", (unsigned long long)n, (int)ct.r); + c->map = ct.info.pMappedData, c->size = n, c->used = 0; r->nchunks++; } *buf = c->buf; @@ -390,22 +509,39 @@ static const VkFormat tex_format[40] = { VK_FORMAT_A2B10G10R10_UNORM_PACK32, VK_FORMAT_A2B10G10R10_UINT_PACK32, VK_FORMAT_B10G11R11_UFLOAT_PACK32}; static struct { int used, di; VkFormat fmt; VkImage image; VmaAllocation alloc; VkImageView view, sview; } SD[48]; +static PlatMutex dummy_lock = PLAT_MUTEX_INIT; // the contexts make stand-ins on first use + static VkImageView storage_dummy(int di, int code) { VkFormat fmt = code > 0 && code < 40 ? tex_format[code] : VK_FORMAT_R8G8B8A8_UNORM; if (fmt == VK_FORMAT_R8G8B8A8_UNORM || dinfo[di].s != VK_SAMPLE_COUNT_1_BIT) return Z.img[di].sview; + VkImageView v = Z.img[di].sview; + plat_mutex_lock(&dummy_lock); for (int i = 0; i < 48; i++) { if (!SD[i].used) { SD[i].used = 1, SD[i].di = di, SD[i].fmt = fmt; make_dummy(di, fmt, &SD[i].image, &SD[i].alloc, &SD[i].view, &SD[i].sview); - return SD[i].sview; + v = SD[i].sview; + break; + } + if (SD[i].di == di && SD[i].fmt == fmt) { + v = SD[i].sview; + break; } - if (SD[i].di == di && SD[i].fmt == fmt) return SD[i].sview; } - return Z.img[di].sview; + plat_mutex_unlock(&dummy_lock); + return v; } +static void dummies_make(void); + static void dummies_init(void) { - if (Z.ready) return; + if (__atomic_load_n(&Z.ready, __ATOMIC_ACQUIRE)) return; + plat_mutex_lock(&dummy_lock); + if (!Z.ready) dummies_make(); + plat_mutex_unlock(&dummy_lock); +} + +static void dummies_make(void) { VkCtx *c = vk_ctx(); VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = ZERO_BYTES, .usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | @@ -429,11 +565,18 @@ static void dummies_init(void) { for (int i = 0; i < DV_COUNT; i++) make_dummy(i, VK_FORMAT_R8G8B8A8_UNORM, &Z.img[i].image, &Z.img[i].alloc, &Z.img[i].view, &Z.img[i].sview); - Z.ready = 1; + __atomic_store_n(&Z.ready, 1, __ATOMIC_RELEASE); } static void twins_release(VkCtx *c); +static void rs_forget(VkExec *ex) { // the pipeline infos the lists are keyed by go with the device + if (!ex->draw) return; + nvn_hm_clear(&ex->draw->rs_lists, free); + ex->draw->rs_last = NULL, ex->draw->rs_last_l = NULL; + memset(ex->draw->PC, 0, sizeof ex->draw->PC), ex->draw->PM.valid = 0; +} + // Releases the stand-ins and the depth twins. The device must be idle; the game path leaves them to process exit like // the pipelines. void vk_exec_draw_shutdown(void) { @@ -451,8 +594,7 @@ void vk_exec_draw_shutdown(void) { vmaDestroyImage(c->vma, SD[i].image, SD[i].alloc); } memset(SD, 0, sizeof SD); - nvn_hm_clear(&rs_lists, free); - rs_last = NULL, rs_last_l = NULL; + vk_exec_contexts(rs_forget); for (int i = 0; i < nspare_pool; i++) vkDestroyDescriptorPool(c->device, spare_pool[i], NULL); for (int i = 0; i < nspare_chunk; i++) vmaDestroyBuffer(c->vma, spare_chunk[i].buf, spare_chunk[i].alloc); nspare_pool = nspare_chunk = 0; @@ -651,13 +793,6 @@ void vk_exec_image_gone(VkImage image) { // ---- program to shaders -// (program, stage, answers) -> shader, 4-way set-associative, most recently used first. A miss costs a SHA-256 of the -// whole shader code (gfx_shader.c req_init), slow on CPUs without SHA instructions: a field frame looks up ~3000 and -// the 2048-slot direct-mapped table this replaced missed 3% of them on conflicts. -#define SHADER_SETS 2048 -#define SHADER_WAYS 4 -typedef struct { uint64_t serial; int stage; int used; GfxShaderAnswers a; const GfxShader *s; } ShaderEnt; -static ShaderEnt shader_tab[SHADER_SETS][SHADER_WAYS]; static int find_stage_scan(const prog_t *p, int nvn_stage, const uint8_t **ctl, const uint8_t **code, uint32_t *len) { for (uint32_t i = 0; i < p->nstage && i < 6; i++) { @@ -673,15 +808,10 @@ static int find_stage_scan(const prog_t *p, int nvn_stage, const uint8_t **ctl, return 0; } -// Which shader of a program serves each NVN stage, and its code length, per program serial (new at every -// nvnProgramSetShaders): the scan reads two words of every stage's control block, cold guest cache lines, and runs -// several times a draw. Direct-mapped; a collision only scans again. -#define STAGEM_N 512 -static struct { uint64_t serial; int8_t idx[6]; uint32_t len[6]; } stagem[STAGEM_N]; -static int find_stage(const prog_t *p, int nvn_stage, const uint8_t **ctl, const uint8_t **code, uint32_t *len) { +static int find_stage(VkExecDraw *D, const prog_t *p, int nvn_stage, const uint8_t **ctl, const uint8_t **code, uint32_t *len) { if (!p->serial || (unsigned)nvn_stage >= 6) return find_stage_scan(p, nvn_stage, ctl, code, len); - typeof(stagem[0]) *m = &stagem[p->serial * 0x9E3779B97F4A7C15ull >> 55]; + typeof(D->stagem[0]) *m = &D->stagem[p->serial * 0x9E3779B97F4A7C15ull >> 55]; if (m->serial != p->serial) { m->serial = p->serial; memset(m->idx, -1, sizeof m->idx); @@ -699,14 +829,13 @@ static int find_stage(const prog_t *p, int nvn_stage, const uint8_t **ctl, const return 1; } -// Texture and image types of a stage's first 32 slots, each kept with the handle and the generation and slot token it -// was looked up under: a slot is looked up again, a hash probe, only when the handle changed or its slot may resolve -// otherwise (slot_still). -static struct { uint64_t tex[32], img[32], ttok[32], itok[32]; uint32_t tgen[32], igen[32]; uint8_t tt[32], it[32]; } AC[6]; // vk_texture_slot_token of the slot a texture or image handle names; an empty handle always resolves to nothing. static uint64_t token_of(uint64_t h) { return h ? vk_texture_slot_token((int)(h & 0xfffff)) : 1; } +// A chunk asks for this handle at every op: what it resolves to depends on the op (vk_texture_slot_moves). +static int slot_moves(const VkExec *ex, uint64_t h) { return ex->chunked && h && vk_texture_slot_moves((int)(h & 0xfffff)); } + // Whether what handle h resolved to under the global generation *egen and its slot token etok (both read before that // lookup) still holds at global generation gen: nothing at all changed since, or not its slot (the token, read only when // the global generation moved, about every 16 draws). A token hit moves *egen to gen. @@ -719,22 +848,65 @@ static int slot_still(uint32_t *egen, uint64_t etok, uint64_t h, uint32_t gen) { } static int shader_memo_on(void); -// The slots of (program, stage) whose types its translation reads (gfx_shader_slot_mask), learned from its first -// shader; ~0u while unknown. The types of other slots select the same shader, so answers_for leaves them zero instead -// of looking up all 64 handles. Direct-mapped by serial and stage; a collision only relearns. -#define SLOTM_N 4096 -static struct { uint64_t serial; int st; uint32_t m; } slotm[SLOTM_N]; static uint32_t slotm_idx(uint64_t serial, int st) { return (uint32_t)((serial * 8 + (uint64_t)st) * 0x9E3779B97F4A7C15ull >> 52); } -static uint32_t slot_mask_of(uint64_t serial, int st) { - const typeof(slotm[0]) *e = &slotm[slotm_idx(serial, st)]; +static uint32_t slot_mask_of(VkExecDraw *D, uint64_t serial, int st) { + const typeof(D->slotm[0]) *e = &D->slotm[slotm_idx(serial, st)]; return serial && e->serial == serial && e->st == st ? e->m : ~0u; } -static void slot_mask_learn(uint64_t serial, int st, const GfxShader *s) { - if (!serial || slot_mask_of(serial, st) != ~0u) return; - slotm[slotm_idx(serial, st)] = (typeof(slotm[0])){serial, st, gfx_shader_slot_mask(s)}; +static void slot_mask_learn(VkExecDraw *D, uint64_t serial, int st, const GfxShader *s) { + if (!serial || slot_mask_of(D, serial, st) != ~0u) return; + D->slotm[slotm_idx(serial, st)] = (typeof(D->slotm[0])){serial, st, gfx_shader_slot_mask(s)}; +} + +// The shader of each (program, stage) a context translated, for the state scan of a chunked replay +// (vk_exec_draw_writes): which storage buffers a stage writes is a property of its code, the same in every translation. +// s NULL: the program has no such stage, or it failed to translate. Shared by the contexts and read by the scan without +// a lock: a writer takes wt_lock and clears the key while it changes the entry, a reader checks the key before and +// after reading the shader. A program not found holds the chunks after its draw (vk_exec.c scan_writes), so the table +// is large and set-associative: a field frame in 1.2.1 draws with programs whose serials reach some 25,000 within a +// minute, and the 4096 direct-mapped entries this replaced missed in nearly every chunk, which then waited for the one +// before it. A full set gives up its oldest program (the lowest serial). +#define WT_WAYS 8 +#define WT_SETS (1 << 14) +static struct { uint64_t key; const GfxShader *s; } WT[WT_SETS][WT_WAYS]; +static PlatMutex wt_lock = PLAT_MUTEX_INIT; + +static uint64_t wt_key(uint64_t serial, int st) { return serial << 4 | (uint64_t)st << 1 | 1; } + +static typeof(WT[0][0]) *wt_set(uint64_t k) { return WT[(k * 0x9E3779B97F4A7C15ull) >> (64 - 14)]; } + +static int wt_way(const typeof(WT[0][0]) *set, uint64_t k) { + for (int w = 0; w < WT_WAYS; w++) + if (__atomic_load_n(&set[w].key, __ATOMIC_ACQUIRE) == k) return w; + return -1; +} + +static void wt_learn(uint64_t serial, int st, const GfxShader *s) { + uint64_t k = wt_key(serial, st); + typeof(WT[0][0]) *set = wt_set(k); + if (!serial || wt_way(set, k) >= 0) return; + plat_mutex_lock(&wt_lock); + if (wt_way(set, k) < 0) { + int w = 0; + for (int i = 1; i < WT_WAYS; i++) + if (set[i].key < set[w].key) w = i; // an empty way (key 0) first, else the oldest + __atomic_store_n(&set[w].key, 0, __ATOMIC_RELEASE); + __atomic_store_n(&set[w].s, s, __ATOMIC_RELEASE); + __atomic_store_n(&set[w].key, k, __ATOMIC_RELEASE); + } + plat_mutex_unlock(&wt_lock); +} + +static int wt_find(uint64_t serial, int st, const GfxShader **s) { + uint64_t k = wt_key(serial, st); + const typeof(WT[0][0]) *set = wt_set(k); + int w = serial ? wt_way(set, k) : -1; + if (w < 0) return 0; + *s = __atomic_load_n(&set[w].s, __ATOMIC_ACQUIRE); + return __atomic_load_n(&set[w].key, __ATOMIC_ACQUIRE) == k; } static uint8_t type_of_handle(uint64_t h) { // texture lock held @@ -746,6 +918,7 @@ static uint8_t type_of_handle(uint64_t h) { // texture lock held // The answers of stage st for the slots in mask; the types of other slots stay zero. static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, GfxShaderAnswers *a) { + VkExecDraw *D = ex->draw; memset(a, 0, sizeof *a); if (st == NVN_STAGE_VS) vk_pipeline_answers(&ex->pipe, a, NULL); else if (st == NVN_STAGE_FS) vk_pipeline_answers(&ex->pipe, NULL, a); @@ -755,7 +928,7 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G memcpy(&l, ctl + 0x748 + 4 * k, 4); a->local[k] = (uint16_t)(l ? l : 1); } - typeof(AC[0]) *c = &AC[st]; + typeof(D->AC[0]) *c = &D->AC[st]; // The generation and tokens are read before the lookups (one that creates an entry changes them) and unlocked: a // registration racing with them is ordered after this draw, and a slot stored under older ones is only looked up // again. @@ -764,10 +937,10 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G for (uint32_t m = mask; m; m &= m - 1) { int slot = __builtin_ctz(m); uint64_t h = ex->tex[st][slot], hi = ex->img[st][slot]; - int tmiss = c->tex[slot] != h || !c->ttok[slot] || !slot_still(&c->tgen[slot], c->ttok[slot], h, gen); - int imiss = c->img[slot] != hi || !c->itok[slot] || !slot_still(&c->igen[slot], c->itok[slot], hi, gen); + int tmiss = c->tex[slot] != h || !c->ttok[slot] || slot_moves(ex, h) || !slot_still(&c->tgen[slot], c->ttok[slot], h, gen); + int imiss = c->img[slot] != hi || !c->itok[slot] || slot_moves(ex, hi) || !slot_still(&c->igen[slot], c->itok[slot], hi, gen); uint64_t tt = tmiss ? token_of(h) : 0, it = imiss ? token_of(hi) : 0; - if ((tmiss || imiss || check) && !locked) vk_texture_lock(), locked = 1, BS.ac_miss++; + if ((tmiss || imiss || check) && !locked) vk_texture_lock(), locked = 1, D->BS.ac_miss++; if (tmiss) c->tt[slot] = type_of_handle(h), c->tex[slot] = h, c->ttok[slot] = tt, c->tgen[slot] = gen; if (imiss) c->it[slot] = type_of_handle(hi), c->img[slot] = hi, c->itok[slot] = it, c->igen[slot] = gen; if (check && (type_of_handle(h) != c->tt[slot] || type_of_handle(hi) != c->it[slot])) // TOTK_SHADER_MEMO=check @@ -780,23 +953,32 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G // *present: the program has this stage. Returns NULL when it has not or the translation failed. pending (graphics // stages, vk_pipeline_async_on): a shader not translated yet is queued and NULL returned with *pending = 1. await: a // shader queued earlier is waited for (gfx_shader_await) instead. -static int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion) +static _Thread_local int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion) +// Chunks of one submission note their scene draws for DLSS's motion one at a time: the list, its memos and the +// last-frame block tables (vk_mvprev.c, vk_mvsurvey.c) are shared; vk_mvobj_execute reads the list once the chunks +// before DLSS's evaluation are recorded (vk_exec.c run_job). +static PlatMutex mo_lock = PLAT_MUTEX_INIT; static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pending, int await) { + VkExecDraw *D = ex->draw; const uint8_t *ctl, *code; uint32_t len; *present = 0; if (pending) *pending = 0; - if (!ex->prog || !find_stage(ex->prog, st, &ctl, &code, &len)) return NULL; + if (!ex->prog) return NULL; + if (!find_stage(ex->draw, ex->prog, st, &ctl, &code, &len)) { + wt_learn(ex->prog->serial, st, NULL); + return NULL; + } *present = 1; GfxShaderAnswers a; uint64_t serial = ex->prog->serial; - answers_for(ex, st, ctl, slot_mask_of(serial, st), &a); + answers_for(ex, st, ctl, slot_mask_of(D, serial, st), &a); a.mvprev = (uint8_t)(sf_mvprev && st == NVN_STAGE_VS); ShaderEnt *set = NULL; if (serial) { uint64_t h = (serial * 0x9E3779B97F4A7C15ull ^ (uint64_t)st) ^ nvn_hm_hash(&a, sizeof a); // a is zero-padded - set = shader_tab[(h ^ h >> 29) & (SHADER_SETS - 1)]; + set = D->shader_tab[(h ^ h >> 29) & (SHADER_SETS - 1)]; for (int w = 0; w < SHADER_WAYS && set[w].used; w++) { ShaderEnt *e = &set[w]; if (e->serial != serial || e->stage != st || memcmp(&e->a, &a, sizeof a)) continue; @@ -805,6 +987,7 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin memmove(&set[1], &set[0], w * sizeof *set); set[0] = hit; } + wt_learn(serial, st, set[0].s); // again after WT gave the entry up return set[0].s; } } @@ -813,7 +996,8 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin : pending && (vk_pipeline_async_on() || a.mvprev) ? gfx_shader_get_async(gs, ctl, code, len, &a, pending) : gfx_shader_get(gs, ctl, code, len, NULL, &a); if (pending && *pending) return NULL; - if (s) slot_mask_learn(serial, st, s); + if (s) slot_mask_learn(D, serial, st, s); + wt_learn(serial, st, s); if (!s) warn_once(W_SHADER, "a stage %d shader of program %llu failed to translate: draws with it are skipped", st, (unsigned long long)serial); else if (set) { // in front, the least recently used way drops out @@ -834,16 +1018,16 @@ static VkDescriptorBufferInfo zero_info(uint64_t max) { } // vk_mem_count, also kept per (class, use) while a graphics build runs (RS). -static void count_use(int cls, int use, uint64_t n) { +static void count_use(VkExecDraw *D, int cls, int use, uint64_t n) { vk_mem_count(cls, use, n); - if (!RS.rec) return; - for (int i = 0; i < RS.ncnt; i++) - if (RS.cnt[i].cls == cls && RS.cnt[i].use == use) { - RS.cnt[i].bytes += n; + if (!D->RS.rec) return; + for (int i = 0; i < D->RS.ncnt; i++) + if (D->RS.cnt[i].cls == cls && D->RS.cnt[i].use == use) { + D->RS.cnt[i].bytes += n; return; } - if (RS.ncnt < (int)(sizeof RS.cnt / sizeof RS.cnt[0])) RS.cnt[RS.ncnt++] = (RsCount){cls, use, n}; - else RS.cnt_over = 1; + if (D->RS.ncnt < (int)(sizeof D->RS.cnt / sizeof D->RS.cnt[0])) D->RS.cnt[D->RS.ncnt++] = (RsCount){cls, use, n}; + else D->RS.cnt_over = 1; } // A buffer descriptor for [addr, addr+size) of guest memory: its backing (a pool's VRAM mirror when it has one) when @@ -851,28 +1035,31 @@ static void count_use(int cls, int use, uint64_t n) { // written: the shader stores into it (a mirror's range goes back to guest memory at the end of the submission). static int bind_range(VkExec *ex, uint64_t addr, uint64_t size, uint64_t align, uint64_t max, int use, int written, VkDescriptorBufferInfo *out) { + VkExecDraw *D = ex->draw; VkMemSpan sp; if (!addr || !vk_mem_resolve(addr, 1, &sp)) return 0; uint64_t n = size && size < sp.avail ? size : sp.avail; if (n > max) n = max; if (!n) return 0; if (!written && (use == VK_MUSE_UBO || use == VK_MUSE_SSBO) && ub_bind(ex, addr, n, align, out)) { - count_use(VK_MCLS_SCRATCH, use, n); + count_use(ex->draw, VK_MCLS_SCRATCH, use, n); return 1; } if (sp.offset % align == 0) { VK_MEM_USE(sp); if (written) ub_written(ex, addr, n); *out = (VkDescriptorBufferInfo){sp.buffer, sp.offset, n}; - if (written) vk_mem_gpu_wrote_shader(&sp, n), vk_usnap_drop(addr, n), vk_exec_ub_foreign(ex, addr, n); - count_use(sp.cls, use, n); + if (written) + vk_mem_gpu_wrote_shader(&sp, n), vk_usnap_drop_at(addr, n, ex->chunked ? ex->drop_n : 0), + vk_exec_ub_foreign(ex, addr, n); + count_use(ex->draw, sp.cls, use, n); VK_MEM_BOUND(sp); return 1; } if (sp.alloc) return 0; VkBuffer b; VkDeviceSize o; - RS.volatile_ = 1; + D->RS.volatile_ = 1; memcpy(scratch(ex, n, &b, &o), (const void *)(uintptr_t)addr, n); vk_mem_count(sp.cls, VK_MUSE_CPU_READ, n); vk_mem_count(VK_MCLS_SCRATCH, use, n); @@ -921,25 +1108,26 @@ static int c0_on(void) { // TOTK_FIX_C0=0: c0 reads zeros instead of the drive // c0 from replay scratch; a bank equal to the stage's previous one reuses its range, so it costs no copy. static VkDescriptorBufferInfo c0_info(VkExec *ex, int st) { + VkExecDraw *D = ex->draw; int k = st == NVN_STAGE_VS ? 0 : st == NVN_STAGE_FS ? 1 : 2; // vk_c0_fill reads the storage slots, and the uniform slots and dispatch size for compute or the base instance for // graphics: equal inputs give the same bank, so neither the fill nor the 2 KiB compare is needed. - typeof(B.c0_in[0]) in; + typeof(D->B.c0_in[0]) in; memset(&in, 0, sizeof in); memcpy(in.ssbo, ex->ssbo[st], sizeof in.ssbo); - if (st == NVN_STAGE_CS) memcpy(in.ubo, ex->ubo[st], sizeof in.ubo), memcpy(in.dim, c0_dim, sizeof in.dim); - else in.binst = c0_binst; - if (B.c0_valid[k] && !memcmp(&in, &B.c0_in[k], sizeof in)) - return (VkDescriptorBufferInfo){B.c0_buf[k], B.c0_off[k], VK_C0_BYTES}; - B.c0_in[k] = in; + if (st == NVN_STAGE_CS) memcpy(in.ubo, ex->ubo[st], sizeof in.ubo), memcpy(in.dim, D->c0_dim, sizeof in.dim); + else in.binst = D->c0_binst; + if (D->B.c0_valid[k] && !memcmp(&in, &D->B.c0_in[k], sizeof in)) + return (VkDescriptorBufferInfo){D->B.c0_buf[k], D->B.c0_off[k], VK_C0_BYTES}; + D->B.c0_in[k] = in; uint8_t tmp[VK_C0_BYTES]; - vk_c0_fill(tmp, st, ex->ssbo[st], ex->ubo[st], c0_dim, c0_binst); - if (!B.c0_valid[k] || memcmp(tmp, B.c0[k], VK_C0_BYTES)) { - memcpy(scratch(ex, VK_C0_BYTES, &B.c0_buf[k], &B.c0_off[k]), tmp, VK_C0_BYTES); - memcpy(B.c0[k], tmp, VK_C0_BYTES); - B.c0_valid[k] = 1; + vk_c0_fill(tmp, st, ex->ssbo[st], ex->ubo[st], D->c0_dim, D->c0_binst); + if (!D->B.c0_valid[k] || memcmp(tmp, D->B.c0[k], VK_C0_BYTES)) { + memcpy(scratch(ex, VK_C0_BYTES, &D->B.c0_buf[k], &D->B.c0_off[k]), tmp, VK_C0_BYTES); + memcpy(D->B.c0[k], tmp, VK_C0_BYTES); + D->B.c0_valid[k] = 1; } - return (VkDescriptorBufferInfo){B.c0_buf[k], B.c0_off[k], VK_C0_BYTES}; + return (VkDescriptorBufferInfo){D->B.c0_buf[k], D->B.c0_off[k], VK_C0_BYTES}; } // The buffer range limits, read once: the device's properties are a cold cache line on every binding of every draw. @@ -954,8 +1142,9 @@ static void limits_init(void) { // Per-object motion (section at the end): while it records a draw again, these replace the uniform bindings of the // vertex stage's slots 18-20 (last frame's camera, bones and place) and the fragment stage's slot 3 (its parameters). -static int mo_ovr_on; -static VkDescriptorBufferInfo mo_ovr[2][32]; // [vertex, fragment][slot]; buffer 0: not replaced +// Per thread: the other chunks' draws go on meanwhile and keep their own bindings. +static _Thread_local int mo_ovr_on; +static _Thread_local VkDescriptorBufferInfo mo_ovr[2][32]; // [vertex, fragment][slot]; buffer 0: not replaced static VkDescriptorBufferInfo ubo_info(VkExec *ex, int st, int slot) { if (mo_ovr_on && (st == NVN_STAGE_VS || st == NVN_STAGE_FS) && slot >= 0 && slot < 32 && mo_ovr[st][slot].buffer) @@ -967,7 +1156,7 @@ static VkDescriptorBufferInfo ubo_info(VkExec *ex, int st, int slot) { if (slot == 1 && ex->prog) { // compiler constants stored after the shader code const uint8_t *ctl, *code; uint32_t len, c1; - if (find_stage(ex->prog, st, &ctl, &code, &len)) { + if (find_stage(ex->draw, ex->prog, st, &ctl, &code, &len)) { memcpy(&c1, ctl + 0x700, 4); if (c1 < len && bind_range(ex, (uint64_t)(uintptr_t)code + c1, len - c1, align, max, VK_MUSE_UBO, 0, &bi)) return bi; @@ -975,11 +1164,12 @@ static VkDescriptorBufferInfo ubo_info(VkExec *ex, int st, int slot) { } else if (slot >= 3 && slot - 3 < 18) { const p_bind_buf *b = &ex->ubo[st][slot - 3]; if (b->size && ub_bind(ex, b->addr, b->size < max ? b->size : max, align, &bi)) { // updated in this pass - count_use(VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); + count_use(ex->draw, VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); return bi; } - if (b->size && vk_usnap_find(b->addr, b->size < max ? b->size : max, align, &bi)) { // as of the submit call - count_use(VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); + if (b->size && vk_usnap_find(b->addr, b->size < max ? b->size : max, ex->chunked ? ex->drop_n : UINT32_MAX, align, + &bi)) { // as of the submit call + count_use(ex->draw, VK_MCLS_SCRATCH, VK_MUSE_UBO, bi.range); return bi; } if (bind_range(ex, b->addr, b->size, align, max, VK_MUSE_UBO, 0, &bi)) return bi; @@ -1016,23 +1206,27 @@ static int ubo_ptr_res(const GfxShaderRes *r) { // into the buffer's GPU copy; guest memory gets them only from the write-back when the submission ends. The terrain // rewrites its pointer block this way before every draw, all at one address, so ubo_ptr reads the latest update that // covers the pointer, else memory. Payloads stay in the recorded command buffer for the whole replay. -#define UPD_N 4096 -static struct { - uint64_t fence; // the submission the entries belong to - uint32_t n; // updates noted in it - struct { uint64_t lo, hi; const uint8_t *data; } u[UPD_N]; -} UPD; - -static void upd_note(const VkExec *ex, const p_update_ub *u) { - if (UPD.fence != ex->fence) UPD.fence = ex->fence, UPD.n = 0; - UPD.u[UPD.n++ % UPD_N] = (typeof(UPD.u[0])){u->dst, u->dst + u->size, u->data}; +static void upd_note(VkExec *ex, const p_update_ub *u) { + VkExecDraw *D = ex->draw; + if (ex->chunked) { // the scan listed it + ex->upd_n++; + return; + } + if (D->UPD.fence != ex->fence) D->UPD.fence = ex->fence, D->UPD.n = 0; + D->UPD.u[D->UPD.n++ % UPD_N] = (VkExecUpd){u->dst, u->dst + u->size, u->data}; } // 1 when an update of this submission supplied [a, a+n). static int upd_read(const VkExec *ex, uint64_t a, void *out, size_t n) { - if (UPD.fence != ex->fence) return 0; - for (uint32_t i = UPD.n, end = UPD.n > UPD_N ? UPD.n - UPD_N : 0; i-- > end;) { - const typeof(UPD.u[0]) *e = &UPD.u[i % UPD_N]; + VkExecDraw *D = ex->draw; + const VkExecUpd *u = ex->upd; // a chunk: the scan's list of the submission, of which upd_n precede this op + uint32_t cnt = ex->upd_n, mask = UINT32_MAX; + if (!ex->chunked) { + if (D->UPD.fence != ex->fence) return 0; + u = D->UPD.u, cnt = D->UPD.n, mask = UPD_N - 1; + } + for (uint32_t i = cnt, end = cnt > UPD_N ? cnt - UPD_N : 0; i-- > end;) { + const VkExecUpd *e = &u[i & mask]; if (a >= e->hi || a + n <= e->lo) continue; if (a < e->lo || a + n > e->hi) return 0; // partly covered: memory memcpy(out, e->data + (a - e->lo), n); @@ -1043,9 +1237,10 @@ static int upd_read(const VkExec *ex, uint64_t a, void *out, size_t n) { // Applies the updates of this submission that touch [a, a+n) onto out, oldest first (ub_bind's merged copies). static void upd_apply(const VkExec *ex, uint64_t a, uint8_t *out, size_t n) { - if (UPD.fence != ex->fence) return; - for (uint32_t i = UPD.n > UPD_N ? UPD.n - UPD_N : 0; i < UPD.n; i++) { - const typeof(UPD.u[0]) *e = &UPD.u[i % UPD_N]; + const UpdRing *U = &ex->draw->UPD; + if (U->fence != ex->fence) return; + for (uint32_t i = U->n > UPD_N ? U->n - UPD_N : 0; i < U->n; i++) { + const VkExecUpd *e = &U->u[i % UPD_N]; uint64_t lo = e->lo > a ? e->lo : a, hi = e->hi < a + n ? e->hi : a + n; if (lo < hi) memcpy(out + (lo - a), e->data + (lo - e->lo), hi - lo); } @@ -1062,18 +1257,29 @@ static uint64_t ubo_ptr(const VkExec *ex, int st, const GfxShaderRes *r) { return a; } +// The binding a storage buffer resource names at this point of the replay, NULL when it names none. *ubo: a compute +// shader's uniform binding; up holds the binding of an address read from a uniform block (size 0: up to the end of +// the backing). +static const p_bind_buf *ssbo_bound(const VkExec *ex, int st, const GfxShaderRes *r, p_bind_buf *up, int *ubo) { + int slot = ssbo_slot(r, st, ubo); + if (slot >= 0 && slot < 16) return *ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot]; + if (!ubo_ptr_res(r)) return NULL; + up->addr = ubo_ptr(ex, st, r); + return up; +} + +// vk_caps.c: shaders translated for sbuf_align address the buffer from its address rounded down to it. +static uint64_t sbuf_lead(uint64_t addr) { return vk_caps.sbuf_align ? addr & (vk_caps.sbuf_align - 1) : 0; } + static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes *r, int written) { if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init(); uint64_t max = LIM.sbo_max, align = LIM.sbo_align; VkDescriptorBufferInfo bi; - int ubo, slot = ssbo_slot(r, st, &ubo); + int ubo; p_bind_buf up = {0}; - const p_bind_buf *b = NULL; - if (slot >= 0 && slot < 16) b = ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot]; - else if (ubo_ptr_res(r)) up.addr = ubo_ptr(ex, st, r), b = &up; // size 0: up to the end of the backing + const p_bind_buf *b = ssbo_bound(ex, st, r, &up, &ubo); if (b) { - // vk_caps.c: shaders translated for sbuf_align address the buffer from its address rounded down to it. - uint64_t lead = vk_caps.sbuf_align ? b->addr & (vk_caps.sbuf_align - 1) : 0; + uint64_t lead = sbuf_lead(b->addr); if (bind_range(ex, b->addr - lead, b->size ? b->size + lead : 0, align, max, ubo ? VK_MUSE_UBO : VK_MUSE_SSBO, written, &bi)) { if (written && bi.buffer != VK_NULL_HANDLE) note_written(ex, b->addr - lead, bi.range); @@ -1088,6 +1294,36 @@ static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes * return zero_info(max); } +int vk_exec_draw_writes(const VkExec *s, int dispatch, uint64_t (*out)[2], int max) { + if (!s->prog) return 0; + static const int gfx[] = {NVN_STAGE_VS, NVN_STAGE_FS}, cs[] = {NVN_STAGE_CS}; + const int *st = dispatch ? cs : gfx, nst = dispatch ? 1 : 2; + const GfxShader *sh[2]; + for (int i = 0; i < nst; i++) + if (!wt_find(s->prog->serial, st[i], &sh[i])) return -1; + if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init(); + int n = 0; + for (int i = 0; i < nst; i++) { + const GfxShaderRes *r; + int nr = sh[i] ? gfx_shader_resources(sh[i], GFX_RES_SSBO, &r) : 0; + for (int j = 0; j < nr; j++) { + int ubo; + p_bind_buf up = {0}; + const p_bind_buf *b = (r[j].flags & 1) ? ssbo_bound(s, st[i], &r[j], &up, &ubo) : NULL; + VkMemSpan sp; + if (!b || !b->addr) continue; + uint64_t lead = sbuf_lead(b->addr), addr = b->addr - lead, size = b->size ? b->size + lead : 0; + if (!vk_mem_resolve(addr, 1, &sp)) continue; + uint64_t len = size && size < sp.avail ? size : sp.avail; // as bind_range binds it + if (len > LIM.sbo_max) len = LIM.sbo_max; + if (!len) continue; + if (n == max) return -1; + out[n][0] = addr, out[n][1] = len, n++; + } + } + return n; +} + // ---- feedback loops (maxwell.md 1.9, vk.h VkCtx.feedback) // TotK's G-buffer and terrain shaders sample targets of the pass that renders them, at their own pixel, after earlier // draws of the pass wrote them (albedo, normals, R32F linear depth). Maxwell reads see those writes; in Vulkan the read @@ -1105,8 +1341,6 @@ static int in_fbl(const VkExec *ex, VkImage img) { return 0; } -// The image and the levels [l0, l1) and layers [a0, a1) a texture's view covers. -typedef struct { VkImage image; uint32_t l0, l1, a0, a1; } SubRange; static SubRange sub_of(const VkTexImage *img, const view_t *v, int hv) { SubRange r = {img->image, 0, img->levels, 0, img->layers}; @@ -1142,9 +1376,6 @@ static int td_mode(void) { // The subresource a texture handle samples, per handle, kept while its slot resolves the same (slot_still): a draw's // texture bindings change the whole-scan memo below, but the handles they bind were mostly looked up before, in this // pass or an earlier one. Direct-mapped. -#define FM_N 8192 -static struct { uint64_t h, tok; uint32_t gen; SubRange r; } FM[FM_N]; - static SubRange sampled_sub(uint64_t h) { // texture lock held view_t v; int hv; @@ -1155,6 +1386,7 @@ static SubRange sampled_sub(uint64_t h) { // texture lock held // Attachments of the open pass the fragment shader fs samples; kept while the shader, the texture bindings and the // pass stay the same. static uint32_t feedback_scan(VkExec *ex, const GfxShader *fs) { + VkExecDraw *D = ex->draw; if (ex->fb_fs == fs && ex->fb_serial == ex->tex_serial && ex->fb_pass == ex->pass_serial) return ex->fb_sampled; const GfxShaderRes *res; int n = gfx_shader_resources(fs, GFX_RES_TEX, &res); @@ -1165,8 +1397,8 @@ static uint32_t feedback_scan(VkExec *ex, const GfxShader *fs) { int slot = res[r].slot + k; uint64_t h = slot >= 0 && slot < 128 ? ex->tex[NVN_STAGE_FS][slot] : 0; if (!h) continue; - typeof(FM[0]) *f = &FM[(h * 0x9E3779B97F4A7C15ull) >> 51]; - if (memo && f->h == h && slot_still(&f->gen, f->tok, h, gen)) { + typeof(D->FM[0]) *f = &D->FM[(h * 0x9E3779B97F4A7C15ull) >> 51]; + if (memo && f->h == h && !slot_moves(ex, h) && slot_still(&f->gen, f->tok, h, gen)) { if (memo == 2) { if (!locked) vk_texture_lock(), locked = 1; SubRange c = sampled_sub(h); @@ -1195,7 +1427,7 @@ static uint32_t feedback_prepare(VkExec *ex, const GfxShader *fs) { if (mode == VK_FEEDBACK_OFF || !fs || !ex->in_pass) return 0; uint32_t m = feedback_scan(ex, fs); uint32_t d = m & ex->fb_dirty; - if (d) vk_exec_frame.fb_draws++; + if (d) ex->fr.fb_draws++; if (d & ~(mode == VK_FEEDBACK_LAYOUT ? ex->fbl_mask : 0)) { if (mode == VK_FEEDBACK_LAYOUT) for (uint32_t i = 0; i < 9; i++) { @@ -1204,7 +1436,7 @@ static uint32_t feedback_prepare(VkExec *ex, const GfxShader *fs) { } vk_exec_pass_end(ex); vk_exec_pass_begin(ex); // learned targets now in the feedback layout; the full barrier cleared fb_dirty - vk_exec_frame.fb_restarts++; + ex->fr.fb_restarts++; } return mode == VK_FEEDBACK_LAYOUT ? m & ex->fbl_mask : 0; } @@ -1222,7 +1454,7 @@ static void feedback_barrier(VkExec *ex) { vkCmdPipelineBarrier2(ex->cb, &di); ex->recorded++; ex->fb_dirty = 0; - vk_exec_frame.fb_barriers++; + ex->fr.fb_barriers++; } // Marks the attachments the draw with state s writes: colour targets with a nonzero effective write mask, depth with @@ -1243,29 +1475,29 @@ static void tex_info_full(VkExec *ex, const GfxShaderRes *r, uint64_t h, int sto // for its storage check; a re-upload writes the same image, so the descriptor stays. Draws bind the same textures // thousands of times a frame (15,000 lookups a frame in the monsters fight); a hit skips the slot lookup, the view key // with its hash probe and the sampler lookup. Storage images, which note their binding per submission, are -// not kept. Direct-mapped; texture lock held. TOTK_TEX_MEMO=0 turns it off; =check computes every hit again and stops +// not kept. Direct-mapped; the texture lock only for the lookups (tex_lock). TOTK_TEX_MEMO=0 turns it off; =check computes every hit again and stops // on a difference. -#define TD_N 8192 -static struct { - uint64_t h, tok; - uint32_t gen, epoch, sgen; - uint16_t type; - uint8_t separate, valid; - VkImage image; - VkDescriptorImageInfo out; -} TD[TD_N]; +// The texture lock, taken on the first lookup that needs it (*locked): a memo hit needs none, and the contexts of a +// chunked replay take it from several threads. +static void tex_lock(int *locked) { + if (!*locked) vk_texture_lock(), *locked = 1; +} static void tex_info_lookup(VkExec *ex, const GfxShaderRes *r, uint64_t h, int storage, int separate, - VkDescriptorImageInfo *out) { + VkDescriptorImageInfo *out, int *locked) { + VkExecDraw *D = ex->draw; int td_on = td_mode(); VkImage image; - if (!h || storage || !td_on) return tex_info_full(ex, r, h, storage, separate, out, &image); + if (!h || storage || !td_on || slot_moves(ex, h)) return tex_lock(locked), tex_info_full(ex, r, h, storage, separate, out, &image); uint32_t gen = vk_texture_slots_gen(), ep = vk_texture_epoch(), sg = vk_sampler_gen(); // before any lookup - typeof(TD[0]) *m = &TD[(h * 0x9E3779B97F4A7C15ull ^ (uint64_t)r->type << 7 ^ (uint64_t)separate) >> 51]; + typeof(D->TD[0]) *m = &D->TD[(h * 0x9E3779B97F4A7C15ull ^ (uint64_t)r->type << 7 ^ (uint64_t)separate) >> 51]; if (m->valid && m->h == h && m->type == r->type && m->separate == separate && m->sgen == sg && slot_still(&m->gen, m->tok, h, gen)) { if (m->epoch != ep) { // the slot's first binding in this submission: only its storage check is due - vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL); + if (!vk_texture_slot_checked((int)(h & 0xfffff))) { // another context may have made it + tex_lock(locked); + vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL); + } m->epoch = ep; } *out = m->out; @@ -1273,6 +1505,7 @@ static void tex_info_lookup(VkExec *ex, const GfxShaderRes *r, uint64_t h, int s : VK_IMAGE_LAYOUT_GENERAL; if (td_on == 2) { VkDescriptorImageInfo full; + tex_lock(locked); tex_info_full(ex, r, h, storage, separate, &full, &image); if (full.sampler != out->sampler || full.imageView != out->imageView || full.imageLayout != out->imageLayout) rt_fatal("vk_exec: texture descriptor memo of handle %#llx type %#x differs from a lookup", @@ -1281,6 +1514,7 @@ static void tex_info_lookup(VkExec *ex, const GfxShaderRes *r, uint64_t h, int s return; } uint64_t tok = token_of(h); // before the lookup + tex_lock(locked); tex_info_full(ex, r, h, storage, separate, out, &image); *m = (typeof(*m)){.h = h, .tok = tok, .gen = gen, .epoch = ep, .sgen = sg, .type = r->type, .separate = (uint8_t)separate, .valid = 1, .image = image, .out = *out}; @@ -1327,9 +1561,9 @@ static void tex_info_full(VkExec *ex, const GfxShaderRes *r, uint64_t h, int sto // (https://github.com/Revivedx/SavageWindRecomp, MIT); the code is ours. TOTK_DEPTH_TWIN=0: off. #define TWINS 8 typedef struct { VkImage src, img; VmaAllocation mem; VkImageView view[2]; uint32_t w, h, layers; VkFormat fmt; } Twin; +static PlatMutex twin_lock = PLAT_MUTEX_INIT; // the table and the copy buffer: contexts recording chunks share them static Twin twins[TWINS]; static unsigned twin_next; -static int twin_dispatch; // op_dispatch*: the resources being bound are a dispatch's static VkBuffer twin_buf; static VmaAllocation twin_buf_mem; static VkDeviceSize twin_buf_cap; @@ -1478,17 +1712,24 @@ static Twin *twin_update(VkExec *ex, const VkTexImage *img) { // The image descriptor of slot h; in a dispatch, a compare-sampled colour image the driver cannot compare is swapped // for its depth twin; in a pass larger than it, the scene's light target, once DLSS scaled it this frame, for DLSS's // output (vk_dlaa.c). -static void tex_info(VkExec *ex, const GfxShaderRes *r, uint64_t h, int storage, int separate, VkDescriptorImageInfo *out) { - tex_info_lookup(ex, r, h, storage, separate, out); - if (h && !storage && vk_dlaa_swap_img && ex->in_pass && ex->pass_w > vk_dlaa_swap_img->width + 1 && - vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL) == vk_dlaa_swap_img) - out->imageView = vk_dlaa_swap_view, out->imageLayout = VK_IMAGE_LAYOUT_GENERAL; - if (!twin_dispatch || storage || separate || !h || !(r->type & 0x400)) return; +static void tex_info(VkExec *ex, const GfxShaderRes *r, uint64_t h, int storage, int separate, VkDescriptorImageInfo *out, + int *locked) { + VkExecDraw *D = ex->draw; + tex_info_lookup(ex, r, h, storage, separate, out, locked); + if (h && !storage && vk_dlaa_swap_img && ex->in_pass && ex->pass_w > vk_dlaa_swap_img->width + 1) { + tex_lock(locked); + if (vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL) == vk_dlaa_swap_img) + out->imageView = vk_dlaa_swap_view, out->imageLayout = VK_IMAGE_LAYOUT_GENERAL; + } + if (!D->twin_dispatch || storage || separate || !h || !(r->type & 0x400)) return; VkImageViewType vt = res_view_type(r->type); if (vt != VK_IMAGE_VIEW_TYPE_2D && vt != VK_IMAGE_VIEW_TYPE_2D_ARRAY) return; + tex_lock(locked); const VkTexImage *img = vk_texture_by_id_locked((int)(h & 0xfffff), NULL, NULL); + plat_mutex_lock(&twin_lock); Twin *t = img ? twin_update(ex, img) : NULL; VkImageView v = t ? t->view[vt == VK_IMAGE_VIEW_TYPE_2D_ARRAY] : VK_NULL_HANDLE; + plat_mutex_unlock(&twin_lock); if (v) out->imageView = v, out->imageLayout = VK_IMAGE_LAYOUT_GENERAL; } @@ -1589,33 +1830,35 @@ static void support_fill(uint8_t *b, float w, float h, float point_size) { // and textures (shaders and pipeline would be the same) and those that also have its buffers and viewport (all // descriptors would be the same). BS is declared beside DS. static int bind_stats_on(void) { - if (BS.on < 0) { + static int on = -1; + if (on < 0) { const char *e = cfg_get("TOTK_BIND_STATS"); - BS.on = e && *e && strcmp(e, "0"); + on = e && *e && strcmp(e, "0"); } - return BS.on; + return on; } -static void bind_stats(const VkPipeInfo *info) { +static void bind_stats(VkExecDraw *D, const VkPipeInfo *info) { if (!bind_stats_on() || info->bind_point != VK_PIPELINE_BIND_POINT_GRAPHICS) return; for (uint32_t i = 0; i < info->n_bindings; i++) { const VkPipeBinding *b = &info->bindings[i]; int fs = info->n_sets > 1 ? b->set : !(b->stages & VK_SHADER_STAGE_VERTEX_BIT); - BS.bind[fs]++, BS.desc[fs] += b->count ? b->count : 1; + D->BS.bind[fs]++, D->BS.desc[fs] += b->count ? b->count : 1; } - if (++BS.draws % 20000 == 0) { - double d = (double)BS.draws; + if (++D->BS.draws % 20000 == 0) { + double d = (double)D->BS.draws; rt_log("vk_exec: per draw over %llu draws: vertex %.2f bindings %.2f descriptors, fragment %.2f bindings %.2f descriptors", - (unsigned long long)BS.draws, BS.bind[0] / d, BS.desc[0] / d, BS.bind[1] / d, BS.desc[1] / d); + (unsigned long long)D->BS.draws, D->BS.bind[0] / d, D->BS.desc[0] / d, D->BS.bind[1] / d, D->BS.desc[1] / d); rt_log("vk_exec: per draw: pushes %.3f skipped %.3f, sets written %.3f reused %.3f, texture types %.3f; " "same program and state as the draw before %.3f, also same buffers %.3f; descriptor builds skipped %.3f", - BS.push / d, BS.push_skip / d, BS.set_new / d, BS.set_reuse / d, BS.ac_miss / d, BS.same_prog / d, - BS.same_all / d, BS.bind_skip / d); + D->BS.push / d, D->BS.push_skip / d, D->BS.set_new / d, D->BS.set_reuse / d, D->BS.ac_miss / d, D->BS.same_prog / d, + D->BS.same_all / d, D->BS.bind_skip / d); } } // The bound state a draw reads, compared with the previous draw's (TOTK_BIND_STATS only). static void repeat_stats(const VkExec *ex, const VkPipeInfo *info) { + VkExecDraw *D = ex->draw; static struct { const prog_t *prog; VkPipeState pipe; @@ -1635,8 +1878,8 @@ static void repeat_stats(const VkExec *ex, const VkPipeInfo *info) { memcpy(cur.vp, ex->vp[0], sizeof cur.vp), cur.point_size = ex->point_size; size_t prog_part = offsetof(typeof(cur), info); if (!memcmp(&cur, &last, prog_part)) { - BS.same_prog++; - if (!memcmp((const char *)&cur + prog_part, (const char *)&last + prog_part, sizeof cur - prog_part)) BS.same_all++; + D->BS.same_prog++; + if (!memcmp((const char *)&cur + prog_part, (const char *)&last + prog_part, sizeof cur - prog_part)) D->BS.same_all++; } last = cur; } @@ -1655,6 +1898,7 @@ static int push_skip_on(void) { // TOTK_PUSH_SKIP=0: every graphics draw pushe // bind_resources and its helpers read beyond the pipeline info's fixed bindings and Z's stand-ins: // - the info (bindings, layouts), the program and its serial (the compiler-constant range of UBO slot 1); // - vk_texture_slots_gen, vk_texture_epoch (a slot's image, view and once-per-submit storage check), vk_sampler_gen; +// - in a chunk, the pass: a slot can resolve otherwise from one op to the next (vk_texture_slot_moves), at a SET_RT or a copy; // - vk_mem_gen (buffer and offset of a span, mirror creation and per-submission sync); // - the images in the feedback layout (the sampled layout); // - per binding: the support inputs, c0's storage table and base instance, the bound uniform and storage ranges, the @@ -1668,13 +1912,14 @@ static int push_skip_on(void) { // TOTK_PUSH_SKIP=0: every graphics draw pushe // the last build; vk_exec_draw_new_cb's memset clears it, and every build clears it until it ends. // TOTK_BIND_SKIP=0 builds every draw; =check builds every draw and stops when a draw with equal inputs gets other // writes, census counts or descriptor commands. -static int bind_skip_on(void) { - if (RS.on < 0) { +static int bind_skip_on(void) { // TOTK_BIND_SKIP: 1 skip (default), 2 check, 0 off + static int on = -1; + if (on < 0) { const char *e = cfg_value("TOTK_BIND_SKIP"); - RS.on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; - if (RS.on != 1) rt_log("vk_exec: descriptor build skip %s", RS.on ? "check" : "off"); + on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; + if (on != 1) rt_log("vk_exec: descriptor build skip %s", on ? "check" : "off"); } - return RS.on; + return on; } static void rs_add(RsRange *r, uint32_t *n, const VkExec *ex, const void *at, size_t len) { @@ -1684,11 +1929,12 @@ static void rs_add(RsRange *r, uint32_t *n, const VkExec *ex, const void *at, si } static const RsList *rs_list(const VkExec *ex, const VkPipeInfo *info) { - if (info == rs_last) return rs_last_l; - if (!rs_lists.klen) nvn_hm_init(&rs_lists, sizeof info); - RsList *l = nvn_hm_get(&rs_lists, &info); + VkExecDraw *D = ex->draw; + if (info == D->rs_last) return D->rs_last_l; + if (!D->rs_lists.klen) nvn_hm_init(&D->rs_lists, sizeof info); + RsList *l = nvn_hm_get(&D->rs_lists, &info); if (!l) { - static RsRange tmp[MAX_DESC]; + RsRange *tmp = ex->draw->rs_tmp; RsPtr ptr[RS_PTRS]; uint32_t n = 0, nd = 0, np = 0; int sup = 0, c0 = 0, ok = 1; @@ -1733,9 +1979,9 @@ static const RsList *rs_list(const VkExec *ex, const VkPipeInfo *info) { l->np = (uint8_t)np; memcpy(l->p, ptr, sizeof *ptr * np); memcpy(l->r, tmp, sizeof *tmp * n); - nvn_hm_put(&rs_lists, &info, l); + nvn_hm_put(&D->rs_lists, &info, l); } - rs_last = info, rs_last_l = l; + D->rs_last = info, D->rs_last_l = l; return l; } @@ -1743,9 +1989,10 @@ static const RsList *rs_list(const VkExec *ex, const VkPipeInfo *info) { // are compared only when a bind op ran since they were stored or the pipeline changed: otherwise they are the same. // Check mode compares them anyway and stops when the bind op generation said they could not differ but they do. static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { + VkExecDraw *D = ex->draw; const RsList *l = rs_list(ex, info); - if (!l->ok) return RS.hn = 0, RS.info = NULL, 0; - uint64_t *h = RS.h, diff = 0; + if (!l->ok) return D->RS.hn = 0, D->RS.info = NULL, 0; + uint64_t *h = D->RS.h, diff = 0; uint32_t n = 0; #define PUT(v) do { uint64_t v_ = (uint64_t)(v); diff |= h[n] ^ v_; h[n++] = v_; } while (0) PUT((uintptr_t)info); @@ -1753,7 +2000,8 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { PUT(ex->prog->serial); PUT((uint64_t)vk_texture_slots_gen() << 32 | vk_texture_epoch()); PUT((uint64_t)vk_sampler_gen() << 32 | (uint32_t)ex->nfbl); - PUT(vk_mem_gen() + vk_usnap_gen()); // both only grow + PUT(ex->chunked ? ex->pass_serial : 0); + PUT(vk_mem_gen() + vk_usnap_gen() + ex->drop_n); // all only grow in a submission; drop_n: copies a chunk passed for (int i = 0; i < ex->nfbl && i < 9; i++) PUT((uintptr_t)ex->fbl_img[i]); if (l->sup) { float in[3]; @@ -1764,7 +2012,7 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { PUT(a); PUT(ps); } - if (l->c0) PUT(c0_binst); + if (l->c0) PUT(D->c0_binst); for (int i = 0; i < l->np; i++) PUT(ubo_ptr(ex, l->p[i].st, l->p[i].res)); // The uniform overlays the draw's uniform and storage bindings read (op_update_ub): a draw of an updated block binds // other bytes after each update, every other draw is left to skip @@ -1791,12 +2039,12 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { PUT(0); } #undef PUT - diff |= n != RS.hn; - RS.hn = n; - int fast = RS.info == info && RS.gen == bind_gen; - if (fast && RS.on != 2) return !diff; + diff |= n != D->RS.hn; + D->RS.hn = n; + int fast = D->RS.info == info && D->RS.gen == D->bind_gen; + if (fast && bind_skip_on() != 2) return !diff; const uint8_t *e = (const uint8_t *)ex; - uint8_t *d = RS.body; + uint8_t *d = D->RS.body; int bd = 0; for (uint32_t i = 0; i < l->n; i++) { const RsRange *r = &l->r[i]; @@ -1805,7 +2053,7 @@ static int bind_sig(const VkExec *ex, const VkPipeInfo *info) { d += r->len; } if (fast && bd) rt_fatal("vk_exec: descriptor build skip check: bound slots changed without a bind op"); - RS.info = info, RS.gen = bind_gen; + D->RS.info = info, D->RS.gen = D->bind_gen; return !diff && !bd; } @@ -1829,24 +2077,25 @@ static void write_set(const VkPipeInfo *info, uint32_t s, VkDescriptorSet set, c // The descriptors go into one packed array d, a VkDescSlot per descriptor in binding order, which a set's update // template reads directly (VkPipeInfo.tpl); write structs are filled only when a set has no template. static void bind_resources(VkExec *ex, const VkPipeInfo *info) { - static VkWriteDescriptorSet w[MAX_BINDINGS]; - static VkDescSlot d[MAX_DESC]; - static VkBufferView bv[MAX_DESC]; + VkExecDraw *D = ex->draw; + VkWriteDescriptorSet *w = ex->draw->w; + VkDescSlot *d = ex->draw->d; + VkBufferView *bv = ex->draw->bv; if (info->n_bindings > MAX_BINDINGS) rt_fatal("vk: %u descriptor bindings exceed the replay's %d", info->n_bindings, MAX_BINDINGS); - bind_stats(info); + bind_stats(ex->draw, info); int gfx = info->bind_point == VK_PIPELINE_BIND_POINT_GRAPHICS; - int skip = gfx ? bind_skip_on() : 0, same = 0, was_valid = B.res_valid; + int skip = gfx ? bind_skip_on() : 0, same = 0, was_valid = D->B.res_valid; if (skip) { same = bind_sig(ex, info); if (same && was_valid && skip == 1) { - for (int i = 0; i < RS.ncnt; i++) vk_mem_count(RS.cnt[i].cls, RS.cnt[i].use, RS.cnt[i].bytes); - BS.bind_skip++; + for (int i = 0; i < D->RS.ncnt; i++) vk_mem_count(D->RS.cnt[i].cls, D->RS.cnt[i].use, D->RS.cnt[i].bytes); + D->BS.bind_skip++; ex->recorded++; return; } - B.res_valid = 0; // until the build below ends - if (skip == 2) memcpy(RS.pcnt, RS.cnt, sizeof RS.cnt), RS.pncnt = RS.ncnt; - RS.rec = 1, RS.ncnt = 0, RS.cnt_over = 0, RS.volatile_ = 0, RS.cmds = 0; + D->B.res_valid = 0; // until the build below ends + if (skip == 2) memcpy(D->RS.pcnt, D->RS.cnt, sizeof D->RS.cnt), D->RS.pncnt = D->RS.ncnt; + D->RS.rec = 1, D->RS.ncnt = 0, D->RS.cnt_over = 0, D->RS.volatile_ = 0, D->RS.cmds = 0; } uint32_t n_sets = info->n_sets ? info->n_sets : 1; uint32_t lo[2] = {0, 0}, hi[2] = {0, 0}, d0[2] = {0, 0}; // bindings [lo, hi) and first descriptor per set @@ -1869,19 +2118,19 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { if (b->kind == VK_PIPE_SUPPORT) { float in[3]; support_in(ex, in); - if (!B.sup_valid || memcmp(in, B.sup_in, sizeof in)) { // support_fill reads only these + if (!D->B.sup_valid || memcmp(in, D->B.sup_in, sizeof in)) { // support_fill reads only these uint8_t tmp[SUPPORT_BYTES]; support_fill(tmp, in[0], in[1], in[2]); - memcpy(B.sup_in, in, sizeof in); - if (!B.sup_valid || memcmp(tmp, B.support, SUPPORT_BYTES)) { // a new buffer only when the contents changed + memcpy(D->B.sup_in, in, sizeof in); + if (!D->B.sup_valid || memcmp(tmp, D->B.support, SUPPORT_BYTES)) { // a new buffer only when the contents changed VkBuffer sb; VkDeviceSize so; memcpy(scratch(ex, SUPPORT_BYTES, &sb, &so), tmp, SUPPORT_BYTES); - memcpy(B.support, tmp, SUPPORT_BYTES); - B.sup_buf = sb, B.sup_off = so, B.sup_valid = 1; + memcpy(D->B.support, tmp, SUPPORT_BYTES); + D->B.sup_buf = sb, D->B.sup_off = so, D->B.sup_valid = 1; } } - o->buffer = (VkDescriptorBufferInfo){B.sup_buf, B.sup_off, SUPPORT_BYTES}; + o->buffer = (VkDescriptorBufferInfo){D->B.sup_buf, D->B.sup_off, SUPPORT_BYTES}; } else if (b->kind == GFX_RES_UBO) { for (uint32_t k = 0; k < cnt; k++) o[k].buffer = ubo_info(ex, st, r->slot + (int)k); } else if (b->kind == GFX_RES_SSBO) { @@ -1897,16 +2146,16 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { } } else { kind = 1; - vk_texture_lock(); // tex_info looks up with the texture lock held + int locked = 0; // tex_info takes the texture lock for the lookups that need it for (uint32_t k = 0; k < cnt; k++) { int slot = r->slot + (int)k; uint64_t h = 0; if (img) h = slot >= 0 && slot < 32 ? ex->img[st][slot] : 0; else h = slot >= 0 && slot < 128 ? ex->tex[st][slot] : 0; - tex_info(ex, r, h, img, r->separate, &o[k].image); + tex_info(ex, r, h, img, r->separate, &o[k].image, &locked); o[k].w[2] = (uint32_t)o[k].image.imageLayout; // zero the tail padding } - vk_texture_unlock(); + if (locked) vk_texture_unlock(); } } else { rt_fatal("vk: binding %u has an unknown kind %d", (unsigned)b->binding, (int)b->kind); @@ -1926,10 +2175,10 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { // layout lacks (false VUID-vkCmdDrawIndexed-None-08114, then an access violation inside the layer). A bind at set 1 // disturbs set 0 by its canonical id, which drops the layer's push set: the push after it gets a new one. int set1_first = 0; - if (gfx && info->layout != B.ds_layout) { // another layout: sets bound before may be disturbed - B.ds_layout = info->layout; - B.ds_bound[0] = B.ds_bound[1] = VK_NULL_HANDLE; - B.ds_pushed[0] = B.ds_pushed[1] = 0; + if (gfx && info->layout != D->B.ds_layout) { // another layout: sets bound before may be disturbed + D->B.ds_layout = info->layout; + D->B.ds_bound[0] = D->B.ds_bound[1] = VK_NULL_HANDLE; + D->B.ds_pushed[0] = D->B.ds_pushed[1] = 0; set1_first = n_sets == 2 && vk_ctx()->validation; } for (uint32_t k = 0; k < n_sets; k++) { @@ -1940,24 +2189,24 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { if (set1_first) { // bound only to drop the layer's push set (comment above) VkDescriptorSet set = alloc_set(ex, info->set_layout1); vkCmdBindDescriptorSets(ex->cb, info->bind_point, info->layout, 1, 1, &set, 0, NULL); - B.ds_bound[1] = set, B.ds_pushed[1] = 0; - RS.cmds++; + D->B.ds_bound[1] = set, D->B.ds_pushed[1] = 0; + D->RS.cmds++; } continue; } if (s ? info->push1 : info->push) { uint32_t n = gfx && push_skip_on() ? dbytes : 0; - if (n && B.ds_pushed[s] && DS.pn[s] == n && !memcmp(DS.psig[s], ds, n)) { - BS.push_skip++; + if (n && D->B.ds_pushed[s] && D->DS.pn[s] == n && !memcmp(D->DS.psig[s], ds, n)) { + D->BS.push_skip++; continue; } if (info->tpl[s]) rec_push_tpl(vk_pipeline_push_tpl_fn(), ex->cb, info->tpl[s], info->layout, s, ds, dbytes); else rec_push(vk_pipeline_push_fn(), ex->cb, info->bind_point, info->layout, s, i1 - i0, w + i0); - BS.push++; - RS.cmds++; + D->BS.push++; + D->RS.cmds++; if (gfx) { - B.ds_bound[s] = VK_NULL_HANDLE, B.ds_pushed[s] = n != 0; - if (n) DS.pn[s] = n, memcpy(DS.psig[s], ds, n); + D->B.ds_bound[s] = VK_NULL_HANDLE, D->B.ds_pushed[s] = n != 0; + if (n) D->DS.pn[s] = n, memcpy(D->DS.psig[s], ds, n); } } else if (!gfx || s > 1) { VkDescriptorSet set = alloc_set(ex, s ? info->set_layout1 : info->set_layout); @@ -1965,44 +2214,44 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { vkCmdBindDescriptorSets(ex->cb, info->bind_point, info->layout, s, 1, &set, 0, NULL); } else { VkDescriptorSetLayout sl = s ? info->set_layout1 : info->set_layout; - VkDescriptorSet set = B.ds_last[s]; - if (!set || B.ds_last_layout[s] != sl || DS.n[s] != dbytes || memcmp(DS.sig[s], ds, dbytes)) { + VkDescriptorSet set = D->B.ds_last[s]; + if (!set || D->B.ds_last_layout[s] != sl || D->DS.n[s] != dbytes || memcmp(D->DS.sig[s], ds, dbytes)) { set = alloc_set(ex, sl); write_set(info, s, set, ds, w, i0, i1); - B.ds_last[s] = set, B.ds_last_layout[s] = sl, DS.n[s] = dbytes; - memcpy(DS.sig[s], ds, dbytes); - BS.set_new++; - RS.cmds++; + D->B.ds_last[s] = set, D->B.ds_last_layout[s] = sl, D->DS.n[s] = dbytes; + memcpy(D->DS.sig[s], ds, dbytes); + D->BS.set_new++; + D->RS.cmds++; } else { - BS.set_reuse++; + D->BS.set_reuse++; } - if (B.ds_bound[s] != set) { + if (D->B.ds_bound[s] != set) { vkCmdBindDescriptorSets(ex->cb, info->bind_point, info->layout, s, 1, &set, 0, NULL); - B.ds_bound[s] = set, B.ds_pushed[s] = 0; - RS.cmds++; + D->B.ds_bound[s] = set, D->B.ds_pushed[s] = 0; + D->RS.cmds++; } } } if (skip) { - RS.rec = 0; + D->RS.rec = 0; // An equal next draw records nothing when every push set holds its push (TOTK_PUSH_SKIP on) and nothing made // the build unrepeatable; allocated sets are always bound to the one written last. - int valid = !RS.volatile_ && !RS.cnt_over && RS.hn; + int valid = !D->RS.volatile_ && !D->RS.cnt_over && D->RS.hn; for (uint32_t s = 0; s < n_sets; s++) - if (hi[s] > lo[s] && (s ? info->push1 : info->push) && !B.ds_pushed[s]) valid = 0; + if (hi[s] > lo[s] && (s ? info->push1 : info->push) && !D->B.ds_pushed[s]) valid = 0; if (skip == 2) { // with the pipeline info equal (bind_sig), the packed descriptors determine every write uint32_t n = (uint32_t)(nd * sizeof *d); - int wr = n != RS.ckn || memcmp(RS.ck, d, n); - int cn = RS.ncnt != RS.pncnt || memcmp(RS.cnt, RS.pcnt, sizeof RS.cnt[0] * (size_t)RS.ncnt); - if (same && was_valid && (wr || cn || RS.cmds)) + int wr = n != D->RS.ckn || memcmp(D->RS.ck, d, n); + int cn = D->RS.ncnt != D->RS.pncnt || memcmp(D->RS.cnt, D->RS.pcnt, sizeof D->RS.cnt[0] * (size_t)D->RS.ncnt); + if (same && was_valid && (wr || cn || D->RS.cmds)) rt_fatal("vk_exec: descriptor build skip check: a draw of program %llu with the inputs of the previous one " "gets %s", (unsigned long long)ex->prog->serial, - RS.cmds ? "descriptor commands" : wr ? "other writes" : "other census counts"); - if (same && was_valid) BS.bind_skip++; // the draws a skip would have spared, compared instead - RS.ckn = n; - memcpy(RS.ck, d, n); + D->RS.cmds ? "descriptor commands" : wr ? "other writes" : "other census counts"); + if (same && was_valid) D->BS.bind_skip++; // the draws a skip would have spared, compared instead + D->RS.ckn = n; + memcpy(D->RS.ck, d, n); } - B.res_valid = valid; + D->B.res_valid = valid; } ex->recorded++; } @@ -2010,12 +2259,13 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info) { // ---- draws static void bind_pipeline(VkExec *ex, VkPipelineBindPoint bp, VkPipeline p) { + VkExecDraw *D = ex->draw; int k = bp == VK_PIPELINE_BIND_POINT_COMPUTE; - if (B.pipe[k] == p) return; + if (D->B.pipe[k] == p) return; vkCmdBindPipeline(ex->cb, bp, p); ex->recorded++; - B.pipe[k] = p; - if (!k) B.st_valid = 0; + D->B.pipe[k] = p; + if (!k) D->B.st_valid = 0; } static VkViewport draw_viewport(VkExec *ex) { @@ -2071,50 +2321,62 @@ static int vb_used_on(void) { } static void bind_vertex_buffers(VkExec *ex) { - VkBuffer vb[17]; - VkDeviceSize vo[17], vsz[17], vstr[17]; - int vcls[17] = {0}; - void *vpool[17] = {0}; + VkExecDraw *D = ex->draw; int dyn = vk_pipeline_vertex_input_dynamic(); - // Only the streams the draw's vertex input reads (B.vmask, from vk_pipeline_set_dynamic): the game keeps about ten - // bound and a draw reads a few. The others get the zero buffer, never a stale one: a buffer bound but not read must - // still be a live handle. - uint32_t used = vb_used_on() && B.st_valid ? B.vmask : VK_PIPE_ALL_BINDINGS; - // A stream whose range is the one its bound buffer was resolved from, with no pool or mirror change since - // (vk_mem_gen, read before any resolve), resolves to that buffer again: most draws change one stream or none. - uint64_t gen = vk_mem_gen(); - int same_gen = B.vb_valid && B.vgen == gen; - uint32_t res = 0; - for (int b = 0; b < 17; b++) { - vb[b] = Z.zbuf, vo[b] = 0, vsz[b] = ZERO_BYTES, vstr[b] = b < 16 ? ex->pipe.vstream[b].stride : 0; - if (b == VK_PIPE_ZERO_BINDING || !(used >> b & 1)) continue; - res |= 1u << b; - const NVNbufferRange *r = &ex->vbuf[b]; - if (same_gen && (B.vres >> b & 1) && B.vin[b].address == r->address && B.vin[b].size == r->size) { - vb[b] = B.vb[b], vo[b] = B.vo[b], vsz[b] = B.vsz[b], vcls[b] = B.vcls[b], vpool[b] = B.vpool[b]; - continue; - } - VkMemSpan sp; - if (!r->address || !vk_mem_resolve(r->address, 1, &sp)) continue; - VK_MEM_USE(sp); - uint64_t n = r->size && r->size < sp.avail ? r->size : sp.avail; - vb[b] = sp.buffer, vo[b] = sp.offset, vsz[b] = n, vcls[b] = sp.cls, vpool[b] = sp.pool; - } - memcpy(B.vcls, vcls, sizeof vcls), memcpy(B.vpool, vpool, sizeof vpool); - memcpy(B.vin, ex->vbuf, sizeof B.vin); - B.vgen = gen, B.vres = res; - if (B.vb_valid && !memcmp(B.vb, vb, sizeof vb) && !memcmp(B.vo, vo, sizeof vo) && !memcmp(B.vsz, vsz, sizeof vsz) && - (dyn || !memcmp(B.vstr, vstr, sizeof vstr))) - return; - vkCmdBindVertexBuffers2(ex->cb, 0, 17, vb, vo, vsz, dyn ? NULL : vstr); - ex->recorded++; - for (int b = 0; b < 17; b++) - if (vcls[b]) vk_mem_count(vcls[b], VK_MUSE_VB, vsz[b]); - if (rt_trace & RT_TRACE_PERF) + // Only the streams the draw's vertex input reads (D->B.vmask, from vk_pipeline_set_dynamic): the game keeps about ten + // bound and a draw reads a few. The streams it does not read keep what is bound. Giving them the zero buffer made + // nearly every draw rebind all 17 slots, since draws in a row read different streams; the NVIDIA driver's cost + // for those binds took more than the resolves saved (issue #974: RTX 5060 Laptop, 16 % fps back with vb_used=0). + // A buffer left bound was bound earlier in this command buffer, so the deferred frees keep it alive until the + // submission is done. + uint32_t used = vb_used_on() && D->B.st_valid ? D->B.vmask : VK_PIPE_ALL_BINDINGS; + uint32_t chg = 0; + if (!D->B.vb_valid) { // the command buffer's first bind sets all 17 slots, the zero buffer where nothing resolves for (int b = 0; b < 17; b++) - if (vpool[b]) vk_mem_bound_note(vpool[b]); - memcpy(B.vb, vb, sizeof vb), memcpy(B.vo, vo, sizeof vo), memcpy(B.vsz, vsz, sizeof vsz), memcpy(B.vstr, vstr, sizeof vstr); - B.vb_valid = 1; + D->B.vb[b] = Z.zbuf, D->B.vo[b] = 0, D->B.vsz[b] = ZERO_BYTES, D->B.vstr[b] = b < 16 ? ex->pipe.vstream[b].stride : 0, + D->B.vcls[b] = 0, D->B.vpool[b] = NULL; + D->B.vres = 0, chg = VK_PIPE_ALL_BINDINGS; + } + // A stream whose range is the one its bound buffer was resolved from, with no pool or mirror change since + // (vk_mem_gen, read before any resolve), resolves to that buffer again: most draws change one stream or none. A + // change of vk_mem_gen voids the resolutions of every slot, read by this draw or not. + uint64_t gen = vk_mem_gen(); + if (D->B.vgen != gen) D->B.vres = 0; + D->B.vgen = gen; + for (int b = 0; b < 17; b++) { + if (!(used >> b & 1)) continue; + VkBuffer nb = Z.zbuf; + VkDeviceSize no = 0, nsz = ZERO_BYTES, nstr = b < 16 ? ex->pipe.vstream[b].stride : 0; + int ncls = 0; + void *npool = NULL; + if (b != VK_PIPE_ZERO_BINDING) { + const NVNbufferRange *r = &ex->vbuf[b]; + if ((D->B.vres >> b & 1) && D->B.vin[b].address == r->address && D->B.vin[b].size == r->size) { + nb = D->B.vb[b], no = D->B.vo[b], nsz = D->B.vsz[b], ncls = D->B.vcls[b], npool = D->B.vpool[b]; + } else { + VkMemSpan sp; + if (r->address && vk_mem_resolve(r->address, 1, &sp)) { + VK_MEM_USE(sp); + nb = sp.buffer, no = sp.offset, nsz = r->size && r->size < sp.avail ? r->size : sp.avail; + ncls = sp.cls, npool = sp.pool; + } + D->B.vin[b] = *r, D->B.vres |= 1u << b; + } + } + if (nb != D->B.vb[b] || no != D->B.vo[b] || nsz != D->B.vsz[b] || (!dyn && nstr != D->B.vstr[b])) chg |= 1u << b; + D->B.vb[b] = nb, D->B.vo[b] = no, D->B.vsz[b] = nsz, D->B.vstr[b] = nstr, D->B.vcls[b] = ncls, D->B.vpool[b] = npool; + } + if (!chg) return; + // One bind over the changed slots; the slots between them get what they hold again. + int lo = __builtin_ctz(chg), hi = 31 - __builtin_clz(chg); + vkCmdBindVertexBuffers2(ex->cb, (uint32_t)lo, (uint32_t)(hi - lo + 1), &D->B.vb[lo], &D->B.vo[lo], &D->B.vsz[lo], + dyn ? NULL : &D->B.vstr[lo]); + ex->recorded++; + for (int b = lo; b <= hi; b++) { + if (D->B.vcls[b]) vk_mem_count(D->B.vcls[b], VK_MUSE_VB, D->B.vsz[b]); + if ((rt_trace & RT_TRACE_PERF) && D->B.vpool[b]) vk_mem_bound_note(D->B.vpool[b]); + } + D->B.vb_valid = 1; } // TOTK_PIPE_SKIP: which draws are skipped, rather than waited for, while their shaders or pipeline build. @@ -2149,14 +2411,6 @@ static int pipe_skip_mode(void) { // texture type in answers_for, and stops on a difference. // tex_serial (bumped by every texture and image bind) equal to the key's means the handles were not rebound since // they were compared, so the 1 KiB compare is skipped; a rebind to the same handles still matches by the compare. -typedef struct { - const prog_t *prog; - uint64_t serial; - uint32_t gen, tex_serial; - uint64_t tex[2][32], img[2][32]; - VkPipeState pipe; -} MemoKey; -static struct { int on, valid, has_fs; MemoKey k; const GfxShader *vs, *fs; } SM = {-1}; static void memo_key(const VkExec *ex, uint32_t gen, MemoKey *k) { k->prog = ex->prog, k->serial = ex->prog->serial, k->gen = gen, k->tex_serial = ex->tex_serial; @@ -2177,22 +2431,23 @@ static int memo_match(const VkExec *ex, uint32_t gen, MemoKey *k) { } static int shader_memo_on(void) { - if (SM.on < 0) { + static int on = -1; + if (on < 0) { const char *e = cfg_value("TOTK_SHADER_MEMO"); - SM.on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; + on = !strcmp(e, "check") ? 2 : strcmp(e, "0") != 0; } - return SM.on; + return on; } -static struct { int on, valid; VkPipeState s; const VkPipeInfo *info; } PM = {-1}; - static int pipe_memo_on(void) { - if (PM.on < 0) PM.on = cfg_bool("TOTK_PIPE_MEMO"); - return PM.on; + static int on = -1; + if (on < 0) on = cfg_bool("TOTK_PIPE_MEMO"); + return on; } // Pass, shaders, pipeline, fixed state and resources for a draw. 0: the draw is skipped. static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { + VkExecDraw *D = ex->draw; if (ex->render_disabled) return 0; cb_res(ex); dummies_init(); @@ -2209,9 +2464,10 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (!ex->prog) return warn_once(W_PROGRAM, "a draw without a bound program is skipped"), 0; int has_vs, has_fs, vs_wait = 0, fs_wait = 0, memo = shader_memo_on(); const GfxShader *vs, *fs; - uint32_t gen = memo ? vk_texture_slots_gen() : 0; // before shader_for, whose lookups may bump it - if (memo && SM.valid && memo_match(ex, gen, &SM.k)) { - vs = SM.vs, fs = SM.fs, has_vs = 1, has_fs = SM.has_fs; + // Before shader_for, whose lookups may bump it. In a chunk per pass too, as in bind_sig. + uint32_t gen = memo ? vk_texture_slots_gen() + (ex->chunked ? ex->pass_serial : 0) : 0; + if (memo && D->SM.valid && memo_match(ex, gen, &D->SM.k)) { + vs = D->SM.vs, fs = D->SM.fs, has_vs = 1, has_fs = D->SM.has_fs; if (memo == 2) { int hv, hf, wv, wf; const GfxShader *v2 = shader_for(ex, NVN_STAGE_VS, &hv, &wv, 0), *f2 = shader_for(ex, NVN_STAGE_FS, &hf, &wf, 0); @@ -2219,11 +2475,11 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { rt_fatal("vk_exec: shader memo of program %llu differs from the lookup", (unsigned long long)ex->prog->serial); } } else { - SM.valid = 0; + D->SM.valid = 0; vs = shader_for(ex, NVN_STAGE_VS, &has_vs, &vs_wait, 0); fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &fs_wait, 0); if (memo && vs && !vs_wait && !fs_wait && (!has_fs || fs)) - memo_key(ex, gen, &SM.k), SM.vs = vs, SM.fs = fs, SM.has_fs = has_fs, SM.valid = 1; + memo_key(ex, gen, &D->SM.k), D->SM.vs = vs, D->SM.fs = fs, D->SM.has_fs = has_fs, D->SM.valid = 1; } if (!has_vs) return warn_once(W_NO_VS, "a draw with a program that has no vertex stage is skipped"), 0; // A draw into a target the previous frame did not render (or into none) waits for a pending shader or pipeline: @@ -2252,16 +2508,14 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { // its pipeline: skipping it can leave a black tile. The waits share 50 ms per presented frame // (TOTK_PIPE_SKIP=fullscreen skips them instead). A one-shot draw waits without that budget. Everything else is // skipped while its pipeline is built. - static uint64_t wait_frame; - static uint64_t wait_left_ns; uint32_t wait_ms = 0; uint64_t t0 = 0; if (oneshot > 0) { wait_ms = ONESHOT_WAIT_MS; } else if (skip < 2 && count <= 6 && (uint64_t)sc.extent.width * sc.extent.height * 2 >= (uint64_t)ex->pass_w * ex->pass_h) { uint64_t f = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); - if (f != wait_frame) wait_frame = f, wait_left_ns = 50000000; - wait_ms = (uint32_t)(wait_left_ns / 1000000); + if (f != D->wait_frame) D->wait_frame = f, D->wait_left_ns = 50000000; + wait_ms = (uint32_t)(D->wait_left_ns / 1000000); if (wait_ms) t0 = kern_now_ns(); } // The previous draw's pipeline while its state repeats (PM, TOTK_PIPE_MEMO=0 turns it off): the cache key is made @@ -2270,15 +2524,19 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { VkPipeline p; const char *why = NULL; uint64_t w0 = 0; - if (pipe_memo_on() && PM.valid && !memcmp(&PM.s, &s, sizeof s)) { - info = PM.info, pending = 0; + PipeCache *pc = NULL; + if (pipe_memo_on() && !(D->PM.valid && !memcmp(&D->PM.s, &s, sizeof s))) + pc = &D->PC[nvn_hm_hash(&s, sizeof s) & (PC_N - 1)]; + if (pipe_memo_on() && (!pc || (pc->info && !memcmp(&pc->s, &s, sizeof s)))) { + info = pc ? pc->info : D->PM.info, pending = 0; p = __atomic_load_n(&info->pipeline, __ATOMIC_ACQUIRE); + if (pc) D->PM.valid = 1, D->PM.s = s, D->PM.info = info; } else { w0 = t0 ? t0 : wait_ms ? kern_now_ns() : 0; // the clock is read only for draws that may wait p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms); if (t0) { uint64_t spent = kern_now_ns() - t0; - wait_left_ns = spent < wait_left_ns ? wait_left_ns - spent : 0; + D->wait_left_ns = spent < D->wait_left_ns ? D->wait_left_ns - spent : 0; } why = oneshot > 0 ? "one-shot, shader" : wait_ms ? "full-screen" : NULL; if (pending && oneshot < 0 && !vk_exec_targets_recent(ex, skip >= 1)) { @@ -2287,8 +2545,9 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (!w0) w0 = kern_now_ns(); p = vk_pipeline_graphics_async(&s, &info, &pending, ONESHOT_WAIT_MS); } - PM.valid = !pending && p && info; - if (PM.valid) PM.s = s, PM.info = info; + D->PM.valid = !pending && p && info; + if (D->PM.valid) D->PM.s = s, D->PM.info = info; + if (D->PM.valid && pc) pc->s = s, pc->info = info; } uint64_t waited = w0 ? kern_now_ns() - w0 : 0; static int wait_logs; @@ -2303,27 +2562,32 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { if (pending) return 0; // a background worker is building it: the draw is skipped until it is ready if (!p || !info) return warn_once(W_PIPELINE, "a graphics pipeline could not be created: draws with it are skipped"), 0; bind_pipeline(ex, VK_PIPELINE_BIND_POINT_GRAPHICS, p); - if (!B.st_valid || memcmp(&B.st, &s, sizeof s)) { + if (!D->B.st_valid || memcmp(&D->B.st, &s, sizeof s)) { // TOTK_DYN_DELTA: only the dynamic states whose values changed since the last call in this command buffer. A // pipeline bind clears st_valid but not dyn_valid: every pipeline here declares the same dynamic states, which // the bind leaves as they were (the viewport, scissor and stencil values of record_fixed rely on that too). + // dyn_valid is per context and cleared with D->B for every new command buffer (vk_exec_draw_new_cb). if (DD.mode < 0) DD.mode = rt_knob("DYN_DELTA", "1"); uint32_t cmds = 1; if (RT_KNOB_ON(DD.mode)) { - B.vmask = vk_pipeline_set_dynamic_delta(ex->cb, &s, B.dyn_valid, &cmds); - B.dyn_valid = 1; + D->B.vmask = vk_pipeline_set_dynamic_delta(ex->cb, &s, D->B.dyn_valid, &cmds); + D->B.dyn_valid = 1; } else { - B.vmask = vk_pipeline_set_dynamic(ex->cb, &s); - B.dyn_valid = 0; // the delta's record is stale from here on + D->B.vmask = vk_pipeline_set_dynamic(ex->cb, &s); + D->B.dyn_valid = 0; // the delta's record is stale from here on } - DD.calls++, DD.cmds += cmds, DD.empty += !cmds; + __atomic_fetch_add(&DD.calls, 1, __ATOMIC_RELAXED), __atomic_fetch_add(&DD.cmds, cmds, __ATOMIC_RELAXED); + if (!cmds) __atomic_fetch_add(&DD.empty, 1, __ATOMIC_RELAXED); ex->recorded += cmds != 0; - B.st = s, B.st_valid = 1; - if (ex->mark && vk_exec_frame.draws == ex->mark_draws0) ex->mark->first[0] = vs, ex->mark->first[1] = fs; + D->B.st = s, D->B.st_valid = 1; + if (ex->mark && ex->fr.draws == ex->mark_draws0) ex->mark->first[0] = vs, ex->mark->first[1] = fs; } vk_dlaa_draw(ex); + int mo = ex->chunked && vk_dlaa_available(); + if (mo) plat_mutex_lock(&mo_lock); vk_mvsurvey_draw(ex); vk_mvprev_draw(ex); + if (mo) plat_mutex_unlock(&mo_lock); record_fixed(ex); repeat_stats(ex, info); bind_resources(ex, info); @@ -2512,12 +2776,12 @@ static int mo_desc(VkExec *ex, const VkPipeInfo *info, MoDraw *q) { } else if (b->kind == GFX_RES_SSBO && !(r->flags & 1)) { // read only: instance data of foliage and such for (uint32_t k = 0; k < cnt; k++) o[k].buffer = ssbo_info(ex, st, r, 0); } else if (b->kind == GFX_RES_TEX && b->type != VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER) { - vk_texture_lock(); + int locked = 0; // tex_info takes the texture lock for the lookups that need it for (uint32_t k = 0; k < cnt; k++) { int slot = r->slot + (int)k; - tex_info(ex, r, slot >= 0 && slot < 128 ? ex->tex[st][slot] : 0, 0, r->separate, &o[k].image); + tex_info(ex, r, slot >= 0 && slot < 128 ? ex->tex[st][slot] : 0, 0, r->separate, &o[k].image, &locked); } - vk_texture_unlock(); + if (locked) vk_texture_unlock(); } else { return 0; } @@ -2640,27 +2904,30 @@ static void mo_emit(VkCommandBuffer cb) { static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSize ib_off, uint64_t t0); static void mvobj_draw(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSize ib_off) { - if (MO.failed || !vk_mvprev_on() || !B.st_valid) return; + VkExecDraw *D = ex->draw; + if (MO.failed || !vk_mvprev_on() || !D->B.st_valid) return; static int timed = -1; if (timed < 0) timed = rt_knob("MVOBJ_TIMES", "0"); // TOTK_MVOBJ_TIMES=1: the summary's times (the clock reads cost) - if (!timed) return mvobj_draw_(ex, d, ib, ib_off, 0); - uint64_t t0 = kern_now_ns(); + if (ex->chunked) plat_mutex_lock(&mo_lock); + uint64_t t0 = timed ? kern_now_ns() : 0; mvobj_draw_(ex, d, ib, ib_off, t0); - MO.ns_check += kern_now_ns() - t0; + if (timed) MO.ns_check += kern_now_ns() - t0; + if (ex->chunked) plat_mutex_unlock(&mo_lock); } // TOTK_MVOBJ_BANKS=1 (test): for scene draws, the words of each other uniform bank (c3, c6, c8-c16) that changed // since the same shader's draw last frame, logged at frame 3000: where the time that moves wind and water lives static void mvobj_banks(VkExec *ex) { + VkExecDraw *D = ex->draw; static int on = -1; if (on < 0) on = rt_knob("MVOBJ_BANKS", "0"); - if (!on || !B.st_valid) return; + if (!on || !D->B.st_valid) return; uint64_t frame = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); if (frame < 2999 || frame > 3000) return; static struct { const GfxShader *vs; int bank; uint64_t frame; float w[64]; int n; } tab[4096]; static int logged; const GfxShaderRes *ur; - int nu = gfx_shader_resources(B.st.vs, GFX_RES_UBO, &ur); + int nu = gfx_shader_resources(D->B.st.vs, GFX_RES_UBO, &ur); for (int i = 0; i < nu; i++) { int bank = ur[i].slot; if (bank < 3 || bank == 4 || bank == 5 || bank == 7 || bank > 16) continue; @@ -2668,16 +2935,16 @@ static void mvobj_banks(VkExec *ex) { uint32_t n = u->size < 256 ? (uint32_t)u->size / 4 : 64; const float *now = n ? (const float *)vk_usnap_bytes(u->addr, n * 4) : NULL; if (!now) continue; - uint32_t k = (uint32_t)(((uintptr_t)B.st.vs ^ (uint64_t)bank * 0x9E3779B97F4A7C15ull) * 0xff51afd7ed558ccdull >> 52); + uint32_t k = (uint32_t)(((uintptr_t)D->B.st.vs ^ (uint64_t)bank * 0x9E3779B97F4A7C15ull) * 0xff51afd7ed558ccdull >> 52); if (frame == 2999) { - tab[k].vs = B.st.vs, tab[k].bank = bank, tab[k].frame = frame, tab[k].n = (int)n; + tab[k].vs = D->B.st.vs, tab[k].bank = bank, tab[k].frame = frame, tab[k].n = (int)n; memcpy(tab[k].w, now, n * 4); - } else if (tab[k].vs == B.st.vs && tab[k].bank == bank && tab[k].frame == 2999 && logged < 80) { + } else if (tab[k].vs == D->B.st.vs && tab[k].bank == bank && tab[k].frame == 2999 && logged < 80) { char line[700]; int o = 0, nd = 0; for (int j = 0; j < tab[k].n && j < (int)n && o < 600; j++) if (memcmp(&tab[k].w[j], &now[j], 4)) nd++, o += snprintf(line + o, sizeof line - o, " [%d] %g>%g", j, tab[k].w[j], now[j]); - if (nd) logged++, rt_log("mvbanks: vs %p c%d size %#llx: %d words%s", (const void *)B.st.vs, bank, + if (nd) logged++, rt_log("mvbanks: vs %p c%d size %#llx: %d words%s", (const void *)D->B.st.vs, bank, (unsigned long long)u->size, nd, line); tab[k].frame = 0; } @@ -2732,7 +2999,7 @@ static int mo_prev_block(int index, const p_bind_buf *u, VkDescriptorBufferInfo uint64_t c1 = __builtin_ia32_rdtsc(); MO.cyc[0] += c1 - c0, MO.ncyc++, MO.big += n; if (index >= 0 && index < 14) MO.lk[index][found ? how : MVPREV_NONE]++; - if (!found || !vk_usnap_find(pa, n, LIM.ubo_align, out)) return 0; + if (!found || !vk_usnap_find(pa, n, UINT32_MAX, LIM.ubo_align, out)) return 0; if (how == MVPREV_SELF) return 1; // bound at the same address last frame: not written again const uint8_t *map = vk_usnap_map(), *cur = vk_usnap_bytes(u->addr, n); uint64_t c2 = __builtin_ia32_rdtsc(); @@ -2770,16 +3037,17 @@ static int mo_prev_cached(int index, const p_bind_buf *u, VkDescriptorBufferInfo } static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSize ib_off, uint64_t t0) { + VkExecDraw *D = ex->draw; mvobj_banks(ex); const VkExecTarget *z = &ex->depth; - if (!z->image || z->width < DLAA_SCENE_MIN_W || z->format != VK_FORMAT_D32_SFLOAT || B.st.samples > 1 || B.st.raster_discard) + if (!z->image || z->width < DLAA_SCENE_MIN_W || z->format != VK_FORMAT_D32_SFLOAT || D->B.st.samples > 1 || D->B.st.raster_discard) return; // solid geometry only, which tests and writes the scene depth: effects, particles and screen quads write none, and // drawn again they would put their motion on the surfaces behind them (DLSS takes those from the surface) - if (!B.st.ds.dt || !B.st.ds.dw) return; + if (!D->B.st.ds.dt || !D->B.st.ds.dw) return; // a vertex shader that reads a user bank besides the camera's (c4) may move on its own const GfxShaderRes *ur; - int nu = gfx_shader_resources(B.st.vs, GFX_RES_UBO, &ur), reads = 0; + int nu = gfx_shader_resources(D->B.st.vs, GFX_RES_UBO, &ur), reads = 0; for (int i = 0; i < nu; i++) reads |= ur[i].slot >= 3 && ur[i].slot <= 16 && ur[i].slot != 4; if (!reads) return; MO.cand++; @@ -2791,8 +3059,8 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi // the variant: the translation of the same code and answers, kept per shader once found; it names the banks the // position depends on (0: none, the shader is skipped from then on) static struct { const GfxShader *of, *vs; uint32_t banks; } vmemo[1024]; - uint32_t vi = (uint32_t)((uintptr_t)B.st.vs * 0x9E3779B97F4A7C15ull >> 54); - if (vmemo[vi].of != B.st.vs) { + uint32_t vi = (uint32_t)((uintptr_t)D->B.st.vs * 0x9E3779B97F4A7C15ull >> 54); + if (vmemo[vi].of != D->B.st.vs) { int has, pending = 0; sf_mvprev = 1; const GfxShader *v = shader_for(ex, NVN_STAGE_VS, &has, &pending, 0); @@ -2802,9 +3070,9 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi return; } GfxShaderIo io[32]; - uint32_t extra = gfx_shader_outputs(v, io) & ~gfx_shader_outputs(B.st.vs, io); + uint32_t extra = gfx_shader_outputs(v, io) & ~gfx_shader_outputs(D->B.st.vs, io); uint32_t banks = extra ? gfx_shader_mvprev_banks(v) & 0x1fff8u : 0; - vmemo[vi].of = B.st.vs, vmemo[vi].vs = v, vmemo[vi].banks = banks; + vmemo[vi].of = D->B.st.vs, vmemo[vi].vs = v, vmemo[vi].banks = banks; vk_mvprev_want((banks & ~(1u << 4)) >> 3); // binding index = bank - 3 } const GfxShader *vs = vmemo[vi].vs; @@ -2845,7 +3113,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi exact = e && *e ? strtol(e, NULL, 16) : -1; } if (exact >= 0 && banks != (uint32_t)exact) return; - mo_mask_note(banks, B.st.vs); + mo_mask_note(banks, D->B.st.vs); // last frame's camera and blocks; held still when none of the position's blocks changed (most of the scenery) VkDescriptorBufferInfo cam, prevb[17]; float jit[2]; @@ -2884,7 +3152,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi } if (t0) MO.ns_part[1] += kern_now_ns() - tp, MO.n_part[1]++, tp = kern_now_ns(); GfxShaderIo io[32]; - uint32_t extra = gfx_shader_outputs(vs, io) & ~gfx_shader_outputs(B.st.vs, io); + uint32_t extra = gfx_shader_outputs(vs, io) & ~gfx_shader_outputs(D->B.st.vs, io); const GfxShaderRes *r; if (gfx_shader_resources(vs, GFX_RES_IMG, &r)) { MO.storage++; @@ -2895,7 +3163,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi MO.no_fs++; return; } - VkPipeState ms = B.st; + VkPipeState ms = D->B.st; ms.vs = vs, ms.fs = fs, ms.feedback = 0; memset(ms.color_fmt, 0, sizeof ms.color_fmt); ms.color_fmt[0] = NVN_FORMAT_RG16F; @@ -2907,7 +3175,7 @@ static void mvobj_draw_(VkExec *ex, const p_draw_idx *d, VkBuffer ib, VkDeviceSi // scene depth passes, anything nearer hides it (TOTK_MVOBJ_EQUAL=1: EQUAL, for comparison) static int eq = -1; if (eq < 0) eq = rt_knob("MVOBJ_EQUAL", "0"); - int df = B.st.ds.df; + int df = D->B.st.ds.df; df = eq ? NVN_DEPTH_FUNC_EQUAL : df == NVN_DEPTH_FUNC_LESS ? NVN_DEPTH_FUNC_LEQUAL : df == NVN_DEPTH_FUNC_GREATER ? NVN_DEPTH_FUNC_GEQUAL : df == NVN_DEPTH_FUNC_LEQUAL || df == NVN_DEPTH_FUNC_GEQUAL ? df : NVN_DEPTH_FUNC_EQUAL; @@ -3083,7 +3351,7 @@ int vk_mvobj_execute(VkExec *ex, uint64_t cam, VkImageView mv, VkImageView depth mo_emit(cb); vkCmdEndRendering(cb); // the draws left their pipeline, descriptors, buffers and dynamic state bound: the next game draw sets everything - vk_exec_draw_new_cb(); + vk_exec_draw_new_cb(ex); ex->dirty = VK_EXEC_DIRTY_ALL; ex->recorded += MQ.n; MQ.n = MQ.nst = MQ.nvb = 0, MQ.ndesc = 0, MQ.frame = 0; @@ -3101,9 +3369,10 @@ static void bind_index(VkExec *ex, const void *data, uint64_t bytes, VkIndexType } static void op_draw(VkExec *ex, const p_draw *d) { - c0_binst = (uint32_t)d->base_instance; + VkExecDraw *D = ex->draw; + D->c0_binst = (uint32_t)d->base_instance; if (d->count <= 0 || !prepare_draw(ex, d->prim, d->count, 0)) return; - vk_exec_frame.draws++; + ex->fr.draws++; uint32_t inst = d->instances > 0 ? (uint32_t)d->instances : 1, first = (uint32_t)d->first, n = (uint32_t)d->count; if (d->prim == NVN_DRAW_PRIMITIVE_QUADS && n >= 4) { // quads become two triangles each uint32_t q = n / 4; @@ -3150,12 +3419,13 @@ static int restart_for(const VkExec *ex, int prim, int itype) { } static void op_draw_idx(VkExec *ex, const p_draw_idx *d) { - c0_binst = (uint32_t)d->base_instance; + VkExecDraw *D = ex->draw; + D->c0_binst = (uint32_t)d->base_instance; static const uint32_t isize[3] = {1, 2, 4}; if (d->itype < 0 || d->itype > 2) return warn_once(W_INDEX, "index type %d is unknown: draw skipped", d->itype); int restart = restart_for(ex, d->prim, d->itype); if (d->count <= 0 || !prepare_draw(ex, d->prim, d->count, restart)) return; - vk_exec_frame.draws++; + ex->fr.draws++; uint64_t bytes = (uint64_t)d->count * isize[d->itype]; VkMemSpan sp; if (!vk_mem_resolve(d->ibuf, bytes, &sp)) return warn_once(W_INDEX, "index buffer %#llx is not backed: draws with it are skipped", (unsigned long long)d->ibuf); @@ -3190,7 +3460,8 @@ static void op_draw_idx(VkExec *ex, const p_draw_idx *d) { // dispatch), as nxnative nvn_mtl.m does. The base instance stays out of c0 (0x644): c0 cannot be patched by a transfer // inside a pass; gl_BaseInstance of the Vulkan draw is the argument's. Byte indices need the count to be widened: skipped. static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { - c0_binst = 0; + VkExecDraw *D = ex->draw; + D->c0_binst = 0; if (d->itype != NVN_INDEX_TYPE_UNSIGNED_SHORT && d->itype != NVN_INDEX_TYPE_UNSIGNED_INT) return warn_once(W_INDEX, "indirect draw with index type %d: skipped", d->itype); VkMemSpan ib, ip; @@ -3199,7 +3470,7 @@ static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { if (!vk_mem_resolve(d->ibuf, 1, &ib) || ib.offset % (d->itype == NVN_INDEX_TYPE_UNSIGNED_SHORT ? 2 : 4)) return warn_once(W_INDEX, "index buffer %#llx is not backed or not aligned: indirect draws skipped", (unsigned long long)d->ibuf); if (!prepare_draw(ex, d->prim, 0x7fffffff, restart_for(ex, d->prim, d->itype))) return; - vk_exec_frame.draws++; + ex->fr.draws++; VK_MEM_USE(ip); VK_MEM_USE(ib); vkCmdBindIndexBuffer(ex->cb, ib.buffer, ib.offset, d->itype == NVN_INDEX_TYPE_UNSIGNED_SHORT ? VK_INDEX_TYPE_UINT16 : VK_INDEX_TYPE_UINT32); @@ -3213,6 +3484,7 @@ static void op_draw_idx_ind(VkExec *ex, const p_draw_idx_ind *d) { // ---- dispatch and uniform buffer updates static void op_dispatch(VkExec *ex, const p_dispatch *d) { + VkExecDraw *D = ex->draw; if (d->x <= 0 || d->y <= 0 || d->z <= 0) return; if (!ex->prog) return warn_once(W_PROGRAM, "a dispatch without a bound program is skipped"); vk_contact_flush(ex); @@ -3226,10 +3498,10 @@ static void op_dispatch(VkExec *ex, const p_dispatch *d) { if (!p || !info) return warn_once(W_PIPELINE, "a compute pipeline could not be created: dispatches with it are skipped"); vk_exec_barrier(ex); // ends the pass; earlier writes are visible to the shader bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p); - c0_dim[0] = (uint32_t)d->x, c0_dim[1] = (uint32_t)d->y, c0_dim[2] = (uint32_t)d->z; - twin_dispatch = 1; + D->c0_dim[0] = (uint32_t)d->x, D->c0_dim[1] = (uint32_t)d->y, D->c0_dim[2] = (uint32_t)d->z; + D->twin_dispatch = 1; bind_resources(ex, info); - twin_dispatch = 0; + D->twin_dispatch = 0; VkPassMark *m = dispatch_mark(ex, cs, (uint32_t)d->x, (uint32_t)d->y, (uint32_t)d->z); vkCmdDispatch(ex->cb, (uint32_t)d->x, (uint32_t)d->y, (uint32_t)d->z); vk_pass_mark_end(ex->cb, m); @@ -3240,6 +3512,7 @@ static void op_dispatch(VkExec *ex, const p_dispatch *d) { // DispatchComputeIndirect: vkCmdDispatchIndirect from the pool buffer. The shader reads the group counts from c0 0x200, // so they are copied there on the GPU, into a c0 range no other dispatch uses, before the dispatch. static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { + VkExecDraw *D = ex->draw; if (!ex->prog) return warn_once(W_PROGRAM, "a dispatch without a bound program is skipped"); VkMemSpan ip; if (!vk_mem_resolve(d->ind, sizeof(VkDispatchIndirectCommand), &ip) || ip.offset % 4) @@ -3255,17 +3528,17 @@ static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { vk_exec_barrier(ex); VK_MEM_USE(ip); bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p); - c0_dim[0] = c0_dim[1] = c0_dim[2] = 0; - B.c0_valid[2] = 0; // a fresh c0 range: the copy below must not change one an earlier dispatch reads - twin_dispatch = 1; + D->c0_dim[0] = D->c0_dim[1] = D->c0_dim[2] = 0; + D->B.c0_valid[2] = 0; // a fresh c0 range: the copy below must not change one an earlier dispatch reads + D->twin_dispatch = 1; bind_resources(ex, info); - twin_dispatch = 0; - if (B.c0_valid[2]) { // the shader binds c0 - VkBufferCopy cp = {ip.offset, B.c0_off[2] + 0x200, sizeof(VkDispatchIndirectCommand)}; - vkCmdCopyBuffer(ex->cb, ip.buffer, B.c0_buf[2], 1, &cp); + D->twin_dispatch = 0; + if (D->B.c0_valid[2]) { // the shader binds c0 + VkBufferCopy cp = {ip.offset, D->B.c0_off[2] + 0x200, sizeof(VkDispatchIndirectCommand)}; + vkCmdCopyBuffer(ex->cb, ip.buffer, D->B.c0_buf[2], 1, &cp); ex->recorded++; vk_exec_barrier(ex); - B.c0_valid[2] = 0; // the range no longer holds B.c0[2] + D->B.c0_valid[2] = 0; // the range no longer holds B.c0[2] } VkPassMark *m = dispatch_mark(ex, cs, 0, 0, 0); vkCmdDispatchIndirect(ex->cb, ip.buffer, ip.offset); @@ -3295,6 +3568,9 @@ static void op_dispatch_ind(VkExec *ex, const p_dispatch_ind *d) { // - The descriptor build skip (bind_sig) hashes the overlay each uniform and storage binding of the draw reads // (ov_at), so a draw of an updated block binds the newest bytes and the other draws still skip. The lists are per submission and keep one entry per range (ov_add), and a binding outside their span is // answered in one comparison; when one is full, updates take the old path. +// - A chunked replay (vk_exec.c) takes the old path for every update: OV is one list in replay order, which the chunks +// recording on several threads at once neither share nor could fill in order. Its fence never matches OV.fence, so +// ub_bind, ub_written, ov_at and vk_exec_ub_flush leave its chunks alone. static int ov_on(void) { if (OV.mode < 0) OV.mode = rt_knob("UB_OVERLAY", "1"); @@ -3369,7 +3645,7 @@ static void ov_retire(uint64_t lo, uint64_t hi) { // A GPU write of another kind into [addr, addr+n) in this submission (vk_exec_xfer.c note_gpu_write, storage bindings). void vk_exec_ub_foreign(VkExec *ex, uint64_t addr, uint64_t n) { - if (OV.mode <= 0 || !n) return; + if (OV.mode <= 0 || !n || ex->chunked) return; ov_sub(ex); if (OV.nf == OV_FOREIGN) OV.f_over = 1; else OV.f[OV.nf].lo = addr, OV.f[OV.nf].hi = addr + n, OV.nf++; @@ -3458,7 +3734,7 @@ static int ub_bind(VkExec *ex, uint64_t addr, uint64_t n, uint64_t align, VkDesc VkBuffer b; VkDeviceSize o; uint8_t *dst = scratch(ex, n, &b, &o); - RS.volatile_ = 1; + ex->draw->RS.volatile_ = 1; const uint8_t *base = vk_usnap_bytes(addr, n); VkMemSpan sp; if (base) memcpy(dst, base, n); @@ -3493,7 +3769,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { cb_res(ex); VkBuffer b; VkDeviceSize o; - if (ov_on()) { + if (!ex->chunked && ov_on()) { ov_sub(ex); OV.c_update++; int here = ex->in_pass || ov_out_on(); @@ -3521,7 +3797,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { VkBufferCopy cp = {o, sp.offset, u->size}; vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp); vk_mem_gpu_wrote(&sp, u->size); - vk_usnap_drop(u->dst, u->size); + if (!ex->chunked) vk_usnap_drop(u->dst, u->size); // a chunk: the scan dropped it in its place ex->recorded++; vk_mem_count(sp.cls, VK_MUSE_GPU_WRITE, u->size); note_written(ex, u->dst, u->size); @@ -3538,12 +3814,15 @@ static void bind_handles(uint64_t *dst, int cap, const p_bind_handles *b) { memcpy(&dst[b->first], b->h, sizeof(uint64_t) * (size_t)b->n); } -void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { +// The bind ops: writes into ex alone, so the state scan (vk_exec.c) applies them too. 1 when the op wrote ex->prog, +// ubo, ssbo, tex or img (VkExecDraw.bind_gen). +int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload) { + int gen = 0; switch (op) { case OP_BIND_PROGRAM: { const p_bind_program *b = payload; ex->prog = b->prog, ex->prog_stages = b->stages; - bind_gen++; + gen = 1; break; } case OP_BIND_STATE: { @@ -3586,20 +3865,20 @@ void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { case OP_BIND_UBO: { const p_bind_buf *b = payload; if (b->stage >= 0 && b->stage < 6 && b->index >= 0 && b->index < 18) ex->ubo[b->stage][b->index] = *b; - bind_gen++; + gen = 1; break; } case OP_BIND_SSBO: { const p_bind_buf *b = payload; if (b->stage >= 0 && b->stage < 6 && b->index >= 0 && b->index < 16) ex->ssbo[b->stage][b->index] = *b; - bind_gen++; + gen = 1; break; } case OP_BIND_TEX: { const p_bind_handles *b = payload; bind_handles(b->stage >= 0 && b->stage < 6 ? ex->tex[b->stage] : NULL, 128, b); ex->tex_serial++; - bind_gen++; + gen = 1; break; } case OP_BIND_SAMP: { @@ -3611,9 +3890,21 @@ void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { const p_bind_handles *b = payload; bind_handles(b->stage >= 0 && b->stage < 6 ? ex->img[b->stage] : NULL, 32, b); ex->tex_serial++; // the shader memo's handle check (feedback_scan reads only textures: a rescan at most) - bind_gen++; + gen = 1; break; } + default: rt_fatal("vk: op %s is not a bind op", nvn_op_name(op)); + } + return gen; +} + +void vk_exec_op_draw(VkExec *ex, uint32_t op, const void *payload) { + VkExecDraw *D = ex->draw ? ex->draw : draw_new(ex); + switch (op) { + case OP_BIND_PROGRAM: case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: + case OP_BIND_UBO: case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: + if (vk_exec_bind(ex, op, payload)) D->bind_gen++; + break; case OP_UPDATE_UB: op_update_ub(ex, payload); break; case OP_DRAW: op_draw(ex, payload); break; case OP_DRAW_IDX: op_draw_idx(ex, payload); break; diff --git a/runtime/vk_exec_probe.c b/runtime/vk_exec_probe.c index 9d244cdf..825b47a6 100644 --- a/runtime/vk_exec_probe.c +++ b/runtime/vk_exec_probe.c @@ -968,6 +968,9 @@ int probe_vkexec(void) { const GfxBackend *b = &gfx_vulkan_backend; uint64_t heap = 0; CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB"); + // Out-of-memory waits (oom.c): the first two staging buffers, the first texture image and the first draw-data chunk + // fail; every part below passes through the wait or the retry that follows. + rt_oom_force(RT_OOM_STAGING, 2), rt_oom_force(RT_OOM_IMAGE, 1), rt_oom_force(RT_OOM_CHUNK, 1); uint8_t *guest = (uint8_t *)(uintptr_t)heap; uint64_t first = 0, last = 0; int w = 0, h = 0, inside = 0, xfer_bytes = 0; @@ -989,6 +992,12 @@ int probe_vkexec(void) { vk_exec_xfer_shutdown(); vk_backend_shutdown(); long errors = vk_validation_errors_total(); + int oom_left[3] = {rt_oom_forced_left(RT_OOM_STAGING), rt_oom_forced_left(RT_OOM_IMAGE), rt_oom_forced_left(RT_OOM_CHUNK)}; + if (!bad && !errors && (oom_left[0] || oom_left[1] || oom_left[2])) { + printf("probe-vkexec: FAILED (forced allocation failures never reached: staging %d, image %d, chunk %d)\n", + oom_left[0], oom_left[1], oom_left[2]); + return 1; + } if (bad || errors) { printf("probe-vkexec: FAILED (%s, %ld validation errors)\n", sync_bad ? "a sync check failed" : draw_bad ? "a draw check failed" : bad ? "a clear or xfer check failed" : "checks passed", errors); @@ -1034,5 +1043,7 @@ int probe_vkexec(void) { "finished mark is the last pass, ring records exact); the report reads:\n%s\n", marks_text); printf("probe-vkexec: empty pool part passed (a zero-byte pool at the start of another: no Vulkan objects, lookups " "there find the other pool before and after its Finalize)\n"); + printf("probe-vkexec: out-of-memory part passed (2 failed staging buffers waited for, 1 failed texture image retried, " + "1 failed draw-data chunk waited for)\n"); return 0; } diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index 9cca6405..d3b35b19 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -72,7 +72,7 @@ static void guest_written_cb(void *arg) { // The GPU wrote [addr, addr+size) of CPU-visible memory: tell the texture cache once the fence is reached. static void note_gpu_write(VkExec *ex, uint64_t addr, uint64_t size) { - vk_usnap_drop(addr, size); + if (!ex->chunked) vk_usnap_drop(addr, size); // a chunk: the scan dropped it in its place vk_exec_ub_foreign(ex, addr, size); if (nvn_gpu_only(addr)) return; Range *r = malloc(sizeof *r); @@ -129,13 +129,6 @@ static int oq_state; // 0 not set up, 1 counting available, static VkQueryControlFlags oq_flags; static volatile uint8_t oq_busy[OQ_N]; // from begin until the completion thread read and reset the slot static uint32_t oq_next; // replay: next ring slot -static int oq_counting; // replay: draws are counted (a reset since the last report, or oq_keep) -static int oq_reported; // replay: a samples-passed report since the last reset -static int oq_keep; // replay: the game reported twice without a reset: count every draw -static int oq_open = -1; // replay: the slot whose query is active in the open pass -static uint32_t *oq_seg; // replay: slots closed since the last report or reset -static uint32_t oq_nseg, oq_capseg; -static int oq_lost; // replay: the segment missed a draw (ring full) static uint64_t oq_count; // completion thread: samples since the last reset typedef struct { uint64_t addr; int reset, lost; uint32_t n; uint32_t q[]; } OqSegment; @@ -166,39 +159,49 @@ static int oq_ready(void) { } static void oq_close(VkExec *ex) { - if (oq_open < 0) return; - vkCmdEndQuery(ex->cb, oq_pool, (uint32_t)oq_open); + if (ex->oq_open < 0) return; + vkCmdEndQuery(ex->cb, oq_pool, (uint32_t)ex->oq_open); ex->recorded++; - if (oq_nseg == oq_capseg) { - oq_capseg = oq_capseg ? oq_capseg * 2 : 64; - oq_seg = realloc(oq_seg, sizeof *oq_seg * oq_capseg); - if (!oq_seg) rt_fatal("vk: out of memory"); + if (ex->oq_nseg == ex->oq_capseg) { + ex->oq_capseg = ex->oq_capseg ? ex->oq_capseg * 2 : 64; + ex->oq_seg = realloc(ex->oq_seg, sizeof *ex->oq_seg * ex->oq_capseg); + if (!ex->oq_seg) rt_fatal("vk: out of memory"); } - oq_seg[oq_nseg++] = (uint32_t)oq_open; - oq_open = -1; + ex->oq_seg[ex->oq_nseg++] = (uint32_t)ex->oq_open; + ex->oq_open = -1; } void vk_exec_oq_draw(VkExec *ex) { - if (!oq_counting || oq_open >= 0) return; - uint32_t q = oq_next % OQ_N; - if (__atomic_load_n(&oq_busy[q], __ATOMIC_ACQUIRE)) { + if (!ex->oq_counting || ex->oq_open >= 0) return; + uint32_t q = __atomic_fetch_add(&oq_next, 1, __ATOMIC_RELAXED) % OQ_N; // threads recording chunks take slots too + if (__atomic_exchange_n(&oq_busy[q], 1, __ATOMIC_ACQUIRE)) { warn_once(W_QUERY, "occlusion query ring full: draws left uncounted"); - oq_lost = 1; + ex->oq_lost = 1; return; } - oq_busy[q] = 1; - oq_next++; vkCmdBeginQuery(ex->cb, oq_pool, q, oq_flags); ex->recorded++; - oq_open = (int)q; + ex->oq_open = (int)q; } void vk_exec_oq_pass_end(VkExec *ex) { oq_close(ex); } +static void oq_segment(VkExec *ex, uint64_t addr, int reset); + +// The queries a chunk closed: the completion thread adds them to the count before the next chunk's (vk_exec.c). +void vk_exec_oq_flush(VkExec *ex) { + if (ex->oq_nseg || ex->oq_lost) oq_segment(ex, 0, 0); +} + +void vk_exec_oq_forget(VkExec *ex) { + ex->oq_counting = ex->oq_reported = ex->oq_keep = 0, ex->oq_open = -1; + ex->oq_nseg = 0, ex->oq_lost = 0; +} + void vk_exec_xfer_shutdown(void) { if (oq_state > 0) vkDestroyQueryPool(vk_ctx()->device, oq_pool, NULL); - oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_counting = oq_reported = oq_keep = 0, oq_open = -1; - oq_nseg = 0, oq_lost = 0, oq_count = 0; + oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_count = 0; + vk_exec_contexts(vk_exec_oq_forget); memset((void *)oq_busy, 0, sizeof oq_busy); } @@ -240,30 +243,34 @@ static void oq_cb(void *arg) { // Hands the segment to the completion thread with a report (addr; 0: no CPU-visible memory) or a reset. static void oq_segment(VkExec *ex, uint64_t addr, int reset) { oq_close(ex); - OqSegment *s = malloc(sizeof *s + sizeof *s->q * oq_nseg); + OqSegment *s = malloc(sizeof *s + sizeof *s->q * ex->oq_nseg); if (!s) rt_fatal("vk: out of memory"); - s->addr = addr, s->reset = reset, s->lost = oq_lost, s->n = oq_nseg; - memcpy(s->q, oq_seg, sizeof *s->q * oq_nseg); - oq_nseg = 0, oq_lost = 0; + s->addr = addr, s->reset = reset, s->lost = ex->oq_lost, s->n = ex->oq_nseg; + memcpy(s->q, ex->oq_seg, sizeof *s->q * ex->oq_nseg); + ex->oq_nseg = 0, ex->oq_lost = 0; vk_completion_add(ex->fence, oq_cb, s); } static void reset_counter(VkExec *ex, const p_reset_counter *p) { if (p->type != NVN_COUNTER_TYPE_SAMPLES_PASSED || !oq_ready()) return; oq_segment(ex, 0, 1); - oq_counting = 1, oq_reported = 0; + ex->oq_counting = 1, ex->oq_reported = 0; } // A samples-passed report: the segment goes with it (addr 0: no CPU-visible memory) and counting stops until the next // reset, unless reports came without resets before. +static void oq_report_state(VkExec *ex) { + if (ex->oq_reported && !ex->oq_keep) { + ex->oq_keep = 1; + LOG_ONCE("vk: samples passed reported twice without a reset: counting every draw from now on"); + } + ex->oq_reported = 1; + if (!ex->oq_keep) ex->oq_counting = 0; +} + static void oq_report(VkExec *ex, uint64_t addr) { oq_segment(ex, addr, 0); - if (oq_reported && !oq_keep) { - oq_keep = 1; - rt_log("vk: samples passed reported twice without a reset: counting every draw from now on"); - } - oq_reported = 1; - if (!oq_keep) oq_counting = 0; + oq_report_state(ex); } static void signal_event(VkExec *ex, const p_signal_event *p) { @@ -290,7 +297,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) { *a = p->addr; vk_completion_add(ex->fence, counter_cb, a); } - pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; + if (!ex->chunked) pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; // else the scan's } // ---- conditional rendering @@ -612,8 +619,77 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload) { case OP_RENDER_COND: ex->render_cond = *(const p_render_cond *)payload; ex->has_render_cond = 1; - ex->render_disabled = !cond_enabled(payload); + ex->render_disabled = !(ex->chunked ? ex->cond[ex->cond_i++] : cond_enabled(payload)); break; default: rt_fatal("vk: op %s is not a transfer", nvn_op_name(op)); } } + +static int revive_new(const tex_t *t, uint32_t pos) { return t && t->mem && !vk_texture_revive(t, pos); } + +int vk_exec_xfer_revive(uint32_t op, const void *payload, uint32_t pos) { + if (op == OP_COPY_T2T) { + int n = revive_new(((const p_copy_t2t *)payload)->src, pos); + return revive_new(((const p_copy_t2t *)payload)->dst, pos) | n; + } + if (op == OP_COPY_B2T || op == OP_COPY_T2B) return revive_new(((const p_copy_b2t *)payload)->tex, pos); + if (op == OP_CLEAR_TEX) return revive_new(((const p_clear_tex *)payload)->tex, pos); + return 0; +} + +uint64_t vk_exec_xfer_written(const VkExec *ex, uint32_t op, const void *payload, uint64_t *addr) { + VkMemSpan m; + if (op == OP_COPY_B2B) { + const p_copy_b2b *p = payload; + *addr = p->dst; + return vk_mem_resolve(p->src, 1, &m) && vk_mem_resolve(p->dst, 1, &m) ? p->size : 0; + } + if (op == OP_CLEAR_BUF) { + const p_clear_buf *p = payload; + *addr = p->addr; + return (p->addr | p->size) & 3 || !vk_mem_resolve(p->addr, 1, &m) ? 0 : p->size; + } + // COPY_T2B: the buffer side of copy_buffer_image from the texture descriptor (sub_resolve, buffer_side) + const p_copy_b2t *p = payload; + const tex_t *t = p->tex; + int bw, bh, bpb = t ? nvn_fmt_bpb(t->d.format, &bw, &bh) : 0; + *addr = p->src; + if (!bpb) return 0; + uint32_t level = p->has_view && p->view.set_levels ? p->view.base_level : 0; + uint32_t lw = max1((uint32_t)t->d.width >> level), lh = max1((uint32_t)t->d.height >> level); + const NVNcopyRegion *r = &p->r; + uint32_t w = r->width > 0 ? (uint32_t)r->width : r->xoffset < (int32_t)lw ? lw - (uint32_t)r->xoffset : 0; + uint32_t h = r->height > 0 ? (uint32_t)r->height : r->yoffset < (int32_t)lh ? lh - (uint32_t)r->yoffset : 0; + uint32_t nz = r->depth > 0 ? (uint32_t)r->depth : 1; + uint32_t wb = (w + (uint32_t)bw - 1) / (uint32_t)bw, hb = (h + (uint32_t)bh - 1) / (uint32_t)bh; + if (!wb || !hb) return 0; + uint64_t row = ex->copy_stride > 0 ? (uint64_t)ex->copy_stride : (uint64_t)wb * (uint64_t)bpb; + return row * hb * (nz - 1) + row * (hb - 1) + (uint64_t)wb * (uint64_t)bpb; +} + +int vk_exec_xfer_scan(VkExec *ex, uint32_t op, const void *payload) { + switch (op) { + case OP_RENDER_ENABLE: ex->render_disabled = !((const p_flag *)payload)->value; break; + case OP_RENDER_COND: { + int on = cond_enabled(payload); + ex->render_cond = *(const p_render_cond *)payload; + ex->has_render_cond = 1; + ex->render_disabled = !on; + return on; + } + case OP_RESET_COUNTER: + if (((const p_reset_counter *)payload)->type == NVN_COUNTER_TYPE_SAMPLES_PASSED && oq_ready()) + ex->oq_counting = 1, ex->oq_reported = 0; + break; + case OP_REPORT_COUNTER: { + const p_report_counter *p = payload; + int samples = p->type == NVN_COUNTER_TYPE_SAMPLES_PASSED && oq_ready(); + VkMemSpan m; + if (!nvn_gpu_only(p->addr) && !vk_mem_resolve(p->addr, sizeof(NVNcounterData), &m)) break; // the recording stops + if (samples) oq_report_state(ex); + if (!nvn_gpu_only(p->addr)) pending[pending_head++ % PENDING] = (typeof(pending[0])){p->addr, ex->fence, samples}; + break; + } + } + return 1; +} diff --git a/runtime/vk_hdr.c b/runtime/vk_hdr.c index 806ffd1b..bf968357 100644 --- a/runtime/vk_hdr.c +++ b/runtime/vk_hdr.c @@ -243,7 +243,9 @@ void vk_hdr_shader_swap(uint32_t **spirv, uint64_t *words) { int vk_hdr_track; // vk_exec_draw.c calls vk_hdr_draw: set at the first present with TOTK_HDR=1 static volatile int32_t g_active; // the swapchain is HDR10 (gfx_present.c vk_hdr_set_active) static PlatMutex rec_lock = PLAT_MUTEX_INIT; -static struct { VkImage win, src; VkFormat fmt; } recs[4]; // per presented window image: what the last draw copied +// Per presented window image: what the last draw in replay order copied, and where that draw is. Chunks of a replay +// record at once, so a draw of an earlier chunk can get here after one of a later chunk. +static struct { VkImage win, src; VkFormat fmt; uint64_t fence; uint32_t pos; } recs[4]; static int nrecs; void vk_hdr_set_active(int on) { plat_xchg32(&g_active, on); } @@ -277,7 +279,8 @@ void vk_hdr_draw(VkExec *ex, const VkPipeInfo *info) { } plat_mutex_lock(&rec_lock); for (int i = 0; i < nrecs; i++) - if (recs[i].win == t->image) recs[i].src = src, recs[i].fmt = fmt; + if (recs[i].win == t->image && (ex->fence > recs[i].fence || (ex->fence == recs[i].fence && ex->pos >= recs[i].pos))) + recs[i].src = src, recs[i].fmt = fmt, recs[i].fence = ex->fence, recs[i].pos = ex->pos; plat_mutex_unlock(&rec_lock); } diff --git a/runtime/vk_headroom.c b/runtime/vk_headroom.c index 0d5871cd..e08c2ef2 100644 --- a/runtime/vk_headroom.c +++ b/runtime/vk_headroom.c @@ -12,7 +12,9 @@ // a block per memory type is held back. // Under VRAM pressure no headroom is kept: the spare block counts against the budget that decides whether a pool gets // its VRAM mirror (vk_memory.c), so while use plus two blocks passes 85% of the budget the anchor goes and nothing new -// is allocated. +// is allocated. The same holds when the system runs short of memory (room), and for HR_HOLD_MS after an allocation +// failed anywhere (vk_headroom_release). +// The texture pool (vk_memory.c vk_texture_image) is kept ahead the same way, with its own anchor of half its block. // TOTK_VRAM_HEADROOM=0 turns it off. #include "cfg.h" @@ -22,6 +24,7 @@ #define HR_PERIOD_MS 500 #define HR_SMALL_HEAP (1ull << 30) // VMA_SMALL_HEAP_MAX_SIZE: smaller heaps use blocks of a heap eighth +#define HR_HOLD_MS 30000u static struct { int on, stop; @@ -33,6 +36,11 @@ static struct { volatile int64_t made; // blocks this thread brought in volatile int64_t other; // device memory of 32 MiB or more other threads allocated (VMA blocks, large dedicated) VmaAllocation anchor[64]; // per memory type: the half block that keeps the newest spare block from being deleted + VmaPool pool; // the texture pool, its memory type and block size, and its anchor + uint32_t pool_type; + VkDeviceSize pool_block; + VmaAllocation pool_anchor; + volatile uint64_t hold_until; // plat_now_ms before which no spare block is kept (vk_headroom_release) } H; static void VKAPI_PTR on_allocate(VmaAllocator a, uint32_t type, VkDeviceMemory m, VkDeviceSize size, void *user) { @@ -55,29 +63,36 @@ const VmaDeviceMemoryCallbacks *vk_headroom_callbacks(void) { static VkDeviceSize block_size(const VkCtx *c, uint32_t type) { VkDeviceSize heap = c->mem_props.memoryHeaps[c->mem_props.memoryTypes[type].heapIndex].size; - return heap <= HR_SMALL_HEAP ? heap / 8 : 256ull << 20; // VMA's default preferredLargeHeapBlockSize + return heap <= HR_SMALL_HEAP ? heap / 8 : c->vma_block; // VMA's preferredLargeHeapBlockSize } +// The spare block is memory the game does not use yet; where device memory takes host memory it is kept only while the +// system has four blocks to spare beyond the reserve (vk_mem_host_room). On the RX 9070 the spare blocks held 1.4 GB +// of commit. static int room(const VkCtx *c, uint32_t type, VkDeviceSize block) { VmaBudget b[VK_MAX_MEMORY_HEAPS]; vmaGetHeapBudgets(c->vma, b); const VmaBudget *h = &b[c->mem_props.memoryTypes[type].heapIndex]; - return h->budget && h->usage + 2 * block <= h->budget / 100 * 85; + return h->budget && h->usage + 2 * block <= h->budget / 100 * 85 && vk_mem_host_room() >= 4 * block; } -static void drop_anchor(const VkCtx *c, uint32_t type) { - if (H.anchor[type]) vmaFreeMemory(c->vma, H.anchor[type]); - H.anchor[type] = NULL; +static void drop_anchor(const VkCtx *c, VmaAllocation *anchor) { + if (*anchor) vmaFreeMemory(c->vma, *anchor); + *anchor = NULL; } -static void top_up(const VkCtx *c, uint32_t type) { - VkDeviceSize block = block_size(c, type); - if (!room(c, type, block)) { - drop_anchor(c, type); +// pool NULL: the default pool of the type +static void top_up(const VkCtx *c, uint32_t type, VmaPool pool, VkDeviceSize block, VmaAllocation *anchor) { + // Device-local types only: textures and mirrors grow there. Host memory types are woken by the staging pool's own + // blocks (vk_memory.c vk_staging), while their default pool held 4 MiB: on the RX 9070 the spare blocks of types 1 + // and 3 held 512 MiB of commit for nothing. + if (!(c->mem_props.memoryTypes[type].propertyFlags & VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) || + plat_now_ms() < __atomic_load_n(&H.hold_until, __ATOMIC_RELAXED) || !room(c, type, block)) { + drop_anchor(c, anchor); return; } VkMemoryRequirements req = {.size = block / 2, .alignment = 256, .memoryTypeBits = 1u << type}; - VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_UNKNOWN}; + VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_UNKNOWN, .pool = pool}; VmaAllocation a; int64_t before = H.made; if (vmaAllocateMemory(c->vma, &req, &ai, &a, NULL) != VK_SUCCESS) return; @@ -85,12 +100,13 @@ static void top_up(const VkCtx *c, uint32_t type) { vmaFreeMemory(c->vma, a); return; } - drop_anchor(c, type); - H.anchor[type] = a; + drop_anchor(c, anchor); + *anchor = a; static volatile int32_t logged; if (plat_inc32(&logged) <= 32) - rt_log("vk: headroom: a %.0f MiB block of memory type %u allocated ahead (%lld so far; %lld allocations of 32 MiB " - "or more on other threads)", (double)block / 1048576.0, type, (long long)H.made, (long long)H.other); + rt_log("vk: headroom: a %.0f MiB block of memory type %u%s allocated ahead (%lld so far; %lld allocations of " + "32 MiB or more on other threads)", (double)block / 1048576.0, type, pool ? " (texture pool)" : "", + (long long)H.made, (long long)H.other); } static void headroom_main(void *arg) { @@ -105,10 +121,15 @@ static void headroom_main(void *arg) { plat_mutex_lock(&H.lock); const VkCtx *c = H.stop ? NULL : vk_ctx(); for (uint32_t t = 0; c && t < c->mem_props.memoryTypeCount && t < 64; t++) - if (watch >> t & 1) top_up(c, t); + if (watch >> t & 1) { + top_up(c, t, NULL, block_size(c, t), &H.anchor[t]); + if (H.pool && t == H.pool_type) top_up(c, t, H.pool, H.pool_block, &H.pool_anchor); + } int stop = H.stop; - if (stop && (c = vk_ctx())) - for (uint32_t t = 0; t < 64; t++) drop_anchor(c, t); + if (stop && (c = vk_ctx())) { + for (uint32_t t = 0; t < 64; t++) drop_anchor(c, &H.anchor[t]); + drop_anchor(c, &H.pool_anchor); + } plat_mutex_unlock(&H.lock); if (stop) return; } @@ -120,11 +141,38 @@ void vk_headroom_stop(void) { plat_mutex_lock(&H.lock); H.stop = 1; const VkCtx *c = vk_ctx(); - for (uint32_t t = 0; c && t < 64; t++) drop_anchor(c, t); + for (uint32_t t = 0; c && t < 64; t++) drop_anchor(c, &H.anchor[t]); + if (c) drop_anchor(c, &H.pool_anchor); plat_mutex_unlock(&H.lock); plat_event_set(&H.ev); } +void vk_headroom_pool(VmaPool pool, uint32_t type, VkDeviceSize block) { + if (!H.started) return; + plat_mutex_lock(&H.lock); + const VkCtx *c = vk_ctx(); + if (c) drop_anchor(c, &H.pool_anchor); + H.pool = pool, H.pool_type = type, H.pool_block = block; + plat_mutex_unlock(&H.lock); +} + +// After a failed allocation: the spare blocks go at once (an empty VMA block takes the next allocation of its type or +// is given back), and none is kept for HR_HOLD_MS. H.lock is held only around VMA calls, so the caller may hold any lock +// of ours, just no VMA lock. +void vk_headroom_release(void) { + __atomic_store_n(&H.hold_until, plat_now_ms() + HR_HOLD_MS, __ATOMIC_RELAXED); + if (!H.started) return; + int n = 0; + plat_mutex_lock(&H.lock); + const VkCtx *c = H.stop ? NULL : vk_ctx(); + for (uint32_t t = 0; c && t < 64; t++) + if (H.anchor[t]) drop_anchor(c, &H.anchor[t]), n++; + if (c && H.pool_anchor) drop_anchor(c, &H.pool_anchor), n++; + plat_mutex_unlock(&H.lock); + static volatile int32_t logged; + if (n && plat_inc32(&logged) <= 8) rt_log("vk: headroom: %d spare blocks freed after a failed allocation", n); +} + void vk_headroom_start(void) { if (!H.on || H.started) return; plat_mutex_init(&H.lock); diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 3e30682f..7cd0957e 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -490,12 +490,24 @@ int vk_mem_class_of(uint64_t addr) { // ---------------------------------------------------------------- census static volatile int64_t traffic[VK_MCLS_N][VK_MUSE_N]; +static _Thread_local int64_t (*count_to)[VK_MUSE_N]; + // Every caller is the replay (vk_exec*, under vk_sync_submit's lock), about ten adds per draw: one writer at a time, so -// a plain add, not a locked one. An add racing the census's reset may be lost. +// a plain add, not a locked one. An add racing the census's reset may be lost. Threads recording chunks in parallel +// count into their own arrays (vk_mem_count_local), added with atomics once per chunk. void vk_mem_count(int cls, int use, uint64_t bytes) { - if ((unsigned)cls < VK_MCLS_N && (unsigned)use < VK_MUSE_N) - __atomic_store_n(&traffic[cls][use], __atomic_load_n(&traffic[cls][use], __ATOMIC_RELAXED) + (int64_t)bytes, - __ATOMIC_RELAXED); + if ((unsigned)cls >= VK_MCLS_N || (unsigned)use >= VK_MUSE_N) return; + if (count_to) count_to[cls][use] += (int64_t)bytes; + else __atomic_store_n(&traffic[cls][use], __atomic_load_n(&traffic[cls][use], __ATOMIC_RELAXED) + (int64_t)bytes, + __ATOMIC_RELAXED); +} + +void vk_mem_count_local(int64_t (*acc)[VK_MUSE_N]) { count_to = acc; } + +void vk_mem_count_flush(int64_t (*acc)[VK_MUSE_N]) { + for (int c = 0; c < VK_MCLS_N; c++) + for (int u = 0; u < VK_MUSE_N; u++) + if (acc[c][u]) __atomic_fetch_add(&traffic[c][u], acc[c][u], __ATOMIC_RELAXED), acc[c][u] = 0; } static void flag_names(uint32_t f, char *buf, size_t n) { @@ -559,6 +571,25 @@ void vk_mem_census(uint64_t frame) { n_import_failures, (unsigned long long)count[POOL_PHYS], total[POOL_PHYS] / 1048576.0, (unsigned long long)count[POOL_VIRT], total[POOL_VIRT] / 1048576.0, (unsigned long long)__atomic_load_n(&n_cross_defer, __ATOMIC_RELAXED)); + // The baseline for the commit a crash report prints: a crash far above it points at growth during play. + double proc, used, limit; + plat_commit_info(&proc, &used, &limit); + uint64_t room = vk_mem_host_room(); + rt_log("mem: commit at frame %llu: this process %.0f MB, the system %.0f of %.0f MB; room for device memory %s%.0f MiB", + (unsigned long long)frame, proc, used, limit, room == UINT64_MAX ? "not counted " : "", + room == UINT64_MAX ? 0.0 : room / 1048576.0); + // Per heap, what VMA holds (blocks, and the allocations inside them) beside what the driver counts for the process: + // usage above VMA's blocks is the driver's own memory (pipelines, command buffers, descriptor pools). + VkCtx *vc = vk_ctx(); + if (vc && vc->vma) { + VmaBudget vb[VK_MAX_MEMORY_HEAPS]; + vmaGetHeapBudgets(vc->vma, vb); + for (uint32_t h = 0; h < vc->mem_props.memoryHeapCount && h < VK_MAX_MEMORY_HEAPS; h++) + rt_log("mem: heap %u (flags %#x) at frame %llu: vma %u blocks %.0f MiB, %u allocations %.0f MiB; driver usage " + "%.0f of %.0f MiB", h, vc->mem_props.memoryHeaps[h].flags, (unsigned long long)frame, + vb[h].statistics.blockCount, vb[h].statistics.blockBytes / 1048576.0, vb[h].statistics.allocationCount, + vb[h].statistics.allocationBytes / 1048576.0, vb[h].usage / 1048576.0, vb[h].budget / 1048576.0); + } static const char *const kinds[] = {"", "import", "physical", "virtual"}; for (int j = 0; j < ng; j++) { char f[160]; @@ -590,6 +621,79 @@ static uint64_t vram_limit(void) { return (uint64_t)limit; } +// Device memory that takes host memory. A UMA GPU's memory is the machine's RAM. On Windows the AMD driver charges +// device-local allocations to the process commit one to one (RX 9070: VRAM use and private bytes grew by the same +// 1164, 256 and 256 MiB), so on a 16 GB machine the commit limit is reached while the VRAM budget shows gigabytes free, +// and the driver then fails inside a call (#285: std::bad_alloc in amdvlk64.dll under gpl_link). Whether a discrete +// GPU's driver does so is measured once (vk_mem_host_probe), not assumed per vendor: an RTX 5080 log (#398) showed +// 10.4 GB of process commit at frame 1200, about what the RX 9070 holds with its VRAM counted. Linux overcommits, so +// there only a UMA GPU's physical memory counts (plat_mem_room). +enum { HB_NONE, HB_COMMIT, HB_UMA, HB_PROBE }; +static int host_backed; +static uint64_t room_reserve; +static volatile uint64_t room_cached, room_ms; + +uint64_t vk_mem_host_room(void) { + if (!host_backed) return UINT64_MAX; + uint64_t now = plat_now_ms(), at = __atomic_load_n(&room_ms, __ATOMIC_ACQUIRE); + if (at && now - at < 250) return __atomic_load_n(&room_cached, __ATOMIC_RELAXED); // a query costs a system call + uint64_t commit, phys; + plat_mem_room(&commit, &phys); + uint64_t r = host_backed == HB_UMA && phys < commit ? phys : commit; + r = r == UINT64_MAX ? r : r > room_reserve ? r - room_reserve : 0; + __atomic_store_n(&room_cached, r, __ATOMIC_RELAXED); + __atomic_store_n(&room_ms, now | 1, __ATOMIC_RELEASE); + return r; +} + +int vk_mem_uma(void) { return host_backed == HB_UMA; } + +// A discrete GPU: one device-local allocation of HB_PROBE_SIZE, and the process commit before and after it. The driver +// charges it at allocation or not at all as far as this sees; one that charged only at first use would read as not +// counted, as before the probe. Another thread's allocation in between could only add, and half the size is the line. +#define HB_PROBE_SIZE (128ull << 20) +void vk_mem_host_probe(void) { + VkCtx *c = vk_ctx(); + if (!c || host_backed != HB_PROBE) return; + host_backed = HB_NONE; + uint64_t commit, phys; + plat_mem_room(&commit, &phys); + if (commit == UINT64_MAX) return; // no commit limit (Linux) + uint32_t type = UINT32_MAX; + for (uint32_t i = 0; i < c->mem_props.memoryTypeCount && type == UINT32_MAX; i++) + if ((c->mem_props.memoryTypes[i].propertyFlags & (VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT)) == + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) + type = i; + VkMemoryAllocateInfo ai = {.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .allocationSize = HB_PROBE_SIZE, + .memoryTypeIndex = type}; + uint64_t r0, p0, r1, p1; + VkDeviceMemory m; + plat_process_memory(&r0, &p0); + if (type == UINT32_MAX || !p0 || vkAllocateMemory(c->device, &ai, NULL, &m) != VK_SUCCESS) { + rt_log("mem: device memory against system memory: not measured (no device-local type or allocation failed): not counted"); + return; + } + plat_process_memory(&r1, &p1); + vkFreeMemory(c->device, m, NULL); + int64_t d = (int64_t)(p1 - p0); + host_backed = d >= (int64_t)HB_PROBE_SIZE / 2 ? HB_COMMIT : HB_NONE; + rt_log("mem: device memory against system memory: %s (a %llu MiB device-local allocation took %.0f MiB of process " + "commit), reserve %llu MiB (TOTK_MEM_RESERVE_MIB)", host_backed == HB_COMMIT ? "commit" : "not counted", + (unsigned long long)(HB_PROBE_SIZE >> 20), d / 1048576.0, (unsigned long long)(room_reserve >> 20)); +} + +// The budget is at most the use plus what the system can still give, so the paths that react to VRAM pressure (the +// mirror refusal, released textures, the headroom block) react to a system short of memory too. +static void host_cap(uint64_t used, uint64_t *budget) { + uint64_t room = vk_mem_host_room(); + if (room == UINT64_MAX || used + room >= *budget) return; + *budget = used + room ? used + room : 1; + static volatile int32_t logged; + if (!plat_xchg32(&logged, 1)) + rt_log("mem: system memory limits the device-local budget: %.0f MiB in use, %.0f MiB more beyond a reserve of " + "%.0f MiB (logged once)", used / 1048576.0, room / 1048576.0, room_reserve / 1048576.0); +} + int vk_mem_vram(uint64_t *used, uint64_t *budget, int *exact) { VkCtx *c = vk_ctx(); *used = *budget = 0, *exact = 0; @@ -605,6 +709,7 @@ int vk_mem_vram(uint64_t *used, uint64_t *budget, int *exact) { *used += b.heapUsage[h], *budget += b.heapBudget[h]; *exact = 1; if (limit && *budget > limit) *budget = limit; + if (*budget) host_cap(*used, budget); return *budget != 0; } VmaBudget vb[VK_MAX_MEMORY_HEAPS]; @@ -613,6 +718,7 @@ int vk_mem_vram(uint64_t *used, uint64_t *budget, int *exact) { if (mp->memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT) *used += vb[h].statistics.blockBytes, *budget += mp->memoryHeaps[h].size; if (limit && *budget > limit) *budget = limit; + if (*budget) host_cap(*used, budget); return *budget != 0; } @@ -953,9 +1059,41 @@ static void shadow_copy(PoolRec *r, uint64_t a, uint64_t b, int i) { s_copied += b - a; } +// TOTK_SHADOW_SNAP (default 1 on a native host, 0 under Wine, where it has not been measured): a pool's sync takes its written pages from the per-submission write-watch snapshots +// (snap_regions, shared with the texture checks): each 2 MiB region is queried at most once per guest submission, however +// many pools share it and however many parts of the submission touch them. The GPU then reads a pool as of the first +// part that syncs it, nearer the submit call, the moment the game means (vk_usnap.c). 0 queries each pool at every +// sync, as before. Measured on a Steam Deck at Lookout Landing: 716 pool syncs and 729 queries a frame took 4.8 ms of +// the replay thread to copy 2.3 MiB, the query walk costing about 4 us per MiB there; with the snapshots, about 58 +// queries and 1.7 ms for the same bytes. +static int shadow_snap = -1; +static void shadow_copy_newer(PoolRec *r, uint32_t since); + // shadow_lock and gen_lock held: stamps the pool's written pages and copies those newer than its shadow. static void shadow_sync_pool(PoolRec *r) { uint64_t lo = r->import_lo, hi = r->import_hi; + if (shadow_snap < 0) { + shadow_snap = rt_knob("SHADOW_SNAP", plat_is_wine() ? "0" : "1"); + rt_log("mem: shadow sync: %s (TOTK_SHADOW_SNAP%s)", + shadow_snap ? "per-submission region snapshots" : "a query per pool at every part", + plat_is_wine() ? "; off by default under Wine" : ""); + } + uint32_t epoch = shadow_snap ? vk_texture_epoch() : 0; + uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT; + if (epoch && r1 < SNAP_REGIONS) { + s_queries += snap_regions(r0, r1, epoch); // stamps what the snapshot found, if the region was not seen yet + int untracked = 0; + for (uint64_t k = r0; k <= r1; k++) untracked |= snap_untracked[k]; + uint32_t since = r->sgen; + r->sgen = gen_now; + if (untracked) { // nothing tells which pages changed: all of them + s_untracked++; + shadow_copy(r, lo, hi, 0); + return; + } + shadow_copy_newer(r, since); + return; + } uint32_t g = ++gen_now; int tracked = 1; for (uint64_t a = lo; a < hi;) { // one query per run of regions that are not frozen @@ -974,6 +1112,12 @@ static void shadow_sync_pool(PoolRec *r) { shadow_copy(r, lo, hi, 0); return; } + shadow_copy_newer(r, since); +} + +// shadow_lock and gen_lock held: copies the pool's pages stamped after since into its shadow, in runs. +static void shadow_copy_newer(PoolRec *r, uint32_t since) { + uint64_t lo = r->import_lo, hi = r->import_hi; uint64_t run = 0; for (uint64_t k = lo >> REG_SHIFT; k <= (hi - 1) >> REG_SHIFT; k++) { uint64_t a = k << REG_SHIFT, b = a + (1ull << REG_SHIFT); @@ -1212,6 +1356,10 @@ static VkBufferCopy *mregions; // replay: vkCmdCopyBuffer regions static int cregions; static PlatMutex flush_lock = PLAT_MUTEX_INIT; static int flush_cap = FLUSH_RANGES; // ranges a pool keeps; 1 with TOTK_FLUSH_SPAN=1 +// The first uses of a submission's mirrors, the copies they queue, the GPU writes noted and their snapshots (pend, wbl, snl, mrep): threads +// recording chunks of one submission take it. A mirror synced in this submission is read without it (vk_mem_use): its +// acc_sub is stored last. Before pool_lock, gen_lock and flush_lock. +static PlatMutex mirror_lock = PLAT_MUTEX_INIT; typedef struct { uint64_t subs, syncs, upload_bytes, upload_copies, first_n, first_bytes, wb_bytes, wb_copies, wb_ranges; @@ -1336,16 +1484,21 @@ static void refuse(PoolRec *r, int why, const char *what) { static void mirror_create(PoolRec *r) { VkCtx *c = vk_ctx(); uint64_t n = r->import_hi - r->import_lo; - r->mstate = MS_REFUSED; __atomic_add_fetch(&mirror_gen, 1, __ATOMIC_RELEASE); - if (r->kind != POOL_IMPORT || !r->buf || !n) return; + if (r->kind != POOL_IMPORT || !r->buf || !n) { + __atomic_store_n(&r->mstate, MS_REFUSED, __ATOMIC_RELEASE); + return; + } plat_mutex_lock_shared(&pool_lock); int i = pidx_upper(&pool_index, r->lo) - 1; while (i >= 0 && pool_index.e[i].buf != r) i--; int alone = i >= 0 && (i == 0 || pool_index.maxhi[i - 1] <= pool_index.e[i].lo) && (i + 1 == pool_index.n || pool_index.e[i + 1].lo >= pool_index.e[i].hi); plat_mutex_unlock_shared(&pool_lock); - if (i < 0) return; // finalized meanwhile + if (i < 0) { // finalized meanwhile + __atomic_store_n(&r->mstate, MS_REFUSED, __ATOMIC_RELEASE); + return; + } if (!alone) return refuse(r, MR_OVERLAP, "it overlaps another pool"); uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT; if (r1 >= SNAP_REGIONS) return refuse(r, MR_UNTRACKED, "outside the write-watched address space"); @@ -1383,9 +1536,9 @@ static void mirror_create(PoolRec *r) { mlist[at] = r; nm++; plat_mutex_unlock(&pool_lock); - r->acc_sub = msub; - r->mstate = MS_ON; range_add(&pend, &npend, &cpend, r, 0, n); + __atomic_store_n(&r->acc_sub, msub, __ATOMIC_RELEASE); + __atomic_store_n(&r->mstate, MS_ON, __ATOMIC_RELEASE); mrep.first_n++, mrep.first_bytes += n; plat_inc64(&m_live_n); plat_add64(&m_live_bytes, (int64_t)n); @@ -1426,7 +1579,6 @@ static void pool_retire(void *arg) { // Replay thread, the first access of this submission: queues the copies of the pages written since the last sync. static void mirror_sync(PoolRec *r) { uint64_t t0 = qpc(); - r->acc_sub = msub; uint64_t lo = r->import_lo, hi = r->import_hi; uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT; int excl = 0; @@ -1475,21 +1627,40 @@ static void mirror_sync(PoolRec *r) { } mrep.syncs++; mrep.sync_t += qpc() - t0; + __atomic_store_n(&r->acc_sub, msub, __ATOMIC_RELEASE); +} + +// 1 when r is settled for this submission without mirror_lock: refused, or mirrored and synced (*on). +static int mirror_settled(PoolRec *r, int *on) { + int st = __atomic_load_n(&r->mstate, __ATOMIC_ACQUIRE); + *on = st == MS_ON; + return st == MS_REFUSED || (st == MS_ON && __atomic_load_n(&r->acc_sub, __ATOMIC_ACQUIRE) == msub); } void vk_mem_use(VkMemSpan *sp) { PoolRec *r = sp->pool; - if (r->mstate == MS_NONE) mirror_create(r); - if (r->mstate != MS_ON) return; - if (r->acc_sub != msub) mirror_sync(r); - sp->buffer = r->mbuf; + int on; + if (!mirror_settled(r, &on)) { + plat_mutex_lock(&mirror_lock); + if (r->mstate == MS_NONE) mirror_create(r); + on = r->mstate == MS_ON; + if (on && r->acc_sub != msub) mirror_sync(r); + plat_mutex_unlock(&mirror_lock); + } + if (on) sp->buffer = r->mbuf; } void vk_mem_pick(VkMemSpan *sp) { PoolRec *r = sp->pool; - if (!r || r->mstate != MS_ON) return; - if (r->acc_sub != msub) mirror_sync(r); - sp->buffer = r->mbuf; + int on; + if (!r) return; + if (!mirror_settled(r, &on)) { + plat_mutex_lock(&mirror_lock); + on = r->mstate == MS_ON; + if (on && r->acc_sub != msub) mirror_sync(r); + plat_mutex_unlock(&mirror_lock); + } + if (on) sp->buffer = r->mbuf; } int vk_mem_mirror_probe(uint64_t addr) { @@ -1753,10 +1924,14 @@ static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) { PoolRec *r = sp->pool; if (r && size) gpu_writes(r, sp->offset, size); if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size) return; + plat_mutex_lock(&mirror_lock); // A binding to the end of its pool (no size) can span hundreds of MiB: those stay copies. - if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) + if (shader && !(sp->offset & 3) && !(size & 3) && size <= MERGE_MAX && merging() && snap_add(r, sp->offset, size)) { + plat_mutex_unlock(&mirror_lock); return; + } range_add(&wbl, &nwbl, &cwbl, r, sp->offset, size); + plat_mutex_unlock(&mirror_lock); } void vk_mem_gpu_wrote(const VkMemSpan *sp, uint64_t size) { gpu_wrote(sp, size, 0); } void vk_mem_gpu_wrote_shader(const VkMemSpan *sp, uint64_t size) { gpu_wrote(sp, size, 1); } @@ -1888,10 +2063,19 @@ void vk_mem_mirror_begin(void) { mrep.presnap_t += qpc() - t0; } -int vk_mem_mirror_pending(void) { return npend || snl_copied < nsnl; } +int vk_mem_mirror_pending(void) { + plat_mutex_lock(&mirror_lock); + int n = npend || snl_copied < nsnl; + plat_mutex_unlock(&mirror_lock); + return n; +} void vk_mem_mirror_prologue(VkCommandBuffer cb) { - if (!npend && snl_copied == nsnl) return; + plat_mutex_lock(&mirror_lock); + if (!npend && snl_copied == nsnl) { + plat_mutex_unlock(&mirror_lock); + return; + } mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); copy_ranges(cb, pend, npend, 1, &mrep.upload_bytes, &mrep.upload_copies); @@ -1904,6 +2088,7 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb) { VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); npend = 0; mrep.prologues++; + plat_mutex_unlock(&mirror_lock); } // ---- verify mode @@ -2135,6 +2320,9 @@ void vk_mem_class_log(const VkPhysicalDeviceProperties *p, const VkPhysicalDevic else if (host_heap < 0) cls = "uma", why = "every heap is device-local"; else cls = "discrete", why = "device-local heap separate from host memory"; int discrete = cls[0] == 'd'; + host_backed = !discrete ? HB_UMA : HB_PROBE; // a discrete GPU is measured once the device exists + const char *rs = cfg_get("TOTK_MEM_RESERVE_MIB"); + room_reserve = (rs && *rs ? strtoull(rs, NULL, 10) : 1536) << 20; // Windows, other programs, our own heap char line[2048]; size_t k = (size_t)snprintf(line, sizeof line, "mem: class %s (%s; device type %d %s, %s)", cls, why, t, tn, p->deviceName); for (uint32_t h = 0; h < m->memoryHeapCount; h++) @@ -2154,6 +2342,9 @@ void vk_mem_class_log(const VkPhysicalDeviceProperties *p, const VkPhysicalDevic f & VK_MEMORY_PROPERTY_LAZILY_ALLOCATED_BIT ? " LAZY" : ""); } rt_log("%s", line); + if (host_backed == HB_UMA) + rt_log("mem: device memory against system memory: commit and physical memory (uma), reserve %llu MiB " + "(TOTK_MEM_RESERVE_MIB)", (unsigned long long)(room_reserve >> 20)); mirror_mode(discrete); } @@ -2291,6 +2482,20 @@ void vk_upload_end(void) { plat_rmutex_unlock(&cmd_lock); } static VmaPool stg_pool; static int stg_pool_tried; +typedef struct { const VkBufferCreateInfo *bi; const VmaAllocationCreateInfo *aci; VkBuffer b; VmaAllocation a; + VmaAllocationInfo info; VkResult r; } StgTry; +static int staging_try(void *arg) { + StgTry *t = arg; + vk_mem_collect(); // upload batches and deferred objects the GPU finished meanwhile + t->r = rt_oom_forced(RT_OOM_STAGING) ? VK_ERROR_OUT_OF_DEVICE_MEMORY + : vmaCreateBuffer(vk_ctx()->vma, t->bi, t->aci, &t->b, &t->a, &t->info); + return t->r == VK_SUCCESS; +} + +// A failed staging buffer waits for memory (rt_oom_wait) before it ends the game: the headroom blocks go and the GPU's +// finished batches are freed at once, released textures at the next submit (vk_oom_release). The caller may hold the +// texture lock and cmd_lock (vk_upload_begin), both of which vk_oom_release may take after them, and no VMA lock. The open +// batch is not submitted: its caller records into it after this returns. void *vk_staging(VkDeviceSize size, VkBuffer *buf, VmaAllocation *alloc) { VkCtx *c = vk_ctx(); VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size ? size : 16, @@ -2314,11 +2519,22 @@ void *vk_staging(VkDeviceSize size, VkBuffer *buf, VmaAllocation *alloc) { plat_rmutex_unlock(&cmd_lock); } if (stg_pool && bi.size <= STG_BLOCK / 2) aci.pool = stg_pool; - VmaAllocationInfo info; - VkBuffer b; - VmaAllocation a; - VkResult r = vmaCreateBuffer(c->vma, &bi, &aci, &b, &a, &info); - if (r != VK_SUCCESS) rt_fatal("vk: staging buffer of %llu bytes failed (%d)", (unsigned long long)size, (int)r); + StgTry st = {.bi = &bi, .aci = &aci}; + st.r = rt_oom_forced(RT_OOM_STAGING) ? VK_ERROR_OUT_OF_DEVICE_MEMORY + : vmaCreateBuffer(c->vma, &bi, &aci, &st.b, &st.a, &st.info); + if (st.r != VK_SUCCESS) { + VkResult first = st.r; + if (first == VK_ERROR_OUT_OF_HOST_MEMORY || first == VK_ERROR_OUT_OF_DEVICE_MEMORY) { + vk_oom_release(); + rt_oom_wait("vk: staging buffer", bi.size, staging_try, &st); + } + if (st.r != VK_SUCCESS) + rt_fatal("vk: staging buffer of %llu bytes failed (%d, first %d)", (unsigned long long)size, (int)st.r, + (int)first); + } + VkBuffer b = st.b; + VmaAllocation a = st.a; + VmaAllocationInfo info = st.info; *buf = b; if (alloc) *alloc = a; // the caller destroys it (vmaDestroyBuffer) after vk_upload_flush(1) else { @@ -2373,6 +2589,66 @@ VkResult vk_scratch_buffer(VkDeviceSize size, VkBuffer *buf, VmaAllocation *allo VkDeviceSize vk_upload_held(void) { return open_active ? open_b.held : 0; } +// Own memory of the game's textures (vk_texture.c create_tex): a pool of its own with blocks of TOTK_TEX_BLOCK_MIB +// (default 64; 0 puts textures back in the default pools). In the default pools textures shared 256 MiB blocks with the +// mirrors and other buffers, and a block went back to the driver only once all of them were gone: on the RX 9070, +// after 4.5 minutes of roaming the field, 3 of the 9 device-local blocks were 13-27% full, held by 87-223 small +// textures and buffers each, and the full ones had hundreds of free gaps under 1 MiB, so blocks held 800 MiB more than +// their allocations. Where device memory takes process commit (vk_mem_host_probe) that commit backs nothing. A smaller +// block, holding textures only, empties sooner and wastes less when it does not. With the pool the same 10-minute route +// ended with 807 MiB of blocks beyond their allocations instead of 1037, the texture pool's 22 blocks mostly 75-100% +// full, and the slow frames did not grow. +// An image of a quarter block or more gets memory of its own (a dedicated allocation) that goes back to the driver +// with the image; so do images the driver prefers that for, and images the pool's memory type does not take. VMA +// checks neither for a custom pool (VmaAllocator_T::AllocateMemory takes the pool's type as it is), so the image's +// requirements are read here first. +static VmaPool tex_pool; +static uint32_t tex_type; +static VkDeviceSize tex_block; +static int tex_pool_tried; + +static void tex_pool_init(VkCtx *c, const VkImageCreateInfo *ci) { + plat_once(&cmd_once, cmd_init, NULL); + plat_rmutex_lock(&cmd_lock); + if (!tex_pool_tried) { + tex_block = (VkDeviceSize)cfg_int("TOTK_TEX_BLOCK_MIB") << 20; // schema default 64 + VmaAllocationCreateInfo aci = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; + VmaPoolCreateInfo pi = {.blockSize = tex_block}; + if (tex_block && vmaFindMemoryTypeIndexForImageInfo(c->vma, ci, &aci, &tex_type) == VK_SUCCESS) { + pi.memoryTypeIndex = tex_type; + if (vmaCreatePool(c->vma, &pi, &tex_pool) != VK_SUCCESS) tex_pool = NULL; + } + if (tex_pool) { + vk_headroom_pool(tex_pool, tex_type, tex_block); + rt_log("vk: texture pool: blocks of %llu MiB in memory type %u; images of %llu MiB or more get memory of " + "their own (TOTK_TEX_BLOCK_MIB)", (unsigned long long)(tex_block >> 20), tex_type, + (unsigned long long)(tex_block >> 22)); + } else { + rt_log("vk: texture pool %s: textures go to the default pools (TOTK_TEX_BLOCK_MIB)", + tex_block ? "unavailable" : "off"); + } + __atomic_store_n(&tex_pool_tried, 1, __ATOMIC_RELEASE); + } + plat_rmutex_unlock(&cmd_lock); +} + +VkResult vk_texture_image(const VkImageCreateInfo *ci, VkImage *img, VmaAllocation *alloc) { + VkCtx *c = vk_ctx(); + VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; + if (!__atomic_load_n(&tex_pool_tried, __ATOMIC_ACQUIRE)) tex_pool_init(c, ci); + if (tex_pool) { + VkDeviceImageMemoryRequirements q = {.sType = VK_STRUCTURE_TYPE_DEVICE_IMAGE_MEMORY_REQUIREMENTS, .pCreateInfo = ci}; + VkMemoryDedicatedRequirements dr = {.sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_REQUIREMENTS}; + VkMemoryRequirements2 mr = {.sType = VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2, .pNext = &dr}; + vkGetDeviceImageMemoryRequirements(c->device, &q, &mr); + if (dr.requiresDedicatedAllocation || dr.prefersDedicatedAllocation || mr.memoryRequirements.size >= tex_block / 4) + ai.flags = VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT; + else if (mr.memoryRequirements.memoryTypeBits >> tex_type & 1) + ai = (VmaAllocationCreateInfo){.usage = VMA_MEMORY_USAGE_UNKNOWN, .pool = tex_pool}; // the pool's type + } + return vmaCreateImage(c->vma, ci, &ai, img, alloc, NULL); +} + // At device creation: the staging and scratch pools with their kept blocks, made through a first small buffer each // (freed at once), so the first world frame does not allocate them (staging was first used there once the GPU took // the early texture uploads). @@ -2497,6 +2773,15 @@ void vk_mem_collect(void) { plat_rmutex_unlock(&cmd_lock); } +// Lock order: the texture lock, then cmd_lock (create_tex records under the first), then the headroom lock, which is +// held only around VMA calls. A caller holding cmd_lock takes it again (recursive) and keeps the fence thread's own +// collect (vk_sync.c) out meanwhile, so this collects for it. +void vk_oom_release(void) { + rt_oom_request(); + vk_headroom_release(); + vk_mem_collect(); +} + void vk_mem_stats(VkMemStats *s) { memset(s, 0, sizeof *s); plat_mutex_lock_shared(&pool_lock); @@ -2539,6 +2824,10 @@ void vk_mem_shutdown(void) { if (scr_pool) vmaDestroyPool(c->vma, scr_pool); // deferred scratch buffers went with collect_locked above scr_pool = NULL; scr_pool_tried = 0; + vk_headroom_pool(NULL, 0, 0); // frees the headroom's anchor in the texture pool + if (tex_pool) vmaDestroyPool(c->vma, tex_pool); // textures went with vk_texture_shutdown, deferred ones above + tex_pool = NULL; + tex_pool_tried = 0; plat_rmutex_unlock(&cmd_lock); plat_mutex_lock(&pool_lock); for (int i = 0; i < pool_index.n; i++) destroy_rec(c, pool_index.e[i].buf); diff --git a/runtime/vk_mvsurvey.c b/runtime/vk_mvsurvey.c index 0e782c21..293ddcb8 100644 --- a/runtime/vk_mvsurvey.c +++ b/runtime/vk_mvsurvey.c @@ -157,10 +157,10 @@ void vk_mvsurvey_draw(VkExec *ex) { k->size = (uint32_t)u->size; uint32_t want = u->size < SNAP ? (uint32_t)u->size : SNAP; VkDescriptorBufferInfo bi; - const uint8_t *src = vk_usnap_find(u->addr, want, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; + const uint8_t *src = vk_usnap_find(u->addr, want, UINT32_MAX, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; k->have = src ? want : 0; if (src) memcpy(k->bytes, src, want); - const uint8_t *all = vk_usnap_find(u->addr, u->size, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; + const uint8_t *all = vk_usnap_find(u->addr, u->size, UINT32_MAX, 1, &bi) ? vk_usnap_map() + bi.offset : NULL; k->mats = all ? lead_mats((const float *)all, (uint32_t)(u->size / 4)) : 0; if (all) { Stat *s = &S.st[b]; diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index 57142b21..21aa2de9 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -3223,10 +3223,11 @@ static void dyn_vals(const VkPipeState *s, DynVals *o) { } } -// What the replay's command buffer holds, as vk_pipeline_set_dynamic_delta last recorded it (the replay thread only): -// the values, the inputs of the vertex input and the vertex bindings it reads. Invariant while the caller passes -// valid 1: cb holds DC.v for every dynamic state of P.dyn, colour attachments [0, DC.v.n) included. -static struct { +// What the replay's command buffer holds, as vk_pipeline_set_dynamic_delta last recorded it: the values, the inputs of +// the vertex input and the vertex bindings it reads. Invariant while the caller passes valid 1: cb holds DC.v for every +// dynamic state of P.dyn, colour attachments [0, DC.v.n) included. Per thread: the threads of a chunked replay +// (vk_exec.c) each record their own command buffers, and a new one starts with valid 0. +static _Thread_local struct { DynVals v; const GfxShader *vs; uint32_t vattr[16]; diff --git a/runtime/vk_rec.c b/runtime/vk_rec.c index 036fe370..5f80db80 100644 --- a/runtime/vk_rec.c +++ b/runtime/vk_rec.c @@ -39,15 +39,24 @@ static int rec_live; // producer: rec_begin started the record static int end_mode = -1; // TOTK_REC_END (rec_own) static int sub_pending; // producer: the recorder ended a part that rec_submit has not handed over yet static volatile VkResult r_sub_res; // recorder: the result of its last vkQueueSubmit2, read after the drain +static volatile VkResult r_sub_err; // recorder: the first vkQueueSubmit2 that failed (rec_submit reports it) +static volatile uint64_t q_subs; // producer: submissions queued for the recorder (rec_submit) +static volatile uint64_t r_subs; // recorder: submissions it handed to the driver (submit_run) +static _Thread_local int on_rec; // the calling thread is the recorder +static int rec_async = -1; // TOTK_REC_ASYNC (rec_submit) +static struct { uint64_t n, ns; } RQ_; // waits in rec_queue_wait, logged by rec_stats_log static PlatThread *rec_thread; static int rec_prio; // the submitting thread's priority, which the recorder takes static int rec_chk; // TOTK_REC_CHECK (rec_check_handle) static uint64_t rec_delay_ns; // TOTK_REC_DELAY_US (rec_main) +static uint64_t rec_spin_ns; // TOTK_REC_SPIN_US (rec_main); 0: 4096 pauses +static int rec_spin_ab; // TOTK_REC_SPIN_US=ab: the timed poll in the B blocks of TOTK_AB only static volatile uint64_t part_h0; // Q.put_h at rec_begin: where the packets of the part being recorded start static volatile uint64_t part_fence; // the value of the part being recorded (rec_begin) +static uint64_t r_sleeps; // recorder: sleeps on wake_ev, read by rec_stats_log // Producer statistics, logged by rec_stats_log. -static struct { _Alignas(64) uint64_t packets, words, drains, fallbacks, drain_ns, full_waits, parts, parts_on, submits; } RS_; +static struct { _Alignas(64) uint64_t packets, words, drains, fallbacks, drain_ns, full_waits, parts, parts_on, submits, wakes; } RS_; enum { R_PAD, R_BeginQuery, R_BeginRendering, R_BindDescriptorSets, R_BindIndexBuffer, R_BindPipeline, R_BindVertexBuffers2, @@ -71,15 +80,33 @@ static void exec_packet(uint32_t op, void *p); static void rec_main(void *arg) { (void)arg; plat_thread_priority(rec_prio); + on_rec = 1; + perf_core_thread(PCORE_RECORDER); uint64_t t = 0; for (;;) { uint64_t h = __atomic_load_n(&Q.head, __ATOMIC_ACQUIRE); if (t == h) { - for (int i = 0; i < 4096 && t == h; i++) _mm_pause(), h = __atomic_load_n(&Q.head, __ATOMIC_ACQUIRE); + // Poll for the next packets before sleeping: a sleep costs the producer an event set and the recorder + // an OS wake-up. The poll is timed (TOTK_REC_SPIN_US, default 60) because a pause takes 8-150 cycles + // depending on the CPU: 4096 of them are ~140 us on a Coffee Lake i7 and a few us on Zen 4. With fewer than + // 6 logical processors it stays 4096 pauses. Savage Wind Recomp's recorder polls 60 us the same way. + if (rec_spin_ns && (!rec_spin_ab || rt_ab())) { + uint64_t t0 = now_ns(); + for (unsigned i = 1; t == h; i++) { + _mm_pause(); + h = __atomic_load_n(&Q.head, __ATOMIC_ACQUIRE); + if (!(i & 31) && now_ns() - t0 >= rec_spin_ns) break; + } + } else { + for (int i = 0; i < 4096 && t == h; i++) _mm_pause(), h = __atomic_load_n(&Q.head, __ATOMIC_ACQUIRE); + } if (t == h) { __atomic_store_n(&Q.sleeping, 1, __ATOMIC_SEQ_CST); h = __atomic_load_n(&Q.head, __ATOMIC_SEQ_CST); - if (t == h) plat_event_wait_ns(&wake_ev, 2000000); // a missed wake costs at most 2 ms + if (t == h) { + __atomic_add_fetch(&r_sleeps, 1, __ATOMIC_RELAXED); + plat_event_wait_ns(&wake_ev, 2000000); // a missed wake costs at most 2 ms + } __atomic_store_n(&Q.sleeping, 0, __ATOMIC_RELAXED); continue; } @@ -103,7 +130,7 @@ static void rec_main(void *arg) { static void wake(void) { __atomic_thread_fence(__ATOMIC_SEQ_CST); - if (__atomic_load_n(&Q.sleeping, __ATOMIC_RELAXED)) plat_event_set(&wake_ev); + if (__atomic_load_n(&Q.sleeping, __ATOMIC_RELAXED)) plat_event_set(&wake_ev), RS_.wakes++; } // Room for need words: the tail read last leaves it in all but about one packet in 16 MiB of them. @@ -143,14 +170,21 @@ static void *put(uint32_t op, size_t bytes) { // rec_drain), and the packets that begin and end a part go out at once (pub_now), so the recorder begins and ends // command buffers as early as before; TOTK_REC_CHECK publishes every packet, since rec_check_handle searches the ring // up to head. +// The batch is 64 words (512 bytes) where the recorder polls before it sleeps, 8192 (64 KiB) where it does not (fewer +// than 6 usable logical processors, rec_spin_ns): there it is mostly asleep when a batch goes out, and each publish +// cost the producer an event set and the recorder a wake-up. Savage Wind Recomp measured that at 6.7% of its render +// thread's busy time on 4 cores and moved to 64 KiB there (runtime/nvn/vulkan/record.c rec_init, publish_every). +// TOTK_REC_PUB= sets the batch; =ab: 64 words in the A blocks of TOTK_AB, 8192 in the B blocks. +static int pub_ab; static void pub_now(void) { Q.pub_h = Q.put_h; __atomic_store_n(&Q.head, Q.put_h, __ATOMIC_RELEASE); - if (__atomic_load_n(&Q.sleeping, __ATOMIC_RELAXED)) plat_event_set(&wake_ev); + if (__atomic_load_n(&Q.sleeping, __ATOMIC_RELAXED)) plat_event_set(&wake_ev), RS_.wakes++; } static void pub(void) { - if (Q.put_h - Q.pub_h >= Q.pub_words) pub_now(); + uint64_t w = pub_ab ? (rt_ab() ? 8192 : 64) : Q.pub_words; + if (Q.put_h - Q.pub_h >= w) pub_now(); } void rec_drain(void) { @@ -200,8 +234,13 @@ void rec_begin(VkCommandBuffer cb, uint64_t fence, int crumb, uint64_t n) { rec_chk = rt_knob("REC_CHECK", "0"); // debug switches, read before the recorder starts const char *d = cfg_get("TOTK_REC_DELAY_US"); rec_delay_ns = d && *d && atoi(d) > 0 ? (uint64_t)atoi(d) * 1000 : 0; + const char *sp = cfg_get("TOTK_REC_SPIN_US"); + rec_spin_ab = sp && !strcmp(sp, "ab"); + int spin_us = sp && *sp && !rec_spin_ab ? atoi(sp) : 60; + rec_spin_ns = spin_us > 0 && plat_cpu_usable() >= 6 ? (uint64_t)spin_us * 1000 : 0; const char *pw = cfg_get("TOTK_REC_PUB"); // words per publish (pub); 0: every packet - Q.pub_words = rec_chk ? 0 : pw && *pw ? (uint64_t)atoi(pw) : 64; + pub_ab = !rec_chk && pw && !strcmp(pw, "ab"); + Q.pub_words = rec_chk ? 0 : pw && *pw && !pub_ab ? (uint64_t)atoi(pw) : rec_spin_ns ? 64 : 8192; if (Q.pub_words > RING_WORDS / 8) Q.pub_words = RING_WORDS / 8; rec_thread = plat_thread_new(rec_main, NULL, "vk recorder"); if (!rec_thread) { @@ -211,8 +250,13 @@ void rec_begin(VkCommandBuffer cb, uint64_t fence, int crumb, uint64_t n) { rt_log("vk_rec: the replay's Vulkan commands are made by a recorder thread%s", mode == 2 ? " in the B blocks (TOTK_AB)" : ""); if (rec_chk) rt_log("vk_rec: TOTK_REC_CHECK: deferred and destroyed objects are searched in the ring"); - if (Q.pub_words != 64) rt_log("vk_rec: TOTK_REC_PUB: head published every %llu words", - (unsigned long long)Q.pub_words); + if (rec_spin_ns) + rt_log("vk_rec: the recorder polls %llu us before it sleeps%s (TOTK_REC_SPIN_US)", + (unsigned long long)(rec_spin_ns / 1000), rec_spin_ab ? " in the B blocks" : ""); + else + rt_log("vk_rec: the recorder polls 4096 pauses before it sleeps (TOTK_REC_SPIN_US)"); + if (pub_ab) rt_log("vk_rec: head published every 64 words, 8192 in the B blocks (TOTK_REC_PUB=ab)"); + else rt_log("vk_rec: head published every %llu words (TOTK_REC_PUB)", (unsigned long long)Q.pub_words); if (rec_delay_ns) rt_log("vk_rec: TOTK_REC_DELAY_US: the recorder waits %llu us before each batch", (unsigned long long)(rec_delay_ns / 1000)); } @@ -295,11 +339,20 @@ VkResult rec_submit(const VkSubmitInfo2 *si) { a->ncb = si->commandBufferInfoCount, a->nsig = si->signalSemaphoreInfoCount; for (uint32_t i = 0; i < a->ncb; i++) a->cbi[i] = si->pCommandBufferInfos[i]; if (a->nsig) a->sig = si->pSignalSemaphoreInfos[0]; - pub(); - // Waited for here, so the queue holds every earlier submission when the caller goes on, as with the recorder off: - // the present, uploads, readbacks and device waits on other threads rely on that order. - rec_drain(); + __atomic_store_n(&q_subs, q_subs + 1, __ATOMIC_RELEASE); RS_.submits++; + // TOTK_REC_ASYNC=1 (default): the submitting thread goes on while the recorder catches up with the part and submits + // it; it waited here for 0.29 ms a frame in the town on a Ryzen 7900X, 0.34 ms limited to 4 cores. The queue order + // the present, uploads, readbacks and device waits rely on holds through vk_queue_lock (rec_queue_wait). The values + // the part signals are counted as issued at once (vk_sync.c): waiting for one on the host before its submission is + // legal for a timeline semaphore. =0: waits here for the submission, as before; =ab with the A/B blocks. + if (rec_async < 0) rec_async = rt_knob("REC_ASYNC", "1"); + if (RT_KNOB_ON(rec_async)) { + pub_now(); + return __atomic_load_n(&r_sub_err, __ATOMIC_ACQUIRE); + } + pub(); + rec_drain(); return r_sub_res; } @@ -308,17 +361,37 @@ static void submit_run(const A_Submit *a) { .pCommandBufferInfos = a->cbi, .signalSemaphoreInfoCount = a->nsig, .pSignalSemaphoreInfos = a->nsig ? &a->sig : NULL}; vk_queue_lock(); - r_sub_res = vkQueueSubmit2(vk_ctx()->gfx_queue, 1, &si, VK_NULL_HANDLE); + VkResult r = vkQueueSubmit2(vk_ctx()->gfx_queue, 1, &si, VK_NULL_HANDLE); vk_queue_unlock(); + r_sub_res = r; + if (r != VK_SUCCESS && __atomic_load_n(&r_sub_err, __ATOMIC_RELAXED) == VK_SUCCESS) + __atomic_store_n(&r_sub_err, r, __ATOMIC_RELEASE); + __atomic_store_n(&r_subs, r_subs + 1, __ATOMIC_RELEASE); +} + +void rec_queue_wait(void) { + if (!rec_thread || on_rec) return; + uint64_t want = __atomic_load_n(&q_subs, __ATOMIC_ACQUIRE); + if (__atomic_load_n(&r_subs, __ATOMIC_ACQUIRE) >= want) return; + uint64_t t0 = now_ns(); + for (int i = 0; __atomic_load_n(&r_subs, __ATOMIC_ACQUIRE) < want; i++) + if (i < 20000) _mm_pause(); + else plat_yield(); + __atomic_add_fetch(&RQ_.n, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&RQ_.ns, now_ns() - t0, __ATOMIC_RELAXED); } void rec_stats_log(void) { if (!(rt_trace & RT_TRACE_PERF) || !RS_.parts) return; rt_log("vk_rec: %llu of %llu parts recorded by the recorder: %llu packets (%.1f MiB), %llu drains (%llu for a call " - "without a packet) waited %.2f ms in all, ring full %llu times, %llu submissions made by the recorder", + "without a packet) waited %.2f ms in all, ring full %llu times, %llu submissions made by the recorder, " + "recorder slept %llu times and was woken %llu times, queue users waited for its submissions %llu times (%.2f ms)", (unsigned long long)RS_.parts_on, (unsigned long long)RS_.parts, (unsigned long long)RS_.packets, RS_.words * 8 / 1048576.0, (unsigned long long)RS_.drains, (unsigned long long)RS_.fallbacks, - RS_.drain_ns / 1e6, (unsigned long long)RS_.full_waits, (unsigned long long)RS_.submits); + RS_.drain_ns / 1e6, (unsigned long long)RS_.full_waits, (unsigned long long)RS_.submits, + (unsigned long long)__atomic_exchange_n(&r_sleeps, 0, __ATOMIC_RELAXED), (unsigned long long)RS_.wakes, + (unsigned long long)__atomic_exchange_n(&RQ_.n, 0, __ATOMIC_RELAXED), + __atomic_exchange_n(&RQ_.ns, 0, __ATOMIC_RELAXED) / 1e6); memset(&RS_, 0, sizeof RS_); } @@ -381,6 +454,8 @@ void rec_check_log(void) { } #define OWNED(cb) (rec_live && (cb) == rec_cb) + +int rec_owns(VkCommandBuffer cb) { return OWNED(cb); } // A call the ring takes no packet for: everything queued first, then the call here. #define FALLBACK() (rec_drain(), RS_.fallbacks++) diff --git a/runtime/vk_rec.h b/runtime/vk_rec.h index bbceff90..37e5b5d5 100644 --- a/runtime/vk_rec.h +++ b/runtime/vk_rec.h @@ -5,8 +5,10 @@ // command buffer (uploads, the window, probes) are made at once. A call the ring has no packet for (a structure with a // pNext chain) waits for the recorder to empty the ring and is then made at once on the calling thread. With // TOTK_REC_END=1 (default) the recorder also begins, ends and submits the parts it records, so a part's command buffer -// never changes threads; rec_submit returns once the recorder submitted it. The submitting thread allocates from the -// part pool only while the recorder is idle (vk_sync.c claim_slot). +// never changes threads. rec_submit queues the submission and returns at once (TOTK_REC_ASYNC=1, default; 0: once the +// recorder submitted it); every other user of the queue waits in vk_queue_lock until the recorder submitted what was +// queued before (rec_queue_wait), so queue order stays the order of the calls. The submitting thread allocates from +// the part pool only while the recorder is idle (vk_sync.c claim_slot). // // Every vkCmd* the runtime calls through vk.h goes through the rc_* wrappers below (the macros at the end); the // extension commands called through function pointers have rec_* wrappers of their own. TOTK_VK_RECORDER=0 makes every @@ -24,9 +26,14 @@ void rec_end(void); // waits for the recorder; no-op when rec_ // (1: rec_submit hands the part over); otherwise rec_end and the breadcrumb, and the caller ends cb (0). int rec_end_cb(VkCommandBuffer cb, int crumb, uint64_t n); // vkQueueSubmit2 of si (at most two command buffers, one signal, no wait, no fence) under vk_queue_lock: by the -// recorder after a part it ended, waited for, else at once. +// recorder after a part it ended, else at once. Queued for the recorder, it returns the first failure of an earlier +// submission the recorder made (VK_SUCCESS if none), since this one's result is not known yet. VkResult rec_submit(const VkSubmitInfo2 *si); void rec_drain(void); // waits until the recorder made every call queued so far +// vk_queue_lock, before it takes the lock (not on the recorder, not when the thread holds the lock already): waits +// until the recorder submitted every submission queued so far. +void rec_queue_wait(void); +int rec_owns(VkCommandBuffer cb); // the recorder makes cb's calls: a direct call into cb follows rec_drain void rec_stats_log(void); // TOTK_TRACE=perf: packets, bytes, drains and waits since the last call // TOTK_REC_CHECK=1 (debug, recorder on): vk_memory.c reports each object it defers (destroy 0, stamped tl) or destroys // (destroy 1) with the value the GPU reached (gpu); a packet of the part being recorded that still names it is logged diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index 3b90c026..0493dcfc 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -56,7 +56,7 @@ static VkQueryPool ts_pool; static int ts_state; // 0 not set up, 1 on, -1 off static double ts_period; // ns per tick static uint64_t ts_mask; // valid bits of a timestamp -static uint32_t ts_next; // submitting thread (sub_lock): next pair +static uint32_t ts_next; // next pair (atomic: parts recorded on other threads take pairs too) static volatile uint8_t ts_busy[TS_PAIRS]; // written until the completion thread read the pair static int ts_cur = -1; // the pair of the part being recorded, -1 untimed static uint64_t ts_last_end; // completion thread: end of the latest part read (ticks) @@ -175,6 +175,12 @@ static void ts_read_cb(void *arg) { // completion thread, in submission order free(t); } +static int ts_take(void) { + uint32_t p = __atomic_fetch_add(&ts_next, 1, __ATOMIC_RELAXED) % TS_PAIRS; + if (__atomic_exchange_n(&ts_busy[p], 1, __ATOMIC_ACQUIRE)) { plat_inc64(&ts_missed); return -1; } + return (int)p; +} + // At the start of a command buffer (sub_lock held): the pair it times, -1 untimed. static int ts_open(VkCommandBuffer cb) { if (ts_state != 1 || !perf_gpu_timing()) { @@ -182,10 +188,9 @@ static int ts_open(VkCommandBuffer cb) { ts_setup(vk_ctx()); // the overlay turned full: timestamps from now on if (ts_state != 1) return -1; } - uint32_t p = ts_next % TS_PAIRS; - if (__atomic_load_n(&ts_busy[p], __ATOMIC_ACQUIRE)) { plat_inc64(&ts_missed); return -1; } - ts_next++; - ts_busy[p] = 1; + int t = ts_take(); + if (t < 0) return -1; + uint32_t p = (uint32_t)t; vkCmdResetQueryPool(cb, ts_pool, 2 * p, 2); vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, ts_pool, 2 * p); return (int)p; @@ -203,6 +208,17 @@ static void ts_close(VkCommandBuffer cb, int pair, uint64_t fence) { vk_completion_add(fence, ts_read_cb, t); } +int vk_sync_ts_open(VkCommandBuffer cb) { + if (__atomic_load_n(&ts_state, __ATOMIC_ACQUIRE) != 1 || !perf_gpu_timing()) return -1; + int p = ts_take(); + if (p < 0) return -1; + vkCmdResetQueryPool(cb, ts_pool, 2 * (uint32_t)p, 2); + vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, ts_pool, 2 * (uint32_t)p); + return p; +} + +void vk_sync_ts_close(VkCommandBuffer cb, int pair, uint64_t fence) { ts_close(cb, pair, fence); } + // The pair of the part being recorded. static void ts_begin(VkCommandBuffer cb) { ts_cur = ts_open(cb); } @@ -349,26 +365,56 @@ static CmdSlot *take_slot(VkCtx *c, uint64_t fence) { // submitted right before the part that needs them and timed like a part; VK_NULL_HANDLE when there are none. After the // part was recorded, sub_lock held; may move slots[]. // First brings the shadow storage up to date with the CPU's writes, which needs no command buffer. -static VkCommandBuffer prologue(VkCtx *c, uint64_t fence) { +// direct: the calling thread records, ends and submits it itself while the part being replayed stays the recorder's +// (vk_sync_part, which the caller made the recorder idle for). +static VkCommandBuffer prologue(VkCtx *c, uint64_t fence, int direct) { vk_mem_shadow_sync(); // shadow storage: the CPU's writes go in before the part is submitted if (!vk_mem_mirror_pending()) return VK_NULL_HANDLE; int i = claim_slot(c, fence); // before reading slots: the claim may move it VkCommandBuffer cb = slots[i].cb; - rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts + if (direct) { + VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vkBeginCommandBuffer(cb, &bi); + } else { + rec_begin(cb, fence, 0, 0); // a command buffer of the part pool: the recorder's like the parts + } int pair = ts_open(cb); vk_mem_mirror_prologue(cb); ts_close(cb, pair, fence); - if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); + if (direct || !rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); return cb; } +// A chunk of a chunked replay (vk_exec.c chunks_submit), while the submission's own part is open on the recorder. The +// recorder begins, records, ends and submits the parts it owns (vk_rec.c rec_end_cb), so the prologue stays off it: +// the recorder is made idle first, as the part pool allows one thread at a time, and the prologue is recorded, ended +// and submitted here without taking the recorder's part away. +void vk_sync_part(VkCommandBuffer cb) { + VkCtx *c = vk_ctx(); + vk_upload_flush(0); + rec_drain(); + VkCommandBuffer pro = prologue(c, cur_fence, 1); + VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, + {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}}; + VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1, + .pCommandBufferInfos = pro ? cbi : cbi + 1}; + vk_queue_lock(); + VkResult r = vkQueueSubmit2(c->gfx_queue, 1, &si, VK_NULL_HANDLE); + vk_queue_unlock(); + if (r != VK_SUCCESS) { + vk_device_fault_report(r); + rt_fatal("vk: vkQueueSubmit2 of a part of value %llu failed (%d)", (unsigned long long)cur_fence, (int)r); + } +} + VkCommandBuffer vk_sync_split(void) { VkCtx *c = vk_ctx(); VkCommandBuffer cb = slots[cur_slot].cb; ts_end(cb, cur_fence); if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); // every command of the part is in cb, ended by its recorder vk_upload_flush(0); // the uploads this part asked for precede it in queue order - VkCommandBuffer pro = prologue(c, cur_fence); + VkCommandBuffer pro = prologue(c, cur_fence, 0); VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}}; VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1, @@ -419,7 +465,7 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) { perf_replay(t0); // After the recording: the uploads it asked for (and every earlier one) precede it in queue order. vk_upload_flush(0); - VkCommandBuffer pro = prologue(c, fence); + VkCommandBuffer pro = prologue(c, fence, 0); s = &slots[cur_slot]; VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = s->cb}}; @@ -446,8 +492,11 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) { return fence; } -void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) { - plat_mutex_lock(&st_lock); +static _Thread_local VkDeferred *deferred; + +void vk_completion_defer(VkDeferred *d) { deferred = d; } + +static void queue_put(uint64_t fence, void (*fn)(void *), void *arg) { // st_lock held if (qhead + qn == qcap) { if (qhead) { memmove(queue, queue + qhead, sizeof *queue * (size_t)qn); @@ -463,9 +512,34 @@ void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) { // The completion thread sleeps only while nothing is issued beyond done: a callback for a later value (the replay's, // for the submission it records: counter reports, freed resources) becomes due through the submit, which wakes it. if (fence <= done) plat_cond_broadcast(&cv_work); +} + +void vk_completion_add(uint64_t fence, void (*fn)(void *), void *arg) { + VkDeferred *d = deferred; + if (d) { + if (d->n == d->cap) { + d->cap = d->cap ? d->cap * 2 : 64; + d->items = realloc(d->items, sizeof(Pending) * (size_t)d->cap); + if (!d->items) rt_fatal("vk: out of memory (completion queue)"); + } + ((Pending *)d->items)[d->n++] = (Pending){fence, fn, arg}; + return; + } + plat_mutex_lock(&st_lock); + queue_put(fence, fn, arg); plat_mutex_unlock(&st_lock); } +void vk_completion_flush(VkDeferred *d) { + plat_mutex_lock(&st_lock); + for (int i = 0; i < d->n; i++) { + const Pending *p = &((Pending *)d->items)[i]; + queue_put(p->fence, p->fn, p->arg); + } + plat_mutex_unlock(&st_lock); + d->n = 0; +} + static int fence_wait_impl(uint64_t value, uint64_t timeout_ns); int vk_sync_fence_wait(uint64_t value, uint64_t timeout_ns) { diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index d461f561..68b5a9b5 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -46,6 +46,8 @@ typedef struct Tex { uint32_t wgen; // page generation of the last upload check uint32_t checked; // submit_epoch of the last check from a draw binding int uploaded, released; + uint32_t rev_epoch, rev_pos; // the state scan made it live in that submission at that op (vk_texture_revive) + uint32_t own_epoch, own_pos; // a chunk made it GPU-owned in that submission at that op (vk_texture_gpu_written) uint32_t n_up; // uploads of this entry (vk_texture_debug) int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked struct Tex *next_at; // other entries at the same storage address @@ -62,6 +64,7 @@ typedef struct Tex { uint32_t wt_n; // WriteTexels into the storage since the last upload struct Tex *rel_prev, *rel_next; // while released: the list of released entries, oldest first uint64_t rel_ms; // while released: when (coarse_ms) + uint64_t rel_bytes; // while released: its own memory, counted in rel_bytes_total uint64_t use_seq; // order of the last creation or take-back among all entries uint64_t rel_seq; // while released: use_seq of the newest entry placed before this release; an entry // with a later use_seq on overlapping storage replaced this one's bytes @@ -93,6 +96,7 @@ static SampSlot *sslots; static int nsslots; static nvn_hmap samplers = {.klen = sizeof(samp_par)}; static int nreleased; +static uint64_t rel_bytes_total; // own memory of the released entries static Tex *rel_head, *rel_tail; // released entries in release order (sweep_released drops from the head) static uint64_t n_uploads, n_reuploads; // Re-uploads since the last perf line (tex_lock): all, and those of textures a WriteTexels wrote since their last upload. @@ -122,10 +126,12 @@ static uint64_t n_checks; // draw-binding storage checks (make_fresh_snap) si // which emptied every cache keyed on the global generations. #define AGEN_N 4096 static uint32_t agen[AGEN_N]; +static uint32_t arev[AGEN_N]; // per storage address (hashed): the submit_epoch of the state scan's last revival there // Per slot id, for vk_texture_slot_token without the lock: the registration serial (bits 16-47; 0: never registered) // and the address generation index (bits 0-15), one word so a reader gets a consistent pair. Fixed-size (slot ids are // 20 bits), unlike tslots, which moves when it grows. static uint64_t *sinfo; +static uint32_t *sepoch; // per slot: the submit_epoch its entry was checked in (vk_texture_slot_checked), 0 none static uint32_t reg_serial; static uint32_t agen_idx(uint64_t addr) { return (uint32_t)(addr * 0x9E3779B97F4A7C15ull >> 52); } @@ -143,11 +149,17 @@ static void tex_init(void *p) { __atomic_store_n(&tex_ready, 1, __ATOMIC_RELEASE); return; } +static _Thread_local int tex_depth; // how often this thread holds tex_lock (entry_of releases it for a wait) +static _Thread_local uint32_t tex_settles; // entry_of waits of this thread (vk_texture_by_id_locked) static void lock(void) { if (!__atomic_load_n(&tex_ready, __ATOMIC_ACQUIRE)) plat_once(&tex_once, tex_init, NULL); plat_rmutex_lock(&tex_lock); + tex_depth++; +} +static void unlock(void) { + tex_depth--; + plat_rmutex_unlock(&tex_lock); } -static void unlock(void) { plat_rmutex_unlock(&tex_lock); } static uint32_t umax(uint32_t a, uint32_t b) { return a > b ? a : b; } static uint32_t mip(uint32_t v, unsigned lv) { return umax(v >> lv, 1); } @@ -447,8 +459,19 @@ static Tex *create_tex(const tex_t *t, const TexKey *k, int defer_init) { } } if (!img) { - VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; - VkResult r = vmaCreateImage(c->vma, &ci, &ai, &img, &e->alloc, NULL); + VkResult r = rt_oom_forced(RT_OOM_IMAGE) ? VK_ERROR_OUT_OF_DEVICE_MEMORY + : vk_texture_image(&ci, &img, &e->alloc); + // Out of memory: one more try after the headroom blocks and the finished deferred objects are freed + // (vk_oom_release, which takes cmd_lock after the texture lock held here); released textures go at the next + // submit. No wait: the texture is asked for again at its next use, while a wait would hold the texture lock. + if (r == VK_ERROR_OUT_OF_HOST_MEMORY || r == VK_ERROR_OUT_OF_DEVICE_MEMORY) { + static volatile int32_t n_retried; + vk_oom_release(); + VkResult again = vk_texture_image(&ci, &img, &e->alloc); + log_limited(&n_retried, "image creation failed (%d), tried again after a memory release: %s (%ux%ux%u fmt %#x)", + (int)r, again == VK_SUCCESS ? "created" : "failed again", w, h, depth, d->format); + r = again; + } if (r != VK_SUCCESS) { log_limited(&n_failed, "image creation failed (%d): nvn fmt %#x %ux%ux%u target %d levels %u layers %u", (int)r, d->format, w, h, depth, d->target, levels, layers); @@ -528,7 +551,11 @@ static void own_remove(Tex *e) { plat_mutex_unlock(&owned_lock); } +static uint64_t own_bytes(const Tex *e); + static void rel_append(Tex *e) { + e->rel_bytes = own_bytes(e); + rel_bytes_total += e->rel_bytes; e->rel_next = NULL; e->rel_prev = rel_tail; if (rel_tail) rel_tail->rel_next = e; @@ -537,6 +564,8 @@ static void rel_append(Tex *e) { } static void rel_unlink(Tex *e) { + rel_bytes_total -= e->rel_bytes; + e->rel_bytes = 0; if (e->rel_prev) e->rel_prev->rel_next = e->rel_next; else rel_head = e->rel_next; if (e->rel_next) e->rel_next->rel_prev = e->rel_prev; @@ -634,12 +663,19 @@ static void sweep_released(void) { // CLEAN_AGE_MS: without the wait a fight re-created about 500 of them per 10 s. Any other render target lives only in its // image: dropping one that was merely old brought the texture back empty (black flashes in a Depths boss fight, ~170 // "left uninitialised" aliases a minute). +// +// TOTK_TEX_RELEASED_MIB (default 256; 0 = no cap): the same drops run while the released entries hold more own memory than +// that, under pressure or not. A 10-minute field run on the RX 9070 kept 818 MiB of them, 730 MiB with their storage +// taken by a later texture, while 23579 of 23660 take-backs came within a second and none after 120 s. With the +// cap the same route ended with 216 MiB of them, 1.1 GB less process commit and 1 GB less in VMA blocks; 15 dropped +// textures were created again, and the slow frames stayed the same. #define PRESSURE_STEP 32 #define CLEAN_AGE_MS 5000u #define DEAD_AGE_MS 30000u #define DEAD_QUIET_SUBMITS 600u #define DEAD_RETRY_QUERIES 8 // budget queries to skip after a search that found nothing static int vram_pressure = -1; +static uint64_t released_cap; static uint64_t n_pressure_drops, n_pressure_overwritten; static int tex_by_addr(const void *a, const void *b) { @@ -693,16 +729,19 @@ static int dead_candidate(const Tex *e) { // Drops overwritten entries until `need` bytes are free or the step is used up. The sort runs only under pressure, while // an old enough entry exists, and rests for a while when it finds nothing to drop. Returns the bytes freed. static uint64_t pressure_search_ns; // longest search since the last mem: released line -static uint64_t drop_overwritten(uint64_t need, int *dropped) { +// step: PRESSURE_STEP, or OOM_STEP from oom_release, which neither waits out nor starts the rest. +#define OOM_STEP 256 +static uint64_t drop_overwritten(uint64_t need, int *dropped, int step) { static int rest; - if (rest > 0) { rest--; return 0; } + int urgent = step > PRESSURE_STEP; + if (rest > 0 && !urgent) { rest--; return 0; } if (!rel_head || coarse_ms - rel_head->rel_ms < DEAD_AGE_MS) return 0; uint64_t t0 = plat_now_ns(), freed = 0; AddrIndex x; if (!addr_index(&x)) return 0; - Tex *cand[PRESSURE_STEP]; + Tex *cand[OOM_STEP]; int nc = 0; - for (Tex *e = rel_head; e && nc < PRESSURE_STEP && coarse_ms - e->rel_ms >= DEAD_AGE_MS; e = e->rel_next) { + for (Tex *e = rel_head; e && nc < step && coarse_ms - e->rel_ms >= DEAD_AGE_MS; e = e->rel_next) { if (!dead_candidate(e)) continue; int lo = 0, hi = x.n; // the first entry at e's address, then e among those while (lo < hi) { @@ -720,35 +759,62 @@ static uint64_t drop_overwritten(uint64_t need, int *dropped) { destroy_tex(cand[i]); (*dropped)++, n_pressure_drops++, n_pressure_overwritten++; } - if (!nc) rest = DEAD_RETRY_QUERIES; + if (!nc && !urgent) rest = DEAD_RETRY_QUERIES; uint64_t ns = plat_now_ns() - t0; if (ns > pressure_search_ns) pressure_search_ns = ns; return freed; } +// Clean released entries, oldest first: mirrored ones the GPU never wrote, which come back from their storage. +static uint64_t drop_clean(uint64_t need, uint64_t freed, int *dropped, int step) { + for (Tex *e = rel_head, *next; e && freed < need && *dropped < step && coarse_ms - e->rel_ms >= CLEAN_AGE_MS; e = next) { + next = e->rel_next; + if (!e->pub.mirrored || e->pub.gpu_owned || e->gpu_dirty) continue; + freed += own_bytes(e); + dropped_note(e); + destroy_tex(e); + (*dropped)++, n_pressure_drops++; + } + return freed; +} + static void sweep_pressure(void) { // tex_lock held if (vram_pressure < 0) { const char *s = cfg_get("TOTK_TEX_VRAM_PRESSURE"); vram_pressure = s && *s ? atoi(s) : 90; if (vram_pressure < 0 || vram_pressure > 100) vram_pressure = 90; - rt_log("vk: released textures under VRAM pressure: %s%d%%", vram_pressure ? "dropped above " : "off", vram_pressure); + const char *m = cfg_get("TOTK_TEX_RELEASED_MIB"); + released_cap = (m && *m ? strtoull(m, NULL, 10) : 256) << 20; + rt_log("vk: released textures under VRAM pressure: %s%d%%; above %llu MiB of them: %s (TOTK_TEX_RELEASED_MIB)", + vram_pressure ? "dropped above " : "off", vram_pressure, (unsigned long long)(released_cap >> 20), + released_cap ? "dropped" : "no cap"); } - if (!vram_pressure || !rel_head || submit_epoch & 7) return; // the budget query every 8 submits + if (!rel_head || submit_epoch & 7) return; // the budget query every 8 submits + uint64_t need = released_cap && rel_bytes_total > released_cap ? rel_bytes_total - released_cap : 0; uint64_t used, budget; int exact; - if (!vk_mem_vram(&used, &budget, &exact)) return; - uint64_t line = budget / 100 * (uint64_t)vram_pressure; - if (used <= line) return; - int dropped = 0; - uint64_t need = used - line, freed = drop_overwritten(need, &dropped); - for (Tex *e = rel_head, *next; e && freed < need && dropped < PRESSURE_STEP && coarse_ms - e->rel_ms >= CLEAN_AGE_MS; e = next) { - next = e->rel_next; - if (!e->pub.mirrored || e->pub.gpu_owned || e->gpu_dirty) continue; - freed += own_bytes(e); - dropped_note(e); - destroy_tex(e); - dropped++, n_pressure_drops++; + if (vram_pressure && vk_mem_vram(&used, &budget, &exact)) { + uint64_t line = budget / 100 * (uint64_t)vram_pressure; + if (used > line && used - line > need) need = used - line; } + if (!need) return; + int dropped = 0; + uint64_t freed = drop_overwritten(need, &dropped, PRESSURE_STEP); + drop_clean(need, freed, &dropped, PRESSURE_STEP); +} + +// rt_oom_request (an allocation failed somewhere, oom.c): the same two kinds as sweep_pressure, as many as OOM_STEP +// whatever the budget says, and the headroom blocks. Never GPU-written content that lives only in its image. The memory +// comes back once the submission being recorded is done (destroy_tex defers the images). Texture lock held. +static void oom_release(void) { + int dropped = 0; + uint64_t freed = drop_overwritten(UINT64_MAX, &dropped, OOM_STEP); + int overwritten = dropped; + freed = drop_clean(UINT64_MAX, freed, &dropped, OOM_STEP); + vk_headroom_release(); + static volatile int32_t n_logged; + log_limited(&n_logged, "out of memory: %d released textures dropped (%d overwritten, %d clean), %.0f MiB, %d kept", + dropped, overwritten, dropped - overwritten, freed / 1048576.0, nreleased); } // ---------------------------------------------------------------- uploads @@ -1231,10 +1297,20 @@ static int alias_compatible(const Tex *e, const Tex *src) { return eb && eb == sb && ew == sw && eh == sh; } -// src holds content the storage bytes do not: the GPU wrote it and, for a mirrored texture, it was not written back -// and the CPU has not rewritten the storage since (the new entry's upload then reads the newer bytes). -static int alias_has_gpu_content(const Tex *src) { - if (!src->pub.gpu_owned) return 0; +// Whether e is live at op pos of a chunked replay: the state scan is ahead of the chunks, and an entry it revived at a +// later op is still released for this one. Two 4K targets at one address, RGBA8 and R8 with memory of their own, were +// swapped whenever the scan got past the second's SET_RT before a chunk sampling the storage got its turn, and large +// parts of the scene flickered. UINT32_MAX: the state now. +static int live_at(const Tex *e, uint32_t pos) { + return !e->released && !(e->rev_epoch == submit_epoch && e->rev_pos > pos); +} + +// src holds content the storage bytes do not, at op pos of a chunked replay (UINT32_MAX: the state now): the GPU wrote +// it and, for a mirrored texture, it was not written back and the CPU has not rewritten the storage since (the new +// entry's upload then reads the newer bytes). A chunk at a later op that rendered src first does not count: in replay +// order that render comes after the new entry's first use. +static int alias_has_gpu_content(const Tex *src, uint32_t pos) { + if (!src->pub.gpu_owned || (src->own_epoch == submit_epoch && src->own_pos > pos)) return 0; if (!src->pub.mirrored) return 1; uint32_t now; return src->gpu_dirty && !storage_rewritten(src, src->gpu_gen, &now); @@ -1288,10 +1364,45 @@ static int alias_copy(Tex *e, const Tex *src) { return 1; } +static void entry_live(Tex *e) { + if (!e->released) return; + takeback_n[e->pub.gpu_owned != 0][age_bucket(coarse_ms - e->rel_ms)]++; + e->use_seq = ++use_counter; + e->released = 0, nreleased--, rel_unlink(e), tex_gen_bump(e->key.addr); +} + +// A live colour entry at e's storage of e's size, as of op pos: the new entry may take its content (alias_copy). +static int alias_candidate(const TexKey *k, const tex_t *t, uint32_t pos) { + for (const Tex *p = nvn_hm_get(&by_addr, &k->addr); p; p = p->next_at) + if (live_at(p, pos) && !p->f.depth && !p->f.stencil && p->t.L.size == t->L.size) return 1; + return 0; +} + static Tex *entry_of(const tex_t *t) { TexKey k; make_key(t, &k); Tex *e = nvn_hm_get(&by_key, &k); + uint32_t pos = vk_exec_replay_pos(); + // A chunk of a chunked replay makes the entry as of its own op: whether it takes the content of a texture at its + // storage (alias_copy) depends on what the chunks before it render there, and those may still be recording on + // other threads. Taken from a render they had not recorded yet, the new image got the storage's old bytes instead: + // on an RX 7800 XT the grass near Link was missing in 3 of 29 launches with 4 threads (a map texture drawn once + // per tile, kept from then on). With a live colour texture of its size at the storage, the chunk waits until the + // chunks before it are recorded, the texture lock released for the wait (as often as this thread holds it). + if (!e && pos != UINT32_MAX && tex_alias != 0 && alias_candidate(&k, t, pos) && vk_exec_chunk_behind()) { + tex_t own = *t; // t may point into tslots, which a registration moves while the lock is released + t = &own; + int d = tex_depth; + for (int i = 0; i < d; i++) unlock(); + vk_exec_chunk_settle(); + for (int i = 0; i < d; i++) lock(); + tex_settles++; + if ((e = nvn_hm_get(&by_key, &k))) { // another chunk made it meanwhile + entry_live(e); + e->t = *t; + return e; + } + } if (!e) { if (tex_alias < 0) { tex_alias = cfg_bool("TOTK_TEX_ALIAS"); @@ -1309,8 +1420,9 @@ static Tex *entry_of(const tex_t *t) { nx = p->next_at; if (p->released) { destroy_tex(p); continue; } alias = 1; + if (!live_at(p, pos)) continue; // the scan revives it at a later op: released at this chunk's if (tex_alias && (!src || (same_but_swizzle(&p->key, &k) && !same_but_swizzle(&src->key, &k))) && - alias_compatible(e, p) && alias_has_gpu_content(p)) + alias_compatible(e, p) && alias_has_gpu_content(p, pos)) src = p; } if (alias) perf_count(PERF_CNT_TEX_ALIAS); @@ -1336,11 +1448,7 @@ static Tex *entry_of(const tex_t *t) { nvn_hm_put(&by_key, &k, e); tex_gen_bump(k.addr); } - if (e->released) { - takeback_n[e->pub.gpu_owned != 0][age_bucket(coarse_ms - e->rel_ms)]++; - e->use_seq = ++use_counter; - e->released = 0, nreleased--, rel_unlink(e), tex_gen_bump(k.addr); - } + entry_live(e); e->t = *t; return e; } @@ -1366,6 +1474,25 @@ const VkTexImage *vk_texture_get(const tex_t *t) { return e ? &e->pub : NULL; } +// The replay's state scan (vk_exec.c): the entry vk_texture_get would return, neither created nor made current; the +// recording does that. It is revived here, in replay order, and remembers the op it happened at: a chunk may bind a +// finalized target's slot before the chunk that renders it reached its SET_RT, or long after the scan went past a later +// target at the same storage, and slot_resolve has to answer as the replay would at the chunk's own op (live_at). +const VkTexImage *vk_texture_revive(const tex_t *t, uint32_t pos) { + if (!t || !t->mem) return NULL; + TexKey k; + make_key(t, &k); + lock(); + Tex *e = nvn_hm_get(&by_key, &k); + if (e && e->released) { + e->rev_epoch = submit_epoch, e->rev_pos = pos; + __atomic_store_n(&arev[agen_idx(k.addr)], submit_epoch, __ATOMIC_RELEASE); + entry_live(e); + } + unlock(); + return e ? &e->pub : NULL; +} + // Render targets the CPU reads without NVN's invalidate (vk_texture_write_back_small); small_pending: one may be dirty. // wb_limit: targets up to this size (bytes) always go back when the CPU waits for a fence, TOTK_WB_FENCE_LIMIT, read // once. A larger one goes back unless the GPU wrote it in each of the last WB_RUN frames (wb_wanted). wb_trace: @@ -1398,6 +1525,10 @@ void vk_texture_gpu_written(const tex_t *t) { Tex *e = entry_of(t); if (e) { make_fresh(e); // a partial render keeps what the storage held before + if (!e->pub.gpu_owned) { // entry_of's alias choice of a chunk at an earlier op leaves this render out + uint32_t pos = vk_exec_replay_pos(); + e->own_epoch = pos == UINT32_MAX ? 0 : submit_epoch, e->own_pos = pos; + } e->pub.gpu_owned = 1; uint64_t f = vk_exec_recording_fence(); if (f > e->wr_fence) e->wr_fence = f; @@ -1468,7 +1599,9 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) { } if (!sinfo) { uint64_t *p = calloc(1 << 20, sizeof *p); - if (!p) rt_fatal("vk: out of memory for texture slot tokens"); + uint32_t *q = calloc(1 << 20, sizeof *q); + if (!p || !q) rt_fatal("vk: out of memory for texture slot tokens"); + __atomic_store_n(&sepoch, q, __ATOMIC_RELEASE); __atomic_store_n(&sinfo, p, __ATOMIC_RELEASE); } TexSlot *s = &tslots[id]; @@ -1484,15 +1617,15 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) { unlock(); } -// The entry slot s resolves to, and the view and has_view a caller gets with it (the slot's, with the descriptor's -// swizzle when the entry is another descriptor at the same storage). NULL when the slot names no texture. -static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) { +// The entry slot s resolves to at op pos, and the view and has_view a caller gets with it (the slot's, with the +// descriptor's swizzle when the entry is another descriptor at the same storage). NULL when the slot names no texture. +static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view, uint32_t pos) { *view = s->v; *has_view = s->has_view; TexKey k; make_key(&s->t, &k); Tex *e = nvn_hm_get(&by_key, &k); - if (!e || e->released) { // MNG hle_nvn.c: a finalized slot follows the live texture at its storage + if (!e || !live_at(e, pos)) { // MNG hle_nvn.c: a finalized slot follows the live texture at its storage // Prefer the live entry that differs from the slot descriptor only in its swizzle: transient render targets // share storage, and the newest one at the address is usually another size. The map screen samples its // 2250x1080 map target through an (R,G,B,1) descriptor and followed a 960x60 target at the same address. @@ -1505,13 +1638,14 @@ static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) { Tex *own = e, *any = NULL; e = NULL; for (Tex *p = nvn_hm_get(&by_addr, &k.addr); p; p = p->next_at) { - if (p->released || linear_format(p->key.d.format) != linear_format(k.d.format)) continue; + if (!live_at(p, pos) || linear_format(p->key.d.format) != linear_format(k.d.format)) continue; if (!any) any = p; if (same_but_swizzle(&p->key, &k)) { e = p; break; } } if (!e) e = any ? any : own; } - if (!e) e = s->t.mem ? entry_of(&s->t) : NULL; + TexSlot copy; // entry_of may release the lock (a chunk's wait), and a registration then moves tslots + if (!e && s->t.mem) copy = *s, s = ©, e = entry_of(&s->t); // A followed texture is another descriptor at the same storage: its image holds the content, but sampling takes the // slot descriptor's swizzle. The UI renders text into RGBA8 and samples it through a (A,A,A,A) descriptor; with the // render target's RGBA swizzle the text blur saw R = 1 everywhere and drew solid boxes behind tips and labels. @@ -1553,14 +1687,38 @@ const VkTexImage *vk_texture_by_id(int id, view_t *view, int *has_view) { const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view) { if (id < 0 || id >= ntslots || !tslots[id].used) return NULL; TexSlot *s = &tslots[id]; - uint32_t g = agen[sinfo[id] & 0xffff]; + uint32_t idx = sinfo[id] & 0xffff, g = agen[idx]; + uint32_t pos = arev[idx] == submit_epoch ? vk_exec_replay_pos() : UINT32_MAX; + if (pos != UINT32_MAX) { // what the storage holds depends on the op: chunks at other ops share no answer in s + view_t v; + int hv; + Tex *f = slot_resolve(s, &v, &hv, pos); + s = &tslots[id]; // slot_resolve may have released the lock + if (f && (!f->uploaded || f->checked != submit_epoch)) { + make_fresh_snap(f); + f->checked = submit_epoch; + } + if (view) *view = has_view ? v : s->v; + if (has_view) *has_view = view ? hv : s->has_view; + return f ? &f->pub : NULL; + } if (!s->ce || s->cgen != g) { // resolved again: an entry at its address was created, dropped or released, or the view_t v; // slot registered (s->ce cleared); an empty result is not kept: image creation may int hv; // succeed later - s->ce = slot_resolve(s, &v, &hv); // entry_of may create an entry and bump g: g read before it + uint32_t settles = tex_settles; + Tex *r = slot_resolve(s, &v, &hv, UINT32_MAX); // entry_of may create an entry and bump g: g read before it + s = &tslots[id]; + if (settles != tex_settles) { // the lock was released for a wait: the slot may be another texture by now + if (r && (!r->uploaded || r->checked != submit_epoch)) make_fresh_snap(r), r->checked = submit_epoch; + if (view) *view = has_view ? v : s->v; + if (has_view) *has_view = view ? hv : s->has_view; + return r ? &r->pub : NULL; + } + s->ce = r; s->cview = v, s->chas_view = (uint8_t)hv; s->cgen = g; s->cepoch = 0; + __atomic_store_n(&sepoch[id], 0, __ATOMIC_RELAXED); } Tex *e = s->ce; // Bindings of one submit share one check, answered from the submit's write-watch snapshot (guest writes during a @@ -1572,6 +1730,7 @@ const VkTexImage *vk_texture_by_id_locked(int id, view_t *view, int *has_view) { e->checked = submit_epoch; } s->cepoch = submit_epoch; + __atomic_store_n(&sepoch[id], submit_epoch, __ATOMIC_RELEASE); } if (view) *view = has_view ? s->cview : s->v; if (has_view) *has_view = view ? s->chas_view : s->has_view; @@ -1593,11 +1752,26 @@ int vk_texture_slot_swz(int id, uint8_t swz[4]) { uint32_t vk_texture_epoch(void) { return __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); } +int vk_texture_slot_moves(int id) { + const uint64_t *si = __atomic_load_n(&sinfo, __ATOMIC_ACQUIRE); + if (!si || id < 0 || id >= (1 << 20)) return 0; + uint64_t v = __atomic_load_n(&si[id], __ATOMIC_RELAXED); + return __atomic_load_n(&arev[v & 0xffff], __ATOMIC_ACQUIRE) == __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); +} + +int vk_texture_slot_checked(int id) { + const uint32_t *e = __atomic_load_n(&sepoch, __ATOMIC_ACQUIRE); + return e && id >= 0 && id < (1 << 20) && + __atomic_load_n(&e[id], __ATOMIC_ACQUIRE) == __atomic_load_n(&submit_epoch, __ATOMIC_RELAXED); +} + void vk_texture_new_submit(void) { lock(); if (!++submit_epoch) submit_epoch = 1; coarse_ms = plat_now_ms(); sweep_pressure(); + if (__atomic_load_n(&rt_oom_req, __ATOMIC_RELAXED) && __atomic_exchange_n(&rt_oom_req, 0, __ATOMIC_ACQUIRE)) + oom_release(); static uint64_t log_at; uint64_t f = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED) / 60; if ((rt_trace & RT_TRACE_PERF) && f != log_at && tex_alias > 0) { @@ -1624,8 +1798,9 @@ void vk_texture_new_submit(void) { (unsigned long long)f * 60, (unsigned long long)ck, (unsigned long long)n_checks, (unsigned long long)calls, (unsigned long long)untr); n_checks = 0; - rt_log("perf: frame %llu: released textures kept %d, dropped under VRAM pressure %llu (%llu overwritten)", - (unsigned long long)f * 60, nreleased, (unsigned long long)n_pressure_drops, + rt_log("perf: frame %llu: released textures kept %d (%.0f MiB), dropped under VRAM pressure or the cap %llu " + "(%llu overwritten)", (unsigned long long)f * 60, nreleased, rel_bytes_total / 1048576.0, + (unsigned long long)n_pressure_drops, (unsigned long long)n_pressure_overwritten); n_pressure_drops = n_pressure_overwritten = 0; } @@ -1921,7 +2096,12 @@ static void write_back_n(Tex **v, int n) { if (np) vk_upload_flush(1); for (int i = 0; i < np; i++) { WbPart *q = &p[i]; - uint8_t *buf = malloc(q->g.img * (size_t)q->g.nz); + size_t bytes = q->g.img * (size_t)q->g.nz; + uint8_t *buf = malloc(bytes); + if (!buf) { // waits for memory with the texture lock held, after the release that needs no submit + vk_oom_release(); + buf = rt_realloc_wait(NULL, bytes, "vk: texture write-back buffer"); + } if (!buf) rt_fatal("vk: out of memory for a texture write-back"); texels_read_take(q->e, &q->g, &q->t, buf); storage_texels(&q->e->t, &q->g, &q->r, buf, 1); @@ -2417,8 +2597,8 @@ void vk_texture_shutdown(void) { by_addr = (nvn_hmap){.klen = 8}; samplers = (nvn_hmap){.klen = sizeof(samp_par)}; free(tslots), free(sslots); - tslots = NULL, sslots = NULL, ntslots = nsslots = 0, nreleased = 0; + tslots = NULL, sslots = NULL, ntslots = nsslots = 0, nreleased = 0, rel_bytes_total = 0; rel_head = rel_tail = NULL; - if (sinfo) memset(sinfo, 0, sizeof *sinfo << 20); // no slot is registered: tokens 0 + if (sinfo) memset(sinfo, 0, sizeof *sinfo << 20), memset(sepoch, 0, sizeof *sepoch << 20); // no slot is registered: tokens 0 unlock(); } diff --git a/runtime/vk_usnap.c b/runtime/vk_usnap.c index 4e9744b9..660aa724 100644 --- a/runtime/vk_usnap.c +++ b/runtime/vk_usnap.c @@ -11,11 +11,13 @@ // that submission's write-back, the mirror holds them until then. // - A GPU write within the submission (UpdateUniformBuffer, a storage buffer a shader writes, a buffer copy or fill, // an image-to-buffer copy: vk_usnap_drop) drops the copies it touches from then on, so later draws read the result. +// A chunked replay (vk_exec.c) records the ops out of order: its scan drops at the position of the op +// (vk_usnap_drop_at, scan_writes for the shaders' stores), and a chunk finds a copy only at a position before that. // - The arena goes back to a free list once the submission's fence is done (completion thread). // The design follows MacNativeGames' TOTK_UBO_SNAPSHOT (games/totk/third_party/d2r_gfx/gfx/nvn/nvn_mtl.m usnap_take, // usnap_drop; MIT, Copyright (c) 2026 jhaizhou); the code is ours. // The copy is taken where vk_exec_record starts: at the submit call when the replay runs on the submitting thread, later -// with TOTK_REPLAY_ASYNC. TOTK_UBO_SNAPSHOT=0 binds live memory as before; =ab alternates (TOTK_AB). +// with TOTK_REPLAY_ASYNC; a chunked replay takes it on a recording thread while its scan runs. TOTK_UBO_SNAPSHOT=0 binds live memory as before; =ab alternates (TOTK_AB). #include #include @@ -42,7 +44,7 @@ typedef struct Arena { } Arena; typedef struct { uint64_t lo, hi; } Rng; -typedef struct { uint64_t lo, hi, off; int dropped; } Reg; +typedef struct { uint64_t lo, hi, off, drop_at; } Reg; // drop_at: dropped from this position on; UINT64_MAX not typedef struct { uint64_t lo, hi, fence; } Wrote; static struct { @@ -62,6 +64,7 @@ static struct { static Arena *free_arenas; static PlatMutex free_lock = PLAT_MUTEX_INIT; +static PlatMutex drop_lock = PLAT_MUTEX_INIT; // threads recording chunks of one submission drop concurrently static void *grow(void *p, int *cap, size_t elem) { int n = *cap ? *cap * 2 : 1024; @@ -279,7 +282,7 @@ void vk_usnap_begin(const uint64_t *handles, int n, uint64_t fence) { continue; } if (U.nreg == U.creg) U.reg = grow(U.reg, &U.creg, sizeof *U.reg); - U.reg[U.nreg++] = (Reg){lo, hi, total, 0}; + U.reg[U.nreg++] = (Reg){lo, hi, total, UINT64_MAX}; total += (hi - lo + ALIGN - 1) & ~(uint64_t)(ALIGN - 1); } if (!U.nreg) return; @@ -313,29 +316,38 @@ static int first_above(uint64_t a) { return lo; } -int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t align, VkDescriptorBufferInfo *out) { +int vk_usnap_find(uint64_t addr, uint64_t n, uint64_t at, uint64_t align, VkDescriptorBufferInfo *out) { if (!U.on || !n) return 0; int i = first_above(addr); if (i >= U.nreg) return 0; const Reg *r = &U.reg[i]; - if (r->dropped || addr < r->lo || addr + n > r->hi) return 0; + if (__atomic_load_n(&r->drop_at, __ATOMIC_RELAXED) <= at || addr < r->lo || addr + n > r->hi) return 0; uint64_t off = r->off + (addr - r->lo); if (off % align) return 0; *out = (VkDescriptorBufferInfo){U.arena->buf, off, n}; - U.hits++; + __atomic_fetch_add(&U.hits, 1, __ATOMIC_RELAXED); return 1; } -void vk_usnap_drop(uint64_t addr, uint64_t n) { +static void drop(uint64_t addr, uint64_t n, uint64_t at) { if (!U.mode || !n) return; + plat_mutex_lock(&drop_lock); if (U.nwr == U.cwr) U.wr = grow(U.wr, &U.cwr, sizeof *U.wr); U.wr[U.nwr++] = (Wrote){addr, addr + n, U.fence}; - if (!U.on) return; - for (int i = first_above(addr); i < U.nreg && U.reg[i].lo < addr + n; i++) - if (!U.reg[i].dropped) U.reg[i].dropped = 1, U.drops++, U.gen++; + if (U.on) + for (int i = first_above(addr); i < U.nreg && U.reg[i].lo < addr + n; i++) + if (at < U.reg[i].drop_at) { + if (U.reg[i].drop_at == UINT64_MAX) U.drops++; + __atomic_store_n(&U.reg[i].drop_at, at, __ATOMIC_RELAXED); + __atomic_add_fetch(&U.gen, 1, __ATOMIC_RELAXED); + } + plat_mutex_unlock(&drop_lock); } -uint64_t vk_usnap_gen(void) { return U.gen; } +void vk_usnap_drop(uint64_t addr, uint64_t n) { drop(addr, n, 0); } +void vk_usnap_drop_at(uint64_t addr, uint64_t n, uint32_t at) { drop(addr, n, at); } + +uint64_t vk_usnap_gen(void) { return __atomic_load_n(&U.gen, __ATOMIC_RELAXED); } // the host copy where there is one (DLSS), else the device copy static uint8_t *host_copy(void) { return U.arena->cpu ? U.arena->cpu : U.arena->map; } @@ -344,7 +356,7 @@ const uint8_t *vk_usnap_map(void) { return U.arena ? host_copy() : NULL; } const uint8_t *vk_usnap_bytes(uint64_t addr, uint64_t n) { VkDescriptorBufferInfo bi; - return vk_usnap_find(addr, n, 1, &bi) ? host_copy() + bi.offset : NULL; + return vk_usnap_find(addr, n, UINT32_MAX, 1, &bi) ? host_copy() + bi.offset : NULL; } // The arenas back on the free list (vk_exec_draw_shutdown: the device is idle, every completion ran). diff --git a/tests/release_linux_apprun_test.py b/tests/release_linux_apprun_test.py new file mode 100644 index 00000000..8aaa7c7f --- /dev/null +++ b/tests/release_linux_apprun_test.py @@ -0,0 +1,57 @@ +"""The AppImage entry must ignore the user's site-packages (issue 985). + +AppRun starts the bundled Python with `-m installer.gui`. Without PYTHONNOUSERSITE, a PyPI package named `installer` +in the user site shadows the project's own installer package and the window fails with "No module named installer.gui". + +The test reads the export lines from AppRun, then runs Python with that environment against a fake user site holding +a shadowing `installer` package (no `gui`) and a fake bundle holding the real layout. The bundle is added the way +totk-release.pth adds it, after the user site, so the user site wins unless AppRun turns it off. +""" +import os +import subprocess +import sys +import tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) + +import make_release_linux # noqa: E402 + +line = 'export PYTHONNOUSERSITE=1' +lines = make_release_linux.APPRUN.splitlines() + +# Environment from AppRun's export lines. Values that refer to $HERE point into the mount and are not needed here. +env = dict(os.environ) +env.pop('PYTHONNOUSERSITE', None) +for text in lines: + if text.startswith('export ') and '=' in text and '$' not in text: + name, value = text[len('export '):].split('=', 1) + env[name] = value +env.pop('PYTHONPATH', None) + +with tempfile.TemporaryDirectory() as tmp: + tmp = Path(tmp) + user_base = tmp / 'userbase' + env['PYTHONUSERBASE'] = str(user_base) + user_site = Path(subprocess.run( + [sys.executable, '-c', 'import site; print(site.getusersitepackages())'], + env=env, capture_output=True, text=True, check=True, cwd=tmp).stdout.strip()) + shadow = user_site / 'installer' + shadow.mkdir(parents=True) + (shadow / '__init__.py').write_text('# stands in for the PyPI "installer" package: no gui submodule\n') + + bundle = tmp / 'bundle' + (bundle / 'installer').mkdir(parents=True) + (bundle / 'installer' / '__init__.py').write_text('') + (bundle / 'installer' / 'gui.py').write_text('print("gui imported")\n') + + code = 'import site, sys; site.addsitedir(sys.argv[1]); import installer.gui' + result = subprocess.run([sys.executable, '-c', code, str(bundle)], + env=env, capture_output=True, text=True, cwd=tmp) + assert result.returncode == 0, f'installer.gui not importable with AppRun env: {result.stderr.strip()}' + +# Checked last so the import behaviour is what fails on the base commit; this line guards the mechanism. +assert line in lines, f'AppRun does not set {line!r}; AppRun lines: {lines}' + +print('AppRun sets PYTHONNOUSERSITE=1 and bundled installer.gui imports past a user-site shadow') diff --git a/tests/romfs_prompts_test.py b/tests/romfs_prompts_test.py index a21cdd50..0705170d 100644 --- a/tests/romfs_prompts_test.py +++ b/tests/romfs_prompts_test.py @@ -13,6 +13,7 @@ import importlib.util import io import json import random +import shutil import struct import subprocess import sys @@ -299,7 +300,7 @@ LETTERS_2 = {ord('T'): (560, polygon((230, 0), (330, 0), (330, 600), (520, 600), # The other characters the PC prompts' keys show (button_icons.PC_LABELS of the defaults): a block with a counter, each # its own size. LETTERS_2.update({ord(ch): (560, lambda pen, i=i: [rect(pen, 60, 0, 300 + 10 * i, 700), rect(pen, 120, 100, 240, 600, True)]) - for i, ch in enumerate('ACDEFMQSW1234⇧G')}) + for i, ch in enumerate('ACDEFGHIMQSW1234')}) PC_PANES = {'A': 'E', 'B': '↑', 'X': '▬', 'Y': 'L'} # the diamond's letters with the default keys @@ -572,6 +573,34 @@ class PromptsTest(unittest.TestCase): self.assertEqual(proc.returncode, 0, proc.stderr) self.assertEqual((out / 'manifest.tsv').read_bytes(), (self.outs['v121', 'xbox'][0] / 'manifest.tsv').read_bytes()) + def test_reuse_links_the_files_that_match_and_writes_the_same_overlay(self): + romfs = self.romfs['v143'] + pc, pc_summary = self.outs['v143', 'pc'] + source = self.base / 'reuse' / 'source' # a copy, so the class's folder is never linked or rewritten + shutil.copytree(pc, source) + out = self.base / 'reuse' / 'pc' + proc = romfs.run(out, '--prompts', 'pc', '--reuse', str(source)) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertEqual((out / 'manifest.tsv').read_bytes(), (pc / 'manifest.tsv').read_bytes()) + self.assertEqual(sorted(json.loads(proc.stdout.strip().splitlines()[-1])['reused']), sorted(manifest(pc))) + self.assertEqual(pc_summary['reused'], []) + # From the Xbox prompts only what both share is taken; the rest is made, and the overlay is the same. + xbox = self.outs['v143', 'xbox'][0] + out = self.base / 'reuse' / 'from-xbox' + proc = romfs.run(out, '--prompts', 'pc', '--reuse', str(xbox)) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertEqual((out / 'manifest.tsv').read_bytes(), (pc / 'manifest.tsv').read_bytes()) + shared = {p for p, row in manifest(pc).items() if manifest(xbox).get(p) == row} + self.assertEqual(set(json.loads(proc.stdout.strip().splitlines()[-1])['reused']), shared) + # A new run into the source folder replaces its files instead of writing through the links. + before = {p: (self.base / 'reuse' / 'pc' / p).read_bytes() for p in manifest(pc)} + ini = self.base / 'reuse' / 'totk.ini' + ini.write_text('[input]\nbind_A = key:F\n', encoding='utf-8') + proc = romfs.run(source, '--prompts', 'pc', '--binds', str(ini)) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertNotEqual((source / 'manifest.tsv').read_bytes(), (pc / 'manifest.tsv').read_bytes()) + self.assertEqual({p: (self.base / 'reuse' / 'pc' / p).read_bytes() for p in manifest(pc)}, before) + def test_unrecognised_romfs_fails_before_writing(self): cases = {'no text fonts': (dict(text_fonts=False), icons.TEXT_FONTS[0]), 'no size entry': (dict(restbl_skips=(REGIONAL_FONT,)), 'has no entry in the resource size table'), diff --git a/tools/button_icons.py b/tools/button_icons.py index 813a174c..05b4dba0 100644 --- a/tools/button_icons.py +++ b/tools/button_icons.py @@ -195,7 +195,7 @@ PC_BINDS = { } # What a key shows: its letter, a symbol or a short name (key names as in totk.ini; the rest show their name). PC_LABELS = { - 'lshift': '⇧', 'rshift': '⇧', 'lctrl': 'CTRL', 'rctrl': 'CTRL', 'lalt': 'ALT', 'tab': 'TAB', + 'lshift': 'SHIFT', 'rshift': 'SHIFT', 'lctrl': 'CTRL', 'rctrl': 'CTRL', 'lalt': 'ALT', 'tab': 'TAB', 'enter': 'ENT', 'backspace': 'BKSP', 'capslock': 'CAPS', 'insert': 'INS', 'delete': 'DEL', 'home': 'HOME', 'end': 'END', 'pageup': 'PGUP', 'pagedown': 'PGDN', 'up': '↑', 'down': '↓', 'left': '←', 'right': '→', 'num_add': '+', 'num_sub': '-', 'num_mul': '*', 'num_div': '/', 'num_decimal': '.', @@ -284,6 +284,13 @@ def pc_icons(g, adv, binds=PC_BINDS): polygon(ReverseContourPen(pen), [(bx0, by0), (bx1, by0), (bx1, by1), (bx1 - t, by1), (bx1 - t, by0 + t), (bx0 + t, by0 + t), (bx0 + t, by1), (bx0, by1)]) return + if key in ('lshift', 'rshift'): # a solid arrow and the name under it: the arrow alone read as a direction + cx, ab, at, hw, sw = (x0 + x1) / 2, y0 + h * 0.46, y0 + h * 0.86, w * 0.20, w * 0.09 + neck = ab + h * 0.18 + polygon(ReverseContourPen(pen), [(cx - sw, ab), (cx + sw, ab), (cx + sw, neck), (cx + hw, neck), (cx, at), + (cx - hw, neck), (cx - sw, neck)]) + g.hole(pen, 'SHIFT', cx, y0 + h * 0.25, h * 0.21, w * 0.84) + return label = PC_LABELS.get(key, key.upper()) cap = h * (0.52 if len(label) == 1 else 0.36 if len(label) == 2 else 0.26) g.hole(pen, label, (x0 + x1) / 2, (y0 + y1) / 2, cap, w * 0.74) diff --git a/tools/make_release_linux.py b/tools/make_release_linux.py index e3ba2182..10a62204 100644 --- a/tools/make_release_linux.py +++ b/tools/make_release_linux.py @@ -426,6 +426,8 @@ APPRUN = '''#!/bin/sh # The AppImage entry: the setup window, run by the bundled Python from the read-only mount (no bytecode written there). HERE="$(dirname "$(readlink -f "$0")")" export PYTHONDONTWRITEBYTECODE=1 +# Ignore ~/.local/lib/pythonX.Y/site-packages: a user-installed package such as PyPI's `installer` would shadow ours +export PYTHONNOUSERSITE=1 export TCL_LIBRARY="$HERE/python/lib/tcl9.0" export TK_LIBRARY="$HERE/python/lib/tk9.0" exec "$HERE/python/bin/python3" -m installer.gui "$@" diff --git a/tools/make_romfs_patches.py b/tools/make_romfs_patches.py index f9139543..0a3946d1 100644 --- a/tools/make_romfs_patches.py +++ b/tools/make_romfs_patches.py @@ -45,6 +45,10 @@ from the game's own files on the player's machine, nothing added (tools/button_i guide's images (A, B, X, Y, L, ZL and the stick's two). The one-letter panes around the action prompt name the keys with one character of the Normal font (E, an arrow for Shift, a bar for Space, L and R for the mouse buttons). +--reuse : an overlay made before, such as romfs-patches-pc for romfs-patches-pc-x. Each file of it whose +contents match the new one is hard-linked (copied where links fail) instead of compressed again; the summary's reused +lists them. The output is byte for byte the same as without it. + Input: build/extracted//romfs (romfs.bin and index.json, verified by tools/extract.py). Output: build//romfs-patches/ by default (romfs-patches-