diff --git a/docs/settings.md b/docs/settings.md index 0d4687ba..0f88cf37 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -143,7 +143,12 @@ Many rows below are developer tools that only act together with another setting | `dyn_delta` (default `1`) | Records only the pipeline dynamic states (cull mode, depth and stencil tests, vertex input, blend enables, equations and write masks, ...) whose values changed since the last draw in the command buffer, also across pipeline changes; `0` records the full set at every change of pipeline state, as before; `ab` alternates with the `ab` blocks. | Fewer commands per pipeline change on the render and recorder threads; a driver that loses dynamic state across pipeline binds would draw with stale state. `TOTK_TRACE=perf` prints the calls and commands per submission. | Caution | | `feedback` | `0` samples a target that an earlier draw of the same pass wrote with no barrier; `restart` ends and restarts the pass behind a full barrier; otherwise the feedback-loop layout is used when the GPU supports it. | `0` brings back the old behaviour that the barrier fixed, so the picture can break in passes that read a target they draw into; `restart` costs extra passes. | Risky | | `flush_span` (default `0`) | `1` copies one span from the lowest to the highest flushed address of a pool, as an older version did. | Copies far more data to the GPU in crowded towns: an estimated 30-50 ms of GPU time per large copy over PCIe 3.0 x8. | Caution | -| `frame_gen` (default `0`) | `1` creates AMD FSR 3 frame interpolation and optical flow (FidelityFX SDK 1.1.4) at start, on a 1920x1080 placeholder size or `output_res`. Read at start. | Work in progress: the contexts are made and destroyed, nothing generates frames yet. Costs start-up time and GPU memory while set. | Caution | +| `frame_gen` (default `0`) | `1` brings up AMD FSR 3 frame interpolation and optical flow (FidelityFX SDK 1.1.4) at start; with DLSS off the replay follows the scene and records the camera motion vectors, the picture before the HUD and the interpolation of a frame between each two the game draws (`runtime/vk_fg.cpp`). Off with DLSS, `replay_workers` above 0, `sync_ahead = 1` or HDR output, and while a menu background or a loading or event fade is open. Read at start. | Work in progress: the generated frames are made but not shown (`fg_test` shows them for checks). Costs about 1.5 ms of GPU and 0.55 ms of the replay thread per frame at 1080p (RX 9070, 7900X). | Caution | +| `fg_test` | Frame generation checks: `1` presents the generated frame in place of the game's, `2` the picture before the HUD. Read at start. | Shows the frames out of their order: for checks only. | Caution | +| `fg_debug` | `1`: the generated frames hold the SDK's debug view (motion fields, optical flow, disocclusion). Read at start. | For checks only. | Caution | +| `fg_shot` | Frame generation shots: a present number, or `w` of world time; the window texture, the generated frame and the picture before the HUD of three frames from there go to `fg_shot_dir` as `fg__.png`. | Writes PNGs. | Safe | +| `fg_shot_dir` | Folder for the `fg_shot` PNGs (default the working folder). | Only moves the output. | Safe | +| `fg_times` (default `1`) | `0` turns off the GPU timestamps of frame generation's steps; every 600 generated frames a log line gives their GPU times and the replay thread's recording time (median, p99). | Timestamps cost next to nothing. | Safe | | `shadow_snap` (default `1`) | Shadow storage (Linux, Wine): `1` syncs the pools from per-submission write-watch snapshots, each 2 MiB region queried at most once per submission and shared with the texture checks; `0` queries each pool at every part of a submission, as before. | `1` cut the replay thread's write tracking from about 4.4 to 1.3 ms a frame on a Steam Deck; `0` is for comparison runs. Under Wine/Proton the default is `0` until it has been measured there; `1` turns it on. No effect where shadow storage is off (Windows by default). | Caution | | `freeze_idle` (default `60`) | Number of submissions a 2 MiB region stays unwritten before it is made read-only; `0` turns the freeze off. | A larger value keeps more regions writable and costs more write tracking; a write to a frozen region costs about 60 microseconds. | Caution | | `gpl_draw_build` (default `1`) | `0` leaves the missing shader libraries of a waiting draw to its background job instead of building them in parallel at once. | Draws whose pipeline is not ready are skipped meanwhile, until the job finishes. | Caution | diff --git a/runtime/cfg.c b/runtime/cfg.c index f7018745..b9e49f98 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -42,7 +42,7 @@ static const char *const known[] = { "BIND_ZL", "BIND_ZR", "BARRIER_MERGE", "BLOCK_VIEWS_3D", "CLIMB_FIX", "CLOTH_REALTIME", "COLOR_BRIGHTNESS", "COLOR_CONTRAST", "COLOR_GAMMA", "COLOR_PRESET", "COLOR_SATURATION", "CONTACT_DEBUG", "CONTACT_SHADOWS", "CORE_DIAG", "CRIT", "CRIT_OUT", "DESC_TEMPLATE", "DESWIZZLE", "DEVICE_PROFILE", "DIALOGUE_CLICK", "DISCORD_APP_ID", "DISCORD_IMAGE", "DISCORD_RPC", "DLAA", "DLAA_CAM_JITTER", "DLAA_EFFECTS_AFTER", "DLAA_JITTER", "DLAA_MV", "DLAA_MASKDEBUG", "DLAA_MVDEBUG", "DLAA_OBJMV", "DLAA_SURVEY", "DLAA_TIMES", "DLSS", "DLSS_CHAR_MASK", "DLSS_FIXED_EXPOSURE", "DLSS_PRESET", "DLSS_REACTIVE", "DUMP", "DUMP_BUF", "DUMP_DIR", - "DUMP_LOG", "DYNAMIC_RES", "DYN_DELTA", "EXEFS", "FEEDBACK", "FIXED_RES_PATCH", "FIX_C0", "FIX_ZERO", "FLIGHT", "FLIGHT_AT", "FLIGHT_DIR", "FLIGHT_FRAMES", + "DUMP_LOG", "DYNAMIC_RES", "DYN_DELTA", "EXEFS", "FEEDBACK", "FG_DEBUG", "FG_SHOT", "FG_SHOT_DIR", "FG_TEST", "FG_TIMES", "FIXED_RES_PATCH", "FIX_C0", "FIX_ZERO", "FLIGHT", "FLIGHT_AT", "FLIGHT_DIR", "FLIGHT_FRAMES", "FLIGHT_SPIKE_MS", "FLIGHT_THUMBS", "FLUSH_SPAN", "FOV", "FOV_PATCH", "FPS60", "FPS_LIMIT", "FRAME_GEN", "FRAME_QUEUE", "FREECAM_KEY", "FREECAM_PAD", "FREECAM_PATCH", "FREEZE_IDLE", "FREE_CORES", "FULLSCREEN", "FXAA", diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index 7a418463..b34b2708 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -30,6 +30,7 @@ #include "vk_exec.h" #include "platform/platform.h" #include "vk_dlaa.h" +#include "vk_fg.h" #define WEAK __attribute__((weak)) @@ -124,6 +125,7 @@ static void present_image(const tex_t *t, const int crop[4], uint64_t at_ns, uin PresentJob j = {.image = img->image, .format = img->format, .w = (int)img->width, .h = (int)img->height, .frame = frame, .at = at_ns, .queued = queued, .value = value, .pid = pid}; vk_hdr_present_source(img, &j.image, &j.format); // TOTK_HDR: the float picture the game copied into it, same size + vk_fg_present_swap(&j.image, &j.format); // TOTK_FG_TEST (frame generation checks): another picture in its place if (crop) { j.has_crop = 1; for (int i = 0; i < 4; i++) j.crop[i] = crop[i]; @@ -338,6 +340,7 @@ static void replay_log(uint64_t frame, int total) { void gfx_exit_log(void) { static int32_t done; rec_check_log(); + vk_fg_exit_log(); vk_texture_eager_log(); vk_mem_shadow_read_log(); if (gfx_replay_async() && r_started && !plat_xchg32(&done, 1)) replay_log(game_frames, 1); diff --git a/runtime/vk_dlaa.c b/runtime/vk_dlaa.c index c24104a9..50ea7274 100644 --- a/runtime/vk_dlaa.c +++ b/runtime/vk_dlaa.c @@ -22,7 +22,10 @@ #include "platform/platform.h" #include "vk.h" #include "vk_dlaa.h" +#include "input.h" +#include "nvn_api.h" #include "vk_exec.h" +#include "vk_fg.h" #include "vk_rec.h" #include "dlaa_mv_spv.h" @@ -301,6 +304,15 @@ static struct { int ids_ok; unsigned cfg_seen; // cfg_version when preset was read uint64_t last_eval_ns; // plat_now_ns of the previous evaluation (FrameTimeDeltaInMsec) + // frame generation with DLSS off (vk_fg.cpp): the scene is followed for it alone ("frame generation" below) + int fg; // frame generation follows the scene this frame (vk_fg_wanted), read at each present + int bloom; // this frame: the bloom chain began after the scene ended + int hud, hud_wait, fin, gen; // this frame: the picture before the HUD was taken / hud_t is the scaling pass's + // target and its scaling draw came / the copy into the window texture came / the + // generation ran (vk_dlaa_sub_end) + VkExecTarget hud_t, win; // the output-sized target the scene is scaled into; the window texture + float cam_dr[2]; // the viewport depth range of the draw cam_addr was noted at + int fg_mv_ok; // the frame's camera motion is in the motion image (fg_motion) } D = {.mv_ok = -1, .cfg_seen = ~0u}; #define DLSS_SIZES (sizeof D.sizes / sizeof D.sizes[0]) @@ -308,6 +320,9 @@ const VkTexImage *vk_dlaa_swap_img; VkImageView vk_dlaa_swap_view; static int dl_active(void) { return N.available && !D.failed && D.on; } +// The scene is followed (its G-buffer, light target, end and camera): for DLSS, or with DLSS off for frame generation. +static int track(void) { return dl_active() || D.fg; } +int vk_dlaa_fg_on; // D.fg for vk_exec_draw.c's per-draw test (vk_dlaa_hud) static void dl_fail(const char *what, NgxResult r) { rt_log("dlaa: %s failed (%08x): DLAA off for this run", what, (unsigned)r); @@ -350,29 +365,11 @@ static void free_size(struct Size *e) { // the target) and motion vectors (RG16F at the scene size, render pixels), both kept in GENERAL like every image of the // back end. TotK's dynamic resolution switches between a few sizes; each keeps its feature, the least recently used // goes for a new one. -static struct Size *size_for(VkCommandBuffer cb, uint32_t w, uint32_t h, uint32_t ow, uint32_t oh) { - struct Size *e = NULL, *lru = &D.sizes[0]; - for (int i = 0; i < 5; i++) { - if (D.sizes[i].w == w && D.sizes[i].h == h && D.sizes[i].ow == ow && D.sizes[i].oh == oh) return &D.sizes[i]; - if (!D.sizes[i].out && !e) e = &D.sizes[i]; - if (D.sizes[i].used < lru->used) lru = &D.sizes[i]; - } - if (!e) { - if (D.last == lru) D.last = NULL; - free_size(e = lru); - } - VkImageUsageFlags u = VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | - VK_IMAGE_USAGE_TRANSFER_DST_BIT; - VkImageUsageFlags u8 = VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_SAMPLED_BIT; - if (!image2d(ow, oh, VK_FORMAT_R16G16B16A16_SFLOAT, u, &e->out, &e->out_mem, &e->out_view) || - !image2d(w, h, VK_FORMAT_R16G16_SFLOAT, u | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT, &e->mv, &e->mv_mem, &e->mv_view) || - !image2d(w, h, VK_FORMAT_R8_UNORM, u8, &e->mask, &e->mask_mem, &e->mask_view) || - !image2d(w, h, VK_FORMAT_R8_UNORM, u8, &e->chars[0], &e->chars_mem[0], &e->chars_view[0]) || - !image2d(w, h, VK_FORMAT_R8_UNORM, u8, &e->chars[1], &e->chars_mem[1], &e->chars_view[1])) - return NULL; +// An entry is taken once it has its motion image. out: DLSS will write the output image; frame generation without DLSS +// (fg_motion) needs none, and an entry it made gets one when DLSS first runs at that size. +static void to_general(VkCommandBuffer cb, const VkImage *im, uint32_t n) { VkImageMemoryBarrier2 ib[5]; - VkImage im[5] = {e->out, e->mv, e->mask, e->chars[0], e->chars[1]}; - for (int i = 0; i < 5; i++) + for (uint32_t i = 0; i < n; i++) ib[i] = (VkImageMemoryBarrier2){.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, @@ -381,9 +378,42 @@ static struct Size *size_for(VkCommandBuffer cb, uint32_t w, uint32_t h, uint32_ .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .image = im[i], .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}}; - VkDependencyInfo di = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 5, + VkDependencyInfo di = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = n, .pImageMemoryBarriers = ib}; vkCmdPipelineBarrier2(cb, &di); +} + +static struct Size *size_for(VkCommandBuffer cb, uint32_t w, uint32_t h, uint32_t ow, uint32_t oh, int out) { + VkImageUsageFlags u = VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | + VK_IMAGE_USAGE_TRANSFER_DST_BIT; + VkImageUsageFlags u8 = VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_SAMPLED_BIT; + struct Size *e = NULL, *lru = &D.sizes[0]; + for (size_t i = 0; i < DLSS_SIZES; i++) { + struct Size *z = &D.sizes[i]; + if (z->mv && z->w == w && z->h == h && z->ow == ow && z->oh == oh) { + if (out && !z->out) { + if (!image2d(ow, oh, VK_FORMAT_R16G16B16A16_SFLOAT, u, &z->out, &z->out_mem, &z->out_view)) return NULL; + to_general(cb, &z->out, 1); + } + return z; + } + if (!z->mv && !e) e = z; + if (z->used < lru->used) lru = z; + } + if (!e) { + if (D.last == lru) D.last = NULL; + free_size(e = lru); + } + if ((out && !image2d(ow, oh, VK_FORMAT_R16G16B16A16_SFLOAT, u, &e->out, &e->out_mem, &e->out_view)) || + !image2d(w, h, VK_FORMAT_R16G16_SFLOAT, u | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT, &e->mv, &e->mv_mem, &e->mv_view) || + !image2d(w, h, VK_FORMAT_R8_UNORM, u8, &e->mask, &e->mask_mem, &e->mask_view) || + !image2d(w, h, VK_FORMAT_R8_UNORM, u8, &e->chars[0], &e->chars_mem[0], &e->chars_view[0]) || + !image2d(w, h, VK_FORMAT_R8_UNORM, u8, &e->chars[1], &e->chars_mem[1], &e->chars_view[1])) { + free_size(e); // the slot is free again, not an entry without its images + return NULL; + } + VkImage im[5] = {e->mv, e->mask, e->chars[0], e->chars[1], e->out}; + to_general(cb, im, out ? 5 : 4); e->w = w, e->h = h, e->ow = ow, e->oh = oh; return e; } @@ -566,9 +596,10 @@ static int inv4(const double m[16], double o[16]) { } // The pass, recorded through the recorder like the replay's own commands. 0: no camera this frame (the vectors stay -// zero and DLSS starts a new history). bias > 0 with this frame's G-buffer classes (D.ids) also writes e->mask, the +// zero and DLSS starts a new history). objects: the per-object pass and the TOTK_MV_CHECK hooks after the camera's +// (DLSS only). bias > 0 with this frame's G-buffer classes (D.ids) also writes e->mask, the // bias mask (dlaa_mv.comp); *masked says whether it did. -static int mv_pass(VkExec *ex, struct Size *e, VkImageView zv, float jx, float jy, float bias, int *masked) { +static int mv_pass(VkExec *ex, struct Size *e, VkImageView zv, float jx, float jy, float bias, int *masked, int objects) { *masked = 0; VkCommandBuffer cb = ex->cb; uint32_t w = e->w, h = e->h; @@ -659,6 +690,11 @@ static int mv_pass(VkExec *ex, struct Size *e, VkImageView zv, float jx, float j vkCmdDispatch(cb, (w + 7) / 8, (h + 7) / 8, 1); full_barrier(cb); times_mark(cb, T_MV1); + if (!objects) { + D.mv_frames++; + e->chars_ok = 0; + return 1; + } vk_mvcheck_camera(cb, e->mv, w, h, jx, jy); // TOTK_MV_CHECK (test) // the characters' own motion over the camera's (vk_exec_draw.c, per-object motion) uint64_t c0 = T.on > 0 ? kern_now_ns() : 0; @@ -881,11 +917,12 @@ static void note_camera(const VkExec *ex) { __atomic_store_n(&D.jit_used, j, __ATOMIC_RELAXED); } D.cam_addr = b->addr; + D.cam_dr[0] = ex->depth_range[0], D.cam_dr[1] = ex->depth_range[1]; } } void vk_dlaa_draw(VkExec *ex) { - if (!dl_active() || ended_at(ex)) return; + if (!track() || ended_at(ex)) return; static int survey = -1; // TOTK_DLAA_SURVEY=1 (test): what is drawn into the armed light target if (survey < 0) survey = rt_knob("DLAA_SURVEY", "0"); static uint64_t dfn[16]; // scene draws writing depth, by depth test (reversed depth: GREATER/GEQUAL) @@ -920,7 +957,7 @@ void vk_dlaa_draw(VkExec *ex) { } void vk_dlaa_scan_draw(const VkExec *ex) { - if (dl_active() && !D.ended) note_camera(ex); + if (track() && !D.ended) note_camera(ex); } // The jitter moves on right after DLAA ran, not at the present: the replay can record the next frame's scene before @@ -996,7 +1033,21 @@ static int in_list(const uint64_t *l, int n, uint64_t a) { return 0; } -void vk_dlaa_game_present(void) { __atomic_add_fetch(&CJ.presents, 1, __ATOMIC_RELEASE); } +void vk_dlaa_game_present(void) { + __atomic_add_fetch(&CJ.presents, 1, __ATOMIC_RELEASE); + // frame generation stops while a menu's background (the map, the pause menu; AppMap_00 is open all through play: + // the minimap) or a loading or event fade is up (the game's screens, read on its thread); + // the frame after it starts without history + if (!vk_fg_active()) return; + static const char *const ui[] = {"AppMenuBG_00", "LoadingFade_00", "EventFade_00"}; + static int last = -1; + int m = 0; + for (int i = 0; i < 3; i++) m |= menu_ui_open(ui[i]) << i; + if (m != last) rt_log("fg: screens open: %s%s%s%s", m & 1 ? "menu background (map, pause menu) " : "", m & 2 ? "loading fade " : "", + m & 4 ? "event fade" : "", m ? "" : "none (frame generation may run)"); + last = m; + vk_fg_menu(m != 0); +} // The game's thread, after a 64-byte memcpy: lr is the copy's return address into the game, dst where it copied to, off // the member's offset in the block. f_70000e31d4 takes each offset from the shader's layout of the block (x8 at both @@ -1187,10 +1238,10 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { vk_exec_barrier(ex); // what drew the scene is done (no pass is open here) if (D.preset_dirty) { // another model: features are created with it, so every size gets a new one for (size_t i = 0; i < DLSS_SIZES; i++) - if (D.sizes[i].out) free_size(&D.sizes[i]); + if (D.sizes[i].mv) free_size(&D.sizes[i]); D.last = NULL, D.preset_dirty = 0; } - struct Size *e = size_for(cb, w, h, ow, oh); + struct Size *e = size_for(cb, w, h, ow, oh, 1); if (!e) return dl_fail("work image creation", 0); if (e != D.last) D.reset = 1, D.why[0]++, e->chars_ok = 0; // another size's history is from another moment if (!D.params && !NGX_OK(N.alloc_params(&D.params))) return dl_fail("parameter allocation", 0); @@ -1201,7 +1252,7 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { static int no_mv = -1; if (no_mv < 0) no_mv = !rt_knob("DLAA_MV", "1"); // TOTK_DLAA_MV=0: zero motion (test) int masked = 0; - if (no_mv || !mv_pass(ex, e, zv, jx, jy, char_bias(), &masked)) { + if (no_mv || !mv_pass(ex, e, zv, jx, jy, char_bias(), &masked, 1)) { VkClearColorValue zero = {{0, 0, 0, 0}}; VkImageSubresourceRange all = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}; vkCmdClearColorImage(cb, e->mv, VK_IMAGE_LAYOUT_GENERAL, &zero, 1, &all); @@ -1424,7 +1475,11 @@ static void fx_watch(const VkExec *ex) { int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { const VkExecTarget *c = &ex->color[0], *z = &ex->depth; if (D.fx_open && dl_active()) fx_watch(ex); - if (!dl_active() || D.done) return 0; + // frame generation's picture before the HUD comes after the bloom chain (vk_dlaa_hud): with DLAA before the effects + // the scene ends earlier, and the volumetric fog's composite, a full-screen pass without depth into the light + // target, would pass for the HUD's first + if (D.fg && D.armed && D.ended && !D.bloom && bloom_start(ex, D.color.width)) D.bloom = 1; + if (!track() || D.done) return 0; // The G-buffer: several colour targets with a D32F depth of at least DLAA_SCENE_MIN_W wide. Only the light target drawn after it // arms the frame; at night the game draws into the light target alone with the scene depth before the G-buffer too, // and DLAA then ran at a small pass ahead of the whole scene, which reached the screen jittered. @@ -1480,7 +1535,7 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { c->samples <= 1 && !c->is3d && scene_large(c)) { int first = !D.armed; D.color = *c, D.depth = *z, D.armed = 1, D.nodepth = VK_NULL_HANDLE; - if (!first) return 0; + if (!first || !dl_active()) return 0; *j = (VkExecJob){.kind = VK_JOB_DLSS_MASK, .scene = *c}; // the lit scene, before the transparents (reactive mask) return 1; } @@ -1493,11 +1548,12 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { if (!bloom_start(ex, D.color.width)) return 0; D.end_fence = ex->fence, D.end_pos = ex->pos; __atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE); + D.bloom = 1; unsigned o[2], s[2]; res_sizes(o, s); - if (D.color.width < o[0]) return 0; // wait for the scaling pass + if (dl_active() && D.color.width < o[0]) return 0; // wait for the scaling pass D.done = 1; - *j = (VkExecJob){.kind = VK_JOB_DLSS}; // DLAA in place + *j = (VkExecJob){.kind = VK_JOB_DLSS}; // DLAA in place; without DLSS frame generation's motion, at any size return 1; } // the game's scaling pass: one colour target of the output size, no depth @@ -1512,6 +1568,125 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) { return 0; } +// ---------------------------------------------------------------- frame generation (vk_fg.cpp), DLSS off +// The scene is followed as for DLSS (vk_dlaa_plan: armed on the light target, ended at the bloom chain's first pass), +// without the jitter, the per-object motion, the reactive mask or NGX. Three points of the frame, in the sequential +// replay: +// 1. The scene's end (fg_motion, the VK_JOB_DLSS point): the camera motion vectors at the scene's size (mv_pass with +// the camera alone; Savage Wind's fg_mv.frag does the same reprojection, as dlaa_mv.comp already did here) and the +// SDK's prepare, which reads them with the scene depth while that still holds the scene. +// 2. The picture before the HUD (vk_dlaa_hud, before a draw): measured in a frame dump of 1.4.3 (PR #1089, a 1920x1080 +// scene and output), after the bloom chain the game draws its bloom composite and FXAA into the light target with +// the scene depth attached; then the HUD, in passes on the light target alone without depth, with the HUD's own +// small targets in between; and one last draw copies the light target into the 8-bit window texture. With a scene +// smaller than the output the game's scaling draw comes first, alone, into an output-sized target without depth, +// and the HUD is drawn over that. So, after the scene's end and the bloom chain's start, the picture is what the +// HUD's target holds before the first draw into the light target in a pass without depth (the scene as large as +// the output), before the second draw into the scaling pass's target (smaller), or, in a frame without HUD (a +// cutscene), before the draw into an 8-bit target of the output's size, the copy into the window texture. +// (The rule is PR #1089's vk_dlaa_hud_plan, by xellow78-lgtm.) +// 3. The submission's end after the copy into the window texture (vk_dlaa_sub_end): the optical flow and the +// interpolation, which read the window texture as the picture with the HUD. +static void fg_motion(VkExec *ex) { + const VkExecTarget *t = &D.color, *z = &D.depth; + uint32_t w = t->width, h = t->height; + VkCommandBuffer cb = ex->cb; + uint64_t c0 = kern_now_ns(); + D.done = 1; + vk_exec_barrier(ex); // what drew the scene is done (no pass is open here) + D.jdepth = z->image; // the scene depth note_camera knows the scene's draws by in the next frame + static int said; + if (z->level || z->base_layer || z->format != VK_FORMAT_D32_SFLOAT || z->width != w || z->height != h) { + if (!said) said = 1, rt_log("fg: the scene depth is not the first level and layer of a D32F image of the light " + "target's size (format %d, level %u, layer %u): no frame generation", + (int)z->format, z->level, z->base_layer); + return; + } + struct Size *e = size_for(cb, w, h, w, h, 0); + if (!e) { + if (!said) said = 1, rt_log("fg: no memory for the motion image at %ux%u: no frame generation", w, h); + return; + } + e->used = D.evals + 1; + VkImageView zv = vk_exec_target_view(z->img, z->format, VK_IMAGE_VIEW_TYPE_2D, VK_IMAGE_ASPECT_DEPTH_BIT, z->level, + z->base_layer, 1, 0); + unsigned o[2], sc[2]; + res_sizes(o, sc); + VkFgScene s = {.depth = z->image, .mv = e->mv, .w = w, .h = h, .ow = o[0], .oh = o[1]}; + // the camera as the snapshot holds it (rows 0-10: view, view-projection, projection), read before mv_pass keeps + // this frame's view-projection as last frame's + const float *cam = D.cam_addr ? (const float *)vk_usnap_bytes(D.cam_addr, 11 * 16) : NULL; + int cut = cam ? vk_fg_camera(cam, nvn_depth_minus_w(), D.cam_dr, &s) : 1; + vk_fg_mark(cb, FG_T_MV0); + int masked = 0, have = cam && mv_pass(ex, e, zv, 0.0f, 0.0f, 0.0f, &masked, 0); + if (!have) { // no camera or no last frame: zero motion, and the SDK starts over + VkClearColorValue zero = {{0, 0, 0, 0}}; + VkImageSubresourceRange all = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}; + vkCmdClearColorImage(cb, e->mv, VK_IMAGE_LAYOUT_GENERAL, &zero, 1, &all); + full_barrier(cb); + } + vk_fg_mark(cb, FG_T_MV1); + s.reset = !have || cut; + D.fg_mv_ok = vk_fg_prepare(cb, &s); + ex->recorded += 6; + vk_exec_draw_new_cb(ex); // the passes bound pipelines and descriptors of their own: the next draw binds everything again + vk_fg_cpu(kern_now_ns() - c0); +} + +static int rgba8(VkFormat f) { + return f == VK_FORMAT_R8G8B8A8_SRGB || f == VK_FORMAT_R8G8B8A8_UNORM || f == VK_FORMAT_B8G8R8A8_SRGB || + f == VK_FORMAT_B8G8R8A8_UNORM; +} + +void vk_dlaa_hud(VkExec *ex) { + if (!D.fg_mv_ok || !D.armed || !D.ended || !D.bloom || D.fin || ex->chunked) return; + const VkExecTarget *c = &ex->color[0]; + if (ex->pass_ncolor != 1 || !c->image || ex->depth.image || c->level || c->is3d || c->samples > 1) return; + unsigned o[2], sc[2]; + res_sizes(o, sc); + if (c->width != o[0] || c->height != o[1]) return; + if (D.hud) { // after the picture: the copy into the window texture + if (rgba8(c->format)) D.win = *c, D.fin = 1; + return; + } + int scaled = D.color.width < o[0]; + const VkExecTarget *pic = scaled ? &D.hud_t : &D.color; + if (scene_color(c->format)) { + if (scaled && !D.hud_wait) { // the scaling draw + D.hud_t = *c, D.hud_wait = 1; + return; + } + if (c->image != pic->image) return; + } else if (!rgba8(c->format) || (scaled && !D.hud_wait) || D.color.width > o[0]) { + return; + } + D.hud = 1; + if (rgba8(c->format)) D.win = *c, D.fin = 1; // no HUD this frame: this draw is the copy + uint64_t c0 = kern_now_ns(); + VkExecTarget p = *pic; + vk_exec_pass_end(ex); + vk_exec_barrier(ex); // what drew the picture is done + vk_fg_hudless(ex->cb, p.image, p.format, p.level, p.base_layer, p.width, p.height); + ex->recorded += 4; + ex->dirty = VK_EXEC_DIRTY_ALL; + vk_exec_pass_begin(ex); + vk_fg_cpu(kern_now_ns() - c0); +} + +void vk_dlaa_sub_end(VkExec *ex) { + if (!D.fin || D.gen || ex->chunked) return; + D.gen = 1; + uint64_t c0 = kern_now_ns(); + vk_exec_barrier(ex); // the window texture is drawn + const VkExecTarget *w = &D.win; + if (!w->level && !w->base_layer) + vk_fg_generate(ex->cb, ex->fence, __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED) + 1, w->image, w->format, + w->width, w->height); + ex->recorded += 4; + vk_exec_draw_new_cb(ex); // the SDK bound pipelines and descriptor sets of its own + vk_fg_cpu(kern_now_ns() - c0); +} + // The scan resolved its targets without making entries (vk_texture_revive); a record made on a chunk takes their images // now, as the scan's pointers may be stale by then. static void target_now(VkExecTarget *t) { @@ -1526,6 +1701,7 @@ void vk_dlaa_exec(VkExec *ex, const VkExecJob *j) { if (D.ids_ok) target_now(&D.ids); } if (j->kind == VK_JOB_DLSS_MASK) return mask_snapshot(ex, &t); + if (!dl_active()) return fg_motion(ex); if (!t.tex) return evaluate(ex, NULL); D.up = 1; evaluate(ex, &t); @@ -1575,9 +1751,13 @@ void vk_dlaa_frame(void) { dlss_mode(); preset_check(); int on = N.available && cfg_bool("TOTK_DLAA"); - if (on != D.on) { - rt_log("dlaa: %s", on ? "on" : "off"); - D.on = on; + vk_fg_frame(); + int fg = vk_fg_wanted(on && !D.failed); + D.bloom = D.hud = D.hud_wait = D.fin = D.gen = D.fg_mv_ok = 0; + if (on != D.on || fg != D.fg) { + if (on != D.on) rt_log("dlaa: %s", on ? "on" : "off"); + if (fg != D.fg) rt_log("dlaa: the scene is %s for frame generation", fg ? "followed" : "no longer followed"); + D.on = on, D.fg = vk_dlaa_fg_on = fg; // either way the next DLAA frame starts a new history from an unjittered frame D.reset = 1, D.cam_prev_ok = 0, D.jit_ok = D.jit_used = D.armed = D.done = D.gbuf = D.ended = D.up = 0; D.fx_seen = D.fx_open = D.fx_bad = 0, D.fx_steady = 0; @@ -1585,8 +1765,8 @@ void vk_dlaa_frame(void) { D.hold = 0; return; } - if (!dl_active()) return; - D.fx_on = cfg_bool("TOTK_DLAA_EFFECTS_AFTER"); + if (!track()) return; + D.fx_on = dl_active() && cfg_bool("TOTK_DLAA_EFFECTS_AFTER"); // DLSS's point alone if (!D.done) D.frames_missed++; else D.fx_steady = D.fx_seen ? D.fx_steady + (D.fx_steady < FX_STEADY) : 0; D.fx_seen = D.fx_open = D.fx_bad = 0; diff --git a/runtime/vk_dlaa.h b/runtime/vk_dlaa.h index 1275c27e..c9cd320c 100644 --- a/runtime/vk_dlaa.h +++ b/runtime/vk_dlaa.h @@ -25,6 +25,12 @@ void vk_dlaa_pass(VkExec *ex); int vk_dlaa_plan(const VkExec *ex, VkExecJob *j); void vk_dlaa_exec(VkExec *ex, const VkExecJob *j); void vk_dlaa_frame(void); +// Frame generation without DLSS (vk_dlaa.c, "frame generation"), sequential replay: vk_dlaa_hud before each draw once +// its pass has begun (vk_exec_draw.c prepare_draw, while vk_dlaa_fg_on), vk_dlaa_sub_end at the end of each +// submission's replay with no pass open (vk_exec.c vk_exec_record). +extern int vk_dlaa_fg_on; +void vk_dlaa_hud(VkExec *ex); +void vk_dlaa_sub_end(VkExec *ex); // vk_exec_draw.c, each draw before its fixed state is recorded: notes the scene camera block (vk_dlaa_draw), and moves // the viewport of the scene's draws by the frame's jitter (vk_dlaa_viewport, when the viewport is recorded). void vk_dlaa_draw(VkExec *ex); diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index 0ef4a57c..c16812cb 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -1787,6 +1787,7 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t for (int i = 0; i < n; i++) walk(ex, nvn_op_first(handles[i]), 0, NULL); } vk_exec_pass_end(ex); + if (vk_dlaa_fg_on) vk_dlaa_sub_end(ex); // frame generation after the frame's window texture (vk_dlaa.c) x_cur = NULL; if (ex->recorded) vk_exec_barrier(ex); // later submissions see this one's writes if (n) ex->recorded += vk_mem_mirror_end(ex->cb, fence), vk_usnap_end(fence); // VRAM mirror bytes the GPU wrote go back to guest memory diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index a87bd64c..42a2ff86 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -3340,6 +3340,7 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) { } if (!ex->in_pass) ex->dirty |= VK_EXEC_DIRTY_VIEWPORT | VK_EXEC_DIRTY_SCISSOR; vk_exec_pass_begin(ex); + if (vk_dlaa_fg_on) vk_dlaa_hud(ex); // frame generation: the picture before the HUD's first draw (vk_dlaa.c) VkRect2D sc = vk_exec_scissor(ex); if (!sc.extent.width || !sc.extent.height) return 0; if (!ex->prog) return warn_once(W_PROGRAM, "a draw without a bound program is skipped"), 0; diff --git a/runtime/vk_fg.cpp b/runtime/vk_fg.cpp index a6a7abe7..27ec3360 100644 --- a/runtime/vk_fg.cpp +++ b/runtime/vk_fg.cpp @@ -3,15 +3,27 @@ // interpolation swapchain (Windows only, and it would own the present). // // Ported from Savage Wind (https://github.com/Revivedx/SavageWindRecomp @ bf6146c14c37fee8781a601e5a544b93dff6d75e, -// MIT), runtime/nvn/vulkan/framegen.cpp: the blob lookup, the backend set-up and the contexts as it makes them, which is -// what the SDK's ffx-api provider (ffx_provider_framegeneration.cpp) does: a shared backend context for the resources -// the two effects exchange (optical flow vectors, scene change detection, dilated depth and motion vectors, the -// previous frame's reconstructed depth, two of each that alternate), the optical flow context and the frame -// interpolation context on a second backend interface. This stage only creates and destroys them; nothing is -// dispatched yet. +// MIT), runtime/nvn/vulkan/framegen.cpp: the blob lookup, the backend set-up, the contexts as it makes them (what the +// SDK's ffx-api provider, ffx_provider_framegeneration.cpp, does: a shared backend context for the resources the two +// effects exchange, the optical flow context and the frame interpolation context on a second backend interface) and +// the parameters of fg_dispatch's three calls; its exec.c vk_fg_generate for the camera, the cut test and the HUD-less +// copy's encoding. +// +// Per frame, all in the replay's command buffer (vk_dlaa.c finds the places, with DLSS off): +// 1. vk_fg_prepare at the scene's end: the SDK dilates the scene's depth and the camera motion vectors (made there by +// vk_dlaa.c mv_pass, at the scene's size), and reconstructs the previous frame's depth from them. +// 2. vk_fg_hudless before the HUD's first draw: the game's float picture, blitted into an R8G8B8A8_SRGB image (the +// blit encodes as the game's last draw encodes the picture into its sRGB window texture) and copied byte for byte +// into an RGBA8 UNORM image, which the SDK reads. Its bytes then equal the window texture's wherever the HUD is not, +// which is how the SDK tells the HUD apart (it compares the two) and keeps it out of the interpolation. +// 3. vk_fg_generate at the end of the submission that drew the window texture: the optical flow over the HUD-less +// picture, then the interpolation between the last frame and this one, with this frame's window texture as the +// picture with the HUD, into an RGBA8 image holding bytes as the window texture does (sRGB transfer function). +// Presenting it is not done here (TOTK_FG_TEST shows it in place of the frame, for checks only). #include "vk_fg.h" #include +#include #include #include #include @@ -58,14 +70,62 @@ FfxErrorCode ffxSetFrameGenerationConfigToSwapchainVK(FfxFrameGenerationConfig c enum { R_OF_VECTOR, R_OF_SCD, R_DILATED_DEPTH, R_DILATED_MV = R_DILATED_DEPTH + 2, R_PREV_DEPTH = R_DILATED_MV + 2, R_COUNT = R_PREV_DEPTH + 2 }; +// What the contexts are made for; a change makes them again (all 32-bit fields: compared with memcmp) +typedef struct { + uint32_t display_w, display_h; // the window texture's (the interpolated frame's) + uint32_t render_w, render_h; // the scene's (depth and motion vectors) + VkFormat color_format; // the window texture's bytes (its UNORM twin) + uint32_t inverted, infinite; +} FgConfig; + +struct Img { + VkImage img; + VmaAllocation mem; +}; + +#define TSLOTS 8u // frames of timestamps in flight: a slot is read when it comes round again +#define CPU_RING 65536u // frames of replay-thread times kept for the median and p99 +#define LOG_EVERY 600u // generated frames between two summary lines + static struct { - bool live; + bool live, failed; + FgConfig cfg; FfxInterface shared, fi_if; void *scratch_shared, *scratch_fi; uint32_t shared_id; FfxOpticalflowContext of; FfxFrameInterpolationContext fi; FfxResourceInternal res[R_COUNT]; + unsigned toggle; + uint64_t frame_id, last_prep_ns, made; + // settings read at start (vk_fg_init) + int workers, sync_ahead, hdr, test, debug, times; + uint64_t shot_at; // TOTK_FG_SHOT: the first present shot, or (shot_w) the first after that world time + int64_t shot_w; + char shot_dir[260]; + unsigned said; // vk_fg_wanted's reasons logged (bits) + volatile int menu; // vk_fg_menu + // this frame + bool prepared, hl_ok, gen_ok, chain, reset; + float near_m, far_m, fov_y, dt_ms; + VkFormat win_fmt; // the window texture's format as last seen (UNORM twin) + int k; // the pair of hudless/out images this frame writes + VkImage show_out, show_hl; // the frame just ended (vk_fg_frame), for vk_fg_present_swap + // images + uint32_t iw, ih; + Img hl_srgb, hl[2], out[2]; + // timing + VkQueryPool qp; + uint32_t qslot, qmask[TSLOTS]; + double gpu[4]; // ms summed: camera motion, prepare, hudless copy, optical flow + interpolation + double gpu_run[4]; + uint64_t timed, timed_run; + uint64_t cpu_frame; // ns this frame + uint32_t *cpu_us; // per frame with frame generation work, microseconds + uint64_t ncpu; + uint64_t gens, resets, skipped, gens_logged; + uint64_t drain_ns, sdk_ns, split_frames; // since the last summary: recorder drains, the SDK's own recording + uint64_t drain_run, sdk_run, prep_ns, prep_run; // prep: the prepare's share of sdk } g; #define TRY(x) do { FfxErrorCode e_ = (x); if (e_ != FFX_OK) { rt_log("fg: %s failed: %#x", #x, (unsigned)e_); return false; } } while (0) @@ -112,35 +172,62 @@ static bool make_backend(VkCtx *c, FfxInterface *i, void **scratch, size_t conte return true; } -extern "C" void vk_fg_release(void) { - if (!g.live && !g.scratch_shared && !g.scratch_fi) return; - VkCtx *c = vk_ctx(); - if (c && c->device) vkDeviceWaitIdle(c->device); +// The SDK's objects (not this file's images): the device is idle or the contexts were never used. +static void sdk_destroy(void) { if (g.live) { for (int i = 0; i < R_COUNT; i++) g.shared.fpDestroyResource(&g.shared, g.res[i], g.shared_id); ffxFrameInterpolationContextDestroy(&g.fi); ffxOpticalflowContextDestroy(&g.of); g.shared.fpDestroyBackendContext(&g.shared, g.shared_id); - rt_log("fg: frame interpolation contexts destroyed"); } scratch_free(g.scratch_shared); scratch_free(g.scratch_fi); + g.scratch_shared = g.scratch_fi = nullptr; + memset(&g.of, 0, sizeof g.of); + memset(&g.fi, 0, sizeof g.fi); + memset(g.res, 0, sizeof g.res); + g.live = false; +} + +static void img_free(VkCtx *c, Img *m) { + if (m->img) vmaDestroyImage(c->vma, m->img, m->mem); + *m = Img{}; +} + +static void times_log(const char *what, const double *gpu, uint64_t timed); +static void cpu_log(const char *what, uint64_t from); + +extern "C" void vk_fg_release(void) { + VkCtx *c = vk_ctx(); + if (!c || !c->device) return; + if (!g.live && !g.scratch_shared && !g.hl_srgb.img && !g.qp) return; + vkDeviceWaitIdle(c->device); + vk_fg_exit_log(); + if (g.live) rt_log("fg: frame interpolation contexts destroyed"); + sdk_destroy(); + img_free(c, &g.hl_srgb); + for (int i = 0; i < 2; i++) img_free(c, &g.hl[i]), img_free(c, &g.out[i]); + if (g.qp) vkDestroyQueryPool(c->device, g.qp, NULL); + free(g.cpu_us); memset(&g, 0, sizeof g); } -static bool create(VkCtx *c, uint32_t w, uint32_t h) { +static bool create(VkCtx *c, const FgConfig *cf) { + g.cfg = *cf; if (!make_backend(c, &g.shared, &g.scratch_shared, 1) || !make_backend(c, &g.fi_if, &g.scratch_fi, 2)) return false; TRY(g.shared.fpCreateBackendContext(&g.shared, FFX_EFFECT_SHAREDAPIBACKEND, nullptr, &g.shared_id)); FfxOpticalflowContextDescription od = {}; od.backendInterface = g.fi_if; - od.resolution = {w, h}; + od.resolution = {cf->display_w, cf->display_h}; TRY(ffxOpticalflowContextCreate(&g.of, &od)); FfxFrameInterpolationContextDescription fd = {}; fd.backendInterface = g.fi_if; - fd.maxRenderSize = {w, h}; - fd.displaySize = {w, h}; - fd.backBufferFormat = ffxGetSurfaceFormatVK(VK_FORMAT_R8G8B8A8_UNORM); - fd.previousInterpolationSourceFormat = ffxGetSurfaceFormatVK(VK_FORMAT_R8G8B8A8_UNORM); + fd.flags = (cf->inverted ? FFX_FRAMEINTERPOLATION_ENABLE_DEPTH_INVERTED : 0) | + (cf->infinite ? FFX_FRAMEINTERPOLATION_ENABLE_DEPTH_INFINITE : 0); + fd.maxRenderSize = {cf->render_w, cf->render_h}; + fd.displaySize = {cf->display_w, cf->display_h}; + fd.backBufferFormat = ffxGetSurfaceFormatVK(cf->color_format); + fd.previousInterpolationSourceFormat = ffxGetSurfaceFormatVK(VK_FORMAT_R8G8B8A8_UNORM); // the HUD-less copy's TRY(ffxFrameInterpolationContextCreate(&g.fi, &fd)); FfxOpticalflowSharedResourceDescriptions ord = {}; TRY(ffxOpticalflowGetSharedResourceDescriptions(&g.of, &ord)); @@ -157,21 +244,534 @@ static bool create(VkCtx *c, uint32_t w, uint32_t h) { return true; } +static int knob_int(const char *key) { + const char *v = cfg_get(key); + return v && *v ? atoi(v) : 0; +} + extern "C" void vk_fg_init(void) { if (!cfg_bool("TOTK_FRAME_GEN")) return; VkCtx *c = vk_ctx(); if (!c) return; - // The size is a placeholder until the dispatch stage sizes the contexts from the real output and scene. + // The first contexts at the output size from the settings; vk_fg_prepare makes them again for the real scene uint32_t w = 1920, h = 1080; sscanf(cfg_value("TOTK_OUTPUT_RES"), "%ux%u", &w, &h); if (w < 64 || h < 64 || w > 16384 || h > 16384) w = 1920, h = 1080; - if (!create(c, w, h)) { + FgConfig cf = {w, h, w, h, VK_FORMAT_R8G8B8A8_UNORM, 0, 0}; + if (!create(c, &cf)) { rt_log("fg: frame interpolation unavailable (the contexts could not be made)"); g.live = false; // (a half-made set is not destroyed through its context functions: only its memory is freed) - vk_fg_release(); + sdk_destroy(); + g.failed = true; return; } - rt_log("fg: FSR 3 frame interpolation contexts created at %ux%u (FidelityFX SDK 1.1.4)", w, h); + g.win_fmt = VK_FORMAT_R8G8B8A8_UNORM; + g.workers = cfg_int("TOTK_REPLAY_WORKERS"); + g.sync_ahead = cfg_bool("TOTK_SYNC_AHEAD"); + g.hdr = vk_hdr_wanted(); + g.test = knob_int("TOTK_FG_TEST"); + g.debug = knob_int("TOTK_FG_DEBUG"); + g.times = !cfg_get("TOTK_FG_TIMES") || knob_int("TOTK_FG_TIMES"); + const char *sh = cfg_get("TOTK_FG_SHOT"); + if (sh && (sh[0] == 'w' || sh[0] == 'W')) g.shot_w = (int64_t)(atof(sh + 1) * 1e6); + else g.shot_at = (uint64_t)knob_int("TOTK_FG_SHOT"); + const char *d = cfg_get("TOTK_FG_SHOT_DIR"); + snprintf(g.shot_dir, sizeof g.shot_dir, "%s", d && *d ? d : "."); + g.cpu_us = (uint32_t *)calloc(CPU_RING, sizeof *g.cpu_us); + if (g.times) { + VkQueryPoolCreateInfo qi = {VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO, NULL, 0, VK_QUERY_TYPE_TIMESTAMP, + TSLOTS * FG_T_N, 0}; + if (vkCreateQueryPool(c->device, &qi, NULL, &g.qp) != VK_SUCCESS) g.qp = VK_NULL_HANDLE; + } + rt_log("fg: FSR 3 frame interpolation contexts created at %ux%u (FidelityFX SDK 1.1.4)%s%s", w, h, + g.test == 1 ? "; TOTK_FG_TEST=1: the generated frame is shown in place of the game's" + : g.test == 2 ? "; TOTK_FG_TEST=2: the picture before the HUD is shown in place of the game's" : "", + g.debug ? "; TOTK_FG_DEBUG: the SDK's debug view" : ""); } extern "C" int vk_fg_active(void) { return g.live; } + +extern "C" void vk_fg_menu(int open) { __atomic_store_n(&g.menu, open, __ATOMIC_RELAXED); } + +extern "C" int vk_fg_wanted(int dlss) { + if (!g.live || g.failed) return 0; + // the inputs are this frame's, recorded in replay order: one frame in flight more or chunks recording side by + // side are not handled yet + int why = dlss ? 1 : g.workers > 0 ? 2 : g.sync_ahead ? 3 : g.hdr ? 4 : 0; + if (why) { + if (!(g.said & 1u << why)) { + g.said |= 1u << why; + static const char *const r[] = {"", "DLSS runs (dlaa = 1)", "the replay records in chunks (replay_workers > 0)", + "sync_ahead = 1", "HDR output (hdr = 1)"}; + rt_log("fg: frame generation off while %s", r[why]); + } + return 0; + } + return !__atomic_load_n(&g.menu, __ATOMIC_RELAXED); +} + +// ---------------------------------------------------------------- the camera +static struct { + bool have, logged; + float pos[3], fwd[3]; + int inverted, infinite; +} cam; + +extern "C" int vk_fg_camera(const float *r, int minus_w, const float dr[2], VkFgScene *s) { + // rows 0-2: the view, 3x4 by rows (right, up, back; the translation in .w); rows 7-10: the projection + const float *v0 = r, *v1 = r + 4, *v2 = r + 8, *py = r + 8 * 4, *pz = r + 9 * 4, *pw = r + 10 * 4; + for (int k = 0; k < 3; k++) { + s->right[k] = v0[k], s->up[k] = v1[k], s->fwd[k] = -v2[k]; + s->pos[k] = -(v0[k] * v0[3] + v1[k] * v1[3] + v2[k] * v2[3]); + } + s->fov_y = 2.0f * atanf(1.0f / (py[1] > 1e-3f ? py[1] : 1.0f)); + // A point on the view axis at distance t: its clip z and w, the depth before the viewport's range (d01) and the + // device depth the depth image holds + auto d01 = [&](double t) { + double zc = -pz[2] * t + pz[3], wc = -pw[2] * t + pw[3]; + double z = wc != 0 ? zc / wc : 0; + return minus_w ? z * 0.5 + 0.5 : z; + }; + auto dev = [&](double t) { return dr[0] + (dr[1] - dr[0]) * d01(t); }; + int inv01 = d01(1.0) > d01(100.0); + s->inverted = dev(1.0) > dev(100.0); + // the planes: where d01 reaches its near and far ends, in clip z: t = (tz * pw.w - pz.w) / (tz * pw.z - pz.z) + auto plane = [&](double end) { + double tz = minus_w ? end * 2 - 1 : end, den = tz * pw[2] - pz[2]; + return fabs(den) > 1e-12 ? (tz * pw[3] - pz[3]) / den : -1.0; + }; + double tn = plane(inv01 ? 1.0 : 0.0), tf = plane(inv01 ? 0.0 : 1.0); + s->infinite = !(tf > 0 && tf < 1e7); + s->near_m = tn > 0 ? (float)tn : 0.1f; + s->far_m = s->infinite ? 1e7f : (float)tf; + if (!cam.logged || s->inverted != cam.inverted || s->infinite != cam.infinite) { + cam.logged = true, cam.inverted = s->inverted, cam.infinite = s->infinite; + rt_log("fg: camera from the scene block: near %.4g m, far %s%.6g m, vertical field of view %.1f degrees; depth %s " + "(NVN depth mode %s, viewport depth range %g..%g; device depth at 1 m %.6f, 100 m %.6f, 10 km %.6f); " + "projection z row %g %g %g %g, w row %g %g %g %g; view axes |right| %.3f |up| %.3f |back| %.3f; camera " + "at %.1f %.1f %.1f", s->near_m, s->infinite ? "infinite, " : "", tf, s->fov_y * 57.29578f, + s->inverted ? "inverted (1 near)" : "standard (0 near)", minus_w ? "-w..w" : "0..w", dr[0], dr[1], + dev(1.0), dev(100.0), dev(10000.0), pz[0], pz[1], pz[2], pz[3], pw[0], pw[1], pw[2], pw[3], + sqrtf(v0[0] * v0[0] + v0[1] * v0[1] + v0[2] * v0[2]), sqrtf(v1[0] * v1[0] + v1[1] * v1[1] + v1[2] * v1[2]), + sqrtf(v2[0] * v2[0] + v2[1] * v2[1] + v2[2] * v2[2]), s->pos[0], s->pos[1], s->pos[2]); + if (!minus_w || fabsf(dr[0]) > 1e-6f || fabsf(dr[1] - 1.0f) > 1e-6f) + rt_log("fg: the camera motion pass (shaders/dlaa_mv.comp) takes depth as z_ndc * 0.5 + 0.5 over 0..1: this " + "depth mode or range does not fit it, its vectors are off for anything the camera moves past"); + } + // a cut: no history (a jump of over 10 m, or the view turned over ~35 degrees in a frame) + float dp = 0, dot = 0; + for (int k = 0; k < 3; k++) + dp += (s->pos[k] - cam.pos[k]) * (s->pos[k] - cam.pos[k]), dot += s->fwd[k] * cam.fwd[k]; + int cut = !cam.have || dp > 100.0f || dot < 0.82f; + cam.have = true; + memcpy(cam.pos, s->pos, sizeof cam.pos), memcpy(cam.fwd, s->fwd, sizeof cam.fwd); + return cut; +} + +// ---------------------------------------------------------------- timing +static void times_log(const char *what, const double *gpu, uint64_t timed) { + double n = timed ? (double)timed : 1.0; + rt_log("fg times %s: gpu camera motion %.3f ms, prepare %.3f ms, HUD-less copy %.3f ms, optical flow + " + "interpolation %.3f ms (%llu frames timed)", what, gpu[0] / n, gpu[1] / n, gpu[2] / n, gpu[3] / n, + (unsigned long long)timed); +} + +static int cmp_u32(const void *a, const void *b) { + uint32_t x = *(const uint32_t *)a, y = *(const uint32_t *)b; + return x < y ? -1 : x > y; +} + +static void cpu_log(const char *what, uint64_t from) { + uint64_t n = g.ncpu - from; + if (!n || !g.cpu_us) return; + if (n > CPU_RING) n = CPU_RING, from = g.ncpu - n; + uint32_t *v = (uint32_t *)malloc(n * sizeof *v); + if (!v) return; + for (uint64_t i = 0; i < n; i++) v[i] = g.cpu_us[(from + i) % CPU_RING]; + qsort(v, n, sizeof *v, cmp_u32); + rt_log("fg cpu %s: replay-thread recording %.3f ms median, %.3f ms p99, %.3f ms max a frame (%llu frames)", what, + v[n / 2] / 1e3, v[n * 99 / 100] / 1e3, v[n - 1] / 1e3, (unsigned long long)n); + free(v); +} + +extern "C" void vk_fg_mark(VkCommandBuffer cb, int which) { + if (!g.qp) return; + if (which == FG_T_MV0) { // the frame's slot: its last frame's results are read, then it is reset + g.qslot = (g.qslot + 1) % TSLOTS; + if (g.qmask[g.qslot] == (1u << FG_T_N) - 1) { + uint64_t q[FG_T_N]; + if (vkGetQueryPoolResults(vk_ctx()->device, g.qp, g.qslot * FG_T_N, FG_T_N, sizeof q, q, sizeof q[0], + VK_QUERY_RESULT_64_BIT) == VK_SUCCESS) { + double ms = vk_ctx()->props.limits.timestampPeriod / 1e6; + double d[4] = {(double)(q[FG_T_MV1] - q[FG_T_MV0]) * ms, (double)(q[FG_T_PREP1] - q[FG_T_PREP0]) * ms, + (double)(q[FG_T_HL1] - q[FG_T_HL0]) * ms, (double)(q[FG_T_FI1] - q[FG_T_FI0]) * ms}; + for (int i = 0; i < 4; i++) g.gpu[i] += d[i], g.gpu_run[i] += d[i]; + g.timed++, g.timed_run++; + } + } + g.qmask[g.qslot] = 0; + vkCmdResetQueryPool(cb, g.qp, g.qslot * FG_T_N, FG_T_N); + } + vkCmdWriteTimestamp2(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, g.qp, g.qslot * FG_T_N + (uint32_t)which); + g.qmask[g.qslot] |= 1u << which; +} + +extern "C" void vk_fg_cpu(uint64_t ns) { g.cpu_frame += ns; } + +// ---------------------------------------------------------------- images +static void full_barrier(VkCommandBuffer cb) { + VkMemoryBarrier2 mb = {VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, NULL, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT}; + VkDependencyInfo di = {}; + di.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, di.memoryBarrierCount = 1, di.pMemoryBarriers = &mb; + vkCmdPipelineBarrier2(cb, &di); +} + +static bool img_make(VkCtx *c, VkCommandBuffer cb, Img *m, uint32_t w, uint32_t h, VkFormat f, VkImageUsageFlags u) { + VkImageCreateInfo ii = {}; + ii.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, ii.imageType = VK_IMAGE_TYPE_2D, ii.format = f; + ii.extent = {w, h, 1}, ii.mipLevels = 1, ii.arrayLayers = 1, ii.samples = VK_SAMPLE_COUNT_1_BIT; + ii.tiling = VK_IMAGE_TILING_OPTIMAL, ii.usage = u, ii.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; + VmaAllocationCreateInfo ai = {}; + ai.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE; + if (vmaCreateImage(c->vma, &ii, &ai, &m->img, &m->mem, NULL) != VK_SUCCESS) return m->img = VK_NULL_HANDLE, false; + VkImageMemoryBarrier2 b = {}; + b.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2; + b.srcStageMask = b.dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT; + b.dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT; + b.oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, b.newLayout = VK_IMAGE_LAYOUT_GENERAL; + b.srcQueueFamilyIndex = b.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED; + b.image = m->img, b.subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}; + VkDependencyInfo di = {}; + di.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, di.imageMemoryBarrierCount = 1, di.pImageMemoryBarriers = &b; + vkCmdPipelineBarrier2(cb, &di); + return true; +} + +// The HUD-less copies and the generated frames at w x h, two of each (the present of the frame before may still read +// the other pair). Another size: the old ones go once every present and submission before is done. +static bool images_for(VkCommandBuffer cb, uint32_t w, uint32_t h) { + if (g.hl_srgb.img && g.iw == w && g.ih == h) return true; + VkCtx *c = vk_ctx(); + if (g.hl_srgb.img) { + vk_present_drain(); + vk_queue_lock(); + vkDeviceWaitIdle(c->device); + vk_queue_unlock(); + g.show_out = g.show_hl = VK_NULL_HANDLE; + } + img_free(c, &g.hl_srgb); + for (int i = 0; i < 2; i++) img_free(c, &g.hl[i]), img_free(c, &g.out[i]); + const VkImageUsageFlags t = VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT; + bool ok = img_make(c, cb, &g.hl_srgb, w, h, VK_FORMAT_R8G8B8A8_SRGB, t); + for (int i = 0; i < 2 && ok; i++) + ok = img_make(c, cb, &g.hl[i], w, h, VK_FORMAT_R8G8B8A8_UNORM, t | VK_IMAGE_USAGE_SAMPLED_BIT) && + img_make(c, cb, &g.out[i], w, h, VK_FORMAT_R8G8B8A8_UNORM, + t | VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_STORAGE_BIT); + if (!ok) { + rt_log("fg: no memory for the %ux%u pictures: frame generation off for this run", w, h); + g.failed = true; + return false; + } + g.iw = w, g.ih = h; + return true; +} + +static FfxResource res(VkImage img, VkFormat f, uint32_t w, uint32_t h, bool write, const wchar_t *name) { + VkImageCreateInfo ci = {}; + ci.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO; + ci.imageType = VK_IMAGE_TYPE_2D; + ci.format = f; + ci.extent = {w, h, 1}; + ci.mipLevels = 1; + ci.arrayLayers = 1; + ci.samples = VK_SAMPLE_COUNT_1_BIT; + ci.usage = VK_IMAGE_USAGE_SAMPLED_BIT | (write ? VK_IMAGE_USAGE_STORAGE_BIT : 0); + // (every image of the runtime is in GENERAL: the backend takes both states as GENERAL and leaves them so) + return ffxGetResourceVK((void *)img, ffxGetImageResourceDescriptionVK(img, ci, write ? FFX_RESOURCE_USAGE_UAV + : FFX_RESOURCE_USAGE_READ_ONLY), name, write ? FFX_RESOURCE_STATE_UNORDERED_ACCESS + : FFX_RESOURCE_STATE_GENERIC_READ); +} + +static FfxResource shared(int i) { return g.shared.fpGetResource(&g.shared, g.res[i]); } + +static VkFormat unorm_of(VkFormat f) { // (the bytes as written: an sRGB image is interpolated as stored) + return f == VK_FORMAT_R8G8B8A8_SRGB ? VK_FORMAT_R8G8B8A8_UNORM : f == VK_FORMAT_B8G8R8A8_SRGB ? VK_FORMAT_B8G8R8A8_UNORM : f; +} + +// ---------------------------------------------------------------- the frame +extern "C" int vk_fg_prepare(VkCommandBuffer cb, const VkFgScene *s) { + g.prepared = false; + if (g.failed || !g.scratch_shared) return 0; + FgConfig want = {s->ow, s->oh, s->w, s->h, g.win_fmt, (uint32_t)s->inverted, (uint32_t)s->infinite}; + if (!g.live || memcmp(&g.cfg, &want, sizeof want)) { + // what the old contexts recorded is queued or submitted: every submission done before they go + VkCtx *c = vk_ctx(); + vk_queue_lock(); + vkDeviceWaitIdle(c->device); + vk_queue_unlock(); + sdk_destroy(); + if (!create(c, &want)) { + rt_log("fg: the contexts could not be made again: frame generation off for this run"); + sdk_destroy(); + g.failed = true; + return 0; + } + g.chain = false; + rt_log("fg: contexts %llu made for a %ux%u output from a %ux%u scene, %s depth%s, window texture format %d", + (unsigned long long)++g.made, want.display_w, want.display_h, want.render_w, want.render_h, + want.inverted ? "inverted" : "standard", want.infinite ? " without a far plane" : "", (int)want.color_format); + } + uint64_t now = kern_now_ns(); + double dt = g.last_prep_ns ? (double)(now - g.last_prep_ns) / 1e6 : 0.0; + g.last_prep_ns = now; + g.dt_ms = dt > 0.0 && dt < 100.0 ? (float)dt : 16.67f; + g.near_m = s->near_m, g.far_m = s->far_m, g.fov_y = s->fov_y; + g.toggle ^= 1; + FfxFrameInterpolationPrepareDescription pd = {}; + pd.commandList = ffxGetCommandListVK(cb); + pd.renderSize = {s->w, s->h}; + // the vectors are last frame's window position minus this frame's in scene pixels, y down: the SDK's convention + // at a scale of one (its constant divides by the render size); no jitter without an upscaler + pd.motionVectorScale = {1.0f, 1.0f}; + pd.frameTimeDelta = g.dt_ms; + pd.cameraNear = s->near_m; + pd.cameraFar = s->far_m; + pd.viewSpaceToMetersFactor = 1; // (TotK's units are metres) + pd.cameraFovAngleVertical = s->fov_y; + pd.depth = res(s->depth, VK_FORMAT_D32_SFLOAT, s->w, s->h, false, L"FG_Depth"); + pd.motionVectors = res(s->mv, VK_FORMAT_R16G16_SFLOAT, s->w, s->h, false, L"FG_MotionVectors"); + pd.frameID = ++g.frame_id; + pd.dilatedDepth = shared(R_DILATED_DEPTH + g.toggle); + pd.dilatedMotionVectors = shared(R_DILATED_MV + g.toggle); + pd.reconstructedPrevDepth = shared(R_PREV_DEPTH + g.toggle); + memcpy(pd.cameraPosition, s->pos, sizeof pd.cameraPosition); + memcpy(pd.cameraUp, s->up, sizeof pd.cameraUp); + memcpy(pd.cameraRight, s->right, sizeof pd.cameraRight); + memcpy(pd.cameraForward, s->fwd, sizeof pd.cameraForward); + vk_fg_mark(cb, FG_T_PREP0); + uint64_t t0 = kern_now_ns(); + rec_drain(); // the SDK records into cb itself: the recorder must have made every call queued before + uint64_t t1 = kern_now_ns(); + FfxErrorCode e = ffxFrameInterpolationPrepare(&g.fi, &pd); + uint64_t t2 = kern_now_ns(); + g.drain_ns += t1 - t0, g.sdk_ns += t2 - t1, g.prep_ns += t2 - t1; + vk_fg_mark(cb, FG_T_PREP1); + if (e != FFX_OK) { + rt_log("fg: the SDK's prepare failed (%#x): frame generation off for this run", (unsigned)e); + g.failed = true; + return 0; + } + g.reset = s->reset || !g.chain; + g.prepared = true; + return 1; +} + +extern "C" int vk_fg_hudless(VkCommandBuffer cb, VkImage src, VkFormat fmt, uint32_t level, uint32_t layer, uint32_t w, + uint32_t h) { + (void)fmt; + if (!g.prepared || g.failed || !images_for(cb, w, h)) return 0; + vk_fg_mark(cb, FG_T_HL0); + VkImageBlit b = {}; + b.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, level, layer, 1}; + b.srcOffsets[1] = {(int32_t)w, (int32_t)h, 1}; + b.dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}; + b.dstOffsets[1] = {(int32_t)w, (int32_t)h, 1}; + vkCmdBlitImage(cb, src, VK_IMAGE_LAYOUT_GENERAL, g.hl_srgb.img, VK_IMAGE_LAYOUT_GENERAL, 1, &b, VK_FILTER_NEAREST); + full_barrier(cb); + VkImageCopy cp = {}; + cp.srcSubresource = cp.dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}; + cp.extent = {w, h, 1}; + vkCmdCopyImage(cb, g.hl_srgb.img, VK_IMAGE_LAYOUT_GENERAL, g.hl[g.k].img, VK_IMAGE_LAYOUT_GENERAL, 1, &cp); + full_barrier(cb); + vk_fg_mark(cb, FG_T_HL1); + g.hl_ok = true; + return 1; +} + +// ---- shots (TOTK_FG_SHOT=): the window texture, the generated and the HUD-less picture of three frames from +// there, as PNGs in TOTK_FG_SHOT_DIR (fg__real.png, _gen.png, _hudless.png) +struct Shot { + VkBuffer buf; + VmaAllocation mem; + void *map; + uint32_t w, h; + int bgra; + char path[300]; +}; + +static void shot_write(void *arg) { + Shot *s = (Shot *)arg; + VkCtx *c = vk_ctx(); + vmaInvalidateAllocation(c->vma, s->mem, 0, VK_WHOLE_SIZE); + uint8_t *p = (uint8_t *)s->map; + if (s->bgra) + for (size_t i = 0; i < (size_t)s->w * s->h; i++) { + uint8_t t = p[4 * i]; + p[4 * i] = p[4 * i + 2], p[4 * i + 2] = t; + } + for (size_t i = 0; i < (size_t)s->w * s->h; i++) p[4 * i + 3] = 255; + rt_write_png(s->path, p, s->w, s->h); + rt_log("fg: shot %s", s->path); + vmaDestroyBuffer(c->vma, s->buf, s->mem); + free(s); +} + +static void shot(VkCommandBuffer cb, uint64_t fence, VkImage img, VkFormat f, uint32_t w, uint32_t h, uint64_t frame, + const char *kind) { + VkCtx *c = vk_ctx(); + Shot *s = (Shot *)calloc(1, sizeof *s); + if (!s) return; + VkBufferCreateInfo bi = {}; + bi.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, bi.size = (VkDeviceSize)w * h * 4; + bi.usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT; + VmaAllocationCreateInfo ai = {}; + ai.usage = VMA_MEMORY_USAGE_AUTO_PREFER_HOST; + ai.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT; + VmaAllocationInfo info; + if (vmaCreateBuffer(c->vma, &bi, &ai, &s->buf, &s->mem, &info) != VK_SUCCESS) return free(s); + s->map = info.pMappedData, s->w = w, s->h = h; + s->bgra = f == VK_FORMAT_B8G8R8A8_UNORM || f == VK_FORMAT_B8G8R8A8_SRGB; + snprintf(s->path, sizeof s->path, "%s/fg_%llu_%s.png", g.shot_dir, (unsigned long long)frame, kind); + VkBufferImageCopy r = {}; + r.imageSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}; + r.imageExtent = {w, h, 1}; + vkCmdCopyImageToBuffer(cb, img, VK_IMAGE_LAYOUT_GENERAL, s->buf, 1, &r); + vk_completion_add(fence, shot_write, s); +} + +extern "C" int vk_fg_generate(VkCommandBuffer cb, uint64_t fence, uint64_t frame, VkImage win, VkFormat fmt, uint32_t w, + uint32_t h) { + if (!g.prepared || !g.hl_ok || g.failed) return 0; + VkFormat wf = unorm_of(fmt); + if (wf != VK_FORMAT_R8G8B8A8_UNORM && wf != VK_FORMAT_B8G8R8A8_UNORM) { + static int said; + if (!said) said = 1, rt_log("fg: the window texture's format %d is not 8-bit RGBA: no frame generated", (int)fmt); + return 0; + } + if (wf != g.cfg.color_format || w != g.cfg.display_w || h != g.cfg.display_h || w != g.iw || h != g.ih) { + g.win_fmt = wf; // the next prepare makes the contexts for it + static int said; + if (said++ < 4) + rt_log("fg: window texture %ux%u format %d against contexts for %ux%u format %d, HUD-less picture %ux%u: " + "no frame generated this time", w, h, (int)fmt, g.cfg.display_w, g.cfg.display_h, + (int)g.cfg.color_format, g.iw, g.ih); + return 0; + } + VkImage hl = g.hl[g.k].img, out = g.out[g.k].img; + vk_fg_mark(cb, FG_T_FI0); + uint64_t t0 = kern_now_ns(); + rec_drain(); // the SDK records into cb itself + uint64_t t1 = kern_now_ns(); + FfxCommandList cl = ffxGetCommandListVK(cb); + FfxResource hud_less = res(hl, VK_FORMAT_R8G8B8A8_UNORM, w, h, false, L"FG_HudlessColor"); + FfxOpticalflowDispatchDescription od = {}; + od.commandList = cl; + od.color = hud_less; + od.reset = g.reset; + od.backbufferTransferFunction = FFX_BACKBUFFER_TRANSFER_FUNCTION_SRGB; + od.opticalFlowVector = shared(R_OF_VECTOR); + od.opticalFlowSCD = shared(R_OF_SCD); + FfxErrorCode e = ffxOpticalflowContextDispatch(&g.of, &od); + if (e == FFX_OK) { + FfxFrameInterpolationDispatchDescription dd = {}; + dd.flags = g.debug ? FFX_FRAMEINTERPOLATION_DISPATCH_DRAW_DEBUG_VIEW : 0; + dd.commandList = cl; + dd.displaySize = {w, h}; + dd.renderSize = {g.cfg.render_w, g.cfg.render_h}; + dd.currentBackBuffer = res(win, wf, w, h, false, L"FG_Color"); + dd.currentBackBuffer_HUDLess = hud_less; + dd.output = res(out, VK_FORMAT_R8G8B8A8_UNORM, w, h, true, L"FG_Output"); + dd.interpolationRect = {0, 0, (int)w, (int)h}; + dd.opticalFlowVector = shared(R_OF_VECTOR); + dd.opticalFlowSceneChangeDetection = shared(R_OF_SCD); + dd.opticalFlowBufferSize = {w / 8, h / 8}; // (unused by the shaders: the SDK's provider leaves it too) + dd.opticalFlowScale = {1.0f / w, 1.0f / h}; + dd.opticalFlowBlockSize = 8; + dd.cameraNear = g.near_m; + dd.cameraFar = g.far_m; + dd.cameraFovAngleVertical = g.fov_y; + dd.viewSpaceToMetersFactor = 1; + dd.frameTimeDelta = g.dt_ms; + dd.reset = g.reset; + dd.backBufferTransferFunction = FFX_BACKBUFFER_TRANSFER_FUNCTION_SRGB; + dd.minMaxLuminance[0] = 0; + dd.minMaxLuminance[1] = 1; + dd.frameID = g.frame_id; + dd.dilatedDepth = shared(R_DILATED_DEPTH + g.toggle); + dd.dilatedMotionVectors = shared(R_DILATED_MV + g.toggle); + dd.reconstructedPrevDepth = shared(R_PREV_DEPTH + g.toggle); + e = ffxFrameInterpolationDispatch(&g.fi, &dd); + } + g.drain_ns += t1 - t0, g.sdk_ns += kern_now_ns() - t1; + full_barrier(cb); + vk_fg_mark(cb, FG_T_FI1); + if (e != FFX_OK) { + rt_log("fg: the SDK's optical flow or interpolation failed (%#x): frame generation off for this run", (unsigned)e); + g.failed = true; + return 0; + } + g.gen_ok = true; + g.gens++, g.resets += g.reset; + if (g.gens == 1) + rt_log("fg: first frame generated: %ux%u from a %ux%u scene (near %g m, far %g m, vertical field of view %.1f " + "degrees), window texture format %d", w, h, g.cfg.render_w, g.cfg.render_h, g.near_m, g.far_m, + g.fov_y * 57.29578f, (int)fmt); + if (g.shot_w > 0 && !g.shot_at && frame_step_world_us() >= g.shot_w) g.shot_at = frame; + if (g.shot_at && frame >= g.shot_at && frame < g.shot_at + 3) { + shot(cb, fence, win, fmt, w, h, frame, "real"); + shot(cb, fence, out, VK_FORMAT_R8G8B8A8_UNORM, w, h, frame, g.reset ? "gen-reset" : "gen"); + shot(cb, fence, hl, VK_FORMAT_R8G8B8A8_UNORM, w, h, frame, "hudless"); + full_barrier(cb); + } + return 1; +} + +extern "C" void vk_fg_frame(void) { + if (g.prepared || g.cpu_frame) { + if (g.cpu_us) g.cpu_us[g.ncpu % CPU_RING] = (uint32_t)(g.cpu_frame / 1000); + g.ncpu++; + if (!g.gen_ok) g.skipped++; + } + g.show_out = g.gen_ok ? g.out[g.k].img : VK_NULL_HANDLE; + g.show_hl = g.hl_ok ? g.hl[g.k].img : VK_NULL_HANDLE; + if (g.hl_ok) g.k ^= 1; + g.chain = g.gen_ok; // a frame without a generated picture breaks the chain: the next one starts over + g.prepared = g.hl_ok = g.gen_ok = false; + g.cpu_frame = 0; + if (g.gens >= g.gens_logged + LOG_EVERY) { + static uint64_t from; + rt_log("fg: %llu frames generated (%llu without history), %llu frames prepared without one", (unsigned long long)g.gens, + (unsigned long long)g.resets, (unsigned long long)g.skipped); + if (g.timed) times_log("last frames", g.gpu, g.timed); + cpu_log("last frames", from); + uint64_t n = g.ncpu - from ? g.ncpu - from : 1; + rt_log("fg cpu last frames: of it recorder drains %.3f ms, the SDK's recording %.3f ms (prepare %.3f ms) a frame " + "(means)", g.drain_ns / 1e6 / n, g.sdk_ns / 1e6 / n, g.prep_ns / 1e6 / n); + g.drain_run += g.drain_ns, g.sdk_run += g.sdk_ns, g.prep_run += g.prep_ns, g.drain_ns = g.sdk_ns = g.prep_ns = 0; + from = g.ncpu; + memset(g.gpu, 0, sizeof g.gpu), g.timed = 0; + g.gens_logged = g.gens; + } +} + +extern "C" void vk_fg_exit_log(void) { + static int done; + if (done || !g.ncpu) return; + done = 1; + rt_log("fg: over the run %llu frames generated (%llu without history), %llu frames prepared without one", + (unsigned long long)g.gens, (unsigned long long)g.resets, (unsigned long long)g.skipped); + if (g.timed_run) times_log("over the run", g.gpu_run, g.timed_run); + cpu_log("over the run", 0); + rt_log("fg cpu over the run: of it recorder drains %.3f ms, the SDK's recording %.3f ms (prepare %.3f ms) a frame " + "(means)", (g.drain_run + g.drain_ns) / 1e6 / g.ncpu, (g.sdk_run + g.sdk_ns) / 1e6 / g.ncpu, + (g.prep_run + g.prep_ns) / 1e6 / g.ncpu); +} + +extern "C" int vk_fg_present_swap(VkImage *img, VkFormat *fmt) { + VkImage s = g.test == 1 ? g.show_out : g.test == 2 ? g.show_hl : VK_NULL_HANDLE; + if (!s) return 0; + *img = s, *fmt = VK_FORMAT_R8G8B8A8_UNORM; + return 1; +} diff --git a/runtime/vk_fg.h b/runtime/vk_fg.h index af538486..3ee2ad65 100644 --- a/runtime/vk_fg.h +++ b/runtime/vk_fg.h @@ -1,7 +1,15 @@ // AMD FSR 3 frame generation (vk_fg.cpp): the FidelityFX SDK 1.1.4's frame interpolation and optical flow // (third_party/FidelityFX, MIT) on its Vulkan backend. Plain C interface: the SDK is C++. +// +// The frame's inputs come from the replay (vk_dlaa.c follows the scene for it, with DLSS off): at the scene's end the +// camera motion vectors and the SDK's prepare step (vk_fg_prepare), before the HUD's first draw the picture without +// the HUD (vk_fg_hudless), at the end of the submission that copied the picture into the window texture the optical +// flow and the interpolation (vk_fg_generate). All of it goes into the replay's command buffer. #pragma once +#include +#include + #ifdef __cplusplus extern "C" { #endif @@ -14,6 +22,54 @@ int vk_fg_active(void); // vk_device.c destroy_all, before the device goes away; harmless when nothing was made. void vk_fg_release(void); +// vk_dlaa.c vk_dlaa_frame, once per present: 1 when frame generation follows the scene in the next frame. 0 while +// DLSS runs (dlss), with replay workers or sync_ahead = 1 (the inputs are per frame and recorded in replay order), +// with HDR output, or while the game shows a menu without its world (vk_fg_menu); each reason is logged once. +int vk_fg_wanted(int dlss); +// vk_dlaa.c, the game's thread at each present: 1 while the map or a loading or event fade is open (sampled there, +// read by vk_fg_wanted). +void vk_fg_menu(int open); + +// What the scene end hands over: this frame's depth and camera motion, and the camera. +typedef struct { + VkImage depth, mv; // D32F scene depth (level 0, layer 0) and RG16F camera motion, w x h, GENERAL: last + // frame's window position minus this frame's, scene pixels (vk_dlaa.c mv_pass) + uint32_t w, h; // the scene's size + uint32_t ow, oh; // the output's (the window texture's) + float near_m, far_m, fov_y; // the camera's planes (far 0: none) and vertical field of view (radians) + int inverted, infinite; // device depth 1 is the near plane; the far plane is at infinity + float pos[3], up[3], right[3], fwd[3]; // the camera in world space + int reset; // no history: no camera motion this frame, or a cut (vk_fg_camera) +} VkFgScene; +// The camera from the scene's uniform block rows 0-10 (view 3x4, view-projection, projection, row-major, as in +// vk_dlaa.c note_camera's block), the NVN depth mode (minus_w: clip z -w..w) and the viewport depth range dr: fills s's +// camera fields. Returns 1 at a cut against the last call (over 10 m or 35 degrees, as Savage Wind's frame +// generation tests it), else 0. +int vk_fg_camera(const float *rows, int minus_w, const float dr[2], VkFgScene *s); +// Records the SDK's prepare (dilated depth and motion) into cb, recorder drained here. Makes the contexts again when +// the sizes, the depth convention or the window texture's format changed. 1: done; 0: frame generation off this +// frame (logged). +int vk_fg_prepare(VkCommandBuffer cb, const VkFgScene *s); +// The picture before the HUD: src (level, layer) copied, sRGB-encoded as the game's last draw encodes it into the +// window texture, into an RGBA8 image of the SDK's. No pass open, earlier writes visible (vk_exec_barrier). +int vk_fg_hudless(VkCommandBuffer cb, VkImage src, VkFormat fmt, uint32_t level, uint32_t layer, uint32_t w, uint32_t h); +// The optical flow and the interpolation into cb, after the frame's last draw into the window texture win (fmt, w x h, +// GENERAL, level 0); fence: the submission's timeline value, frame: the present this frame will be (TOTK_FG_SHOT). +// 1: a frame between the last one and this was made. +int vk_fg_generate(VkCommandBuffer cb, uint64_t fence, uint64_t frame, VkImage win, VkFormat fmt, uint32_t w, + uint32_t h); +// GPU timestamps of the frame's steps (TOTK_FG_TIMES, default on with frame generation): FG_T_MV0 resets the frame's +// slot, so it comes first in the frame. +enum { FG_T_MV0, FG_T_MV1, FG_T_PREP0, FG_T_PREP1, FG_T_HL0, FG_T_HL1, FG_T_FI0, FG_T_FI1, FG_T_N }; +void vk_fg_mark(VkCommandBuffer cb, int which); +void vk_fg_cpu(uint64_t ns); // replay-thread time the frame generation recording took, added to the frame's +// vk_dlaa.c vk_dlaa_frame, each present before the frame state is cleared: the frame's counts and times. +void vk_fg_frame(void); +void vk_fg_exit_log(void); // vk_backend.c gfx_exit_log: the run's counts and times, once +// vk_backend.c present_image (testing, TOTK_FG_TEST): 1 = the frame's generated picture, 2 = its picture before the +// HUD in place of the window texture (*img, *fmt). 0: the present is left as it is. +int vk_fg_present_swap(VkImage *img, VkFormat *fmt); + #ifdef __cplusplus } #endif