From 25b200f94c2c0a64bb9bb5a315a7fcfd8c0b9cf5 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Fri, 30 Oct 2026 23:50:02 +0300 Subject: [PATCH 01/34] Draw 3D BC textures through stand-ins and let the recorder end its parts AMD driver 26.10.1 on RDNA2 cards wrote past the end of a 3D BC4 volume made BLOCK_TEXEL_VIEW_COMPATIBLE and 2D_ARRAY_COMPATIBLE, which lost the device at the logo (issue #5). 3D BC textures no longer get block views: draws into a level go to an uncompressed 2D array stand-in that is copied back at the pass end (TOTK_BLOCK_VIEWS_3D=1 restores the old path). Parts the recorder thread recorded and the submitting thread ended turned the scene green, then white, on the same driver once a submission was split. The recorder now writes the closing breadcrumb and ends the part itself (TOTK_REC_END=0 restores the old split). The recorder stays off on AMD by default. Based on the patch Pelipooper posted in issue #5; stand-ins extended to every mip level. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 2 +- runtime/cfg.c | 4 +- runtime/vk_exec.c | 156 +++++++++++++++++++++++++++++++++++++++++ runtime/vk_exec.h | 2 + runtime/vk_exec_draw.c | 1 + runtime/vk_rec.c | 37 +++++++++- runtime/vk_rec.h | 6 +- runtime/vk_sync.c | 8 +-- runtime/vk_texture.c | 13 +++- 9 files changed, 217 insertions(+), 12 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index af1c6605..28a29bf7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -82,7 +82,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app - Vulkan 1.3 back end in place of MacNativeGames' Metal one. At `QueueSubmitCommands` the recorded NVN commands are replayed into Vulkan command buffers (`vk_exec*.c`). - Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`). - Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default because no timing has shown a gain yet. -- Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once, and the submit waits until the ring is empty. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. `TOTK_VK_RECORDER=0` makes every call on the submitting thread. +- Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. `TOTK_VK_RECORDER=0` makes every call on the submitting thread. - Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame that reads them, while the replay still records it. At the start of a submission's replay every uniform range it binds from CPU-written pools is copied, and the draws bind the copy (`vk_usnap.c`, after MacNativeGames' diff --git a/runtime/cfg.c b/runtime/cfg.c index 31eb9213..ed31672a 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -35,7 +35,7 @@ static const char *const known[] = { "BIND_LSTICK_RIGHT", "BIND_LSTICK_UP", "BIND_MINUS", "BIND_PLUS", "BIND_R", "BIND_RCLICK", "BIND_RIGHT", "BIND_RSTICK_DOWN", "BIND_RSTICK_LEFT", "BIND_RSTICK_RIGHT", "BIND_RSTICK_UP", "BIND_SKIP", "BIND_STATS", "BIND_UP", "BIND_X", "BIND_Y", - "BIND_ZL", "BIND_ZR", "CLOTH_REALTIME", "CORE_DIAG", "CRIT", "CRIT_OUT", "DESC_TEMPLATE", "DUMP", "DUMP_BUF", "DUMP_DIR", "DUMP_LOG", + "BIND_ZL", "BIND_ZR", "BLOCK_VIEWS_3D", "CLOTH_REALTIME", "CORE_DIAG", "CRIT", "CRIT_OUT", "DESC_TEMPLATE", "DUMP", "DUMP_BUF", "DUMP_DIR", "DUMP_LOG", "EXEFS", "FEEDBACK", "FIX_C0", "FIX_ZERO", "FLIGHT", "FLIGHT_AT", "FLIGHT_DIR", "FLIGHT_FRAMES", "FLIGHT_SPIKE_MS", "FLIGHT_THUMBS", "FPS60", "FPS_LIMIT", "FREEZE_IDLE", "FREE_CORES", "FULLSCREEN", "GPL", "GPL_DRAW_BUILD", "GPU", "GPU_INDEX", "GPU_NAME", "HIGHFPS_FIX", "INPUT", "INPUT_RECORD", "INPUT_SCRIPT", "LANG", "LIB_PREBUILD", @@ -43,7 +43,7 @@ static const char *const known[] = { "MENU_TEST_DIR", "MENU_TEST_SET", "MINIMAL_LAYOUT", "MIRROR_GAP", "MOTION", "MOUSE_SENS", "MOVIE_HOST", "MOVIE_PACE", "ONE_PER_CORE", "OPERATION_MODE", "OUTPUT_RES", "PAD_LAYOUT", "PATCH_OUTPUT", "PATCH_SCENE", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", "PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES", - "PROFILE_MAIN", "PROFILE_RENDER", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REEXEC", + "PROFILE_MAIN", "PROFILE_RENDER", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_END", "REEXEC", "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHECK", "RESOLVE_CHECK", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index d82c0574..258b38ed 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -23,6 +23,7 @@ #include "host.h" #include "nvn_api.h" #include "nvn_tex.h" +#include "platform/platform.h" #include "tex.h" #include "vk.h" #include "vk_exec.h" @@ -141,11 +142,163 @@ static int view_format(const tex_t *t, const view_t *v, int blocks, VkFormat *ou return 1; } +// ---- 3D BC render targets +// A 3D BC texture has no block views (vk_texture.c create_tex, TOTK_BLOCK_VIEWS_3D): on RDNA2 cards such an image +// made BLOCK_TEXEL_VIEW_COMPATIBLE and 2D_ARRAY_COMPATIBLE took GPU writes past the end of its memory (issue #5). Draws +// into it through an uncompressed view of its blocks (agl's BC4 encoder fills 32x32x32 and 64x64x64 volumes and their +// levels at the logo) go to a stand-in: a 2D array of the blocks of one level in the view's format, one layer per +// depth plane. It starts with the level's content, so a pass that writes part of a plane keeps the rest, and each pass +// end copies the planes it drew into the level (a copy between a compressed and a size-compatible uncompressed format). +typedef struct { VkImage bc, img; VmaAllocation alloc; VkFormat fmt; uint32_t level, w, h, d; } BcStandIn; +#define BC_STANDINS 64 // one per level drawn: 6 volumes of up to 7 levels at the logo +static BcStandIn bcs[BC_STANDINS]; +static int nbcs; +static PlatMutex bcs_lock = PLAT_MUTEX_INIT; // the replay adds and reads; destroy_tex (any thread) removes + +static void image_barrier(VkCommandBuffer cb, VkImage img, VkImageLayout from, uint32_t layers) { + VkImageMemoryBarrier2 ib = {.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .srcStageMask = VK_PIPELINE_STAGE_2_NONE, + .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + .oldLayout = from, .newLayout = VK_IMAGE_LAYOUT_GENERAL, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = img, .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, layers}}; + VkDependencyInfo di = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 1, .pImageMemoryBarriers = &ib}; + vkCmdPipelineBarrier2(cb, &di); +} + +static void memory_barrier(VkCommandBuffer cb, VkPipelineStageFlags2 src, VkAccessFlags2 sa, VkPipelineStageFlags2 dst, + VkAccessFlags2 da) { + VkMemoryBarrier2 mb = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER_2, .srcStageMask = src, .srcAccessMask = sa, + .dstStageMask = dst, .dstAccessMask = da}; + VkDependencyInfo di = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; + vkCmdPipelineBarrier2(cb, &di); +} + +// The stand-in of level level of a 3D BC image for draws through view format fmt, made on first use; 0 when none can +// be made (logged). Replay thread. +static int bc_standin(const VkTexImage *img, uint32_t level, VkFormat fmt, BcStandIn *out) { + static int warned; + plat_mutex_lock(&bcs_lock); + int full = nbcs == BC_STANDINS; + for (int i = 0; i < nbcs; i++) + if (bcs[i].bc == img->image && bcs[i].level == level && bcs[i].fmt == fmt) { + *out = bcs[i]; + plat_mutex_unlock(&bcs_lock); + return 1; + } + plat_mutex_unlock(&bcs_lock); + if (full) { + if (!warned++) rt_log("vk: more than %d 3D BC render targets: the others are left out", BC_STANDINS); + return 0; + } + uint32_t lw = level_extent(img->width, level), lh = level_extent(img->height, level); + BcStandIn s = {img->image, VK_NULL_HANDLE, NULL, fmt, level, (lw + 3) / 4, (lh + 3) / 4, + level_extent(img->depth, level)}; + VkImageCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .imageType = VK_IMAGE_TYPE_2D, .format = fmt, + .extent = {s.w, s.h, 1}, .mipLevels = 1, .arrayLayers = s.d, .samples = VK_SAMPLE_COUNT_1_BIT, + .tiling = VK_IMAGE_TILING_OPTIMAL, + .usage = VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | + VK_IMAGE_USAGE_TRANSFER_DST_BIT, + .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED}; + VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; + if (vmaCreateImage(vk_ctx()->vma, &ci, &ai, &s.img, &s.alloc, NULL) != VK_SUCCESS) { + if (!warned++) rt_log("vk: stand-in for a 3D BC render target could not be created: left out"); + return 0; + } + // In the upload batch, which goes to the queue before this part: GENERAL, then the texture's blocks. + VkCommandBuffer cb = vk_upload_begin(); + image_barrier(cb, s.img, VK_IMAGE_LAYOUT_UNDEFINED, s.d); + memory_barrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, + VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); + VkImageCopy rg = {.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, level, 0, 1}, + .dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, s.d}, + .extent = {lw, lh, s.d}}; // texels of the compressed source + vkCmdCopyImage(cb, img->image, VK_IMAGE_LAYOUT_GENERAL, s.img, VK_IMAGE_LAYOUT_GENERAL, 1, &rg); + memory_barrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); + vk_upload_end(); + plat_mutex_lock(&bcs_lock); + bcs[nbcs++] = s; + plat_mutex_unlock(&bcs_lock); + rt_log("vk: 3D BC render target %ux%ux%u level %u drawn through a %ux%u x %u stand-in (format %d)", img->width, + img->height, img->depth, level, s.w, s.h, s.d, (int)fmt); + *out = s; + return 1; +} + +// vk_exec_draw.c vk_exec_image_gone: the stand-in of a BC image that goes, freed once the GPU is done with it. +void vk_exec_bc_gone(VkImage image) { + plat_mutex_lock(&bcs_lock); + for (int i = 0; i < nbcs; i++) + if (bcs[i].bc == image) { + vk_defer_image(bcs[i].img, bcs[i].alloc); + bcs[i--] = bcs[--nbcs]; + } + plat_mutex_unlock(&bcs_lock); +} + +// At a pass end: the planes the pass drew into a stand-in go to its texture. +static void bc_copy_back(VkExec *ex, const VkExecTarget *t) { + plat_mutex_lock(&bcs_lock); + for (int i = 0; i < nbcs; i++) { + if (bcs[i].img != t->image) continue; + memory_barrier(ex->cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, + VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); + VkImageCopy rg = {.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, t->base_layer, t->layers}, + .dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, bcs[i].level, 0, 1}, + .dstOffset = {0, 0, (int32_t)t->base_layer}, + .extent = {bcs[i].w, bcs[i].h, t->layers}}; // texels of the uncompressed source + vkCmdCopyImage(ex->cb, t->image, VK_IMAGE_LAYOUT_GENERAL, bcs[i].bc, VK_IMAGE_LAYOUT_GENERAL, 1, &rg); + memory_barrier(ex->cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); + ex->recorded++; + break; + } + plat_mutex_unlock(&bcs_lock); +} + +// A draw into a 3D BC image without block views through an uncompressed view of its blocks: the target is the +// image's stand-in (bc_standin). 1 when o is filled (target or left out), 0 for any other target. +static int bc_target(const tex_t *t, const view_t *v, int has_v, const VkTexImage *img, VkExecTarget *o) { + int bw, bh; + if (img->type != VK_IMAGE_TYPE_3D || img->block_views || !nvn_fmt_bpb(t->d.format, &bw, &bh) || bw != 4 || bh != 4) + return 0; + VkFormat fmt; + BcStandIn s; + uint32_t level = has_v && v->set_levels ? v->base_level : 0; + if (level >= img->levels) { + warn_once(W_LEVEL, "render target level %u of a %u-level texture: left out", level, img->levels); + return 1; + } + if (!has_v || !v->set_format || !view_format(t, v, 1, &fmt) || !bc_standin(img, level, fmt, &s)) { + warn_once(W_BLOCK_VIEW, "render target format %#x (%ux%u) through view format %#x: no block view, left out", + t->d.format, t->d.width, t->d.height, has_v && v->set_format ? v->format : 0); + return 1; + } + uint32_t base = 0, layers = s.d; + if (v->set_layers) { + base = v->min_layer; + layers = v->nlayers ? v->nlayers : (base < s.d ? s.d - base : 0); + } + if (base >= s.d || !layers) return 1; + if (base + layers > s.d) layers = s.d - base; + o->image = s.img; // not img->image: view_of makes this command buffer's view of the stand-in + o->img = img; + o->format = fmt; + o->view_type = layers > 1 ? VK_IMAGE_VIEW_TYPE_2D_ARRAY : VK_IMAGE_VIEW_TYPE_2D; + o->aspect = VK_IMAGE_ASPECT_COLOR_BIT; + o->level = 0, o->base_layer = base, o->layers = layers; + o->width = s.w, o->height = s.h, o->samples = 1; + o->nvn_format = v->format; + return 1; +} + static void resolve_target(const tex_t *t, const view_t *v, int has_v, int depth_slot, VkExecTarget *o) { memset(o, 0, sizeof *o); o->tex = t; if (!t) return; const VkTexImage *img = vk_texture_get(t); + if (img && !depth_slot && bc_target(t, v, has_v, img, o)) return; // a 3D BC image: its stand-in VkImageUsageFlags need = depth_slot ? VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT : VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT; int is3d = img && img->type == VK_IMAGE_TYPE_3D && !depth_slot; // slices drawn through 2D views (2D_ARRAY_COMPATIBLE) if (!img || !(img->usage & need) || (img->type != VK_IMAGE_TYPE_2D && !is3d)) { @@ -469,6 +622,9 @@ void vk_exec_pass_end(VkExec *ex) { vkCmdEndRendering(ex->cb); ex->recorded++; ex->in_pass = 0; + for (int i = 0; i < ex->pass_ncolor && i < 8; i++) // stand-ins of 3D BC targets go to their textures + if (ex->color[i].image && ex->color[i].img && ex->color[i].img->image != ex->color[i].image) + bc_copy_back(ex, &ex->color[i]); if (vk_dump_armed) vk_dump_pass_end(ex); uint64_t stamp = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED) + 1; for (int i = 0; i < 8; i++) diff --git a/runtime/vk_exec.h b/runtime/vk_exec.h index c81d5c61..6236929e 100644 --- a/runtime/vk_exec.h +++ b/runtime/vk_exec.h @@ -124,6 +124,8 @@ void vk_exec_inline_end(int reopen); // buffers): the replay continues in a new one (vk_sync_split). void vk_exec_draw_new_cb(void); VkBuffer vk_exec_inline_buffer(VkDeviceSize size); +// vk_exec.c: the stand-in a 3D BC render target is drawn through goes with its image (vk_exec_image_gone). +void vk_exec_bc_gone(VkImage image); // Scissor 0 clipped to the render area of the open pass; extent 0: nothing to draw. VkRect2D vk_exec_scissor(const VkExec *ex); void vk_exec_noatt_dims(const VkExec *ex, uint32_t *w, uint32_t *h); // pass size when no attachment is set diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index 59ff9cbb..436ba0a2 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -564,6 +564,7 @@ VkImageView vk_exec_target_view(const VkTexImage *img, VkFormat format, VkImageV } void vk_exec_image_gone(VkImage image) { + vk_exec_bc_gone(image); if (!views_by_image.klen) return; for (ViewNode *n = nvn_hm_del(&views_by_image, &image), *next; n; n = next) { next = n->next; diff --git a/runtime/vk_rec.c b/runtime/vk_rec.c index 4ebfeb79..6ccd30d3 100644 --- a/runtime/vk_rec.c +++ b/runtime/vk_rec.c @@ -38,7 +38,7 @@ enum { R_SetDepthWriteEnable, R_SetFrontFace, R_SetPrimitiveRestartEnable, R_SetPrimitiveTopology, R_SetRasterizerDiscardEnable, R_SetScissorWithCount, R_SetStencilCompareMask, R_SetStencilOp, R_SetStencilReference, R_SetStencilTestEnable, R_SetStencilWriteMask, R_SetViewportWithCount, R_WriteTimestamp, R_WriteTimestamp2, - R_PushTpl, R_Push, R_VertexInput, R_Bool, R_PolygonMode, R_BlendEnable, R_BlendEquation, R_WriteMask, + R_PushTpl, R_Push, R_VertexInput, R_Bool, R_PolygonMode, R_BlendEnable, R_BlendEquation, R_WriteMask, R_EndCb, }; static uint64_t now_ns(void) { @@ -131,7 +131,8 @@ void rec_begin(VkCommandBuffer cb) { if (mode < 0) { // Default off on AMD GPUs: with the recorder their drivers lost the device within the first frames or at menu // opens, 1 boot in 3 on an RX 9070 XT, and TOTK_VK_RECORDER=0 ended it for three users (issue #5); under RADV - // the kernel saw an illegal opcode at the end of a command buffer (issue #40). The cause is not found. + // the kernel saw an illegal opcode at the end of a command buffer (issue #40). Letting the recorder end its own + // parts (rec_end_cb) fixed the white scene on an RX 6700 XT; the default stays off until more AMD cards run it. const char *set = cfg_get("TOTK_VK_RECORDER"); mode = rt_knob("VK_RECORDER", vk_caps.amd ? "0" : "1"); if (!mode && vk_caps.amd && !(set && *set)) @@ -162,6 +163,32 @@ void rec_end(void) { rec_live = 0; } +// TOTK_REC_END (default 1): the thread that recorded the part also ends it, after the breadcrumb crumb (kind 0: none) +// that vk_sync.c writes at the end of a submission. On an RX 6700 XT (driver 26.10.1, issue #5) parts the recorder +// recorded and the submitting thread ended broke when a submission was split (vk_sync.c vk_sync_split): the GPU hung +// or the scene turned green, then white. The handoff in rec_drain orders the two threads, so Vulkan allows the old +// split; the driver did not cope with it. TOTK_REC_END=0 brings the old split back. +typedef struct { int kind; uint64_t n; } A_EndCb; +int rec_end_cb(VkCommandBuffer cb, int crumb, uint64_t n) { + static int mode = -1; + if (mode < 0) { + mode = rt_knob("REC_END", "1"); + rt_log("vk_rec: a part the recorder records is ended by %s (TOTK_REC_END=%d)", + mode ? "the recorder" : "the submitting thread", mode); + } + if (!rec_live || cb != rec_cb || !mode) { + rec_end(); + if (crumb) vk_crumb(cb, crumb, n); + return 0; + } + A_EndCb *a = put(R_EndCb, sizeof *a); + *a = (A_EndCb){crumb, n}; + pub(); + rec_drain(); + rec_live = 0; + return 1; +} + void rec_stats_log(void) { if (!(rt_trace & RT_TRACE_PERF) || !RS_.parts) return; rt_log("vk_rec: %llu of %llu parts recorded by the recorder: %llu packets (%.1f MiB), %llu drains (%llu for a call " @@ -588,6 +615,12 @@ static void exec_packet(uint32_t op, void *p) { case R_BlendEnable: { A_Targets *a = p; ((PFN_vkCmdSetColorBlendEnableEXT)a->fn)(cb, a->first, a->n, a->v); break; } case R_BlendEquation: { A_Targets *a = p; ((PFN_vkCmdSetColorBlendEquationEXT)a->fn)(cb, a->first, a->n, a->v); break; } case R_WriteMask: { A_Targets *a = p; ((PFN_vkCmdSetColorWriteMaskEXT)a->fn)(cb, a->first, a->n, a->v); break; } + case R_EndCb: { + A_EndCb *a = p; + if (a->kind) vk_crumb(cb, a->kind, a->n); + vkEndCommandBuffer(cb); + break; + } default: rt_fatal("vk_rec: unknown packet %u", op); } } diff --git a/runtime/vk_rec.h b/runtime/vk_rec.h index afbb2937..d4054e1e 100644 --- a/runtime/vk_rec.h +++ b/runtime/vk_rec.h @@ -4,7 +4,8 @@ // ring, in call order, and the recorder makes the real call in that order; calls into any other command buffer (uploads, // the window, probes) are made at once. A call the ring has no packet for (a structure with a pNext chain) waits for // the recorder to empty the ring and is then made at once on the calling thread. rec_end returns once the recorder made -// every call, so ending, submitting and the command pool stay on the submitting thread. +// every call. vk_sync.c ends a part through rec_end_cb: the recorder makes the last calls into a part it recorded, +// vkEndCommandBuffer included; submitting and the command pool stay on the submitting thread. // // Every vkCmd* the runtime calls through vk.h goes through the rc_* wrappers below (the macros at the end); the // extension commands called through function pointers have rec_* wrappers of their own. TOTK_VK_RECORDER=0 makes every @@ -16,6 +17,9 @@ void rec_begin(VkCommandBuffer cb); // the submitting thread, sub_lock held void rec_end(void); // waits for the recorder; no-op when rec_begin did not start it +// The end of part cb: the breadcrumb crumb (0: none, vk_crumbs.c), then vkEndCommandBuffer, both made by the recorder +// when it records cb, and the wait for it (1); otherwise rec_end and the breadcrumb, and the caller ends cb (0). +int rec_end_cb(VkCommandBuffer cb, int crumb, uint64_t n); void rec_drain(void); // waits until the recorder made every call queued so far void rec_stats_log(void); // TOTK_TRACE=perf: packets, bytes, drains and waits since the last call diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index 671a803e..6fb05a5b 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -293,9 +293,8 @@ static VkCommandBuffer prologue(VkCtx *c, uint64_t fence) { VkCommandBuffer vk_sync_split(void) { VkCtx *c = vk_ctx(); VkCommandBuffer cb = slots[cur_slot].cb; - rec_end(); // every command of the part is in cb ts_end(cb, cur_fence); - vkEndCommandBuffer(cb); + if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); // every command of the part is in cb, ended by its recorder vk_upload_flush(0); // the uploads this part asked for precede it in queue order VkCommandBuffer pro = prologue(c, cur_fence); VkCommandBufferSubmitInfo cbi[2] = {{.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = pro}, @@ -348,11 +347,10 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) { rec_begin(s->cb); int recorded = vk_exec_record(s->cb, handles, n, fence); s = &slots[cur_slot]; // the last part (vk_sync_split) - rec_end(); - vk_crumb(s->cb, VK_CRUMB_SUBMIT_END, fence); // the recorder is done with the part if (recorded) ts_end(s->cb, fence); else ts_drop(); - vkEndCommandBuffer(s->cb); + // The recorder is done with the part: the end-of-submission crumb, then the end, by the recorder when it records. + if (!rec_end_cb(s->cb, VK_CRUMB_SUBMIT_END, fence)) vkEndCommandBuffer(s->cb); perf_replay(t0); // After the recording: the uploads it asked for (and every earlier one) precede it in queue order. vk_upload_flush(0); diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index 9631add8..f12e7934 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -328,7 +328,18 @@ static Tex *create_tex(const tex_t *t, const TexKey *k, int defer_init) { .samples = (VkSampleCountFlagBits)samples, .tiling = VK_IMAGE_TILING_OPTIMAL, .usage = usage, .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED}; int bw, bh, block_views = 0; - if (!ds && samples == 1 && f.vk == f.ideal && nvn_fmt_bpb(d->format, &bw, &bh) && bw == 4 && bh == 4) { + // TOTK_BLOCK_VIEWS_3D (default 0): a 3D BC image also gets block views. On RDNA2 cards (driver 26.10.1, issue #5) + // the 32x32x32 BC4 volume the game creates at the logo, made BLOCK_TEXEL_VIEW_COMPATIBLE and 2D_ARRAY_COMPATIBLE, + // took a GPU write just past the end of its memory: device lost at the first submit after it when the image ended + // its memory block. Draws into a 3D BC image go through a stand-in instead (vk_exec.c bc_target). + static int bv3d = -1; + if (bv3d < 0) { + bv3d = rt_knob("BLOCK_VIEWS_3D", "0"); + rt_log("vk: 3D BC textures %s (TOTK_BLOCK_VIEWS_3D=%d)", bv3d ? "get block views (renderable)" : "have no block views", + bv3d); + } + if (!ds && samples == 1 && f.vk == f.ideal && nvn_fmt_bpb(d->format, &bw, &bh) && bw == 4 && bh == 4 && + (type != VK_IMAGE_TYPE_3D || bv3d)) { VkImageCreateInfo bc = ci; // BC formats: renderable through views of their blocks (block_views_ok) bc.usage |= VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT; bc.flags |= VK_IMAGE_CREATE_BLOCK_TEXEL_VIEW_COMPATIBLE_BIT | VK_IMAGE_CREATE_EXTENDED_USAGE_BIT; From ed1973a68b449c5712c2da1eaec785605c75c11f Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 00:05:54 +0300 Subject: [PATCH 02/34] Add a stick dead zone setting for drifting gamepads stick_deadzone (0 to 50 percent, default 8) replaces the fixed per-axis cutoff of 2500. The zone is round and the travel past it is scaled to start at 0, so a drifting stick rests centred and full deflection still reaches full. The Controls tab shows it as a slider; --probe-cfg checks the math (input_pad_stick). Co-Authored-By: Claude Opus 5.5 --- docs/controls.md | 2 ++ runtime/cfg.c | 6 +++--- runtime/cfg_probe.c | 11 +++++++++++ runtime/input_map.c | 9 +++++++++ runtime/input_map.h | 4 ++++ runtime/input_win.c | 27 +++++++++++++++++++-------- runtime/ui/menu.cpp | 1 + runtime/ui_text.c | 4 ++++ runtime/ui_text.h | 1 + totk.ini.example | 3 +++ 10 files changed, 57 insertions(+), 11 deletions(-) diff --git a/docs/controls.md b/docs/controls.md index 00f5a5d1..b80fc89c 100644 --- a/docs/controls.md +++ b/docs/controls.md @@ -68,6 +68,8 @@ Mouse motion (raw input) becomes right-stick deflection. Each presented frame ta `TOTK_MOUSE_SENS` (0 to 10, default 1; higher turns faster, 0 turns the mouse camera and the cursor capture off). While the window is the foreground window the cursor is hidden and confined to the client area. It is released on focus loss, minimise, while F11 changes the window, during a size or move loop and while a text entry is open. Esc releases it by hand: the cursor shows, the mouse camera and mouse buttons stop, and the window can be moved, minimised or closed. A mouse click inside the client area captures it again; that click does not reach the game. Activating the window by a click on its title bar or border does not capture, so it can be dragged or closed right away; Alt+Tab or a click in the client area captures as before. `TOTK_INPUT=none` disables the capture too. +`TOTK_STICK_DEADZONE` (0 to 50, default 8) is the gamepad stick dead zone in percent of full deflection, the same for both sticks. It is round: a stick inside the circle reads centred, and the rest of its travel is scaled to start at 0 at the edge, so full deflection still reaches full. Raise it for a stick that drifts. The game's own dead zone applies after it. + The stick direction bits (`StickLLeft` to `StickRDown`) are derived from the sticks at 50% deflection. A gamepad plugged in later is picked up within one second; with none connected the log says `input: no gamepad`. ## Scripted input diff --git a/runtime/cfg.c b/runtime/cfg.c index 31eb9213..322917ab 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -47,7 +47,7 @@ static const char *const known[] = { "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHECK", "RESOLVE_CHECK", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", - "SHADER_MEMO", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "SWKBD_ANSWER", + "SHADER_MEMO", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "SWKBD_ANSWER", "SWKBD_TEST", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "UBO_SNAPSHOT", "UPSCALER", "VK_DYN_VERTEX", "VK_NO_FLOAT64", "VK_PRESENT", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VS_OUTS", "WB_FENCE_LIMIT", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_PROCS", "ZSTD_CHECK", "ZSTD_NATIVE", @@ -63,7 +63,7 @@ static const struct { const char *key, *section; } sections[] = { {"MENU_LANG", "language"}, {"LANG", "language"}, {"OPERATION_MODE", "language"}, {"VOLUME", "audio"}, {"MOTION", "input"}, {"RUMBLE_STRENGTH", "input"}, {"RUMBLE_CURVE", "input"}, {"RUMBLE_TRIGGERS", "input"}, - {"PAD_LAYOUT", "input"}, {"MOUSE_SENS", "input"}, {"INPUT_RECORD", "input"}, + {"PAD_LAYOUT", "input"}, {"STICK_DEADZONE", "input"}, {"MOUSE_SENS", "input"}, {"INPUT_RECORD", "input"}, {"BIND_A", "input"}, {"BIND_B", "input"}, {"BIND_X", "input"}, {"BIND_Y", "input"}, {"BIND_LCLICK", "input"}, {"BIND_RCLICK", "input"}, {"BIND_L", "input"}, {"BIND_R", "input"}, {"BIND_ZL", "input"}, {"BIND_ZR", "input"}, {"BIND_PLUS", "input"}, {"BIND_MINUS", "input"}, {"BIND_LEFT", "input"}, {"BIND_UP", "input"}, @@ -104,7 +104,7 @@ static const CfgMeta metas[] = { {"VOLUME", "100", "0..100", LIVE}, {"MOTION", "1", "0|1", LIVE}, {"RUMBLE_STRENGTH", "130", "0..300", LIVE}, {"RUMBLE_CURVE", "0.6", "0.2..1.5", LIVE}, {"RUMBLE_TRIGGERS", "0", "0|1", LIVE}, {"PAD_LAYOUT", "auto", "auto|nintendo|xbox", LIVE}, - {"MOUSE_SENS", "1", "0..10", LIVE}, + {"STICK_DEADZONE", "8", "0..50", LIVE}, {"MOUSE_SENS", "1", "0..10", LIVE}, {"MENU_LANG", "", "en|ru", LIVE}, // Advanced: scheduling {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, diff --git a/runtime/cfg_probe.c b/runtime/cfg_probe.c index 3e4ee435..0fc40a42 100644 --- a/runtime/cfg_probe.c +++ b/runtime/cfg_probe.c @@ -261,6 +261,17 @@ int probe_cfg(void) { CHECK((int64_t)st[0] * st[0] + (int64_t)st[1] * st[1] <= 32768ll * 32768, "mouse: clamped to the unit circle"); input_mouse_stick(30, 0, 1.0 / 60, 0, st); CHECK(st[0] == 0 && st[1] == 0, "mouse: sensitivity 0 is off"); + input_pad_stick(3000, -2000, 0.15f, st); + CHECK(st[0] == 0 && st[1] == 0, "pad stick: an 11% drift inside a 15% dead zone reads centred"); + input_pad_stick(4000, -3000, 0, st); + CHECK(st[0] == 4000 && st[1] == -3000, "pad stick: dead zone 0 passes the stick unchanged"); + input_pad_stick(32767, 0, 0.15f, st); + CHECK(st[0] == 32767 && st[1] == 0, "pad stick: full deflection still reaches full"); + input_pad_stick(-32768, -32768, 0.15f, st); + CHECK(st[0] < 0 && st[1] < 0 && st[0] >= -32767 && (int64_t)st[0] * st[0] + (int64_t)st[1] * st[1] <= 32768ll * 32768, + "pad stick: a square gate's corner stays on the unit circle"); + input_pad_stick(0, 32767 / 2 + 1, 0.2f, st); + CHECK(abs(st[1] - 32767 * 3 / 8) <= 3, "pad stick: half deflection past a 20% zone is (0.5 - 0.2) / 0.8 of full"); // Gamepad layouts (input_map.c) CHECK(input_pad_layout_pick("auto", 1) == PAD_LAYOUT_NINTENDO && input_pad_layout_pick(NULL, 0) == PAD_LAYOUT_XBOX, "pad layout: automatic is Nintendo for an east button labelled A, Xbox otherwise"); diff --git a/runtime/input_map.c b/runtime/input_map.c index 7874e7e1..5c5e06c8 100644 --- a/runtime/input_map.c +++ b/runtime/input_map.c @@ -325,6 +325,15 @@ void input_mouse_stick(int32_t dx, int32_t dy, double dt_s, float sens, int32_t out[1] = (int32_t)lrint(y); } +void input_pad_stick(int32_t x, int32_t y, float dead, int32_t out[2]) { + double fx = x / (double)STICK_MAX, fy = y / (double)STICK_MAX, m = sqrt(fx * fx + fy * fy); + out[0] = out[1] = 0; + if (m <= dead) return; + double k = ((m > 1 ? 1 : m) - dead) / (1 - dead) / m; // past the edge (a square gate's corner) the length is 1 + out[0] = (int32_t)lrint(fmax(-1, fmin(1, fx * k)) * STICK_MAX); + out[1] = (int32_t)lrint(fmax(-1, fmin(1, fy * k)) * STICK_MAX); +} + // Minus and Plus held together open or close the menu. *held: latched from the poll where both go down until neither is // down (the caller keeps it). Every poll of the latch reports neither to the game, so releasing one button first does not // leak the other; Minus alone before Plus reaches the game unchanged. With the menu open the game sees a neutral pad. diff --git a/runtime/input_map.h b/runtime/input_map.h index c4c4425b..fe524fa6 100644 --- a/runtime/input_map.h +++ b/runtime/input_map.h @@ -91,6 +91,10 @@ void input_map_stick(const uint8_t down[KEY_COUNT], int stick, int32_t out[2]); // Mouse motion in raw counts over dt_s seconds to right-stick deflection: full deflection at 3000 / sens counts/s. // Clamped to the unit circle, up (negative dy) is +y. void input_mouse_stick(int32_t dx, int32_t dy, double dt_s, float sens, int32_t out[2]); +// A gamepad stick (x, y in -32768..32767, up is +y) through a round dead zone of `dead` (fraction of full deflection, +// 0..0.5): inside the circle the stick reads 0, and the rest of the travel is scaled to start at 0 at the edge, so a +// drifting stick rests at 0 and full deflection still reaches full. The result stays inside the unit square. +void input_pad_stick(int32_t x, int32_t y, float dead, int32_t out[2]); // ---- the remapping rows' edits, on a copy of input_map. Pure. // action's keys (table order) / its gamepad buttons in layout, at most max; returns how many. diff --git a/runtime/input_win.c b/runtime/input_win.c index 61bb7dea..096fef53 100644 --- a/runtime/input_win.c +++ b/runtime/input_win.c @@ -19,7 +19,7 @@ #include "ui/menu.h" #include "vk.h" -enum { STICK_MAX = 32767, STICK_DEAD = 2500, STICK_DIR = 16384, TRIGGER_ON = 16384, RESCAN_MS = 1000 }; +enum { STICK_MAX = 32767, STICK_DIR = 16384, TRIGGER_ON = 16384, RESCAN_MS = 1000 }; static SRWLOCK lock = SRWLOCK_INIT; static int state; // 0 not started, 1 running, -1 disabled or SDL failed @@ -428,9 +428,21 @@ static void mouse_camera(int32_t out[2]) { out[1] = held[1]; } -static int32_t axis(int16_t v) { - int32_t a = v < -STICK_MAX ? -STICK_MAX : v; - return a > -STICK_DEAD && a < STICK_DEAD ? 0 : a; +static float stick_dead = -1; // TOTK_STICK_DEADZONE as a fraction of full deflection; live (cfg_version) +static unsigned stick_dead_seen; + +static float stick_dead_get(void) { + unsigned ver = cfg_version(); + if (stick_dead >= 0 && ver == stick_dead_seen) return stick_dead; + stick_dead_seen = ver; + double v = cfg_double("TOTK_STICK_DEADZONE"); + stick_dead = (float)(v < 0 ? 0 : v > 50 ? 50 : v) / 100.f; + return stick_dead; +} + +// One gamepad stick through the dead zone; SDL's Y grows downwards, the Switch's upwards. +static void stick(SDL_GamepadAxis ax, SDL_GamepadAxis ay, float dead, int32_t out[2]) { + input_pad_stick(SDL_GetGamepadAxis(pad, ax), -(int32_t)SDL_GetGamepadAxis(pad, ay), dead, out); } // Open the first gamepad when none is open or the open one went away; logs once per change. @@ -545,10 +557,9 @@ static int pad_read(uint64_t *b, int32_t sl[2], int32_t sr[2]) { uint8_t pd[PAD_COUNT] = {0}; if (!pad_physical(pd)) return 0; *b |= input_map_pad_buttons(pd, pad_layout_get()); - sl[0] = axis(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_LEFTX)); - sl[1] = -axis(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_LEFTY)); - sr[0] = axis(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_RIGHTX)); - sr[1] = -axis(SDL_GetGamepadAxis(pad, SDL_GAMEPAD_AXIS_RIGHTY)); + float dead = stick_dead_get(); + stick(SDL_GAMEPAD_AXIS_LEFTX, SDL_GAMEPAD_AXIS_LEFTY, dead, sl); + stick(SDL_GAMEPAD_AXIS_RIGHTX, SDL_GAMEPAD_AXIS_RIGHTY, dead, sr); return 1; } diff --git a/runtime/ui/menu.cpp b/runtime/ui/menu.cpp index 7d52baa9..99466940 100644 --- a/runtime/ui/menu.cpp +++ b/runtime/ui/menu.cpp @@ -573,6 +573,7 @@ static void tab_controls(float scale) { opt_slider("TOTK_RUMBLE_STRENGTH", UI_MENU_RUMBLE_STRENGTH, 0, 300, UI_MENU_RUMBLE_STRENGTH_DESC, 10, "%.0f %%"); opt_slider("TOTK_RUMBLE_CURVE", UI_MENU_RUMBLE_CURVE, 0.2f, 1.5f, UI_MENU_RUMBLE_CURVE_DESC, 0.05f, "%.2f"); opt_toggle("TOTK_RUMBLE_TRIGGERS", UI_MENU_RUMBLE_TRIGGERS, UI_MENU_RUMBLE_TRIGGERS_DESC); + opt_slider("TOTK_STICK_DEADZONE", UI_MENU_STICK_DEADZONE, 0, 50, UI_MENU_STICK_DEADZONE_DESC, 1, "%.0f %%"); opt_slider("TOTK_MOUSE_SENS", UI_MENU_MOUSE_SENS, 0, 10, UI_MENU_MOUSE_SENS_DESC, 0.1f, "%.1f"); binds_rows(scale); opt_end(); diff --git a/runtime/ui_text.c b/runtime/ui_text.c index 0de0818f..b17ebb02 100644 --- a/runtime/ui_text.c +++ b/runtime/ui_text.c @@ -97,6 +97,8 @@ static const char *const text_en[UI_TEXT_COUNT] = { "Nintendo for controllers with Nintendo labels and Xbox for the others.", [UI_MENU_PAD_LAYOUT_AUTO] = "Automatic", [UI_MENU_PAD_LAYOUT_NINTENDO] = "Nintendo (A right)", [UI_MENU_PAD_LAYOUT_XBOX] = "Xbox (A bottom)", [UI_MENU_PAD_LAYOUT_IN_USE] = "in use: %s", + [UI_MENU_STICK_DEADZONE] = "Stick dead zone", + [UI_MENU_STICK_DEADZONE_DESC] = "How far a gamepad stick moves before the game sees it. Raise it if the character or the camera moves on its own.", [UI_MENU_MOUSE_SENS] = "Mouse camera sensitivity", [UI_MENU_MOUSE_SENS_DESC] = "How fast the mouse turns the camera.", @@ -315,6 +317,8 @@ static const char *const text_ru[UI_TEXT_COUNT] = { "Nintendo для контроллеров с подписями Nintendo, Xbox для остальных.", [UI_MENU_PAD_LAYOUT_AUTO] = "Автоматически", [UI_MENU_PAD_LAYOUT_NINTENDO] = "Nintendo (A справа)", [UI_MENU_PAD_LAYOUT_XBOX] = "Xbox (A снизу)", [UI_MENU_PAD_LAYOUT_IN_USE] = "сейчас: %s", + [UI_MENU_STICK_DEADZONE] = "Мёртвая зона стиков", + [UI_MENU_STICK_DEADZONE_DESC] = "Насколько нужно отклонить стик геймпада, чтобы игра это заметила. Увеличьте, если персонаж или камера двигаются сами.", [UI_MENU_MOUSE_SENS] = "Чувствительность мыши", [UI_MENU_MOUSE_SENS_DESC] = "Как быстро мышь поворачивает камеру.", diff --git a/runtime/ui_text.h b/runtime/ui_text.h index 96e10618..24975758 100644 --- a/runtime/ui_text.h +++ b/runtime/ui_text.h @@ -57,6 +57,7 @@ typedef enum { UI_MENU_RUMBLE_TRIGGERS, UI_MENU_RUMBLE_TRIGGERS_DESC, UI_MENU_PAD_LAYOUT, UI_MENU_PAD_LAYOUT_DESC, UI_MENU_PAD_LAYOUT_AUTO, UI_MENU_PAD_LAYOUT_NINTENDO, UI_MENU_PAD_LAYOUT_XBOX, UI_MENU_PAD_LAYOUT_IN_USE, + UI_MENU_STICK_DEADZONE, UI_MENU_STICK_DEADZONE_DESC, UI_MENU_MOUSE_SENS, UI_MENU_MOUSE_SENS_DESC, // Controls tab: the remapping rows (ui/binds_page.cpp) diff --git a/totk.ini.example b/totk.ini.example index 30c3a2af..522e247b 100644 --- a/totk.ini.example +++ b/totk.ini.example @@ -150,6 +150,9 @@ # west X, north Y, as an Xbox pad labels them; cross A on a PlayStation pad) or auto (nintendo for a pad with Nintendo # labels, xbox for the others). The bind_ lines below apply on top. Changes while the game runs take effect at once. # pad_layout = auto +# Gamepad stick dead zone in percent of full deflection (0 to 50). Raise it for a stick that drifts: inside the circle +# the stick reads centred, and the rest of its travel is scaled to start there. +# stick_deadzone = 8 # Mouse camera sensitivity (0 to 10; 0 = no mouse camera and no cursor capture). # mouse_sens = 1 # Write the input of this run to a file that replays as a script (docs/controls.md, "Scripted input"). Default: off. From 42a1d32710b290d55e56e68f8986df7d4c217156 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 00:20:41 +0300 Subject: [PATCH 03/34] Skip audio that piles up in the output ring so the sound delay stays short A stalled audio device (a wireless headset waking up, a change of the default device) filled the 340 ms ring, the render thread then dropped new frames and the queue stayed full for the rest of the session: about a third of a second of delay on top of the device's own. ring_pull now skips what is queued beyond the prefill, the device's request and 40 ms. The 5 s level line (trace = audio) reports the queue and the skipped frames. With the SDL audio thread suspended for 1.5 s: before, 325-340 ms queued for the rest of the run; after, 10 ms. Co-Authored-By: Claude Opus 5.5 --- runtime/audio_mixer.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/runtime/audio_mixer.c b/runtime/audio_mixer.c index bbb29ae8..a34885c7 100644 --- a/runtime/audio_mixer.c +++ b/runtime/audio_mixer.c @@ -9,6 +9,9 @@ // single-producer single-consumer ring. The SDL audio thread drains that ring without taking any lock: an empty ring // plays silence and never blocks the render thread or the guest. TOTK_AUDIO=none (or TOTK_AUDIO_OFF) mixes without a // device; TOTK_AUDIO_PREFILL_MS (0..500, default 20) is the queued audio the output waits for after start or underrun. +// The render thread runs on the host clock and the device on its own, and after a stall the render thread catches up in +// a burst, so audio can pile up in the ring; the SDL thread skips what is queued beyond the prefill plus QUEUE_SLACK_MS, +// which keeps the delay from the game to the speakers at prefill plus one device period instead of up to the ring size. #include #include #include @@ -26,6 +29,7 @@ #define MAX_FRAME 1024 #define RING_FRAMES 16384 // stereo frames, about 340 ms at 48 kHz +#define QUEUE_SLACK_MS 40 // queued audio above prefill plus the device's request that ring_pull skips #define SPLITTER_UNUSED 0xffffffffu #define dm_guest_write_prepare(p, n) ((void)(p), (void)(n)) // guest memory is plain host memory on Windows @@ -43,7 +47,7 @@ typedef struct Mixer { SplitInfo *split; int nsplit; SplitDest *dest; int ndest; SDL_AudioStream *stream; int device_ok, rate; float ring[RING_FRAMES * 2]; - _Atomic uint64_t wr, rd, underruns, overruns, missing_frames, callback_frames, callback_nonzero; + _Atomic uint64_t wr, rd, underruns, overruns, missing_frames, callback_frames, callback_nonzero, skipped_frames; _Atomic int playing; int ready; // ready: the callback's own state _Atomic uint64_t flush_until; _Atomic unsigned stop_epoch; unsigned callback_epoch; EffectState *effect; int *effect_order; unsigned effect_reported; @@ -107,16 +111,22 @@ static void ring_pull(Mixer *m, float *out, uint32_t frames) { atomic_store_explicit(&m->rd, wr, memory_order_release); return; } + uint64_t prefill = (uint64_t)m->frame * (uint64_t)m->prefill; + if (prefill < frames) prefill = frames; + if (prefill > RING_FRAMES) prefill = RING_FRAMES; if (!m->ready) { - uint64_t prefill = (uint64_t)m->frame * (uint64_t)m->prefill; - if (prefill < frames) prefill = frames; - if (prefill > RING_FRAMES) prefill = RING_FRAMES; if (have < prefill) { memset(out, 0, (size_t)frames * 2 * sizeof(float)); return; } m->ready = 1; } + uint64_t keep = prefill + frames, slack = (uint64_t)m->rate * QUEUE_SLACK_MS / 1000; + if (have > keep + slack) { // audio piled up (render burst, clock drift): skip the oldest, keep prefill + request + atomic_fetch_add(&m->skipped_frames, have - keep); + rd = wr - keep; + have = keep; + } uint32_t n = frames < have ? frames : (uint32_t)have; uint64_t nonzero = 0; for (uint32_t i = 0; i < n; i++) { @@ -209,12 +219,14 @@ static void level_note(AudioRenderer *r, Mixer *m, const float *lr, int frames) rt_log("audio: first nonzero output after %llu frames", (unsigned long long)r->elapsed_frames); } if (m->level_frames >= (uint64_t)r->param.sample_rate * 5) { + uint64_t wr = atomic_load(&m->wr), queued = wr - atomic_load(&m->rd); rt_log("audio: level rms %.4f peak %.4f over %.1f s; written %llu frames, underruns %llu, overruns %llu, " - "missing frames %llu, callback frames %llu (nonzero %llu)", + "missing frames %llu, callback frames %llu (nonzero %llu), queued %.1f ms, skipped %llu frames", sqrt(m->level_sum / (double)m->level_count), m->level_peak, m->level_frames / (double)r->param.sample_rate, - (unsigned long long)atomic_load(&m->wr), (unsigned long long)atomic_load(&m->underruns), + (unsigned long long)wr, (unsigned long long)atomic_load(&m->underruns), (unsigned long long)atomic_load(&m->overruns), (unsigned long long)atomic_load(&m->missing_frames), - (unsigned long long)atomic_load(&m->callback_frames), (unsigned long long)atomic_load(&m->callback_nonzero)); + (unsigned long long)atomic_load(&m->callback_frames), (unsigned long long)atomic_load(&m->callback_nonzero), + queued * 1000.0 / r->param.sample_rate, (unsigned long long)atomic_load(&m->skipped_frames)); m->level_sum = m->level_peak = 0; m->level_count = m->level_frames = 0; } From 7d89fde2f0fd1cde07285d08b303aeb3c8cfe9a2 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 00:22:08 +0300 Subject: [PATCH 04/34] Drop the six-axis samples from before StartSixAxisSensor motion_restart only flagged the next sample to start a fresh frame, so a read in the frame of the Start returned the previous session's 16 states as live, and for the next 60 ms 15 of the 16 were from before it, with the angle jumping back to zero between two reads. Motion aim was dead when the bow was first drawn and lagged after that. A restart now hides the older samples and zeroes the angle; sampling numbers keep counting. --probe-services checks a restart in the middle of a turn: 16 old states in the Start frame before, none after. Reported with the cause and a harness in #104. Co-Authored-By: Claude Opus 5.5 --- runtime/motion_fusion.c | 13 +++++++++---- runtime/ov_probe.c | 14 +++++++++++++- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/runtime/motion_fusion.c b/runtime/motion_fusion.c index 0b4c56b7..76df5064 100644 --- a/runtime/motion_fusion.c +++ b/runtime/motion_fusion.c @@ -7,8 +7,9 @@ // axis, direction = the controller's three axes in a reference frame whose z is up. The orientation is integrated from // the gyro and, while the controller is not being accelerated (|a| within 15 % of 1 G), pulled toward the measured // gravity (Mahony, gain 0.5/s: about 2 s), as the Switch's sensor fusion does; yaw has no reference and drifts with the -// gyro's bias. The newest MOTION_RING samples are kept; reads are newest first. The feeding thread writes, guest -// threads read, both under the instance's SRW lock. +// gyro's bias. The newest MOTION_RING samples are kept; reads are newest first. A restart (StartSixAxisSensor) drops +// the samples before it: reads return nothing until the next sample, which starts a fresh frame and angle; sampling +// numbers go on. The feeding thread writes, guest threads read, both under the instance's SRW lock. #include #include #include @@ -24,6 +25,7 @@ struct MotionFusion { SRWLOCK lock; MotionState ring[MOTION_RING]; uint64_t produced, last_ns; + uint64_t first; // produced at the last restart: older samples are not read double R[3][3]; // controller -> reference; row i = reference axis i in controller coordinates double angle[3]; int restart; @@ -131,8 +133,8 @@ int motion_read(MotionFusion *m, MotionState *out, int max, uint64_t now) { if (max <= 0) return 0; int n = 0; AcquireSRWLockShared(&m->lock); - if (m->produced && (now < m->last_ns || now - m->last_ns < STALE_NS)) { - uint64_t have = m->produced < MOTION_RING ? m->produced : MOTION_RING; + if (m->produced > m->first && (now < m->last_ns || now - m->last_ns < STALE_NS)) { + uint64_t have = m->produced - m->first < MOTION_RING ? m->produced - m->first : MOTION_RING; n = (int)(have < (uint64_t)max ? have : (uint64_t)max); for (int i = 0; i < n; i++) out[i] = m->ring[(m->produced - 1 - (uint64_t)i) % MOTION_RING]; } @@ -143,5 +145,8 @@ int motion_read(MotionFusion *m, MotionState *out, int max, uint64_t now) { void motion_restart(MotionFusion *m) { AcquireSRWLockExclusive(&m->lock); m->restart = 1; + m->first = m->produced; + m->last_ns = 0; // the first sample after the restart integrates nothing across the stop + memset(m->angle, 0, sizeof m->angle); ReleaseSRWLockExclusive(&m->lock); } diff --git a/runtime/ov_probe.c b/runtime/ov_probe.c index 4668bf0a..c0ffa181 100644 --- a/runtime/ov_probe.c +++ b/runtime/ov_probe.c @@ -552,9 +552,21 @@ static void probe_feedback(uint32_t pid, uint32_t style) { near3(ms[0].dir + 3, -1, 0, 0, 0.01) && near3(ms[0].dir + 6, 0, 0, 1, 0.01), "fusion quarter turn: %d states, sampling %lld, angle %.4f rev, x axis (%.3f %.3f %.3f)", n, (long long)ms[0].sampling, (double)ms[0].angle[2], (double)ms[0].dir[0], (double)ms[0].dir[1], (double)ms[0].dir[2]); + double quarter = ms[0].angle[2]; CHECK(motion_read(m, ms, 16, t + 300000000ull) == 0, "fusion states 300 ms after the last sample"); + // StartSixAxisSensor again in the middle of the turn (the bow drawn again): the states from before it are gone, and + // the first one after it starts a fresh angle in a fresh frame; sampling numbers go on. + for (int i = 0; i < 250; i++) motion_feed(m, down, turn, t += 4000000); + motion_restart(m); + int same = motion_read(m, ms, 16, t + 1000000); + motion_feed(m, down, turn, t += 4000000); + int next = motion_read(m, ms, 16, t + 1000000); + CHECK(same == 0 && next == 1 && ms[0].sampling == 502 && fabs(ms[0].angle[2]) < 0.002 && + near3(ms[0].dir, 1, 0, 0, 0.01), + "fusion restart: %d states in the frame of the Start, %d after one sample (sampling %lld, angle %.4f rev)", same, + next, (long long)ms[0].sampling, (double)ms[0].angle[2]); printf("hid feedback: vibration handles %#x %#x, six-axis %#x, at rest when stopped; fusion quarter turn %.4f rev\n", - vh[0], vh[1], sh[0], (double)ms[0].angle[2]); + vh[0], vh[1], sh[0], quarter); } static void probe_input(void) { From 605662ca87613bfcfaf13045fd26c5c7487742d8 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 00:25:05 +0300 Subject: [PATCH 05/34] Offer 1280x720 and 1600x900 as game resolutions in the menu The game resolution row started at 1920x1080 although resolution.c accepts sizes from 1280x720, so weak GPUs and integrated graphics had to edit totk.ini to draw the game smaller. A 1280x720 run on 1.4.3 renders the field with a 1067x600 scene and an intact interface. Co-Authored-By: Claude Opus 5.5 --- runtime/ui/menu.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/runtime/ui/menu.cpp b/runtime/ui/menu.cpp index 7d52baa9..162894d5 100644 --- a/runtime/ui/menu.cpp +++ b/runtime/ui/menu.cpp @@ -414,10 +414,10 @@ static void row_window_size(void) { opt_choice_list("TOTK_WINDOW_SIZE", UI_MENU_WINDOW_SIZE, pv, pn, n, UI_MENU_WINDOW_SIZE_DESC); } -// The game resolution (resolution.c): 16:9 sizes from the game's own 1920x1080 up, and the current value when it is -// another size. +// The game resolution (resolution.c): 16:9 sizes from its smallest, 1280x720 (for weak GPUs), up, and the current value +// when it is another size. static void row_output_res(void) { - static const char *const vals[] = {"1920x1080", "2560x1440", "3200x1800", "3840x2160"}; + static const char *const vals[] = {"1280x720", "1600x900", "1920x1080", "2560x1440", "3200x1800", "3840x2160"}; enum { N = sizeof vals / sizeof *vals }; char shown[N][32]; const char *names[N]; From a76224f8a6ce85221af6df21b45281a7eafdfd73 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 00:42:03 +0300 Subject: [PATCH 06/34] Fail the install when romfs/ is gone or unreadable after cleanup An install whose romfs/ went empty and unreadable after the extraction had verified it (a folder Windows was still deleting) ended with result ok, and the game stopped at its first file read. Cleanup now checks that romfs.bin and index.json are still there, match the verified size and hash record, and that romfs.bin opens; otherwise it fails with cleanup.romfs_unreadable, which tells the player to restart Windows and run the installer on the same folder (resume extracts the game data again). Reported in #107. tests/installer_test.py CleanupTest: an emptied, removed or resized romfs passed cleanup before, fails now. Co-Authored-By: Claude Opus 5.5 --- installer/steps.py | 16 +++++++++++++ installer/text/en.json | 1 + installer/text/ru.json | 1 + tests/installer_test.py | 52 ++++++++++++++++++++++++++++++++++++++++- 4 files changed, 69 insertions(+), 1 deletion(-) diff --git a/installer/steps.py b/installer/steps.py index 3984f784..9d31fb13 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -661,8 +661,24 @@ class Cleanup(Step): remove(stale) for name in ('gen', 'build', 'check', 'shader_list.tsv'): remove(ctx.temp / name) + romfs_problem(dest / 'romfs', expected_update(ctx)) return {} +def romfs_problem(romfs, expected): + """Raise cleanup.romfs_unreadable unless romfs/ still holds the verified image. The game opens romfs.bin at start; + an install whose romfs/ went empty and unreadable after the extraction verified it (a folder Windows was still + deleting, #107) reported success and the game stopped at its first file read.""" + try: + ok = extract.romfs_verified(romfs, expected) and (romfs / 'romfs.bin').stat().st_size == expected['romfs']['size'] + if ok: + with open(romfs / 'romfs.bin', 'rb') as f: + f.read(1) + except (OSError, ValueError): + ok = False + if not ok: + raise InstallError('cleanup.romfs_unreadable', path=str(romfs)) + + def all_steps(): return [Check(), Extract(), Lift(), Compile(), Prepare(), Cleanup()] diff --git a/installer/text/en.json b/installer/text/en.json index 5964a0f6..3425f5a5 100644 --- a/installer/text/en.json +++ b/installer/text/en.json @@ -75,6 +75,7 @@ "prepare.shader_list_missing": "The shader list is missing.", "prepare.romfs_patches_failed": "Writing the game data patches failed (exit code {code}). Details: {log}", "prepare.shaders_failed": "Preparing the shaders failed (exit code {code}). Details: {log}", + "cleanup.romfs_unreadable": "The game data in {path} is missing or cannot be read after the install. Restart Windows, then run the installer again on the same folder: it extracts the game data again and keeps the rest.", "cleanup.ini_failed": "totk.ini cannot be written: {detail}", "warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.", diff --git a/installer/text/ru.json b/installer/text/ru.json index c3a87db8..b1dc33ae 100644 --- a/installer/text/ru.json +++ b/installer/text/ru.json @@ -75,6 +75,7 @@ "prepare.shader_list_missing": "Нет списка шейдеров.", "prepare.romfs_patches_failed": "Не удалось записать патчи данных игры (код выхода {code}). Подробности: {log}", "prepare.shaders_failed": "Не удалось подготовить шейдеры (код выхода {code}). Подробности: {log}", + "cleanup.romfs_unreadable": "Данные игры в {path} пропали или не читаются после установки. Перезагрузите Windows и запустите установщик снова на ту же папку: он заново распакует данные игры, остальное сохранится.", "cleanup.ini_failed": "Не удаётся записать totk.ini: {detail}", "warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.", diff --git a/tests/installer_test.py b/tests/installer_test.py index d7404929..036d897a 100644 --- a/tests/installer_test.py +++ b/tests/installer_test.py @@ -14,7 +14,7 @@ from types import SimpleNamespace ROOT = Path(__file__).resolve().parent.parent sys.path.insert(0, str(ROOT)) -from installer import ini, runner, state # noqa: E402 +from installer import ini, runner, state, steps # noqa: E402 from installer.events import Reporter # noqa: E402 from installer.layout import Layout # noqa: E402 @@ -241,6 +241,56 @@ class IniTest(unittest.TestCase): self.assertEqual(ini.get_language(text), 'ja') +class CleanupTest(unittest.TestCase): + """The real Cleanup step on a finished install whose romfs/ is intact, emptied or gone (#107).""" + + def setUp(self): + self.steps = steps + self.tmp = tempfile.TemporaryDirectory() + base = Path(self.tmp.name) + self.dest, temp = base / 'dest', base / 'temp' + (self.dest / 'romfs').mkdir(parents=True) + temp.mkdir() + image = b'romfs image' + (self.dest / 'romfs' / 'romfs.bin').write_bytes(image) + digest = steps.extract.sha256(self.dest / 'romfs' / 'romfs.bin') + (self.dest / 'romfs' / 'index.json').write_text(json.dumps({'romfs_size': len(image), 'romfs_sha256': digest})) + self.expected = {'romfs': {'size': len(image), 'sha256': digest}} + self.ctx = SimpleNamespace(reporter=Reporter(out=io.StringIO(), log_path=base / 'x.log'), dest=self.dest, + temp=temp, lang='en-US', state={}, + layout=SimpleNamespace(ini_example=ROOT / 'totk.ini.example')) + self.saved = steps.expected_update + steps.expected_update = lambda ctx: self.expected + + def tearDown(self): + self.steps.expected_update = self.saved + self.ctx.reporter.close() + self.tmp.cleanup() + + def run_cleanup(self): + try: + self.steps.Cleanup().run(self.ctx) + return 'ok' + except self.steps.InstallError as e: + return e.id + + def test_intact_romfs_passes(self): + self.assertEqual(self.run_cleanup(), 'ok') + + def test_emptied_romfs_fails(self): + for p in (self.dest / 'romfs').iterdir(): + p.unlink() + self.assertEqual(self.run_cleanup(), 'cleanup.romfs_unreadable') + + def test_missing_romfs_fails(self): + self.steps.remove(self.dest / 'romfs') + self.assertEqual(self.run_cleanup(), 'cleanup.romfs_unreadable') + + def test_romfs_of_another_size_fails(self): + (self.dest / 'romfs' / 'romfs.bin').write_bytes(b'short') + self.assertEqual(self.run_cleanup(), 'cleanup.romfs_unreadable') + + class EventsTest(unittest.TestCase): def test_lines_are_json_and_the_log_is_english(self): with tempfile.TemporaryDirectory() as d: From 9016b1e21072e35f85b3dcd1758d47431d32151e Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 00:46:24 +0300 Subject: [PATCH 07/34] Add a virtual test gamepad for stick tests without a controller TOTK_TEST_PAD=,,, attaches an SDL virtual gamepad at start with its sticks at those raw values, so the dead zone and the gamepad path can be tested in a run. On the field save: an 11% drift reached the game as 3.3% at the default 8% zone (the old per-axis cutoff passed 9% on x and dropped y) and not at all at 15% (Link stood still); full deflection reached it as full at 15%. Co-Authored-By: Claude Opus 5.5 --- docs/controls.md | 2 ++ runtime/cfg.c | 2 +- runtime/input_win.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/docs/controls.md b/docs/controls.md index b80fc89c..a351fdcd 100644 --- a/docs/controls.md +++ b/docs/controls.md @@ -87,6 +87,8 @@ The stick direction bits (`StickLLeft` to `StickRDown`) are derived from the sti `TOTK_INPUT_RECORD=` (`input_record` in `totk.ini`, or Input recording in the menu) writes the merged input in the same format, one line per button press and per stick segment: events before the world clock starts by tick, the rest timed; a button or stick held when the world clock starts is cut there and goes on as a timed event. The file replays as a script, which makes it the repro to attach to a bug report. +`TOTK_TEST_PAD=,,,` attaches an SDL virtual gamepad at start whose sticks rest at these raw SDL axis values (-32768 to 32767, down is +y), so a drifting or held stick can be tested without a controller: the values go through the gamepad mapping and `TOTK_STICK_DEADZONE` like a real pad's, and `TOTK_INPUT_RECORD` shows what reached the game. + ## RenderDoc capture (blocked) `TOTK_RDOC=[,...]` (`runtime/rdoc_win.c`; `TOTK_RDOC_DIR`, `TOTK_RDOC_DLL`) loads RenderDoc before `vkCreateInstance`. RenderDoc hides `VK_EXT_external_memory_host`, which the back end requires for guest pool imports (`runtime/vk_memory.c`), so device creation fails under it. A capture needs a copy-based pool fallback first. diff --git a/runtime/cfg.c b/runtime/cfg.c index 322917ab..336b24e3 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -48,7 +48,7 @@ static const char *const known[] = { "REPLAY_CHECK", "RESOLVE_CHECK", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", "SHADER_MEMO", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "SWKBD_ANSWER", - "SWKBD_TEST", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "UBO_SNAPSHOT", "UPSCALER", "VK_DYN_VERTEX", "VK_NO_FLOAT64", + "SWKBD_TEST", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "UBO_SNAPSHOT", "UPSCALER", "VK_DYN_VERTEX", "VK_NO_FLOAT64", "VK_PRESENT", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VS_OUTS", "WB_FENCE_LIMIT", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_PROCS", "ZSTD_CHECK", "ZSTD_NATIVE", }; diff --git a/runtime/input_win.c b/runtime/input_win.c index 096fef53..d4f0e78d 100644 --- a/runtime/input_win.c +++ b/runtime/input_win.c @@ -515,12 +515,44 @@ static void binds_refresh(void) { input_map_configure(); } +// TOTK_TEST_PAD=,,, (test runs): an SDL virtual gamepad whose sticks rest at these raw SDL axis values +// (-32768..32767, down is +y), attached before the first scan so it is the gamepad the game reads: drift and full +// deflection go through the same mapping and dead zone as a real pad. +static void test_pad_attach(void) { + static SDL_Joystick *joy; // kept open: a virtual joystick's axes are set through an open handle + const char *e = cfg_get("TOTK_TEST_PAD"); + int v[4]; + if (!e || !*e) return; + if (sscanf(e, "%d,%d,%d,%d", &v[0], &v[1], &v[2], &v[3]) != 4) { + rt_log("input: TOTK_TEST_PAD=%s is not ,,,; no test gamepad", e); + return; + } + SDL_VirtualJoystickDesc d; + SDL_INIT_INTERFACE(&d); + d.type = SDL_JOYSTICK_TYPE_GAMEPAD; + d.naxes = SDL_GAMEPAD_AXIS_COUNT; + d.nbuttons = SDL_GAMEPAD_BUTTON_COUNT; + d.name = "TotK test gamepad"; + SDL_JoystickID id = SDL_AttachVirtualJoystick(&d); + joy = id ? SDL_OpenJoystick(id) : NULL; + if (!joy) { + rt_log("input: test gamepad not attached (%s)", SDL_GetError()); + return; + } + static const SDL_GamepadAxis axes[4] = {SDL_GAMEPAD_AXIS_LEFTX, SDL_GAMEPAD_AXIS_LEFTY, SDL_GAMEPAD_AXIS_RIGHTX, + SDL_GAMEPAD_AXIS_RIGHTY}; + for (int i = 0; i < 4; i++) + SDL_SetJoystickVirtualAxis(joy, axes[i], (Sint16)(v[i] < -32768 ? -32768 : v[i] > 32767 ? 32767 : v[i])); + rt_log("input: test gamepad attached, sticks at %d,%d %d,%d", v[0], v[1], v[2], v[3]); +} + static void start(void) { binds_refresh(); const char *e = cfg_get("TOTK_INPUT"); if (e && !strcmp(e, "none")) { state = -1; rt_log("input: disabled (TOTK_INPUT=none)"); return; } if (!SDL_Init(SDL_INIT_GAMEPAD)) { state = -1; rt_log("input: SDL gamepad init failed (%s), keyboard only", SDL_GetError()); } else state = 1; + if (state == 1) test_pad_attach(); if (state == 1) { scan_pads(); if (!pad) rt_log("input: no gamepad (keyboard only until one is connected)"); From ada341e2b11472a65454d783ac4a019ff8101a50 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 07:12:48 +0300 Subject: [PATCH 08/34] Pace audio rendering by the output device The render thread ran on its own 5 ms timer while the device consumed on its own clock, so the queue crept: on DirectSound it grew 2.7 ms every 5 s, which fills the 340 ms ring in about ten minutes of play. The render thread now renders the next frame when the device has taken the ring below max(prefill, two device periods), as the audio hardware paces the Switch's DSP and as LASRecomp and SavageWindRecomp pace their renderers by the SDL stream. Without a device, while the renderer is stopped and while the device takes nothing for 50 ms (at least four periods) it keeps the timer, so a stalled device does not stall the game; the ring trim then drops what piled up, measured from the target. 60 s on the monsters save: WASAPI 15 ms queued, DirectSound 27-40 ms without growth, no underruns. With the SDL audio thread suspended for 1.5 s the game held 60 fps and the queue was back at 15 ms after one trim. Co-Authored-By: Claude Opus 5.5 --- runtime/audio_mixer.c | 44 ++++++++++++++++++++++++++++++++++++------- runtime/audren.c | 31 ++++++++++++++++++++---------- runtime/audren.h | 4 ++++ 3 files changed, 62 insertions(+), 17 deletions(-) diff --git a/runtime/audio_mixer.c b/runtime/audio_mixer.c index a34885c7..6dc625b8 100644 --- a/runtime/audio_mixer.c +++ b/runtime/audio_mixer.c @@ -9,9 +9,13 @@ // single-producer single-consumer ring. The SDL audio thread drains that ring without taking any lock: an empty ring // plays silence and never blocks the render thread or the guest. TOTK_AUDIO=none (or TOTK_AUDIO_OFF) mixes without a // device; TOTK_AUDIO_PREFILL_MS (0..500, default 20) is the queued audio the output waits for after start or underrun. -// The render thread runs on the host clock and the device on its own, and after a stall the render thread catches up in -// a burst, so audio can pile up in the ring; the SDL thread skips what is queued beyond the prefill plus QUEUE_SLACK_MS, -// which keeps the delay from the game to the speakers at prefill plus one device period instead of up to the ring size. +// The device's clock paces the render thread, as the audio hardware paces the Switch's DSP (audio_mixer_pace_ns): a +// frame is rendered when the ring holds less than the output target, max(prefill, two device periods), so the queue and +// the delay from the game to the speakers stay there and the host and device clocks cannot drift apart. LASRecomp and +// SavageWindRecomp pace their renderers by the SDL stream the same way. Without a device, while the renderer is +// stopped and while the device takes nothing (a stall), the render thread keeps real time on its own timer; what piled +// up meanwhile is skipped on the SDL thread once the device resumes (more than QUEUE_SLACK_MS above the target plus the +// device's request), as MacNativeGames' audio_mixer.c trims its ring after a device stall. #include #include #include @@ -29,7 +33,8 @@ #define MAX_FRAME 1024 #define RING_FRAMES 16384 // stereo frames, about 340 ms at 48 kHz -#define QUEUE_SLACK_MS 40 // queued audio above prefill plus the device's request that ring_pull skips +#define QUEUE_SLACK_MS 40 // queued audio above the target plus the device's request that ring_pull skips +#define STALL_MS 50 // the device took nothing this long (at least 4 periods): the timer paces #define SPLITTER_UNUSED 0xffffffffu #define dm_guest_write_prepare(p, n) ((void)(p), (void)(n)) // guest memory is plain host memory on Windows @@ -49,6 +54,9 @@ typedef struct Mixer { float ring[RING_FRAMES * 2]; _Atomic uint64_t wr, rd, underruns, overruns, missing_frames, callback_frames, callback_nonzero, skipped_frames; _Atomic int playing; int ready; // ready: the callback's own state + int target, period; // output target and device period, frames (output_open) + uint64_t stall_ns; + _Atomic uint64_t pulled_ns; // kern_now_ns of the device's last request, 0 before the first _Atomic uint64_t flush_until; _Atomic unsigned stop_epoch; unsigned callback_epoch; EffectState *effect; int *effect_order; unsigned effect_reported; uint64_t *sink_address; uint32_t *sink_size, *sink_offset, *sink_old, *sink_written; @@ -121,8 +129,9 @@ static void ring_pull(Mixer *m, float *out, uint32_t frames) { } m->ready = 1; } - uint64_t keep = prefill + frames, slack = (uint64_t)m->rate * QUEUE_SLACK_MS / 1000; - if (have > keep + slack) { // audio piled up (render burst, clock drift): skip the oldest, keep prefill + request + uint64_t keep = (prefill > (uint64_t)m->target ? prefill : (uint64_t)m->target) + frames; + uint64_t slack = (uint64_t)m->rate * QUEUE_SLACK_MS / 1000; + if (have > keep + slack) { // piled up while the device stalled: skip the oldest, keep target + request atomic_fetch_add(&m->skipped_frames, have - keep); rd = wr - keep; have = keep; @@ -149,6 +158,7 @@ static void SDLCALL sdl_pull(void *user, SDL_AudioStream *stream, int additional (void)total; Mixer *m = user; float tmp[2 * 1024]; + atomic_store_explicit(&m->pulled_ns, kern_now_ns(), memory_order_release); while (additional > 0) { uint32_t frames = (uint32_t)additional / (2 * sizeof(float)); if (!frames) break; @@ -182,8 +192,28 @@ static void output_open(Mixer *m, int rate) { return; } m->device_ok = 1; + SDL_AudioSpec dev; + int period = 0; + if (!SDL_GetAudioDeviceFormat(SDL_GetAudioStreamDevice(m->stream), &dev, &period) || period <= 0) period = m->frame; + m->period = period; + m->target = m->frame * m->prefill > 2 * period ? m->frame * m->prefill : 2 * period; + if (m->target > RING_FRAMES / 2) m->target = RING_FRAMES / 2; + uint64_t period_ns = (uint64_t)period * 1000000000ull / (uint64_t)rate; + m->stall_ns = 4 * period_ns > STALL_MS * 1000000ull ? 4 * period_ns : STALL_MS * 1000000ull; const char *name = SDL_GetAudioDeviceName(SDL_GetAudioStreamDevice(m->stream)); - rt_log("audio: SDL device '%s' opened, %d Hz stereo float, prefill %d ms", name ? name : "?", rate, m->prefill * 5); + rt_log("audio: SDL device '%s' opened, %d Hz stereo float, prefill %d ms, device period %d frames; the device paces " + "rendering at %.1f ms queued", name ? name : "?", rate, m->prefill * 5, period, m->target * 1000.0 / rate); +} + +int64_t audio_mixer_pace_ns(AudioRenderer *r, uint64_t now) { + Mixer *m = r->mixer; + if (!m || !m->device_ok || !atomic_load_explicit(&m->playing, memory_order_acquire)) return -1; + uint64_t pulled = atomic_load_explicit(&m->pulled_ns, memory_order_acquire); + if (!pulled || (now > pulled && now - pulled > m->stall_ns)) return -1; + uint64_t have = atomic_load(&m->wr) - atomic_load_explicit(&m->rd, memory_order_acquire); + if (have < (uint64_t)m->target) return 0; + int64_t ns = (int64_t)((have - (uint64_t)m->target + 1) * 1000000000ull / (uint64_t)m->rate); + return ns < 500000 ? 500000 : ns; // until the device has taken it below the target } // Render thread only. diff --git a/runtime/audren.c b/runtime/audren.c index f031d031..8a379c0b 100644 --- a/runtime/audren.c +++ b/runtime/audren.c @@ -316,7 +316,8 @@ static void request_update(AudioRenderer *r, const uint8_t *input, uint64_t in_s ReleaseSRWLockExclusive(&r->lock); } -// ---- render tick: one frame every sample_count / sample_rate seconds, on a host thread +// ---- render tick: one frame each time the output device has taken the ring below its target (audio_mixer_pace_ns), +// or every sample_count / sample_rate seconds of the host clock without a device, while stopped or while it stalls static void advance_voice(AudioRenderer *r, ServerVoice *v, double out_seconds) { if (!v->in.in_use || v->state != SV_STARTED || !v->in.sample_rate) return; double want = v->fraction + out_seconds * v->in.sample_rate * (v->in.pitch > 0 ? v->in.pitch : 1.0); @@ -348,6 +349,12 @@ static void advance_voice(AudioRenderer *r, ServerVoice *v, double out_seconds) } } +static void wait_ns(HANDLE timer, uint64_t ns) { + LARGE_INTEGER due = {.QuadPart = -(LONGLONG)((ns + 99) / 100)}; // relative, 100 ns units + if (!SetWaitableTimer(timer, &due, 0, NULL, NULL, FALSE)) rt_fatal("audren: SetWaitableTimer failed (error %lu)", GetLastError()); + WaitForSingleObject(timer, INFINITE); +} + static DWORD WINAPI render_thread(void *p) { AudioRenderer *r = p; SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_HIGHEST); @@ -358,6 +365,19 @@ static DWORD WINAPI render_thread(void *p) { const double frame = (double)r->param.sample_count / r->param.sample_rate; uint64_t next = kern_now_ns(); while (InterlockedCompareExchange(&r->running, 0, 0)) { + uint64_t now = kern_now_ns(); + int64_t pace = audio_mixer_pace_ns(r, now); + if (pace > 0) { + wait_ns(timer, (uint64_t)pace); + continue; + } + if (pace == 0) { + next = now + frame_ns; // where the timer goes on if the device stops pacing + } else { + if (next > now) wait_ns(timer, next - now); + else if (now - next > 100000000ull) next = now; // fell far behind (debugger, sleep): resynchronise + next += frame_ns; + } AcquireSRWLockExclusive(&r->lock); if (r->state == 0) { voices_prepare(r); @@ -374,15 +394,6 @@ static DWORD WINAPI render_thread(void *p) { } ReleaseSRWLockExclusive(&r->lock); kern_event_signal(r->event); - next += frame_ns; - uint64_t now = kern_now_ns(); - if (next > now) { - LARGE_INTEGER due = {.QuadPart = -(LONGLONG)((next - now + 99) / 100)}; // relative, 100 ns units - if (!SetWaitableTimer(timer, &due, 0, NULL, NULL, FALSE)) - rt_fatal("audren: SetWaitableTimer failed (error %lu)", GetLastError()); - WaitForSingleObject(timer, INFINITE); - } else if (now - next > 100000000ull) - next = now; // fell far behind (debugger, sleep): resynchronise } CloseHandle(timer); return 0; diff --git a/runtime/audren.h b/runtime/audren.h index 92da770d..6dc74bc7 100644 --- a/runtime/audren.h +++ b/runtime/audren.h @@ -180,5 +180,9 @@ void audio_mixer_update_effects(AudioRenderer *r); void audio_mixer_effect_result(AudioRenderer *r, int id, void *output, unsigned size); uint32_t audio_mixer_sink_written(AudioRenderer *r, int id); void audio_mixer_render(AudioRenderer *r, double seconds); // after audio_voices_prepare, before voices advance +// Device pacing for the render thread at kern_now_ns `now`: 0 renders the next frame now, a positive value waits that +// many ns (the ring is at its target), -1 leaves the pace to the render thread's timer (no device, renderer stopped, +// device stalled). +int64_t audio_mixer_pace_ns(AudioRenderer *r, uint64_t now); int audio_buffer_empty(const AudrenWaveBuffer *b); // no samples to decode: played as soon as it is reached void audio_device_handler(IpcSession *s, const IpcRequest *rq, IpcResponse *rs); // IAudioDevice From 31fff747767155a039f8a0a3f276fc36e35a671a Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Sat, 31 Oct 2026 07:23:04 +0300 Subject: [PATCH 09/34] Log the memory commit charge with every fatal error #94 ended on a realloc of 1 MiB that failed with 64 GB of commit allowed, and the log could not tell whether the process had grown towards the limit or the system ran short. rt_fatal now adds one line with the process's private bytes and the system's commit in use and limit (plat_commit_info). Co-Authored-By: Claude Opus 5.5 --- runtime/dispatch.c | 6 ++++++ runtime/platform/platform.h | 3 +++ runtime/platform/win32.c | 12 ++++++++++++ 3 files changed, 21 insertions(+) diff --git a/runtime/dispatch.c b/runtime/dispatch.c index e9bc46a9..71e326ab 100644 --- a/runtime/dispatch.c +++ b/runtime/dispatch.c @@ -12,6 +12,7 @@ // is also fixed at init; they are looked up only after the flat table misses. #include "dispatch.h" #include "host.h" +#include "platform/platform.h" #include #include @@ -51,6 +52,11 @@ void rt_fatal(const char *fmt, ...) { rt_vlogf(fmt, ap); rt_log_raw("\n", 1); va_end(ap); + // A failed allocation (malloc, realloc, VirtualAlloc) means the commit limit was reached: this line tells a leak of + // ours (process private bytes near the limit) from a machine short of page file. + double proc, used, limit; + plat_commit_info(&proc, &used, &limit); + rt_log("fatal: memory commit: this process %.0f MB, the system %.0f of %.0f MB", proc, used, limit); fflush(stdout); fflush(stderr); _Exit(70); diff --git a/runtime/platform/platform.h b/runtime/platform/platform.h index 17898d8f..c5761651 100644 --- a/runtime/platform/platform.h +++ b/runtime/platform/platform.h @@ -116,6 +116,9 @@ void plat_cpu_load(PlatLoad *l, float *total_pct, float *proc_pct, float *temp_c void plat_gpu_load(PlatLoad *l, float *load_pct, float *vram_used_mb); // Physical memory in MB: in use and installed on the machine, and this process's working set. void plat_mem_info(float *used_mb, float *total_mb, float *proc_mb); +// Commit charge in MB: this process's private bytes, the system's commit in use and its limit (RAM plus page files). +// An allocation fails when the system's reaches the limit, however much physical memory is free. -1 when unknown. +void plat_commit_info(double *proc_mb, double *used_mb, double *limit_mb); // ---- child processes: a program with stdin and stdout piped to us (stderr discarded), no console window. typedef struct PlatChild PlatChild; diff --git a/runtime/platform/win32.c b/runtime/platform/win32.c index c053d4d5..6ab2fbdb 100644 --- a/runtime/platform/win32.c +++ b/runtime/platform/win32.c @@ -613,6 +613,18 @@ void plat_mem_info(float *used_mb, float *total_mb, float *proc_mb) { if (K32GetProcessMemoryInfo(GetCurrentProcess(), &pmc, sizeof pmc)) *proc_mb = (float)(pmc.WorkingSetSize >> 20); } +void plat_commit_info(double *proc_mb, double *used_mb, double *limit_mb) { + *proc_mb = *used_mb = *limit_mb = -1; + MEMORYSTATUSEX ms = {.dwLength = sizeof ms}; + if (GlobalMemoryStatusEx(&ms)) { + *limit_mb = (double)(ms.ullTotalPageFile >> 20); + *used_mb = (double)((ms.ullTotalPageFile - ms.ullAvailPageFile) >> 20); + } + PROCESS_MEMORY_COUNTERS_EX pmc; + if (K32GetProcessMemoryInfo(GetCurrentProcess(), (PROCESS_MEMORY_COUNTERS *)&pmc, sizeof pmc)) + *proc_mb = (double)(pmc.PrivateUsage >> 20); +} + // ---- child processes struct PlatChild { From 2fcb09d609f47030e37abb3abcd995a24995d432 Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:26:13 +0200 Subject: [PATCH 10/34] Read shader translations, use masks and stored inputs from shader-cache.pack before the loose files --- runtime/gfx_shader.c | 147 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 136 insertions(+), 11 deletions(-) diff --git a/runtime/gfx_shader.c b/runtime/gfx_shader.c index 4df23fea..15e2343a 100644 --- a/runtime/gfx_shader.c +++ b/runtime/gfx_shader.c @@ -18,6 +18,11 @@ // (64 hex digits) overrides it. A translator change therefore changes every cache key without a hand-made KEY_VERSION // bump; KEY_VERSION is for changes on this side (key layout, how the output is used). // +// Pack: tools/pack_shader_cache.py folds the .spv, .res.json, .use and in/*.in files into /shader-cache.pack +// (one file instead of four per shader: ~117,000 files after an install) and deletes them; the installer runs it after +// preparing the shaders. The pack is read-only here. Every read looks in the pack first and then for the loose file, +// and new translations are written as loose files as before, until the tool runs again (section "cache pack"). +// // Stored inputs: every translation also writes /in/.in (magic "TKSI"; stage, the full answers, the // control section, the code up to its length, c1), so a pipeline key record (vk_pipeline.c), which names its shaders by // input id, can be translated again under a new translator without guest memory (gfx_shader_load_input, @@ -125,6 +130,94 @@ static INIT_ONCE paths_once = INIT_ONCE_STATIC_INIT; #define read_file plat_file_read #define write_file plat_file_write +// ---- cache pack (/shader-cache.pack, written by tools/pack_shader_cache.py) +// Little-endian: "TKSPACK1", u32 count, u32 0, then count entries sorted by (kind, name), then the data. name is the +// 32 bytes the loose file's 64 hex digits spell. The index is read once (48 bytes a file: ~5.6 MB after an install); +// the data is read per lookup with positioned reads on one handle, safe from every thread. A pack that fails a check is +// ignored as a whole and the loose files are used. +enum { PACK_SPV = 1, PACK_RES = 2, PACK_USE = 3, PACK_IN = 4 }; +typedef struct { uint8_t name[32]; uint32_t kind, size; uint64_t offset; } PackEnt; +_Static_assert(sizeof(PackEnt) == 48, "shader-cache.pack entry layout"); +static HANDLE pack_file = INVALID_HANDLE_VALUE; +static PackEnt *pack_ix; +static uint32_t pack_n; + +static int pack_cmp(const PackEnt *a, uint32_t kind, const uint8_t name[32]) { + if (a->kind != kind) return a->kind < kind ? -1 : 1; + return memcmp(a->name, name, 32); +} + +static int pack_read_at(uint64_t offset, void *out, uint32_t n) { + OVERLAPPED o = {0}; + o.Offset = (DWORD)offset, o.OffsetHigh = (DWORD)(offset >> 32); + DWORD got = 0; + return ReadFile(pack_file, out, n, &got, &o) && got == n; +} + +// Opens /shader-cache.pack and reads its index (init_paths). Nothing happens when there is none. +static void pack_open(void) { + char p[PATH_CAP + 32]; + wchar_t w[PATH_CAP + 32]; + snprintf(p, sizeof p, "%s\\shader-cache.pack", cache_dir); + if (!MultiByteToWideChar(CP_UTF8, 0, p, -1, w, PATH_CAP + 32)) return; + HANDLE h = CreateFileW(w, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL, NULL); + if (h == INVALID_HANDLE_VALUE) return; + LARGE_INTEGER size; + uint8_t head[16]; + uint32_t n = 0; + const char *why = NULL; + PackEnt *ix = NULL; + pack_file = h; + if (!GetFileSizeEx(h, &size) || !pack_read_at(0, head, sizeof head) || memcmp(head, "TKSPACK1", 8)) why = "header"; + if (!why) { + memcpy(&n, head + 8, 4); + if (n > (1u << 24) || (uint64_t)n * sizeof *ix + sizeof head > (uint64_t)size.QuadPart || + !(ix = malloc((size_t)n * sizeof *ix + 1))) + why = "index size"; + else if (!pack_read_at(sizeof head, ix, n * (uint32_t)sizeof *ix)) why = "index read"; + } + for (uint32_t i = 0; !why && i < n; i++) { + if (ix[i].offset + ix[i].size > (uint64_t)size.QuadPart) why = "entry out of bounds"; + else if (i && pack_cmp(&ix[i - 1], ix[i].kind, ix[i].name) >= 0) why = "index order"; + } + if (why) { + rt_log("gfx_shader: %s is not a usable cache pack (%s); reading the loose files", p, why); + free(ix); + CloseHandle(h); + pack_file = INVALID_HANDLE_VALUE; + return; + } + pack_ix = ix, pack_n = n; + rt_log("gfx_shader: cache pack %s: %u files", p, n); +} + +static const PackEnt *pack_find(uint32_t kind, const uint8_t name[32]) { + size_t lo = 0, hi = pack_n; + while (lo < hi) { + size_t mid = lo + (hi - lo) / 2; + int c = pack_cmp(&pack_ix[mid], kind, name); + if (!c) return &pack_ix[mid]; + if (c < 0) lo = mid + 1; + else hi = mid; + } + return NULL; +} + +// A packed file as plat_file_read returns a loose one: malloc'd, NUL after the last byte; NULL when absent or empty. +static uint8_t *pack_read(uint32_t kind, const uint8_t name[32], uint64_t *size) { + const PackEnt *e = pack_find(kind, name); + uint8_t *buf = e && e->size ? malloc((size_t)e->size + 1) : NULL; + if (!buf) return NULL; + if (!pack_read_at(e->offset, buf, e->size)) { + free(buf); + return NULL; + } + buf[e->size] = 0; + *size = e->size; + return buf; +} + // Appends a UTF-8 string as a JSON string body: ASCII as is, quotes and backslashes escaped, the rest as \uXXXX // (UTF-16 units), so the service's stdin encoding does not matter. static void json_esc(char **o, const char *s) { @@ -218,6 +311,7 @@ static BOOL CALLBACK init_paths(PINIT_ONCE once, PVOID param, PVOID *ctx) { if (cache_cfg && *cache_cfg && strlen(cache_cfg) < sizeof cache_dir) strcpy(cache_dir, cache_cfg); else if (exe_dir[0]) snprintf(cache_dir, sizeof cache_dir, "%s\\shader-cache", exe_dir); xlate_identity(); + if (cache_dir[0]) pack_open(); use_preload_start(); return TRUE; } @@ -468,15 +562,15 @@ static int spirv_complete(const uint32_t *w, uint64_t n) { return i == n; } -// Reads .spv and .res.json; NULL if either is missing or malformed (the caller translates again and -// overwrites both). iid: the input id, NULL when the caller sets it. -static GfxShader *disk_load(const uint8_t key[32], const uint8_t *iid) { +// Reads .spv and .res.json from the pack (packed 1) or the loose files; NULL if either is missing or +// malformed. iid: the input id, NULL when the caller sets it. +static GfxShader *disk_load_from(const uint8_t key[32], const uint8_t *iid, int packed) { static volatile LONG n_bad; char hex[65], p[PATH_CAP + 80]; key_hex(key, hex); uint64_t spv_size = 0, res_size = 0; snprintf(p, PATH_CAP + 80, "%s\\%s.res.json", cache_dir, hex); - uint8_t *res = read_file(p, &res_size); + uint8_t *res = packed ? pack_read(PACK_RES, key, &res_size) : read_file(p, &res_size); if (!res) return NULL; if (!reflection_valid((const char *)res, res_size)) { LONG n = InterlockedIncrement(&n_bad); @@ -485,7 +579,7 @@ static GfxShader *disk_load(const uint8_t key[32], const uint8_t *iid) { return NULL; } snprintf(p, PATH_CAP + 80, "%s\\%s.spv", cache_dir, hex); - uint8_t *spv = read_file(p, &spv_size); + uint8_t *spv = packed ? pack_read(PACK_SPV, key, &spv_size) : read_file(p, &spv_size); uint32_t magic = 0; if (spv && spv_size >= 20) memcpy(&magic, spv, 4); if (!spv || spv_size % 4 || magic != 0x07230203 || !spirv_complete((const uint32_t *)spv, spv_size / 4)) { @@ -503,6 +597,13 @@ static GfxShader *disk_load(const uint8_t key[32], const uint8_t *iid) { return s; } +// The pack, then the loose files (a newer translation of a key the pack holds a bad copy of); NULL when neither has a +// valid pair (the caller translates again and writes loose files). +static GfxShader *disk_load(const uint8_t key[32], const uint8_t *iid) { + GfxShader *s = pack_n ? disk_load_from(key, iid, 1) : NULL; + return s ? s : disk_load_from(key, iid, 0); +} + // ---- request building static const char *const tex_names[] = {NULL, "1d", "1darray", "buffer", "2d", "2darray", "2dms", "2dmsarray", "3d", @@ -751,10 +852,25 @@ static DWORD WINAPI use_preload(void *arg) { (void)arg; uint64_t t0 = plat_now_ns(); char pattern[PATH_CAP + 16], p[PATH_CAP + 80]; + int n = 0, bad = 0; + for (uint32_t i = 0; i < pack_n; i++) { // the packed masks; the loose files below add the masks found since + uint32_t u[5]; + UseFile **slot, *x; + if (pack_ix[i].kind != PACK_USE) continue; + if (pack_ix[i].size != sizeof u || !pack_read_at(pack_ix[i].offset, u, sizeof u) || u[0] != USE_MAGIC || + u[1] != USE_VERSION) { + bad++; + continue; + } + if (!*(slot = use_file_slot(pack_ix[i].name)) && (x = malloc(sizeof *x))) { + memcpy(x->name, pack_ix[i].name, 32); + x->next = NULL, x->m.slots = u[2], x->m.attrs = u[3], x->m.fouts = u[4]; + *slot = x, n++; + } + } snprintf(pattern, sizeof pattern, "%s\\*.use", cache_dir); WIN32_FIND_DATAA fd; HANDLE f = FindFirstFileExA(pattern, FindExInfoBasic, &fd, FindExSearchNameMatch, NULL, FIND_FIRST_EX_LARGE_FETCH); - int n = 0, bad = 0; if (f != INVALID_HANDLE_VALUE) { do { UseFile e = {0}; @@ -824,7 +940,8 @@ static int use_find(const uint8_t h[32], UseMask *m) { char p[PATH_CAP + 80]; use_path_of(name, p); uint64_t n = 0; - uint32_t *f = (uint32_t *)read_file(p, &n); + uint32_t *f = (uint32_t *)pack_read(PACK_USE, name, &n); + if (!f) f = (uint32_t *)read_file(p, &n); known = f && n == 20 && f[0] == USE_MAGIC && f[1] == USE_VERSION; if (known) m->slots = f[2], m->attrs = f[3], m->fouts = f[4]; free(f); @@ -1105,7 +1222,7 @@ static void in_store(const uint8_t iid[32], int stage, const uint8_t *control, c const GfxShaderAnswers *a) { char p[PATH_CAP + 80], tmp[PATH_CAP + 120]; in_path(iid, p); - if (file_exists(p)) return; + if (pack_find(PACK_IN, iid) || file_exists(p)) return; GfxShaderAnswers zero = {0}; InHeader h = {IN_MAGIC, IN_VERSION, (uint32_t)stage, r->length, r->c1_len, sizeof(GfxShaderAnswers)}; size_t n = sizeof h + sizeof zero + CONTROL_SIZE + r->length + r->c1_len; @@ -1127,7 +1244,8 @@ static int in_load(const uint8_t iid[32], Inputs *in) { char p[PATH_CAP + 80]; in_path(iid, p); uint64_t n = 0; - uint8_t *f = read_file(p, &n); + uint8_t *f = pack_read(PACK_IN, iid, &n); + if (!f) f = read_file(p, &n); InHeader h; if (!f || n < sizeof h) return free(f), 0; memcpy(&h, f, sizeof h); @@ -1425,7 +1543,7 @@ static DWORD WINAPI warm_worker(void *arg) { key_of_input(iid, key); key_hex(key, hex); snprintf(p, PATH_CAP + 80, "%s\\%s.spv", cache_dir, hex); - if (file_exists(p)) InterlockedIncrement(&w->other_cached); + if (pack_find(PACK_SPV, key) || file_exists(p)) InterlockedIncrement(&w->other_cached); else gfx_shader_load_input(iid, 1); } InterlockedIncrement64(&w->done); @@ -1463,7 +1581,14 @@ static size_t other_inputs(const uint8_t (*named)[32], size_t n_named, uint8_t ( char dir[PATH_CAP + 80]; snprintf(dir, sizeof dir, "%s\\in", cache_dir); InList l = {.named = named, .n_named = n_named}; - plat_file_list(dir, ".in", in_collect, &l); + char name[68]; + for (uint32_t i = 0; i < pack_n; i++) { // the packed inputs, under the name their loose file would have + if (pack_ix[i].kind != PACK_IN) continue; + key_hex(pack_ix[i].name, name); + memcpy(name + 64, ".in", 4); + if (!in_collect(name, &l)) break; + } + plat_file_list(dir, ".in", in_collect, &l); // in_store writes no loose file for a packed input: no duplicates return *out = l.v, l.n; } From bc84ac0b1fc6e62949aa83ccb6830ed3b5f3e343 Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:27:03 +0200 Subject: [PATCH 11/34] Add tools/pack_shader_cache.py, which folds the shader cache's loose files into shader-cache.pack --- tools/pack_shader_cache.py | 140 +++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 tools/pack_shader_cache.py diff --git a/tools/pack_shader_cache.py b/tools/pack_shader_cache.py new file mode 100644 index 00000000..0b326d5d --- /dev/null +++ b/tools/pack_shader_cache.py @@ -0,0 +1,140 @@ +"""Fold the loose files of a shader cache into /shader-cache.pack and delete them. + +The translator writes four files per shader into the cache (runtime/gfx_shader.c): .spv, .res.json, +.use and in/.in, about 117,000 files after an install, which are slow to copy, back up, scan and sync. +totk.exe reads the pack first and the loose files after it, and keeps writing new translations as loose files, so +this tool can run again at any time the game is closed: it merges the existing pack with the loose files (a loose file +wins over a packed one of the same name) and writes the pack anew. The installer runs it after preparing the shaders. + +Layout, little-endian (gfx_shader.c, section "cache pack"): b'TKSPACK1', u32 count, u32 0, then count entries of 48 +bytes sorted by (kind, name): name (the 32 bytes the file name's 64 hex digits spell), u32 kind (1 .spv, 2 .res.json, +3 .use, 4 in/*.in), u32 size, u64 offset; then the data. + + python tools\\pack_shader_cache.py \\shader-cache [--keep] + +--keep leaves the loose files in place. Close the game first: a running totk.exe has read the old pack's index. +""" +import argparse +import os +import re +import struct +import sys +from pathlib import Path + +MAGIC = b'TKSPACK1' +PACK = 'shader-cache.pack' +KINDS = {'.spv': 1, '.res.json': 2, '.use': 3, '.in': 4} +NAME = re.compile(r'([0-9a-f]{64})(\.spv|\.res\.json|\.use|\.in)') +USE_SIZE = 20 # gfx_shader.c use_store: magic, version, three masks + + +def loose_files(cache): + """{(kind, name): path} of the cache's loose files; *.in only in in/.""" + found = {} + for folder, kinds in ((cache, (1, 2, 3)), (cache / 'in', (4,))): + if not folder.is_dir(): + continue + with os.scandir(folder) as it: + for e in it: + m = NAME.fullmatch(e.name) + if not m or not e.is_file(): + continue + kind = KINDS[m.group(2)] + if kind in kinds: + found[(kind, bytes.fromhex(m.group(1)))] = Path(e.path) + return found + + +def read_pack(path): + """{(kind, name): (offset, size)} of an existing pack; {} when there is none or it fails a check.""" + try: + f = open(path, 'rb') + except FileNotFoundError: + return {} + with f: + size = os.fstat(f.fileno()).st_size + head = f.read(16) + if len(head) != 16 or head[:8] != MAGIC: + print(f'{path}: not a cache pack, rebuilt from the loose files only', flush=True) + return {} + count = struct.unpack_from(' 0 + + +def pack(cache, keep=False): + cache = Path(cache) + out = cache / PACK + packed = read_pack(out) + loose = loose_files(cache) + keys = sorted(set(packed) | set(loose)) + tmp = out.with_name(PACK + '.tmp') + index, written, skipped, offset = [], [], 0, 16 + 48 * len(keys) + old = open(out, 'rb') if packed else None + try: + with open(tmp, 'wb') as f: + f.write(bytes(offset)) # header and index, written once the offsets are known + for key in keys: + data = None + if key in loose: # a valid loose file, else the packed copy, else nothing + try: + data = loose[key].read_bytes() + except OSError: + pass + if data is not None and valid(key[0], data): + written.append(loose[key]) + else: + data = None + if data is None and key in packed: + old.seek(packed[key][0]) + data = old.read(packed[key][1]) + if data is None or not valid(key[0], data): + skipped += 1 + continue + f.write(data) + index.append(struct.pack('<32sIIQ', key[1], key[0], len(data), offset)) + offset += len(data) + f.seek(0) + f.write(MAGIC + struct.pack(' Date: Fri, 30 Oct 2026 07:27:51 +0200 Subject: [PATCH 12/34] Test tools/pack_shader_cache.py on a fake shader cache --- tests/pack_shader_cache_test.py | 111 ++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 tests/pack_shader_cache_test.py diff --git a/tests/pack_shader_cache_test.py b/tests/pack_shader_cache_test.py new file mode 100644 index 00000000..53356ef7 --- /dev/null +++ b/tests/pack_shader_cache_test.py @@ -0,0 +1,111 @@ +"""tools/pack_shader_cache.py on a fake shader cache: the pack layout gfx_shader.c reads (sorted index, bounds, data), +loose files folded in and deleted, a second run merging new loose files, a loose file winning over its packed copy, +malformed files skipped and --keep. No game files; stdlib only. + +Run: python tests/pack_shader_cache_test.py +""" +import hashlib +import struct +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / 'tools')) +import pack_shader_cache as psc # noqa: E402 + +USE = struct.pack('<5I', 0x4d554b54, 1, 3, 0, 1) # gfx_shader.c use_store + + +def name(seed): + return hashlib.sha256(seed.encode()).hexdigest() + + +def read_pack(path): + """{(kind, name hex): bytes}, checking what gfx_shader.c pack_open checks.""" + b = path.read_bytes() + assert b[:8] == psc.MAGIC + count = struct.unpack_from('= 16 + 48 * count + assert last is None or (last[0], last[1]) < (kind, nm), 'index not sorted by (kind, name)' + last = (kind, nm) + out[(kind, nm.hex())] = b[offset:offset + size] + return out + + +class PackTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.cache = Path(self.tmp.name) / 'shader-cache' + (self.cache / 'in').mkdir(parents=True) + + def tearDown(self): + self.tmp.cleanup() + + def shader(self, seed, spv=b'\x03\x02\x23\x07spirv', res=b'{"x":1}'): + k = name(seed) + (self.cache / f'{k}.spv').write_bytes(spv + seed.encode()) + (self.cache / f'{k}.res.json').write_bytes(res) + (self.cache / f'{name("use" + seed)}.use').write_bytes(USE) + (self.cache / 'in').mkdir(exist_ok=True) # in_store makes it again after a pack removed it + (self.cache / 'in' / f'{name("in" + seed)}.in').write_bytes(b'TKSI' + seed.encode()) + return k + + def test_pack_holds_every_file_and_deletes_the_loose_ones(self): + keys = [self.shader(str(i)) for i in range(50)] + (self.cache / 'notes.txt').write_text('kept') + (self.cache / f'{keys[0]}.code').write_bytes(b'kept too') + n, new, skipped, _ = psc.pack(self.cache) + self.assertEqual((n, new, skipped), (200, 200, 0)) + got = read_pack(self.cache / psc.PACK) + self.assertEqual(got[(1, keys[3])], b'\x03\x02\x23\x07spirv3') + self.assertEqual(got[(2, keys[3])], b'{"x":1}') + self.assertEqual(got[(3, name('use3'))], USE) + self.assertEqual(got[(4, name('in3'))], b'TKSI3') + left = sorted(p.name for p in self.cache.iterdir()) + self.assertEqual(left, sorted([psc.PACK, 'notes.txt', f'{keys[0]}.code'])) # in/ removed when empty + + def test_second_run_merges_new_loose_files_and_loose_wins(self): + a = self.shader('a') + psc.pack(self.cache) + b = self.shader('b') + (self.cache / f'{a}.res.json').write_bytes(b'{"newer":1}') + n, new, _, _ = psc.pack(self.cache) + got = read_pack(self.cache / psc.PACK) + self.assertEqual(n, 8) + self.assertEqual(new, 5) + self.assertEqual(got[(2, a)], b'{"newer":1}') + self.assertEqual(got[(1, b)], b'\x03\x02\x23\x07spirvb') + + def test_malformed_files_are_skipped_or_replaced_by_the_packed_copy(self): + a = self.shader('a') + psc.pack(self.cache) + (self.cache / f'{name("usea")}.use').write_bytes(b'short') # bad loose copy: the packed one stays + (self.cache / f'{name("bad")}.use').write_bytes(b'short') # bad and nowhere else: skipped + (self.cache / f'{name("empty")}.spv').write_bytes(b'') + n, _, skipped, _ = psc.pack(self.cache) + got = read_pack(self.cache / psc.PACK) + self.assertEqual(got[(3, name('usea'))], USE) + self.assertNotIn((3, name('bad')), got) + self.assertEqual((n, skipped), (4, 2)) + self.assertTrue((self.cache / f'{name("bad")}.use').exists()) # never deleted unless packed + + def test_keep_leaves_the_loose_files(self): + k = self.shader('a') + psc.pack(self.cache, keep=True) + self.assertTrue((self.cache / f'{k}.spv').exists()) + self.assertEqual(len(read_pack(self.cache / psc.PACK)), 4) + + def test_a_broken_pack_is_rebuilt_from_the_loose_files(self): + self.shader('a') + (self.cache / psc.PACK).write_bytes(b'garbage') + n, _, _, _ = psc.pack(self.cache) + self.assertEqual(n, 4) + + +if __name__ == '__main__': + unittest.main() From 00da267dcd092f75a6c040aeca3b8b7d287e9a49 Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:29:01 +0200 Subject: [PATCH 13/34] Run tools/pack_shader_cache.py at the end of the prepare step --- installer/steps.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/installer/steps.py b/installer/steps.py index 9d31fb13..b2109fa1 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -587,7 +587,8 @@ class Prepare(Step): def key(self, ctx): layout = ctx.layout xlate_id = layout.xlate.with_suffix('.id') - tools = [layout.tools / n for n in ('prepare_shaders.py', 'shader_census.py', 'make_romfs_patches.py')] + tools = [layout.tools / n for n in ('prepare_shaders.py', 'shader_census.py', 'make_romfs_patches.py', + 'pack_shader_cache.py')] return fingerprint('prepare-1', ctx.keys['compile'], layout.xlate, xlate_id, shader_list_text(layout), *tools) def current(self, ctx, record): @@ -638,6 +639,12 @@ class Prepare(Step): dest / 'exefs', '--list', shader_list], parse) if code or not (dest / 'shader-prepared.bin').is_file(): raise InstallError('prepare.shaders_failed', code=code, log=str(log)) + + # One shader-cache.pack instead of four files per shader (~117,000): totk.exe reads the pack first. + rep.progress('prepare', 'pack', force=True) + code, log = run_tool(ctx, 'prepare', layout.tools / 'pack_shader_cache.py', [dest / 'shader-cache']) + if code: + raise InstallError('prepare.pack_failed', code=code, log=str(log)) return {'xlate_sha256': file_sha256(xlate)} From dc5bd01a152bb8c7fabc9e39fc469b90c1f5ebfd Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:29:42 +0200 Subject: [PATCH 14/34] List tools/pack_shader_cache.py among the files a release needs --- installer/layout.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/installer/layout.py b/installer/layout.py index 655c7c22..affcb63b 100644 --- a/installer/layout.py +++ b/installer/layout.py @@ -72,7 +72,8 @@ class Layout: def missing(self): """Paths the steps need that do not exist.""" need = [self.python, self.tools / 'extract.py', self.tools / 'build_game.py', self.tools / 'prepare_shaders.py', - self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', self.clang, + self.tools / 'pack_shader_cache.py', self.tools / 'make_romfs_patches.py', self.recomp / 'gen.py', + self.clang, self.deps / 'include', self.deps / 'lib', self.runtime, self.runtime_lib, self.ninja, self.hactool, self.xlate, *self.dlls, *(f(v) for v in SUPPORTED for f in (self.patches, self.expected))] if self.shader_list: From 06ae18fe933fa280d8b5973ec09369d72fe0d40a Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:30:32 +0200 Subject: [PATCH 15/34] Add the English texts of the pack phase and its failure --- installer/text/en.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/installer/text/en.json b/installer/text/en.json index 3425f5a5..5e8a5912 100644 --- a/installer/text/en.json +++ b/installer/text/en.json @@ -26,6 +26,7 @@ "phase.prepare.seed": "Reading the shaders from the game data", "phase.prepare.translate": "Translating the shaders", "phase.prepare.libraries": "Building the shaders for the graphics driver", + "phase.prepare.pack": "Packing the shader cache into one file", "phase.cleanup.ini": "Writing totk.ini", "phase.cleanup.temp": "Removing temporary files", @@ -76,6 +77,7 @@ "prepare.romfs_patches_failed": "Writing the game data patches failed (exit code {code}). Details: {log}", "prepare.shaders_failed": "Preparing the shaders failed (exit code {code}). Details: {log}", "cleanup.romfs_unreadable": "The game data in {path} is missing or cannot be read after the install. Restart Windows, then run the installer again on the same folder: it extracts the game data again and keeps the rest.", + "prepare.pack_failed": "Packing the shader cache failed (exit code {code}). Details: {log}", "cleanup.ini_failed": "totk.ini cannot be written: {detail}", "warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.", From 82af209b2b3a9f80ebf301878aabb35010837fd9 Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:31:03 +0200 Subject: [PATCH 16/34] Add the Russian texts of the pack phase and its failure --- installer/text/ru.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/installer/text/ru.json b/installer/text/ru.json index b1dc33ae..fa4d3a76 100644 --- a/installer/text/ru.json +++ b/installer/text/ru.json @@ -26,6 +26,7 @@ "phase.prepare.seed": "Чтение шейдеров из данных игры", "phase.prepare.translate": "Трансляция шейдеров", "phase.prepare.libraries": "Сборка шейдеров для драйвера видеокарты", + "phase.prepare.pack": "Упаковка кэша шейдеров в один файл", "phase.cleanup.ini": "Запись totk.ini", "phase.cleanup.temp": "Удаление временных файлов", @@ -76,6 +77,7 @@ "prepare.romfs_patches_failed": "Не удалось записать патчи данных игры (код выхода {code}). Подробности: {log}", "prepare.shaders_failed": "Не удалось подготовить шейдеры (код выхода {code}). Подробности: {log}", "cleanup.romfs_unreadable": "Данные игры в {path} пропали или не читаются после установки. Перезагрузите Windows и запустите установщик снова на ту же папку: он заново распакует данные игры, остальное сохранится.", + "prepare.pack_failed": "Не удалось упаковать кэш шейдеров (код выхода {code}). Подробности: {log}", "cleanup.ini_failed": "Не удаётся записать totk.ini: {detail}", "warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.", From 58b15ed06977cd91a5b17a2dbc2c9f4f4b5c2393 Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:31:33 +0200 Subject: [PATCH 17/34] Copy tools/pack_shader_cache.py into the release --- tools/make_release.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/make_release.py b/tools/make_release.py index 9fd9d33c..9c3c5f92 100644 --- a/tools/make_release.py +++ b/tools/make_release.py @@ -98,7 +98,7 @@ LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'} # tools/ files the installer steps run (and what they import), copied from the repo. TOOLS_FILES = ['build_game.py', 'build_shaders.py', 'check_content.py', 'extract.py', 'hostmem.py', 'make_romfs_patches.py', - 'prepare_shaders.py', 'romfs_index.py', 'shader_census.py', 'shader_list.py'] + 'pack_shader_cache.py', 'prepare_shaders.py', 'romfs_index.py', 'shader_census.py', 'shader_list.py'] TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes RUNTIME_SUFFIXES = {'.h', '.inc', '.ico'} # .ico: the installer window icon From 4d871c7920490ca82b3e024317424db396803300 Mon Sep 17 00:00:00 2001 From: Pasha Kagan Date: Fri, 30 Oct 2026 07:31:56 +0200 Subject: [PATCH 18/34] Describe the shader cache pack in the installer documentation --- docs/installer.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/installer.md b/docs/installer.md index b6157b83..a3158d28 100644 --- a/docs/installer.md +++ b/docs/installer.md @@ -24,7 +24,7 @@ The install folder (`main.c` `portable_paths()` reads this layout; it stays port exefs/, romfs/ extracted from the player's files profile/ saves logs/ totk.log of the last run, totk.1.log to totk.4.log of the runs before it - shader and pipeline caches prepared by the setup + shader and pipeline caches prepared by the setup; shader-cache/shader-cache.pack holds the translations totk-install.json what built this folder: release version, input hashes, finished steps ``` @@ -50,7 +50,9 @@ keeps `profile/`. Saves carry over both ways: 1.4.3 converts a 1.2.1 save on loa hashes and the game version's patch page) and link them with the prebuilt runtime library into `totk.exe` (`build_game.py --runtime-lib`; about 3 min on 24 cores). 5. **Prepare** the RomFS patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`, - `totk.exe --prepare-shaders`) in the install folder. + `totk.exe --prepare-shaders`) in the install folder, then fold the shader cache's loose files (four per shader, about + 117,000) into `shader-cache/shader-cache.pack` (`tools/pack_shader_cache.py`). totk.exe reads the pack first and + writes the translations it makes in play as loose files; running the tool again with the game closed folds those in. 6. **Clean up** the temp folder (generated C and objects, about 1.3 GB). A cancelled or failed setup resumes from the last finished step (`totk-install.json`). From 70a83837b0ca76001df310e482a209935ddb64b4 Mon Sep 17 00:00:00 2001 From: Seth <29849850+shin2344234@users.noreply.github.com> Date: Fri, 30 Oct 2026 19:54:59 -0500 Subject: [PATCH 19/34] Let the back paddles be bound SDL reports the back buttons of an Xbox Elite, DualSense Edge or Steam Controller as four paddle buttons, but the pad table stopped at the triggers, so they could not be mapped. They are now pad inputs lpaddle1, rpaddle1, lpaddle2 and rpaddle2, with no default action, bindable from totk.ini or the remapping rows. --- docs/controls.md | 2 +- runtime/cfg_probe.c | 7 +++++++ runtime/input_map.c | 2 +- runtime/input_map.h | 5 ++++- runtime/input_win.c | 3 +++ runtime/ui/binds_page.cpp | 3 ++- runtime/ui_text.c | 4 ++++ runtime/ui_text.h | 2 +- totk.ini.example | 1 + 9 files changed, 24 insertions(+), 5 deletions(-) diff --git a/docs/controls.md b/docs/controls.md index a351fdcd..0cceb44a 100644 --- a/docs/controls.md +++ b/docs/controls.md @@ -46,7 +46,7 @@ The *Controls* tab of the settings menu lists, under the button layout, every Sw The tab writes `totk.ini` lines, one per Switch button or stick direction: `bind_