From 152b00494b8afc5176ee7d40c5e9042b0e56ad8e Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 09:25:15 +0300 Subject: [PATCH 1/8] The release compiles plugin_init.c per install: it reads the install's plugin records from patches.h (#845) make_release.py stopped at the runtime library: plugin_init.c includes patches.h, a per-version header, so the library build refused it. Built per install like patches.c, it sees the RT_PLUGINS and RT_PLUGIN_WRITES of the install's own rebuild. Co-Authored-By: Claude Opus 5.5 --- tools/build_game.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/build_game.py b/tools/build_game.py index 84e93682..38b356e8 100644 --- a/tools/build_game.py +++ b/tools/build_game.py @@ -188,8 +188,10 @@ DEFAULT_DEPS = ROOT / 'toolchain' / 'deps-mingw' # Runtime sources compiled per install, not in the library: romfs.c includes romfs_config.h, patches.c and ov_zstd.c # the patches.h of the player's game version (its patch page is the page after that version's main; ov_zstd.c's host # functions sit on it). mouse_direct.c and ov_ability.c also register host callbacks relative to that version's patch -# page. A library source that includes one of PER_VERSION_HEADERS stops the library build (library_sources). -INSTALL_SIDE = {'romfs.c', 'patches.c', 'ov_zstd.c', 'mouse_direct.c', 'ov_ability.c'} +# page. plugin_init.c reads the install's own plugin records from patches.h (RT_PLUGINS, RT_PLUGIN_WRITES): a mod plugin +# the rebuild built in. A library source that includes one of PER_VERSION_HEADERS stops the library build +# (library_sources). +INSTALL_SIDE = {'romfs.c', 'patches.c', 'ov_zstd.c', 'mouse_direct.c', 'ov_ability.c', 'plugin_init.c'} PER_VERSION_HEADERS = ('"patches.h"', '"romfs_config.h"') LIB_NAME = 'libtotkrt.a' AR = UCRT / 'llvm-ar.exe' if WIN else on_path('llvm-ar') From b9b8f3dce1183c8ffa49976f4ad8b22655cd3162 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 09:25:24 +0300 Subject: [PATCH 2/8] Each mod plugin is built or refused on its own: one that is refused, crashes or hangs costs only itself (#845) tools/exefs_plugins.py ran every plugin's init in one service run and refused them all when one was refused. Now a run without inits lists the plugin modules, then each plugin runs in service runs of its own (TOTK_PLUGIN_ONLY, the others stay loaded): a refused svc, a crash (the FATAL line is the reason), a timeout or a failed check refuses that plugin alone, with its reason in plugins.status; the rest are built in. 4 plugins take 2.3 s. At start, plugins_start compares each plugin right after its own init and takes its branches out of main's .text before the next one, so each init sees the game's bytes, as in its service run. Only that init's aliases are flushed: the start stays at 113 ms for Full Inventory Expansion + Quick Weapon Durability (107 ms before). The installer's second lift keeps every plugin of the service runs in the layout when one is built: the records hold that layout's addresses. A refused plugin stays loaded and never starts. A mod with a built plugin and a refused one, used without its code, runs the built one (mods.c; --probe-mods case T). Co-Authored-By: Claude Opus 5.5 --- docs/mods.md | 20 +++-- installer/steps.py | 5 +- runtime/mods.c | 5 +- runtime/mods_probe.c | 40 ++++++--- runtime/plugin_init.c | 147 ++++++++++++++++++--------------- tests/exefs_plugins_test.py | 77 +++++++++++++++++ tests/installer_test.py | 35 ++++++-- tools/exefs_plugins.py | 159 ++++++++++++++++++++++++------------ 8 files changed, 340 insertions(+), 148 deletions(-) diff --git a/docs/mods.md b/docs/mods.md index 089207b9..9e0f1d76 100644 --- a/docs/mods.md +++ b/docs/mods.md @@ -103,12 +103,15 @@ Inventory Expansion's sort fixer, Quick Weapon Durability). On the Switch the pl redirects some of the game's instructions into its own code. The port does the same with the plugin's code compiled ahead of time, after the design in [#845](https://github.com/valentderah/totk-pc-recomp/issues/845) by itaskaev2: -1. The rebuild (the same one as for code patches above) compiles the plugin with the game, starts the game's code - once in a service mode that runs only the plugin's start-up (`totk.exe --plugin-init`), twice, and records which - instructions it redirects. Each redirect becomes a code site of the mod, built into `totk.exe`. -2. At every start the plugin's start-up runs again, before the game's own start-up code, in the same setting. Its - redirects are compared with the record of the rebuild. When they match, the mod's code sites go on. When they do - not, the plugin stays off for this run, the mod's row says why, and the next start runs the rebuild. +1. The rebuild (the same one as for code patches above) compiles the plugins with the game, then starts the game's + code in a service mode that runs only one plugin's start-up (`totk.exe --plugin-init`), twice per plugin, and + records which instructions it redirects. Each redirect becomes a code site of the mod, built into `totk.exe`. Each + plugin is judged on its own: one that is refused, or whose start-up crashes or never returns, stays off with its + reason, and the other plugins are built in. +2. At every start each plugin's start-up runs again, before the game's own start-up code, in the same setting: it + sees the game's own instructions, as in its service run. Its redirects are compared with the record of the + rebuild. When they match, the mod's code sites go on. When they do not, the plugin stays off for this run, the + mod's row says why, and the next start runs the rebuild. 3. The plugin keeps what its start-up made (its memory, its tables), so its code works as on the Switch. The start costs about 50 ms per plugin; the game runs no differently without a plugin. @@ -119,8 +122,9 @@ The rebuild refuses a plugin when its start-up or the code its redirects reach n system calls other than the few its start-up needs (memory queries and maps, the clock, debug output), library calls outside a reviewed list (memory and string functions, the heap, text formatting), threads, services or input; a redirect that is not one of exlaunch's hook shapes (inline hook, replace hook, call-site rewrite); or two runs of its -start-up that differ in code. The row then says **Its plugin ... cannot run in the port** with the reason, and the -mod stays off; **Use without its code** uses the rest of it. UltraCam stays refused: it hooks the graphics API, +start-up that differ in code; or a start-up that crashes the service run or does not return within two minutes. The +row then says **Its plugin ... cannot run in the port** with the reason, and the mod stays off; **Use without its +code** uses the rest of it, including its other plugins that were built in. UltraCam stays refused: it hooks the graphics API, reads input and opens sockets. `plugin_trace = 1` in `totk.ini` logs the first call into each of a plugin's hooks (`plugins: hook entry ... reached`). diff --git a/installer/steps.py b/installer/steps.py index 08ff4baa..2510e376 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -1009,7 +1009,10 @@ class Lift(Step): skipped['warn.plugins_skipped'] = log else: patches = ctx.temp / 'patches-plugins.txt' - lift_plugins = built_plugins(plugins_status(ctx), plugins, ctx.dest / 'mods') + # With one plugin built, the layout keeps every plugin of the service runs: the records hold addresses + # of that layout. A refused plugin stays loaded at play, its init never runs (no patch set). + if built_plugins(plugins_status(ctx), plugins, ctx.dest / 'mods'): + lift_plugins = plugins if lift_plugins: args += ['--plugins', ','.join(str(f) for f in lift_plugins)] code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py', args + ['--patches', patches], parse) diff --git a/runtime/mods.c b/runtime/mods.c index 5d7667da..0d972f51 100644 --- a/runtime/mods.c +++ b/runtime/mods.c @@ -446,11 +446,12 @@ static int scan_cb(const char *name, int kind, uint64_t size, void *ctx) { int unsupported = exefs_check(m, path); // A mod whose code part the port cannot use is off as a whole: its RomFS files may expect that code. Used without // its code (its switch, or mods_without_code = 1 for every mod), the code modules (m->code) and the ExeFS patch - // files the port cannot apply are left out and the rest of the mod is used, when it has a rest. + // files the port cannot apply are left out and the rest of the mod is used, when it has a rest. Each plugin of a + // mod is built or refused on its own (tools/exefs_plugins.py), so a built plugin is such a rest too. m->no_code = listed(cfg_get("TOTK_MODS_NO_CODE"), m->name); m->without_code = m->no_code || cfg_bool("TOTK_MODS_WITHOUT_CODE"); int usable = m->nsets || m->ndata || m->kind & (MOD_ROMFS | MOD_CHEATS); - int code_off = m->code[0] && (!m->without_code || !(m->kind & (MOD_ROMFS | MOD_EXEFS | MOD_CHEATS))); + int code_off = m->code[0] && (!m->without_code || !(m->kind & (MOD_ROMFS | MOD_EXEFS | MOD_CHEATS) || m->nplugins)); m->problem = strchr(m->name, '|') ? MOD_BAD_NAME : code_off ? MOD_PLUGIN : unsupported && (!m->without_code || !usable) ? MOD_EXEFS_UNSUPPORTED diff --git a/runtime/mods_probe.c b/runtime/mods_probe.c index 473ea1e5..0d1e22c0 100644 --- a/runtime/mods_probe.c +++ b/runtime/mods_probe.c @@ -212,6 +212,8 @@ static const char *refused_plugins(void) { // Q an ExeFS patch file on P's site // R a plugin the build has no line for waits for the rebuild // S RomFS + a refused plugin off, the reason kept; used without its code, the rest runs +// T a built plugin + a refused one each judged alone: off for the refused one, its reason kept; used +// without its code, the built one's set runs // Then P's start differs from the build (mods_plugin_off): its set goes off, Q's code runs at the site, the conflict // goes, a stale line marks P's file, and the next scan has P wait for the rebuild. static void probe_plugins(const char *root) { @@ -221,18 +223,22 @@ static void probe_plugins(const char *root) { char q[160]; snprintf(q, sizeof q, "Q/exefs/%s.ips", bid); static const char *const files[] = {"P/romfs/p.bin", "P/exefs/subsdk9", "R/exefs/subsdk9", "S/romfs/s.bin", - "S/exefs/subsdk9"}; + "S/exefs/subsdk9", "T/exefs/subsdk8", "T/exefs/subsdk9"}; for (size_t i = 0; i < sizeof files / sizeof *files; i++) put(root, files[i]); put(root, q); - static char names[3][16]; - static const char *const table[] = {"fps60", names[1], names[2]}; + static char names[4][16]; + static const char *const table[] = {"fps60", names[1], names[2], names[3]}; set_name(names[1], sizeof names[1], "P/exefs/subsdk9"); set_name(names[2], sizeof names[2], q); - char status[256], plugins[256]; + set_name(names[3], sizeof names[3], "T/exefs/subsdk9"); + char status[256], plugins[512]; snprintf(status, sizeof status, "%08X\t%08X\tcode\t-\t\n", fnv(q, strlen(q)), fnv("probe", 5)); snprintf(plugins, sizeof plugins, "%08X\t%08X\tplugin\tfps60\tplugin0, 7 hooks\n" - "%08X\t%08X\tunsupported\t-\tthe code its hooks reach has svc 0x40\n", fnv("P/exefs/subsdk9", 15), - fnv("probe", 5), fnv("S/exefs/subsdk9", 15), fnv("probe", 5)); + "%08X\t%08X\tunsupported\t-\tthe code its hooks reach has svc 0x40\n" + "%08X\t%08X\tplugin\t-\tplugin1, 2 hooks\n" + "%08X\t%08X\tunsupported\t-\tthe run was refused: svc 0x1 at plugin2+0x50a0\n", fnv("P/exefs/subsdk9", 15), + fnv("probe", 5), fnv("S/exefs/subsdk9", 15), fnv("probe", 5), fnv("T/exefs/subsdk9", 15), fnv("probe", 5), + fnv("T/exefs/subsdk8", 15), fnv("probe", 5)); static const int16_t s0[] = {1, 2}; static const RtPatchPick picks[] = {{"main", 0x100, s0, 2}}; mods_probe_cache(status, ""); @@ -244,12 +250,13 @@ static void probe_plugins(const char *root) { cfg_set("TOTK_MODS_ORDER", round ? "Q|P" : "P|Q"); mods_rescan(); mods_scan(); - rt_patch_probe_table(table, 3, picks, 1); + rt_patch_probe_table(table, 4, picks, 1); rt_patch_switch(0, 1); mods_exefs_apply(0, 0, 0, NULL); - const ModInfo *pm = by_name("P"), *qm = by_name("Q"), *rm = by_name("R"), *sm = by_name("S"); - CHECK(pm && qm && rm && sm, "plugin scan: four mods"); - if (!pm || !qm || !rm || !sm) break; + const ModInfo *pm = by_name("P"), *qm = by_name("Q"), *rm = by_name("R"), *sm = by_name("S"), + *tm = by_name("T"); + CHECK(pm && qm && rm && sm && tm, "plugin scan: five mods"); + if (!pm || !qm || !rm || !sm || !tm) break; CHECK(pm->problem == MOD_OK && pm->nsets == 1 && pm->nplugins == 1 && pm->plugin_hooks == 7 && !pm->code[0] && EQ(pm->overrides, "fps60"), "a built plugin: its set, its hooks, the built-in set it replaces"); CHECK(rt_patch_enabled(1) && !rt_patch_enabled(0), "the plugin's set on, the built-in set it replaces off"); @@ -261,25 +268,32 @@ static void probe_plugins(const char *root) { CHECK(rm->problem == MOD_EXEFS_LATER && rm->pending == 1, "a plugin the build lacks waits for the rebuild"); CHECK(sm->problem == MOD_PLUGIN && EQ(sm->code, "subsdk9") && strstr(sm->plugin_note, "svc 0x40"), "a refused plugin keeps its mod off and says why"); + CHECK(tm->problem == MOD_PLUGIN && tm->nplugins == 1 && tm->plugin_hooks == 2 && EQ(tm->code, "subsdk8") && + strstr(tm->plugin_note, "svc 0x1") && !rt_patch_enabled(3), + "a built plugin beside a refused one: the mod off, the refused one's reason on its row"); } // P above Q, then P's start differs from the build cfg_set("TOTK_MODS_ORDER", "P|Q"); mods_rescan(); mods_scan(); - rt_patch_probe_table(table, 3, picks, 1); + rt_patch_probe_table(table, 4, picks, 1); mods_exefs_apply(0, 0, 0, NULL); mods_plugin_off(1, "probe: 1 of the words the build recorded were not written"); const ModInfo *pm = by_name("P"); CHECK(!rt_patch_enabled(1) && rt_patch_pick_winner(0) == 1, "a plugin off at start: the other mod's patch runs"); CHECK(pm && strstr(pm->plugin_off, "were not written"), "its row says why"); CHECK(mods_conflict_count() == 0, "its conflict goes with it"); - cfg_set("TOTK_MODS_NO_CODE", "S"); + cfg_set("TOTK_MODS_NO_CODE", "S|T"); mods_rescan(); mods_scan(); + rt_patch_probe_table(table, 4, picks, 1); + mods_exefs_apply(0, 0, 0, NULL); pm = by_name("P"); - const ModInfo *sm = by_name("S"); + const ModInfo *sm = by_name("S"), *tm = by_name("T"); CHECK(pm && pm->problem == MOD_EXEFS_LATER && pm->pending == 1, "the next start: its file waits for the rebuild"); CHECK(sm && sm->problem == MOD_OK && sm->plugin_note[0], "a refused plugin used without its code: the rest runs"); + CHECK(tm && tm->problem == MOD_OK && tm->plugin_note[0] && rt_patch_enabled(3), + "a built plugin beside a refused one, used without its code: the built one's set runs"); rt_patch_probe_table(NULL, 0, NULL, 0); cfg_set("TOTK_MODS_NO_CODE", ""); cfg_set("TOTK_MODS_ORDER", ""); diff --git a/runtime/plugin_init.c b/runtime/plugin_init.c index 227f6621..ffbfcd3a 100644 --- a/runtime/plugin_init.c +++ b/runtime/plugin_init.c @@ -2,9 +2,11 @@ // this runtime's own kernel and sdk. The design follows #845 by itaskaev2, with the init run natively instead of under // Unicorn, and run again at every start instead of an image taken after one run at build time. // -// The service run: totk.exe --plugin-init DIR runs every plugin's DT_INIT, records what the init did and exits before -// any game module's constructor, nnMain, graphics or audio. The record is the input of tools/exefs_plugins.py, which -// checks it and writes the plugin's patch set (its hook sites, recomp/patches.py `plugin` lines). +// The service run: totk.exe --plugin-init DIR runs the plugins' DT_INITs, records what the inits did and exits before +// any game module's constructor, nnMain, graphics or audio. TOTK_PLUGIN_ONLY names the one plugin module whose init +// runs ("-": none); the others stay loaded, as they are at play. tools/exefs_plugins.py runs each plugin on its own, +// so a plugin that is refused, crashes or hangs costs only itself; it checks each record and writes the plugin's patch +// set (its hook sites, recomp/patches.py `plugin` lines). // // At every start (plugins_start, from the module-init callback in boot.c): when the build has a plugin whose patch set // its mod switched on, sdk's DT_INIT runs first, then the plugin's, in the same environment as in the service run, and @@ -15,8 +17,11 @@ // its GOT, its alias addresses. What the init wrote into the game modules (its hook branches) and the plugin's .text // after it (its stubs and trampolines) are compared with the record the build used (patches.h RT_PLUGIN_WRITES and // RT_PLUGINS): equal, the plugin's set stays on and the generated hook sites call into it; different, the set goes off -// for this run, the reason is logged and the mod waits for a rebuild (mods.c mods_plugin_off). The branches written -// into main's .text are taken back afterwards: the lifted code is the code that runs, and main keeps the game's bytes. +// for this run, the reason is logged and the mod waits for a rebuild (mods.c mods_plugin_off). Each plugin is compared +// right after its own init, and the branches it wrote into main's .text are taken back before the next init: the next +// plugin sees the game's own bytes, as in its own service run (two plugins hooking one site each relocate the game's +// instruction; the generated site runs the one the Mods tab order picks). The lifted code is the code that runs, and +// main keeps the game's bytes. // Without a plugin in the build none of this runs (RT_PLUGIN_COUNT 0). // // What exlaunch's init needs from the process, and how this run gives it: @@ -37,10 +42,10 @@ // Map/UnmapProcessMemory); any other one, such as CreateSession and ReplyAndReceive of exlaunch's process-handle // fallback, ends the run with the reason in the record. // -// The record, in DIR: record.json (modules, the writes into every game module as words, the alias calls, svc and -// import counts, the tick offset, a refusal), plugin.img (the plugin's image after its init, all segments) and -// plugin.pre.img (the same after loading and relocation, before the init). TOTK_PLUGIN_TICK_OFFSET adds to every -// GetSystemTick, so two runs pick different aliases. +// The record, in DIR: record.json (modules, which plugins' inits ran, the writes into every game module as words, the +// alias calls, svc and import counts, the tick offset, a refusal), plugin.img (the plugin's image after its init, +// all segments) and plugin.pre.img (the same after loading and relocation, before the init). +// TOTK_PLUGIN_TICK_OFFSET adds to every GetSystemTick, so two runs pick different aliases. #include #include #include @@ -93,6 +98,7 @@ static uint64_t svc_count[0x80], info_count[16], info_65001; static struct { const char *name; uint64_t calls; } imports[MAX_IMPORTS]; static int nimports; static uint8_t **pre; // per loaded module: its image before the plugins' inits +static uint8_t *ran; // service run, per loaded module: its init ran static uint64_t align_up(uint64_t v, uint64_t a) { return (v + a - 1) & ~(a - 1); } @@ -344,6 +350,7 @@ static void write_record(const char *refused) { i ? "," : "", m->name, (unsigned long long)m->base, (unsigned long long)m->size, (unsigned long long)m->text_end, (unsigned long long)m->data_off, (unsigned long long)m->init, m->plugin ? "true" : "false"); + if (m->plugin) fprintf(f, ", \"ran\": %s", ran && ran[i] ? "true" : "false"); if (m->plugin && !refused) { char img[64], pre_img[64], hex[65]; snprintf(img, sizeof img, "%s.img", m->name); @@ -460,9 +467,17 @@ void plugin_init_run(Cpu *c, const RtLoaded *sdk) { if (!pre[i]) rt_fatal("plugin-init: out of memory for a copy of %s", m->name); memcpy(pre[i], (const void *)(uintptr_t)m->base, m->size); } + const char *only = cfg_get("TOTK_PLUGIN_ONLY"); + ran = calloc((size_t)ldr_count(), 1); + if (!ran) rt_fatal("plugin-init: out of memory"); for (int i = ldr_count() - 1; i >= 0; i--) { const RtLoaded *m = ldr_module(i); if (!m->plugin) continue; + if (only && *only && strcmp(only, m->name)) { + rt_log("plugin-init: %s stays loaded, its init does not run (TOTK_PLUGIN_ONLY=%s)", m->name, only); + continue; + } + ran[i] = 1; if (!m->init) refuse("a plugin without DT_INIT"); rt_log("plugin-init: ENTER %s DT_INIT at %#llx (%llu init_array entries)", m->name, (unsigned long long)m->init, (unsigned long long)(m->init_array_size / 8)); @@ -546,17 +561,59 @@ static void trace_entries(void) { rt_log("plugins: trace: %d hook entries counted", ntraced); } -// Takes back the words the inits changed in the sealed part (.text, .rodata) of a game module: main keeps the game's -// own bytes, the lifted code being what runs. -static void restore_sealed(const RtLoaded *m, const uint8_t *was) { - for (uint64_t off = 0; off < m->data_off; off += PAGE) { - const uint8_t *now = (const uint8_t *)(uintptr_t)(m->base + off); - if (!memcmp(now, was + off, PAGE)) continue; - if (!plat_mem_protect(m->base + off, PAGE, PLAT_PROT_RW)) rt_fatal("plugins: cannot unseal %s+%#llx", m->name, - (unsigned long long)off); - memcpy((void *)(uintptr_t)(m->base + off), was + off, PAGE); - plat_mem_protect(m->base + off, PAGE, PLAT_PROT_READ); +// Takes back a page an init changed in the sealed part (.text, .rodata) of a game module: main keeps the game's own +// bytes, the lifted code being what runs. +static void restore_page(const RtLoaded *m, uint64_t off, const uint8_t *was) { + if (!plat_mem_protect(m->base + off, PAGE, PLAT_PROT_RW)) rt_fatal("plugins: cannot unseal %s+%#llx", m->name, + (unsigned long long)off); + memcpy((void *)(uintptr_t)(m->base + off), was, PAGE); + plat_mem_protect(m->base + off, PAGE, PLAT_PROT_READ); +} + +// After plugin i's init: what it wrote into the game modules, word by word, against the record of the build (its +// entries of RT_PLUGIN_WRITES). The first difference goes into why (unless a reason is there already). Then the game +// modules are made what they were before the init: the sealed part (.text, .rodata) gets the game's bytes back, a +// write elsewhere stays and goes into pre, so the next plugin's init is compared only with what it writes itself. +// The aliases this init mapped (from first_alias on) are flushed first; an earlier init's were flushed after it. +static void check_writes(int plugin, int first_alias, char *why, size_t cap) { + for (int i = first_alias; i < naliases; i++) + if (aliases[i].live) flush(&aliases[i]); + uint8_t seen[NWRITES > 0 ? NWRITES : 1]; + memset(seen, 0, sizeof seen); + int unexpected = 0, missing = 0; + char first[160] = ""; + for (int i = 0; i < ldr_count(); i++) { + const RtLoaded *m = ldr_module(i); + if (!pre[i]) continue; + uint8_t *now = (uint8_t *)(uintptr_t)m->base; + for (uint64_t off = 0; off < m->size; off += PAGE) { + if (!memcmp(now + off, pre[i] + off, PAGE)) continue; + for (uint64_t w = off; w < off + PAGE; w += 4) { + uint32_t a, b; + memcpy(&a, pre[i] + w, 4); + memcpy(&b, now + w, 4); + if (a == b) continue; + int k = 0; + while (k < NWRITES && !(write_tab[k].plugin == plugin && write_tab[k].off == w && + !strcmp(write_tab[k].module, m->name) && write_tab[k].old == a && + write_tab[k].now == b)) + k++; + if (k < NWRITES) { + seen[k] = 1; + } else if (!unexpected++) { + snprintf(first, sizeof first, "%s+%#llx %08x -> %08x", m->name, (unsigned long long)w, a, b); + } + } + if (off < m->data_off) restore_page(m, off, pre[i] + off); + else memcpy(pre[i] + off, now + off, PAGE); + } } + for (int k = 0; k < NWRITES; k++) missing += write_tab[k].plugin == plugin && !seen[k]; + if (why[0]) return; + if (unexpected) + snprintf(why, cap, "its init wrote %d words the build did not record (first %s)", unexpected, first); + else if (missing) + snprintf(why, cap, "%d of the words the build recorded were not written", missing); } #endif @@ -597,62 +654,22 @@ int plugins_start(Cpu *c, const RtLoaded *sdk) { const RtLoaded *m = loaded(plugin_tab[i].module); rt_log("plugins: ENTER %s DT_INIT at %#llx", m->name, (unsigned long long)m->init); failed[0] = 0; + int first_alias = naliases; running = 1; rt_trace |= RT_TRACE_PLUGIN; rt_call(c, m->init, 0); rt_trace &= ~(unsigned)RT_TRACE_PLUGIN; running = 0; if (failed[0]) snprintf(why[i], sizeof why[i], "its init asked for what the port does not serve: %s", failed); - } - for (int i = 0; i < naliases; i++) - if (aliases[i].live) flush(&aliases[i]); - // What the inits wrote into the game modules, word by word, against the record of the build. - uint8_t seen[NWRITES > 0 ? NWRITES : 1]; - memset(seen, 0, sizeof seen); - int unexpected = 0; - char first[160] = ""; - for (int i = 0; i < ldr_count(); i++) { - const RtLoaded *m = ldr_module(i); - if (!pre[i]) continue; - const uint8_t *now = (const uint8_t *)(uintptr_t)m->base; - for (uint64_t off = 0; off < m->size; off += PAGE) { - if (!memcmp(now + off, pre[i] + off, PAGE)) continue; - for (uint64_t w = off; w < off + PAGE; w += 4) { - uint32_t a, b; - memcpy(&a, pre[i] + w, 4); - memcpy(&b, now + w, 4); - if (a == b) continue; - int k = 0; - while (k < NWRITES && !(write_tab[k].off == w && !strcmp(write_tab[k].module, m->name) && - write_tab[k].old == a && write_tab[k].now == b)) - k++; - if (k < NWRITES) { - seen[k] = 1; - } else if (!unexpected++) { - snprintf(first, sizeof first, "%s+%#llx %08x -> %08x", m->name, (unsigned long long)w, a, b); - } - } - } - restore_sealed(m, pre[i]); - free(pre[i]); - } - free(pre); - pre = NULL; - for (int i = 0; i < RT_PLUGIN_COUNT; i++) { - if (!run[i] || why[i][0]) continue; - const RtLoaded *m = loaded(plugin_tab[i].module); - int missing = 0; - for (int k = 0; k < NWRITES; k++) missing += write_tab[k].plugin == i && !seen[k]; + check_writes(i, first_alias, why[i], sizeof why[i]); char hex[65]; sha256_hex((const void *)(uintptr_t)m->base, m->text_end, hex); - if (unexpected) - snprintf(why[i], sizeof why[i], "the inits wrote %d words the build did not record (first %s)", unexpected, - first); - else if (missing) - snprintf(why[i], sizeof why[i], "%d of the words the build recorded were not written", missing); - else if (strcmp(hex, plugin_tab[i].sha256)) + if (!why[i][0] && strcmp(hex, plugin_tab[i].sha256)) snprintf(why[i], sizeof why[i], "its code after the init is not the one the build lifted"); } + for (int i = 0; i < ldr_count(); i++) free(pre[i]); + free(pre); + pre = NULL; mem_query_hook = NULL; // the game's own QueryMemory sees its address space as without plugins rt_process_memory = NULL; // a callback that maps process memory stops with a named fatal (svc.c) int on = 0; diff --git a/tests/exefs_plugins_test.py b/tests/exefs_plugins_test.py index 7687d6c9..5243569f 100644 --- a/tests/exefs_plugins_test.py +++ b/tests/exefs_plugins_test.py @@ -379,6 +379,83 @@ class Lines(unittest.TestCase): self.assertFalse(any(x.startswith(('orig', 'stub')) for x in out)) +# ---- one service run per plugin +class PerPlugin(unittest.TestCase): + """main() with three plugins in three mods and a stand-in for the service run: plugin0 hooks its site, plugin1's + init ends the run without a record (a crash), plugin2's run is refused (an svc). Each gets its own status line; + only plugin0's set goes into the patch file.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.d = Path(tmp.name) + self.files = [] + for name in ('A', 'B', 'C'): + f = self.d / 'mods' / name / 'exefs' / 'subsdk9' + f.parent.mkdir(parents=True) + f.write_bytes(nso(PLUGIN_TEXT) + name.encode()) + self.files.append(f) + (self.d / 'patches.txt').write_text('set fps60 TOTK_FPS60 1\n') + self.runs = [] + saved = xp.run, xp.hook_impls + self.addCleanup(lambda: (setattr(xp, 'run', saved[0]), setattr(xp, 'hook_impls', saved[1]))) + xp.run, xp.hook_impls = self.fake_run, lambda rec: IMPLS + + def fake_run(self, exe, out, tick_offset, timeout, only): + self.runs.append(only) + out.mkdir(parents=True, exist_ok=True) + if only == 'plugin1': + return 0xC0000005, 'its init ended the service run without a record (exit 0xc0000005)' + rec = record(writes=[] if only != 'plugin0' else None) + rec['modules'][2]['file'] = str(self.files[0]) + for i in (1, 2): + rec['modules'].append(dict(rec['modules'][2], name=f'plugin{i}', base=PLUGIN + i * 0x100000, + file=str(self.files[i]))) + for m in rec['modules'][2:]: + m['ran'] = m['name'] == only + if only == 'plugin2': + rec['refused'] = 'svc 0x1 at plugin2+0x50a0' + (out / 'plugin0.img').write_bytes(rec.pop('images')['plugin0']) + rec['modules'][2]['image'] = 'plugin0.img' + (out / 'record.json').write_text(json.dumps(rec)) + return 3 if rec['refused'] else 0, None + + def test_each_plugin_alone(self): + out, status, patched = self.d / 'rec', self.d / 'plugins.status', self.d / 'patches-out.txt' + argv = sys.argv + sys.argv = ['exefs_plugins.py', '--exe', 'totk.exe', '--out', str(out), '--mods', str(self.d / 'mods'), + '--patches', str(self.d / 'patches.txt'), '--patches-out', str(patched), '--status', str(status)] + try: + code = xp.main() + finally: + sys.argv = argv + self.assertEqual(code, 1) + self.assertEqual(self.runs, ['-', 'plugin0', 'plugin0', 'plugin1', 'plugin2']) + lines = {line.split('\t')[0]: line.split('\t') for line in status.read_text().splitlines()} + key = {n: xp.fnv(f'{n}/exefs/subsdk9'.encode()) for n in 'ABC'} + self.assertEqual(lines[key['A']][2:], ['plugin', '-', 'plugin0, 1 hooks']) + self.assertEqual(lines[key['B']][2], 'unsupported') + self.assertIn('ended the service run', lines[key['B']][4]) + self.assertEqual(lines[key['C']][2], 'unsupported') + self.assertIn('svc 0x1', lines[key['C']][4]) + text = patched.read_text() + self.assertIn(f'set mod_{key["A"].lower()} ', text) + self.assertNotIn(f'mod_{key["B"].lower()}', text) + self.assertNotIn(f'mod_{key["C"].lower()}', text) + self.assertIn((out / 'plugin0' / 'run1' / 'plugin0.img').resolve().as_posix(), text) + + def test_ran_view_leaves_out_plugins_that_did_not_run(self): + rec = record() + rec['modules'].append(dict(rec['modules'][2], name='plugin1', base=PLUGIN + 0x100000, ran=False)) + rec['modules'][2]['ran'] = True + rec['images']['plugin1'] = rec['images']['plugin0'] + view = xp.ran_view(rec) + self.assertEqual([m['name'] for m in view['modules'] if m['plugin']], ['plugin0']) + self.assertEqual(set(view['images']), {'plugin0'}) + whole = record() # no "ran": every init ran, the record is its own view + self.assertIs(xp.ran_view(whole), whole) + + # ---- the generator on those lines def code_nso(words_at, size_text=0x5000): """An NSO as nso() whose .text holds words_at {offset: word}.""" diff --git a/tests/installer_test.py b/tests/installer_test.py index f890a1c8..57d09a32 100644 --- a/tests/installer_test.py +++ b/tests/installer_test.py @@ -383,10 +383,14 @@ class LiftRunTest(unittest.TestCase): self.assertEqual(self.outputs(), [False, False]) self.assertIn('warn.exefs_skipped', self.ctx.reporter.out.getvalue()) - def plugin_setup(self, kind): - """A mod with an exlaunch plugin; the fake service run writes its status line of that kind.""" + def plugin_setup(self, kind, more=()): + """A mod with an exlaunch plugin (SQ), and one more mod per (name, kind) of more; the fake service run writes + each plugin's status line of its kind.""" + kinds = {'SQ': kind, **dict(more)} + for name in kinds: + (self.dest / 'mods' / name / 'exefs').mkdir(parents=True, exist_ok=True) + (self.dest / 'mods' / name / 'exefs' / 'subsdk9').write_bytes(b'NSO0' + name.encode()) plugin = self.dest / 'mods' / 'SQ' / 'exefs' / 'subsdk9' - plugin.write_bytes(b'NSO0') self.ctx.layout = SimpleNamespace(**vars(self.ctx.layout), llvm='llvm', runtime='runtime', ninja='ninja', deps=None, sysroot=None, runtime_lib_for=lambda cpu: None, pgo=lambda version: None, expected=lambda version: 'expected.json') @@ -399,11 +403,14 @@ class LiftRunTest(unittest.TestCase): Path(args[args.index('--patches-out') + 1]).write_text('patches\n') status = Path(args[args.index('--status') + 1]) status.parent.mkdir(parents=True, exist_ok=True) - h = 0x811C9DC5 # FNV-1a 32 of "SQ/exefs/subsdk9", as tools/exefs_plugins.py keys it - for b in b'SQ/exefs/subsdk9': - h = ((h ^ b) * 0x01000193) & 0xFFFFFFFF - status.write_text(f'{h:08X}\t00000000\t{kind}\t-\tnote\n') - return (1 if kind == 'unsupported' else 0), log + lines = [] + for name, k in kinds.items(): + h = 0x811C9DC5 # FNV-1a 32 of "/exefs/subsdk9", as tools/exefs_plugins.py keys it + for b in f'{name}/exefs/subsdk9'.encode(): + h = ((h ^ b) * 0x01000193) & 0xFFFFFFFF + lines.append(f'{h:08X}\t00000000\t{k}\t-\tnote\n') + status.write_text(''.join(lines)) + return (1 if 'unsupported' in kinds.values() else 0), log return code, log steps.run_tool = run_tool return plugin @@ -429,6 +436,18 @@ class LiftRunTest(unittest.TestCase): self.assertTrue(steps.plugins_status(self.ctx).is_file()) self.assertNotIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue()) + def test_refused_plugin_stays_in_the_layout_of_a_built_one(self): + # one plugin built, one refused: the lift with the hook sites keeps both in the layout (the records hold the + # addresses of the service runs' layout); the refused one has no patch set, its status line says why + self.plugin_setup('plugin', [('ZZ', 'unsupported')]) + steps.Lift().run(self.ctx) + last = self.calls[4][1] + mods = self.dest / 'mods' + self.assertEqual(last[last.index('--plugins') + 1], + ','.join(str(mods / n / 'exefs' / 'subsdk9') for n in ('SQ', 'ZZ'))) + self.assertIn('unsupported', steps.plugins_status(self.ctx).read_text()) + self.assertNotIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue()) + def test_failed_plugin_build_warns(self): self.plugin_setup('plugin') self.fail = {'build_game.py'} diff --git a/tools/exefs_plugins.py b/tools/exefs_plugins.py index a0ef82ff..73421bdb 100644 --- a/tools/exefs_plugins.py +++ b/tools/exefs_plugins.py @@ -4,8 +4,8 @@ a plugin's hook sites. After the design in #845 by itaskaev2; the init runs on t same environment (plugin_init.c plugins_start), so the plugin keeps real state (its heap blocks, its GOT, its alias addresses). The record says what the build expects that start-up to write. - python tools/exefs_plugins.py --exe build/-x/totk.exe --out # two runs, their diff, the check - python tools/exefs_plugins.py --check /run1 # the check of one record + python tools/exefs_plugins.py --exe build/-x/totk.exe --out # per plugin: two runs, diff, check + python tools/exefs_plugins.py --check //run1 # the check of one record python tools/exefs_plugins.py --exe --out --mods --patches --patches-out --status # and the patch lines for the generator @@ -20,11 +20,16 @@ file hash: FNV-1a 32 of "/"; conten `plugin` (note: the module and its hook count) or `unsupported` (note: the first reason); built-in sets: the port's own sets with a site the plugin hooks too ("-" none), which the runtime switches off while the mod is on. -The exe must be built with the plugin in its layout (recomp/gen.py --plugins). --out gets run1/ and run2/, one record -each; run2 adds a large offset to every GetSystemTick (TOTK_PLUGIN_TICK_OFFSET), so the init picks other alias -addresses. The words that differ between the two records are listed with what they look like, not refused: an init -keeps its alias addresses and the regions GetInfo gave it in its .data, and a plugin's callbacks must not use them -(the generator's check of the code the callbacks reach, still to come). +The exe must be built with the plugins in its layout (recomp/gen.py --plugins). Each plugin runs in service runs of +its own (TOTK_PLUGIN_ONLY): the other plugins stay loaded, their inits do not run. A plugin that is refused, crashes, +hangs or fails the check is refused alone, with its reason; the others are built in. At every start the game runs each +init against the game's own bytes as well (runtime/plugin_init.c plugins_start takes a plugin's writes back before the +next init), so a plugin's own record is what its start must match. --out gets list/ (a run without inits, which names +the plugin modules), and /run1 and /run2 for each plugin; run2 adds a large offset to every +GetSystemTick (TOTK_PLUGIN_TICK_OFFSET), so the init picks other alias addresses. The words that differ between the two +records are listed with what they look like, not refused: an init keeps its alias addresses and the regions GetInfo +gave it in its .data, and a plugin's callbacks must not use them. A difference in code (a write into a game module, +the plugin's .text or .rodata) refuses the plugin: its start could not be compared with the record. The check (check_record) accepts a record only when every rule holds; each broken rule is one reason: - the run was not refused (an svc outside the served set, such as CreateSession of exlaunch's process-handle @@ -90,6 +95,19 @@ def u(v): return int(v, 16) if isinstance(v, str) else v +def ran_view(rec): + """The record as seen from the plugins whose inits ran: a plugin that only stayed loaded (TOTK_PLUGIN_ONLY) counts + as a module that is no plugin, so its exports and code are not checked as this record's. A record without "ran" + (every init ran) comes back as it is.""" + if all(m.get('ran', True) for m in rec['modules'] if m['plugin']): + return rec + view = dict(rec) + view['modules'] = [dict(m, plugin=False) if m['plugin'] and not m.get('ran', True) else m for m in rec['modules']] + names = {m['name'] for m in view['modules'] if m['plugin']} + view['images'] = {k: v for k, v in rec['images'].items() if k in names} + return view + + # ---- instructions def sext(v, bits): return v - (1 << bits) if v >> (bits - 1) & 1 else v @@ -433,35 +451,35 @@ def plugin_files(mods): return out -def emit(rec, hooks, reasons, record_dir, args): - """--patches-out and --status from the record (accepted or not).""" +def emit(results, args): + """--patches-out and --status from the checked plugins. results: one (module, its NSO file, its record or None, + the record's directory, hooks, reasons) per plugin of the layout; no reasons: accepted.""" text = args.patches.read_text() if args.patches else '' builtin = builtin_sites(text) status, sets = [], [] files = {str(f.resolve()): (m, rel, f) for m, rel, f in plugin_files(args.mods)} - plugins = [m for m in rec['modules'] if m['plugin']] - for p in plugins: - hit = files.get(str(Path(p.get('file', '')).resolve())) + for module, file, rec, record_dir, hooks, reasons in results: + hit = files.get(str(Path(file or '').resolve())) if not hit: - print(f'plugins: {p["name"]}: {p.get("file")} is not under {args.mods}') + print(f'plugins: {module}: {file} is not under {args.mods}') continue mod, rel, f = hit key, content = fnv(f'{mod}/{rel}'.encode()), fnv(f.read_bytes()) if reasons: - note = reasons[0].replace('\t', ' ') + note = ' '.join(reasons[0].split()) status.append(f'{key}\t{content}\tunsupported\t-\t{note}') print(f'plugins: unsupported {mod}/{rel} ({note})') continue name = f'mod_{key.lower()}' - own = hooks_of(hooks, p['name']) + own = hooks_of(hooks, module) sites = [int(h['site'].split('+')[1], 16) for h in own] over = sorted({b for a in sites for b in builtin.get(a, [])}) label = f'{mod}/{rel}'.encode('ascii', 'replace').decode() sets.append([f'set {name} TOTK_MODPATCH_{key} 0 ordered # {label}'] + plugin_lines(rec, own, name, record_dir, - p['name'])) - status.append(f'{key}\t{content}\tplugin\t{",".join(over) or "-"}\t{p["name"]}, {len(own)} hooks') - print(f'plugins: plugin {mod}/{rel} as {p["name"]}, {len(own)} hooks' + (f', over {", ".join(over)}' - if over else '')) + module)) + status.append(f'{key}\t{content}\tplugin\t{",".join(over) or "-"}\t{module}, {len(own)} hooks') + print(f'plugins: plugin {mod}/{rel} as {module}, {len(own)} hooks' + (f', over {", ".join(over)}' + if over else '')) if sets: text += '\n# Mod plugins (tools/exefs_plugins.py)\n' + '\n'.join(line for s in sets for line in s) + '\n' args.patches_out.write_text(text) @@ -469,7 +487,7 @@ def emit(rec, hooks, reasons, record_dir, args): args.status.write_text('\n'.join(status) + ('\n' if status else ''), encoding='utf-8') -# ---- two runs +# ---- the service runs def where_is(v, rec): """What an address-like value points into, for the diff listing.""" for m in rec['modules']: @@ -524,17 +542,34 @@ def diff_records(a, b): return out -def run(exe, out, tick_offset, timeout): +def run(exe, out, tick_offset, timeout, only): + """One service run into out, only module `only`'s init running ("-": none). (exit code or None, why the run gave + no record or None); a refused run writes its record too.""" out.mkdir(parents=True, exist_ok=True) - env = dict(os.environ, TOTK_PLUGIN_TICK_OFFSET=f'{tick_offset:#x}', TOTK_BACKGROUND_WINDOW='1', - TOTK_ASSUME_FOCUS='1') - r = subprocess.run([str(exe), '--plugin-init', str(out)], env=env, capture_output=True, text=True, - errors='replace', timeout=timeout) + for f in out.iterdir(): # an earlier run's record must not pass for this one's + if f.is_file() and (f.name in ('record.json', 'run.log') or f.suffix == '.img'): + f.unlink() + env = dict(os.environ, TOTK_PLUGIN_TICK_OFFSET=f'{tick_offset:#x}', TOTK_PLUGIN_ONLY=only, + TOTK_BACKGROUND_WINDOW='1', TOTK_ASSUME_FOCUS='1') + try: + r = subprocess.run([str(exe), '--plugin-init', str(out)], env=env, capture_output=True, text=True, + errors='replace', timeout=timeout) + except subprocess.TimeoutExpired as e: + log = b''.join(x if isinstance(x, bytes) else x.encode() for x in (e.stdout, e.stderr) if x) + (out / 'run.log').write_text(log.decode(errors='replace'), encoding='utf-8') + return None, f'its init did not return within {timeout} s' (out / 'run.log').write_text(r.stdout + r.stderr, encoding='utf-8') - return r.returncode + if not (out / 'record.json').is_file(): + # the runtime's crash report names the fault in a FATAL line (an access violation, an abort) + fatal = next((line.split('FATAL:', 1)[1].strip() for line in (r.stdout + r.stderr).splitlines() + if 'FATAL:' in line), None) + return r.returncode, (f'its init crashed the service run: {fatal}' if fatal else + f'its init ended the service run without a record (exit {r.returncode & 0xFFFFFFFF:#x})') + return r.returncode, None def report(rec, label): + rec = ran_view(rec) impls = hook_impls(rec) hooks, reasons = check_record(rec, impls) more, notes = reach_check(rec, hooks, impls) @@ -559,12 +594,40 @@ def report(rec, label): return hooks, reasons +def plugin_runs(exe, out, module, timeout): + """The two service runs of one plugin (run2 with the tick offset) into out/: (its first record or None, + hooks, reasons).""" + recs = [] + for i, offset in enumerate((0, TICK_OFFSET), 1): + d = out / module / f'run{i}' + code, why = run(exe, d, offset, timeout, module) + if why: + print(f'{module} run {i}: {why}; see {d / "run.log"}') + return None, [], [why] + recs.append(load_record(d)) + print(f'{module} run {i}: exit {code}, tick offset {offset:#x}') + if recs[0].get('refused'): # refused: a second run says nothing more + hooks, bad = report(recs[0], f'{module} run 1') + return recs[0], hooks, bad + hooks, bad = report(recs[0], f'{module} run 1') + diff = diff_records(ran_view(recs[0]), ran_view(recs[1])) + print(f'{module} run 1 against run 2: {len(diff)} differences' + (':' if diff else '')) + for line in diff: + print(f' {line}') + # Words of a game module or of the plugin's .text that differ between the runs: the start-up could not be compared + # with the record. Its .data and .bss may differ (alias addresses). + unstable = [line for line in diff if line.startswith('write ') or '(.text)' in line or '(.rodata)' in line] + if unstable: + bad = bad + [f'two runs of the init differ in code: {unstable[0]}'] + return recs[0], hooks, bad + + def main(): ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - ap.add_argument('--exe', type=Path, help='totk.exe built with the plugin in its layout') - ap.add_argument('--out', type=Path, help='directory for run1/ and run2/') + ap.add_argument('--exe', type=Path, help='totk.exe built with the plugins in its layout') + ap.add_argument('--out', type=Path, help='directory for list/ and /run1, run2') ap.add_argument('--check', type=Path, help='check the record in this directory only') - ap.add_argument('--timeout', type=int, default=300) + ap.add_argument('--timeout', type=int, default=120, help='seconds a service run may take') ap.add_argument('--mods', type=Path, help='the mods folder the plugins come from') ap.add_argument('--patches', type=Path, help='the patch file so far') ap.add_argument('--patches-out', type=Path, help='the patch file with the plugins\' sets added') @@ -578,32 +641,26 @@ def main(): rec = load_record(args.check) hooks, bad = report(rec, str(args.check)) if emitting: - emit(rec, hooks, bad, args.check, args) + emit([(m['name'], m.get('file'), rec, args.check, hooks, bad) for m in ran_view(rec)['modules'] + if m['plugin']], args) return 1 if bad else 0 if not args.exe or not args.out: ap.error('--exe and --out, or --check') - recs = [] - for i, offset in enumerate((0, TICK_OFFSET), 1): - d = args.out / f'run{i}' - code = run(args.exe, d, offset, args.timeout) - if not (d / 'record.json').is_file(): - print(f'run {i}: no record (exit {code}); see {d / "run.log"}') - return 1 - recs.append(load_record(d)) - print(f'run {i}: exit {code}, tick offset {offset:#x}') - hooks, bad = report(recs[0], 'run 1') - diff = diff_records(*recs) - print(f'run 1 against run 2: {len(diff)} differences' + (':' if diff else '')) - for line in diff: - print(f' {line}') - # Words of a game module or of a plugin's .text that differ between the runs: the start-up could not be compared - # with the record. Its .data and .bss may differ (alias addresses). - unstable = [line for line in diff if line.startswith('write ') or '(.text)' in line or '(.rodata)' in line] - if unstable: - bad = bad + [f'two runs of the init differ in code: {unstable[0]}'] + # A run without any init names the plugin modules of the exe's layout and checks that the service run works at all: + # without it no plugin can be built. + code, why = run(args.exe, args.out / 'list', 0, args.timeout, '-') + if why: + print(f'the service run without inits gave no record: {why}; see {args.out / "list" / "run.log"}') + return 2 + listing = load_record(args.out / 'list') + results = [] + for m in (m for m in listing['modules'] if m['plugin']): + rec, hooks, bad = plugin_runs(args.exe, args.out, m['name'], args.timeout) + print(f'{m["name"]}: ' + (f'refused: {bad[0]}' if bad else f'accepted, {len(hooks)} hooks')) + results.append((m['name'], m.get('file'), rec, args.out / m['name'] / 'run1', hooks, bad)) if emitting: - emit(recs[0], hooks, bad, args.out / 'run1', args) - return 1 if bad else 0 + emit(results, args) + return 1 if any(r[5] for r in results) else 0 if __name__ == '__main__': From 77f34a05f5140720e2850d4104d6b1a79ffd45cb Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 09:33:49 +0300 Subject: [PATCH 3/8] A plugin build whose plugins all stay off gives the game its own QueryMemory back (#845) plugins_start returned before clearing mem_query_hook and rt_process_memory when no plugin's set was on (the mod off, removed, or its file changed), so the game ran with the service run's view of memory: host ranges Inaccessible, the rtld stand-in below main. Co-Authored-By: Claude Opus 5.5 --- runtime/plugin_init.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/runtime/plugin_init.c b/runtime/plugin_init.c index ffbfcd3a..20204f66 100644 --- a/runtime/plugin_init.c +++ b/runtime/plugin_init.c @@ -636,7 +636,11 @@ int plugins_start(Cpu *c, const RtLoaded *sdk) { } run[i] = 1, nrun++; } - if (!nrun) return 0; + if (!nrun) { // every plugin's mod off or gone: the game sees its address space as without plugins + mem_query_hook = NULL; + rt_process_memory = NULL; + return 0; + } uint64_t t0 = plat_now_ms(); rt_log("boot: ENTER %s DT_INIT first: a mod plugin's init runs before the game's constructors", sdk->name); rt_call(c, sdk->init, 0); From 81c004094db45a76b647adf6b8d5c0b82cfe64fa Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 09:52:37 +0300 Subject: [PATCH 4/8] The installer's plugin service run finds its DLLs: they go beside the totk.exe it builds in temp (#845) On the release path the service run's totk.exe in /build stopped before main with 0xc0000135 (SDL3.dll not found): the DLLs are copied next to totk.exe in the install folder only by the compile step, after the lift. Every plugin was skipped with warn.plugins_skipped. The fake-tool tests did not see it; installer_test now checks the copy. exefs_plugins_test covers run()'s hang and crash reasons. Co-Authored-By: Claude Opus 5.5 --- installer/steps.py | 3 +++ tests/exefs_plugins_test.py | 16 ++++++++++++++++ tests/installer_test.py | 8 +++++++- 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/installer/steps.py b/installer/steps.py index 2510e376..70211bf0 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -997,6 +997,9 @@ class Lift(Step): if not code: code, log = run_tool(ctx, 'lift', layout.tools / 'build_game.py', build_args(ctx, gen)) if not code: + # the service run's totk.exe loads its DLLs from its own folder, as the installed one does + for dll in layout.dlls: + replace_file(dll, ctx.temp / 'build' / dll.name) code, log = run_tool(ctx, 'lift', layout.tools / 'exefs_plugins.py', [ '--exe', ctx.temp / 'build' / ('totk' + EXE), '--out', ctx.temp / 'plugins', '--mods', ctx.dest / 'mods', '--patches', patches, '--patches-out', ctx.temp / 'patches-plugins.txt', diff --git a/tests/exefs_plugins_test.py b/tests/exefs_plugins_test.py index 5243569f..071df40b 100644 --- a/tests/exefs_plugins_test.py +++ b/tests/exefs_plugins_test.py @@ -13,6 +13,7 @@ import struct import sys import tempfile import unittest +from types import SimpleNamespace from pathlib import Path ROOT = Path(__file__).resolve().parent.parent @@ -398,6 +399,7 @@ class PerPlugin(unittest.TestCase): (self.d / 'patches.txt').write_text('set fps60 TOTK_FPS60 1\n') self.runs = [] saved = xp.run, xp.hook_impls + self.real_run = xp.run self.addCleanup(lambda: (setattr(xp, 'run', saved[0]), setattr(xp, 'hook_impls', saved[1]))) xp.run, xp.hook_impls = self.fake_run, lambda rec: IMPLS @@ -444,6 +446,20 @@ class PerPlugin(unittest.TestCase): self.assertNotIn(f'mod_{key["C"].lower()}', text) self.assertIn((out / 'plugin0' / 'run1' / 'plugin0.img').resolve().as_posix(), text) + def test_run_reasons(self): + # the real run(): a hang becomes its own reason, a crash names the FATAL line + saved = xp.subprocess.run + self.addCleanup(setattr, xp.subprocess, 'run', saved) + + def hang(cmd, **kw): + raise xp.subprocess.TimeoutExpired(cmd, kw['timeout'], output=b'started', stderr=None) + xp.subprocess.run = hang + self.assertEqual(self.real_run(None, self.d / 'h', 0, 7, 'plugin0'), (None, 'its init did not return within 7 s')) + xp.subprocess.run = lambda cmd, **kw: SimpleNamespace(returncode=0xC0000005, stdout='x\nFATAL: access violation ' + '(0xc0000005) at host 1\n', stderr='') + code, why = self.real_run(None, self.d / 'c', 0, 7, 'plugin0') + self.assertEqual(why, 'its init crashed the service run: access violation (0xc0000005) at host 1') + def test_ran_view_leaves_out_plugins_that_did_not_run(self): rec = record() rec['modules'].append(dict(rec['modules'][2], name='plugin1', base=PLUGIN + 0x100000, ran=False)) diff --git a/tests/installer_test.py b/tests/installer_test.py index 57d09a32..4f8c8941 100644 --- a/tests/installer_test.py +++ b/tests/installer_test.py @@ -391,7 +391,11 @@ class LiftRunTest(unittest.TestCase): (self.dest / 'mods' / name / 'exefs').mkdir(parents=True, exist_ok=True) (self.dest / 'mods' / name / 'exefs' / 'subsdk9').write_bytes(b'NSO0' + name.encode()) plugin = self.dest / 'mods' / 'SQ' / 'exefs' / 'subsdk9' - self.ctx.layout = SimpleNamespace(**vars(self.ctx.layout), llvm='llvm', runtime='runtime', ninja='ninja', + dll = self.dest.parent / 'release-bin' / 'SDL3.dll' + dll.parent.mkdir(exist_ok=True) + dll.write_bytes(b'MZ') + (self.ctx.temp / 'build').mkdir(parents=True, exist_ok=True) + self.ctx.layout = SimpleNamespace(**vars(self.ctx.layout), llvm='llvm', runtime='runtime', ninja='ninja', dlls=[dll], deps=None, sysroot=None, runtime_lib_for=lambda cpu: None, pgo=lambda version: None, expected=lambda version: 'expected.json') saved = self.run_tool @@ -426,6 +430,8 @@ class LiftRunTest(unittest.TestCase): self.assertEqual(last[last.index('--plugins') + 1], str(plugin)) self.assertEqual(last[last.index('--patches') + 1], str(self.ctx.temp / 'patches-plugins.txt')) self.assertTrue(steps.plugins_status(self.ctx).is_file()) + # the service run's totk.exe finds its DLLs beside it + self.assertTrue((self.ctx.temp / 'build' / 'SDL3.dll').is_file()) def test_refused_plugin_is_not_lifted(self): # a refused plugin: its status line stays (the Mods tab says why), the game is lifted without it From c97ceb64511fe84ac6eaf78886ba36cc27e4f7bc Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 10:35:32 +0300 Subject: [PATCH 5/8] A plugin stage that failed runs again once a release changes the installer: steps.py is in the lift key with plugins (#845) After warn.plugins_skipped the lift key still matched its inputs, so the next setup run (with the fixed release) found lift and compile current and never built the plugin. installer/steps.py runs that stage, so it joins the key when the mods hold a plugin. Co-Authored-By: Claude Opus 5.5 --- installer/steps.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/installer/steps.py b/installer/steps.py index 70211bf0..44387eb4 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -929,9 +929,10 @@ class Lift(Step): # fingerprint takes a file's name and content only; the paths under mods/ go in too, because tools/cheats.py # and tools/exefs_patches.py name each patch set after its mod folder, so a renamed folder needs a lift # A mod plugin is built by running its init on a totk.exe of this build (tools/exefs_plugins.py), so the - # runtime is in the key too when there is one. + # runtime is in the key too when there is one, and this file, which runs that stage: a plugin stage that failed + # (warn.plugins_skipped, the game built without the plugins) runs again once a release changes it. files = cheat_files(ctx) + exefs_files(ctx) + plugin_files(ctx) - extra = [ctx.layout.tools / 'exefs_plugins.py', runtime_key(ctx)] if plugin_files(ctx) else [] + extra = [ctx.layout.tools / 'exefs_plugins.py', Path(__file__), runtime_key(ctx)] if plugin_files(ctx) else [] return fingerprint('lift-5', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches(game_version(ctx)), ctx.layout.tools / 'cheats.py', ctx.layout.tools / 'exefs_patches.py', *extra, *files, *(f.relative_to(ctx.dest / 'mods').as_posix() for f in files)) From c477b67a0993421eabdb87819df498bcc5320107 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 10:35:49 +0300 Subject: [PATCH 6/8] The runtime library's module tables hold 32 modules, not the release's MOD_COUNT: an install's plugins fit (#845) loader.c and memory.c sized their tables by MOD_COUNT of the layout.h make_release.py built libtotkrt.a against (3: main, subsdk0, sdk). An install whose rebuild adds a plugin has 4, and its service run stopped at once with "loader: 4 modules in the layout"; dev builds compile those files against their own layout and never saw it. RT_MAX_MODULES (host.h) is the bound now; the loader still refuses a layout above it. Co-Authored-By: Claude Opus 5.5 --- runtime/host.h | 3 +++ runtime/loader.c | 8 ++++---- runtime/memory.c | 4 ++-- tools/make_release.py | 3 ++- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/runtime/host.h b/runtime/host.h index d4f8c060..8bb2804a 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -207,6 +207,9 @@ typedef struct { void ldr_load(const char *exefs_dir); // load, relocate (overrides applied), seal; once extern GuestFn ldr_plugin_import; // set before ldr_load (plugin_init.c): every JUMP_SLOT // import of a plugin binds to this observer +// The module tables of the runtime library: it is built once per release (tools/build_game.py --make-runtime-lib), +// before an install's layout (recomp/layout.py) adds its mod plugins after sdk, so they do not use that MOD_COUNT. +#define RT_MAX_MODULES 32 int ldr_count(void); const RtLoaded *ldr_module(int i); // order: main, subsdk0, sdk uint64_t ldr_symbol(const char *name); // exported definition, 0 if none diff --git a/runtime/loader.c b/runtime/loader.c index 6c698ffb..b1911fe7 100644 --- a/runtime/loader.c +++ b/runtime/loader.c @@ -47,11 +47,11 @@ typedef struct { uint64_t *plt_slot, *plt_stub, nplt; // PLT stubs sorted by GOT slot (ldr_plt_stub), built on first use // statistics uint64_t kinds[K_COUNT], sym_import, sym_own; - uint32_t imports, import_to[MOD_COUNT], weak_unresolved, strong_unresolved; + uint32_t imports, import_to[RT_MAX_MODULES], weak_unresolved, strong_unresolved; uint32_t init_off; } Mod; -static Mod mods[MOD_COUNT]; +static Mod mods[RT_MAX_MODULES]; GuestFn ldr_plugin_import; static int nmods; static uint64_t identity_refs, stubs; @@ -503,7 +503,7 @@ static void seal(Mod *m) { void ldr_load(const char *exefs_dir) { if (nmods) rt_fatal("ldr_load called twice"); if (!mem_layout()->pool) rt_fatal("ldr_load before mem_init"); - if (rt_layout_count > MOD_COUNT) rt_fatal("loader: %d modules in the layout", rt_layout_count); + if (rt_layout_count > RT_MAX_MODULES) rt_fatal("loader: %d modules in the layout", rt_layout_count); int n = 0; for (int i = 0; i < rt_layout_count; i++) { // A mod plugin's file that is gone or changed since the build is left out at play time: its patch set cannot @@ -622,7 +622,7 @@ uint64_t ldr_plt_stub(const char *module, const char *name) { typedef struct { const char *name; uint64_t kinds[K_COUNT], total, sym_import, sym_own; - uint32_t imports, to[MOD_COUNT], weak, init_off, init_array; + uint32_t imports, to[RT_MAX_MODULES], weak, init_off, init_array; } Expect; static const Expect expect_121[] = { {"main", {508816, 1415, 32, 641}, 510904, 2088, 0, 675, {0, 16, 647}, 12, 0x30, 10673}, diff --git a/runtime/memory.c b/runtime/memory.c index c452c398..5df53916 100644 --- a/runtime/memory.c +++ b/runtime/memory.c @@ -57,7 +57,7 @@ static uint8_t *alias_pages; // 1 per mapped 4 KiB page of the alias regi typedef struct { uint64_t dst, src, size; } StackMap; static StackMap stack_maps[MAX_STACK_MAPS]; static unsigned nstack_maps; -static uint8_t *module_perm[MOD_COUNT]; // per data page of each module: perm | ATTR_PERMISSION_LOCKED +static uint8_t *module_perm[RT_MAX_MODULES]; // per data page of each module: perm | ATTR_PERMISSION_LOCKED static uint64_t align_up(uint64_t v, uint64_t a) { return (v + a - 1) & ~(a - 1); } @@ -530,7 +530,7 @@ uint32_t mem_unmap_physical(uint64_t a, uint64_t n) { return alias_map(1, a, n); // as taken (Code, RX), so the SDK never picks it. static void taken_gap(uint64_t a, uint64_t *base, uint64_t *size) { uint64_t lo = 0, hi = RT_ADDRESS_LIMIT; - RtRange known[3 + MOD_COUNT + 2]; + RtRange known[3 + RT_MAX_MODULES + 2]; int n = 0; known[n++] = (RtRange){L.aslr.base, low_size}; known[n++] = L.alias; diff --git a/tools/make_release.py b/tools/make_release.py index 2122ccd2..5cc9ede6 100644 --- a/tools/make_release.py +++ b/tools/make_release.py @@ -25,7 +25,8 @@ runtime/ and bin/ (PATH holds the Windows system directory and those), then runs Both need one game version extracted and lifted first (tools/extract.py, recomp/gen.py): generated/v121 or, without it, generated/v143, and for --prove its build/extracted/ (they are read, nothing is written to them). The -runtime library reads only what both versions' layout.h share (MOD_COUNT, MOD_MAIN_BASE), so either serves. +runtime library reads only what both versions' layout.h share (MOD_MAIN_BASE; its module tables hold RT_MAX_MODULES, so an +install's layout may add mod plugins), so either serves. A release also needs build/xlate built from the current sources (tools/build_xlate.py); the script stops first if not. From aa184f9fd27f5246ce38211f0af15809533a9cf3 Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 10:35:49 +0300 Subject: [PATCH 7/8] A mod plugin the build has whose file is gone starts the rebuild, which goes back to a build without it (#845) Removing a plugin mod left its module in the layout: the loader skipped the missing file and the game ran, but the rebuild that drops the plugin never came. The loader counts layout plugins whose file no longer exists (ldr_plugins_gone) and rebuild_if_needed treats them like files waiting for the rebuild. A changed file was already pending through its mod's status line. On the 1.4.3 install: The Second Quest removed, the start asked for the rebuild, the setup lifted and compiled the no-plugin build in 30.5 s, the game reached gameplay. Co-Authored-By: Claude Opus 5.5 --- runtime/cheats.c | 13 +++++++------ runtime/host.h | 1 + runtime/loader.c | 14 ++++++++++++++ 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/runtime/cheats.c b/runtime/cheats.c index 11991dc3..b96962c8 100644 --- a/runtime/cheats.c +++ b/runtime/cheats.c @@ -517,8 +517,9 @@ static void plain_name(char *s) { // ---- rebuild: code cheats and ExeFS patches the build lacks // The setup writes build-cache\rebuild.txt (installer/steps.py write_rebuild_command), the command that starts its // window in rebuild mode on this install folder, one argument per line. When an enabled mod has code cheats this build -// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), the game runs it and exits; the setup lifts -// and compiles them in and starts the game again. +// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), or the build has a mod plugin whose file is +// gone (ldr_plugins_gone: its mod was removed), the game runs it and exits; the setup lifts and compiles them in (or the +// plugin out) and starts the game again. // build-cache\rebuild.skip holds the key of the cheats a rebuild was started for: a rebuild that failed or was // cancelled leaves them unbuilt, and the same set does not start it again. TOTK_MODS_REBUILD=off never starts it. // Scripted runs (--seconds, --boot) never start it. @@ -532,7 +533,7 @@ static void rebuild_if_needed(const char *dir) { key = (key ^ fnv(c->info.key, strlen(c->info.key))) * 0x01000193u; key = (key ^ c->content) * 0x01000193u; } - int patches = mods_exefs_pending(&key); + int patches = mods_exefs_pending(&key) + ldr_plugins_gone(&key); if (!need && !patches) return; char cmd_path[PLAT_PATH_MAX + 64], skip_path[PLAT_PATH_MAX + 64], skip[16] = "", want[16]; snprintf(cmd_path, sizeof cmd_path, "%sbuild-cache" SEP "rebuild.txt", dir); @@ -548,7 +549,7 @@ static void rebuild_if_needed(const char *dir) { char *cmd = why ? NULL : read_text(cmd_path); if (!why && !cmd) why = "no build-cache/rebuild.txt (a build without the setup)"; if (why) { - rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; no rebuild: %s", need, + rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; no rebuild: %s", need, patches, why); free(cmd); return; @@ -565,12 +566,12 @@ static void rebuild_if_needed(const char *dir) { } argv[argc] = NULL; if (argc < 2 || !plat_file_write(skip_path, want, strlen(want))) { - rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; cannot start the " + rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; cannot start the " "rebuild (%s)", need, patches, cmd_path); free(cmd); return; } - rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; starting the rebuild: %s", + rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; starting the rebuild: %s", need, patches, argv[0]); int ok = plat_spawn(argv, dir); free(cmd); diff --git a/runtime/host.h b/runtime/host.h index 8bb2804a..401ba3ec 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -211,6 +211,7 @@ extern GuestFn ldr_plugin_import; // set before ldr_loa // before an install's layout (recomp/layout.py) adds its mod plugins after sdk, so they do not use that MOD_COUNT. #define RT_MAX_MODULES 32 int ldr_count(void); +int ldr_plugins_gone(uint32_t *key); // layout plugins whose file is gone; mixes their paths into key const RtLoaded *ldr_module(int i); // order: main, subsdk0, sdk uint64_t ldr_symbol(const char *name); // exported definition, 0 if none uint64_t ldr_plt_stub(const char *module, const char *name); // PLT stub whose GOT slot holds name, 0 if none diff --git a/runtime/loader.c b/runtime/loader.c index b1911fe7..635d4313 100644 --- a/runtime/loader.c +++ b/runtime/loader.c @@ -86,12 +86,26 @@ static uint8_t *read_path_verified(const char *path, const char *want, size_t *l return p; } +// Mod plugins of the layout whose file is gone (their mod was removed): a rebuild without them is due (cheats.c +// rebuild_if_needed), keyed by their paths. +static int plugins_gone; +static uint32_t plugins_gone_key = 0x811C9DC5u; + +int ldr_plugins_gone(uint32_t *key) { + if (plugins_gone) *key = (*key ^ plugins_gone_key) * 0x01000193u; + return plugins_gone; +} + // 1 when the plugin file of lay is the one the build lifted; logged when not. static int plugin_file_ok(const RtLayoutEntry *lay) { size_t len; uint8_t *p = read_file(lay->file, &len); if (!p) { rt_log("loader: %s: cannot read %s; the plugin is left out", lay->name, lay->file); + if (!plat_file_exists(lay->file)) { + plugins_gone++; + for (const char *c = lay->file; *c; c++) plugins_gone_key = (plugins_gone_key ^ (uint8_t)*c) * 0x01000193u; + } return 0; } uint8_t d[32]; From 791e077c549d75d5d691d90a95f5fe5e05e0d86d Mon Sep 17 00:00:00 2001 From: Valiantsin Dzerakh Date: Tue, 17 Nov 2026 10:36:10 +0300 Subject: [PATCH 8/8] docs/mods.md: removing a plugin mod starts the rebuild without it (#845) Co-Authored-By: Claude Opus 5.5 --- docs/mods.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/mods.md b/docs/mods.md index 9e0f1d76..c29503ec 100644 --- a/docs/mods.md +++ b/docs/mods.md @@ -114,6 +114,8 @@ ahead of time, after the design in [#845](https://github.com/valentderah/totk-pc mod's row says why, and the next start runs the rebuild. 3. The plugin keeps what its start-up made (its memory, its tables), so its code works as on the Switch. The start costs about 50 ms per plugin; the game runs no differently without a plugin. +4. Removing a plugin mod from the mods folder starts the rebuild at the next start, which builds the game without + it (half a minute on 12 cores with the lift cache). A plugin mod switched off on the Mods tab only stays off. The Mods tab order decides between a plugin and another mod's patch on the same instruction, as for code patches. A plugin site over one of the port's own patches turns that built-in patch off while the mod is on.