Merge task/plugins2: each mod plugin is built or refused on its own, and the release builds plugins in on a player's install (#845)

Per-plugin service runs (a refused, crashed or hung plugin costs only itself) and four fixes the real setup path
needed: plugin_init.c compiled per install, the service run's DLLs beside its exe, module tables sized for plugins in
the runtime library, and a failed plugin stage rerun after a fixed release. Checked with a release install: The
Second Quest built in and played with 7 hooks on, and removed again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-11-17 10:37:03 +03:00
commit d33d8d2e2f
14 files changed
+413 -167

No files matched your search

+14 -8
View File
@@ -103,14 +103,19 @@ Inventory Expansion's sort fixer, Quick Weapon Durability). On the Switch the pl
redirects some of the game's instructions into its own code. The port does the same with the plugin's code compiled
ahead of time, after the design in [#845](https://github.com/valentderah/totk-pc-recomp/issues/845) by itaskaev2:
1. The rebuild (the same one as for code patches above) compiles the plugin with the game, starts the game's code
once in a service mode that runs only the plugin's start-up (`totk.exe --plugin-init`), twice, and records which
instructions it redirects. Each redirect becomes a code site of the mod, built into `totk.exe`.
2. At every start the plugin's start-up runs again, before the game's own start-up code, in the same setting. Its
redirects are compared with the record of the rebuild. When they match, the mod's code sites go on. When they do
not, the plugin stays off for this run, the mod's row says why, and the next start runs the rebuild.
1. The rebuild (the same one as for code patches above) compiles the plugins with the game, then starts the game's
code in a service mode that runs only one plugin's start-up (`totk.exe --plugin-init`), twice per plugin, and
records which instructions it redirects. Each redirect becomes a code site of the mod, built into `totk.exe`. Each
plugin is judged on its own: one that is refused, or whose start-up crashes or never returns, stays off with its
reason, and the other plugins are built in.
2. At every start each plugin's start-up runs again, before the game's own start-up code, in the same setting: it
sees the game's own instructions, as in its service run. Its redirects are compared with the record of the
rebuild. When they match, the mod's code sites go on. When they do not, the plugin stays off for this run, the
mod's row says why, and the next start runs the rebuild.
3. The plugin keeps what its start-up made (its memory, its tables), so its code works as on the Switch. The start
costs about 50 ms per plugin; the game runs no differently without a plugin.
4. Removing a plugin mod from the mods folder starts the rebuild at the next start, which builds the game without
it (half a minute on 12 cores with the lift cache). A plugin mod switched off on the Mods tab only stays off.
The Mods tab order decides between a plugin and another mod's patch on the same instruction, as for code patches. A
plugin site over one of the port's own patches turns that built-in patch off while the mod is on.
@@ -119,8 +124,9 @@ The rebuild refuses a plugin when its start-up or the code its redirects reach n
system calls other than the few its start-up needs (memory queries and maps, the clock, debug output), library calls
outside a reviewed list (memory and string functions, the heap, text formatting), threads, services or input; a
redirect that is not one of exlaunch's hook shapes (inline hook, replace hook, call-site rewrite); or two runs of its
start-up that differ in code. The row then says **Its plugin ... cannot run in the port** with the reason, and the
mod stays off; **Use <mod> without its code** uses the rest of it. UltraCam stays refused: it hooks the graphics API,
start-up that differ in code; or a start-up that crashes the service run or does not return within two minutes. The
row then says **Its plugin ... cannot run in the port** with the reason, and the mod stays off; **Use <mod> without its
code** uses the rest of it, including its other plugins that were built in. UltraCam stays refused: it hooks the graphics API,
reads input and opens sockets.
`plugin_trace = 1` in `totk.ini` logs the first call into each of a plugin's hooks (`plugins: hook entry ... reached`).
+10 -3
View File
@@ -929,9 +929,10 @@ class Lift(Step):
# fingerprint takes a file's name and content only; the paths under mods/ go in too, because tools/cheats.py
# and tools/exefs_patches.py name each patch set after its mod folder, so a renamed folder needs a lift
# A mod plugin is built by running its init on a totk.exe of this build (tools/exefs_plugins.py), so the
# runtime is in the key too when there is one.
# runtime is in the key too when there is one, and this file, which runs that stage: a plugin stage that failed
# (warn.plugins_skipped, the game built without the plugins) runs again once a release changes it.
files = cheat_files(ctx) + exefs_files(ctx) + plugin_files(ctx)
extra = [ctx.layout.tools / 'exefs_plugins.py', runtime_key(ctx)] if plugin_files(ctx) else []
extra = [ctx.layout.tools / 'exefs_plugins.py', Path(__file__), runtime_key(ctx)] if plugin_files(ctx) else []
return fingerprint('lift-5', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches(game_version(ctx)),
ctx.layout.tools / 'cheats.py', ctx.layout.tools / 'exefs_patches.py', *extra, *files,
*(f.relative_to(ctx.dest / 'mods').as_posix() for f in files))
@@ -997,6 +998,9 @@ class Lift(Step):
if not code:
code, log = run_tool(ctx, 'lift', layout.tools / 'build_game.py', build_args(ctx, gen))
if not code:
# the service run's totk.exe loads its DLLs from its own folder, as the installed one does
for dll in layout.dlls:
replace_file(dll, ctx.temp / 'build' / dll.name)
code, log = run_tool(ctx, 'lift', layout.tools / 'exefs_plugins.py', [
'--exe', ctx.temp / 'build' / ('totk' + EXE), '--out', ctx.temp / 'plugins', '--mods',
ctx.dest / 'mods', '--patches', patches, '--patches-out', ctx.temp / 'patches-plugins.txt',
@@ -1009,7 +1013,10 @@ class Lift(Step):
skipped['warn.plugins_skipped'] = log
else:
patches = ctx.temp / 'patches-plugins.txt'
lift_plugins = built_plugins(plugins_status(ctx), plugins, ctx.dest / 'mods')
# With one plugin built, the layout keeps every plugin of the service runs: the records hold addresses
# of that layout. A refused plugin stays loaded at play, its init never runs (no patch set).
if built_plugins(plugins_status(ctx), plugins, ctx.dest / 'mods'):
lift_plugins = plugins
if lift_plugins:
args += ['--plugins', ','.join(str(f) for f in lift_plugins)]
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py', args + ['--patches', patches], parse)
+7 -6
View File
@@ -517,8 +517,9 @@ static void plain_name(char *s) {
// ---- rebuild: code cheats and ExeFS patches the build lacks
// The setup writes build-cache\rebuild.txt (installer/steps.py write_rebuild_command), the command that starts its
// window in rebuild mode on this install folder, one argument per line. When an enabled mod has code cheats this build
// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), the game runs it and exits; the setup lifts
// and compiles them in and starts the game again.
// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), or the build has a mod plugin whose file is
// gone (ldr_plugins_gone: its mod was removed), the game runs it and exits; the setup lifts and compiles them in (or the
// plugin out) and starts the game again.
// build-cache\rebuild.skip holds the key of the cheats a rebuild was started for: a rebuild that failed or was
// cancelled leaves them unbuilt, and the same set does not start it again. TOTK_MODS_REBUILD=off never starts it.
// Scripted runs (--seconds, --boot) never start it.
@@ -532,7 +533,7 @@ static void rebuild_if_needed(const char *dir) {
key = (key ^ fnv(c->info.key, strlen(c->info.key))) * 0x01000193u;
key = (key ^ c->content) * 0x01000193u;
}
int patches = mods_exefs_pending(&key);
int patches = mods_exefs_pending(&key) + ldr_plugins_gone(&key);
if (!need && !patches) return;
char cmd_path[PLAT_PATH_MAX + 64], skip_path[PLAT_PATH_MAX + 64], skip[16] = "", want[16];
snprintf(cmd_path, sizeof cmd_path, "%sbuild-cache" SEP "rebuild.txt", dir);
@@ -548,7 +549,7 @@ static void rebuild_if_needed(const char *dir) {
char *cmd = why ? NULL : read_text(cmd_path);
if (!why && !cmd) why = "no build-cache/rebuild.txt (a build without the setup)";
if (why) {
rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; no rebuild: %s", need,
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; no rebuild: %s", need,
patches, why);
free(cmd);
return;
@@ -565,12 +566,12 @@ static void rebuild_if_needed(const char *dir) {
}
argv[argc] = NULL;
if (argc < 2 || !plat_file_write(skip_path, want, strlen(want))) {
rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; cannot start the "
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; cannot start the "
"rebuild (%s)", need, patches, cmd_path);
free(cmd);
return;
}
rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; starting the rebuild: %s",
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; starting the rebuild: %s",
need, patches, argv[0]);
int ok = plat_spawn(argv, dir);
free(cmd);
+4
View File
@@ -207,7 +207,11 @@ typedef struct {
void ldr_load(const char *exefs_dir); // load, relocate (overrides applied), seal; once
extern GuestFn ldr_plugin_import; // set before ldr_load (plugin_init.c): every JUMP_SLOT
// import of a plugin binds to this observer
// The module tables of the runtime library: it is built once per release (tools/build_game.py --make-runtime-lib),
// before an install's layout (recomp/layout.py) adds its mod plugins after sdk, so they do not use that MOD_COUNT.
#define RT_MAX_MODULES 32
int ldr_count(void);
int ldr_plugins_gone(uint32_t *key); // layout plugins whose file is gone; mixes their paths into key
const RtLoaded *ldr_module(int i); // order: main, subsdk0, sdk
uint64_t ldr_symbol(const char *name); // exported definition, 0 if none
uint64_t ldr_plt_stub(const char *module, const char *name); // PLT stub whose GOT slot holds name, 0 if none
+18 -4
View File
@@ -47,11 +47,11 @@ typedef struct {
uint64_t *plt_slot, *plt_stub, nplt; // PLT stubs sorted by GOT slot (ldr_plt_stub), built on first use
// statistics
uint64_t kinds[K_COUNT], sym_import, sym_own;
uint32_t imports, import_to[MOD_COUNT], weak_unresolved, strong_unresolved;
uint32_t imports, import_to[RT_MAX_MODULES], weak_unresolved, strong_unresolved;
uint32_t init_off;
} Mod;
static Mod mods[MOD_COUNT];
static Mod mods[RT_MAX_MODULES];
GuestFn ldr_plugin_import;
static int nmods;
static uint64_t identity_refs, stubs;
@@ -86,12 +86,26 @@ static uint8_t *read_path_verified(const char *path, const char *want, size_t *l
return p;
}
// Mod plugins of the layout whose file is gone (their mod was removed): a rebuild without them is due (cheats.c
// rebuild_if_needed), keyed by their paths.
static int plugins_gone;
static uint32_t plugins_gone_key = 0x811C9DC5u;
int ldr_plugins_gone(uint32_t *key) {
if (plugins_gone) *key = (*key ^ plugins_gone_key) * 0x01000193u;
return plugins_gone;
}
// 1 when the plugin file of lay is the one the build lifted; logged when not.
static int plugin_file_ok(const RtLayoutEntry *lay) {
size_t len;
uint8_t *p = read_file(lay->file, &len);
if (!p) {
rt_log("loader: %s: cannot read %s; the plugin is left out", lay->name, lay->file);
if (!plat_file_exists(lay->file)) {
plugins_gone++;
for (const char *c = lay->file; *c; c++) plugins_gone_key = (plugins_gone_key ^ (uint8_t)*c) * 0x01000193u;
}
return 0;
}
uint8_t d[32];
@@ -503,7 +517,7 @@ static void seal(Mod *m) {
void ldr_load(const char *exefs_dir) {
if (nmods) rt_fatal("ldr_load called twice");
if (!mem_layout()->pool) rt_fatal("ldr_load before mem_init");
if (rt_layout_count > MOD_COUNT) rt_fatal("loader: %d modules in the layout", rt_layout_count);
if (rt_layout_count > RT_MAX_MODULES) rt_fatal("loader: %d modules in the layout", rt_layout_count);
int n = 0;
for (int i = 0; i < rt_layout_count; i++) {
// A mod plugin's file that is gone or changed since the build is left out at play time: its patch set cannot
@@ -622,7 +636,7 @@ uint64_t ldr_plt_stub(const char *module, const char *name) {
typedef struct {
const char *name;
uint64_t kinds[K_COUNT], total, sym_import, sym_own;
uint32_t imports, to[MOD_COUNT], weak, init_off, init_array;
uint32_t imports, to[RT_MAX_MODULES], weak, init_off, init_array;
} Expect;
static const Expect expect_121[] = {
{"main", {508816, 1415, 32, 641}, 510904, 2088, 0, 675, {0, 16, 647}, 12, 0x30, 10673},
+2 -2
View File
@@ -57,7 +57,7 @@ static uint8_t *alias_pages; // 1 per mapped 4 KiB page of the alias regi
typedef struct { uint64_t dst, src, size; } StackMap;
static StackMap stack_maps[MAX_STACK_MAPS];
static unsigned nstack_maps;
static uint8_t *module_perm[MOD_COUNT]; // per data page of each module: perm | ATTR_PERMISSION_LOCKED
static uint8_t *module_perm[RT_MAX_MODULES]; // per data page of each module: perm | ATTR_PERMISSION_LOCKED
static uint64_t align_up(uint64_t v, uint64_t a) { return (v + a - 1) & ~(a - 1); }
@@ -530,7 +530,7 @@ uint32_t mem_unmap_physical(uint64_t a, uint64_t n) { return alias_map(1, a, n);
// as taken (Code, RX), so the SDK never picks it.
static void taken_gap(uint64_t a, uint64_t *base, uint64_t *size) {
uint64_t lo = 0, hi = RT_ADDRESS_LIMIT;
RtRange known[3 + MOD_COUNT + 2];
RtRange known[3 + RT_MAX_MODULES + 2];
int n = 0;
known[n++] = (RtRange){L.aslr.base, low_size};
known[n++] = L.alias;
+3 -2
View File
@@ -446,11 +446,12 @@ static int scan_cb(const char *name, int kind, uint64_t size, void *ctx) {
int unsupported = exefs_check(m, path);
// A mod whose code part the port cannot use is off as a whole: its RomFS files may expect that code. Used without
// its code (its switch, or mods_without_code = 1 for every mod), the code modules (m->code) and the ExeFS patch
// files the port cannot apply are left out and the rest of the mod is used, when it has a rest.
// files the port cannot apply are left out and the rest of the mod is used, when it has a rest. Each plugin of a
// mod is built or refused on its own (tools/exefs_plugins.py), so a built plugin is such a rest too.
m->no_code = listed(cfg_get("TOTK_MODS_NO_CODE"), m->name);
m->without_code = m->no_code || cfg_bool("TOTK_MODS_WITHOUT_CODE");
int usable = m->nsets || m->ndata || m->kind & (MOD_ROMFS | MOD_CHEATS);
int code_off = m->code[0] && (!m->without_code || !(m->kind & (MOD_ROMFS | MOD_EXEFS | MOD_CHEATS)));
int code_off = m->code[0] && (!m->without_code || !(m->kind & (MOD_ROMFS | MOD_EXEFS | MOD_CHEATS) || m->nplugins));
m->problem = strchr(m->name, '|') ? MOD_BAD_NAME
: code_off ? MOD_PLUGIN
: unsupported && (!m->without_code || !usable) ? MOD_EXEFS_UNSUPPORTED
+27 -13
View File
@@ -212,6 +212,8 @@ static const char *refused_plugins(void) {
// Q an ExeFS patch file on P's site
// R a plugin the build has no line for waits for the rebuild
// S RomFS + a refused plugin off, the reason kept; used without its code, the rest runs
// T a built plugin + a refused one each judged alone: off for the refused one, its reason kept; used
// without its code, the built one's set runs
// Then P's start differs from the build (mods_plugin_off): its set goes off, Q's code runs at the site, the conflict
// goes, a stale line marks P's file, and the next scan has P wait for the rebuild.
static void probe_plugins(const char *root) {
@@ -221,18 +223,22 @@ static void probe_plugins(const char *root) {
char q[160];
snprintf(q, sizeof q, "Q/exefs/%s.ips", bid);
static const char *const files[] = {"P/romfs/p.bin", "P/exefs/subsdk9", "R/exefs/subsdk9", "S/romfs/s.bin",
"S/exefs/subsdk9"};
"S/exefs/subsdk9", "T/exefs/subsdk8", "T/exefs/subsdk9"};
for (size_t i = 0; i < sizeof files / sizeof *files; i++) put(root, files[i]);
put(root, q);
static char names[3][16];
static const char *const table[] = {"fps60", names[1], names[2]};
static char names[4][16];
static const char *const table[] = {"fps60", names[1], names[2], names[3]};
set_name(names[1], sizeof names[1], "P/exefs/subsdk9");
set_name(names[2], sizeof names[2], q);
char status[256], plugins[256];
set_name(names[3], sizeof names[3], "T/exefs/subsdk9");
char status[256], plugins[512];
snprintf(status, sizeof status, "%08X\t%08X\tcode\t-\t\n", fnv(q, strlen(q)), fnv("probe", 5));
snprintf(plugins, sizeof plugins, "%08X\t%08X\tplugin\tfps60\tplugin0, 7 hooks\n"
"%08X\t%08X\tunsupported\t-\tthe code its hooks reach has svc 0x40\n", fnv("P/exefs/subsdk9", 15),
fnv("probe", 5), fnv("S/exefs/subsdk9", 15), fnv("probe", 5));
"%08X\t%08X\tunsupported\t-\tthe code its hooks reach has svc 0x40\n"
"%08X\t%08X\tplugin\t-\tplugin1, 2 hooks\n"
"%08X\t%08X\tunsupported\t-\tthe run was refused: svc 0x1 at plugin2+0x50a0\n", fnv("P/exefs/subsdk9", 15),
fnv("probe", 5), fnv("S/exefs/subsdk9", 15), fnv("probe", 5), fnv("T/exefs/subsdk9", 15), fnv("probe", 5),
fnv("T/exefs/subsdk8", 15), fnv("probe", 5));
static const int16_t s0[] = {1, 2};
static const RtPatchPick picks[] = {{"main", 0x100, s0, 2}};
mods_probe_cache(status, "");
@@ -244,12 +250,13 @@ static void probe_plugins(const char *root) {
cfg_set("TOTK_MODS_ORDER", round ? "Q|P" : "P|Q");
mods_rescan();
mods_scan();
rt_patch_probe_table(table, 3, picks, 1);
rt_patch_probe_table(table, 4, picks, 1);
rt_patch_switch(0, 1);
mods_exefs_apply(0, 0, 0, NULL);
const ModInfo *pm = by_name("P"), *qm = by_name("Q"), *rm = by_name("R"), *sm = by_name("S");
CHECK(pm && qm && rm && sm, "plugin scan: four mods");
if (!pm || !qm || !rm || !sm) break;
const ModInfo *pm = by_name("P"), *qm = by_name("Q"), *rm = by_name("R"), *sm = by_name("S"),
*tm = by_name("T");
CHECK(pm && qm && rm && sm && tm, "plugin scan: five mods");
if (!pm || !qm || !rm || !sm || !tm) break;
CHECK(pm->problem == MOD_OK && pm->nsets == 1 && pm->nplugins == 1 && pm->plugin_hooks == 7 && !pm->code[0] &&
EQ(pm->overrides, "fps60"), "a built plugin: its set, its hooks, the built-in set it replaces");
CHECK(rt_patch_enabled(1) && !rt_patch_enabled(0), "the plugin's set on, the built-in set it replaces off");
@@ -261,25 +268,32 @@ static void probe_plugins(const char *root) {
CHECK(rm->problem == MOD_EXEFS_LATER && rm->pending == 1, "a plugin the build lacks waits for the rebuild");
CHECK(sm->problem == MOD_PLUGIN && EQ(sm->code, "subsdk9") && strstr(sm->plugin_note, "svc 0x40"),
"a refused plugin keeps its mod off and says why");
CHECK(tm->problem == MOD_PLUGIN && tm->nplugins == 1 && tm->plugin_hooks == 2 && EQ(tm->code, "subsdk8") &&
strstr(tm->plugin_note, "svc 0x1") && !rt_patch_enabled(3),
"a built plugin beside a refused one: the mod off, the refused one's reason on its row");
}
// P above Q, then P's start differs from the build
cfg_set("TOTK_MODS_ORDER", "P|Q");
mods_rescan();
mods_scan();
rt_patch_probe_table(table, 3, picks, 1);
rt_patch_probe_table(table, 4, picks, 1);
mods_exefs_apply(0, 0, 0, NULL);
mods_plugin_off(1, "probe: 1 of the words the build recorded were not written");
const ModInfo *pm = by_name("P");
CHECK(!rt_patch_enabled(1) && rt_patch_pick_winner(0) == 1, "a plugin off at start: the other mod's patch runs");
CHECK(pm && strstr(pm->plugin_off, "were not written"), "its row says why");
CHECK(mods_conflict_count() == 0, "its conflict goes with it");
cfg_set("TOTK_MODS_NO_CODE", "S");
cfg_set("TOTK_MODS_NO_CODE", "S|T");
mods_rescan();
mods_scan();
rt_patch_probe_table(table, 4, picks, 1);
mods_exefs_apply(0, 0, 0, NULL);
pm = by_name("P");
const ModInfo *sm = by_name("S");
const ModInfo *sm = by_name("S"), *tm = by_name("T");
CHECK(pm && pm->problem == MOD_EXEFS_LATER && pm->pending == 1, "the next start: its file waits for the rebuild");
CHECK(sm && sm->problem == MOD_OK && sm->plugin_note[0], "a refused plugin used without its code: the rest runs");
CHECK(tm && tm->problem == MOD_OK && tm->plugin_note[0] && rt_patch_enabled(3),
"a built plugin beside a refused one, used without its code: the built one's set runs");
rt_patch_probe_table(NULL, 0, NULL, 0);
cfg_set("TOTK_MODS_NO_CODE", "");
cfg_set("TOTK_MODS_ORDER", "");
+87 -66
View File
@@ -2,9 +2,11 @@
// this runtime's own kernel and sdk. The design follows #845 by itaskaev2, with the init run natively instead of under
// Unicorn, and run again at every start instead of an image taken after one run at build time.
//
// The service run: totk.exe --plugin-init DIR runs every plugin's DT_INIT, records what the init did and exits before
// any game module's constructor, nnMain, graphics or audio. The record is the input of tools/exefs_plugins.py, which
// checks it and writes the plugin's patch set (its hook sites, recomp/patches.py `plugin` lines).
// The service run: totk.exe --plugin-init DIR runs the plugins' DT_INITs, records what the inits did and exits before
// any game module's constructor, nnMain, graphics or audio. TOTK_PLUGIN_ONLY names the one plugin module whose init
// runs ("-": none); the others stay loaded, as they are at play. tools/exefs_plugins.py runs each plugin on its own,
// so a plugin that is refused, crashes or hangs costs only itself; it checks each record and writes the plugin's patch
// set (its hook sites, recomp/patches.py `plugin` lines).
//
// At every start (plugins_start, from the module-init callback in boot.c): when the build has a plugin whose patch set
// its mod switched on, sdk's DT_INIT runs first, then the plugin's, in the same environment as in the service run, and
@@ -15,8 +17,11 @@
// its GOT, its alias addresses. What the init wrote into the game modules (its hook branches) and the plugin's .text
// after it (its stubs and trampolines) are compared with the record the build used (patches.h RT_PLUGIN_WRITES and
// RT_PLUGINS): equal, the plugin's set stays on and the generated hook sites call into it; different, the set goes off
// for this run, the reason is logged and the mod waits for a rebuild (mods.c mods_plugin_off). The branches written
// into main's .text are taken back afterwards: the lifted code is the code that runs, and main keeps the game's bytes.
// for this run, the reason is logged and the mod waits for a rebuild (mods.c mods_plugin_off). Each plugin is compared
// right after its own init, and the branches it wrote into main's .text are taken back before the next init: the next
// plugin sees the game's own bytes, as in its own service run (two plugins hooking one site each relocate the game's
// instruction; the generated site runs the one the Mods tab order picks). The lifted code is the code that runs, and
// main keeps the game's bytes.
// Without a plugin in the build none of this runs (RT_PLUGIN_COUNT 0).
//
// What exlaunch's init needs from the process, and how this run gives it:
@@ -37,10 +42,10 @@
// Map/UnmapProcessMemory); any other one, such as CreateSession and ReplyAndReceive of exlaunch's process-handle
// fallback, ends the run with the reason in the record.
//
// The record, in DIR: record.json (modules, the writes into every game module as words, the alias calls, svc and
// import counts, the tick offset, a refusal), plugin<N>.img (the plugin's image after its init, all segments) and
// plugin<N>.pre.img (the same after loading and relocation, before the init). TOTK_PLUGIN_TICK_OFFSET adds to every
// GetSystemTick, so two runs pick different aliases.
// The record, in DIR: record.json (modules, which plugins' inits ran, the writes into every game module as words, the
// alias calls, svc and import counts, the tick offset, a refusal), plugin<N>.img (the plugin's image after its init,
// all segments) and plugin<N>.pre.img (the same after loading and relocation, before the init).
// TOTK_PLUGIN_TICK_OFFSET adds to every GetSystemTick, so two runs pick different aliases.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -93,6 +98,7 @@ static uint64_t svc_count[0x80], info_count[16], info_65001;
static struct { const char *name; uint64_t calls; } imports[MAX_IMPORTS];
static int nimports;
static uint8_t **pre; // per loaded module: its image before the plugins' inits
static uint8_t *ran; // service run, per loaded module: its init ran
static uint64_t align_up(uint64_t v, uint64_t a) { return (v + a - 1) & ~(a - 1); }
@@ -344,6 +350,7 @@ static void write_record(const char *refused) {
i ? "," : "", m->name, (unsigned long long)m->base, (unsigned long long)m->size,
(unsigned long long)m->text_end, (unsigned long long)m->data_off, (unsigned long long)m->init,
m->plugin ? "true" : "false");
if (m->plugin) fprintf(f, ", \"ran\": %s", ran && ran[i] ? "true" : "false");
if (m->plugin && !refused) {
char img[64], pre_img[64], hex[65];
snprintf(img, sizeof img, "%s.img", m->name);
@@ -460,9 +467,17 @@ void plugin_init_run(Cpu *c, const RtLoaded *sdk) {
if (!pre[i]) rt_fatal("plugin-init: out of memory for a copy of %s", m->name);
memcpy(pre[i], (const void *)(uintptr_t)m->base, m->size);
}
const char *only = cfg_get("TOTK_PLUGIN_ONLY");
ran = calloc((size_t)ldr_count(), 1);
if (!ran) rt_fatal("plugin-init: out of memory");
for (int i = ldr_count() - 1; i >= 0; i--) {
const RtLoaded *m = ldr_module(i);
if (!m->plugin) continue;
if (only && *only && strcmp(only, m->name)) {
rt_log("plugin-init: %s stays loaded, its init does not run (TOTK_PLUGIN_ONLY=%s)", m->name, only);
continue;
}
ran[i] = 1;
if (!m->init) refuse("a plugin without DT_INIT");
rt_log("plugin-init: ENTER %s DT_INIT at %#llx (%llu init_array entries)", m->name, (unsigned long long)m->init,
(unsigned long long)(m->init_array_size / 8));
@@ -546,17 +561,59 @@ static void trace_entries(void) {
rt_log("plugins: trace: %d hook entries counted", ntraced);
}
// Takes back the words the inits changed in the sealed part (.text, .rodata) of a game module: main keeps the game's
// own bytes, the lifted code being what runs.
static void restore_sealed(const RtLoaded *m, const uint8_t *was) {
for (uint64_t off = 0; off < m->data_off; off += PAGE) {
const uint8_t *now = (const uint8_t *)(uintptr_t)(m->base + off);
if (!memcmp(now, was + off, PAGE)) continue;
if (!plat_mem_protect(m->base + off, PAGE, PLAT_PROT_RW)) rt_fatal("plugins: cannot unseal %s+%#llx", m->name,
(unsigned long long)off);
memcpy((void *)(uintptr_t)(m->base + off), was + off, PAGE);
plat_mem_protect(m->base + off, PAGE, PLAT_PROT_READ);
// Takes back a page an init changed in the sealed part (.text, .rodata) of a game module: main keeps the game's own
// bytes, the lifted code being what runs.
static void restore_page(const RtLoaded *m, uint64_t off, const uint8_t *was) {
if (!plat_mem_protect(m->base + off, PAGE, PLAT_PROT_RW)) rt_fatal("plugins: cannot unseal %s+%#llx", m->name,
(unsigned long long)off);
memcpy((void *)(uintptr_t)(m->base + off), was, PAGE);
plat_mem_protect(m->base + off, PAGE, PLAT_PROT_READ);
}
// After plugin i's init: what it wrote into the game modules, word by word, against the record of the build (its
// entries of RT_PLUGIN_WRITES). The first difference goes into why (unless a reason is there already). Then the game
// modules are made what they were before the init: the sealed part (.text, .rodata) gets the game's bytes back, a
// write elsewhere stays and goes into pre, so the next plugin's init is compared only with what it writes itself.
// The aliases this init mapped (from first_alias on) are flushed first; an earlier init's were flushed after it.
static void check_writes(int plugin, int first_alias, char *why, size_t cap) {
for (int i = first_alias; i < naliases; i++)
if (aliases[i].live) flush(&aliases[i]);
uint8_t seen[NWRITES > 0 ? NWRITES : 1];
memset(seen, 0, sizeof seen);
int unexpected = 0, missing = 0;
char first[160] = "";
for (int i = 0; i < ldr_count(); i++) {
const RtLoaded *m = ldr_module(i);
if (!pre[i]) continue;
uint8_t *now = (uint8_t *)(uintptr_t)m->base;
for (uint64_t off = 0; off < m->size; off += PAGE) {
if (!memcmp(now + off, pre[i] + off, PAGE)) continue;
for (uint64_t w = off; w < off + PAGE; w += 4) {
uint32_t a, b;
memcpy(&a, pre[i] + w, 4);
memcpy(&b, now + w, 4);
if (a == b) continue;
int k = 0;
while (k < NWRITES && !(write_tab[k].plugin == plugin && write_tab[k].off == w &&
!strcmp(write_tab[k].module, m->name) && write_tab[k].old == a &&
write_tab[k].now == b))
k++;
if (k < NWRITES) {
seen[k] = 1;
} else if (!unexpected++) {
snprintf(first, sizeof first, "%s+%#llx %08x -> %08x", m->name, (unsigned long long)w, a, b);
}
}
if (off < m->data_off) restore_page(m, off, pre[i] + off);
else memcpy(pre[i] + off, now + off, PAGE);
}
}
for (int k = 0; k < NWRITES; k++) missing += write_tab[k].plugin == plugin && !seen[k];
if (why[0]) return;
if (unexpected)
snprintf(why, cap, "its init wrote %d words the build did not record (first %s)", unexpected, first);
else if (missing)
snprintf(why, cap, "%d of the words the build recorded were not written", missing);
}
#endif
@@ -579,7 +636,11 @@ int plugins_start(Cpu *c, const RtLoaded *sdk) {
}
run[i] = 1, nrun++;
}
if (!nrun) return 0;
if (!nrun) { // every plugin's mod off or gone: the game sees its address space as without plugins
mem_query_hook = NULL;
rt_process_memory = NULL;
return 0;
}
uint64_t t0 = plat_now_ms();
rt_log("boot: ENTER %s DT_INIT first: a mod plugin's init runs before the game's constructors", sdk->name);
rt_call(c, sdk->init, 0);
@@ -597,62 +658,22 @@ int plugins_start(Cpu *c, const RtLoaded *sdk) {
const RtLoaded *m = loaded(plugin_tab[i].module);
rt_log("plugins: ENTER %s DT_INIT at %#llx", m->name, (unsigned long long)m->init);
failed[0] = 0;
int first_alias = naliases;
running = 1;
rt_trace |= RT_TRACE_PLUGIN;
rt_call(c, m->init, 0);
rt_trace &= ~(unsigned)RT_TRACE_PLUGIN;
running = 0;
if (failed[0]) snprintf(why[i], sizeof why[i], "its init asked for what the port does not serve: %s", failed);
}
for (int i = 0; i < naliases; i++)
if (aliases[i].live) flush(&aliases[i]);
// What the inits wrote into the game modules, word by word, against the record of the build.
uint8_t seen[NWRITES > 0 ? NWRITES : 1];
memset(seen, 0, sizeof seen);
int unexpected = 0;
char first[160] = "";
for (int i = 0; i < ldr_count(); i++) {
const RtLoaded *m = ldr_module(i);
if (!pre[i]) continue;
const uint8_t *now = (const uint8_t *)(uintptr_t)m->base;
for (uint64_t off = 0; off < m->size; off += PAGE) {
if (!memcmp(now + off, pre[i] + off, PAGE)) continue;
for (uint64_t w = off; w < off + PAGE; w += 4) {
uint32_t a, b;
memcpy(&a, pre[i] + w, 4);
memcpy(&b, now + w, 4);
if (a == b) continue;
int k = 0;
while (k < NWRITES && !(write_tab[k].off == w && !strcmp(write_tab[k].module, m->name) &&
write_tab[k].old == a && write_tab[k].now == b))
k++;
if (k < NWRITES) {
seen[k] = 1;
} else if (!unexpected++) {
snprintf(first, sizeof first, "%s+%#llx %08x -> %08x", m->name, (unsigned long long)w, a, b);
}
}
}
restore_sealed(m, pre[i]);
free(pre[i]);
}
free(pre);
pre = NULL;
for (int i = 0; i < RT_PLUGIN_COUNT; i++) {
if (!run[i] || why[i][0]) continue;
const RtLoaded *m = loaded(plugin_tab[i].module);
int missing = 0;
for (int k = 0; k < NWRITES; k++) missing += write_tab[k].plugin == i && !seen[k];
check_writes(i, first_alias, why[i], sizeof why[i]);
char hex[65];
sha256_hex((const void *)(uintptr_t)m->base, m->text_end, hex);
if (unexpected)
snprintf(why[i], sizeof why[i], "the inits wrote %d words the build did not record (first %s)", unexpected,
first);
else if (missing)
snprintf(why[i], sizeof why[i], "%d of the words the build recorded were not written", missing);
else if (strcmp(hex, plugin_tab[i].sha256))
if (!why[i][0] && strcmp(hex, plugin_tab[i].sha256))
snprintf(why[i], sizeof why[i], "its code after the init is not the one the build lifted");
}
for (int i = 0; i < ldr_count(); i++) free(pre[i]);
free(pre);
pre = NULL;
mem_query_hook = NULL; // the game's own QueryMemory sees its address space as without plugins
rt_process_memory = NULL; // a callback that maps process memory stops with a named fatal (svc.c)
int on = 0;
+93
View File
@@ -13,6 +13,7 @@ import struct
import sys
import tempfile
import unittest
from types import SimpleNamespace
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
@@ -379,6 +380,98 @@ class Lines(unittest.TestCase):
self.assertFalse(any(x.startswith(('orig', 'stub')) for x in out))
# ---- one service run per plugin
class PerPlugin(unittest.TestCase):
"""main() with three plugins in three mods and a stand-in for the service run: plugin0 hooks its site, plugin1's
init ends the run without a record (a crash), plugin2's run is refused (an svc). Each gets its own status line;
only plugin0's set goes into the patch file."""
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.d = Path(tmp.name)
self.files = []
for name in ('A', 'B', 'C'):
f = self.d / 'mods' / name / 'exefs' / 'subsdk9'
f.parent.mkdir(parents=True)
f.write_bytes(nso(PLUGIN_TEXT) + name.encode())
self.files.append(f)
(self.d / 'patches.txt').write_text('set fps60 TOTK_FPS60 1\n')
self.runs = []
saved = xp.run, xp.hook_impls
self.real_run = xp.run
self.addCleanup(lambda: (setattr(xp, 'run', saved[0]), setattr(xp, 'hook_impls', saved[1])))
xp.run, xp.hook_impls = self.fake_run, lambda rec: IMPLS
def fake_run(self, exe, out, tick_offset, timeout, only):
self.runs.append(only)
out.mkdir(parents=True, exist_ok=True)
if only == 'plugin1':
return 0xC0000005, 'its init ended the service run without a record (exit 0xc0000005)'
rec = record(writes=[] if only != 'plugin0' else None)
rec['modules'][2]['file'] = str(self.files[0])
for i in (1, 2):
rec['modules'].append(dict(rec['modules'][2], name=f'plugin{i}', base=PLUGIN + i * 0x100000,
file=str(self.files[i])))
for m in rec['modules'][2:]:
m['ran'] = m['name'] == only
if only == 'plugin2':
rec['refused'] = 'svc 0x1 at plugin2+0x50a0'
(out / 'plugin0.img').write_bytes(rec.pop('images')['plugin0'])
rec['modules'][2]['image'] = 'plugin0.img'
(out / 'record.json').write_text(json.dumps(rec))
return 3 if rec['refused'] else 0, None
def test_each_plugin_alone(self):
out, status, patched = self.d / 'rec', self.d / 'plugins.status', self.d / 'patches-out.txt'
argv = sys.argv
sys.argv = ['exefs_plugins.py', '--exe', 'totk.exe', '--out', str(out), '--mods', str(self.d / 'mods'),
'--patches', str(self.d / 'patches.txt'), '--patches-out', str(patched), '--status', str(status)]
try:
code = xp.main()
finally:
sys.argv = argv
self.assertEqual(code, 1)
self.assertEqual(self.runs, ['-', 'plugin0', 'plugin0', 'plugin1', 'plugin2'])
lines = {line.split('\t')[0]: line.split('\t') for line in status.read_text().splitlines()}
key = {n: xp.fnv(f'{n}/exefs/subsdk9'.encode()) for n in 'ABC'}
self.assertEqual(lines[key['A']][2:], ['plugin', '-', 'plugin0, 1 hooks'])
self.assertEqual(lines[key['B']][2], 'unsupported')
self.assertIn('ended the service run', lines[key['B']][4])
self.assertEqual(lines[key['C']][2], 'unsupported')
self.assertIn('svc 0x1', lines[key['C']][4])
text = patched.read_text()
self.assertIn(f'set mod_{key["A"].lower()} ', text)
self.assertNotIn(f'mod_{key["B"].lower()}', text)
self.assertNotIn(f'mod_{key["C"].lower()}', text)
self.assertIn((out / 'plugin0' / 'run1' / 'plugin0.img').resolve().as_posix(), text)
def test_run_reasons(self):
# the real run(): a hang becomes its own reason, a crash names the FATAL line
saved = xp.subprocess.run
self.addCleanup(setattr, xp.subprocess, 'run', saved)
def hang(cmd, **kw):
raise xp.subprocess.TimeoutExpired(cmd, kw['timeout'], output=b'started', stderr=None)
xp.subprocess.run = hang
self.assertEqual(self.real_run(None, self.d / 'h', 0, 7, 'plugin0'), (None, 'its init did not return within 7 s'))
xp.subprocess.run = lambda cmd, **kw: SimpleNamespace(returncode=0xC0000005, stdout='x\nFATAL: access violation '
'(0xc0000005) at host 1\n', stderr='')
code, why = self.real_run(None, self.d / 'c', 0, 7, 'plugin0')
self.assertEqual(why, 'its init crashed the service run: access violation (0xc0000005) at host 1')
def test_ran_view_leaves_out_plugins_that_did_not_run(self):
rec = record()
rec['modules'].append(dict(rec['modules'][2], name='plugin1', base=PLUGIN + 0x100000, ran=False))
rec['modules'][2]['ran'] = True
rec['images']['plugin1'] = rec['images']['plugin0']
view = xp.ran_view(rec)
self.assertEqual([m['name'] for m in view['modules'] if m['plugin']], ['plugin0'])
self.assertEqual(set(view['images']), {'plugin0'})
whole = record() # no "ran": every init ran, the record is its own view
self.assertIs(xp.ran_view(whole), whole)
# ---- the generator on those lines
def code_nso(words_at, size_text=0x5000):
"""An NSO as nso() whose .text holds words_at {offset: word}."""
+34 -9
View File
@@ -383,11 +383,19 @@ class LiftRunTest(unittest.TestCase):
self.assertEqual(self.outputs(), [False, False])
self.assertIn('warn.exefs_skipped', self.ctx.reporter.out.getvalue())
def plugin_setup(self, kind):
"""A mod with an exlaunch plugin; the fake service run writes its status line of that kind."""
def plugin_setup(self, kind, more=()):
"""A mod with an exlaunch plugin (SQ), and one more mod per (name, kind) of more; the fake service run writes
each plugin's status line of its kind."""
kinds = {'SQ': kind, **dict(more)}
for name in kinds:
(self.dest / 'mods' / name / 'exefs').mkdir(parents=True, exist_ok=True)
(self.dest / 'mods' / name / 'exefs' / 'subsdk9').write_bytes(b'NSO0' + name.encode())
plugin = self.dest / 'mods' / 'SQ' / 'exefs' / 'subsdk9'
plugin.write_bytes(b'NSO0')
self.ctx.layout = SimpleNamespace(**vars(self.ctx.layout), llvm='llvm', runtime='runtime', ninja='ninja',
dll = self.dest.parent / 'release-bin' / 'SDL3.dll'
dll.parent.mkdir(exist_ok=True)
dll.write_bytes(b'MZ')
(self.ctx.temp / 'build').mkdir(parents=True, exist_ok=True)
self.ctx.layout = SimpleNamespace(**vars(self.ctx.layout), llvm='llvm', runtime='runtime', ninja='ninja', dlls=[dll],
deps=None, sysroot=None, runtime_lib_for=lambda cpu: None,
pgo=lambda version: None, expected=lambda version: 'expected.json')
saved = self.run_tool
@@ -399,11 +407,14 @@ class LiftRunTest(unittest.TestCase):
Path(args[args.index('--patches-out') + 1]).write_text('patches\n')
status = Path(args[args.index('--status') + 1])
status.parent.mkdir(parents=True, exist_ok=True)
h = 0x811C9DC5 # FNV-1a 32 of "SQ/exefs/subsdk9", as tools/exefs_plugins.py keys it
for b in b'SQ/exefs/subsdk9':
h = ((h ^ b) * 0x01000193) & 0xFFFFFFFF
status.write_text(f'{h:08X}\t00000000\t{kind}\t-\tnote\n')
return (1 if kind == 'unsupported' else 0), log
lines = []
for name, k in kinds.items():
h = 0x811C9DC5 # FNV-1a 32 of "<mod>/exefs/subsdk9", as tools/exefs_plugins.py keys it
for b in f'{name}/exefs/subsdk9'.encode():
h = ((h ^ b) * 0x01000193) & 0xFFFFFFFF
lines.append(f'{h:08X}\t00000000\t{k}\t-\tnote\n')
status.write_text(''.join(lines))
return (1 if 'unsupported' in kinds.values() else 0), log
return code, log
steps.run_tool = run_tool
return plugin
@@ -419,6 +430,8 @@ class LiftRunTest(unittest.TestCase):
self.assertEqual(last[last.index('--plugins') + 1], str(plugin))
self.assertEqual(last[last.index('--patches') + 1], str(self.ctx.temp / 'patches-plugins.txt'))
self.assertTrue(steps.plugins_status(self.ctx).is_file())
# the service run's totk.exe finds its DLLs beside it
self.assertTrue((self.ctx.temp / 'build' / 'SDL3.dll').is_file())
def test_refused_plugin_is_not_lifted(self):
# a refused plugin: its status line stays (the Mods tab says why), the game is lifted without it
@@ -429,6 +442,18 @@ class LiftRunTest(unittest.TestCase):
self.assertTrue(steps.plugins_status(self.ctx).is_file())
self.assertNotIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue())
def test_refused_plugin_stays_in_the_layout_of_a_built_one(self):
# one plugin built, one refused: the lift with the hook sites keeps both in the layout (the records hold the
# addresses of the service runs' layout); the refused one has no patch set, its status line says why
self.plugin_setup('plugin', [('ZZ', 'unsupported')])
steps.Lift().run(self.ctx)
last = self.calls[4][1]
mods = self.dest / 'mods'
self.assertEqual(last[last.index('--plugins') + 1],
','.join(str(mods / n / 'exefs' / 'subsdk9') for n in ('SQ', 'ZZ')))
self.assertIn('unsupported', steps.plugins_status(self.ctx).read_text())
self.assertNotIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue())
def test_failed_plugin_build_warns(self):
self.plugin_setup('plugin')
self.fail = {'build_game.py'}
+4 -2
View File
@@ -188,8 +188,10 @@ DEFAULT_DEPS = ROOT / 'toolchain' / 'deps-mingw'
# Runtime sources compiled per install, not in the library: romfs.c includes romfs_config.h, patches.c and ov_zstd.c
# the patches.h of the player's game version (its patch page is the page after that version's main; ov_zstd.c's host
# functions sit on it). mouse_direct.c and ov_ability.c also register host callbacks relative to that version's patch
# page. A library source that includes one of PER_VERSION_HEADERS stops the library build (library_sources).
INSTALL_SIDE = {'romfs.c', 'patches.c', 'ov_zstd.c', 'mouse_direct.c', 'ov_ability.c'}
# page. plugin_init.c reads the install's own plugin records from patches.h (RT_PLUGINS, RT_PLUGIN_WRITES): a mod plugin
# the rebuild built in. A library source that includes one of PER_VERSION_HEADERS stops the library build
# (library_sources).
INSTALL_SIDE = {'romfs.c', 'patches.c', 'ov_zstd.c', 'mouse_direct.c', 'ov_ability.c', 'plugin_init.c'}
PER_VERSION_HEADERS = ('"patches.h"', '"romfs_config.h"')
LIB_NAME = 'libtotkrt.a'
AR = UCRT / 'llvm-ar.exe' if WIN else on_path('llvm-ar')
+108 -51
View File
@@ -4,8 +4,8 @@ a plugin's hook sites. After the design in #845 by itaskaev2; the init runs on t
same environment (plugin_init.c plugins_start), so the plugin keeps real state (its heap blocks, its GOT, its alias
addresses). The record says what the build expects that start-up to write.
python tools/exefs_plugins.py --exe build/<ver>-x/totk.exe --out <dir> # two runs, their diff, the check
python tools/exefs_plugins.py --check <dir>/run1 # the check of one record
python tools/exefs_plugins.py --exe build/<ver>-x/totk.exe --out <dir> # per plugin: two runs, diff, check
python tools/exefs_plugins.py --check <dir>/<plugin>/run1 # the check of one record
python tools/exefs_plugins.py --exe <exe> --out <dir> --mods <mods folder> --patches <patch file>
--patches-out <patch file> --status <file> # and the patch lines for the generator
@@ -20,11 +20,16 @@ file hash: FNV-1a 32 of "<mod folder name>/<path of the NSO in the mod>"; conten
`plugin` (note: the module and its hook count) or `unsupported` (note: the first reason); built-in sets: the port's
own sets with a site the plugin hooks too ("-" none), which the runtime switches off while the mod is on.
The exe must be built with the plugin in its layout (recomp/gen.py --plugins). --out gets run1/ and run2/, one record
each; run2 adds a large offset to every GetSystemTick (TOTK_PLUGIN_TICK_OFFSET), so the init picks other alias
addresses. The words that differ between the two records are listed with what they look like, not refused: an init
keeps its alias addresses and the regions GetInfo gave it in its .data, and a plugin's callbacks must not use them
(the generator's check of the code the callbacks reach, still to come).
The exe must be built with the plugins in its layout (recomp/gen.py --plugins). Each plugin runs in service runs of
its own (TOTK_PLUGIN_ONLY): the other plugins stay loaded, their inits do not run. A plugin that is refused, crashes,
hangs or fails the check is refused alone, with its reason; the others are built in. At every start the game runs each
init against the game's own bytes as well (runtime/plugin_init.c plugins_start takes a plugin's writes back before the
next init), so a plugin's own record is what its start must match. --out gets list/ (a run without inits, which names
the plugin modules), and <plugin>/run1 and <plugin>/run2 for each plugin; run2 adds a large offset to every
GetSystemTick (TOTK_PLUGIN_TICK_OFFSET), so the init picks other alias addresses. The words that differ between the two
records are listed with what they look like, not refused: an init keeps its alias addresses and the regions GetInfo
gave it in its .data, and a plugin's callbacks must not use them. A difference in code (a write into a game module,
the plugin's .text or .rodata) refuses the plugin: its start could not be compared with the record.
The check (check_record) accepts a record only when every rule holds; each broken rule is one reason:
- the run was not refused (an svc outside the served set, such as CreateSession of exlaunch's process-handle
@@ -90,6 +95,19 @@ def u(v):
return int(v, 16) if isinstance(v, str) else v
def ran_view(rec):
"""The record as seen from the plugins whose inits ran: a plugin that only stayed loaded (TOTK_PLUGIN_ONLY) counts
as a module that is no plugin, so its exports and code are not checked as this record's. A record without "ran"
(every init ran) comes back as it is."""
if all(m.get('ran', True) for m in rec['modules'] if m['plugin']):
return rec
view = dict(rec)
view['modules'] = [dict(m, plugin=False) if m['plugin'] and not m.get('ran', True) else m for m in rec['modules']]
names = {m['name'] for m in view['modules'] if m['plugin']}
view['images'] = {k: v for k, v in rec['images'].items() if k in names}
return view
# ---- instructions
def sext(v, bits):
return v - (1 << bits) if v >> (bits - 1) & 1 else v
@@ -433,35 +451,35 @@ def plugin_files(mods):
return out
def emit(rec, hooks, reasons, record_dir, args):
"""--patches-out and --status from the record (accepted or not)."""
def emit(results, args):
"""--patches-out and --status from the checked plugins. results: one (module, its NSO file, its record or None,
the record's directory, hooks, reasons) per plugin of the layout; no reasons: accepted."""
text = args.patches.read_text() if args.patches else ''
builtin = builtin_sites(text)
status, sets = [], []
files = {str(f.resolve()): (m, rel, f) for m, rel, f in plugin_files(args.mods)}
plugins = [m for m in rec['modules'] if m['plugin']]
for p in plugins:
hit = files.get(str(Path(p.get('file', '')).resolve()))
for module, file, rec, record_dir, hooks, reasons in results:
hit = files.get(str(Path(file or '').resolve()))
if not hit:
print(f'plugins: {p["name"]}: {p.get("file")} is not under {args.mods}')
print(f'plugins: {module}: {file} is not under {args.mods}')
continue
mod, rel, f = hit
key, content = fnv(f'{mod}/{rel}'.encode()), fnv(f.read_bytes())
if reasons:
note = reasons[0].replace('\t', ' ')
note = ' '.join(reasons[0].split())
status.append(f'{key}\t{content}\tunsupported\t-\t{note}')
print(f'plugins: unsupported {mod}/{rel} ({note})')
continue
name = f'mod_{key.lower()}'
own = hooks_of(hooks, p['name'])
own = hooks_of(hooks, module)
sites = [int(h['site'].split('+')[1], 16) for h in own]
over = sorted({b for a in sites for b in builtin.get(a, [])})
label = f'{mod}/{rel}'.encode('ascii', 'replace').decode()
sets.append([f'set {name} TOTK_MODPATCH_{key} 0 ordered # {label}'] + plugin_lines(rec, own, name, record_dir,
p['name']))
status.append(f'{key}\t{content}\tplugin\t{",".join(over) or "-"}\t{p["name"]}, {len(own)} hooks')
print(f'plugins: plugin {mod}/{rel} as {p["name"]}, {len(own)} hooks' + (f', over {", ".join(over)}'
if over else ''))
module))
status.append(f'{key}\t{content}\tplugin\t{",".join(over) or "-"}\t{module}, {len(own)} hooks')
print(f'plugins: plugin {mod}/{rel} as {module}, {len(own)} hooks' + (f', over {", ".join(over)}'
if over else ''))
if sets:
text += '\n# Mod plugins (tools/exefs_plugins.py)\n' + '\n'.join(line for s in sets for line in s) + '\n'
args.patches_out.write_text(text)
@@ -469,7 +487,7 @@ def emit(rec, hooks, reasons, record_dir, args):
args.status.write_text('\n'.join(status) + ('\n' if status else ''), encoding='utf-8')
# ---- two runs
# ---- the service runs
def where_is(v, rec):
"""What an address-like value points into, for the diff listing."""
for m in rec['modules']:
@@ -524,17 +542,34 @@ def diff_records(a, b):
return out
def run(exe, out, tick_offset, timeout):
def run(exe, out, tick_offset, timeout, only):
"""One service run into out, only module `only`'s init running ("-": none). (exit code or None, why the run gave
no record or None); a refused run writes its record too."""
out.mkdir(parents=True, exist_ok=True)
env = dict(os.environ, TOTK_PLUGIN_TICK_OFFSET=f'{tick_offset:#x}', TOTK_BACKGROUND_WINDOW='1',
TOTK_ASSUME_FOCUS='1')
r = subprocess.run([str(exe), '--plugin-init', str(out)], env=env, capture_output=True, text=True,
errors='replace', timeout=timeout)
for f in out.iterdir(): # an earlier run's record must not pass for this one's
if f.is_file() and (f.name in ('record.json', 'run.log') or f.suffix == '.img'):
f.unlink()
env = dict(os.environ, TOTK_PLUGIN_TICK_OFFSET=f'{tick_offset:#x}', TOTK_PLUGIN_ONLY=only,
TOTK_BACKGROUND_WINDOW='1', TOTK_ASSUME_FOCUS='1')
try:
r = subprocess.run([str(exe), '--plugin-init', str(out)], env=env, capture_output=True, text=True,
errors='replace', timeout=timeout)
except subprocess.TimeoutExpired as e:
log = b''.join(x if isinstance(x, bytes) else x.encode() for x in (e.stdout, e.stderr) if x)
(out / 'run.log').write_text(log.decode(errors='replace'), encoding='utf-8')
return None, f'its init did not return within {timeout} s'
(out / 'run.log').write_text(r.stdout + r.stderr, encoding='utf-8')
return r.returncode
if not (out / 'record.json').is_file():
# the runtime's crash report names the fault in a FATAL line (an access violation, an abort)
fatal = next((line.split('FATAL:', 1)[1].strip() for line in (r.stdout + r.stderr).splitlines()
if 'FATAL:' in line), None)
return r.returncode, (f'its init crashed the service run: {fatal}' if fatal else
f'its init ended the service run without a record (exit {r.returncode & 0xFFFFFFFF:#x})')
return r.returncode, None
def report(rec, label):
rec = ran_view(rec)
impls = hook_impls(rec)
hooks, reasons = check_record(rec, impls)
more, notes = reach_check(rec, hooks, impls)
@@ -559,12 +594,40 @@ def report(rec, label):
return hooks, reasons
def plugin_runs(exe, out, module, timeout):
"""The two service runs of one plugin (run2 with the tick offset) into out/<module>: (its first record or None,
hooks, reasons)."""
recs = []
for i, offset in enumerate((0, TICK_OFFSET), 1):
d = out / module / f'run{i}'
code, why = run(exe, d, offset, timeout, module)
if why:
print(f'{module} run {i}: {why}; see {d / "run.log"}')
return None, [], [why]
recs.append(load_record(d))
print(f'{module} run {i}: exit {code}, tick offset {offset:#x}')
if recs[0].get('refused'): # refused: a second run says nothing more
hooks, bad = report(recs[0], f'{module} run 1')
return recs[0], hooks, bad
hooks, bad = report(recs[0], f'{module} run 1')
diff = diff_records(ran_view(recs[0]), ran_view(recs[1]))
print(f'{module} run 1 against run 2: {len(diff)} differences' + (':' if diff else ''))
for line in diff:
print(f' {line}')
# Words of a game module or of the plugin's .text that differ between the runs: the start-up could not be compared
# with the record. Its .data and .bss may differ (alias addresses).
unstable = [line for line in diff if line.startswith('write ') or '(.text)' in line or '(.rodata)' in line]
if unstable:
bad = bad + [f'two runs of the init differ in code: {unstable[0]}']
return recs[0], hooks, bad
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--exe', type=Path, help='totk.exe built with the plugin in its layout')
ap.add_argument('--out', type=Path, help='directory for run1/ and run2/')
ap.add_argument('--exe', type=Path, help='totk.exe built with the plugins in its layout')
ap.add_argument('--out', type=Path, help='directory for list/ and <plugin>/run1, run2')
ap.add_argument('--check', type=Path, help='check the record in this directory only')
ap.add_argument('--timeout', type=int, default=300)
ap.add_argument('--timeout', type=int, default=120, help='seconds a service run may take')
ap.add_argument('--mods', type=Path, help='the mods folder the plugins come from')
ap.add_argument('--patches', type=Path, help='the patch file so far')
ap.add_argument('--patches-out', type=Path, help='the patch file with the plugins\' sets added')
@@ -578,32 +641,26 @@ def main():
rec = load_record(args.check)
hooks, bad = report(rec, str(args.check))
if emitting:
emit(rec, hooks, bad, args.check, args)
emit([(m['name'], m.get('file'), rec, args.check, hooks, bad) for m in ran_view(rec)['modules']
if m['plugin']], args)
return 1 if bad else 0
if not args.exe or not args.out:
ap.error('--exe and --out, or --check')
recs = []
for i, offset in enumerate((0, TICK_OFFSET), 1):
d = args.out / f'run{i}'
code = run(args.exe, d, offset, args.timeout)
if not (d / 'record.json').is_file():
print(f'run {i}: no record (exit {code}); see {d / "run.log"}')
return 1
recs.append(load_record(d))
print(f'run {i}: exit {code}, tick offset {offset:#x}')
hooks, bad = report(recs[0], 'run 1')
diff = diff_records(*recs)
print(f'run 1 against run 2: {len(diff)} differences' + (':' if diff else ''))
for line in diff:
print(f' {line}')
# Words of a game module or of a plugin's .text that differ between the runs: the start-up could not be compared
# with the record. Its .data and .bss may differ (alias addresses).
unstable = [line for line in diff if line.startswith('write ') or '(.text)' in line or '(.rodata)' in line]
if unstable:
bad = bad + [f'two runs of the init differ in code: {unstable[0]}']
# A run without any init names the plugin modules of the exe's layout and checks that the service run works at all:
# without it no plugin can be built.
code, why = run(args.exe, args.out / 'list', 0, args.timeout, '-')
if why:
print(f'the service run without inits gave no record: {why}; see {args.out / "list" / "run.log"}')
return 2
listing = load_record(args.out / 'list')
results = []
for m in (m for m in listing['modules'] if m['plugin']):
rec, hooks, bad = plugin_runs(args.exe, args.out, m['name'], args.timeout)
print(f'{m["name"]}: ' + (f'refused: {bad[0]}' if bad else f'accepted, {len(hooks)} hooks'))
results.append((m['name'], m.get('file'), rec, args.out / m['name'] / 'run1', hooks, bad))
if emitting:
emit(recs[0], hooks, bad, args.out / 'run1', args)
return 1 if bad else 0
emit(results, args)
return 1 if any(r[5] for r in results) else 0
if __name__ == '__main__':
+2 -1
View File
@@ -25,7 +25,8 @@ runtime/ and bin/ (PATH holds the Windows system directory and those), then runs
Both need one game version extracted and lifted first (tools/extract.py, recomp/gen.py): generated/v121 or, without
it, generated/v143, and for --prove its build/extracted/<version> (they are read, nothing is written to them). The
runtime library reads only what both versions' layout.h share (MOD_COUNT, MOD_MAIN_BASE), so either serves.
runtime library reads only what both versions' layout.h share (MOD_MAIN_BASE; its module tables hold RT_MAX_MODULES, so an
install's layout may add mod plugins), so either serves.
A release also needs build/xlate built from the current sources (tools/build_xlate.py); the script stops first if not.