Store the data writes of mods' ExeFS patch files into main before the game runs

An IPS, IPS32 or pchtxt file for main that writes main's .rodata, .data or .bss was refused as a whole. Such writes
need no lifting: tools/exefs_patches.py now puts them in build-cache/exefs.data, and runtime/mods.c stores them into
the loaded main before any game code runs, as emulators and Atmosphere apply a patch file to the module before its
dynamic linker runs. A DT_RELR word gets main's base added; bytes over words an explicit-addend relocation fills change
nothing and are dropped; a write into the dynamic linker's tables, a lazily bound function pointer or outside main's
image keeps the file unsupported. A mod whose data lines do not all fit the build keeps its code patches off too.

The mods are applied from the bottom one up (mods_exefs_apply), so where two mods write the same bytes the higher one
wins, as it does for files. --probe-mods checks it with two mods writing one byte.

The lifted code reads some of main's bytes at lift time (the jump tables). recomp/gen.py lists those ranges in
<gen>/main/folded.txt, and the setup checks the data writes against them after the lift: a file that writes them
becomes unsupported, since the change would not reach the generated code. Generated C without mods is unchanged.

This is the data part of #790 (ericdallo's ExeFS work), split out on its own; the plugin part follows separately.

Co-Authored-By: itaskaev2 <165683571+itaskaev2@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and itaskaev2 committed 2026-11-06 15:51:42 +01:00
1 parent df9ba65a91
commit fa94e36549
13 files changed
+1001 -102

No files matched your search

+2
View File
@@ -164,6 +164,8 @@ class Main:
def __init__(self, path):
n = Nso(str(path))
self.image = bytes(n.image)
n.image = self.image # one copy of the image; tools/exefs_patches.py reads nso's dynamic info
self.nso = n
self.text = n.text # (offset, size)
self.code_end = n.rodata[0] # .text and its padding up to .rodata, where code caves go
self.lifter = Lifter(BASE, 'c')
+256 -44
View File
@@ -1,11 +1,16 @@
"""ExeFS patches of the mods folder to code patch sets: IPS, IPS32 and IPSwitch (.pchtxt) files for the game's main.
"""ExeFS patches of the mods folder to code patch sets and data writes: IPS, IPS32 and IPSwitch (.pchtxt) files for
the game's main.
Lifted code cannot be byte-patched when the game runs, so a patch file's writes into main's code become `site` lines
(recomp/patches.py), one patch set per file, which the runtime switches on while its mod is on (runtime/mods.c). The
writes go through the same rules as the mods' code cheats (tools/cheats.py convert): an instruction replaced in place,
a branch into a code cave the file writes into main's zero padding inlined at the branch.
a branch into a code cave the file writes into main's zero padding inlined at the branch. Writes into main's data
(.rodata, .data, .bss) need no lifting: the runtime stores them into the loaded module before the game runs, as
emulators and Atmosphere apply a patch file to the module before its dynamic linker runs (data_runs).
python tools/exefs_patches.py --exefs <dir> --mods <mods folder> --patches <patch file> --out <patch file> --status <file>
python tools/exefs_patches.py --exefs <dir> --mods <mods folder> --patches <patch file> --out <patch file>
--status <file> --data <file>
python tools/exefs_patches.py --exefs <dir> --status <file> --data <file> --folded <gen>/main/folded.txt
The files are <mod>/exefs/*.ips and *.pchtxt (or the Atmosphere places of the exefs folder, as runtime/mods.c finds
them) and <mod>/atmosphere/exefs_patches/<name>/*.ips. A file is for main when its build id matches main's: an IPS
@@ -13,19 +18,37 @@ file's name, a pchtxt file's @nsobid line (trailing zeros may be left out). Offs
offsets count the NSO header (0x100 bytes) before main's image; a pchtxt offset adds its `@flag offset_shift` first.
A pchtxt patch line applies between @enabled and @disabled; @stop ends the file.
--out gets the --patches file followed by one set per usable file (`mod_<hash>`, switch TOTK_MODPATCH_<HASH>, default
off) and its sites. --status gets one line per file, mods by name, tab separated:
--out gets the --patches file followed by one set per file with sites (`mod_<hash>`, switch TOTK_MODPATCH_<HASH>,
default off) and its sites. --status gets one line per file, mods by name, tab separated:
<file hash> <content hash> <kind> <built-in sets> <note>
file hash: FNV-1a 32 of "<mod folder name>/<path in the mod>" (forward slashes); content hash: the same over the file's
bytes, so the runtime sees a file that changed after the build. kind:
code a set of sites
code a set of sites, and data writes when the file has some
data data writes only
other for another game version or module: not used
empty no write into main for this version (or only the game's own words)
unsupported <note> says why: a write into main's data, a write the site rules cannot express
empty no write into main for this version (or only the game's own bytes)
unsupported <note> says why: a write the site rules cannot express, a write outside main's image, into the dynamic
linker's tables or into a pointer it binds lazily, or (after the lift) into bytes the lifted code reads
as constants
built-in sets: the port's own sets (the --patches file's sets that are no cheats) with a site this file patches too,
comma separated, or "-": the runtime switches them off while the mod is on, so the mod's code runs.
comma separated, or "-": the runtime switches them off while the mod is on, so the mod's code runs. The note of a code
or data file with data writes is `data <bytes>`, the bytes its data lines hold: the runtime keeps the mod's patches off
when the data file does not hold them all.
--data gets one line per run of data bytes of the code and data files, tab separated (runtime/mods.c reads it):
<file hash> <content hash> <offset in main, hex> <bytes, hex> [base]
`base` marks a DT_RELR word: 8 bytes, little-endian, the word before relocation; the runtime adds main's base, as the
dynamic linker does. Without --data a data write makes its file unsupported.
The lifted code reads some of main's .text and .rodata at lift time instead of at run time: the jump tables
(recomp/jumptables.py), whose targets become labels of the generated C. A data write over such bytes would not reach
the lifted code. recomp/gen.py lists the ranges it read in <gen>/<module>/folded.txt; the second form above, run after
the lift, makes every file with a data write into those ranges unsupported (its status line says so and its data lines
go). Without that list, every data write into .rodata counts as one into read bytes.
A file the rules cannot express is unsupported as a whole: half of a patch is worse than none.
@@ -39,9 +62,12 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from cheats import TITLE, Main, convert, fnv # noqa: E402
from nso import (DT_JMPREL, DT_NULL, DT_PLTRELSZ, DT_RELA, DT_RELASZ, DT_RELR, DT_RELRSZ, DT_STRSZ, # noqa: E402
DT_STRTAB, DT_SYMTAB, R_AARCH64_JUMP_SLOT)
NSO_HEADER = 0x100
EXEFS_PLACES = ('exefs', f'atmosphere/contents/{TITLE}/exefs', f'contents/{TITLE}/exefs', f'{TITLE}/exefs')
RUN_MAX = 256 # bytes per line of the data file (runtime/mods.c reads lines of at most 1024 characters)
def patch_files(mods):
@@ -159,6 +185,99 @@ def word_writes(main, writes):
return words
# ---- writes into data
def relocated(n):
"""({offset: type} of the words main's explicit-addend relocations (DT_RELA, DT_JMPREL) fill, {offset} of its
DT_RELR words): the 8-byte words the dynamic linker writes at start (runtime/loader.c relocate_table,
relocate_relr)."""
rela = {}
for table, size in ((DT_RELA, DT_RELASZ), (DT_JMPREL, DT_PLTRELSZ)):
if table in n.dyn:
for i in range(n.dyn[size] // 24):
off, info, _ = struct.unpack_from('<QQq', n.image, n.dyn[table] + 24 * i)
rela[off] = info & 0xFFFFFFFF
relr, where = set(), 0
for i in range(n.dyn.get(DT_RELRSZ, 0) // 8):
e = struct.unpack_from('<Q', n.image, n.dyn[DT_RELR] + 8 * i)[0]
if not e & 1: # an address; the bitmap after it covers the 63 words that follow
relr.add(e)
where = e + 8
else:
relr.update(where + 8 * k for k in range(63) if e >> (k + 1) & 1)
where += 63 * 8
return rela, relr
def linker_tables(n):
"""[(start, end)] of main's .dynamic, relocation, symbol and string tables: the dynamic linker reads them at start,
so a write into them would change what it does, which no store into the loaded module can follow."""
end = n.dynamic
while struct.unpack_from('<Q', n.image, end)[0] != DT_NULL:
end += 16
out = [(n.dynamic, end + 16)]
for table, size in ((DT_RELA, DT_RELASZ), (DT_JMPREL, DT_PLTRELSZ), (DT_RELR, DT_RELRSZ), (DT_STRTAB, DT_STRSZ)):
if table in n.dyn and size in n.dyn:
out.append((n.dyn[table], n.dyn[table] + n.dyn[size]))
if DT_SYMTAB in n.dyn:
out.append((n.dyn[DT_SYMTAB], n.dyn[DT_SYMTAB] + 24 * n.symbol_count()))
return out
class MainData:
"""What a write into main's data meets: main's image from .rodata to the end of .bss, the words the dynamic linker
fills and the tables it reads."""
def __init__(self, main):
n = main.nso
self.image, self.lo, self.size, self.data = main.image, main.code_end, n.size, n.data[0]
self.rela, self.relr = relocated(n)
self.linker = linker_tables(n)
def data_runs(md, writes):
"""The data part of one file ({offset: byte} outside main's code), applied as emulators and Atmosphere apply a patch
file: to the module before its dynamic linker runs. Returns ([(offset, bytes, rebase)], dropped):
- a byte no relocation covers is stored as it is (runs of consecutive bytes, at most RUN_MAX long);
- a DT_RELR word holds its addend: the word with the file's bytes in it, rebase set (main's base gets added);
- the linker overwrites a word an explicit-addend relocation fills whatever the file wrote there: such bytes
change nothing and only count in dropped.
Bytes equal to main's are left out. ValueError for a write outside main's image (below .rodata: the NSO header),
into the dynamic linker's tables, or into a function pointer it binds lazily (JUMP_SLOT): on the Switch it holds
the file's value plus main's base until the first call binds it, which no store before the start can reproduce."""
plain, words, dropped = {}, {}, 0
for off in sorted(writes):
if not md.lo <= off < md.size:
raise ValueError(f'it writes outside main\'s image at main+{off:X}')
if any(a <= off < b for a, b in md.linker):
raise ValueError(f'it writes the dynamic linker\'s tables at main+{off:X}')
w = next((w for w in range(off - 7, off + 1) if w in md.rela or w in md.relr), None)
if w is None:
plain[off] = writes[off]
elif w in md.relr:
words.setdefault(w, bytearray(md.image[w:w + 8]))[off - w] = writes[off]
elif md.rela[w] == R_AARCH64_JUMP_SLOT:
raise ValueError(f'it writes a function pointer the dynamic linker binds lazily at main+{w:X}')
else:
dropped += 1
runs = []
for off in sorted(o for o, b in plain.items() if b != md.image[o]):
last = runs[-1] if runs else None
if last and last[0] + len(last[1]) == off and len(last[1]) < RUN_MAX:
runs[-1] = (last[0], last[1] + bytes([plain[off]]), False)
else:
runs.append((off, bytes([plain[off]]), False))
runs += [(w, bytes(b), True) for w, b in words.items() if bytes(b) != md.image[w:w + 8]]
return sorted(runs), dropped
def data_line(key, content, run):
off, data, rebase = run
return f'{key}\t{content}\t{off:x}\t{data.hex()}' + ('\tbase' if rebase else '')
def builtin_sites(text):
"""{offset in main: [set name]} of the patch file's sets that are no cheats or mod patches."""
out = {}
@@ -169,41 +288,45 @@ def builtin_sites(text):
return out
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--exefs', type=Path, required=True)
ap.add_argument('--mods', type=Path, required=True, help='the mods folder next to totk.exe')
ap.add_argument('--patches', type=Path, required=True, help='the patch file so far')
ap.add_argument('--out', type=Path, required=True, help='the patch file with the mods\' ExeFS patches added')
ap.add_argument('--status', type=Path, required=True)
args = ap.parse_args()
sys.stdout.reconfigure(errors='replace')
def classify(m, md, bid, path, raw):
"""(kind, note, sites, runs, dropped) of one patch file: path gives its name and suffix, raw its bytes, bid main's
build id; md None: no data file, so a write into main's data makes the file unsupported."""
try:
if path.suffix.lower() == '.ips':
for_main = same_build(path.stem, bid)
writes = parse_ips(raw) if for_main else {}
else:
pid, writes = parse_pchtxt(raw.decode('utf-8-sig', errors='replace'))
for_main = pid is not None and same_build(pid, bid)
if not for_main:
return 'other', '', {}, [], 0
code = {o: b for o, b in writes.items() if m.in_code(o - o % 4)}
data = {o: b for o, b in writes.items() if o not in code}
if data and md is None:
raise ValueError(f'it writes main\'s data at main+{min(data):X} (only code patches are supported)')
sites = convert(m, word_writes(m, code)) if code else {}
runs, dropped = data_runs(md, data) if data else ([], 0)
return 'code' if sites else 'data' if runs else 'empty', '', sites, runs, dropped
except ValueError as e:
return 'unsupported', str(e), {}, [], 0
DATA_HEADER = '# file hash, content hash, offset in main, bytes[, base] (tools/exefs_patches.py)'
def build(args):
m = Main(args.exefs / 'main')
raw = (args.exefs / 'main').read_bytes()
bid = raw[0x40:0x60].hex()
md = MainData(m) if args.data else None
bid = m.nso.build_id
text = args.patches.read_text()
builtin = builtin_sites(text)
files = patch_files(args.mods)
sets, sites, status, kinds = [], [], [], {}
sets, sites, status, data_lines, kinds = [], [], [], [], {}
for mod, rel, f in files:
data = f.read_bytes()
key, content = fnv(f'{mod}/{rel}'.encode()), fnv(data)
kind, note, over, file_sites = 'code', '', [], {}
try:
if f.suffix.lower() == '.ips':
for_main = same_build(f.stem, bid)
writes = parse_ips(data) if for_main else {}
else:
pid, writes = parse_pchtxt(data.decode('utf-8-sig', errors='replace'))
for_main = pid is not None and same_build(pid, bid)
if not for_main:
kind = 'other'
else:
file_sites = convert(m, word_writes(m, writes))
if not file_sites:
kind = 'empty'
except ValueError as e:
kind, note, file_sites = 'unsupported', str(e), {}
raw = f.read_bytes()
key, content = fnv(f'{mod}/{rel}'.encode()), fnv(raw)
kind, note, file_sites, runs, dropped = classify(m, md, bid, f, raw)
over = []
if file_sites:
name = f'mod_{key.lower()}'
label = f'{mod}/{rel}'.encode('ascii', 'replace').decode()
@@ -211,19 +334,108 @@ def main():
for a, repl in sorted(file_sites.items()):
sites.append(f'site {name} main {a:x} {m.word(a):08x} {" ".join(f"{w:08x}" for w in repl)}')
over += [s for s in builtin.get(a, []) if s not in over]
data_lines += [data_line(key, content, r) for r in runs]
data_bytes = sum(len(r[1]) for r in runs)
kinds[kind] = kinds.get(kind, 0) + 1
status.append(f'{key}\t{content}\t{kind}\t{",".join(over) or "-"}\t{note.replace(chr(9), " ")}')
print(f'exefs: {kind:11} {mod}/{rel}' + (f' ({note})' if note else '')
+ (f' ({len(file_sites)} sites, over {", ".join(over)})' if over else
f' ({len(file_sites)} sites)' if file_sites else ''))
status_note = f'data {data_bytes}' if runs else note.replace(chr(9), ' ')
status.append(f'{key}\t{content}\t{kind}\t{",".join(over) or "-"}\t{status_note}')
parts = [note] if note else []
if file_sites:
parts.append(f'{len(file_sites)} sites' + (f', over {", ".join(over)}' if over else ''))
if runs:
parts.append(f'{data_bytes} bytes of main\'s data')
if dropped:
parts.append(f'{dropped} bytes into words the dynamic linker fills, which change nothing')
print(f'exefs: {kind:11} {mod}/{rel}' + (f' ({"; ".join(parts)})' if parts else ''))
if sets:
text += ('\n# ExeFS patches of the mods folder (tools/exefs_patches.py)\n' + '\n'.join(sets) + '\n'
+ '\n'.join(sites) + '\n')
args.out.write_text(text)
args.status.parent.mkdir(parents=True, exist_ok=True)
args.status.write_text('\n'.join(status) + ('\n' if status else ''), encoding='utf-8')
if args.data:
args.data.parent.mkdir(parents=True, exist_ok=True)
args.data.write_text('\n'.join([DATA_HEADER] + data_lines) + '\n', encoding='utf-8')
print(f'exefs: {len(files)} patch files: ' + ', '.join(f'{n} {k}' for k, n in sorted(kinds.items()))
+ f'; {len(sites)} sites')
+ f'; {len(sites)} sites, {len(data_lines)} data runs')
def read_ranges(path):
"""[(start, end)] of a folded.txt (recomp/gen.py): hex module offsets, one range per line, '#' comments."""
out = []
for line in path.read_text().splitlines():
f = line.split('#', 1)[0].split()
if len(f) == 2:
out.append((int(f[0], 16), int(f[1], 16)))
return sorted(out)
def data_offset(main_path):
"""The offset of main's .data segment in its image, from the NSO header (no decompression)."""
with open(main_path, 'rb') as f:
head = f.read(0x40)
return struct.unpack_from('<III', head, 0x30)[1]
def recheck(args):
"""After the lift: every file with a data write into bytes the lifted code read at lift time becomes unsupported."""
if not args.status.is_file() or not args.data.is_file():
print('exefs: no status or data file: nothing to check')
return
folded = read_ranges(args.folded) if args.folded.is_file() else None
data_off = data_offset(args.exefs / 'main')
lines = args.data.read_text(encoding='utf-8').splitlines()
bad = {}
for line in lines:
f = line.split('\t')
if line.startswith('#') or len(f) < 4 or f[0] in bad:
continue
a = int(f[2], 16)
b = a + len(f[3]) // 2
if folded is None and a < data_off:
bad[f[0]] = (f'it writes main+{a:X} in .rodata, and the lift left no list of the bytes it reads as '
f'constants')
elif folded is not None:
hit = next((s for s, e in folded if s < b and a < e), None)
if hit is not None:
bad[f[0]] = (f'it writes main+{max(a, hit):X}, which the lifted code reads when the game is built '
f'(a jump table): the change would not reach it')
if not bad:
print(f'exefs: data writes checked against {"the bytes the lift read" if folded is not None else ".rodata"}: '
f'none meets them')
return
status = []
for line in args.status.read_text(encoding='utf-8').splitlines():
f = line.split('\t')
if len(f) >= 5 and f[0] in bad:
line = '\t'.join([f[0], f[1], 'unsupported', '-', bad[f[0]]])
print(f'exefs: unsupported file {f[0]} ({bad[f[0]]})')
status.append(line)
args.status.write_text('\n'.join(status) + '\n', encoding='utf-8')
kept = [line for line in lines if line.split('\t')[0] not in bad]
args.data.write_text('\n'.join(kept) + '\n', encoding='utf-8')
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--exefs', type=Path, required=True)
ap.add_argument('--mods', type=Path, help='the mods folder next to totk.exe')
ap.add_argument('--patches', type=Path, help='the patch file so far')
ap.add_argument('--out', type=Path, help='the patch file with the mods\' ExeFS patches added')
ap.add_argument('--status', type=Path, required=True)
ap.add_argument('--data', type=Path, help='the data writes of the mods\' patch files (runtime/mods.c reads them)')
ap.add_argument('--folded', type=Path, help='after the lift: <gen>/main/folded.txt; checks --status and --data '
'against it instead of building')
args = ap.parse_args()
sys.stdout.reconfigure(errors='replace')
if args.folded:
if not args.data:
ap.error('--folded needs --data')
recheck(args)
else:
if not (args.mods and args.patches and args.out):
ap.error('--mods, --patches and --out are needed to build')
build(args)
if __name__ == '__main__':