Files
totk-pc-recomp/tools/make_romfs_patches.py
T
Valiantsin DzerakhandClaude Opus 5.5 4a59088f04 Without an output_res the game makes the missing folders for the display's shape, so a prompts change on a wide display brings that shape back at the next start (#180)
The default output_res keeps the display's shape only while that shape has the
prompts' folder (cfg.c shape_keeps_prompts). After a switch to PlayStation
prompts on a 21:9 display the default was 1920x1080, so the start and the menu
asked only for the 16:9 folder and the game stayed 16:9. cfg_screen_size gives
the display's size while the player set no output_res.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:46:52 +03:00

646 lines
37 KiB
Python

# Adapted from MacNativeGames games/totk/scripts/make_romfs_patches.py @ ebe621d (MIT, Copyright (c) 2026
# MacNativeGames contributors).
"""Build the 60 fps RomFS overlay for TotK 1.1.0, 1.2.1 or 1.4.3 (runtime/romfs.c, TOTK_ROMFS_PATCHES).
UI/LayoutArchive/Common.Product.<110 or 140>.Nin_NX_NVN.blarc.zs: the weapon / shield / material quick menus capture the game
picture for their background when the panes C_CaptureBG_00, C_CaptureClearBG_00 and P_CaptureClearBG_00 become
visible at animation frame 1.0 (anim/ShortCutBG_00_In.bflan), AutoBuilder the same with C_CaptureBG_00
(anim/AutoBuilder_00_In.bflan). UI animations advance 0.5 frames per game frame at 60 fps, so the capture lands after
the world stopped drawing and the background can be black. The visibility keys move to frame 0.0: the live world is
captured at once (upstream checked weapon, shield, material and ability wheel at 60 fps, and 30 fps).
The game-over flow fix of the same upstream script is 1.0.0 only (1.1.0, 1.2.1 and 1.4.3 run DmF_SY_GameOver.110) and is
left out.
--aspect <w>x<h> (a game resolution of another shape than 16:9, such as 3440x1440 or 1280x800) also lays the UI out
for that shape (tools/ultrawide_layout.py): menus keep their 16:9 proportions in the middle, full-screen backgrounds
cover the screen and the HUD moves to the screen's corners (wider) or its top and bottom edges (narrower). totk.exe
loads that overlay from romfs-patches-<w>x<h> when the game resolution is that size (runtime/romfs.c overlay_init);
the camera's side of it is the patch set aspect (recomp/patches-v121.txt, patches-v143.txt).
--hud 21:9 | 16:9 (with --aspect, a screen wider than that; default full) keeps the HUD within the middle 21:9 or 16:9 of
the screen instead of its corners (totk.ini hud_width). The folder is romfs-patches-<w>x<h>-hud21x9 or -hud16x9, which
runtime/romfs.c overlay_init loads in place of romfs-patches-<w>x<h> when hud_width asks for it.
--prompts xbox | xbox-label: the same overlay with Xbox button icons in the game's prompts, for prompts = xbox
(runtime/romfs.c overlay_init loads romfs-patches-xbox or romfs-patches-xbox-label in place of romfs-patches). Drawn
from the game's own files on the player's machine, nothing added (tools/button_icons.py, which needs fontTools):
- both: L, R, ZL, ZR, Plus, Minus and the d-pad become LB, RB, LT, RT, Menu, View and a cross in the button font
(scft/External.bfotf, the same font in each Font/Font*.Nin_NX_NVN.bfarc.zs), and L and ZL in the button guide's images
(Nt_KeyTexL_00 and Nt_KeyTexZL_00 of timg/__Combined.bntx in the Common archive). The fonts grow, so the overlay also
carries the resource size table (System/Resource/ResourceSizeTable.Product.<n>...rsizetable.zs) with their entries
raised by as much.
- xbox, for pad_layout = nintendo. That layout plays an Xbox pad by position: the button that does Switch A is
labelled B, so the face buttons' prompts are swapped to name the labels, A with B and X with Y:
- the action prompt (blyt/DoCommand_00.bflyt) draws the letter images Nt_KeyTexA/B/X/Y_00 of timg/__Combined.bntx in
the Common archive: their image data is swapped (same format, size and tiling). Around it, a diamond names the
other face buttons with one-letter text panes (T_B_00, T_X_00, ...). The game fills them from the messages of the
same name in LayoutMsg/DoCommand_00.msbt of each Mals/<language> archive, over the layout's own text: both have
their letters swapped the same way. The diamond's shape (Nt_KeyBtnSubBase*) shows the button's position, which
stays right.
- every other prompt: the glyphs U+E040 A, U+E041 B, U+E042 X, U+E043 Y of the button font trade places.
- xbox-label, for pad_layout = xbox and auto. Those layouts put Switch A on the button labelled A (auto picks xbox for
every pad not labelled the Nintendo way), so the face buttons keep their letters. The action prompt's diamond is
laid out as an Xbox pad's (bflyt_xbox_diamond): A at the bottom, B on the right, X on the left, Y at the top, the
button the prompt names lit in its place with its bar.
--prompts playstation | playstation-xbox: the same overlay with PlayStation button icons in the game's prompts, for
prompts = playstation (romfs-patches-playstation for pad_layout = nintendo, romfs-patches-playstation-xbox for xbox and
auto). The face buttons become the circle, cross, triangle and square shapes (button_icons.PS_FACE: with nintendo A is
the circle and B the cross, as the buttons sit; with xbox and auto A is the cross and B the circle), the shoulders and
triggers L1 R1 L2 R2, Plus and Minus Options and Share, and the d-pad keeps its cross. The game's text fonts get the
four shapes, so the diamond's letters become those characters; playstation-xbox lays the diamond out as xbox-label
does, since the face buttons sit where an Xbox pad's do. Drawn from the game's own files like the Xbox prompts
(tools/button_icons.py).
--xbox-prompts is the older name of --prompts xbox. With --aspect, romfs-patches-<style>-<w>x<h>: the same prompts with
that shape's UI layout.
--prompts pc: keyboard and mouse prompts, for prompts = pc (romfs-patches-pc, or romfs-patches-pc-<w>x<h> with
--aspect): every button of the button font is drawn as the key or mouse button the keyboard defaults put on it
(button_icons.pc_icons; --binds <totk.ini> takes the first key of its bind_ lines instead), and so are the button
guide's images (A, B, X, Y, L, ZL and the stick's two). In the font Y is T, the key that presses it next to the left
mouse button, which is A in the game's menus (button_icons.PC_TEXT_KEYS); the button guide keeps the left button. The
one-letter panes around the action prompt name the keys with one character of the Normal font (E, an arrow for Shift,
a bar for Space, L and R for the mouse buttons).
--reuse <folder>: an overlay made before, such as romfs-patches-pc for romfs-patches-pc-<w>x<h>. Each file of it whose
contents match the new one is hard-linked (copied where links fail) instead of compressed again; the summary's reused
lists them. The output is byte for byte the same as without it.
Input: build/extracted/<version>/romfs (romfs.bin and index.json, verified by tools/extract.py). Output:
build/<version>/romfs-patches/<path> by default (romfs-patches-<style> with --prompts, -<w>x<h> with --aspect,
and -hud<a>x<b> with --hud), and <out>/manifest.tsv (path, size, SHA-256 of the overlay, SHA-256 of the original). Run with .venv-py314.
The files go into <out>.part-<pid> beside <out>, which is renamed to <out> once complete (publish), so a folder is
never seen half written and two runs for one folder do not mix their files. Without --background an older <out> is
removed first (the setup and a source build replace it). --background is the game's (runtime/romfs.c
romfs_folders_prepare, for a folder the settings need and the install lacks): it runs at a lower priority, does
nothing when <out> has a manifest at its start, and drops its own result when another run finished <out> first, so a
folder a running game loaded (whose files totk.exe checks again on every open) is never written into. A .part folder
left by a run that was stopped is removed by the next run for that folder once it is an hour old.
"""
import argparse
import compression.zstd as zstd
import ctypes
import glob
import hashlib
import importlib.util
import json
import os
import re
import shutil
import struct
import time
from pathlib import Path
root = Path(__file__).resolve().parents[1]
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--version', choices=('v110', 'v121', 'v143'), default='v121')
ap.add_argument('--romfs', type=Path, help='default build/extracted/<version>/romfs')
ap.add_argument('--out', type=Path,
help='default build/<version>/romfs-patches[-<style>][-<w>x<h>][-hud<a>x<b>]')
ap.add_argument('--aspect', help='<w>x<h> of a game resolution of another shape than 16:9: also lay the UI out for it')
ap.add_argument('--hud', choices=('full', '21:9', '16:9'), default='full',
help='with --aspect on a wider screen: the HUD stays within the middle 21:9 or 16:9 (default full: the '
'screen\'s corners); the default --out gets a -hud21x9 or -hud16x9 suffix')
ap.add_argument('--prompts', choices=('xbox', 'xbox-label', 'playstation', 'playstation-xbox', 'pc'),
help='Xbox or PlayStation button icons in the prompts: xbox for pad_layout = nintendo (A shown as B, '
'X as Y), xbox-label for pad_layout = xbox or auto (the face buttons keep their letters), '
'playstation for the PlayStation shapes on the Nintendo layout and playstation-xbox for xbox and '
'auto, pc for the keyboard and mouse')
ap.add_argument('--binds', type=Path, help='--prompts pc: a totk.ini whose bind_ lines the key icons follow')
ap.add_argument('--xbox-prompts', dest='prompts', action='store_const', const='xbox', help='the same as --prompts xbox')
ap.add_argument('--reuse', type=Path,
help='an overlay folder made before: its files with the same contents are linked, not compressed again')
ap.add_argument('--background', action='store_true',
help='started by the game: lower priority, and an <out> that has a manifest is kept, not replaced')
args = ap.parse_args()
args.romfs = args.romfs or root / 'build/extracted' / args.version / 'romfs'
width = height = None
if args.aspect:
match = re.fullmatch(r'([1-9][0-9]*)x([1-9][0-9]*)', args.aspect.lower())
if not match: ap.error('--aspect must be positive integer dimensions <w>x<h>')
width, height = map(int, match.groups())
if width * 9 == height * 16: ap.error('--aspect must have another shape than 16:9')
args.aspect = f'{width}x{height}'
HUD_CAP = {'full': None, '21:9': 21 / 9, '16:9': 16 / 9}[args.hud]
if HUD_CAP is not None:
if not args.aspect: ap.error('--hud needs --aspect')
if HUD_CAP >= width / height * 0.99:
ap.error(f'--hud {args.hud} is not narrower than a {width}x{height} screen (that is --hud full)')
if not args.out:
# Match runtime/romfs.c overlay_init, preserving the default 16:9 overlay.
name = f'romfs-patches-{args.prompts}' if args.prompts else 'romfs-patches'
if args.aspect: name += '-' + args.aspect
if args.hud != 'full': name += '-hud' + args.hud.replace(':', 'x')
args.out = root / 'build' / args.version / name
PART_STALE_S = 3600 # a .part folder untouched this long belongs to a run that was stopped
def finished(folder):
return (folder / 'manifest.tsv').is_file()
def newest_mtime(folder):
times = [folder.stat().st_mtime]
for path in folder.rglob('*'):
try:
times.append(path.stat().st_mtime)
except OSError:
pass
return max(times)
def remove_stale_parts(out):
"""Remove the <out>.part-* folders of runs that were stopped (none of their files changed for PART_STALE_S)."""
for part in out.parent.glob(glob.escape(out.name) + '.part-*'):
try:
if part.is_dir() and time.time() - newest_mtime(part) > PART_STALE_S:
shutil.rmtree(part, ignore_errors=True)
except OSError:
pass
def lower_priority():
"""Below-normal priority, so the game that started this run keeps its cores."""
try:
if os.name == 'nt':
kernel32 = ctypes.windll.kernel32
kernel32.SetPriorityClass(kernel32.GetCurrentProcess(), 0x4000) # BELOW_NORMAL_PRIORITY_CLASS
else:
os.nice(10)
except (OSError, AttributeError):
pass
if args.background:
if finished(args.out): # another run made it since the game looked
print(json.dumps(dict(status='pass', kept=str(args.out))))
raise SystemExit(0)
lower_priority()
args.out.parent.mkdir(parents=True, exist_ok=True)
remove_stale_parts(args.out)
part = args.out.with_name(f'{args.out.name}.part-{os.getpid()}') # the files go here first (publish)
shutil.rmtree(part, ignore_errors=True)
index = json.loads((args.romfs / 'index.json').read_text())
assert index['status'] == 'pass'
files = {f['path']: f for f in index['files']}
def sarc_nodes(data):
"""(name, node offset, absolute start, end) of every SARC member, in SFAT order."""
assert data[:4] == b'SARC' and data[6:8] == b'\xff\xfe'
header, size, start = struct.unpack_from('<HxxII', data, 4)
assert size == len(data) and data[header:header + 4] == b'SFAT'
count = struct.unpack_from('<H', data, header + 6)[0]
names = header + 12 + count * 16
assert data[names:names + 4] == b'SFNT'
names += 8
out = []
for i in range(count):
node = header + 12 + i * 16
_, attribute, lo, hi = struct.unpack_from('<4I', data, node)
offset = names + (attribute & 0xffffff) * 4
out.append((data[offset:data.index(0, offset, start)].decode(), node, start + lo, start + hi))
return out
def sarc_members(data):
"""name -> (absolute start, end) of every SARC member."""
return {name: (lo, hi) for name, _, lo, hi in sarc_nodes(data)}
def flvi_keys(data, base):
"""(pane name, absolute key offset, keys) of every FLVI (visibility) step track in the BFLAN at data[base:]."""
assert data[base:base + 4] == b'FLAN' and data[base + 4:base + 6] == b'\xff\xfe'
pos, out = base + 0x14, []
for _ in range(struct.unpack_from('<H', data, base + 0x10)[0]):
magic, size = data[pos:pos + 4], struct.unpack_from('<I', data, pos + 4)[0]
if magic == b'pai1':
count, table = struct.unpack_from('<H', data, pos + 14)[0], pos + struct.unpack_from('<I', data, pos + 16)[0]
for e in range(count):
eo = pos + struct.unpack_from('<I', data, table + 4 * e)[0]
name = data[eo:eo + 28].split(b'\0')[0].decode()
for t in range(data[eo + 28]):
to = eo + struct.unpack_from('<I', data, eo + 32 + 4 * t)[0]
if data[to:to + 4] != b'FLVI':
continue
for g in range(data[to + 4]):
go = to + struct.unpack_from('<I', data, to + 8 + 4 * g)[0]
assert data[go + 2] == 1, 'visibility tracks are step curves'
ko = go + struct.unpack_from('<I', data, go + 8)[0]
keys = [struct.unpack_from('<fH', data, ko + 8 * k)
for k in range(struct.unpack_from('<H', data, go + 4)[0])]
out.append((name, ko, keys))
pos += size
return out
def bntx_swap(data, base, pairs):
"""Swap the image data of each (name, name) pair of textures in the BNTX at data[base:] (names without the ^
suffix). Both of a pair must have the same header from flags to layout (format, size, tiling, mips) and image
size, so the bytes of one are a valid image of the other."""
assert data[base:base + 4] == b'BNTX' and data[base + 0x20:base + 0x24] == b'NX '
count, table = struct.unpack_from('<IQ', data, base + 0x24)
found = {}
for k in range(count):
info = base + struct.unpack_from('<Q', data, base + table + 8 * k)[0]
assert data[info:info + 4] == b'BRTI'
name = base + struct.unpack_from('<Q', data, info + 0x60)[0]
name = data[name + 2:name + 2 + struct.unpack_from('<H', data, name)[0]].decode().split('^')[0]
mips = base + struct.unpack_from('<Q', data, info + 0x70)[0]
shape = bytes(data[info + 0x10:info + 0x3c]) + bytes(data[info + 0x50:info + 0x54])
start, size = base + struct.unpack_from('<Q', data, mips)[0], struct.unpack_from('<I', data, info + 0x50)[0]
found[name] = (shape, start, size)
for a, b in pairs:
assert a in found and b in found, f'{a} or {b} missing from the texture list'
(shape_a, at_a, size), (shape_b, at_b, _) = found[a], found[b]
assert shape_a == shape_b, f'{a} and {b} differ in format, size or tiling'
data[at_a:at_a + size], data[at_b:at_b + size] = data[at_b:at_b + size], data[at_a:at_a + size]
def bflyt_letters(data, base, end, swap):
"""Swap the text of the one-letter text panes T_<letter>_<n> (A, B, X or Y) in the BFLYT at data[base:end] by
swap (text -> text, one UTF-16 code unit: a letter or a PlayStation shape); the names of the panes changed. Same
length (one character and its terminator)."""
assert data[base:base + 4] == b'FLYT'
pos, changed = base + struct.unpack_from('<H', data, base + 6)[0], []
while pos + 8 <= end:
magic, size = data[pos:pos + 4], struct.unpack_from('<I', data, pos + 4)[0]
if magic == b'txt1':
name = data[pos + 12:pos + 36].split(b'\0')[0].decode()
length = struct.unpack_from('<H', data, pos + 0x56)[0]
at = pos + struct.unpack_from('<I', data, pos + 0x64)[0]
letter = data[at:at + 2].decode('utf-16-le') if length == 4 and at > pos else ''
if letter in swap and re.fullmatch(r'T_[ABXY]_\d+', name):
data[at:at + 2] = swap[letter].encode('utf-16-le')
changed.append(name)
if not size:
break
pos += size
return changed
def bflyt_panes(data, base, end):
"""name -> absolute offset of each pane section (pan1, pic1, txt1, wnd1, ...) of the BFLYT at data[base:end], and
name -> absolute offset of each material of its mat1, and its texture names (txl1)."""
assert data[base:base + 4] == b'FLYT'
pos, panes, mats, textures = base + struct.unpack_from('<H', data, base + 6)[0], {}, {}, []
while pos + 8 <= end:
magic, size = data[pos:pos + 4], struct.unpack_from('<I', data, pos + 4)[0]
if magic in (b'pan1', b'pic1', b'txt1', b'wnd1', b'bnd1', b'prt1', b'ali1', b'scr1'):
panes[data[pos + 12:pos + 36].split(b'\0')[0].decode()] = pos
elif magic == b'mat1':
for i in range(struct.unpack_from('<H', data, pos + 8)[0]):
at = pos + struct.unpack_from('<I', data, pos + 12 + 4 * i)[0]
mats[data[at:at + 28].split(b'\0')[0].decode()] = at
elif magic == b'txl1':
for i in range(struct.unpack_from('<H', data, pos + 8)[0]):
at = pos + 12 + struct.unpack_from('<I', data, pos + 12 + 4 * i)[0]
textures.append(data[at:data.index(0, at)].decode())
if not size:
break
pos += size
return panes, mats, textures
# The action prompt's diamond (blyt/DoCommand_00.bflyt) for pad_layout = xbox and auto, where Switch A, B, X and Y are
# the buttons labelled so on an Xbox pad: A at the bottom, B on the right, X on the left, Y at the top. The layout has
# one diamond per button the prompt can name (N_A_00, N_B_00, N_X_00, N_Y_00; N_StopA_00 is A's while held), each
# with its button and bar (N_Btn<K>_00), the three other letters (T_<letter>_<n>) and a base image (P_Base<K>_00)
# that lights the button's side. A Switch button's place on an Xbox pad is that of its partner on a Switch (A of B,
# X of Y), so each piece takes the place and look its partner's piece has: the bar of a button on the left reaches
# left (Y's on a Switch), the others reach right.
DIAMOND_SLOTS = {'A': (39.0, 1.0), 'B': (2.0, -39.0), 'X': (1.0, 39.0), 'Y': (-38.0, 0.0)} # the letters' places
DIAMOND_PARTNER = {'A': 'B', 'B': 'A', 'X': 'Y', 'Y': 'X'}
# (pane, the pane whose place it takes): the buttons with their bars, and the flashes of a repeated press (Rush)
DIAMOND_MOVES = [('N_BtnA_00', 'N_BtnB_00'), ('N_BtnA_01', 'N_BtnB_00'), ('N_CompanionBtnA_00', 'N_BtnB_00'),
('N_BtnB_00', 'N_BtnA_00'), ('N_BtnX_00', 'N_BtnY_00'), ('N_BtnY_00', 'N_BtnX_00')]
DIAMOND_MOVES += [(f'P_Capture{k}_{n:02}', f'P_Capture{DIAMOND_PARTNER[k]}_{n:02}') for k in 'ABXY' for n in (1, 2, 3)]
# X and Y trade the side their bar reaches to: the flash, the bar (Y's mirrored) and the text (Y's right-aligned)
DIAMOND_MIRROR = [('P_CaptureX_00', 'P_CaptureY_00'), ('W_BaseX_00', 'W_BaseY_00'), ('T_GuideX_00', 'T_GuideY_00'),
('T_GuideX_TW', 'T_GuideY_TW')]
# the base image: its texture (by its material) and its rotation and scale
DIAMOND_BASES = [('P_BaseA_00', 'P_BaseB_00'), ('P_BaseA_01', 'P_BaseB_00'), ('P_BaseB_00', 'P_BaseA_00'),
('P_BaseX_00', 'P_BaseY_00'), ('P_BaseY_00', 'P_BaseX_00')]
def bflyt_xbox_diamond(data, base, end):
"""Lay the action prompt's diamond in the BFLYT at data[base:end] (blyt/DoCommand_00.bflyt) out as an Xbox pad's
face buttons (DIAMOND_* above); returns the panes changed. A pane section holds its origin at +0x09, translation
(3 floats) at +0x2C, rotation (3) at +0x38 and scale (2) at +0x44; a text pane its text position at +0x5C; a
picture pane its material index at +0x64, and a material its first texture's index at +0x2C."""
panes, mats, textures = bflyt_panes(data, base, end)
old = bytes(data) # every piece takes its partner's place as the game has it
changed = []
for name, at in panes.items():
m = re.fullmatch(r'T_([ABXY])_\d+', name)
if m: # a letter around a diamond: its partner's place
struct.pack_into('<2f', data, at + 0x2C, *DIAMOND_SLOTS[DIAMOND_PARTNER[m[1]]])
changed.append(name)
for name, like in DIAMOND_MOVES:
data[panes[name] + 0x2C:panes[name] + 0x34] = old[panes[like] + 0x2C:panes[like] + 0x34]
changed.append(name)
# A's hold flashes (N_LoopA_03) sit 129 right of the diamond, as A's repeated-press ones: the same move as those
dx, dy = (a - b for a, b in zip(struct.unpack_from('<2f', old, panes['P_CaptureB_01'] + 0x2C),
struct.unpack_from('<2f', old, panes['P_CaptureA_01'] + 0x2C)))
x, y = struct.unpack_from('<2f', old, panes['N_LoopA_03'] + 0x2C)
struct.pack_into('<2f', data, panes['N_LoopA_03'] + 0x2C, x + dx, y + dy)
changed.append('N_LoopA_03')
for a, b in DIAMOND_MIRROR:
pa, pb = panes[a], panes[b]
for lo, hi in ((0x09, 0x0A), (0x2C, 0x4C)) + (((0x5C, 0x5D),) if old[pa:pa + 4] == b'txt1' else ()):
data[pa + lo:pa + hi], data[pb + lo:pb + hi] = old[pb + lo:pb + hi], old[pa + lo:pa + hi]
changed += [a, b]
in_order = list(mats.values())
def material(name): # the offset of a picture pane's material
return in_order[struct.unpack_from('<H', old, panes[name] + 0x64)[0]]
for name, like in DIAMOND_BASES:
assert old[panes[name]:panes[name] + 4] == b'pic1', name
data[panes[name] + 0x38:panes[name] + 0x4C] = old[panes[like] + 0x38:panes[like] + 0x4C]
mine, theirs = material(name), material(like)
data[mine + 0x2C:mine + 0x2E] = old[theirs + 0x2C:theirs + 0x2E]
assert textures[struct.unpack_from('<H', data, mine + 0x2C)[0]].startswith('Nt_KeyBtnSubBase'), name
changed.append(name)
return changed
def msbt_letters(data, swap):
"""Swap the one-letter messages labelled T_<letter>_<n> (A, B, X or Y) in the UTF-16 MSBT data (a bytearray) by
swap (text -> text, one UTF-16 code unit: a letter or a PlayStation shape); the labels changed. The game sets a
layout's text panes from the messages of the same name in Mals/<language>/LayoutMsg/<layout>.msbt, over the text
stored in the layout."""
assert data[:8] == b'MsgStdBn' and data[0x0c] == 1, 'not a UTF-16 MSBT'
sections, pos = {}, 0x20
for _ in range(struct.unpack_from('<H', data, 0x0e)[0]):
size = struct.unpack_from('<I', data, pos + 4)[0]
sections[bytes(data[pos:pos + 4])] = (pos + 16, size)
pos = (pos + 16 + size + 15) & ~15
labels, (base, _) = {}, sections[b'LBL1']
for g in range(struct.unpack_from('<I', data, base)[0]):
count, at = struct.unpack_from('<II', data, base + 4 + 8 * g)
at += base
for _ in range(count):
length = data[at]
labels[struct.unpack_from('<I', data, at + 1 + length)[0]] = data[at + 1:at + 1 + length].decode()
at += 1 + length + 4
(base, size), changed = sections[b'TXT2'], []
count = struct.unpack_from('<I', data, base)[0]
offsets = list(struct.unpack_from(f'<{count}I', data, base + 4)) + [size]
for i in range(count):
at, length = base + offsets[i], offsets[i + 1] - offsets[i]
letter = data[at:at + 2].decode('utf-16-le') if length == 4 else ''
if letter in swap and re.fullmatch(r'T_[ABXY]_\d+', labels.get(i, '')):
data[at:at + 2] = swap[letter].encode('utf-16-le')
changed.append(labels[i])
return changed
def sarc_replace(data, new):
"""The SARC data with the members named in new (name -> bytes) replaced. The header, SFAT and SFNT stay; the
members are laid out again in their old order, each at an offset with at least the alignment its old one had."""
start = struct.unpack_from('<I', data, 12)[0]
out = bytearray(data[:start])
for name, node, lo, hi in sorted(sarc_nodes(data), key=lambda n: n[2]):
align = min(lo & -lo, 0x2000) if lo else 0x2000
out += bytes(-len(out) % align)
body = new.get(name, data[lo:hi])
struct.pack_into('<II', out, node + 8, len(out) - start, len(out) - start + len(body))
out += body
struct.pack_into('<I', out, 8, len(out))
return bytes(out)
COMMON = {'v110': '110', 'v121': '110', 'v143': '140'} # the version in the name of each game's Common archive
ARCHIVE = f'UI/LayoutArchive/Common.Product.{COMMON[args.version]}.Nin_NX_NVN.blarc.zs'
MEMBERS = {
'anim/ShortCutBG_00_In.bflan': ('C_CaptureBG_00', 'C_CaptureClearBG_00', 'P_CaptureClearBG_00'),
'anim/AutoBuilder_00_In.bflan': ('C_CaptureBG_00',),
}
KEY_TEXTURES = (('Nt_KeyTexA_00', 'Nt_KeyTexB_00'), ('Nt_KeyTexX_00', 'Nt_KeyTexY_00'))
KEY_GLYPHS = ((0xE040, 0xE041), (0xE042, 0xE043)) # A B, X Y in scft/External.bfotf
KEY_LETTERS = {'A': 'B', 'B': 'A', 'X': 'Y', 'Y': 'X'} # the face-button diamond around the action prompt
FONTS = sorted(p for p in files if re.fullmatch(r'Font/Font(_\w+)?\.Nin_NX_NVN\.bfarc\.zs', p))
# The message archive of each language: the newest version in the RomFS (Mals/USen.Product.121.sarc.zs on 1.2.1).
MALS = {}
for p in files:
m = re.fullmatch(r'Mals/(\w+)\.Product\.(\d+)\.sarc\.zs', p)
if m and int(m[2]) > MALS.get(m[1], (0, ''))[0]:
MALS[m[1]] = (int(m[2]), p)
MALS = sorted(p for _, p in MALS.values())
LAYOUT_MSG = 'LayoutMsg/DoCommand_00.msbt'
# The resource size table (System/Resource/ResourceSizeTable.Product.<n>...rsizetable.zs, matched as loosely as
# MacNativeGames make_button_icons.py does), likewise the newest version in the RomFS.
RESTBL = max((p for p in files if p.startswith('System/Resource/ResourceSizeTable.') and p.endswith('.rsizetable.zs')),
key=lambda p: int((re.search(r'\.(\d+)\.', p) or [0, 0])[1]), default=None)
by_position = args.prompts == 'xbox' # the face buttons' prompts swapped to the labels (pad_layout = nintendo)
pc = args.prompts == 'pc'
ps = args.prompts in ('playstation', 'playstation-xbox')
ps_layout = 'nintendo' if args.prompts == 'playstation' else 'xbox'
letter_panes = by_position or pc or ps # the one-letter panes around the action prompt change
if args.prompts:
assert FONTS, 'no Font/Font*.Nin_NX_NVN.bfarc.zs in the RomFS'
assert MALS or not letter_panes, 'no Mals/<language>.Product.<version>.sarc.zs in the RomFS'
assert RESTBL, 'no System/Resource/ResourceSizeTable.Product.<n>...rsizetable.zs in the RomFS'
spec = importlib.util.spec_from_file_location('button_icons', Path(__file__).with_name('button_icons.py'))
icons = importlib.util.module_from_spec(spec)
spec.loader.exec_module(icons)
assert icons.TEXT_ARCHIVE in FONTS, f'no {icons.TEXT_ARCHIVE} in the RomFS'
with (args.romfs / 'romfs.bin').open('rb') as image:
def read(path):
f = files[path]
image.seek(f['offset'])
b = image.read(f['size'])
assert len(b) == f['size']
return b
dictionaries = {}
pack = zstd.decompress(read('Pack/ZsDic.pack.zs'))
for name, (lo, hi) in sarc_members(pack).items():
d = zstd.ZstdDict(pack[lo:hi])
dictionaries[d.dict_id] = d
prompt_files = (FONTS + (MALS if letter_panes else []) + [RESTBL]) if args.prompts else []
originals = {path: read(path) for path in [ARCHIVE] + prompt_files}
raw = originals[ARCHIVE]
info = zstd.get_frame_info(raw)
dictionary = dictionaries[info.dictionary_id]
original = zstd.decompress(raw, zstd_dict=dictionary)
blarc = bytearray(original)
assert len(blarc) == info.decompressed_size
members = sarc_members(blarc)
patched = {}
for member, panes in MEMBERS.items():
lo, hi = members[member]
changed = []
for name, ko, keys in flvi_keys(blarc, lo):
if name not in panes:
continue
assert [(f, v) for f, v in keys] == [(0.0, 0), (1.0, 1)], (member, name, keys)
struct.pack_into('<f', blarc, ko + 8, 0.0)
changed.append(name)
assert sorted(changed) == sorted(panes), (member, changed)
patched[member] = changed
layout = []
if args.aspect:
spec = importlib.util.spec_from_file_location('ultrawide_layout', Path(__file__).with_name('ultrawide_layout.py'))
ultrawide = importlib.util.module_from_spec(spec)
spec.loader.exec_module(ultrawide)
layout = ultrawide.apply(blarc, members, width, height, HUD_CAP)
diff = [i for i in range(len(original)) if original[i] != blarc[i]]
touched = [members[m] for m in (*MEMBERS, *layout)]
assert diff and all(any(lo <= i < hi for lo, hi in touched) for i in diff), diff
outputs = {ARCHIVE: bytes(blarc)} # RomFS path -> new contents (before compression)
prompts = None
if args.prompts:
sizes = {ARCHIVE: len(original)} # RomFS path -> size of the original's contents
def unpack(path):
frame = zstd.get_frame_info(originals[path])
data = zstd.decompress(originals[path], zstd_dict=dictionaries.get(frame.dictionary_id))
sizes[path] = len(data)
return data
archives = {path: unpack(path) for path in FONTS}
text = sarc_members(archives[icons.TEXT_ARCHIVE])
missing = [name for name in icons.TEXT_FONTS if name not in text]
assert not missing, f'no {", ".join(missing)} in {icons.TEXT_ARCHIVE}'
glyphs = icons.text_glyphs(archives[icons.TEXT_ARCHIVE][slice(*text[name])] for name in icons.TEXT_FONTS)
advances = icons.font_advances(archives[icons.TEXT_ARCHIVE][slice(*text[icons.FONT])])
if pc:
ini = args.binds.read_text(encoding='utf-8', errors='replace') if args.binds else None
binds = icons.pc_binds(ini)
drawn = icons.pc_icons(glyphs, advances, binds)
# The text prompts show Y as T (button_icons.PC_TEXT_KEYS); the button guide's images keep the left click
font_drawn = icons.pc_icons(glyphs, advances, icons.pc_binds(ini, text=True))
pane_letters = {letter: icons.pc_letter(binds[letter]) for letter in 'ABXY'}
elif ps:
drawn = icons.playstation_icons(glyphs, advances, ps_layout)
for shape in set(icons.PS_FACE[ps_layout].values()):
glyphs.outline(shape) # the diamond's text needs the shape in the game's text fonts
font_drawn = drawn
pane_letters = icons.PS_FACE[ps_layout]
else:
drawn = font_drawn = icons.xbox_icons(glyphs)
pane_letters = KEY_LETTERS
bntx = members['timg/__Combined.bntx'][0]
letters = {}
if by_position:
bntx_swap(blarc, bntx, KEY_TEXTURES)
diamond = []
if args.prompts in ('xbox-label', 'playstation-xbox'):
diamond = bflyt_xbox_diamond(blarc, *members['blyt/DoCommand_00.bflyt'])
if letter_panes:
for member, (lo, hi) in members.items():
if member.endswith('.bflyt'):
changed = bflyt_letters(blarc, lo, hi, pane_letters)
if changed:
letters[member] = changed
assert 'blyt/DoCommand_00.bflyt' in letters, 'no letter panes in blyt/DoCommand_00.bflyt'
hud = icons.bntx_redraw(blarc, bntx, icons.PC_HUD_TEXTURES if pc else icons.PS_HUD_TEXTURES if ps else icons.HUD_TEXTURES,
drawn)
outputs[ARCHIVE] = bytes(blarc)
fonts = {} # each distinct font is drawn once (1.4.3's four archives hold the same one)
for path, bfarc in archives.items():
lo, hi = sarc_members(bfarc)[icons.FONT]
otf, key = icons.bfotf_decode(bfarc[lo:hi])
if otf not in fonts:
fonts[otf] = icons.redraw_font(otf, font_drawn, KEY_GLYPHS if by_position else ())
outputs[path] = sarc_replace(bfarc, {icons.FONT: icons.bfotf_encode(fonts[otf], key, bfarc[lo:hi])})
messages = {}
for path in MALS if letter_panes else []:
mals = bytearray(unpack(path))
lo, hi = sarc_members(mals)[LAYOUT_MSG]
msbt = bytearray(mals[lo:hi])
changed = msbt_letters(msbt, pane_letters)
assert len(changed) == sum(len(v) for v in letters.values()), (path, changed)
mals[lo:hi] = msbt
outputs[path] = bytes(mals)
messages[path] = len(changed)
# An archive that grew needs as much more room in the resource size table, or the game stops when it loads it.
grown = {path: len(data) - sizes[path] for path, data in outputs.items() if len(data) > sizes[path]}
if grown:
outputs[RESTBL] = icons.grow_restbl(unpack(RESTBL), grown)
prompts = dict(style=args.prompts, textures=KEY_TEXTURES if by_position else [], letters=letters,
diamond=len(diamond), messages=messages, hud=hud, icons=len(drawn), fonts=FONTS, grown=grown)
def reusable(path, contents, dictionary):
"""The --reuse folder's copy of path when it holds these contents under the same dictionary, else None. The level 19
compression is most of a run (about 80 s for the four fonts of 1.4.3), and romfs-patches-<style>-<w>x<h> differs
from romfs-patches-<style> in the Common archive alone, so the setup makes the fifteen shapes from the 16:9
folder."""
if not args.reuse or not (args.reuse / path).is_file():
return None
data = (args.reuse / path).read_bytes()
try:
same = (zstd.get_frame_info(data).dictionary_id == zstd.get_frame_info(originals[path]).dictionary_id
and zstd.decompress(data, zstd_dict=dictionary) == contents)
except zstd.ZstdError:
return None
return data if same else None
def publish(part, out, sizes, keep_existing):
"""Rename the complete folder part to out. Each file the manifest lists must be there with its size first: a
setup's clean-up that removed part's files under a running generator must not leave a manifest without them.
keep_existing (--background): an out that has a manifest by now stays and part is dropped; an out without one is
an unfinished folder of an older generator, which nothing loads. Without it out is replaced. Returns whether part
became out."""
for path, size in sizes.items():
assert (part / path).is_file() and (part / path).stat().st_size == size, f'{part / path} is missing or changed'
for attempt in range(3):
if keep_existing and finished(out):
shutil.rmtree(part, ignore_errors=True)
return False
if out.exists() and not keep_existing:
shutil.rmtree(out)
elif out.exists():
# Moved aside, never removed in place: another run may publish between the check and this rename, and a
# folder moved aside with a manifest goes back (a removed one could be the one a game just loaded).
stale = out.with_name(f'{out.name}.part-stale-{os.getpid()}')
try:
os.rename(out, stale)
except OSError:
pass
else:
if finished(stale):
os.rename(stale, out)
continue
shutil.rmtree(stale, ignore_errors=True)
try:
os.rename(part, out)
return True
except OSError: # another run renamed its folder into place after the check: the next pass looks again
if attempt == 2:
raise
shutil.rmtree(part, ignore_errors=True)
return False
lines, reused, sizes = [], [], {}
try:
for path, contents in outputs.items():
raw = originals[path]
dictionary = dictionaries.get(zstd.get_frame_info(raw).dictionary_id)
target = part / path
target.parent.mkdir(parents=True, exist_ok=True)
data = reusable(path, contents, dictionary)
if data is not None:
try: # a hard link takes no room; a volume without them gets a copy
os.link(args.reuse / path, target)
except OSError:
target.write_bytes(data)
reused.append(path)
else:
data = zstd.compress(contents, level=19, zstd_dict=dictionary)
assert zstd.decompress(data, zstd_dict=dictionary) == contents
assert zstd.get_frame_info(data).dictionary_id == zstd.get_frame_info(raw).dictionary_id
target.write_bytes(data)
sizes[path] = len(data)
lines.append('\t'.join([path, str(len(data)), hashlib.sha256(data).hexdigest(), hashlib.sha256(raw).hexdigest()]))
(part / 'manifest.tsv').write_text('# path\tsize\tsha256\toriginal_sha256 (tools/make_romfs_patches.py)\n'
+ ''.join(line + '\n' for line in lines))
published = publish(part, args.out, sizes, args.background)
except BaseException:
shutil.rmtree(part, ignore_errors=True)
raise
print(json.dumps(dict(status='pass', path=ARCHIVE, members=patched, changed_bytes=len(diff),
ultrawide_layout=dict(aspect=args.aspect, hud=args.hud, members=len(layout)) if args.aspect else None,
prompts=prompts, reused=reused, **({} if published else {'kept': str(args.out)}))))