mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-19 04:15:57 +00:00
Merge dev into task/pr1263
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
e031ac596a
50 files changed
+3918
-315
No files matched your search
@@ -111,6 +111,15 @@ of them.
|
||||
$env:SCCACHE = "path\to\sccache.exe"
|
||||
.venv\Scripts\python tools\build_game.py --version <ver>
|
||||
```
|
||||
|
||||
A build into another `--out` (a task build, or a private lift of a patch change with `--gen generated\<ver>-<tag>`)
|
||||
hardlinks the objects of the main build `build\<ver>` and compiles only `runtime/` and the generated files whose
|
||||
compile differs. An object is reused when the main `build.ninja` has the same compiler, flags and rule, the main
|
||||
build is up to date, and every file its compile read according to the main Ninja deps log (the `.c`, the `runtime/`
|
||||
and generated headers, the system headers) has the same bytes in this build's trees and has not changed since the
|
||||
object was written, and neither has the compiler. A lift that changes one patch site then compiles one or two chunks
|
||||
instead of all of them. The build prints `reused N of M generated objects`; `--verbose` lists the others with the
|
||||
reason, and `--full-build` compiles everything.
|
||||
`--patches` builds in the frame-rate, resolution and other patch sets, switchable at run time. The link copies
|
||||
`SDL3.dll`, `libiconv-2.dll` and `libopus-0.dll` next to the exe.
|
||||
For Xbox button prompts, run `make_romfs_patches.py` twice more, with `--prompts xbox` and `--prompts xbox-label`:
|
||||
|
||||
+63
-39
@@ -1,8 +1,9 @@
|
||||
# Installer
|
||||
|
||||
A public release that turns the player's own game files into a playable TotK folder. The release carries no game
|
||||
data and no code derived from the game: the installer builds the game on the player's machine. The installed folder
|
||||
holds only the game; the installer and its tools stay outside it.
|
||||
data and no code derived from the game: the installer builds the game on the player's machine. On Windows the
|
||||
installed folder also holds the build kit, the installer's tools the game rebuilds itself with for code mods, so the
|
||||
setup folder can be deleted after the install and the install folder moved.
|
||||
|
||||
## What the player does
|
||||
|
||||
@@ -16,7 +17,8 @@ holds only the game; the installer and its tools stay outside it.
|
||||
needed for Chinese, Korean and Traditional Chinese text;
|
||||
- the game language and the install folder.
|
||||
3. Waits for the setup, one time. The window shows each step and its progress.
|
||||
4. Plays `totk.exe` from the install folder. The installer and the source NSP or XCI (or NSZ, XCZ) can be deleted.
|
||||
4. Plays `totk.exe` from the install folder. The setup folder and the source NSP or XCI (or NSZ, XCZ) can be deleted,
|
||||
and the install folder can be moved: code mods rebuild the game with the build kit inside it.
|
||||
|
||||
The install folder (`main.c` `portable_paths()` reads this layout; it stays portable):
|
||||
|
||||
@@ -32,16 +34,25 @@ The install folder (`main.c` `portable_paths()` reads this layout; it stays port
|
||||
logs/ totk.log of the last run, totk.1.log to totk.4.log of the runs before it
|
||||
shader and pipeline caches prepared by the setup; shader-cache/shader-cache.pack holds the translations
|
||||
totk-install.json what built this folder: release version, input hashes, finished steps
|
||||
build-kit/ Windows: the release folder without TotK-Setup.exe (about 0.65 GB); the game's rebuild
|
||||
for code mods runs the setup window from it
|
||||
build-cache/ what the rebuild built in (cheats.status, exefs.status, plugins.status), rebuild.txt,
|
||||
and work/: the generated C, the lift cache and the objects (about 1.6 GB, Windows)
|
||||
```
|
||||
|
||||
## Updating the port
|
||||
|
||||
The player runs a newer installer on the same folder. It reads `totk-install.json`, keeps `romfs/`, `exefs/`,
|
||||
`shared-fonts/`, `profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the
|
||||
shaders. A newer release that changes nothing game-derived skips the rebuild. The window starts with the last install folder
|
||||
and the base, update, keys and firmware paths it ran with (paths only, never the keys file's contents), kept in
|
||||
`%LOCALAPPDATA%\TotK Recomp\setup.json` (`$XDG_STATE_HOME/totk-recomp/setup.json` elsewhere). Entries that no longer exist
|
||||
stay empty. A reinstall or a switch to another game version starts filled in.
|
||||
`shared-fonts/`, `profile/` and `totk.ini` after checking their hashes, replaces `build-kit/` with its own files,
|
||||
rebuilds `totk.exe` and re-prepares the shaders. The setup is an installer and updater: nothing after the install needs
|
||||
its folder. An install made by a setup before the build kit kept its temp folder in that setup's folder
|
||||
(`<setup>/temp/<install name>-<hash>`, the setup `build-cache/rebuild.txt` named); the first newer setup moves it into
|
||||
`build-cache/work` when both are on one disk (on another it starts empty) and writes the new `rebuild.txt`. Until then
|
||||
the old `rebuild.txt` keeps using the old setup folder. A newer release that changes nothing game-derived skips the
|
||||
rebuild. The window starts with the last install folder and the base, update, keys and firmware paths it ran with (paths
|
||||
only, never the keys file's contents), kept in `%LOCALAPPDATA%\TotK Recomp\setup.json`
|
||||
(`$XDG_STATE_HOME/totk-recomp/setup.json` elsewhere). Entries that no longer exist stay empty. A reinstall or a switch
|
||||
to another game version starts filled in.
|
||||
|
||||
Given an update file other than the recorded one (1.1.0, then 1.2.1, then 1.4.3), the installer extracts and builds again from it
|
||||
and keeps `profile/`. Saves carry over both ways between 1.2.1 and 1.4.3: 1.4.3 converts a 1.2.1 save on load, and 1.2.1
|
||||
@@ -53,21 +64,28 @@ loads the saves 1.4.3 wrote.
|
||||
`tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a
|
||||
first install, 20 GiB after it), the CPU is at least x86-64-v2 (SSE4.2; with AVX2/FMA the faster `v3` build is used,
|
||||
otherwise the `v2` one), a Vulkan 1.3 GPU exists. Each failure is one clear message.
|
||||
2. **Fonts** with Chinese, Korean or Traditional Chinese text and a firmware folder: extract the Switch system fonts
|
||||
2. **Kit** (Windows): copy the release folder without `TotK-Setup.exe` into `build-kit/` in the install folder
|
||||
(`installer/steps.py` `Kit`, `installer/layout.py` `KIT_ITEMS`); every step after it runs its tools from there, so
|
||||
the commands the build records (the compiler's path in the Ninja file) are the ones the game's rebuild runs later.
|
||||
A newer setup replaces the kit; the kit's own rebuild keeps it. The steps' keys fingerprint file contents, not
|
||||
paths (`runtime_key` leaves out the lines of `clang --version` that name its folder), so a rebuild from the kit
|
||||
or from a moved install folder keeps every step the setup recorded.
|
||||
3. **Fonts** with Chinese, Korean or Traditional Chinese text and a firmware folder: extract the Switch system fonts
|
||||
from the firmware into `shared-fonts/` (MacNativeGames' `extract_shared_fonts.py`, which verifies every NCA layer
|
||||
and BFTTF header). Without a firmware folder the setup warns and goes on; the game then lacks those glyphs.
|
||||
3. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
|
||||
4. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt` or `patches-v143.txt`) into a temp folder next
|
||||
to the installer. The code cheats of the mods' cheat files and the mods' ExeFS patch files become patch sets first
|
||||
(`tools/cheats.py`, `tools/exefs_patches.py`, status in `build-cache/cheats.status` and `exefs.status`); a file the
|
||||
tools refuse costs those cheats or patches, not the install. The patch files' writes into main's data go to
|
||||
`build-cache/exefs.data`, which the game stores into main at start; after the lift they are checked against the
|
||||
bytes the lifted code read as constants (`<gen>/main/folded.txt`, its jump tables).
|
||||
5. **Compile** the generated C, `modules.c`, `romfs.c`, `patches.c` and `ov_zstd.c` (they bake the per-install layout, RomFS
|
||||
4. **Extract** ExeFS and RomFS into the install folder and verify the hashes.
|
||||
5. **Lift** the ExeFS to C (`recomp/gen.py`, `recomp/patches-v121.txt` or `patches-v143.txt`) into the temp folder,
|
||||
`build-cache/work` in the install folder (`.setup-temp` on Linux, `--temp DIR` elsewhere). The code cheats of the
|
||||
mods' cheat files and the mods' ExeFS patch files become patch sets first (`tools/cheats.py`,
|
||||
`tools/exefs_patches.py`, status in `build-cache/cheats.status` and `exefs.status`); a file the tools refuse costs
|
||||
those cheats or patches, not the install. The patch files' writes into main's data go to `build-cache/exefs.data`,
|
||||
which the game stores into main at start; after the lift they are checked against the bytes the lifted code read as
|
||||
constants (`<gen>/main/folded.txt`, its jump tables).
|
||||
6. **Compile** the generated C, `modules.c`, `romfs.c`, `patches.c` and `ov_zstd.c` (they bake the per-install layout, RomFS
|
||||
hashes and the game version's patch page) and link
|
||||
them with the prebuilt runtime library for the host's level (`libtotkrt.a` or `libtotkrt-v2.a`) into `totk.exe`
|
||||
(`build_game.py --runtime-lib`; about 3 min on 24 cores).
|
||||
6. **Prepare** (only when the runtime, the translator or the game version changed, not for a new lift) the RomFS
|
||||
7. **Prepare** (only when the runtime, the translator or the game version changed, not for a new lift) the RomFS
|
||||
patches and the shaders (`tools/make_romfs_patches.py`, `tools/prepare_shaders.py`,
|
||||
`totk.exe --prepare-shaders`) in the install folder, then fold the shader cache's loose files (four per shader, about
|
||||
117,000) into `shader-cache/shader-cache.pack` (`tools/pack_shader_cache.py`). totk.exe reads the pack first and
|
||||
@@ -82,7 +100,7 @@ loads the saves 1.4.3 wrote.
|
||||
When the graphics driver crashes while building the shaders, `prepare_shaders.py` runs that part again without the
|
||||
variants it was building ([building.md](building.md), step 6), and the setup finishes with a warning that names how
|
||||
many were left out.
|
||||
7. **Shapes**: the UI laid out for each screen shape this player needs, `romfs-patches-<w>x<h>`
|
||||
8. **Shapes**: the UI laid out for each screen shape this player needs, `romfs-patches-<w>x<h>`
|
||||
(`--aspect`), and with Xbox or PC prompts in `totk.ini` that layout with those prompts too,
|
||||
`romfs-patches-<style>-<w>x<h>` (`--prompts <style> --aspect --reuse romfs-patches-<style>`, which hard-links every
|
||||
file but the UI archive; for `pc` with the same `--binds` as the PC prompts step, whose key the step follows). The shapes are those of every connected display (Windows: `EnumDisplayDevices`; Linux:
|
||||
@@ -93,39 +111,44 @@ loads the saves 1.4.3 wrote.
|
||||
again when the displays, the prompts or the patches change; at a game resolution whose shape has no folder the
|
||||
game draws 16:9 with bars, and the settings menu marks such sizes and can write the shape's folders in the
|
||||
background for the next start.
|
||||
8. **DLSS** (the window's tick box, on by default, or `--dlss`; Windows with an NVIDIA RTX GPU): download
|
||||
9. **DLSS** (the window's tick box, on by default, or `--dlss`; Windows with an NVIDIA RTX GPU): download
|
||||
`nvngx_dlss.dll` from NVIDIA's own DLSS SDK repository (github.com/NVIDIA/DLSS, a pinned tag, NVIDIA RTX SDK
|
||||
license) into the install folder and check its size and SHA-256 (`steps.py`, `Dlss`). The release does not ship
|
||||
the library; NGX loads it from the folder of totk.exe for `dlaa` and the DLSS modes. A DLSS library already in the
|
||||
folder (a newer one the player put there) is kept. Without an RTX GPU the step only logs and does nothing; a failed
|
||||
download warns, the game runs without DLSS, and the next run of the setup tries again.
|
||||
9. **Clean up** the temp folder, except the generated C and the objects (about 1.3 GB), and write
|
||||
10. **Clean up** the temp folder, except the generated C and the objects (about 1.3 GB), and write
|
||||
`build-cache/rebuild.txt` in the install folder.
|
||||
|
||||
A cancelled or failed setup resumes from the last finished step (`totk-install.json`).
|
||||
|
||||
## Rebuilding for the mods
|
||||
|
||||
The generated C and the objects stay in the temp folder. The next lift lifts again only the chunks whose inputs
|
||||
changed (the lift cache of `recomp/gen.py`: the patch sites inside a chunk, with their set indices) and rewrites only
|
||||
those whose C changed, and Ninja recompiles only those. A new or edited code cheat costs a few seconds of lift
|
||||
(discovery and a hash of the kept chunks; 7 s for one changed cheat word on 12 cores, against 102 s for the whole lift)
|
||||
and the compile of the chunks that hold its sites, not a full compile, and the shaders stay prepared. A cheat set added
|
||||
or removed before others in the patch file moves their set indices, so the chunks of their sites are lifted too. A
|
||||
change to `recomp/*.py` lifts everything.
|
||||
The generated C and the objects stay in the temp folder, inside the install folder. The next lift lifts again only the
|
||||
chunks whose inputs changed (the lift cache of `recomp/gen.py`: the patch sites inside a chunk, with their set indices)
|
||||
and rewrites only those whose C changed, and Ninja recompiles only those. A new or edited code cheat costs a few seconds
|
||||
of lift (discovery and a hash of the kept chunks; 7 s for one changed cheat word on 12 cores, against 102 s for the
|
||||
whole lift) and the compile of the chunks that hold its sites, not a full compile, and the shaders stay prepared. A
|
||||
cheat set added or removed before others in the patch file moves their set indices, so the chunks of their sites are
|
||||
lifted too. A change to `recomp/*.py` lifts everything.
|
||||
|
||||
`build-cache/rebuild.txt` holds the command that opens the setup window in its rebuild mode on the install folder
|
||||
(`installer/gui.py --rebuild <install folder>`, started through `python -c` so the release folder is on `sys.path`
|
||||
whatever the working folder), one argument per line. When an enabled mod has code cheats or ExeFS patch files the build
|
||||
lacks, `totk.exe` runs it at start and exits (`runtime/cheats.c` `rebuild_if_needed`); the
|
||||
window runs the setup on the folder at once and starts the game again when it succeeds. A failed or cancelled rebuild
|
||||
offers to start the game with the code it has; `build-cache/rebuild.skip` then keeps the same files from starting it
|
||||
again, until "Rebuild the game now" on the Cheats or Mods tab. A command that does not start at all leaves no
|
||||
`rebuild.skip`, so the next start tries it again. `[mods] mods_rebuild = off` turns it off. The rebuild needs the
|
||||
release folder the setup ran from: a moved or deleted release folder means running a setup again. On Linux the command
|
||||
starts the AppImage file itself (`$APPIMAGE --rebuild <install folder>`), because the AppImage's mounted folder is gone
|
||||
once the setup exits; a moved or deleted AppImage means running a setup again. Installs set up by an AppImage before
|
||||
this change hold a command into that mount: running the new setup once on the install folder writes the new one.
|
||||
whatever the working folder), one argument per line. On Windows it names the build kit relative to the install folder
|
||||
(`build-kit\python\pythonw.exe`, `build-kit`, `--rebuild .`): `totk.exe` takes the program from its own folder and
|
||||
starts the command there, so no path of the setup or of the install folder is stored. When an enabled mod has code
|
||||
cheats or ExeFS patch files the build lacks, `totk.exe` runs it at start and exits (`runtime/cheats.c`
|
||||
`rebuild_if_needed`); the window runs the setup on the folder at once and starts the game again when it succeeds. A
|
||||
failed or cancelled rebuild offers to start the game with the code it has; `build-cache/rebuild.skip` then keeps the
|
||||
same files from starting it again, until "Rebuild the game now" on the Cheats or Mods tab. A command that does not start
|
||||
at all leaves no `rebuild.skip`, so the next start tries it again. `[mods] mods_rebuild = off` turns it off. A moved
|
||||
install folder keeps its incremental rebuild: the kit's builds run `tools/build_game.py --portable`, whose Ninja file
|
||||
and deps log name every path relative to the kit (3.5 s to rebuild after a move instead of a full compile). A build with
|
||||
a mod plugin names the plugin's file by its absolute path (`recomp/layout.py`), so after a move the game finds it gone
|
||||
and runs the rebuild once. On Linux the command starts the AppImage file itself (`$APPIMAGE --rebuild <install
|
||||
folder>`), because the AppImage's mounted folder is gone once the setup exits; a moved or deleted AppImage means running
|
||||
a setup again. Installs set up by an AppImage before this change hold a command into that mount: running the new setup
|
||||
once on the install folder writes the new one.
|
||||
|
||||
## Command line
|
||||
|
||||
@@ -170,7 +193,8 @@ python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR
|
||||
The unpacked release; `tools/make_release.py` assembles it and `--zip` packs it:
|
||||
|
||||
```
|
||||
TotK-Setup.exe starts python/pythonw.exe -m installer.gui (installer/launcher/launcher.c, no console)
|
||||
TotK-Setup.exe starts python/pythonw.exe -m installer.gui (installer/launcher/launcher.c, no console); the
|
||||
rest is what the Kit step copies into the install folder's build-kit/
|
||||
installer/ the installer: steps, CLI, window, en/ru/zh-Hans text catalog
|
||||
python/ Python 3.14.9 embeddable + numpy, lz4, cryptography, cffi, pycparser (+ tkinter for the window)
|
||||
recomp/, tools/ the lifter and the build, extraction and shader scripts the steps run (copied from the repo)
|
||||
|
||||
+45
-6
@@ -49,8 +49,8 @@ named ones.
|
||||
|
||||
### Mods with code the port cannot run
|
||||
|
||||
Some mods bring code the port cannot run: their own Switch code (a subsdk plugin or a replaced main, as UltraCam ships)
|
||||
or a code patch in a form the port cannot apply. Such a mod stays off as a whole, because its other files may expect
|
||||
Some mods bring code the port cannot run: their own Switch code (a replaced main, or a subsdk plugin the rebuild
|
||||
refused, as it refuses UltraCam's) or a code patch in a form the port cannot apply. Such a mod stays off as a whole, because its other files may expect
|
||||
that code. Under its row the Mods tab then shows **Use <mod> without its code** (off by default). Switched on, the code
|
||||
the port cannot run is left out and the rest of the mod loads after a restart: its game files, cheats and the code
|
||||
patches that work. The mod's row says which part is left out and why. If the game misbehaves with it, switch it off
|
||||
@@ -80,9 +80,11 @@ ahead of time, so these patches are built into `totk.exe`:
|
||||
2. Until the rebuild succeeds, the whole mod stays off, its other files included.
|
||||
3. After the rebuild the mod switches on and off from the Mods tab like any other, without another rebuild.
|
||||
|
||||
The rebuild needs the setup folder the game was installed from. When that folder was moved or deleted, run
|
||||
`TotK-Setup.exe` again on the same install folder: it does the same job. `mods_rebuild = off` under `[mods]` in
|
||||
`totk.ini` turns the automatic rebuild off.
|
||||
The rebuild runs from `build-kit` in the game folder, which the setup put there: the setup folder can be deleted
|
||||
and the game folder moved. After a move, a game with a code plugin mod rebuilds once at its next start. A game
|
||||
installed by an older setup (no `build-kit` in its folder) still rebuilds from that setup folder until a newer
|
||||
setup runs once on it.
|
||||
`mods_rebuild = off` under `[mods]` in `totk.ini` turns the automatic rebuild off.
|
||||
|
||||
A code patch over one of the port's own patches (the 60 FPS patch, the resolution patch and the other entries under
|
||||
**Built into the port**) turns that built-in patch off while the mod is on. The mod's row names it.
|
||||
@@ -96,6 +98,41 @@ A patch file may also change the game's data (constants and tables in its `.roda
|
||||
stored into the loaded game at start, before it runs. When two mods write the same bytes, the higher one wins,
|
||||
as with files.
|
||||
|
||||
## Code plugins (exefs subsdk)
|
||||
|
||||
Some mods ship their own Switch code as an exlaunch plugin: a file `exefs\subsdk<N>` (The Second Quest's MSFix, Full
|
||||
Inventory Expansion's sort fixer, Quick Weapon Durability). On the Switch the plugin starts with the game and
|
||||
redirects some of the game's instructions into its own code. The port does the same with the plugin's code compiled
|
||||
ahead of time, after the design in [#845](https://github.com/valentderah/totk-pc-recomp/issues/845) by itaskaev2:
|
||||
|
||||
1. The rebuild (the same one as for code patches above) compiles the plugins with the game, then starts the game's
|
||||
code in a service mode that runs only one plugin's start-up (`totk.exe --plugin-init`), twice per plugin, and
|
||||
records which instructions it redirects. Each redirect becomes a code site of the mod, built into `totk.exe`. Each
|
||||
plugin is judged on its own: one that is refused, or whose start-up crashes or never returns, stays off with its
|
||||
reason, and the other plugins are built in.
|
||||
2. At every start each plugin's start-up runs again, before the game's own start-up code, in the same setting: it
|
||||
sees the game's own instructions, as in its service run. Its redirects are compared with the record of the
|
||||
rebuild. When they match, the mod's code sites go on. When they do not, the plugin stays off for this run, the
|
||||
mod's row says why, and the next start runs the rebuild.
|
||||
3. The plugin keeps what its start-up made (its memory, its tables), so its code works as on the Switch. The start
|
||||
costs about 50 ms per plugin; the game runs no differently without a plugin.
|
||||
4. Removing a plugin mod from the mods folder starts the rebuild at the next start, which builds the game without
|
||||
it (half a minute on 12 cores with the lift cache). A plugin mod switched off on the Mods tab only stays off.
|
||||
|
||||
The Mods tab order decides between a plugin and another mod's patch on the same instruction, as for code patches. A
|
||||
plugin site over one of the port's own patches turns that built-in patch off while the mod is on.
|
||||
|
||||
The rebuild refuses a plugin when its start-up or the code its redirects reach needs what the port does not give it:
|
||||
system calls other than the few its start-up needs (memory queries and maps, the clock, debug output), library calls
|
||||
outside a reviewed list (memory and string functions, the heap, text formatting), threads, services or input; a
|
||||
redirect that is not one of exlaunch's hook shapes (inline hook, replace hook, call-site rewrite); or two runs of its
|
||||
start-up that differ in code; or a start-up that crashes the service run or does not return within two minutes. The
|
||||
row then says **Its plugin ... cannot run in the port** with the reason, and the mod stays off; **Use <mod> without its
|
||||
code** uses the rest of it, including its other plugins that were built in. UltraCam stays refused: it hooks the graphics API,
|
||||
reads input and opens sockets.
|
||||
|
||||
`plugin_trace = 1` in `totk.ini` logs the first call into each of a plugin's hooks (`plugins: hook entry ... reached`).
|
||||
|
||||
## Cheats
|
||||
|
||||
A cheat file is part of a mod: `mods\<mod>\cheats\<build id>.txt`, switched cheat by cheat on the Cheats tab. Cheats
|
||||
@@ -134,7 +171,9 @@ keeps its textures at a quarter of the size.
|
||||
| This folder has no romfs, exefs or cheats files. | The archive was unpacked one level too deep (`mods\Mod\Mod\romfs`), or it holds no game files. | Move the folders up so `romfs` sits directly in the mod's folder. |
|
||||
| ... files carry another game version in their names. | The files are named for another update (`.Product.<number>.`) and the game's own files have no such name. They are still served under those names, as on a console, because the game sometimes asks for one; another update's resource size table is the exception and only lends its entries. | Nothing if the mod works. If it does nothing, download its release for your game update. |
|
||||
| Its code patches (exefs) are made for another game version. | The patch files name another update's build id. | Same: get the release for your update. |
|
||||
| This mod brings its own game code (a subsdk plugin or a replaced main). | Mods such as UltraCam ship compiled Switch code. The port cannot load it. | The whole mod stays off. **Use <mod> without its code** under its row uses the rest of it, which may expect the missing code (see Mods with code the port cannot run). |
|
||||
| This mod brings its own game code (a subsdk plugin or a replaced main). | The mod replaces the game's main code, which the port cannot load. | The whole mod stays off. **Use <mod> without its code** under its row uses the rest of it, which may expect the missing code (see Mods with code the port cannot run). |
|
||||
| Its plugin ... cannot run in the port: ... | The rebuild refused the mod's code plugin; the reason follows (see Code plugins). | The whole mod stays off. **Use <mod> without its code** under its row uses the rest of it. |
|
||||
| Its plugin stays off this time: it started differently from the build. | The plugin redirected other instructions at this start than in the rebuild, for example after the mod was updated in place. | Nothing: the next start runs the rebuild. |
|
||||
| This mod patches the game code (exefs) in a way the port cannot apply. | The patch changes code in a form the port cannot build, or bytes the port cannot change at start. The reason follows on the row. | The whole mod stays off. **Use <mod> without its code** under its row leaves that patch file out and uses the rest of the mod. |
|
||||
| The folder name contains \|. | The mod list uses `\|` between names. | Rename the folder. |
|
||||
|
||||
|
||||
@@ -24,6 +24,8 @@ Many rows below are developer tools that only act together with another setting
|
||||
| `core_diag` | `1` records why ready guest threads wait for a virtual core and for how long, with a histogram every 60 presents. | Turns on host-call and SVC depth tracking and writes many log lines (one per present over 25 ms); frames get slower. | Caution |
|
||||
| `barrier_merge` (default `0`) | Experimental. `1` keeps a render pass open through the game's Barrier calls that ask only for fragment ordering and texture invalidation, when no draw of the pass can write a storage buffer or image; the next draw that samples a written target gets a by-region barrier in the pass (the feedback layout). In scenes with about a thousand such barriers a frame this cuts the passes from about 1200 to 130 and the GPU time from 24.5 to 7.8 ms. | The game's barrier orders every pixel, the in-pass by-region barrier only the pixel a draw writes: a driver that runs it per screen tile can show flicker or squares at the edges of water, decals or particles. | Caution |
|
||||
| `menu_cursor_log` | `1` logs what the menu cursor code reads at every screen change. | Only adds log lines. | Safe |
|
||||
| `plugin_tick_offset` (default `0`) | Added to every GetSystemTick in the plugin service run (`totk.exe --plugin-init DIR`), so `tools/exefs_plugins.py` can run a plugin's init twice with other ticks and compare the records. | Has no effect outside `--plugin-init`; a plugin's init at the start of the game gets the plain ticks. | Safe |
|
||||
| `plugin_trace` (default `0`) | `1` logs the first call of each entry of a mod plugin's hooks (its callbacks, replace and call targets) as `plugins: hook entry ... reached`. | Each such call goes through a counting step; nothing changes without a mod plugin. | Safe |
|
||||
| `perf_slow_ms` (default `50`) | Frames at least this many milliseconds long get a `perf: slow frame` log line with the time blocked per cause. | Changes how many frames are logged; a small value fills the log. | Safe |
|
||||
| `replay_check` | `1` reads every texture read-back a second time after a full drain of the replay queue and compares the two. | Stalls the replay queue at every read-back, so frames get much slower; mismatches are only counted and logged. | Caution |
|
||||
| `resolve_check` | `1` compares every cached pool lookup with the locked lookup and stops on a difference. | Slows every pool lookup; the game stops if the two ever differ. | Caution |
|
||||
@@ -215,6 +217,7 @@ These names are code patch sets built into the generated game code (`recomp/patc
|
||||
| --- | --- | --- | --- |
|
||||
| `ability_guard` (default `1`) | Patch set that checks the target a Recall or Ultrahand pick hands to the game (1.4.3 main+0x1bd01c4, 1.2.1 main+0x1622bf0) and drops it for that frame where the game's own code would fault on it: the holder, the actor or its component unreadable, or the word the game's reference writes not writable (`runtime/ov_ability.c`). Every other target goes on unchanged; a target that fails UltraTears' live-actor test is only logged (`ability: target kept although ...`). | `0` runs the game's code as shipped: the random crash reading `0xffffffffffffffff` during an ability pick can return. | Risky |
|
||||
| `aspect_patch` (default `0`) | Patch set that feeds the camera the output's aspect ratio when it is not 16:9. | The resolution code switches this set on or off itself at start, so a value here is replaced; the code does not settle what a manual value does. | Unknown |
|
||||
| `audio_drain_fix` (default `1`) | Patch set that sends the game's audio thread back to its command ring after each list (1.2.1 main+0x883780, 1.4.3 main+0x149244), so it takes every queued list per 5 ms audio frame instead of one (#337). Game 1.1.0 has no site and keeps the game's code. | `0` runs the game's code as shipped: above about 200 game updates a second the ring stays full and the game's guard (1.2.1 main+0x9bcdc4) switches the sound commands off until the game is restarted. The set acts only when more than one list is queued, so below that rate it seldom changes anything. Measured at `fps_limit = 240`, seconds with sound: on 1.2.1 5 of 118 with `0` and 108 of 118 with `1`; on 1.4.3 2 of 118 with `0` and 110 of 119 with `1`. | Safe |
|
||||
| `fixed_res_patch` (default `0`) | Patch set that holds the game's dynamic resolution at 100 percent (the scene tier). | The resolution code switches this set from `dynamic_res` (on at `0`); a manual value is replaced. | Unknown |
|
||||
| `fov_patch` (default `0`) | Patch set that scales the camera's vertical view angle by the field-of-view setting. | The resolution code switches this set itself whenever the field of view is not 100 percent; a manual value is replaced. | Unknown |
|
||||
| `mouse_direct_patch` (default `1`) | Patch set of the hooks in the game's camera code behind `mouse_mode = direct` on game 1.4.3. | `0` stops `mouse_mode = direct` from taking effect; other mouse modes are unaffected. | Safe |
|
||||
|
||||
+7
-17
@@ -12,7 +12,6 @@ line, 3 cancelled (Ctrl+C or Ctrl+Break).
|
||||
Windows with an NVIDIA RTX GPU (steps.py, Dlss).
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
@@ -22,7 +21,7 @@ from . import layout as layouts
|
||||
from . import state as statefile
|
||||
from .events import Reporter
|
||||
from .runner import Context, run_steps
|
||||
from .steps import all_steps, migrate_state
|
||||
from .steps import adopt_old_temp, all_steps, migrate_state
|
||||
|
||||
EXIT = {'ok': 0, 'failed': 1, 'cancelled': 3}
|
||||
|
||||
@@ -33,17 +32,6 @@ class Parser(argparse.ArgumentParser):
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
# The temp folder inside the install folder when the layout names none (the Linux release); the check step does not
|
||||
# count it when it requires an empty install folder
|
||||
TEMP_IN_DEST = '.setup-temp'
|
||||
|
||||
|
||||
def temp_folder(base, dest):
|
||||
"""One temp folder per install folder, so two installs never share generated files."""
|
||||
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
|
||||
return base / f'{dest.name or "root"}-{tag}'
|
||||
|
||||
|
||||
def cancel(signum, frame):
|
||||
raise KeyboardInterrupt
|
||||
|
||||
@@ -58,8 +46,8 @@ def main(argv=None):
|
||||
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
|
||||
ap.add_argument('--dest', type=Path, required=True, help='install folder')
|
||||
ap.add_argument('--root', type=Path, help='release folder or repository (default: the folder above installer/)')
|
||||
ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release on Windows, .setup-temp in '
|
||||
'the install folder on Linux, build/install-temp in a repository)')
|
||||
ap.add_argument('--temp', type=Path, help='temp folder (default: build-cache/work in the install folder on Windows, '
|
||||
'.setup-temp there on Linux, build/install-temp in a repository)')
|
||||
ap.add_argument('--verify', action='store_true', help='hash every kept file, also those this release made')
|
||||
ap.add_argument('--dlss', action='store_true', help="download NVIDIA's nvngx_dlss.dll (NVIDIA RTX SDK license) next "
|
||||
'to the game, for dlaa and the DLSS modes')
|
||||
@@ -72,10 +60,12 @@ def main(argv=None):
|
||||
signal.signal(getattr(signal, name), cancel)
|
||||
layout = layouts.find(args.root)
|
||||
dest = args.dest.resolve()
|
||||
base = args.temp or layout.temp
|
||||
temp = temp_folder(base.resolve(), dest) if base else dest / TEMP_IN_DEST
|
||||
temp = layouts.temp_for(layout, dest, args.temp)
|
||||
adopted = adopt_old_temp(layout, dest, temp)
|
||||
log = temp / 'logs' / time.strftime('install-%Y%m%d-%H%M%S.log')
|
||||
rep = Reporter(log_path=log)
|
||||
if adopted:
|
||||
rep.log(adopted)
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
state = statefile.load(dest)
|
||||
|
||||
+3
-1
@@ -746,7 +746,9 @@ def main(argv=None):
|
||||
return selftest(layout, args.shots)
|
||||
fix_tcl_paths()
|
||||
root = tk.Tk()
|
||||
App(root, args.lang or windows_ui_language(), layout, rebuild=args.rebuild)
|
||||
# resolved now: the game starts the build kit's rebuild with the install folder as '.' and the working folder
|
||||
rebuild = args.rebuild.resolve() if args.rebuild else None
|
||||
App(root, args.lang or windows_ui_language(), layout, rebuild=rebuild)
|
||||
root.mainloop()
|
||||
return 0
|
||||
|
||||
|
||||
@@ -27,7 +27,8 @@ TEXT = Path(__file__).resolve().parent / 'text'
|
||||
UI_LANGUAGES = ('en', 'ru', 'pt-BR', 'zh-Hans')
|
||||
# what the language switch in the window's corner shows for each
|
||||
UI_LANGUAGE_LABELS = {'en': 'English', 'ru': 'Русский', 'pt-BR': 'Português (Brasil)', 'zh-Hans': '简体中文'}
|
||||
STEP_ORDER = ('check', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss', 'cleanup')
|
||||
STEP_ORDER = ('check', 'kit', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss',
|
||||
'cleanup')
|
||||
# message parameter -> catalog id prefix of the names that replace its raw values
|
||||
PARAM_NAMES = {'lang': 'lang.', 'known': 'lang.', 'step': 'step.', 'input': 'param.input.'}
|
||||
LANG_RU = 0x19 # primary language id of Russian in a Windows LANGID
|
||||
|
||||
+46
-6
@@ -16,7 +16,13 @@ Release (docs/installer.md "Release layout"), root = the unpacked folder:
|
||||
without one the game is built at -O1
|
||||
totk.ini.example
|
||||
VERSION the release version, one line
|
||||
temp/ the default temp folder (created)
|
||||
|
||||
Build kit (Windows release): the setup copies the release folder without TotK-Setup.exe into <install>/build-kit
|
||||
(KIT_ITEMS, steps.py Kit) and runs every later step from it, so the game's rebuild for its mods
|
||||
(build-cache/rebuild.txt, relative to the install folder) needs neither the setup folder nor its path: the setup can be
|
||||
deleted and the install folder moved. The kit is a release folder itself (release_layout). The temp folder with the
|
||||
rebuild's working state (gen/, build/, the lift cache) is <install>/build-cache/work (TEMP_IN_DEST), so it moves
|
||||
with the install and survives a newer setup's kit.
|
||||
|
||||
Linux release (the AppImage's folder, docs/installer.md "Linux release"): python/bin/python3 (python-build-standalone,
|
||||
with tkinter), llvm/ with clang, lld and llvm-ar plus llvm/sysroot (the glibc headers and libraries the game links
|
||||
@@ -26,9 +32,11 @@ Dev mode, root = a repository checkout (no python/python.exe): build/py314-embed
|
||||
Python the release ships, with the same packages), else this Python; toolchain/llvm-mingw-*/ and toolchain/deps-mingw;
|
||||
runtime/ as the sources with build/install-rtlib/libtotkrt.a and libtotkrt-v2.a (tools/build_game.py --make-runtime-lib
|
||||
--out build/install-rtlib, once per --cpu); MSYS2 UCRT64 ninja.exe; toolchain/hactool; build/xlate/totk_xlate.exe; the shader list from
|
||||
refs/MacNativeGames; build/pgo/pgo-<v>.profdata when there is one; build/install-temp. The version is VERSION with -dev and the commit: 0.1.0-dev+973c626. On Linux,
|
||||
dev mode takes clang, lld and ninja from PATH and the system's libraries (no deps folder, no SDL3 copy).
|
||||
refs/MacNativeGames; build/pgo/pgo-<v>.profdata when there is one; build/install-temp/<install name>-<hash>. The
|
||||
version is VERSION with -dev and the commit: 0.1.0-dev+973c626. Dev mode and the Linux release install no build kit:
|
||||
the repository, or the AppImage file, is what the rebuild runs. On Linux, dev mode takes clang, lld and ninja from PATH and the system's libraries (no deps folder, no SDL3 copy).
|
||||
"""
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -40,6 +48,14 @@ WIN = os.name == 'nt'
|
||||
EXE = '.exe' if WIN else ''
|
||||
UCRT = Path(r'C:\msys64\ucrt64\bin')
|
||||
SUPPORTED = ('1.2.1', '1.4.3', '1.1.0') # updates the installer builds; the check identifies the player's by hash
|
||||
KIT = 'build-kit' # the build kit's folder in the install folder
|
||||
# What the build kit holds of a release folder (tools/make_release.py writes them): all but TotK-Setup.exe. licenses/
|
||||
# and THIRD_PARTY_NOTICES.md go along with the binaries they cover.
|
||||
KIT_ITEMS = ('python', 'llvm', 'runtime', 'bin', 'recomp', 'tools', 'data', 'installer', 'licenses',
|
||||
'THIRD_PARTY_NOTICES.md', 'totk.ini.example', 'VERSION')
|
||||
# The temp folder inside the install folder, for a layout that names none (every release); the check step does not
|
||||
# count it when it requires an empty install folder. Linux keeps the name its installs already use.
|
||||
TEMP_IN_DEST = Path('build-cache', 'work') if WIN else Path('.setup-temp')
|
||||
|
||||
|
||||
def version_tag(version):
|
||||
@@ -139,9 +155,10 @@ def release_layout(root):
|
||||
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll') if WIN else (root / 'bin' / 'libSDL3.so.0',),
|
||||
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example',
|
||||
pgo_dir=root / 'data' if WIN else None,
|
||||
# Linux: None, the install folder's .setup-temp (installer/cli.py): the AppImage is mounted read-only, and on
|
||||
# a Steam Deck /home is often too full for the generated C while the install folder is on the SD card
|
||||
temp=root / 'temp' if WIN else None,
|
||||
# None: the install folder's TEMP_IN_DEST (temp_for). Windows: the rebuild's working state stays with the install
|
||||
# and its build kit. Linux: the AppImage is mounted read-only, and on a Steam Deck /home is often too full for
|
||||
# the generated C while the install folder is on the SD card
|
||||
temp=None,
|
||||
sysroot=None if WIN else root / 'llvm' / 'sysroot')
|
||||
|
||||
|
||||
@@ -191,3 +208,26 @@ def find(root=None):
|
||||
"""The layout of root (default: the folder above this package)."""
|
||||
root = Path(root or Path(__file__).resolve().parent.parent).resolve()
|
||||
return release_layout(root) if is_release(root) else dev_layout(root)
|
||||
|
||||
|
||||
def uses_kit(layout):
|
||||
"""True when the setup installs a build kit (Windows release); dev mode and the Linux AppImage run from where they
|
||||
are."""
|
||||
return WIN and not layout.dev
|
||||
|
||||
|
||||
def kit_folder(dest):
|
||||
return Path(dest) / KIT
|
||||
|
||||
|
||||
def temp_folder(base, dest):
|
||||
"""One temp folder per install folder below a shared base (dev mode, --temp), so two installs never share
|
||||
generated files."""
|
||||
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
|
||||
return base / f'{dest.name or "root"}-{tag}'
|
||||
|
||||
|
||||
def temp_for(layout, dest, override=None):
|
||||
"""The temp folder of a run on dest: below --temp or the layout's, else TEMP_IN_DEST in the install folder."""
|
||||
base = override or layout.temp
|
||||
return temp_folder(Path(base).resolve(), dest) if base else dest / TEMP_IN_DEST
|
||||
@@ -49,6 +49,10 @@ class Step:
|
||||
"""Do the work. Returns fields to record with the step."""
|
||||
return {}
|
||||
|
||||
def settle(self, ctx):
|
||||
"""After the step ran or was skipped: a step whose output the later steps run from (the build kit) points the
|
||||
context at it."""
|
||||
|
||||
|
||||
class Context:
|
||||
def __init__(self, layout, dest, temp, reporter, options, state=None, verify=False):
|
||||
@@ -121,6 +125,7 @@ def run_steps(steps, ctx):
|
||||
for s in steps:
|
||||
if not ctx.runs[s.id]:
|
||||
rep.step(s.id, 'skipped', reason=ctx.reasons[s.id])
|
||||
s.settle(ctx)
|
||||
continue
|
||||
current = s.id
|
||||
rep.step(s.id, 'started')
|
||||
@@ -133,6 +138,7 @@ def run_steps(steps, ctx):
|
||||
ctx.save() # the first save, after the check, creates the install folder
|
||||
rep.step(s.id, 'done', seconds=seconds)
|
||||
current = None
|
||||
s.settle(ctx)
|
||||
ctx.state['release'] = ctx.layout.release
|
||||
ctx.save()
|
||||
return 'ok'
|
||||
|
||||
+239
-27
@@ -1,5 +1,5 @@
|
||||
"""The ten setup steps of docs/installer.md: check, fonts, extract, lift, compile, prepare, pc_prompts, shapes, dlss,
|
||||
cleanup.
|
||||
"""The eleven setup steps of docs/installer.md: check, kit, fonts, extract, lift, compile, prepare, pc_prompts, shapes,
|
||||
dlss, cleanup.
|
||||
|
||||
Extraction runs in this process through tools/extract.py (the package's sibling tools/, which ships with it); fonts
|
||||
runs the vendored extract_shared_fonts.py on the firmware folder; lift, compile and prepare run the bundled Python on
|
||||
@@ -11,18 +11,21 @@ Install folder: exefs/, romfs/ (romfs.bin, index.json), profile/, shared-fonts/
|
||||
and its DLLs, nvngx_dlss.dll (with --dlss), romfs-index.bin, romfs-patches/, romfs-patches-xbox/,
|
||||
romfs-patches-xbox-label/, romfs-patches-pc/, romfs-patches-<w>x<h>/ and romfs-patches-<style>-<w>x<h>/ (for the
|
||||
screen shapes and the prompts style this player needs, installer/screens.py), totk_xlate.exe, the
|
||||
shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json.
|
||||
Temp folder (<temp>/<dest name>-<hash>): check/ (ExeFS of both inputs, the host check), unpacked/ (the NSP of an NSZ
|
||||
or XCZ input until the extraction is done), gen/ (C), build/ (objects), logs/. The cleanup keeps gen/, build/ and
|
||||
shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json, and on Windows build-kit/ (Kit): the release
|
||||
folder the steps after the kit step run from, and the game's rebuild with them.
|
||||
Temp folder (build-cache/work in the install folder on Windows, .setup-temp there on Linux, <temp>/<dest name>-<hash>
|
||||
with --temp or in a repository; installer/layout.py temp_for): check/ (ExeFS of both inputs, the host check),
|
||||
unpacked/ (the NSP of an NSZ or XCZ input until the extraction is done), gen/ (C), build/ (objects), logs/. The cleanup keeps gen/, build/ and
|
||||
logs/: the next run lifts into the same gen/, where recomp/gen.py lifts again only the chunks whose inputs changed (its
|
||||
lift cache, gen/<mod>/lift_cache.json) and rewrites only those whose C changed, and Ninja in build/ recompiles only
|
||||
those. A change of the mods' code cheats then costs a few seconds of lift and the chunks holding its sites, not a full
|
||||
lift or compile.
|
||||
|
||||
Install folder build-cache/: cheats.status (tools/cheats.py), exefs.status and exefs.data (tools/exefs_patches.py) and
|
||||
Install folder build-cache/: cheats.status (tools/cheats.py), exefs.status and exefs.data (tools/exefs_patches.py),
|
||||
plugins.status (tools/exefs_plugins.py; plugins.stale is the game's list of plugins whose start differed) and
|
||||
rebuild.txt, the command that starts this setup in its rebuild mode on the install folder (installer/gui.py --rebuild),
|
||||
one argument per line; totk.exe runs it at start when an enabled mod has code cheats the build lacks
|
||||
(runtime/cheats.c).
|
||||
one argument per line, relative to the install folder for the build kit; totk.exe runs it at start when an enabled mod
|
||||
has code cheats the build lacks (runtime/cheats.c).
|
||||
"""
|
||||
import hashlib
|
||||
import http.client
|
||||
@@ -39,6 +42,7 @@ import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from . import ini, screens
|
||||
from . import layout as layouts
|
||||
from .layout import EXE, SUPPORTED, version_tag
|
||||
from .runner import InstallError, Step
|
||||
|
||||
@@ -52,6 +56,7 @@ from extract_exefs import (common_tickets, container_entries, load_keys, nca_ctr
|
||||
GiB = 1 << 30
|
||||
DEST_EXTRA = 4 * GiB # beside the RomFS: shader and pipeline caches (about 2 GB), exefs, totk.exe, seed files
|
||||
TEMP_NEED = 3 * GiB # generated C (about 0.9 GB) and objects
|
||||
KIT_NEED = 1 * GiB # the build kit (Kit), about 0.65 GB
|
||||
HOSTCHECK = Path(__file__).resolve().parent / 'hostcheck.c'
|
||||
STAMP = '.install-key'
|
||||
|
||||
@@ -597,6 +602,8 @@ class Check(Step):
|
||||
dest_need -= romfs.stat().st_size
|
||||
if ctx.will_run('fonts'):
|
||||
dest_need += FONTS_NEED
|
||||
if ctx.will_run('kit') and layouts.uses_kit(ctx.layout):
|
||||
dest_need += KIT_NEED
|
||||
if ctx.will_run('prepare') or ctx.will_run('extract'):
|
||||
dest_need += DEST_EXTRA
|
||||
if ctx.will_run('lift') or ctx.will_run('compile'):
|
||||
@@ -649,6 +656,114 @@ class Check(Step):
|
||||
ctx.state['version'] = found
|
||||
|
||||
|
||||
# ---- 1b. build kit
|
||||
|
||||
class Kit(Step):
|
||||
"""The build kit (installer/layout.py, Windows release): the release folder without TotK-Setup.exe, copied into
|
||||
<install>/build-kit. Every step after this one runs its tools from there (settle), and the game's rebuild for its
|
||||
mods starts the setup window from there, so the setup folder can go once the install is done and the install
|
||||
folder can move. A newer setup replaces the kit; the kit's own rebuild keeps it. The steps' keys fingerprint file
|
||||
contents, not paths, so the kit, a copy of the setup's files, keeps every step the setup recorded."""
|
||||
|
||||
id = 'kit'
|
||||
|
||||
def key(self, ctx):
|
||||
return fingerprint('kit-1', ctx.layout.release if layouts.uses_kit(ctx.layout) else 'none')
|
||||
|
||||
def current(self, ctx, record):
|
||||
if not layouts.uses_kit(ctx.layout):
|
||||
return True
|
||||
kit = layouts.kit_folder(ctx.dest)
|
||||
if not layouts.is_release(kit):
|
||||
return False
|
||||
found = layouts.release_layout(kit)
|
||||
return found.release == ctx.layout.release and not found.missing(getattr(ctx, 'cpu', 'v3'))
|
||||
|
||||
def run(self, ctx):
|
||||
layout, rep = ctx.layout, ctx.reporter
|
||||
if not layouts.uses_kit(layout):
|
||||
return {}
|
||||
kit = layouts.kit_folder(ctx.dest)
|
||||
if layout.root.resolve() == kit.resolve(): # the kit's own rebuild: it is in place
|
||||
return {}
|
||||
new, old = kit.with_name(layouts.KIT + '.new'), kit.with_name(layouts.KIT + '.old')
|
||||
try:
|
||||
remove(new) # what a cancelled or failed run left
|
||||
remove(old)
|
||||
except OSError as e:
|
||||
raise InstallError('kit.in_use', path=str(e.filename or old)) from None
|
||||
files = []
|
||||
for name in layouts.KIT_ITEMS:
|
||||
src = layout.root / name
|
||||
if src.is_dir():
|
||||
files += [f for f in sorted(src.rglob('*')) if f.is_file() and '__pycache__' not in f.parts]
|
||||
elif src.is_file():
|
||||
files.append(src)
|
||||
total, done = size_of(files), 0
|
||||
rep.progress('kit', 'copy', 0, total, 'bytes', force=True)
|
||||
try:
|
||||
for f in files:
|
||||
dst = new / f.relative_to(layout.root)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(f, dst)
|
||||
done += f.stat().st_size
|
||||
rep.progress('kit', 'copy', done, total, 'bytes')
|
||||
except OSError as e:
|
||||
raise InstallError('kit.copy_failed', path=str(e.filename or new), detail=str(e)) from None
|
||||
# An open rebuild window, or a layout generator the game's settings menu started, holds files of the old kit:
|
||||
# then nothing is replaced, and the message asks to close the game and run the setup again
|
||||
moved = False
|
||||
try:
|
||||
if kit.exists():
|
||||
os.replace(kit, old)
|
||||
moved = True
|
||||
os.replace(new, kit)
|
||||
except OSError:
|
||||
if moved and not kit.exists():
|
||||
os.replace(old, kit) # the game keeps the kit it had
|
||||
raise InstallError('kit.in_use', path=str(kit)) from None
|
||||
try:
|
||||
remove(old)
|
||||
except OSError as e:
|
||||
rep.log(f'kit: the old kit stays until the next run: {e}')
|
||||
rep.log(f'kit: {len(files)} files, {total / GiB:.2f} GiB in {kit}')
|
||||
return {'bytes': total}
|
||||
|
||||
def settle(self, ctx):
|
||||
if not layouts.uses_kit(ctx.layout):
|
||||
return
|
||||
kit = layouts.kit_folder(ctx.dest).resolve()
|
||||
if ctx.layout.root.resolve() != kit:
|
||||
ctx.layout = layouts.release_layout(kit)
|
||||
ctx.reporter.log(f'kit: the next steps run from {kit}')
|
||||
|
||||
|
||||
def adopt_old_temp(layout, dest, temp):
|
||||
"""An install by a setup before the build kit has its temp folder (generated C, lift cache, objects) in that
|
||||
setup's folder: <setup>/temp/<install name>-<hash>, the setup build-cache/rebuild.txt names. The first run of a
|
||||
setup with the build kit moves it into the install folder when both are on one disk, and the steps' keys decide
|
||||
what of it still holds. On another disk the new temp folder starts empty: an update recompiles everything anyway
|
||||
(the compiler's path is in every compile command), so a copy would save only part of the lift.
|
||||
Runs before the run's log opens in temp; returns the line for the log, None when there was nothing to move."""
|
||||
if not layouts.uses_kit(layout) or temp != dest / layouts.TEMP_IN_DEST or temp.exists():
|
||||
return None
|
||||
try:
|
||||
lines = (dest / 'build-cache' / 'rebuild.txt').read_text(encoding='utf-8').splitlines()
|
||||
except OSError:
|
||||
return None
|
||||
if len(lines) < 4 or lines[1] != '-c' or not os.path.isabs(lines[3]):
|
||||
return None
|
||||
old = layouts.temp_folder(Path(lines[3]) / 'temp', dest)
|
||||
if not (old / 'gen').is_dir():
|
||||
return None
|
||||
try:
|
||||
temp.parent.mkdir(parents=True, exist_ok=True)
|
||||
os.replace(old, temp)
|
||||
return f'temp: moved the temp folder of the setup in {lines[3]} to {temp}'
|
||||
except OSError as e:
|
||||
return f'temp: the temp folder {old} stays where it is ({e}); {temp} starts empty'
|
||||
|
||||
|
||||
# ---- 2. fonts
|
||||
|
||||
class Fonts(Step):
|
||||
@@ -866,6 +981,43 @@ def exefs_files(ctx):
|
||||
return sorted(out)
|
||||
|
||||
|
||||
def plugin_files(ctx):
|
||||
"""The exlaunch plugins of the mods folder (each mod's exefs/subsdk*, in the first ExeFS place, as
|
||||
tools/exefs_plugins.py finds them), by mod name, for the lift key and the plugin stage."""
|
||||
mods = ctx.dest / 'mods'
|
||||
if not mods.is_dir():
|
||||
return []
|
||||
out = []
|
||||
for mod in sorted((m for m in mods.iterdir() if m.is_dir()), key=lambda m: m.name):
|
||||
exefs = next((mod / w for w in EXEFS_PLACES if (mod / w).is_dir()), None)
|
||||
if exefs:
|
||||
out += sorted((f for f in exefs.iterdir() if f.is_file() and f.name.lower().startswith('subsdk')),
|
||||
key=lambda f: f.name.lower())
|
||||
return out
|
||||
|
||||
|
||||
def plugins_status(ctx):
|
||||
"""The build's plugin status file (tools/exefs_plugins.py --status), which runtime/mods.c reads."""
|
||||
return ctx.dest / 'build-cache' / 'plugins.status'
|
||||
|
||||
|
||||
def built_plugins(status, files, mods):
|
||||
"""The files of the plugins the status file lists as built (kind `plugin`), in the order of files. The key is
|
||||
FNV-1a 32 of "<mod folder>/<path in the mod>", as tools/exefs_plugins.py and runtime/mods.c compute it."""
|
||||
if not status.is_file():
|
||||
return []
|
||||
built = {line.split('\t')[0] for line in status.read_text(encoding='utf-8').splitlines()
|
||||
if len(line.split('\t')) >= 3 and line.split('\t')[2] == 'plugin'}
|
||||
|
||||
def key(f):
|
||||
rel = f.relative_to(mods).as_posix()
|
||||
h = 0x811C9DC5
|
||||
for b in rel.encode():
|
||||
h = ((h ^ b) * 0x01000193) & 0xFFFFFFFF
|
||||
return f'{h:08X}'
|
||||
return [f for f in files if key(f) in built]
|
||||
|
||||
|
||||
def exefs_status(ctx):
|
||||
"""The build's ExeFS patch status file (tools/exefs_patches.py --status), which runtime/mods.c reads."""
|
||||
return ctx.dest / 'build-cache' / 'exefs.status'
|
||||
@@ -890,9 +1042,13 @@ class Lift(Step):
|
||||
def key(self, ctx):
|
||||
# fingerprint takes a file's name and content only; the paths under mods/ go in too, because tools/cheats.py
|
||||
# and tools/exefs_patches.py name each patch set after its mod folder, so a renamed folder needs a lift
|
||||
files = cheat_files(ctx) + exefs_files(ctx)
|
||||
# A mod plugin is built by running its init on a totk.exe of this build (tools/exefs_plugins.py), so the
|
||||
# runtime is in the key too when there is one, and this file, which runs that stage: a plugin stage that failed
|
||||
# (warn.plugins_skipped, the game built without the plugins) runs again once a release changes it.
|
||||
files = cheat_files(ctx) + exefs_files(ctx) + plugin_files(ctx)
|
||||
extra = [ctx.layout.tools / 'exefs_plugins.py', Path(__file__), runtime_key(ctx)] if plugin_files(ctx) else []
|
||||
return fingerprint('lift-5', ctx.keys['extract'], ctx.layout.recomp, ctx.layout.patches(game_version(ctx)),
|
||||
ctx.layout.tools / 'cheats.py', ctx.layout.tools / 'exefs_patches.py', *files,
|
||||
ctx.layout.tools / 'cheats.py', ctx.layout.tools / 'exefs_patches.py', *extra, *files,
|
||||
*(f.relative_to(ctx.dest / 'mods').as_posix() for f in files))
|
||||
|
||||
def current(self, ctx, record):
|
||||
@@ -943,12 +1099,50 @@ class Lift(Step):
|
||||
else:
|
||||
patches = ctx.temp / out
|
||||
args = ['--exefs', ctx.dest / 'exefs', '--out', gen]
|
||||
# The mods' exlaunch plugins (tools/exefs_plugins.py, after the design in #845 by itaskaev2): a lift with the
|
||||
# plugins in the layout, a totk.exe of it in build/ (the compile step's folder, so it recompiles only what
|
||||
# the hook sites change after), the plugin service run, and the plugins' patch sets for the lift below. A
|
||||
# refused plugin gets its reason in plugins.status; a failed stage costs the plugins, not the install.
|
||||
plugins = plugin_files(ctx)
|
||||
drop([plugins_status(ctx), ctx.dest / 'build-cache' / 'plugins.stale'])
|
||||
lift_plugins = []
|
||||
if plugins:
|
||||
listed = ['--plugins', ','.join(str(f) for f in plugins)]
|
||||
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py', args + ['--patches', patches] + listed, parse)
|
||||
if not code:
|
||||
code, log = run_tool(ctx, 'lift', layout.tools / 'build_game.py', build_args(ctx, gen))
|
||||
if not code:
|
||||
# the service run's totk.exe loads its DLLs from its own folder, as the installed one does
|
||||
for dll in layout.dlls:
|
||||
replace_file(dll, ctx.temp / 'build' / dll.name)
|
||||
code, log = run_tool(ctx, 'lift', layout.tools / 'exefs_plugins.py', [
|
||||
'--exe', ctx.temp / 'build' / ('totk' + EXE), '--out', ctx.temp / 'plugins', '--mods',
|
||||
ctx.dest / 'mods', '--patches', patches, '--patches-out', ctx.temp / 'patches-plugins.txt',
|
||||
'--status', plugins_status(ctx)])
|
||||
# exit 1 with a status file: refused plugins, their reasons in it; the rest goes on
|
||||
if plugins_status(ctx).is_file() and (ctx.temp / 'patches-plugins.txt').is_file():
|
||||
code = 0
|
||||
if code:
|
||||
drop([plugins_status(ctx)])
|
||||
skipped['warn.plugins_skipped'] = log
|
||||
else:
|
||||
patches = ctx.temp / 'patches-plugins.txt'
|
||||
# With one plugin built, the layout keeps every plugin of the service runs: the records hold addresses
|
||||
# of that layout. A refused plugin stays loaded at play, its init never runs (no patch set).
|
||||
if built_plugins(plugins_status(ctx), plugins, ctx.dest / 'mods'):
|
||||
lift_plugins = plugins
|
||||
if lift_plugins:
|
||||
args += ['--plugins', ','.join(str(f) for f in lift_plugins)]
|
||||
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py', args + ['--patches', patches], parse)
|
||||
if code and patches != plain:
|
||||
for tool, out, outputs, inputs, warn in stages:
|
||||
if inputs and warn not in skipped:
|
||||
drop(outputs)
|
||||
skipped[warn] = log
|
||||
if lift_plugins:
|
||||
drop([plugins_status(ctx)])
|
||||
skipped.setdefault('warn.plugins_skipped', log)
|
||||
args = ['--exefs', ctx.dest / 'exefs', '--out', gen]
|
||||
code, log = run_tool(ctx, 'lift', layout.recomp / 'gen.py', args + ['--patches', plain], parse)
|
||||
if code:
|
||||
raise InstallError('lift.failed', code=code, log=str(log))
|
||||
@@ -976,12 +1170,32 @@ def runtime_key(ctx):
|
||||
"""Fingerprint of what totk.exe is built from besides the lifted code: the runtime, the toolchain and the build
|
||||
tools. Shader preparation depends on these, not on the code cheats a lift adds."""
|
||||
layout, cpu = ctx.layout, getattr(ctx, 'cpu', 'v3')
|
||||
clang = subprocess.run([str(layout.clang), '--version'], capture_output=True, text=True).stdout
|
||||
# without the lines that name clang's folder: the build kit of a moved install folder is the same toolchain
|
||||
clang = '\n'.join(line for line in subprocess.run([str(layout.clang), '--version'], capture_output=True,
|
||||
text=True).stdout.splitlines()
|
||||
if not line.startswith(('InstalledDir:', 'Configuration file:')))
|
||||
tools = [layout.tools / n for n in ('build_game.py', 'romfs_index.py', 'check_content.py', 'check_isa.py')]
|
||||
return fingerprint('runtime-1', cpu, layout.runtime_lib_for(cpu), layout.runtime, layout.deps or 'system',
|
||||
layout.sysroot or 'host', *tools, clang, *[d.name for d in layout.dlls])
|
||||
|
||||
|
||||
def build_args(ctx, gen):
|
||||
"""tools/build_game.py's arguments for a totk.exe of the lifted C in gen, built in <temp>/build."""
|
||||
layout, dest = ctx.layout, ctx.dest
|
||||
cpu = getattr(ctx, 'cpu', 'v3')
|
||||
lib = layout.runtime_lib_for(cpu)
|
||||
pgo = layout.pgo(game_version(ctx))
|
||||
return ['--gen', gen, '--out', ctx.temp / 'build', '--toolchain', layout.llvm, '--cpu', cpu,
|
||||
*(['--deps', layout.deps] if layout.deps else []), *(['--sysroot', layout.sysroot] if layout.sysroot else []),
|
||||
'--runtime', layout.runtime, *(['--runtime-lib', lib] if lib else ()), *(['--pgo', pgo] if pgo else ()),
|
||||
'--ninja', layout.ninja,
|
||||
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
|
||||
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
||||
'--profile', dest / 'profile',
|
||||
# the build kit and the temp folder move with the install folder: Ninja's paths relative to the kit
|
||||
*(['--portable'] if layouts.uses_kit(layout) else [])]
|
||||
|
||||
|
||||
class Compile(Step):
|
||||
id = 'compile'
|
||||
|
||||
@@ -1012,16 +1226,7 @@ class Compile(Step):
|
||||
|
||||
rep.progress('compile', 'compile', force=True)
|
||||
cpu = getattr(ctx, 'cpu', 'v3')
|
||||
lib = layout.runtime_lib_for(cpu)
|
||||
pgo = layout.pgo(game_version(ctx))
|
||||
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
|
||||
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--cpu', cpu,
|
||||
*(['--deps', layout.deps] if layout.deps else []), *(['--sysroot', layout.sysroot] if layout.sysroot else []),
|
||||
'--runtime', layout.runtime, *(['--runtime-lib', lib] if lib else ()), *(['--pgo', pgo] if pgo else ()),
|
||||
'--ninja', layout.ninja,
|
||||
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
|
||||
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
||||
'--profile', dest / 'profile'], parse)
|
||||
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', build_args(ctx, ctx.temp / 'gen'), parse)
|
||||
exe = build / ('totk' + EXE)
|
||||
if code and link_output_locked(log):
|
||||
raise InstallError('compile.exe_locked', path=str(exe), log=str(log))
|
||||
@@ -1364,10 +1569,11 @@ class Cleanup(Step):
|
||||
return {}
|
||||
|
||||
|
||||
# Puts the release folder (the argument after -c) on sys.path and runs installer.gui with the rest. `-m installer.gui`
|
||||
# finds the package only through the release Python's ._pth: the Linux release's Python and the dev-mode embeddable one
|
||||
# lack that entry, and totk.exe starts the command in the install folder.
|
||||
REBUILD_BOOT = ('import runpy, sys; sys.path.insert(0, sys.argv.pop(1)); '
|
||||
# Puts the release folder or the build kit (the argument after -c, made absolute: the kit's is relative to the install
|
||||
# folder) on sys.path and runs installer.gui with the rest. `-m installer.gui` finds the package only through the
|
||||
# release Python's ._pth: the Linux release's Python and the dev-mode embeddable one lack that entry, and totk.exe
|
||||
# starts the command in the install folder.
|
||||
REBUILD_BOOT = ('import os, runpy, sys; sys.path.insert(0, os.path.abspath(sys.argv.pop(1))); '
|
||||
"runpy.run_module('installer.gui', run_name='__main__', alter_sys=True)")
|
||||
|
||||
|
||||
@@ -1386,7 +1592,13 @@ def rebuild_command(ctx):
|
||||
python = Path(sys.executable) if layout.dev else layout.python
|
||||
if os.name == 'nt' and python.with_name('pythonw.exe').is_file():
|
||||
python = python.with_name('pythonw.exe')
|
||||
return [str(python), '-c', REBUILD_BOOT, str(layout.root), '--root', str(layout.root), '--rebuild', str(ctx.dest)]
|
||||
root, dest = str(layout.root), str(ctx.dest)
|
||||
if layouts.uses_kit(layout) and layout.root.resolve() == layouts.kit_folder(ctx.dest).resolve():
|
||||
# the build kit: paths relative to the install folder, which totk.exe resolves against its own folder and
|
||||
# starts the command in, so a moved install folder keeps its rebuild
|
||||
python = Path(layouts.KIT) / python.relative_to(layout.root)
|
||||
root, dest = layouts.KIT, '.'
|
||||
return [str(python), '-c', REBUILD_BOOT, root, '--root', root, '--rebuild', dest]
|
||||
|
||||
|
||||
def write_rebuild_command(ctx):
|
||||
@@ -1414,4 +1626,4 @@ def romfs_problem(romfs, expected):
|
||||
|
||||
|
||||
def all_steps():
|
||||
return [Check(), Fonts(), Extract(), Lift(), Compile(), Prepare(), PcPrompts(), Shapes(), Dlss(), Cleanup()]
|
||||
return [Check(), Kit(), Fonts(), Extract(), Lift(), Compile(), Prepare(), PcPrompts(), Shapes(), Dlss(), Cleanup()]
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
"step.check": "Checking the computer and the game files",
|
||||
"step.kit": "Copying the build kit into the game folder",
|
||||
"step.fonts": "Extracting the system fonts",
|
||||
"step.extract": "Extracting the game",
|
||||
"step.lift": "Translating the game code to C",
|
||||
@@ -17,6 +18,7 @@
|
||||
"phase.check.unpack_base": "Unpacking the compressed base game",
|
||||
"phase.check.unpack_update": "Unpacking the compressed update",
|
||||
"phase.check.versions": "Checking the game versions",
|
||||
"phase.kit.copy": "Copying the tools the game rebuilds itself with for code mods",
|
||||
"phase.fonts.extract": "Extracting the shared fonts from the firmware",
|
||||
"phase.fonts.copy": "Copying the shared fonts into the install folder",
|
||||
"phase.extract.hash_base": "Computing the checksum of the base game",
|
||||
@@ -95,6 +97,8 @@
|
||||
"lift.failed": "Translating the game code failed (exit code {code}). Details: {log}",
|
||||
"compile.failed": "Compiling {game_exe} failed (exit code {code}). Details: {log}",
|
||||
"compile.exe_in_use": "{path} is in use. Close the game and run the installer again.",
|
||||
"kit.in_use": "{path} is in use. Close the game and any open setup or rebuild window, then run the installer again.",
|
||||
"kit.copy_failed": "The build kit cannot be copied to {path}: {detail}",
|
||||
"compile.exe_locked": "{path} cannot be written: another program holds it, a game started from that folder or an antivirus that flagged the file. Close the game, add the setup's folder and the install folder to the antivirus exclusions, restart Windows, then run the installer again. Details: {log}",
|
||||
"prepare.xlate_in_use": "{path} is in use. Close the game, end any totk_xlate process left from an earlier run, then run the installer again.",
|
||||
"prepare.shader_list_missing": "The shader list is missing.",
|
||||
@@ -108,6 +112,7 @@
|
||||
"warn.npdm_differs": "main.npdm differs from the reference {version} dump; only the code modules are used.",
|
||||
"warn.cheats_skipped": "The cheats of the mods folder that change the game's code could not be built; the game installs without them. Details: {log}",
|
||||
"warn.exefs_skipped": "The mods' ExeFS patches (.ips, .pchtxt) could not be built; the game installs without them. Details: {log}",
|
||||
"warn.plugins_skipped": "The mods' code plugins (exefs subsdk) could not be built; the game installs without them. Details: {log}",
|
||||
"warn.xbox_prompts_failed": "The Xbox or PC button prompts could not be prepared (exit code {code}); the game keeps the Switch prompts. Details: {log}",
|
||||
"warn.shaders_skipped": "The graphics driver crashed while building some shaders, so {count} were left out of the preparation. The game builds them when it first needs them, and the driver may crash there too; a driver update may fix it. Details: {log}",
|
||||
"warn.kernel_old": "Linux {version} lacks the write tracking the game relies on (6.7 or later); the game may run very slowly. Updating the system fixes this.",
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
"step.check": "Verificando o computador e os arquivos do jogo",
|
||||
"step.kit": "Copiando o kit de compilação para a pasta do jogo",
|
||||
"step.fonts": "Extraindo as fontes do sistema",
|
||||
"step.extract": "Extraindo o jogo",
|
||||
"step.lift": "Traduzindo o código do jogo para C",
|
||||
@@ -17,6 +18,7 @@
|
||||
"phase.check.unpack_base": "Descompactando o jogo base compactado",
|
||||
"phase.check.unpack_update": "Descompactando a atualização compactada",
|
||||
"phase.check.versions": "Verificando as versões do jogo",
|
||||
"phase.kit.copy": "Copiando as ferramentas com que o jogo se recompila para mods de código",
|
||||
"phase.fonts.extract": "Extraindo as fontes compartilhadas do firmware",
|
||||
"phase.fonts.copy": "Copiando as fontes compartilhadas para a pasta de instalação",
|
||||
"phase.extract.hash_base": "Calculando o checksum do jogo base",
|
||||
@@ -95,6 +97,8 @@
|
||||
"lift.failed": "Falha ao traduzir o código do jogo (código de saída {code}). Detalhes: {log}",
|
||||
"compile.failed": "Falha ao compilar o {game_exe} (código de saída {code}). Detalhes: {log}",
|
||||
"compile.exe_in_use": "{path} está em uso. Feche o jogo e execute o instalador de novo.",
|
||||
"kit.in_use": "{path} está em uso. Feche o jogo e qualquer janela de instalação ou recompilação aberta, depois execute o instalador de novo.",
|
||||
"kit.copy_failed": "Não foi possível copiar o kit de compilação para {path}: {detail}",
|
||||
"compile.exe_locked": "Não é possível gravar {path}: outro programa o está usando, um jogo aberto dessa pasta ou um antivírus que marcou o arquivo. Feche o jogo, adicione a pasta do instalador e a pasta de instalação às exceções do antivírus, reinicie o Windows e execute o instalador de novo. Detalhes: {log}",
|
||||
"prepare.xlate_in_use": "{path} está em uso. Feche o jogo, encerre qualquer processo totk_xlate que tenha sobrado de uma execução anterior e execute o instalador de novo.",
|
||||
"prepare.shader_list_missing": "A lista de shaders está faltando.",
|
||||
@@ -107,6 +111,7 @@
|
||||
"warn.lang_needs_fonts": "Textos em {lang} precisam das fontes de sistema do Switch, e nenhuma foi informada. O jogo não inicia neste idioma até que sejam adicionadas. Escolha a pasta do firmware (Contents/registered) ou veja \"Switch system fonts\" em docs/installer.md.",
|
||||
"warn.cheats_skipped": "Não foi possível compilar os cheats da pasta de mods que alteram o código do jogo; o jogo é instalado sem eles. Detalhes: {log}",
|
||||
"warn.exefs_skipped": "Não foi possível compilar os patches ExeFS dos mods (.ips, .pchtxt); o jogo é instalado sem eles. Detalhes: {log}",
|
||||
"warn.plugins_skipped": "Não foi possível compilar os plugins de código dos mods (exefs subsdk); o jogo é instalado sem eles. Detalhes: {log}",
|
||||
"warn.npdm_differs": "O main.npdm difere do dump de referência da versão {version}; só os módulos de código são usados.",
|
||||
"warn.xbox_prompts_failed": "Não foi possível preparar os ícones de botão do Xbox ou do PC (código de saída {code}); o jogo mantém os ícones do Switch. Detalhes: {log}",
|
||||
"warn.shaders_skipped": "O driver gráfico travou ao compilar alguns shaders, então {count} ficaram fora da preparação. O jogo os compila quando precisar deles pela primeira vez, e o driver pode travar ali também; uma atualização do driver pode resolver. Detalhes: {log}",
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
"step.check": "Проверка компьютера и игровых файлов",
|
||||
"step.kit": "Копирование набора сборки в папку игры",
|
||||
"step.fonts": "Извлечение системных шрифтов",
|
||||
"step.extract": "Распаковка игры",
|
||||
"step.lift": "Перевод кода игры в C",
|
||||
@@ -17,6 +18,7 @@
|
||||
"phase.check.unpack_base": "Распаковка сжатой базовой игры",
|
||||
"phase.check.unpack_update": "Распаковка сжатого обновления",
|
||||
"phase.check.versions": "Проверка версий игры",
|
||||
"phase.kit.copy": "Копирование инструментов, которыми игра пересобирает себя для модов с кодом",
|
||||
"phase.fonts.extract": "Извлечение общих шрифтов из прошивки",
|
||||
"phase.fonts.copy": "Копирование общих шрифтов в папку установки",
|
||||
"phase.extract.hash_base": "Подсчёт контрольной суммы базовой игры",
|
||||
@@ -95,6 +97,8 @@
|
||||
"lift.failed": "Не удалось перевести код игры (код выхода {code}). Подробности: {log}",
|
||||
"compile.failed": "Не удалось собрать {game_exe} (код выхода {code}). Подробности: {log}",
|
||||
"compile.exe_in_use": "Файл {path} занят. Закройте игру и запустите установку снова.",
|
||||
"kit.in_use": "{path} занят другой программой. Закройте игру и все окна установки или пересборки, затем запустите установку снова.",
|
||||
"kit.copy_failed": "Не удалось скопировать набор сборки в {path}: {detail}",
|
||||
"compile.exe_locked": "Файл {path} не удаётся записать: его держит другая программа — игра, запущенная из этой папки, или антивирус, который пометил файл. Закройте игру, добавьте папку установщика и папку установки в исключения антивируса, перезагрузите Windows и запустите установку снова. Подробности: {log}",
|
||||
"prepare.xlate_in_use": "Файл {path} занят. Закройте игру, завершите процесс totk_xlate, оставшийся от прошлого запуска, и запустите установку снова.",
|
||||
"prepare.shader_list_missing": "Нет списка шейдеров.",
|
||||
@@ -108,6 +112,7 @@
|
||||
"warn.npdm_differs": "main.npdm отличается от эталонного дампа {version}; используются только модули с кодом.",
|
||||
"warn.cheats_skipped": "Не удалось собрать читы из папки модов, меняющие код игры; игра установится без них. Подробности: {log}",
|
||||
"warn.exefs_skipped": "Не удалось собрать ExeFS-патчи модов (.ips, .pchtxt); игра установится без них. Подробности: {log}",
|
||||
"warn.plugins_skipped": "Не удалось собрать плагины кода модов (exefs subsdk); игра установится без них. Подробности: {log}",
|
||||
"warn.xbox_prompts_failed": "Не удалось подготовить подсказки кнопок Xbox или PC (код выхода {code}); игра оставит подсказки Switch. Подробности: {log}",
|
||||
"warn.shaders_skipped": "Графический драйвер аварийно завершился при сборке части шейдеров, поэтому {count} из них пропущены при подготовке. Игра соберёт их, когда они впервые понадобятся, и драйвер может упасть и там; обновление драйвера может это исправить. Подробности: {log}",
|
||||
"warn.kernel_old": "В Linux {version} нет отслеживания записи, на которое опирается игра (нужна версия 6.7 или новее); игра может работать очень медленно. Обновление системы это исправит.",
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
"step.check": "检查电脑和游戏文件",
|
||||
"step.kit": "正在把构建工具包复制到游戏文件夹",
|
||||
"step.fonts": "正在提取系统字体",
|
||||
"step.extract": "解包游戏",
|
||||
"step.lift": "将游戏代码转换为 C 代码",
|
||||
@@ -17,6 +18,7 @@
|
||||
"phase.check.unpack_base": "正在解压压缩的游戏本体",
|
||||
"phase.check.unpack_update": "正在解压压缩的更新",
|
||||
"phase.check.versions": "检查游戏版本",
|
||||
"phase.kit.copy": "正在复制游戏为代码模组重新构建自身所用的工具",
|
||||
"phase.fonts.extract": "正在从固件中提取共享字体",
|
||||
"phase.fonts.copy": "正在将共享字体复制到安装文件夹",
|
||||
"phase.extract.hash_base": "计算游戏本体的校验和",
|
||||
@@ -95,6 +97,8 @@
|
||||
"lift.failed": "游戏代码转换失败(退出代码 {code})。详情:{log}",
|
||||
"compile.failed": "编译 {game_exe} 失败(退出代码 {code})。详情:{log}",
|
||||
"compile.exe_in_use": "{path} 正在使用中。请关闭游戏后重新运行安装程序。",
|
||||
"kit.in_use": "{path} 正在被占用。请关闭游戏以及所有打开的安装或重新构建窗口,再重新运行安装程序。",
|
||||
"kit.copy_failed": "无法把构建工具包复制到 {path}:{detail}",
|
||||
"compile.exe_locked": "无法写入 {path}:另一个程序正在占用它,可能是从该文件夹启动的游戏,或标记了该文件的杀毒软件。请关闭游戏,将安装程序文件夹和安装文件夹加入杀毒软件的排除项,重启 Windows,然后重新运行安装程序。详情:{log}",
|
||||
"prepare.xlate_in_use": "{path} 正在被占用。请关闭游戏,结束上次残留的 totk_xlate 进程,再重新运行安装程序。",
|
||||
"prepare.shader_list_missing": "缺少着色器列表。",
|
||||
@@ -107,6 +111,7 @@
|
||||
"warn.lang_needs_fonts": "{lang}文本需要 Switch 系统字体,但未提供。在添加这些字体之前,游戏无法以该语言启动。请选择固件文件夹(Contents/registered),或参阅 docs/installer.md 中的“Switch system fonts”一节。",
|
||||
"warn.cheats_skipped": "无法构建 mods 文件夹中修改游戏代码的金手指;游戏将不含它们进行安装。详情:{log}",
|
||||
"warn.exefs_skipped": "无法构建模组的 ExeFS 补丁(.ips、.pchtxt);游戏将不含它们进行安装。详情:{log}",
|
||||
"warn.plugins_skipped": "无法构建模组的代码插件(exefs subsdk);游戏将不含它们进行安装。详情:{log}",
|
||||
"warn.npdm_differs": "main.npdm 与参考的 {version} 导出不同;只使用其中的代码模块。",
|
||||
"warn.xbox_prompts_failed": "无法准备 Xbox 或 PC 按键提示(退出代码 {code});游戏将继续使用 Switch 按键提示。详情:{log}",
|
||||
"warn.shaders_skipped": "显卡驱动在编译部分着色器时崩溃,因此有 {count} 个着色器未被预先准备。游戏会在首次需要时编译它们,驱动在那时也可能崩溃;更新驱动可能会解决。详情:{log}",
|
||||
|
||||
+113
-22
@@ -3,7 +3,11 @@
|
||||
|
||||
Usage: gen.py --exefs build/extracted/v121/exefs --out generated/v121 [--modules main,subsdk0,sdk] [--jobs N]
|
||||
[--limit N] [--chunk-fns N] [--chunk-bytes B] [--big-fn B] [--patches FILE] [--v-locals K [--v-trust]]
|
||||
[--no-cache] [--fn-enter-all]
|
||||
[--no-cache] [--fn-enter-all] [--plugins NSO[,NSO...]]
|
||||
|
||||
--plugins places a mod's code modules (exlaunch plugins) after sdk as plugin0..N (recomp/layout.py) and lifts them with
|
||||
the game modules; their DT_INIT and init_array entries are seeds too (Module.init_functions). Without it the output is
|
||||
what it was before plugins existed.
|
||||
|
||||
Output, under --out:
|
||||
layout.h, layout.json module bases (recomp/layout.py)
|
||||
@@ -63,8 +67,8 @@ import patches # noqa: E402
|
||||
import regs # noqa: E402
|
||||
import vlocals # noqa: E402
|
||||
from check_content import GENERATED_MARKER # noqa: E402
|
||||
from layout import layout, write as write_layout # noqa: E402
|
||||
from lifter import hx # noqa: E402
|
||||
from layout import layout, plugin_files, write as write_layout # noqa: E402
|
||||
from lifter import Insn, hx # noqa: E402
|
||||
from module import Module # noqa: E402
|
||||
|
||||
TERMINATORS = ('b', 'br', 'ret', 'trap')
|
||||
@@ -85,6 +89,30 @@ def fname(a):
|
||||
return f'f_{a:x}'
|
||||
|
||||
|
||||
def orig_name(a):
|
||||
"""The unpatched copy of the function at a that a plugin's replace hook reaches as its original (patches.py `orig`)."""
|
||||
return f'fo_{a:x}'
|
||||
|
||||
|
||||
def open_module(name, path, base, table, plugin=False):
|
||||
"""Module name for lifting: a plugin with a `plugin` line of the patch table (recomp/patches.py) is lifted from its
|
||||
image after the init, its inline-hook stubs returning where they would branch to their trampolines."""
|
||||
p = table.plugins.get(name)
|
||||
if not p:
|
||||
return Module(name, path, base, 'c', plugin=plugin)
|
||||
image = Path(p['image']).read_bytes()
|
||||
mod = Module(name, path, base, 'c', plugin=True, image=image, rets=[o + 12 for o in p['stubs']],
|
||||
seeds=sorted(p['stubs'] | p['seeds']))
|
||||
t0, t1 = mod.nso.text[0], mod.nso.text[0] + mod.nso.text[1]
|
||||
if hashlib.sha256(image[t0:t1]).hexdigest() != p['sha256']:
|
||||
raise patches.PatchError(f'{name}: {p["image"]} is not the image whose .text the patch table names')
|
||||
for o in p['stubs']:
|
||||
w = int.from_bytes(image[o + 12:o + 16], 'little')
|
||||
if not mod.in_text(base + o) or w >> 26 != 0x05:
|
||||
raise patches.PatchError(f'{name}+{o:#x}: no inline-hook stub (its fourth word {w:08x} is no b)')
|
||||
return mod
|
||||
|
||||
|
||||
def header(what):
|
||||
return f'// {GENERATED_MARKER}: {what}, generated by recomp/gen.py from game code. Do not edit or commit.\n'
|
||||
|
||||
@@ -229,12 +257,13 @@ class Emitter:
|
||||
the C (an argument here, or anything else it reads) must go into the lift cache's key: chunk_key."""
|
||||
|
||||
def __init__(self, mod, starts, patched=None, reg_locals=regs.LOCALS, callee_saved=True, spins=None,
|
||||
v_locals=vlocals.LOCALS, v_trust=False, hooks=None, fn_enter_all=False, picks=None):
|
||||
v_locals=vlocals.LOCALS, v_trust=False, hooks=None, fn_enter_all=False, picks=None, origs=None):
|
||||
self.mod = mod
|
||||
self.starts = starts
|
||||
self.patched = patched or {} # {address: [(set index, [replacement words])]} (recomp/patches.py)
|
||||
self.picks = picks or {} # {contested site: (pick index, {ordered set: value})} (recomp/patches.py)
|
||||
self.spins = spins or {} # {spin wait back-edge: its head} (recomp/patches.py `spin`)
|
||||
self.origs = origs or set() # functions also emitted unpatched, as orig_name() (recomp/patches.py `orig`)
|
||||
self.reg_locals = reg_locals # recomp/regs.py: registers per function in C locals
|
||||
self.kept = regs.CALLEE_SAVED if callee_saved else frozenset() # not reloaded after a call
|
||||
self.v_locals = v_locals # recomp/vlocals.py: V registers per function in C locals (0: all in c->v)
|
||||
@@ -290,9 +319,12 @@ class Emitter:
|
||||
alternative may branch (bl, b, b.cond / cbz / tbz, ret)."""
|
||||
out = []
|
||||
for _, words in self.patched[a]:
|
||||
lifted = [self.mod.lifter.lift(w, a) for w in words]
|
||||
# ('call', target): a plugin's inline hook (recomp/patches.py @call), before the site's own instruction
|
||||
lifted = [Insn('hook', target=w[1]) if isinstance(w, tuple) else self.mod.lifter.lift(w, a) for w in words]
|
||||
for j, r in enumerate(lifted):
|
||||
last = j == len(lifted) - 1
|
||||
if r.kind == 'hook' and not last:
|
||||
continue
|
||||
if r.kind != 'op' and not (last and r.kind in ('bl', 'b', 'bcond', 'ret')):
|
||||
raise patches.PatchError(f'{a:#x}: replacement word {words[j]:08x} is a {r.kind}')
|
||||
out.append(lifted)
|
||||
@@ -371,6 +403,22 @@ class Emitter:
|
||||
raise patches.PatchError(f'{a:#x}: a patched call that uses flags')
|
||||
return reads, alts[0][1]
|
||||
|
||||
def unpatched_copy(self, entry, end, refs):
|
||||
"""C of the function [entry, end) without its patched sites, named orig_name(entry): what a plugin's replace
|
||||
hook calls as the original (its trampoline, recomp/patches.py `orig`). Its own fall-back fb_ is renamed too."""
|
||||
kept = self.patched
|
||||
self.patched = {a: alts for a, alts in kept.items() if not entry <= a < end}
|
||||
try:
|
||||
lines, r = self.function(entry, end)
|
||||
finally:
|
||||
self.patched = kept
|
||||
refs |= r
|
||||
text = '\n'.join(lines).replace(f'fb_{entry:x}(', f'fbo_{entry:x}(')
|
||||
head = f'void {fname(entry)}(Cpu *c) {{'
|
||||
if head not in text:
|
||||
raise patches.PatchError(f'{entry:#x}: no function head to rename for its unpatched copy')
|
||||
return text.replace(head, f'void {orig_name(entry)}(Cpu *c) {{', 1)
|
||||
|
||||
def function(self, entry, end):
|
||||
"""(lines of C, referenced function starts) for the guest function [entry, end)."""
|
||||
mod, starts = self.mod, self.starts
|
||||
@@ -453,6 +501,12 @@ class Emitter:
|
||||
return f'if ({fx(i, r.cond)}) {{ {jump(r.target, i)} }}'
|
||||
if r.kind == 'ret':
|
||||
return f'{spill}return;'
|
||||
if r.kind == 'hook':
|
||||
# a call that writes no x30, the whole Cpu current before it and read back after it: the stub saves and
|
||||
# restores x30 itself, and the callback may change any register through exlaunch's hook context
|
||||
refs.add(r.target)
|
||||
self.stats['plugin_hook_calls'] += 1
|
||||
return f'{spill}{fname(r.target)}(c);{reload}'
|
||||
raise patches.PatchError(f'{a:#x}: patched instruction is a {r.kind}')
|
||||
|
||||
def patch(i, ins, a):
|
||||
@@ -548,6 +602,8 @@ def chunk_text(name, number, emitter, fns):
|
||||
lines, r = emitter.function(entry, end)
|
||||
body.append('\n'.join(lines))
|
||||
refs |= r
|
||||
if entry in emitter.origs:
|
||||
body.append(emitter.unpatched_copy(entry, end, refs))
|
||||
# Every referenced function gets a prototype, including ones defined later in this chunk.
|
||||
protos = ''.join(f'void {fname(r)}(Cpu *c);\n' for r in sorted(refs))
|
||||
if emitter.uses_patches:
|
||||
@@ -614,7 +670,7 @@ def _emitter(name):
|
||||
e = _W['emitters'].get(name)
|
||||
if e is None:
|
||||
path, base = _W['mods'][name]
|
||||
mod = Module(name, path, base, 'c')
|
||||
mod = open_module(name, path, base, _W['patches'])
|
||||
# Module.read is how the jump tables (recomp/jumptables.py _eval) read .text and .rodata at lift time: the
|
||||
# bytes it returns become part of the generated code, so <mod>/folded.txt lists them
|
||||
read, reads = mod.read, _W['reads'].setdefault(name, [])
|
||||
@@ -627,7 +683,8 @@ def _emitter(name):
|
||||
_W['patches'].for_module(name, base),
|
||||
spins=_W['patches'].spins_for(name, base),
|
||||
hooks=_W['patches'].hooks_for(name, base),
|
||||
picks=_W['patches'].picks_for(name, base), **_W['opts'])
|
||||
picks=_W['patches'].picks_for(name, base),
|
||||
origs=_W['patches'].origs_for(name, base), **_W['opts'])
|
||||
return e
|
||||
|
||||
|
||||
@@ -672,13 +729,19 @@ def _emit_chunk(task):
|
||||
|
||||
# ------------------------------------------------------------------------------------------------ driver
|
||||
|
||||
def table_text(name, addrs):
|
||||
def table_text(name, addrs, extra=None):
|
||||
"""<mod>_table.c: the function starts and their C; extra {address: C function name} adds entries of functions
|
||||
lifted elsewhere (a plugin's trampoline reaching the unpatched copy of a game function, recomp/patches.py `orig`)."""
|
||||
extra = extra or {}
|
||||
names = {a: fname(a) for a in addrs if a not in extra}
|
||||
names.update(extra)
|
||||
addrs = sorted(names)
|
||||
lines = [header(f'{name} function table'), '#include "cpu.h"\n\n']
|
||||
lines += [f'void {fname(a)}(Cpu *c);\n' for a in addrs]
|
||||
lines += [f'void {names[a]}(Cpu *c);\n' for a in addrs]
|
||||
lines.append(f'\nconst uint64_t {name}_fn_addrs[] = {{\n')
|
||||
lines.append(',\n'.join(hx(a) for a in addrs))
|
||||
lines.append(f'\n}};\nconst GuestFn {name}_fn_ptrs[] = {{\n')
|
||||
lines.append(',\n'.join(fname(a) for a in addrs))
|
||||
lines.append(',\n'.join(names[a] for a in addrs))
|
||||
lines.append(f'\n}};\nconst uint64_t {name}_fn_count = {len(addrs)};\n')
|
||||
return ''.join(lines)
|
||||
|
||||
@@ -742,7 +805,7 @@ def in_span(addrs, lo, hi):
|
||||
return addrs[bisect.bisect_left(addrs, lo):bisect.bisect_left(addrs, hi)]
|
||||
|
||||
|
||||
def chunk_key(mod_key, number, fns, patched, site_addrs, spins, spin_addrs, hook_addrs, picks=None):
|
||||
def chunk_key(mod_key, number, fns, patched, site_addrs, spins, spin_addrs, hook_addrs, picks=None, origs=None):
|
||||
"""Key of one chunk: its module, number (in its header) and functions, and the patch sites (set index and
|
||||
replacement words, @page resolved, so the patch page is in them), the pick index and values of its contested
|
||||
sites, spin waits and hooked functions inside its address range; hook_addrs None (FN_ENTER in every function) keys
|
||||
@@ -750,10 +813,15 @@ def chunk_key(mod_key, number, fns, patched, site_addrs, spins, spin_addrs, hook
|
||||
here, with a case in tests/lift_cache_test.py."""
|
||||
lo, hi = fns[0][0], fns[-1][1]
|
||||
in_chunk = in_span(site_addrs, lo, hi)
|
||||
sites = [[a, [[k, words] for k, words in patched[a]]] for a in in_chunk]
|
||||
sites = [[a, [[k, [list(w) if isinstance(w, tuple) else w for w in words]] for k, words in patched[a]]]
|
||||
for a in in_chunk]
|
||||
hooks = None if hook_addrs is None else in_span(hook_addrs, lo, hi)
|
||||
contested = [[a, picks[a][0], sorted(picks[a][1].items())] for a in in_chunk if a in (picks or {})]
|
||||
return _key(mod_key, number, fns, sites, [[b, spins[b]] for b in in_span(spin_addrs, lo, hi)], hooks, contested)
|
||||
parts = [mod_key, number, fns, sites, [[b, spins[b]] for b in in_span(spin_addrs, lo, hi)], hooks, contested]
|
||||
copies = [e for e, _ in fns if e in (origs or ())]
|
||||
if copies: # none: the key it had before plugins
|
||||
parts.append(copies)
|
||||
return _key(*parts)
|
||||
|
||||
|
||||
def _entry_check(file, entry):
|
||||
@@ -853,7 +921,10 @@ def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--exefs', type=Path, required=True)
|
||||
ap.add_argument('--out', type=Path, required=True)
|
||||
ap.add_argument('--modules', default='main,subsdk0,sdk')
|
||||
ap.add_argument('--modules', default='main,subsdk0,sdk', help='modules to lift (default main,subsdk0,sdk and every '
|
||||
'--plugins module)')
|
||||
ap.add_argument('--plugins', default='', help='comma-separated plugin NSOs (exefs/subsdk* of a mod), in mod load '
|
||||
'order: lifted as plugin0..N after sdk')
|
||||
ap.add_argument('--jobs', type=int, help='emit workers (default: by free memory and CPU count)')
|
||||
ap.add_argument('--limit', type=int, default=0, help='emit only the first N functions of each module')
|
||||
ap.add_argument('--chunk-fns', type=int, default=3000, help='functions per chunk at most')
|
||||
@@ -887,11 +958,13 @@ def main():
|
||||
gen_key = generator_key(opts)
|
||||
|
||||
t0 = time.perf_counter()
|
||||
mods = layout(args.exefs)
|
||||
write_layout(mods, args.out)
|
||||
plugins = [Path(p) for p in args.plugins.split(',') if p]
|
||||
mods = layout(args.exefs, plugins)
|
||||
files = plugin_files(plugins)
|
||||
write_layout(mods, args.out, files)
|
||||
table = patches.load(args.patches, mods) if args.patches else patches.Table()
|
||||
write_if_changed(args.out / 'patches.h', patches.header_text(table, GENERATED_MARKER))
|
||||
want = args.modules.split(',')
|
||||
want = args.modules.split(',') + list(files)
|
||||
unknown = set(want) - {n for n, _, _ in mods}
|
||||
if unknown:
|
||||
raise SystemExit(f'not in {args.exefs}: {", ".join(sorted(unknown))}')
|
||||
@@ -901,8 +974,8 @@ def main():
|
||||
if name not in want:
|
||||
continue
|
||||
t = time.perf_counter()
|
||||
path = args.exefs / name
|
||||
mod = Module(name, path, base, 'c')
|
||||
path = Path(files[name][0]) if name in files else args.exefs / name
|
||||
mod = open_module(name, path, base, table, plugin=name in files)
|
||||
patches.check(table, mod, name)
|
||||
starts, dstats = discover(mod)
|
||||
est = estimate_sizes(mod, starts)
|
||||
@@ -912,8 +985,25 @@ def main():
|
||||
chunks = partition(est[:n], args.chunk_fns, args.chunk_bytes, args.big_fn)
|
||||
moddir = args.out / name
|
||||
moddir.mkdir(parents=True, exist_ok=True)
|
||||
extra = table.orig_entries(name, {n: b for n, b, _ in mods})
|
||||
if extra: # a trampoline reaches another function's unpatched copy, not a lift of its own words
|
||||
keep = ~np.isin(abs_starts, np.array(sorted(extra), dtype=abs_starts.dtype))
|
||||
starts, est, abs_starts = starts[keep], est[keep], abs_starts[keep]
|
||||
n = len(starts) if not args.limit else min(args.limit, len(starts))
|
||||
ends = np.append(abs_starts[1:], mod.hi)
|
||||
chunks = partition(est[:n], args.chunk_fns, args.chunk_bytes, args.big_fn)
|
||||
missing = sorted(base + o for o in (table.plugins.get(name) or {}).get('seeds', ())
|
||||
if base + o not in set(abs_starts.tolist()))
|
||||
if missing:
|
||||
raise SystemExit(f'{name}: the seeds {", ".join(hex(a) for a in missing)} are no function starts')
|
||||
mkey = module_key(gen_key, name, path, base, abs_starts)
|
||||
if table.plugin_inputs(name):
|
||||
mkey = _key(mkey, table.plugin_inputs(name), _sha256(Path(table.plugins[name]['image']).read_bytes()))
|
||||
patched, spins, picks = table.for_module(name, base), table.spins_for(name, base), table.picks_for(name, base)
|
||||
origs = table.origs_for(name, base)
|
||||
bad = sorted(origs - set(abs_starts.tolist()))
|
||||
if bad:
|
||||
raise SystemExit(f'{args.patches}: orig {", ".join(hex(a) for a in bad)} is no function start of {name}')
|
||||
site_addrs, spin_addrs = sorted(patched), sorted(spins)
|
||||
hooks = None if args.fn_enter_all else table.hooks_for(name, base)
|
||||
stray = sorted(set(hooks or ()) - set(abs_starts.tolist()))
|
||||
@@ -923,11 +1013,11 @@ def main():
|
||||
hook_addrs = None if hooks is None else sorted(hooks)
|
||||
for number, (i0, i1) in enumerate(chunks):
|
||||
fns = list(zip(abs_starts[i0:i1].tolist(), ends[i0:i1].tolist()))
|
||||
key = chunk_key(mkey, number, fns, patched, site_addrs, spins, spin_addrs, hook_addrs, picks)
|
||||
key = chunk_key(mkey, number, fns, patched, site_addrs, spins, spin_addrs, hook_addrs, picks, origs)
|
||||
tasks.append((int(est[i0:i1].sum()), (name, number, str(moddir / f'{name}_{number:04d}.c'), fns, key)))
|
||||
paths[name] = (str(path), base)
|
||||
starts_abs[name] = abs_starts
|
||||
info[name] = {'base': base, 'lo': mod.lo, 'hi': mod.hi, 'emitted': abs_starts[:n].tolist(),
|
||||
info[name] = {'base': base, 'lo': mod.lo, 'hi': mod.hi, 'emitted': abs_starts[:n].tolist(), 'extra': extra,
|
||||
'chunks': len(chunks), 'discover': dstats, 'discover_seconds': time.perf_counter() - t,
|
||||
'moddir': moddir}
|
||||
print(f'{name}: {dstats["functions"]} functions ({dstats["seeds"]} seeds, {dstats["rounds"]} rounds), '
|
||||
@@ -992,7 +1082,8 @@ def main():
|
||||
stats.update(r['stats'])
|
||||
unsupported.update(r['unsupported'])
|
||||
t = time.perf_counter()
|
||||
table_written = write_if_changed(moddir / f'{name}_table.c', table_text(name, d['emitted']))
|
||||
table_written = write_if_changed(moddir / f'{name}_table.c', table_text(
|
||||
name, d['emitted'], {t: orig_name(f) for t, f in d['extra'].items()}))
|
||||
write_if_changed(moddir / 'folded.txt', folded_text(name, merge_ranges(folded[name])))
|
||||
removed = remove_stale(moddir, name, d['chunks'], errors)
|
||||
manifest = {
|
||||
|
||||
+35
-13
@@ -6,9 +6,12 @@ calls layout() and writes the result into generated/<version>/layout.h and layou
|
||||
build read.
|
||||
|
||||
Modules are placed in Horizon's load order (rtld is dropped; the host links the modules):
|
||||
main, subsdk0..9, sdk. Each base is aligned to 2 MiB. Everything must stay below 2^39, the limit of the addresses
|
||||
IPC buffers can encode.
|
||||
main, subsdk0..9, sdk. A mod's code modules (exlaunch plugins such as an exefs/subsdk9) follow sdk as plugin0, plugin1,
|
||||
... in mod load order, so the game modules keep their bases with or without them (after the plugin design in #845 by
|
||||
itaskaev2). Each base is aligned to 2 MiB. Everything must stay below 2^39, the limit of the addresses IPC buffers can
|
||||
encode.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
@@ -24,14 +27,18 @@ def align_up(v, a=ALIGN):
|
||||
return (v + a - 1) & ~(a - 1)
|
||||
|
||||
|
||||
def layout(exefs):
|
||||
"""[(name, base, size)] for the modules present in the ExeFS directory, in load order."""
|
||||
def plugin_name(i):
|
||||
return f'plugin{i}'
|
||||
|
||||
|
||||
def layout(exefs, plugins=()):
|
||||
"""[(name, base, size)] for the modules present in the ExeFS directory, then the plugin NSOs (paths, in mod load
|
||||
order) as plugin0..N, in load order."""
|
||||
exefs = Path(exefs)
|
||||
paths = [(name, exefs / name) for name in ORDER if (exefs / name).exists()]
|
||||
paths += [(plugin_name(i), Path(p)) for i, p in enumerate(plugins)]
|
||||
out, base = [], FIRST_BASE
|
||||
for name in ORDER:
|
||||
path = exefs / name
|
||||
if not path.exists():
|
||||
continue
|
||||
for name, path in paths:
|
||||
size = Nso(str(path)).size
|
||||
out.append((name, base, size))
|
||||
base = align_up(base + size)
|
||||
@@ -40,12 +47,27 @@ def layout(exefs):
|
||||
return out
|
||||
|
||||
|
||||
def write(mods, outdir):
|
||||
"""layout.json for Python tools and layout.h for the runtime."""
|
||||
def plugin_files(plugins):
|
||||
"""{plugin<N>: (absolute path, SHA-256 of the NSO)} for layout.json: the build and the runtime read a plugin from its
|
||||
mod folder, not from the ExeFS directory, and check that it is the file the code was lifted from."""
|
||||
return {plugin_name(i): (Path(p).resolve().as_posix(), hashlib.sha256(Path(p).read_bytes()).hexdigest())
|
||||
for i, p in enumerate(plugins)}
|
||||
|
||||
|
||||
def write(mods, outdir, files=None):
|
||||
"""layout.json for Python tools and layout.h for the runtime. files: plugin_files() for the plugin entries, which
|
||||
carry them as "file" and "sha256"; without plugins both files are what they were before plugins existed."""
|
||||
files = files or {}
|
||||
outdir = Path(outdir)
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
(outdir / 'layout.json').write_text(json.dumps(
|
||||
[{'name': n, 'base': f'{b:#x}', 'size': f'{s:#x}'} for n, b, s in mods], indent=1) + '\n')
|
||||
|
||||
def entry(n, b, s):
|
||||
e = {'name': n, 'base': f'{b:#x}', 'size': f'{s:#x}'}
|
||||
if n in files:
|
||||
e['file'], e['sha256'] = files[n]
|
||||
return e
|
||||
|
||||
(outdir / 'layout.json').write_text(json.dumps([entry(n, b, s) for n, b, s in mods], indent=1) + '\n')
|
||||
lines = ['// Module layout chosen by recomp/layout.py. Generated; do not edit.', '#pragma once', '']
|
||||
for n, b, s in mods:
|
||||
lines.append(f'#define MOD_{n.upper()}_BASE {b:#x}ull')
|
||||
@@ -56,6 +78,6 @@ def write(mods, outdir):
|
||||
|
||||
|
||||
def read(outdir):
|
||||
"""[(name, base, size)] from a layout.json written by write()."""
|
||||
"""[(name, base, size)] from a layout.json written by write() (a plugin's file and hash left out)."""
|
||||
data = json.loads((Path(outdir) / 'layout.json').read_text())
|
||||
return [(m['name'], int(m['base'], 16), int(m['size'], 16)) for m in data]
|
||||
+37
-4
@@ -10,8 +10,9 @@ from collections import OrderedDict
|
||||
import numpy as np
|
||||
|
||||
from lifter import Insn, Lifter, Unsupported, hx
|
||||
from nso import Nso, R_AARCH64_RELATIVE
|
||||
from nso import DT_INIT, DT_INIT_ARRAY, DT_INIT_ARRAYSZ, Nso, R_AARCH64_RELATIVE
|
||||
|
||||
RET = 0xD65F03C0
|
||||
LIFT_CACHE = 1 << 16 # lifted instructions kept per module; analyses revisit nearby addresses only
|
||||
|
||||
|
||||
@@ -33,12 +34,27 @@ def _read_encoded(img, off, enc, pc_base):
|
||||
|
||||
|
||||
class Module:
|
||||
def __init__(self, name, path, base, simd='c'):
|
||||
def __init__(self, name, path, base, simd='c', plugin=False, image=None, rets=(), seeds=()):
|
||||
"""image, rets, seeds: a plugin's lift (recomp/patches.py `plugin`): its .text is taken from image (the plugin's
|
||||
image after its init, holding the hook stubs and trampolines the file has as zeros), the words at the image
|
||||
offsets rets become `ret` (an inline-hook stub's b to its trampoline), and the image offsets seeds are function
|
||||
starts too."""
|
||||
self.name = name
|
||||
self.nso = Nso(str(path))
|
||||
self.base = base
|
||||
self.plugin = plugin # a mod's code module (recomp/layout.py plugin<N>): its init functions are seeds too
|
||||
n = self.nso
|
||||
self.image = bytes(n.image)
|
||||
if image is not None:
|
||||
t0, t1 = n.text[0], n.text[0] + n.text[1]
|
||||
if len(image) < t1:
|
||||
raise ValueError(f'{name}: the image after the init is shorter than its .text')
|
||||
img = bytearray(self.image)
|
||||
img[t0:t1] = image[t0:t1]
|
||||
for o in rets:
|
||||
img[o:o + 4] = RET.to_bytes(4, 'little')
|
||||
self.image = bytes(img)
|
||||
self.extra_seeds = [base + o for o in seeds]
|
||||
self.lo = base + n.text[0]
|
||||
self.hi = self.lo + n.text[1]
|
||||
self.words = np.frombuffer(self.image, dtype='<u4', count=n.text[1] // 4, offset=n.text[0])
|
||||
@@ -83,7 +99,7 @@ class Module:
|
||||
def eh_frame_functions(self):
|
||||
"""Function starts from .eh_frame_hdr's binary search table (absolute addresses)."""
|
||||
img, hdr = self.image, self.nso.eh_start
|
||||
if not hdr or hdr >= len(img):
|
||||
if not hdr or hdr >= len(img) or self.nso.eh_end <= hdr: # MOD0 with an empty .eh_frame_hdr range (exlaunch)
|
||||
return []
|
||||
_, ptr_enc, cnt_enc, tbl_enc = img[hdr:hdr + 4]
|
||||
off = hdr + 4
|
||||
@@ -100,9 +116,26 @@ class Module:
|
||||
starts.append(self.base + loc)
|
||||
return starts
|
||||
|
||||
def init_functions(self):
|
||||
"""DT_INIT and the DT_INIT_ARRAY entries (absolute addresses). rtld calls DT_INIT only; the module's DT_INIT
|
||||
runs the array through blr (SDK and exlaunch modules alike), so the entries are seeds too. An entry holds its
|
||||
target as the addend of its R_AARCH64_RELATIVE relocation, or as the word itself when no relocation names it."""
|
||||
dyn, out = self.nso.dyn, []
|
||||
if dyn.get(DT_INIT):
|
||||
out.append(self.base + dyn[DT_INIT])
|
||||
if dyn.get(DT_INIT_ARRAY):
|
||||
addends = {off: add for off, typ, _, add in self.nso.relocs if typ == R_AARCH64_RELATIVE}
|
||||
for off in range(dyn[DT_INIT_ARRAY], dyn[DT_INIT_ARRAY] + dyn.get(DT_INIT_ARRAYSZ, 0), 8):
|
||||
out.append(self.base + addends.get(off, self.nso.u64(off)))
|
||||
return out
|
||||
|
||||
def seeds(self):
|
||||
"""Known function starts: .eh_frame_hdr, exported FUNC symbols, R_AARCH64_RELATIVE targets in .text."""
|
||||
"""Known function starts: .eh_frame_hdr, exported FUNC symbols, R_AARCH64_RELATIVE targets in .text; for a
|
||||
plugin also its init functions (exlaunch's DT_INIT starts inside a function discovery would merge it into)."""
|
||||
starts = set(self.eh_frame_functions())
|
||||
if self.plugin:
|
||||
starts.update(self.init_functions())
|
||||
starts.update(self.extra_seeds)
|
||||
for _, info, shndx, value, _ in self.nso.syms:
|
||||
if shndx and (info & 0xF) == 2:
|
||||
starts.add(self.base + value)
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ from collections import Counter
|
||||
import lz4.block
|
||||
|
||||
DT_NULL, DT_NEEDED, DT_PLTRELSZ, DT_HASH, DT_STRTAB, DT_SYMTAB, DT_RELA, DT_RELASZ = 0, 1, 2, 4, 5, 6, 7, 8
|
||||
DT_STRSZ, DT_JMPREL, DT_RELACOUNT = 10, 23, 0x6FFFFFF9
|
||||
DT_STRSZ, DT_INIT, DT_JMPREL, DT_INIT_ARRAY, DT_INIT_ARRAYSZ, DT_RELACOUNT = 10, 12, 23, 25, 27, 0x6FFFFFF9
|
||||
DT_RELRSZ, DT_RELR, DT_GNU_HASH = 0x23, 0x24, 0x6FFFFEF5
|
||||
R_AARCH64_ABS64, R_AARCH64_GLOB_DAT, R_AARCH64_JUMP_SLOT, R_AARCH64_RELATIVE = 257, 1025, 1026, 1027
|
||||
|
||||
|
||||
@@ -132,6 +132,22 @@ site aspect main 12521cc bd414260 f81f0fea @page:x10 bd403940 f84107ea # ldr s0
|
||||
site aspect main 19b36f0 1e211800 f81f0fea @page:x10 bd403940 f84107ea # fdiv s0, s0, s1 (viewport)
|
||||
site aspect main 1ecdf68 1e211800 f81f0fea @page:x10 bd403940 f84107ea # fdiv s0, s0, s1 (viewport)
|
||||
|
||||
# Field of view (runtime/resolution.c res_fov_live writes TOTK_FOV / 100 to page +60 and switches the set). The same two
|
||||
# camera routines as the aspect set (update main+0xcb4208, setup main+0x1252024; 1.4.3 main+0x628460, main+0xb432bc)
|
||||
# load the camera's vertical view angle (+0x13c, radians) a few words before the ratio and hand it to sead's perspective
|
||||
# projection (+0xa0, then sinf / cosf / tanf of half of it at +0xa4..+0xac); 1.4.3's setup reads the same fields through
|
||||
# x22 = camera + 0xd8. Eight more places fill a perspective projection the same way, 1.4.3 has the same eight, and
|
||||
# neither version patches them: one more inlined copy that scales an angle of another object (main+0x6c5548, 1.4.3
|
||||
# main+0x4e9754), three projection routines (main+0xdde89c, main+0x1141ce8, main+0x1a5e994; 1.4.3 main+0xc02690,
|
||||
# main+0xd42730, main+0x27d53d8) and four that fill a projection on the stack (main+0x7800f0, main+0x7805fc,
|
||||
# main+0xc764cc, main+0x21ee474; 1.4.3 main+0x63e2b4, main+0x63e744, main+0x79344, main+0x2804fec). The replacement is
|
||||
# the one of patches-v143.txt (word for word the sites of patches-v110.txt): it loads the angle as before and
|
||||
# multiplies it by the factor on the page; s8 holds the factor because the next instruction (fmul s8, s0, s1) overwrites
|
||||
# it, and x10 is kept on the stack.
|
||||
set fov TOTK_FOV_PATCH 0
|
||||
site fov main cb4ad8 bd413e60 f81f0fea @page:x10 bd413e60 bd403d48 1e280800 f84107ea # ldr s0, [x19, #0x13c] (camera update)
|
||||
site fov main 1252194 bd413e60 f81f0fea @page:x10 bd413e60 bd403d48 1e280800 f84107ea # ldr s0, [x19, #0x13c] (camera setup)
|
||||
|
||||
# Texture pool offsets past 2 GiB (MacNativeGames ca311e2d). The texture loader keeps a texture's place in its pool as a
|
||||
# 32-bit offset (main+0xad4294: sub x25, x9, x10, stored as the word w25) and widens it as a signed number for
|
||||
# nvnTextureBuilderSetStorage's offset and for SetPackagedTextureData's source (the pool's CPU address + offset). The
|
||||
@@ -1151,3 +1167,20 @@ site optref main 1622bec b4000188 937ffd09 8a290108 b4000188 # cbz x8 -> asr x
|
||||
set ability TOTK_ABILITY_GUARD 1
|
||||
site ability main 1622bf0 f9400115 94c424dc # ldr x21, [x8] -> bl page+0xf60
|
||||
site ability main 1622bf4 b4000155 aa0903f5 b4000155 # cbz x21 -> mov x21, x9; cbz x21 (x9 dead after it)
|
||||
|
||||
# Sound command lists above 200 fps (issue #337). The game thread hands its sound commands to the audio thread in
|
||||
# lists: 16 of them, of which 4 are always held (one being filled by the game thread, the others on the audio thread's
|
||||
# side), so at most 12 wait in the command ring. The game thread submits at most one list per update (main+0x9bd144,
|
||||
# when a free list is there), and the audio thread takes one per pass of its loop, one 5 ms audio frame: after a list
|
||||
# main+0x883780 goes on past the ring's check (b 0x883318). Above about 200 updates a second the ring stays full. A
|
||||
# guard in the game (main+0x9bcdc4, at the end of each update) counts the updates that find the ring of free lists
|
||||
# empty and exactly 12 lists in the command ring (0x9bce1c cmp w9, #0xc); the count goes back to 0 only when a free
|
||||
# list is there. At every multiple of 300 (10 s at 30 fps, 1.25 s at 240) it clears the sound command switch, the byte
|
||||
# at [[engine + 0x1a0] + 8] (main+0x2268bfc), and stops every active voice (main+0x226f29c). The switch is set to 1 only
|
||||
# by its constructor (main+0xf31ce8, from the engine's init main+0xf31530), so the game stays silent until it is
|
||||
# restarted. The site sends the audio thread back to the ring's check (0x8832e8, where the game's own 0x883c18 also
|
||||
# branches; x21, the engine, is not written between the check and the site), so it takes every queued list in the pass
|
||||
# and the ring does not stay full. Measured on 1.2.1 at fps_limit 240, same save and input script: sound in 5 of 118 s
|
||||
# with the set off, in 108 of 118 s with it on.
|
||||
set audiodrain TOTK_AUDIO_DRAIN_FIX 1
|
||||
site audiodrain main 883780 17fffee6 17fffeda # b 0x883318 -> b 0x8832e8
|
||||
@@ -1177,3 +1177,10 @@ site report main a2310 531f7821 530f7c24 531f7821 35000184 # lsr w4, w1, #15;
|
||||
# hook. main+0x585bd0 sets the render size (runtime/resolution.c SET_RENDER_SIZE): res_fn_hit takes the renderer from
|
||||
# the game's own call at start, so a DLSS mode can change while the game runs.
|
||||
hook main 585bd0
|
||||
|
||||
# Sound command lists above 200 fps (issue #337): as in patches-v121.txt. The audio thread's check of the command ring
|
||||
# is main+0x148aec, which wants x8 = the engine (as 0x148ae0 loads it from [x19]); after a list the site goes back to
|
||||
# it instead of on to 0x148b14, with the two instructions the game itself has at 0x149700. Measured on 1.4.3 at
|
||||
# fps_limit 240, same save and input script: sound in 2 of 118 s with the set off, in 110 of 119 s with it on.
|
||||
set audiodrain TOTK_AUDIO_DRAIN_FIX 1
|
||||
site audiodrain main 149244 17fffe34 f9400268 17fffe2a # b 0x148b14 -> ldr x8, [x19]; b 0x148aec
|
||||
+126
-7
@@ -10,6 +10,12 @@ Lifted code cannot be byte-patched when the game runs, so a patch is applied whe
|
||||
|
||||
hook <module> <function> [<function> ...]
|
||||
|
||||
plugin <set> <module> <text sha256> <image>
|
||||
expect <set> <module> <offset> <old> <new>
|
||||
stub <module> <offset>
|
||||
seed <module> <offset> [<offset> ...]
|
||||
orig <set> <module> <offset> <function module> <function offset>
|
||||
|
||||
<address> is a module offset (hex), <original> the instruction word the module has there (checked; a mismatch stops
|
||||
the generator), each <replacement> an instruction word (hex) or `@page:x<n>`, which is `adrp x<n>, <patch page>`
|
||||
encoded for the site's address. The patch page is the first 4 KiB page after main's image (layout.py leaves the rest
|
||||
@@ -44,6 +50,28 @@ and the direct branches back to it (module offsets). Each back-edge (checked: a
|
||||
<head>) is lifted as `rt_spin_gate(c, <head>); goto <head>`, which keeps every register and only adds time
|
||||
(runtime/kernel.c kern_spin_gate).
|
||||
|
||||
The plugin lines carry a mod's exlaunch plugin (layout.py plugin<N>), whose init runs at every start
|
||||
(runtime/plugin_init.c) and whose hooks are sites of its set (tools/exefs_plugins.py writes them from the plugin service
|
||||
run's record; the design follows #845 by itaskaev2):
|
||||
- `plugin`: <module> belongs to the set <set>; it is lifted from <image>, the plugin's image after its init (the
|
||||
service run's plugin<N>.img), whose .text holds the hook stubs and trampolines the NSO file has as zeros. <text
|
||||
sha256> is that .text's hash: the runtime compares it with the .text its own run of the init leaves, and keeps the
|
||||
set off when they differ.
|
||||
- `expect`: one word the init writes into a game module (the hook branches into main's .text); the runtime compares
|
||||
the words its run of the init wrote with these, all or nothing per plugin.
|
||||
- `stub`: an exlaunch inline-hook entry stub (stur x30; bl impl; ldur x30; b trampoline; .quad callback). Its fourth
|
||||
word is lifted as a `ret`: the generated site calls the stub and runs its own original instruction after it
|
||||
(`@call` below), so the trampoline (the original relocated, b site+4) never runs.
|
||||
- `seed`: function starts of the module the lift must have (the callbacks, the hook impls, branch targets).
|
||||
- `orig`: the plugin calls its replace hook's original through the trampoline at <module>+<offset>; the generator
|
||||
lifts the function at <function module>+<function offset> a second time without its patched sites
|
||||
(fo_<address>) and lists that copy at the trampoline's address in <module>'s function table, so the call reaches
|
||||
the original code and not the hook again.
|
||||
A replacement word `@call:<module>+<offset>` is a call of the function there that writes no x30: every register in
|
||||
the Cpu at the call and all read back after it (the callback may change any of them, as exlaunch's hook context
|
||||
allows); the flags stay as they were. It may stand before the site's last word, which is then the original
|
||||
instruction.
|
||||
|
||||
A `hook` line names guest functions (module offsets of function starts; gen.py stops on any other address) whose
|
||||
entry the runtime watches: their lifted C starts with FN_ENTER (runtime/cpu.h), a load of g_fnprof and a predicted
|
||||
branch to fnprof_hit while the runtime sets it. A table with hook lines puts FN_ENTER into these functions only; a table
|
||||
@@ -73,13 +101,36 @@ class Table:
|
||||
from here (for_module, picks_for, spins_for, hooks_for) is in the lift cache's key (recomp/gen.py chunk_key); a new
|
||||
kind of line the Emitter reads must go there too."""
|
||||
|
||||
def __init__(self, sets=(), sites=None, page=0, spins=None, hooks=None, ordered=()):
|
||||
def __init__(self, sets=(), sites=None, page=0, spins=None, hooks=None, ordered=(), plugins=None):
|
||||
self.sets = list(sets)
|
||||
self.sites = sites or {}
|
||||
self.page = page
|
||||
self.spins = spins or {} # {module: {back-edge offset: head offset}}
|
||||
self.hooks = hooks or {} # {module: {function offset}}; empty: FN_ENTER in every function
|
||||
self.ordered = set(ordered)
|
||||
# {module: {'set', 'sha256', 'image', 'expect': [(module, offset, old, new)], 'stubs': {offset},
|
||||
# 'seeds': {offset}, 'origs': {trampoline offset: (function module, function offset)}}}; empty without
|
||||
# plugin lines
|
||||
self.plugins = plugins or {}
|
||||
self.bases = {} # {module: base} of the layout load() read
|
||||
|
||||
def plugin_inputs(self, name):
|
||||
"""What the lift of module name takes from its plugin lines (for the lift cache's key), or None."""
|
||||
p = self.plugins.get(name)
|
||||
if not p:
|
||||
return None
|
||||
return [p['image'], p['sha256'], sorted(p['stubs']), sorted(p['seeds']), sorted(p['origs'])]
|
||||
|
||||
def origs_for(self, name, base):
|
||||
"""The absolute addresses of module name's functions that a plugin's `orig` line copies."""
|
||||
return {base + fo for p in self.plugins.values() for fm, fo in p['origs'].values() if fm == name}
|
||||
|
||||
def orig_entries(self, name, bases):
|
||||
"""{trampoline address in module name: absolute address of the function it reaches} (`orig`)."""
|
||||
p = self.plugins.get(name)
|
||||
if not p:
|
||||
return {}
|
||||
return {bases[name] + t: bases[fm] + fo for t, (fm, fo) in p['origs'].items()}
|
||||
|
||||
def picks(self):
|
||||
"""The contested sites, by module name and offset: [(module, offset, [ordered set index in file order])]; the
|
||||
@@ -110,19 +161,30 @@ class Table:
|
||||
"""{absolute back-edge address: absolute head address}."""
|
||||
return {base + b: base + h for b, h in self.spins.get(name, {}).items()}
|
||||
|
||||
def for_module(self, name, base):
|
||||
"""{absolute address: [(set index, [words])]} with @page resolved."""
|
||||
def for_module(self, name, base, bases=None):
|
||||
"""{absolute address: [(set index, [words])]} with @page resolved; an @call word becomes ('call', absolute
|
||||
address), which needs bases ({module: base})."""
|
||||
out = {}
|
||||
|
||||
def word(a, w):
|
||||
if not isinstance(w, tuple):
|
||||
return w
|
||||
if w[0] == 'page':
|
||||
return adrp(a, self.page, w[1])
|
||||
known = bases or self.bases
|
||||
if w[1] not in known:
|
||||
raise PatchError(f'{name}+{a - base:#x}: @call into {w[1]}, which is not in the layout')
|
||||
return ('call', known[w[1]] + w[2])
|
||||
|
||||
for off, alts in self.sites.get(name, {}).items():
|
||||
a = base + off
|
||||
out[a] = [(k, [adrp(a, self.page, w[1]) if isinstance(w, tuple) else w for w in repl])
|
||||
for k, repl, _ in alts]
|
||||
out[a] = [(k, [word(a, w) for w in repl]) for k, repl, _ in alts]
|
||||
return out
|
||||
|
||||
|
||||
def load(path, mods):
|
||||
"""Table from a patch file; mods is layout() output [(name, base, size)]."""
|
||||
sets, index, sites, spins, hooks, ordered = [], {}, {}, {}, {}, set()
|
||||
sets, index, sites, spins, hooks, ordered, plugins = [], {}, {}, {}, {}, set(), {}
|
||||
bases = {n: (b, s) for n, b, s in mods}
|
||||
for no, raw in enumerate(Path(path).read_text().splitlines(), 1):
|
||||
f = raw.split('#', 1)[0].split()
|
||||
@@ -146,6 +208,11 @@ def load(path, mods):
|
||||
for w in f[5:]:
|
||||
if w.startswith('@page:x'):
|
||||
repl.append(('page', int(w[7:])))
|
||||
elif w.startswith('@call:') and '+' in w:
|
||||
target, at = w[6:].split('+', 1)
|
||||
if target not in bases:
|
||||
raise PatchError(f'{where}: @call into {target}, which is not in the layout')
|
||||
repl.append(('call', target, int(at, 16)))
|
||||
else:
|
||||
repl.append(int(w, 16))
|
||||
alts = sites.setdefault(f[2], {}).setdefault(off, [])
|
||||
@@ -163,6 +230,15 @@ def load(path, mods):
|
||||
if b in mod:
|
||||
raise PatchError(f'{where}: {f[1]}+{b:#x} is a back-edge twice')
|
||||
mod[b] = head
|
||||
elif f[0] == 'plugin' and len(f) == 5:
|
||||
if f[1] not in index or f[2] not in bases:
|
||||
raise PatchError(f'{where}: unknown set {f[1]} or module {f[2]}')
|
||||
if f[2] in plugins:
|
||||
raise PatchError(f'{where}: module {f[2]} has two plugin lines')
|
||||
plugins[f[2]] = {'set': index[f[1]], 'sha256': f[3], 'image': f[4], 'expect': [], 'stubs': set(),
|
||||
'seeds': set(), 'origs': {}}
|
||||
elif f[0] in ('expect', 'stub', 'seed', 'orig'):
|
||||
plugin_line(f, where, index, bases, plugins)
|
||||
elif f[0] == 'hook' and len(f) >= 3:
|
||||
if f[1] not in bases:
|
||||
raise PatchError(f'{where}: module {f[1]} not in the layout')
|
||||
@@ -174,7 +250,36 @@ def load(path, mods):
|
||||
nxt = min((b for _, b, _ in mods if b > base), default=None)
|
||||
if nxt is not None and page + PAGE > nxt:
|
||||
raise PatchError(f'no room for the patch page after main ({page:#x})')
|
||||
return Table(sets, sites, page, spins, hooks, ordered)
|
||||
table = Table(sets, sites, page, spins, hooks, ordered, plugins)
|
||||
table.bases = {n: b for n, (b, _) in bases.items()}
|
||||
return table
|
||||
|
||||
|
||||
def plugin_line(f, where, index, bases, plugins):
|
||||
"""An expect, stub, seed or orig line into the plugin entry it belongs to (its `plugin` line comes first)."""
|
||||
def plugin(name):
|
||||
if name not in plugins:
|
||||
raise PatchError(f'{where}: {name} has no plugin line before this one')
|
||||
return plugins[name]
|
||||
|
||||
if f[0] == 'expect' and len(f) == 6:
|
||||
if f[1] not in index or f[2] not in bases:
|
||||
raise PatchError(f'{where}: unknown set {f[1]} or module {f[2]}')
|
||||
owner = [p for p in plugins.values() if p['set'] == index[f[1]]]
|
||||
if not owner:
|
||||
raise PatchError(f'{where}: set {f[1]} owns no plugin')
|
||||
owner[0]['expect'].append((f[2], int(f[3], 16), int(f[4], 16), int(f[5], 16)))
|
||||
elif f[0] == 'stub' and len(f) == 3:
|
||||
plugin(f[1])['stubs'].add(int(f[2], 16))
|
||||
elif f[0] == 'seed' and len(f) >= 3:
|
||||
plugin(f[1])['seeds'].update(int(o, 16) for o in f[2:])
|
||||
elif f[0] == 'orig' and len(f) == 6:
|
||||
p = plugin(f[2])
|
||||
if f[1] not in index or p['set'] != index[f[1]] or f[4] not in bases:
|
||||
raise PatchError(f'{where}: orig of set {f[1]} for {f[2]}, which another set owns, or module {f[4]}')
|
||||
p['origs'][int(f[3], 16)] = (f[4], int(f[5], 16))
|
||||
else:
|
||||
raise PatchError(f'{where}: cannot parse {" ".join(f)!r}')
|
||||
|
||||
|
||||
def check(table, mod, name):
|
||||
@@ -210,6 +315,20 @@ def header_text(table, marker):
|
||||
'// X(index, name, env switch, default)',
|
||||
'#define RT_PATCH_SETS(X) ' + ' '.join(f'X({i}, {n}, "{e}", {d})' for i, (n, e, d) in
|
||||
enumerate(table.sets)), '']
|
||||
if table.plugins: # none: the header is what it was before plugins
|
||||
names = sorted(table.plugins)
|
||||
writes = [f'X({i}, "{m}", {o:#x}, {old:#010x}u, {new:#010x}u)' for i, n in enumerate(names)
|
||||
for m, o, old, new in sorted(table.plugins[n]['expect'])]
|
||||
lines += [f'#define RT_PLUGIN_COUNT {len(names)}',
|
||||
'// X(index, module, its set, SHA-256 of its .text after the init)',
|
||||
'#define RT_PLUGINS(X) ' + ' '.join(f'X({i}, "{n}", {table.plugins[n]["set"]}, '
|
||||
f'"{table.plugins[n]["sha256"]}")' for i, n in enumerate(names)),
|
||||
f'#define RT_PLUGIN_WRITE_COUNT {len(writes)}',
|
||||
'// X(plugin index, module, offset, old word, new word): what the init writes into game modules',
|
||||
'#define RT_PLUGIN_WRITES(X) ' + ' '.join(writes),
|
||||
'// X(module, offset): the hook entries (seed lines) TOTK_PLUGIN_TRACE counts (runtime/plugin_init.c)',
|
||||
'#define RT_PLUGIN_ROOTS(X) ' + ' '.join(f'X("{n}", {o:#x})' for n in names
|
||||
for o in sorted(table.plugins[n]['seeds'])), '']
|
||||
picks = table.picks()
|
||||
if picks: # none: the header is what it was before ordered sets
|
||||
lines += [f'#define RT_PATCH_PICK_COUNT {len(picks)}',
|
||||
|
||||
+39
-5
@@ -30,6 +30,7 @@
|
||||
#define CHECKPOINT_RET (RT_V143 ? 0xbdb48cull : RT_V110 ? 0xeb8c14ull : 0xe7a954ull)
|
||||
#define NVN_RET (RT_V143 ? 0xbe0360ull : RT_V110 ? 0xebcdecull : 0xe7eb2cull)
|
||||
#define CTOR_DONE 2 // a module's constructor flag after its DT_INIT ran the init_array
|
||||
#define CTOR_WAITING 1 // after a DT_INIT that musl deferred (sdk's, called before main's)
|
||||
|
||||
// The init_array sizes (1.1.0 and 1.2.1: counted in the modules; 1.4.3: DT_INIT_ARRAYSZ); MacNativeGames' boot check
|
||||
// expects the same. The three tables carry the same rows, so cb_module_inits walks them by the size of one.
|
||||
@@ -40,6 +41,7 @@ static const InitCount inits_143[] = {{"main", 10717}, {"subsdk0", 45}, {"sdk",
|
||||
#define expected_inits (RT_V143 ? inits_143 : RT_V110 ? inits_110 : inits_121)
|
||||
|
||||
static volatile int32_t inits_done, checkpoint_seen;
|
||||
static int sdk_first; // a mod plugin ran: sdk's DT_INIT came first (cb_module_inits)
|
||||
uint64_t rt_guest_start_ns;
|
||||
|
||||
static const RtLoaded *module_named(const char *name) {
|
||||
@@ -77,15 +79,28 @@ static void cb_loader_ready(Cpu *c) {
|
||||
// afterwards (MacNativeGames loader.c:458-462).
|
||||
static void cb_module_inits(Cpu *c) {
|
||||
if (plat_cas32(&inits_done, 0, 0)) rt_fatal("boot: module constructors requested twice");
|
||||
if (rt_config()->plugin_init) plugin_init_run(c, module_named("sdk")); // the plugin service run ends there
|
||||
// A mod plugin's init runs after sdk's and before the game's (plugin_init.c plugins_start): sdk's DT_INIT then
|
||||
// comes first, as on the Switch; without a plugin the order stays main, subsdk0, sdk. Called first, sdk's DT_INIT
|
||||
// only registers sdk with its own musl (__nnmusl_init_dso returns 1: deferred), and main's DT_INIT then finishes
|
||||
// sdk from musl's list of waiting modules (the log shows sdk's RELRO protected and 44 pages of sdk's data written
|
||||
// during it on 1.4.3): sdk's flag stays 1 and its DT_INIT is not called again (a second call would protect its
|
||||
// locked RELRO again, and the SDK aborts).
|
||||
sdk_first = plugins_start(c, module_named("sdk"));
|
||||
for (unsigned k = 0; k < sizeof inits_121 / sizeof *inits_121; k++) {
|
||||
const RtLoaded *m = module_named(expected_inits[k].name);
|
||||
uint64_t entries = m->init_array_size / 8;
|
||||
if (!m->init) rt_fatal("boot: %s has no DT_INIT", m->name);
|
||||
rt_log("boot: ENTER %s DT_INIT (%llu init_array entries)", m->name, (unsigned long long)entries);
|
||||
rt_call(c, m->init, 0);
|
||||
int deferred = sdk_first && m == module_named("sdk");
|
||||
if (!deferred) {
|
||||
rt_log("boot: ENTER %s DT_INIT (%llu init_array entries)", m->name, (unsigned long long)entries);
|
||||
rt_call(c, m->init, 0);
|
||||
}
|
||||
uint32_t state = ctor_state(m);
|
||||
rt_logf("boot: DT_INIT %s state=%u (%llu)\n", m->name, state, (unsigned long long)entries);
|
||||
if (state != CTOR_DONE) rt_fatal("boot: %s DT_INIT left its constructor flag at %u, not 2", m->name, state);
|
||||
if (state != (deferred ? CTOR_WAITING : CTOR_DONE))
|
||||
rt_fatal("boot: %s DT_INIT left its constructor flag at %u, not %d", m->name, state,
|
||||
deferred ? CTOR_WAITING : CTOR_DONE);
|
||||
if (entries != expected_inits[k].entries)
|
||||
rt_fatal("boot: %s has %llu init_array entries, this game version has %llu", m->name, (unsigned long long)entries,
|
||||
(unsigned long long)expected_inits[k].entries);
|
||||
@@ -118,8 +133,9 @@ void rt_boot_event(const char *what, Cpu *c, const char *detail) {
|
||||
rt_fatal("boot: checkpoint %s reached from %s, expected main+%#llx", detail, at,
|
||||
(unsigned long long)CHECKPOINT_RET);
|
||||
if (!plat_cas32(&inits_done, 0, 0)) rt_fatal("boot: checkpoint %s before DT_INIT", detail);
|
||||
for (int i = 0; i < ldr_count(); i++)
|
||||
if (ctor_state(ldr_module(i)) != CTOR_DONE)
|
||||
for (int i = 0; i < ldr_count(); i++) // a plugin's init does not run at play time (plugin_init.c)
|
||||
if (!ldr_module(i)->plugin && ctor_state(ldr_module(i)) != CTOR_DONE &&
|
||||
!(sdk_first && !strcmp(ldr_module(i)->name, "sdk") && ctor_state(ldr_module(i)) == CTOR_WAITING))
|
||||
rt_fatal("boot: checkpoint %s: %s constructor flag reads %u", detail, ldr_module(i)->name,
|
||||
ctor_state(ldr_module(i)));
|
||||
plat_xchg32(&checkpoint_seen, 1);
|
||||
@@ -341,11 +357,29 @@ static void prewarm_main(void *arg) {
|
||||
return;
|
||||
}
|
||||
|
||||
// --plugin-init DIR: the modules load and nn::init::Start runs as at boot, and its module-init callback runs sdk's
|
||||
// DT_INIT and each plugin's instead of the game's constructors, writes the record and exits (plugin_init.c). No game
|
||||
// constructor, nnMain, shader prewarm, graphics or audio runs.
|
||||
void rt_boot_plugin_init(void) {
|
||||
check_npdm();
|
||||
plugin_init_setup();
|
||||
ldr_load(rt_config()->exefs);
|
||||
rt_patches_init(); // the generated code reads the patch page
|
||||
cb_addr[0] = hostfn_register("<rtld loader_ready>", cb_loader_ready);
|
||||
cb_addr[1] = hostfn_register("<rtld module_inits>", cb_module_inits);
|
||||
cb_addr[2] = hostfn_register("<rtld module_finis>", cb_module_finis);
|
||||
hostfn_seal();
|
||||
rt_guest_start_ns = kern_now_ns();
|
||||
kern_run_on_guest_main(guest_main);
|
||||
rt_fatal("boot: the guest main thread ended");
|
||||
}
|
||||
|
||||
void rt_boot(void) {
|
||||
check_npdm();
|
||||
if (!rt_config()->shot) vk_pipeline_async_enable(); // a --shot image must not depend on build timing
|
||||
// Shaders and pipelines recorded in earlier runs are made ready while the modules load; the guest waits for them.
|
||||
PlatThread *warm = plat_thread_new(prewarm_main, NULL, "prewarm");
|
||||
plugins_setup(); // a mod plugin's environment (plugin_init.c), before its imports bind
|
||||
ldr_load(rt_config()->exefs);
|
||||
rt_patches_init();
|
||||
mods_exefs_switch(); // after the patch sets' settings: an enabled mod's ExeFS patches on
|
||||
|
||||
+6
-5
@@ -32,7 +32,7 @@ static int save_ok = 1; // the last cfg_save_to wrote (o
|
||||
// logged as unknown. A new variable goes here too, or its ini key logs a harmless "unknown key" line, and gets a
|
||||
// description in totk.ini.example or docs/settings.md (tests/settings_doc_check.py checks both).
|
||||
static const char *const known[] = {
|
||||
"AB", "ABILITY_GUARD", "ADDR_REPORT", "AMIIBO_DIR", "ANISO", "APP_POOL", "ASPECT", "ASPECT_PATCH", "ASSUME_FOCUS", "ASTC", "ASTC_BC", "ASTC_TIME", "ASYNC_PREEMPT", "AUDIO", "AUDIO_CHANNELS", "AUDIO_DEVICE", "AUDIO_OFF", "AUDIO_PREFILL_MS",
|
||||
"AB", "ABILITY_GUARD", "ADDR_REPORT", "AMIIBO_DIR", "ANISO", "APP_POOL", "ASPECT", "ASPECT_PATCH", "ASSUME_FOCUS", "ASTC", "ASTC_BC", "ASTC_TIME", "ASYNC_PREEMPT", "AUDIO", "AUDIO_CHANNELS", "AUDIO_DEVICE", "AUDIO_DRAIN_FIX", "AUDIO_OFF", "AUDIO_PREFILL_MS",
|
||||
"BACKGROUND_WINDOW", "BIND_A", "BIND_B", "BIND_DOWN", "BIND_L", "BIND_LCLICK", "BIND_LEFT", "BIND_LSTICK_DOWN", "BIND_LSTICK_LEFT",
|
||||
"BIND_LSTICK_RIGHT", "BIND_LSTICK_UP", "BIND_MINUS",
|
||||
"BIND_PLUS", "BIND_R", "BIND_RCLICK", "BIND_RIGHT", "BIND_RSTICK_DOWN", "BIND_RSTICK_LEFT", "BIND_RSTICK_RIGHT", "BIND_RSTICK_UP",
|
||||
@@ -56,7 +56,7 @@ static const char *const known[] = {
|
||||
"MEM_RESERVE_MIB", "MENU_CURSOR", "MENU_CURSOR_LOG", "MENU_TEST_DIR", "MENU_TEST_SET", "MINIMAL_LAYOUT", "MIRROR_GAP", "MODS_DIR", "MODS_DISABLED", "MODS_NO_CODE", "MODS_ORDER", "MODS_REBUILD", "MODS_WITHOUT_CODE", "MOTION", "MOUSE_AIM_SCALE", "MOUSE_MODE", "MOUSE_DIRECT_PATCH", "MOUSE_SENS", "MOVIE_ASPECT", "MOVIE_HOST", "MOVIE_PACE",
|
||||
"MVOBJ_ALL", "MVOBJ_BANKS", "MVOBJ_EQUAL", "MVOBJ_EXACT", "MVOBJ_EXEC", "MVOBJ_NOC4", "MVOBJ_ONLY", "MVOBJ_TIMES", "MVPREV_CHECK", "MVPREV_GROW", "MVPREV_SKIP", "MVPREV_TRUST", "MV_CHECK",
|
||||
"MV_SURVEY",
|
||||
"ONE_PER_CORE", "OPERATION_MODE", "OPTREF_GUARD", "OUTPUT_RES", "PAD_LAYOUT", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_SCENE", "PCORE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", "POINTER_IMAGE",
|
||||
"ONE_PER_CORE", "OPERATION_MODE", "OPTREF_GUARD", "OUTPUT_RES", "PAD_LAYOUT", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_SCENE", "PCORE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", "PLUGIN_TICK_OFFSET", "PLUGIN_TRACE", "POINTER_IMAGE",
|
||||
"PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_BENCH", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES",
|
||||
"PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_ASYNC", "REC_END", "REC_SPIN_US", "REEXEC", "REFLEX",
|
||||
"RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHUNKS", "REPLAY_CHUNK_DRAWS", "REPLAY_SKEW_US", "REPLAY_STATE_CHECK", "REPLAY_WORKERS",
|
||||
@@ -75,7 +75,7 @@ static const char *const known[] = {
|
||||
static const struct { const char *key, *section; } sections[] = {
|
||||
{"FPS_LIMIT", "frame rate"}, {"FPS60", "frame rate"}, {"FRAME_QUEUE", "frame rate"},
|
||||
{"REALTIME", "frame rate"}, {"CLOTH_REALTIME", "frame rate"}, {"HIGHFPS_FIX", "frame rate"},
|
||||
{"CLIMB_FIX", "frame rate"},
|
||||
{"CLIMB_FIX", "frame rate"}, {"AUDIO_DRAIN_FIX", "frame rate"},
|
||||
{"FULLSCREEN", "display"}, {"WINDOW_SIZE", "display"}, {"PERF_OVERLAY", "display"}, {"VK_PRESENT", "display"},
|
||||
{"GPU_INDEX", "display"}, {"OUTPUT_RES", "display"}, {"RENDER_SCALE", "display"}, {"UPSCALER", "display"},
|
||||
{"SHARPNESS", "display"}, {"DYNAMIC_RES", "display"}, {"ASPECT", "display"}, {"MOVIE_ASPECT", "display"}, {"FOV", "display"}, {"ANISO", "display"},
|
||||
@@ -115,7 +115,7 @@ static const struct { const char *key, *section; } sections[] = {
|
||||
{"OPTREF_GUARD", "performance"}, {"ABILITY_GUARD", "performance"}, {"REPORT_GUARD", "performance"},
|
||||
{"TRACE", "diagnostics"}, {"FLIGHT", "diagnostics"}, {"PASS_CRUMBS", "diagnostics"}, {"ADDR_REPORT", "diagnostics"}, {"FLIGHT_SPIKE_MS", "diagnostics"},
|
||||
{"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"},
|
||||
{"MV_SURVEY", "diagnostics"},
|
||||
{"MV_SURVEY", "diagnostics"}, {"PLUGIN_TRACE", "diagnostics"},
|
||||
{"GAMEPLAY_TWEAKS_ENABLED", "gameplay tweaks"}, {"GAMEPLAY_ULTRAHAND_ROTATE", "gameplay tweaks"},
|
||||
{"GAMEPLAY_ULTRAHAND_C_AXIS", "gameplay tweaks"}, {"GAMEPLAY_ULTRAHAND_DISTANCE", "gameplay tweaks"},
|
||||
{"GAMEPLAY_ITEM_WHEEL", "gameplay tweaks"}, {"GAMEPLAY_ITEM_AD", "gameplay tweaks"},
|
||||
@@ -161,7 +161,7 @@ static const CfgMeta metas[] = {
|
||||
{"REALTIME", "1", "0|1", RESTART}, {"CLOTH_REALTIME", "1", "0|1", RESTART},
|
||||
{"MODS_REBUILD", "auto", "auto|off", RESTART},
|
||||
{"MODS_WITHOUT_CODE", "0", "0|1", RESTART}, // 1: a mod is used without the code module it cannot run (mods.c)
|
||||
{"HIGHFPS_FIX", "1", "0|1", RESTART}, {"CLIMB_FIX", "1", "0|1", RESTART},
|
||||
{"HIGHFPS_FIX", "1", "0|1", RESTART}, {"CLIMB_FIX", "1", "0|1", RESTART}, {"AUDIO_DRAIN_FIX", "1", "0|1", RESTART},
|
||||
{"LANG", "", "ja|en-US|fr|de|it|es|zh-CN|ko|nl|pt|ru|zh-TW|en-GB|fr-CA|es-419|zh-Hans|zh-Hant|pt-BR", RESTART},
|
||||
{"OPERATION_MODE", "docked", "docked|handheld", RESTART},
|
||||
{"ASSUME_FOCUS", "1", "0|1", LIVE}, // read on every focus change (ov_system.c)
|
||||
@@ -234,6 +234,7 @@ static const CfgMeta metas[] = {
|
||||
{"WB_FENCE_LIMIT", "65536", "0..1073741824", RESTART}, {"WB_MERGE", "1", "0|1", RESTART}, {"WB_EXACT", "1", "0|1", RESTART},
|
||||
{"PASS_CRUMBS", "", "0|1", RESTART}, // vk_crumbs.c pass marks; unset: VK_AMD_buffer_marker only
|
||||
{"ADDR_REPORT", "0", "0|1", RESTART}, // vk_device.c GPU address map for device faults
|
||||
{"PLUGIN_TRACE", "0", "0|1", RESTART}, // plugin_init.c: the first call of each mod plugin hook entry
|
||||
{"INPUT_RECORD", "", NULL, RESTART},
|
||||
};
|
||||
|
||||
|
||||
+16
-7
@@ -516,9 +516,13 @@ static void plain_name(char *s) {
|
||||
|
||||
// ---- rebuild: code cheats and ExeFS patches the build lacks
|
||||
// The setup writes build-cache\rebuild.txt (installer/steps.py write_rebuild_command), the command that starts its
|
||||
// window in rebuild mode on this install folder, one argument per line. When an enabled mod has code cheats this build
|
||||
// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), the game runs it and exits; the setup lifts
|
||||
// and compiles them in and starts the game again.
|
||||
// window in rebuild mode on this install folder, one argument per line. The build kit's command (installer/layout.py)
|
||||
// is relative to the install folder: its program is taken from this exe's folder, and the command runs with that
|
||||
// folder as its working folder, against which the setup resolves its other arguments. When an enabled mod has code
|
||||
// cheats this build
|
||||
// lacks (CHEAT_REBUILD) or ExeFS patch files it lacks (mods_exefs_pending), or the build has a mod plugin whose file is
|
||||
// gone (ldr_plugins_gone: its mod was removed), the game runs it and exits; the setup lifts and compiles them in (or the
|
||||
// plugin out) and starts the game again.
|
||||
// build-cache\rebuild.skip holds the key of the cheats a rebuild was started for: a rebuild that failed or was
|
||||
// cancelled leaves them unbuilt, and the same set does not start it again. TOTK_MODS_REBUILD=off never starts it.
|
||||
// Scripted runs (--seconds, --boot) never start it.
|
||||
@@ -532,7 +536,7 @@ static void rebuild_if_needed(const char *dir) {
|
||||
key = (key ^ fnv(c->info.key, strlen(c->info.key))) * 0x01000193u;
|
||||
key = (key ^ c->content) * 0x01000193u;
|
||||
}
|
||||
int patches = mods_exefs_pending(&key);
|
||||
int patches = mods_exefs_pending(&key) + ldr_plugins_gone(&key);
|
||||
if (!need && !patches) return;
|
||||
char cmd_path[PLAT_PATH_MAX + 64], skip_path[PLAT_PATH_MAX + 64], skip[16] = "", want[16];
|
||||
snprintf(cmd_path, sizeof cmd_path, "%sbuild-cache" SEP "rebuild.txt", dir);
|
||||
@@ -548,12 +552,13 @@ static void rebuild_if_needed(const char *dir) {
|
||||
char *cmd = why ? NULL : read_text(cmd_path);
|
||||
if (!why && !cmd) why = "no build-cache/rebuild.txt (a build without the setup)";
|
||||
if (why) {
|
||||
rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; no rebuild: %s", need,
|
||||
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; no rebuild: %s", need,
|
||||
patches, why);
|
||||
free(cmd);
|
||||
return;
|
||||
}
|
||||
const char *argv[32];
|
||||
char program[PLAT_PATH_MAX + 64];
|
||||
int argc = 0;
|
||||
for (char *line = cmd; line && *line && argc < 31;) {
|
||||
char *end = strchr(line, '\n'), *next = end ? end + 1 : NULL;
|
||||
@@ -564,13 +569,17 @@ static void rebuild_if_needed(const char *dir) {
|
||||
line = next;
|
||||
}
|
||||
argv[argc] = NULL;
|
||||
// CreateProcess resolves a relative program against this process's working folder, which a shortcut may set
|
||||
if (argc && !plat_path_absolute(argv[0]) &&
|
||||
snprintf(program, sizeof program, "%s%s", dir, argv[0]) < (int)sizeof program)
|
||||
argv[0] = program;
|
||||
if (argc < 2 || !plat_file_write(skip_path, want, strlen(want))) {
|
||||
rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; cannot start the "
|
||||
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; cannot start the "
|
||||
"rebuild (%s)", need, patches, cmd_path);
|
||||
free(cmd);
|
||||
return;
|
||||
}
|
||||
rt_log("cheats: %d code cheats and %d ExeFS patch files are not built into the game; starting the rebuild: %s",
|
||||
rt_log("cheats: %d code cheats and %d ExeFS patch files or plugins differ from the build; starting the rebuild: %s",
|
||||
need, patches, argv[0]);
|
||||
int ok = plat_spawn(argv, dir);
|
||||
free(cmd);
|
||||
|
||||
@@ -214,6 +214,21 @@ void rt_dispatch(Cpu *c) {
|
||||
__attribute__((musttail)) return f(c);
|
||||
}
|
||||
|
||||
// The plugin trace (plugin_init.c, TOTK_PLUGIN_TRACE): the slot of the function start a runs fn from now on. Once per
|
||||
// traced function, before the game's code runs; the hot path stays as it is. 0 when a is no function start of the
|
||||
// table or fn lies out of the reach of a slot's 32-bit offset.
|
||||
int rt_dispatch_reroute(uint64_t a, GuestFn fn) {
|
||||
uint64_t i = (a - D.lo) >> 2;
|
||||
if (!g_ready || (a & 3) || a < D.lo || i >= D.slots || !D.slot[i]) return 0;
|
||||
uintptr_t f = (uintptr_t)fn;
|
||||
if (f <= D.anchor || f - D.anchor > UINT32_MAX) return 0;
|
||||
uint64_t page = (uint64_t)(uintptr_t)&D.slot[i] & ~0xFFFull;
|
||||
if (!plat_mem_protect(page, 0x1000, PLAT_PROT_RW)) return 0;
|
||||
((uint32_t *)D.slot)[i] = (uint32_t)(f - D.anchor); // the table is sealed read-only, unsealed above
|
||||
plat_mem_protect(page, 0x1000, PLAT_PROT_READ);
|
||||
return 1;
|
||||
}
|
||||
|
||||
RtDispatchStats rt_dispatch_stats(void) {
|
||||
return (RtDispatchStats){D.lo, D.lo + D.slots * 4, D.slots, g_table_bytes, g_functions, g_nhost};
|
||||
}
|
||||
+5
-2
@@ -19,8 +19,10 @@ typedef struct RtModule {
|
||||
} RtModule;
|
||||
|
||||
// The module glue that tools/build_game.py writes for each build (build/<version>/modules.c), from layout.json and
|
||||
// layout.h: every module of the layout, and the modules linked into this executable.
|
||||
typedef struct RtLayoutEntry { const char *name; uint64_t base, size; } RtLayoutEntry;
|
||||
// layout.h: every module of the layout, and the modules linked into this executable. A mod's code module
|
||||
// (recomp/layout.py plugin<N>, after sdk) also names its NSO file and that file's SHA-256; a game module has NULL there
|
||||
// and is read from the ExeFS directory.
|
||||
typedef struct RtLayoutEntry { const char *name; uint64_t base, size; const char *file, *sha256; } RtLayoutEntry;
|
||||
extern const RtLayoutEntry rt_layout[];
|
||||
extern const int rt_layout_count;
|
||||
int rt_linked_modules(RtModule *out, int max);
|
||||
@@ -36,6 +38,7 @@ void rt_dispatch_init(const RtModule *mods, int count);
|
||||
|
||||
// The host function for a guest address, or NULL when nothing is there. Does not run guest code.
|
||||
GuestFn rt_dispatch_lookup(uint64_t pc);
|
||||
int rt_dispatch_reroute(uint64_t a, GuestFn fn); // plugin trace only (dispatch.c); 0 when it cannot
|
||||
|
||||
// "0x7000001234 (main+0x1234)" for a guest address inside a module, else the bare address. Returns buf.
|
||||
const char *rt_guest_name(uint64_t pc, char *buf, int size);
|
||||
|
||||
+30
-2
@@ -25,7 +25,9 @@ void rt_log_raw(const char *p, size_t n);
|
||||
const char *rt_log_path(void); // the log file, NULL when this run writes none
|
||||
enum { RT_TRACE_SVC = 1, RT_TRACE_HOSTCALL = 2, RT_TRACE_IPC = 4, RT_TRACE_FS = 8, RT_TRACE_GFX = 16, RT_TRACE_PERF = 32,
|
||||
RT_TRACE_SCHED = 64, RT_TRACE_AUDIO = 128, RT_TRACE_PRESENT = 256,
|
||||
RT_TRACE_HOSTCOUNT = 512 }; // no TOTK_TRACE name of its own: hostfn_count switches it (host.h)
|
||||
RT_TRACE_HOSTCOUNT = 512, // no TOTK_TRACE name of its own: hostfn_count switches it (host.h)
|
||||
RT_TRACE_PLUGIN = 1024 }; // no name either: the plugin service run (plugin_init.c) sees every SVC through
|
||||
// rt_plugin_svc while it runs a plugin's init
|
||||
extern unsigned rt_trace; // from TOTK_TRACE=svc,host,ipc,fs,gfx,perf,sched,audio,present (or all)
|
||||
// prof_win.c: TOTK_PROFILE=<path> sampling profiler; TOTK_TRACE=perf frame timing (waits of the render thread)
|
||||
// PERF_TEX: texture uploads on the render thread (deswizzle, decode, staging), PERF_ASTC included
|
||||
@@ -150,6 +152,7 @@ uint32_t mem_set_attribute(uint64_t a, uint64_t n, uint32_t mask, uint32_t at);
|
||||
uint32_t mem_map(uint64_t dst, uint64_t src, uint64_t n); // 0x04
|
||||
uint32_t mem_unmap(uint64_t dst, uint64_t src, uint64_t n); // 0x05
|
||||
uint32_t mem_query(uint64_t a, RtMemoryInfo *out); // 0x06
|
||||
extern void (*mem_query_hook)(uint64_t a, RtMemoryInfo *r); // plugin service run only: adjusts mem_query's answer
|
||||
uint32_t mem_map_physical(uint64_t a, uint64_t n); // 0x2C
|
||||
uint32_t mem_unmap_physical(uint64_t a, uint64_t n); // 0x2D
|
||||
int mem_info(uint32_t id, uint64_t *out); // GetInfo memory ids 2-7, 12-17, 21, 22, 28; 0 if not a memory id
|
||||
@@ -199,13 +202,24 @@ int rt_oom_forced_left(int site); // forced failures st
|
||||
typedef struct {
|
||||
const char *name; uint64_t base, size, text_end, data_off; // text_end, data_off: offsets from base
|
||||
uint64_t init, init_array, init_array_size, ctor_flag; // absolute guest addresses; init_array_size in bytes
|
||||
int plugin; // a mod's code module (layout plugin<N>), not a game module
|
||||
} RtLoaded;
|
||||
void ldr_load(const char *exefs_dir); // load, relocate (overrides applied), seal; once
|
||||
extern GuestFn ldr_plugin_import; // set before ldr_load (plugin_init.c): every JUMP_SLOT
|
||||
// import of a plugin binds to this observer
|
||||
// The module tables of the runtime library: it is built once per release (tools/build_game.py --make-runtime-lib),
|
||||
// before an install's layout (recomp/layout.py) adds its mod plugins after sdk, so they do not use that MOD_COUNT.
|
||||
#define RT_MAX_MODULES 32
|
||||
int ldr_count(void);
|
||||
int ldr_plugins_gone(uint32_t *key); // layout plugins whose file is gone; mixes their paths into key
|
||||
const RtLoaded *ldr_module(int i); // order: main, subsdk0, sdk
|
||||
uint64_t ldr_symbol(const char *name); // exported definition, 0 if none
|
||||
uint64_t ldr_plt_stub(const char *module, const char *name); // PLT stub whose GOT slot holds name, 0 if none
|
||||
const char *ldr_describe(uint64_t a, char *buf, int n); // "main+0x1234 (symbol+0x10)", "host <name>", or the address
|
||||
uint64_t ldr_nso_size(const char *dir, const char *file); // image size of an ExeFS NSO (SHA-256 checked)
|
||||
void ldr_map_image(const char *dir, const char *file, uint64_t base, uint64_t *text_end, uint64_t *data_off);
|
||||
// its segments at base (reserved by the caller), not
|
||||
// linked: the plugin service run's rtld (plugin_init.c)
|
||||
uint8_t *ldr_read_verified(const char *dir, const char *file, size_t *len); // whole file, SHA-256 checked against
|
||||
// modules.c (fatal on mismatch); free() it
|
||||
|
||||
@@ -358,13 +372,27 @@ int rt_knob(const char *name, const char *def);
|
||||
void frame_step_present(uint64_t display_ns, int interval); // nvnQueuePresentTexture: the frame's display time and
|
||||
// present interval (60 Hz ticks)
|
||||
// ---- boot (boot.c, main.c)
|
||||
typedef struct { const char *exefs, *romfs, *profile; int boot; double seconds; const char *shot, *gpu; } RtConfig;
|
||||
typedef struct {
|
||||
const char *exefs, *romfs, *profile;
|
||||
int boot;
|
||||
double seconds;
|
||||
const char *shot, *gpu;
|
||||
const char *plugin_init; // --plugin-init DIR: the plugin service run writes its record there (plugin_init.c)
|
||||
} RtConfig;
|
||||
// romfs: directory with romfs.bin and index.json;
|
||||
// frames: --frames N (0 = no limit); shot: --shot
|
||||
// PATH; gpu: TOTK_GPU
|
||||
const RtConfig *rt_config(void);
|
||||
void rt_boot_event(const char *what, Cpu *c, const char *detail); // checkpoint log; "nvn" ends --boot
|
||||
void rt_boot(void); // boot.c
|
||||
void rt_boot_plugin_init(void); // boot.c: --plugin-init, the plugin service run
|
||||
void plugin_init_setup(void); // plugin_init.c: before ldr_load
|
||||
void plugin_init_run(Cpu *c, const RtLoaded *sdk); // plugin_init.c: from the module-init callback; exits
|
||||
void plugins_setup(void); // plugin_init.c: rt_boot, before ldr_load; nothing
|
||||
// without a plugin in the build
|
||||
int plugins_start(Cpu *c, const RtLoaded *sdk); // plugin_init.c: from the module-init callback, before
|
||||
// the game's constructors; 1 when it ran sdk's DT_INIT
|
||||
// (a plugin's init ran after it)
|
||||
// Modes of totk.exe. main.c has run mem_init and parsed the paths; each mode calls ldr_load itself if it needs the
|
||||
// modules, registers its host functions, then seals (hostfn_seal, or implicitly at the first rt_call). Return the
|
||||
// process exit code.
|
||||
|
||||
@@ -38,3 +38,10 @@ extern volatile int kern_test_cv_old_order; // WaitProcessWideKeyAtomi
|
||||
// the key (MNG's pre-fix order, the negative control)
|
||||
extern void (*volatile kern_test_cv_hook)(void); // called right after the lock release, holding the kernel lock
|
||||
extern void (*volatile svc_test_exit_hook)(unsigned code); // svc 0x07 calls it instead of ExitProcess (probes)
|
||||
|
||||
// ---- mod plugin service run (plugin_init.c sets these; the game calls none of the SVCs they serve)
|
||||
extern uint64_t rt_svc_tick_offset; // added to GetSystemTick (0x1E): the service run's second run picks other ticks
|
||||
extern int (*rt_plugin_svc)(Cpu *c, uint32_t imm, uint64_t pc); // with RT_TRACE_PLUGIN: sees each SVC
|
||||
// first; 1: refused, the result is in x0 and the SVC does not run
|
||||
extern uint32_t (*rt_process_memory)(int unmap, uint64_t dst, uint32_t process, uint64_t src, uint64_t size);
|
||||
// Map/UnmapProcessMemory (0x74/0x75); NULL outside the service run: fatal
|
||||
+95
-16
@@ -47,11 +47,12 @@ typedef struct {
|
||||
uint64_t *plt_slot, *plt_stub, nplt; // PLT stubs sorted by GOT slot (ldr_plt_stub), built on first use
|
||||
// statistics
|
||||
uint64_t kinds[K_COUNT], sym_import, sym_own;
|
||||
uint32_t imports, import_to[MOD_COUNT], weak_unresolved, strong_unresolved;
|
||||
uint32_t imports, import_to[RT_MAX_MODULES], weak_unresolved, strong_unresolved;
|
||||
uint32_t init_off;
|
||||
} Mod;
|
||||
|
||||
static Mod mods[MOD_COUNT];
|
||||
static Mod mods[RT_MAX_MODULES];
|
||||
GuestFn ldr_plugin_import;
|
||||
static int nmods;
|
||||
static uint64_t identity_refs, stubs;
|
||||
static int *ovr_used, *obs_used; // per linked table entry, plus probe overrides counted separately
|
||||
@@ -74,6 +75,51 @@ static void hex32(const uint8_t d[32], char out[65]) {
|
||||
for (int i = 0; i < 32; i++) snprintf(out + 2 * i, 3, "%02x", d[i]);
|
||||
}
|
||||
|
||||
static uint8_t *read_path_verified(const char *path, const char *want, size_t *len, const char *hint) {
|
||||
uint8_t *p = read_file(path, len);
|
||||
if (!p) rt_fatal("loader: cannot read %s (%s)", path, hint);
|
||||
uint8_t d[32];
|
||||
char hex[65];
|
||||
rt_sha256(p, *len, d);
|
||||
hex32(d, hex);
|
||||
if (strcmp(hex, want)) rt_fatal("loader: %s has SHA-256 %s, expected %s", path, hex, want);
|
||||
return p;
|
||||
}
|
||||
|
||||
// Mod plugins of the layout whose file is gone (their mod was removed): a rebuild without them is due (cheats.c
|
||||
// rebuild_if_needed), keyed by their paths.
|
||||
static int plugins_gone;
|
||||
static uint32_t plugins_gone_key = 0x811C9DC5u;
|
||||
|
||||
int ldr_plugins_gone(uint32_t *key) {
|
||||
if (plugins_gone) *key = (*key ^ plugins_gone_key) * 0x01000193u;
|
||||
return plugins_gone;
|
||||
}
|
||||
|
||||
// 1 when the plugin file of lay is the one the build lifted; logged when not.
|
||||
static int plugin_file_ok(const RtLayoutEntry *lay) {
|
||||
size_t len;
|
||||
uint8_t *p = read_file(lay->file, &len);
|
||||
if (!p) {
|
||||
rt_log("loader: %s: cannot read %s; the plugin is left out", lay->name, lay->file);
|
||||
if (!plat_file_exists(lay->file)) {
|
||||
plugins_gone++;
|
||||
for (const char *c = lay->file; *c; c++) plugins_gone_key = (plugins_gone_key ^ (uint8_t)*c) * 0x01000193u;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
uint8_t d[32];
|
||||
char hex[65];
|
||||
rt_sha256(p, len, d);
|
||||
free(p);
|
||||
hex32(d, hex);
|
||||
if (strcmp(hex, lay->sha256)) {
|
||||
rt_log("loader: %s: %s changed since the build lifted it; the plugin is left out", lay->name, lay->file);
|
||||
return 0;
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
uint8_t *ldr_read_verified(const char *dir, const char *file, size_t *len) {
|
||||
const char *want = NULL;
|
||||
for (int i = 0; i < rt_exefs_sha256_count; i++)
|
||||
@@ -81,14 +127,7 @@ uint8_t *ldr_read_verified(const char *dir, const char *file, size_t *len) {
|
||||
if (!want) rt_fatal("loader: no expected SHA-256 for %s in modules.c", file);
|
||||
char path[1024];
|
||||
snprintf(path, sizeof path, "%s/%s", dir, file);
|
||||
uint8_t *p = read_file(path, len);
|
||||
if (!p) rt_fatal("loader: cannot read %s (extract the game first: tools/extract.py)", path);
|
||||
uint8_t d[32];
|
||||
char hex[65];
|
||||
rt_sha256(p, *len, d);
|
||||
hex32(d, hex);
|
||||
if (strcmp(hex, want)) rt_fatal("loader: %s has SHA-256 %s, expected %s", path, hex, want);
|
||||
return p;
|
||||
return read_path_verified(path, want, len, "extract the game first: tools/extract.py");
|
||||
}
|
||||
|
||||
// ---- NSO
|
||||
@@ -141,6 +180,32 @@ static void nso_segment(const uint8_t *raw, const Seg *s, uint8_t *dst, const ch
|
||||
}
|
||||
}
|
||||
|
||||
// The plugin service run's stand-in for rtld (plugin_init.c): an ExeFS NSO's image, read and checked like a module but
|
||||
// neither relocated, linked nor sealed. Nothing runs it; exlaunch only reads it while it walks the address space.
|
||||
uint64_t ldr_nso_size(const char *dir, const char *file) {
|
||||
size_t len;
|
||||
uint8_t *raw = ldr_read_verified(dir, file, &len);
|
||||
Seg seg[3];
|
||||
uint64_t size = nso_segments(raw, len, file, seg);
|
||||
free(raw);
|
||||
return size;
|
||||
}
|
||||
|
||||
void ldr_map_image(const char *dir, const char *file, uint64_t base, uint64_t *text_end, uint64_t *data_off) {
|
||||
size_t len;
|
||||
uint8_t *raw = ldr_read_verified(dir, file, &len);
|
||||
Seg seg[3];
|
||||
uint64_t size = nso_segments(raw, len, file, seg);
|
||||
if (!plat_mem_commit(base, size)) rt_fatal("loader: cannot commit %s at %#llx", file, (unsigned long long)base);
|
||||
for (int i = 0; i < 3; i++) {
|
||||
if ((uint64_t)seg[i].mem_off + seg[i].size > size) rt_fatal("loader: %s segment %d exceeds the image", file, i);
|
||||
nso_segment(raw, &seg[i], (uint8_t *)(uintptr_t)(base + seg[i].mem_off), file, i);
|
||||
}
|
||||
free(raw);
|
||||
*text_end = seg[0].mem_off + seg[0].size;
|
||||
*data_off = seg[2].mem_off;
|
||||
}
|
||||
|
||||
// ---- symbols
|
||||
static const char *sym_name(const Mod *m, uint32_t i) { return m->strtab + m->syms[i].name; }
|
||||
static int sym_defined(const Mod *m, uint32_t i) { return m->syms[i].shndx != 0; }
|
||||
@@ -273,9 +338,12 @@ static uint64_t decode_ctor_flag(const Mod *m) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// A plugin (lay->file set) comes from its mod folder, checked against the hash the build took when it lifted it.
|
||||
static void load_module(Mod *m, const char *dir, const RtLayoutEntry *lay) {
|
||||
size_t len;
|
||||
uint8_t *raw = ldr_read_verified(dir, lay->name, &len);
|
||||
uint8_t *raw = lay->file ? read_path_verified(lay->file, lay->sha256, &len, "the mod's code module the build "
|
||||
"lifted; rebuild after changing mods")
|
||||
: ldr_read_verified(dir, lay->name, &len);
|
||||
Seg seg[3];
|
||||
uint64_t size = nso_segments(raw, len, lay->name, seg);
|
||||
if (size != lay->size)
|
||||
@@ -291,6 +359,7 @@ static void load_module(Mod *m, const char *dir, const RtLayoutEntry *lay) {
|
||||
}
|
||||
free(raw);
|
||||
m->pub.name = lay->name;
|
||||
m->pub.plugin = lay->file != NULL;
|
||||
m->pub.base = lay->base;
|
||||
m->pub.size = size;
|
||||
m->pub.text_end = seg[0].mem_off + seg[0].size;
|
||||
@@ -328,7 +397,11 @@ static Res resolve(Mod *m, uint32_t si, int jump) {
|
||||
uint64_t guest = sym_defined(m, si) ? m->pub.base + m->syms[si].value : find_symbol(name, NULL);
|
||||
GuestFn f;
|
||||
Res r;
|
||||
if (jump && (f = hostfn_observer(name))) {
|
||||
if (jump && m->pub.plugin && ldr_plugin_import) { // the plugin service run counts the plugin's calls by name
|
||||
uint64_t to = (f = hostfn_override(name)) ? hostfn_register(name, f) : guest;
|
||||
if (!to) to = hostfn_register(name, unresolved_stub);
|
||||
r = (Res){hostfn_register_observer(name, ldr_plugin_import, to), RES_OBSERVER};
|
||||
} else if (jump && (f = hostfn_observer(name))) {
|
||||
r = (Res){hostfn_register_observer(name, f, guest), RES_OBSERVER};
|
||||
mark_used(1, name);
|
||||
} else if ((f = hostfn_override(name))) {
|
||||
@@ -444,9 +517,15 @@ static void seal(Mod *m) {
|
||||
void ldr_load(const char *exefs_dir) {
|
||||
if (nmods) rt_fatal("ldr_load called twice");
|
||||
if (!mem_layout()->pool) rt_fatal("ldr_load before mem_init");
|
||||
if (rt_layout_count > MOD_COUNT) rt_fatal("loader: %d modules in the layout", rt_layout_count);
|
||||
for (int i = 0; i < rt_layout_count; i++) load_module(&mods[i], exefs_dir, &rt_layout[i]);
|
||||
nmods = rt_layout_count;
|
||||
if (rt_layout_count > RT_MAX_MODULES) rt_fatal("loader: %d modules in the layout", rt_layout_count);
|
||||
int n = 0;
|
||||
for (int i = 0; i < rt_layout_count; i++) {
|
||||
// A mod plugin's file that is gone or changed since the build is left out at play time: its patch set cannot
|
||||
// run (plugin_init.c plugins_start keeps it off); the plugin service run needs it and stops.
|
||||
if (rt_layout[i].file && !rt_config()->plugin_init && !plugin_file_ok(&rt_layout[i])) continue;
|
||||
load_module(&mods[n++], exefs_dir, &rt_layout[i]);
|
||||
}
|
||||
nmods = n;
|
||||
ovr_used = calloc(hostfn_table_size(0) + 1, sizeof(int));
|
||||
obs_used = calloc(hostfn_table_size(1) + 1, sizeof(int));
|
||||
for (int i = 0; i < nmods; i++) {
|
||||
@@ -557,7 +636,7 @@ uint64_t ldr_plt_stub(const char *module, const char *name) {
|
||||
typedef struct {
|
||||
const char *name;
|
||||
uint64_t kinds[K_COUNT], total, sym_import, sym_own;
|
||||
uint32_t imports, to[MOD_COUNT], weak, init_off, init_array;
|
||||
uint32_t imports, to[RT_MAX_MODULES], weak, init_off, init_array;
|
||||
} Expect;
|
||||
static const Expect expect_121[] = {
|
||||
{"main", {508816, 1415, 32, 641}, 510904, 2088, 0, 675, {0, 16, 647}, 12, 0x30, 10673},
|
||||
|
||||
+6
-1
@@ -611,7 +611,7 @@ static int usage(const char *argv0) {
|
||||
"--probe-cfg | --probe-dpi | --probe-mods | --probe-playtime | --probe-kernel | --probe-ipc | --probe-services | --probe-fs | --probe-gfx-abi | "
|
||||
"--probe-nvn | --probe-nvn-tex | --probe-window | --probe-shader | --probe-vkmem | "
|
||||
"--probe-vkexec | --probe-astc | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | "
|
||||
"--prepare-shaders | --warm-shaders | --boot |--seconds S [--shot PNG]]\n", argv0);
|
||||
"--prepare-shaders | --warm-shaders | --boot | --plugin-init DIR | --seconds S [--shot PNG]]\n", argv0);
|
||||
return 2;
|
||||
}
|
||||
|
||||
@@ -700,6 +700,10 @@ int main(int argc, char **argv) {
|
||||
} else if (!strcmp(argv[i], "--shot")) {
|
||||
if (++i == argc) return usage(argv[0]);
|
||||
cfg.shot = argv[i];
|
||||
} else if (!mode && !strcmp(argv[i], "--plugin-init")) { // the plugin service run (plugin_init.c)
|
||||
if (++i == argc) return usage(argv[0]);
|
||||
mode = argv[i - 1];
|
||||
cfg.plugin_init = argv[i];
|
||||
} else if (!mode && !strncmp(argv[i], "--", 2)) {
|
||||
mode = argv[i];
|
||||
} else {
|
||||
@@ -744,5 +748,6 @@ int main(int argc, char **argv) {
|
||||
if (!strcmp(mode, "--prepare-shaders")) return vk_pipeline_prepare_shaders();
|
||||
if (!strcmp(mode, "--warm-shaders")) return vk_pipeline_warm_shaders();
|
||||
if (!strcmp(mode, "--boot")) { cfg.boot = 1; rt_boot(); return 0; }
|
||||
if (!strcmp(mode, "--plugin-init")) { rt_boot_plugin_init(); return 0; }
|
||||
return usage(argv[0]);
|
||||
}
|
||||
+6
-2
@@ -57,7 +57,7 @@ static uint8_t *alias_pages; // 1 per mapped 4 KiB page of the alias regi
|
||||
typedef struct { uint64_t dst, src, size; } StackMap;
|
||||
static StackMap stack_maps[MAX_STACK_MAPS];
|
||||
static unsigned nstack_maps;
|
||||
static uint8_t *module_perm[MOD_COUNT]; // per data page of each module: perm | ATTR_PERMISSION_LOCKED
|
||||
static uint8_t *module_perm[RT_MAX_MODULES]; // per data page of each module: perm | ATTR_PERMISSION_LOCKED
|
||||
|
||||
static uint64_t align_up(uint64_t v, uint64_t a) { return (v + a - 1) & ~(a - 1); }
|
||||
|
||||
@@ -530,7 +530,7 @@ uint32_t mem_unmap_physical(uint64_t a, uint64_t n) { return alias_map(1, a, n);
|
||||
// as taken (Code, RX), so the SDK never picks it.
|
||||
static void taken_gap(uint64_t a, uint64_t *base, uint64_t *size) {
|
||||
uint64_t lo = 0, hi = RT_ADDRESS_LIMIT;
|
||||
RtRange known[3 + MOD_COUNT + 2];
|
||||
RtRange known[3 + RT_MAX_MODULES + 2];
|
||||
int n = 0;
|
||||
known[n++] = (RtRange){L.aslr.base, low_size};
|
||||
known[n++] = L.alias;
|
||||
@@ -562,6 +562,9 @@ static uint64_t run_start(const uint8_t *p, uint64_t i, uint8_t v) { // first
|
||||
return i;
|
||||
}
|
||||
|
||||
// The plugin service run (plugin_init.c) shows rtld, its process-memory aliases and its free pool here.
|
||||
void (*mem_query_hook)(uint64_t a, RtMemoryInfo *r);
|
||||
|
||||
uint32_t mem_query(uint64_t a, RtMemoryInfo *out) {
|
||||
RtMemoryInfo r = {.base = a & ~(PAGE - 1), .size = PAGE};
|
||||
uint64_t low = L.aslr.base;
|
||||
@@ -632,6 +635,7 @@ uint32_t mem_query(uint64_t a, RtMemoryInfo *out) {
|
||||
r.size = hi - lo;
|
||||
}
|
||||
plat_mutex_unlock_shared(&lock);
|
||||
if (mem_query_hook) mem_query_hook(a, &r);
|
||||
if (out) *out = r;
|
||||
return R_OK;
|
||||
}
|
||||
|
||||
+142
-9
@@ -26,9 +26,14 @@
|
||||
// .bss) are lines of build-cache/exefs.data, stored into the loaded main before the game runs (mods_exefs_data), as
|
||||
// emulators apply a patch file to the module before its dynamic linker runs.
|
||||
//
|
||||
// Code modules: an exefs/subsdk* file is a plugin that patches main from its own start-up on the Switch, and a replaced
|
||||
// exefs/main is another game code; the port cannot load either (ModInfo.code names them): their mod stays off unless
|
||||
// it is used without its code, which also leaves out the ExeFS patch files the port cannot apply.
|
||||
// Code modules: an exefs/subsdk* file is a plugin that patches main from its own start-up on the Switch (exlaunch).
|
||||
// The setup builds it in (tools/exefs_plugins.py, after the design in #845 by itaskaev2) and writes its line into
|
||||
// build-cache/plugins.status (the format of exefs.status): `plugin`, its patch set mod_<hash> holds its hook sites and
|
||||
// its init runs at every start (plugin_init.c); `unsupported`, the note says why the build refused it. A plugin without
|
||||
// a line waits for the rebuild like an ExeFS patch file; one whose start differed from the build is listed in
|
||||
// build-cache/plugins.stale (mods_plugin_off) and waits for it too. A replaced exefs/main is another game code the port
|
||||
// cannot load. A code module the port cannot run (ModInfo.code names them) keeps its mod off unless it is used without
|
||||
// its code, which also leaves out the ExeFS patch files the port cannot apply.
|
||||
#include <ctype.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
@@ -92,6 +97,8 @@ static void code_unrun(ModInfo *m, const char *name) {
|
||||
snprintf(m->code + k, sizeof m->code - k, "%s%s", k ? ", " : "", name);
|
||||
}
|
||||
|
||||
static void plugin_file(ModInfo *m, const char *leaf);
|
||||
|
||||
// What the ExeFS folder holds: MOD_EXEFS for patches (.ips, .pchtxt), MOD_SUBSDK for code modules (subsdk* plugins, a
|
||||
// replaced main). main.npdm alone is the process's metadata, not code: it counts as neither.
|
||||
static int exefs_cb(const char *name, int kind, uint64_t size, void *ctx) {
|
||||
@@ -100,7 +107,10 @@ static int exefs_cb(const char *name, int kind, uint64_t size, void *ctx) {
|
||||
if (kind != PLAT_FS_FILE) return 1;
|
||||
if (ends_with_ci(name, ".ips") || ends_with_ci(name, ".pchtxt")) {
|
||||
m->kind |= MOD_EXEFS;
|
||||
} else if (!strncasecmp(name, "subsdk", 6) || !strcasecmp(name, "main")) {
|
||||
} else if (!strncasecmp(name, "subsdk", 6)) {
|
||||
m->kind |= MOD_SUBSDK;
|
||||
plugin_file(m, name);
|
||||
} else if (!strcasecmp(name, "main")) {
|
||||
m->kind |= MOD_SUBSDK;
|
||||
code_unrun(m, name);
|
||||
}
|
||||
@@ -202,6 +212,7 @@ static int same_build(const char *hex, size_t n) {
|
||||
|
||||
// build-cache/<name> next to totk.exe, read once into *text; "" when there is none (mods_probe_cache sets the text).
|
||||
static char *status_text, *data_text;
|
||||
static int no_stale_file; // --probe-mods: plugins.stale stays in memory
|
||||
static const char *build_cache(const char *name, char **text) {
|
||||
if (*text) return *text;
|
||||
char path[2100];
|
||||
@@ -217,14 +228,17 @@ static const char *build_cache(const char *name, char **text) {
|
||||
return *text;
|
||||
}
|
||||
|
||||
static char *plugin_status_text, *plugin_stale_text;
|
||||
static const char *exefs_status(void) { return build_cache("exefs.status", &status_text); }
|
||||
static const char *exefs_data(void) { return build_cache("exefs.data", &data_text); }
|
||||
static const char *plugin_status(void) { return build_cache("plugins.status", &plugin_status_text); }
|
||||
static const char *plugin_stale(void) { return build_cache("plugins.stale", &plugin_stale_text); }
|
||||
|
||||
// The status line of key: its kind, built-in sets and note, NUL-terminated copies; 0 when none.
|
||||
static int status_line(const char *key, const char *content, char *kind, char *over, char *note) {
|
||||
// The line of key in a status text: its kind, built-in sets and note, NUL-terminated copies; 0 when none.
|
||||
static int status_in(const char *text, const char *key, const char *content, char *kind, char *over, char *note) {
|
||||
char want[20];
|
||||
snprintf(want, sizeof want, "%s\t%s\t", key, content);
|
||||
for (const char *line = exefs_status(); *line;) {
|
||||
for (const char *line = text; *line;) {
|
||||
const char *end = strchr(line, '\n');
|
||||
size_t len = end ? (size_t)(end - line) : strlen(line);
|
||||
if (len > 18 && !memcmp(line, want, 18)) {
|
||||
@@ -246,6 +260,69 @@ static int status_line(const char *key, const char *content, char *kind, char *o
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int status_line(const char *key, const char *content, char *kind, char *over, char *note) {
|
||||
return status_in(exefs_status(), key, content, kind, over, note);
|
||||
}
|
||||
|
||||
// The reason plugins.stale gives for key and content, or NULL.
|
||||
static const char *stale_reason(const char *key, const char *content, char *out, size_t cap) {
|
||||
char want[20];
|
||||
snprintf(want, sizeof want, "%s\t%s\t", key, content);
|
||||
for (const char *line = plugin_stale(); *line;) {
|
||||
const char *end = strchr(line, '\n');
|
||||
size_t len = end ? (size_t)(end - line) : strlen(line);
|
||||
if (len > 18 && !memcmp(line, want, 18)) {
|
||||
size_t n = len - 18;
|
||||
if (n && line[len - 1] == '\r') n--;
|
||||
snprintf(out, cap, "%.*s", (int)(n < cap ? n : cap - 1), line + 18);
|
||||
return out;
|
||||
}
|
||||
line = end ? end + 1 : line + len;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
// One exefs/subsdk* file of m: a built plugin joins its patch sets, a refused one is code it cannot run, one the build
|
||||
// lacks (or whose start differed from the build) waits for the rebuild.
|
||||
static void plugin_file(ModInfo *m, const char *leaf) {
|
||||
char path[4300], rel[600];
|
||||
snprintf(path, sizeof path, "%s/%s", m->exefs, leaf);
|
||||
size_t root = strlen(dir) + 1 + strlen(m->name) + 1; // m->exefs is <dir>/<mod>/<exefs place>
|
||||
snprintf(rel, sizeof rel, "%s/%s/%s", m->name, strlen(m->exefs) > root ? m->exefs + root : "exefs", leaf);
|
||||
uint64_t n = 0;
|
||||
char *data = (char *)plat_file_read(path, &n);
|
||||
if (!data) {
|
||||
rt_log("mods: %s: cannot read %s", m->name, path);
|
||||
code_unrun(m, leaf);
|
||||
return;
|
||||
}
|
||||
char key[9], content[9], kind[16], over[256], note[256], stale[256];
|
||||
snprintf(key, sizeof key, "%08X", fnv(rel, strlen(rel)));
|
||||
snprintf(content, sizeof content, "%08X", fnv(data, (size_t)n));
|
||||
free(data);
|
||||
if (!status_in(plugin_status(), key, content, kind, over, note) || stale_reason(key, content, stale, sizeof stale)) {
|
||||
m->pending++;
|
||||
m->pending_key = (m->pending_key ^ fnv(key, 8)) * 0x01000193u;
|
||||
m->pending_key = (m->pending_key ^ fnv(content, 8)) * 0x01000193u;
|
||||
return;
|
||||
}
|
||||
if (!strcmp(kind, "plugin") && m->nsets < (int)(sizeof m->sets / sizeof *m->sets) &&
|
||||
m->nplugins < (int)(sizeof m->plugin_keys / sizeof *m->plugin_keys)) {
|
||||
uint32_t k = (uint32_t)strtoul(key, NULL, 16);
|
||||
m->sets[m->nsets++] = k;
|
||||
m->plugin_keys[m->nplugins] = k;
|
||||
m->plugin_contents[m->nplugins++] = (uint32_t)strtoul(content, NULL, 16);
|
||||
const char *hooks = strstr(note, ", ");
|
||||
m->plugin_hooks += hooks ? atoi(hooks + 2) : 0;
|
||||
size_t at = strlen(m->overrides);
|
||||
if (*over) snprintf(m->overrides + at, sizeof m->overrides - at, "%s%s", at ? "," : "", over);
|
||||
} else {
|
||||
code_unrun(m, leaf);
|
||||
if (!m->plugin_note[0]) snprintf(m->plugin_note, sizeof m->plugin_note, "%s", note);
|
||||
}
|
||||
if (!m->plugin_file[0]) snprintf(m->plugin_file, sizeof m->plugin_file, "%s", leaf);
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
ModInfo *m;
|
||||
const char *folder, *rel; // the folder listed and its path in the mod
|
||||
@@ -369,11 +446,12 @@ static int scan_cb(const char *name, int kind, uint64_t size, void *ctx) {
|
||||
int unsupported = exefs_check(m, path);
|
||||
// A mod whose code part the port cannot use is off as a whole: its RomFS files may expect that code. Used without
|
||||
// its code (its switch, or mods_without_code = 1 for every mod), the code modules (m->code) and the ExeFS patch
|
||||
// files the port cannot apply are left out and the rest of the mod is used, when it has a rest.
|
||||
// files the port cannot apply are left out and the rest of the mod is used, when it has a rest. Each plugin of a
|
||||
// mod is built or refused on its own (tools/exefs_plugins.py), so a built plugin is such a rest too.
|
||||
m->no_code = listed(cfg_get("TOTK_MODS_NO_CODE"), m->name);
|
||||
m->without_code = m->no_code || cfg_bool("TOTK_MODS_WITHOUT_CODE");
|
||||
int usable = m->nsets || m->ndata || m->kind & (MOD_ROMFS | MOD_CHEATS);
|
||||
int code_off = m->code[0] && (!m->without_code || !(m->kind & (MOD_ROMFS | MOD_EXEFS | MOD_CHEATS)));
|
||||
int code_off = m->code[0] && (!m->without_code || !(m->kind & (MOD_ROMFS | MOD_EXEFS | MOD_CHEATS) || m->nplugins));
|
||||
m->problem = strchr(m->name, '|') ? MOD_BAD_NAME
|
||||
: code_off ? MOD_PLUGIN
|
||||
: unsupported && (!m->without_code || !usable) ? MOD_EXEFS_UNSUPPORTED
|
||||
@@ -592,6 +670,14 @@ void mods_probe_cache(const char *status, const char *data) {
|
||||
data_text = dup_text(data);
|
||||
}
|
||||
|
||||
void mods_probe_plugins(const char *status, const char *stale) {
|
||||
free(plugin_status_text);
|
||||
free(plugin_stale_text);
|
||||
plugin_status_text = dup_text(status);
|
||||
plugin_stale_text = dup_text(stale);
|
||||
no_stale_file = status != NULL;
|
||||
}
|
||||
|
||||
void mods_rescan(void) {
|
||||
free(mods);
|
||||
mods = NULL;
|
||||
@@ -770,6 +856,53 @@ void mods_exefs_apply(uint64_t base, uint64_t size, uint64_t code_end,
|
||||
code_conflicts();
|
||||
}
|
||||
|
||||
// build-cache/plugins.stale next to totk.exe: one more line (--probe-mods keeps it in memory).
|
||||
static void stale_append(const char *line) {
|
||||
size_t had = plugin_stale_text ? strlen(plugin_stale_text) : 0, n = strlen(line);
|
||||
char *grown = realloc(plugin_stale_text, had + n + 1);
|
||||
if (grown) {
|
||||
if (!plugin_stale_text) grown[0] = 0;
|
||||
memcpy(grown + had, line, n + 1);
|
||||
plugin_stale_text = grown;
|
||||
}
|
||||
char path[2100];
|
||||
if (no_stale_file || !plat_exe_path(path, sizeof path)) return;
|
||||
char *cut = strrchr(path, '/'), *back = strrchr(path, '\\');
|
||||
if (back > cut) cut = back;
|
||||
if (!cut || snprintf(cut + 1, sizeof path - (size_t)(cut + 1 - path), "build-cache/plugins.stale") >=
|
||||
(int)(sizeof path - (size_t)(cut + 1 - path)))
|
||||
return;
|
||||
FILE *f = fopen(path, "ab");
|
||||
if (!f) {
|
||||
rt_log("mods: cannot write %s", path);
|
||||
return;
|
||||
}
|
||||
fputs(line, f);
|
||||
fclose(f);
|
||||
}
|
||||
|
||||
void mods_plugin_off(int set, const char *why) {
|
||||
const ModInfo *owner = set_owner(set);
|
||||
rt_patch_switch(set, 0);
|
||||
if (!owner) return;
|
||||
ModInfo *m = (ModInfo *)owner;
|
||||
snprintf(m->plugin_off, sizeof m->plugin_off, "%s", why);
|
||||
const char *name = rt_patch_name(set);
|
||||
uint32_t key = name ? (uint32_t)strtoul(name + 4, NULL, 16) : 0;
|
||||
for (int k = 0; k < m->nplugins; k++) {
|
||||
if (m->plugin_keys[k] != key) continue;
|
||||
char line[400];
|
||||
snprintf(line, sizeof line, "%08X\t%08X\t%s\n", key, m->plugin_contents[k], why);
|
||||
stale_append(line);
|
||||
}
|
||||
rt_log("mods: %s: its plugin stays off this run (%s); it waits for the rebuild at the next start", m->name, why);
|
||||
int kept = 0; // the code conflicts counted anew without the set
|
||||
for (int i = 0; i < nconflicts; i++)
|
||||
if (conflicts[i].part != MOD_PART_CODE) conflicts[kept++] = conflicts[i];
|
||||
nconflicts = kept;
|
||||
code_conflicts();
|
||||
}
|
||||
|
||||
void mods_exefs_switch(void) {
|
||||
mods_scan();
|
||||
const RtLoaded *main_mod = NULL;
|
||||
|
||||
@@ -53,6 +53,13 @@ typedef struct {
|
||||
uint32_t data_bytes; // the bytes its usable ExeFS patch files write into main's data, as their
|
||||
// status lines count them ("data <bytes>")
|
||||
char overrides[256]; // the port's own patch sets its patches replace, comma separated
|
||||
// Its exefs/subsdk* plugins (build-cache/plugins.status, tools/exefs_plugins.py): the built ones are patch sets in
|
||||
// sets[] whose init runs at start (plugin_init.c); a refused one is a code module it cannot run (code).
|
||||
int nplugins, plugin_hooks; // built plugins and their hooks
|
||||
uint32_t plugin_keys[4], plugin_contents[4]; // their file and content hashes, for plugins.stale
|
||||
char plugin_file[64]; // the first plugin's file name
|
||||
char plugin_note[256]; // why the build refused a plugin ("" none)
|
||||
char plugin_off[256]; // why its plugin stays off this run: its start differed from the build
|
||||
} ModInfo;
|
||||
|
||||
// What two mods both change, the higher one winning: files (romfs.c), bytes of main's data and code sites
|
||||
@@ -105,5 +112,11 @@ void mods_rescan(void);
|
||||
// The ExeFS patch files of the enabled mods that the build lacks, their FNV-1a mixed into *key (cheats.c starts the
|
||||
// rebuild for them).
|
||||
int mods_exefs_pending(uint32_t *key);
|
||||
// From plugin_init.c plugins_start: the plugin patch set `set` stays off this run (why), its mod's row says so,
|
||||
// build-cache/plugins.stale marks its file for the rebuild at the next start, and the code conflicts are counted anew.
|
||||
void mods_plugin_off(int set, const char *why);
|
||||
// --probe-mods: the texts of build-cache/plugins.status and plugins.stale instead of the files; the stale lines
|
||||
// mods_plugin_off adds stay in memory.
|
||||
void mods_probe_plugins(const char *status, const char *stale);
|
||||
// The hex of the game main's build id (64 digits), "" when exefs/main cannot be read.
|
||||
const char *mods_main_build_id(void);
|
||||
@@ -197,6 +197,111 @@ static void probe_code(const char *root) {
|
||||
remove_tree(root);
|
||||
}
|
||||
|
||||
// build-cache/plugins.status lines (tools/exefs_plugins.py) that refuse the probe's subsdk9 files of E and H: they stay
|
||||
// code the port cannot run.
|
||||
static const char *refused_plugins(void) {
|
||||
static char text[256];
|
||||
snprintf(text, sizeof text, "%08X\t%08X\tunsupported\t-\tthe init called svc 0x40\n"
|
||||
"%08X\t%08X\tunsupported\t-\tthe init called svc 0x40\n", fnv("E/exefs/subsdk9", 15), fnv("probe", 5),
|
||||
fnv("H/exefs/subsdk9", 15), fnv("probe", 5));
|
||||
return text;
|
||||
}
|
||||
|
||||
// Mod plugins (build-cache/plugins.status, plugins.stale; mods.c plugin_file, mods_plugin_off) in a folder of their own:
|
||||
// P RomFS + a built plugin (7 hooks) its set on, ranked by the order; a contested site with Q's patch file
|
||||
// Q an ExeFS patch file on P's site
|
||||
// R a plugin the build has no line for waits for the rebuild
|
||||
// S RomFS + a refused plugin off, the reason kept; used without its code, the rest runs
|
||||
// T a built plugin + a refused one each judged alone: off for the refused one, its reason kept; used
|
||||
// without its code, the built one's set runs
|
||||
// Then P's start differs from the build (mods_plugin_off): its set goes off, Q's code runs at the site, the conflict
|
||||
// goes, a stale line marks P's file, and the next scan has P wait for the rebuild.
|
||||
static void probe_plugins(const char *root) {
|
||||
remove_tree(root);
|
||||
plat_file_mkdir_p(root);
|
||||
const char *bid = *mods_main_build_id() ? mods_main_build_id() : "0";
|
||||
char q[160];
|
||||
snprintf(q, sizeof q, "Q/exefs/%s.ips", bid);
|
||||
static const char *const files[] = {"P/romfs/p.bin", "P/exefs/subsdk9", "R/exefs/subsdk9", "S/romfs/s.bin",
|
||||
"S/exefs/subsdk9", "T/exefs/subsdk8", "T/exefs/subsdk9"};
|
||||
for (size_t i = 0; i < sizeof files / sizeof *files; i++) put(root, files[i]);
|
||||
put(root, q);
|
||||
static char names[4][16];
|
||||
static const char *const table[] = {"fps60", names[1], names[2], names[3]};
|
||||
set_name(names[1], sizeof names[1], "P/exefs/subsdk9");
|
||||
set_name(names[2], sizeof names[2], q);
|
||||
set_name(names[3], sizeof names[3], "T/exefs/subsdk9");
|
||||
char status[256], plugins[512];
|
||||
snprintf(status, sizeof status, "%08X\t%08X\tcode\t-\t\n", fnv(q, strlen(q)), fnv("probe", 5));
|
||||
snprintf(plugins, sizeof plugins, "%08X\t%08X\tplugin\tfps60\tplugin0, 7 hooks\n"
|
||||
"%08X\t%08X\tunsupported\t-\tthe code its hooks reach has svc 0x40\n"
|
||||
"%08X\t%08X\tplugin\t-\tplugin1, 2 hooks\n"
|
||||
"%08X\t%08X\tunsupported\t-\tthe run was refused: svc 0x1 at plugin2+0x50a0\n", fnv("P/exefs/subsdk9", 15),
|
||||
fnv("probe", 5), fnv("S/exefs/subsdk9", 15), fnv("probe", 5), fnv("T/exefs/subsdk9", 15), fnv("probe", 5),
|
||||
fnv("T/exefs/subsdk8", 15), fnv("probe", 5));
|
||||
static const int16_t s0[] = {1, 2};
|
||||
static const RtPatchPick picks[] = {{"main", 0x100, s0, 2}};
|
||||
mods_probe_cache(status, "");
|
||||
mods_probe_plugins(plugins, "");
|
||||
cfg_set("TOTK_MODS_DIR", root);
|
||||
cfg_set("TOTK_MODS_DISABLED", "");
|
||||
cfg_set("TOTK_MODS_NO_CODE", "");
|
||||
for (int round = 0; round < 2; round++) {
|
||||
cfg_set("TOTK_MODS_ORDER", round ? "Q|P" : "P|Q");
|
||||
mods_rescan();
|
||||
mods_scan();
|
||||
rt_patch_probe_table(table, 4, picks, 1);
|
||||
rt_patch_switch(0, 1);
|
||||
mods_exefs_apply(0, 0, 0, NULL);
|
||||
const ModInfo *pm = by_name("P"), *qm = by_name("Q"), *rm = by_name("R"), *sm = by_name("S"),
|
||||
*tm = by_name("T");
|
||||
CHECK(pm && qm && rm && sm && tm, "plugin scan: five mods");
|
||||
if (!pm || !qm || !rm || !sm || !tm) break;
|
||||
CHECK(pm->problem == MOD_OK && pm->nsets == 1 && pm->nplugins == 1 && pm->plugin_hooks == 7 && !pm->code[0] &&
|
||||
EQ(pm->overrides, "fps60"), "a built plugin: its set, its hooks, the built-in set it replaces");
|
||||
CHECK(rt_patch_enabled(1) && !rt_patch_enabled(0), "the plugin's set on, the built-in set it replaces off");
|
||||
CHECK(rt_patch_pick_winner(0) == (round ? 1 : 0), round ? "Q above P: Q's patch runs at their site"
|
||||
: "P above Q: P's plugin hook runs at their site");
|
||||
CHECK(mods_conflict_count() == 1 && mods_conflict_at(0)->part == MOD_PART_CODE &&
|
||||
mods_conflict_at(0)->winner == (round ? qm : pm)->id,
|
||||
"the conflict table: a plugin site against a patch file goes by the order");
|
||||
CHECK(rm->problem == MOD_EXEFS_LATER && rm->pending == 1, "a plugin the build lacks waits for the rebuild");
|
||||
CHECK(sm->problem == MOD_PLUGIN && EQ(sm->code, "subsdk9") && strstr(sm->plugin_note, "svc 0x40"),
|
||||
"a refused plugin keeps its mod off and says why");
|
||||
CHECK(tm->problem == MOD_PLUGIN && tm->nplugins == 1 && tm->plugin_hooks == 2 && EQ(tm->code, "subsdk8") &&
|
||||
strstr(tm->plugin_note, "svc 0x1") && !rt_patch_enabled(3),
|
||||
"a built plugin beside a refused one: the mod off, the refused one's reason on its row");
|
||||
}
|
||||
// P above Q, then P's start differs from the build
|
||||
cfg_set("TOTK_MODS_ORDER", "P|Q");
|
||||
mods_rescan();
|
||||
mods_scan();
|
||||
rt_patch_probe_table(table, 4, picks, 1);
|
||||
mods_exefs_apply(0, 0, 0, NULL);
|
||||
mods_plugin_off(1, "probe: 1 of the words the build recorded were not written");
|
||||
const ModInfo *pm = by_name("P");
|
||||
CHECK(!rt_patch_enabled(1) && rt_patch_pick_winner(0) == 1, "a plugin off at start: the other mod's patch runs");
|
||||
CHECK(pm && strstr(pm->plugin_off, "were not written"), "its row says why");
|
||||
CHECK(mods_conflict_count() == 0, "its conflict goes with it");
|
||||
cfg_set("TOTK_MODS_NO_CODE", "S|T");
|
||||
mods_rescan();
|
||||
mods_scan();
|
||||
rt_patch_probe_table(table, 4, picks, 1);
|
||||
mods_exefs_apply(0, 0, 0, NULL);
|
||||
pm = by_name("P");
|
||||
const ModInfo *sm = by_name("S"), *tm = by_name("T");
|
||||
CHECK(pm && pm->problem == MOD_EXEFS_LATER && pm->pending == 1, "the next start: its file waits for the rebuild");
|
||||
CHECK(sm && sm->problem == MOD_OK && sm->plugin_note[0], "a refused plugin used without its code: the rest runs");
|
||||
CHECK(tm && tm->problem == MOD_OK && tm->plugin_note[0] && rt_patch_enabled(3),
|
||||
"a built plugin beside a refused one, used without its code: the built one's set runs");
|
||||
rt_patch_probe_table(NULL, 0, NULL, 0);
|
||||
cfg_set("TOTK_MODS_NO_CODE", "");
|
||||
cfg_set("TOTK_MODS_ORDER", "");
|
||||
mods_probe_cache(NULL, NULL);
|
||||
mods_probe_plugins(refused_plugins(), "");
|
||||
remove_tree(root);
|
||||
}
|
||||
|
||||
// Mods with code the port cannot run (a code module, an ExeFS patch file it cannot apply): used without that code
|
||||
// by their own switch (mods_no_code) or by mods_without_code for every mod.
|
||||
static void probe_without_code(const char *root) {
|
||||
@@ -292,6 +397,7 @@ int probe_mods(void) {
|
||||
if (back > cut) cut = back;
|
||||
if (!cut) return 1;
|
||||
strcpy(cut + 1, "mods-probe");
|
||||
mods_probe_plugins(refused_plugins(), ""); // not the build's own plugins.status
|
||||
remove_tree(root);
|
||||
plat_file_mkdir_p(root);
|
||||
static const char *const files[] = {
|
||||
@@ -362,6 +468,8 @@ int probe_mods(void) {
|
||||
probe_data(root);
|
||||
probe_without_code(root);
|
||||
probe_code(root);
|
||||
probe_plugins(root);
|
||||
mods_probe_plugins(NULL, NULL);
|
||||
printf("mods: %s\n", fails ? "FAIL" : "ok");
|
||||
int table = restbl_probe();
|
||||
return fails || table ? 1 : 0;
|
||||
|
||||
@@ -379,6 +379,17 @@ static inline char *plat_last_sep(const char *path) {
|
||||
if (*p == '/' || *p == '\\') last = (char *)p;
|
||||
return last;
|
||||
}
|
||||
// 1 when path does not depend on the working directory: it starts with a separator (a root or UNC path) or, on
|
||||
// Windows, with a drive letter and a separator.
|
||||
static inline int plat_path_absolute(const char *path) {
|
||||
if (path[0] == '/' || path[0] == '\\') return 1;
|
||||
#ifdef _WIN32
|
||||
char c = path[0];
|
||||
return ((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z')) && path[1] == ':' && (path[2] == '/' || path[2] == '\\');
|
||||
#else
|
||||
return 0;
|
||||
#endif
|
||||
}
|
||||
// Environment variable as UTF-8 into out; its length, or 0 when unset, empty or too long for cap.
|
||||
size_t plat_getenv(const char *name, char *out, size_t cap);
|
||||
// The user's UI language as a BCP-47 tag ("ru-RU", "en-US"); "" when the system gives none. Valid for the process.
|
||||
|
||||
@@ -0,0 +1,703 @@
|
||||
// Mod plugins' inits (recomp/layout.py plugin<N>, an exlaunch plugin such as a mod's exefs/subsdk9), run natively on
|
||||
// this runtime's own kernel and sdk. The design follows #845 by itaskaev2, with the init run natively instead of under
|
||||
// Unicorn, and run again at every start instead of an image taken after one run at build time.
|
||||
//
|
||||
// The service run: totk.exe --plugin-init DIR runs the plugins' DT_INITs, records what the inits did and exits before
|
||||
// any game module's constructor, nnMain, graphics or audio. TOTK_PLUGIN_ONLY names the one plugin module whose init
|
||||
// runs ("-": none); the others stay loaded, as they are at play. tools/exefs_plugins.py runs each plugin on its own,
|
||||
// so a plugin that is refused, crashes or hangs costs only itself; it checks each record and writes the plugin's patch
|
||||
// set (its hook sites, recomp/patches.py `plugin` lines).
|
||||
//
|
||||
// At every start (plugins_start, from the module-init callback in boot.c): when the build has a plugin whose patch set
|
||||
// its mod switched on, sdk's DT_INIT runs first, then the plugin's, in the same environment as in the service run, and
|
||||
// then the game's constructors. That is the order of the Switch's rtld, which runs the modules' DT_INITs from the last
|
||||
// loaded to the first (oss-rtld, librtld/source/ld.cpp __rtld_modules_init walks g_pAutoLoadList backwards; the list is
|
||||
// filled in address order by __rtld_relocate_modules in librtld/source/relocation.cpp: main, subsdk*, sdk), so sdk
|
||||
// runs before a subsdk plugin and the plugin before the game's main. The plugin keeps what its init made: heap blocks,
|
||||
// its GOT, its alias addresses. What the init wrote into the game modules (its hook branches) and the plugin's .text
|
||||
// after it (its stubs and trampolines) are compared with the record the build used (patches.h RT_PLUGIN_WRITES and
|
||||
// RT_PLUGINS): equal, the plugin's set stays on and the generated hook sites call into it; different, the set goes off
|
||||
// for this run, the reason is logged and the mod waits for a rebuild (mods.c mods_plugin_off). Each plugin is compared
|
||||
// right after its own init, and the branches it wrote into main's .text are taken back before the next init: the next
|
||||
// plugin sees the game's own bytes, as in its own service run (two plugins hooking one site each relocate the game's
|
||||
// instruction; the generated site runs the one the Mods tab order picks). The lifted code is the code that runs, and
|
||||
// main keeps the game's bytes.
|
||||
// Without a plugin in the build none of this runs (RT_PLUGIN_COUNT 0).
|
||||
//
|
||||
// What exlaunch's init needs from the process, and how this run gives it:
|
||||
// - Its module walk (exl::util::FindModules) reads QueryMemory from 0 up and takes every Code RX, Code R, CodeData
|
||||
// RW run as a module, the first being rtld and the second main. Our layout has no rtld and reports the host's
|
||||
// ranges as Code RX (memory.c taken_gap), which would make it take a gap for a module. mem_query_hook shows the
|
||||
// game's own rtld NSO (exefs/rtld, read, never run) right below main, and the host's ranges as Inaccessible.
|
||||
// - It picks a random free range in the ASLR region (GetSystemTick seeds it) for its process-memory aliases. Our
|
||||
// ASLR region is almost all host ranges, so a free pool of POOL_SIZE is reserved after the alias region.
|
||||
// - GetInfo 65001 (svc.c) gives it the current process handle; MapProcessMemory maps main's .text and its own pages
|
||||
// at the picked address. Guest addresses are host addresses and the modules are not file-backed, so an alias is a
|
||||
// copy: its pages hold the source's bytes when mapped, and the words written through it are written into the
|
||||
// source when it is unmapped and when the init returns. That is how the stubs reach the plugin's image and the
|
||||
// hook branches reach main's .text.
|
||||
// - nn::oe::GetDisplayVersion is answered with this build's game version, so the plugin picks its table for the
|
||||
// installed game without the applet services.
|
||||
// Only svcs the init needs are allowed (QueryMemory, GetSystemTick, OutputDebugString, GetInfo 2-5, 12-15 and 65001,
|
||||
// Map/UnmapProcessMemory); any other one, such as CreateSession and ReplyAndReceive of exlaunch's process-handle
|
||||
// fallback, ends the run with the reason in the record.
|
||||
//
|
||||
// The record, in DIR: record.json (modules, which plugins' inits ran, the writes into every game module as words, the
|
||||
// alias calls, svc and import counts, the tick offset, a refusal), plugin<N>.img (the plugin's image after its init,
|
||||
// all segments) and plugin<N>.pre.img (the same after loading and relocation, before the init).
|
||||
// TOTK_PLUGIN_TICK_OFFSET adds to every GetSystemTick, so two runs pick different aliases.
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "cfg.h"
|
||||
#include "dispatch.h"
|
||||
#include "host.h"
|
||||
#include "kernel.h"
|
||||
#include "mods.h"
|
||||
#include "platform/platform.h"
|
||||
|
||||
#if __has_include("patches.h")
|
||||
#include "patches.h"
|
||||
#endif
|
||||
#ifndef RT_PLUGIN_COUNT
|
||||
#define RT_PLUGIN_COUNT 0
|
||||
#endif
|
||||
|
||||
#define PAGE 0x1000ull
|
||||
#define GiB (1ull << 30)
|
||||
#define POOL_SIZE (64 * GiB)
|
||||
#define POOL_ALIGN (4 * GiB)
|
||||
#define RTLD_ALIGN 0x200000ull
|
||||
#define MAX_ALIASES 256
|
||||
#define MAX_IMPORTS 64
|
||||
#define CUR_PROCESS 0xFFFF8001u
|
||||
// Horizon memory states and permissions of QueryMemory, and results.
|
||||
enum { ST_FREE = 0, ST_CODE = 3, ST_CODEDATA = 4, ST_SHARED_CODE = 0xF, ST_INACCESSIBLE = 0x10 };
|
||||
enum { PERM_R = 1, PERM_RW = 3, PERM_RX = 5 };
|
||||
#define R_OK 0u
|
||||
#define R_INVALID_ADDRESS 0xcc01u
|
||||
#define R_INVALID_STATE 0xd401u
|
||||
#define R_INVALID_HANDLE 0xe401u
|
||||
#define R_LIMIT_REACHED 0x10801u
|
||||
#define SYM_DISPLAY_VERSION "_ZN2nn2oe17GetDisplayVersionEPNS0_14DisplayVersionE"
|
||||
|
||||
typedef struct {
|
||||
uint64_t dst, src, size;
|
||||
uint8_t *snap; // the source's bytes when mapped: the words that differ from it were written through the alias
|
||||
int live, unmapped;
|
||||
} Alias;
|
||||
static Alias aliases[MAX_ALIASES];
|
||||
static int naliases;
|
||||
static uint64_t rtld_base, rtld_size, rtld_text_end, rtld_data_off;
|
||||
static uint64_t pool_base;
|
||||
static int running; // a plugin's init runs: svcs and imports are counted
|
||||
static int play; // plugins_start: a refused SVC fails the plugin instead of ending the process
|
||||
static char failed[256]; // play: why the running init was refused
|
||||
static uint64_t svc_count[0x80], info_count[16], info_65001;
|
||||
static struct { const char *name; uint64_t calls; } imports[MAX_IMPORTS];
|
||||
static int nimports;
|
||||
static uint8_t **pre; // per loaded module: its image before the plugins' inits
|
||||
static uint8_t *ran; // service run, per loaded module: its init ran
|
||||
|
||||
static uint64_t align_up(uint64_t v, uint64_t a) { return (v + a - 1) & ~(a - 1); }
|
||||
|
||||
static void clip(uint64_t a, uint64_t *lo, uint64_t *hi, uint64_t b, uint64_t n) {
|
||||
if (!n) return;
|
||||
uint64_t e = b + n;
|
||||
if (e <= a && e > *lo) *lo = e;
|
||||
if (b > a && b < *hi) *hi = b;
|
||||
}
|
||||
|
||||
static void clip_aliases(uint64_t a, uint64_t *lo, uint64_t *hi) {
|
||||
for (int i = 0; i < naliases; i++)
|
||||
if (aliases[i].live) clip(a, lo, hi, aliases[i].dst, aliases[i].size);
|
||||
}
|
||||
|
||||
static int in_module(uint64_t a) {
|
||||
for (int i = 0; i < ldr_count(); i++)
|
||||
if (a - ldr_module(i)->base < ldr_module(i)->size) return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
// ---- QueryMemory as exlaunch expects it (memory.c mem_query_hook)
|
||||
static void query_hook(uint64_t a, RtMemoryInfo *r) {
|
||||
for (int i = 0; i < naliases; i++) {
|
||||
const Alias *al = &aliases[i];
|
||||
if (al->live && a - al->dst < al->size) {
|
||||
*r = (RtMemoryInfo){.base = al->dst, .size = al->size, .state = ST_SHARED_CODE, .perm = PERM_RW};
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (a - rtld_base < rtld_size) {
|
||||
uint64_t off = a - rtld_base, text = align_up(rtld_text_end, PAGE);
|
||||
if (off < text) *r = (RtMemoryInfo){.base = rtld_base, .size = text, .state = ST_CODE, .perm = PERM_RX};
|
||||
else if (off < rtld_data_off)
|
||||
*r = (RtMemoryInfo){.base = rtld_base + text, .size = rtld_data_off - text, .state = ST_CODE, .perm = PERM_R};
|
||||
else
|
||||
*r = (RtMemoryInfo){.base = rtld_base + rtld_data_off, .size = rtld_size - rtld_data_off,
|
||||
.state = ST_CODEDATA, .perm = PERM_RW};
|
||||
return;
|
||||
}
|
||||
uint64_t lo = r->base, hi = r->base + r->size;
|
||||
if (a - pool_base < POOL_SIZE) {
|
||||
lo = pool_base, hi = pool_base + POOL_SIZE;
|
||||
*r = (RtMemoryInfo){.state = ST_FREE};
|
||||
} else if (r->state == ST_CODE && !in_module(a)) { // a host range (memory.c taken_gap)
|
||||
clip(a, &lo, &hi, rtld_base, rtld_size);
|
||||
clip(a, &lo, &hi, pool_base, POOL_SIZE);
|
||||
*r = (RtMemoryInfo){.state = ST_INACCESSIBLE};
|
||||
} else if (r->state != ST_FREE) {
|
||||
return;
|
||||
}
|
||||
clip_aliases(a, &lo, &hi);
|
||||
r->base = lo;
|
||||
r->size = hi - lo;
|
||||
}
|
||||
|
||||
// ---- Map/UnmapProcessMemory (svc.c rt_process_memory)
|
||||
static const RtLoaded *module_holding(uint64_t a, uint64_t n) {
|
||||
for (int i = 0; i < ldr_count(); i++) {
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
if (a >= m->base && n <= m->size && a - m->base <= m->size - n) return m;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
// Writes the words changed through an alias into its source. The source's read-only pages (sealed .text and .rodata)
|
||||
// are made writable for the copy and sealed again.
|
||||
static void flush(Alias *al) {
|
||||
const RtLoaded *m = module_holding(al->src, al->size);
|
||||
const uint8_t *now = (const uint8_t *)(uintptr_t)al->dst;
|
||||
for (uint64_t off = 0; off < al->size; off += PAGE) {
|
||||
if (!memcmp(now + off, al->snap + off, PAGE)) continue;
|
||||
uint64_t page = al->src + off;
|
||||
int sealed = page - m->base < m->data_off;
|
||||
if (sealed && !plat_mem_protect(page, PAGE, PLAT_PROT_RW)) rt_fatal("plugin-init: cannot unseal %#llx", (unsigned long long)page);
|
||||
for (uint64_t w = 0; w < PAGE; w += 4)
|
||||
if (memcmp(now + off + w, al->snap + off + w, 4)) memcpy((void *)(uintptr_t)(page + w), now + off + w, 4);
|
||||
if (sealed && !plat_mem_protect(page, PAGE, PLAT_PROT_READ)) rt_fatal("plugin-init: cannot seal %#llx", (unsigned long long)page);
|
||||
}
|
||||
memcpy(al->snap, now, al->size); // flushed: a later flush writes only what changes after this one
|
||||
}
|
||||
|
||||
static uint32_t process_memory(int unmap, uint64_t dst, uint32_t process, uint64_t src, uint64_t size) {
|
||||
if (process != CUR_PROCESS) return R_INVALID_HANDLE;
|
||||
if (!size || ((dst | src | size) & (PAGE - 1))) return R_INVALID_ADDRESS;
|
||||
if (unmap) {
|
||||
for (int i = 0; i < naliases; i++) {
|
||||
Alias *al = &aliases[i];
|
||||
if (!al->live || al->dst != dst || al->src != src || al->size != size) continue;
|
||||
flush(al);
|
||||
if (!plat_mem_decommit(dst, size)) rt_fatal("plugin-init: cannot unmap the alias at %#llx", (unsigned long long)dst);
|
||||
free(al->snap);
|
||||
al->snap = NULL;
|
||||
al->live = 0;
|
||||
al->unmapped = 1;
|
||||
return R_OK;
|
||||
}
|
||||
return R_INVALID_STATE;
|
||||
}
|
||||
if (!module_holding(src, size)) return R_INVALID_STATE; // only the modules' own memory
|
||||
RtMemoryInfo q;
|
||||
mem_query(dst, &q);
|
||||
if (q.state != ST_FREE || dst + size > q.base + q.size) return R_INVALID_STATE;
|
||||
if (naliases == MAX_ALIASES) return R_LIMIT_REACHED;
|
||||
if (!plat_mem_commit(dst, size)) return R_INVALID_STATE; // not inside one of our reservations
|
||||
Alias *al = &aliases[naliases];
|
||||
al->snap = malloc(size);
|
||||
if (!al->snap) rt_fatal("plugin-init: out of memory for a %#llx byte alias", (unsigned long long)size);
|
||||
memcpy((void *)(uintptr_t)dst, (const void *)(uintptr_t)src, size);
|
||||
memcpy(al->snap, (const void *)(uintptr_t)src, size);
|
||||
*al = (Alias){dst, src, size, al->snap, 1, 0};
|
||||
naliases++;
|
||||
return R_OK;
|
||||
}
|
||||
|
||||
// ---- svcs and imports
|
||||
static void write_record(const char *refused);
|
||||
|
||||
static void refuse(const char *why) {
|
||||
rt_log("plugin-init: refused: %s", why);
|
||||
if (play) { // the init goes on with an error result; the plugin stays off (plugins_start)
|
||||
if (!failed[0]) snprintf(failed, sizeof failed, "%s", why);
|
||||
return;
|
||||
}
|
||||
write_record(why);
|
||||
fflush(NULL);
|
||||
plat_exit(3);
|
||||
}
|
||||
|
||||
#define R_NOT_IMPLEMENTED 0xF601u
|
||||
|
||||
static int plugin_svc(Cpu *c, uint32_t imm, uint64_t pc) {
|
||||
if (!running) return 0;
|
||||
svc_count[imm & 0x7F]++;
|
||||
char why[256], at[128];
|
||||
switch (imm) {
|
||||
case 0x06: case 0x1E: case 0x27: case 0x74: case 0x75: return 0;
|
||||
case 0x29: {
|
||||
uint32_t id = (uint32_t)c->x[1];
|
||||
if ((id >= 2 && id <= 5) || (id >= 12 && id <= 15)) {
|
||||
info_count[id]++;
|
||||
return 0;
|
||||
}
|
||||
if (id == 65001) {
|
||||
info_65001++;
|
||||
return 0;
|
||||
}
|
||||
snprintf(why, sizeof why, "GetInfo id %u at %s", id, ldr_describe(pc, at, sizeof at));
|
||||
refuse(why);
|
||||
c->x[0] = R_NOT_IMPLEMENTED;
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
snprintf(why, sizeof why, "svc %#x at %s (only QueryMemory, GetSystemTick, OutputDebugString, GetInfo 2-5, 12-15 "
|
||||
"and 65001, Map/UnmapProcessMemory are served)", imm, ldr_describe(pc, at, sizeof at));
|
||||
refuse(why);
|
||||
c->x[0] = R_NOT_IMPLEMENTED;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Every JUMP_SLOT import of a plugin binds here (loader.c resolve): counted by name, then the guest definition runs,
|
||||
// except nn::oe::GetDisplayVersion, which gets this build's game version.
|
||||
static void plugin_import(Cpu *c) {
|
||||
const char *name = hostfn_name(c->pc);
|
||||
int i = 0;
|
||||
while (i < nimports && strcmp(imports[i].name, name)) i++;
|
||||
if (i == nimports && nimports < MAX_IMPORTS) imports[nimports++].name = name;
|
||||
if (i < MAX_IMPORTS && running) imports[i].calls++;
|
||||
if (!strcmp(name, SYM_DISPLAY_VERSION)) { // DisplayVersion { char value[16]; }
|
||||
char v[16] = {0};
|
||||
snprintf(v, sizeof v, "%d.%d.%d", rt_game_version / 100, rt_game_version / 10 % 10, rt_game_version % 10);
|
||||
memcpy((void *)(uintptr_t)c->x[0], v, sizeof v);
|
||||
return;
|
||||
}
|
||||
rt_original(c);
|
||||
}
|
||||
|
||||
// ---- the record
|
||||
static FILE *open_out(const char *file, const char *mode) {
|
||||
char path[PLAT_PATH_MAX];
|
||||
snprintf(path, sizeof path, "%s/%s", rt_config()->plugin_init, file);
|
||||
FILE *f = fopen(path, mode);
|
||||
if (!f) rt_fatal("plugin-init: cannot write %s", path);
|
||||
return f;
|
||||
}
|
||||
|
||||
static void write_image(const char *file, const void *p, uint64_t n) {
|
||||
FILE *f = open_out(file, "wb");
|
||||
if (fwrite(p, 1, n, f) != n || fclose(f)) rt_fatal("plugin-init: cannot write %s", file);
|
||||
}
|
||||
|
||||
static void sha256_hex(const void *p, uint64_t n, char out[65]) {
|
||||
uint8_t d[32];
|
||||
rt_sha256(p, n, d);
|
||||
for (int i = 0; i < 32; i++) snprintf(out + 2 * i, 3, "%02x", d[i]);
|
||||
}
|
||||
|
||||
static void json_str(FILE *f, const char *s) {
|
||||
fputc('"', f);
|
||||
for (; *s; s++) {
|
||||
if (*s == '"' || *s == '\\') fputc('\\', f);
|
||||
if ((unsigned char)*s < 0x20) fprintf(f, "\\u%04x", *s);
|
||||
else fputc(*s, f);
|
||||
}
|
||||
fputc('"', f);
|
||||
}
|
||||
|
||||
// The words of each game module that differ from its image before the plugins' inits: (module, offset, old, new).
|
||||
static void write_writes(FILE *f) {
|
||||
int first = 1;
|
||||
for (int i = 0; i < ldr_count(); i++) {
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
if (m->plugin || !pre || !pre[i]) continue;
|
||||
const uint8_t *now = (const uint8_t *)(uintptr_t)m->base;
|
||||
for (uint64_t off = 0; off < m->size; off += PAGE) {
|
||||
if (!memcmp(now + off, pre[i] + off, PAGE)) continue;
|
||||
for (uint64_t w = off; w < off + PAGE; w += 4) {
|
||||
uint32_t a, b;
|
||||
memcpy(&a, pre[i] + w, 4);
|
||||
memcpy(&b, now + w, 4);
|
||||
if (a == b) continue;
|
||||
// fn_start: a lifted function starts at the word (a b there replaces the function)
|
||||
fprintf(f, "%s\n {\"module\": \"%s\", \"offset\": \"%#llx\", \"old\": \"%#010x\", \"new\": \"%#010x\", "
|
||||
"\"fn_start\": %s}", first ? "" : ",", m->name, (unsigned long long)w, a, b,
|
||||
rt_dispatch_lookup(m->base + w) ? "true" : "false");
|
||||
first = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static void write_record(const char *refused) {
|
||||
for (int i = 0; i < naliases; i++)
|
||||
if (aliases[i].live) flush(&aliases[i]);
|
||||
FILE *f = open_out("record.json", "w");
|
||||
// The marker of generated game-derived files (tools/check_content.py): the record never goes into the repository.
|
||||
fprintf(f, "{\n\"marker\": \"%s%s\",\n\"game_version\": %d,\n\"tick_offset\": \"%#llx\",\n\"refused\": ",
|
||||
"TOTK_RECOMP_", "GENERATED_FROM_GAME_CODE", rt_game_version, (unsigned long long)rt_svc_tick_offset);
|
||||
if (refused) json_str(f, refused);
|
||||
else fputs("null", f);
|
||||
fprintf(f, ",\n\"rtld\": {\"base\": \"%#llx\", \"size\": \"%#llx\"},\n\"pool\": {\"base\": \"%#llx\", \"size\": \"%#llx\"},",
|
||||
(unsigned long long)rtld_base, (unsigned long long)rtld_size, (unsigned long long)pool_base,
|
||||
(unsigned long long)POOL_SIZE);
|
||||
fputs("\n\"modules\": [", f);
|
||||
for (int i = 0; i < ldr_count(); i++) {
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
fprintf(f, "%s\n {\"name\": \"%s\", \"base\": \"%#llx\", \"size\": \"%#llx\", \"text_end\": \"%#llx\", "
|
||||
"\"data_off\": \"%#llx\", \"init\": \"%#llx\", \"plugin\": %s",
|
||||
i ? "," : "", m->name, (unsigned long long)m->base, (unsigned long long)m->size,
|
||||
(unsigned long long)m->text_end, (unsigned long long)m->data_off, (unsigned long long)m->init,
|
||||
m->plugin ? "true" : "false");
|
||||
if (m->plugin) fprintf(f, ", \"ran\": %s", ran && ran[i] ? "true" : "false");
|
||||
if (m->plugin && !refused) {
|
||||
char img[64], pre_img[64], hex[65];
|
||||
snprintf(img, sizeof img, "%s.img", m->name);
|
||||
snprintf(pre_img, sizeof pre_img, "%s.pre.img", m->name);
|
||||
write_image(img, (const void *)(uintptr_t)m->base, m->size);
|
||||
if (pre && pre[i]) write_image(pre_img, pre[i], m->size);
|
||||
sha256_hex((const void *)(uintptr_t)m->base, m->size, hex);
|
||||
fprintf(f, ", \"image\": \"%s\", \"image_sha256\": \"%s\", \"pre_image\": \"%s\"", img, hex, pre_img);
|
||||
}
|
||||
if (m->plugin) { // the lifted function starts: a main write may branch straight to one (tools/exefs_plugins.py)
|
||||
RtModule lm[16];
|
||||
int n = rt_linked_modules(lm, 16);
|
||||
for (int k = 0; k < n; k++) {
|
||||
if (strcmp(lm[k].name, m->name)) continue;
|
||||
fputs(", \"functions\": [", f);
|
||||
for (uint64_t j = 0; j < lm[k].count; j++)
|
||||
fprintf(f, "%s\"%#llx\"", j ? ", " : "", (unsigned long long)(lm[k].addrs[j] - m->base));
|
||||
fputc(']', f);
|
||||
}
|
||||
}
|
||||
for (int k = 0; k < rt_layout_count; k++)
|
||||
if (rt_layout[k].file && !strcmp(rt_layout[k].name, m->name)) {
|
||||
fputs(", \"file\": ", f);
|
||||
json_str(f, rt_layout[k].file);
|
||||
fprintf(f, ", \"nso_sha256\": \"%s\"", rt_layout[k].sha256);
|
||||
}
|
||||
fputc('}', f);
|
||||
}
|
||||
fputs("\n],\n\"writes\": [", f);
|
||||
write_writes(f);
|
||||
fputs("\n],\n\"aliases\": [", f);
|
||||
for (int i = 0; i < naliases; i++)
|
||||
fprintf(f, "%s\n {\"dst\": \"%#llx\", \"src\": \"%#llx\", \"size\": \"%#llx\", \"unmapped\": %s}", i ? "," : "",
|
||||
(unsigned long long)aliases[i].dst, (unsigned long long)aliases[i].src,
|
||||
(unsigned long long)aliases[i].size, aliases[i].unmapped ? "true" : "false");
|
||||
fputs("\n],\n\"svcs\": {", f);
|
||||
int first = 1;
|
||||
for (int i = 0; i < 0x80; i++)
|
||||
if (svc_count[i]) fprintf(f, "%s\"%#x\": %llu", first ? "" : ", ", i, (unsigned long long)svc_count[i]), first = 0;
|
||||
fputs("},\n\"get_info\": {", f);
|
||||
first = 1;
|
||||
for (int i = 0; i < 16; i++)
|
||||
if (info_count[i]) fprintf(f, "%s\"%d\": %llu", first ? "" : ", ", i, (unsigned long long)info_count[i]), first = 0;
|
||||
if (info_65001) fprintf(f, "%s\"65001\": %llu", first ? "" : ", ", (unsigned long long)info_65001);
|
||||
fputs("},\n\"imports\": {", f);
|
||||
for (int i = 0; i < nimports; i++) {
|
||||
fputs(i ? ", " : "", f);
|
||||
json_str(f, imports[i].name);
|
||||
fprintf(f, ": %llu", (unsigned long long)imports[i].calls);
|
||||
}
|
||||
fputs("}\n}\n", f);
|
||||
if (fclose(f)) rt_fatal("plugin-init: cannot write record.json");
|
||||
}
|
||||
|
||||
// ---- setup and run
|
||||
static int layout_plugins(void) {
|
||||
int plugins = 0;
|
||||
for (int i = 0; i < rt_layout_count; i++) plugins += rt_layout[i].file != NULL;
|
||||
return plugins;
|
||||
}
|
||||
|
||||
static void environment(void);
|
||||
|
||||
// Before ldr_load: the import observer, the svc hooks, the pool and the stand-in for rtld.
|
||||
void plugin_init_setup(void) {
|
||||
if (!layout_plugins()) rt_fatal("plugin-init: this build's layout has no plugin module (recomp/gen.py --plugins)");
|
||||
plat_file_mkdir_p(rt_config()->plugin_init);
|
||||
const char *off = cfg_get("TOTK_PLUGIN_TICK_OFFSET");
|
||||
rt_svc_tick_offset = off ? strtoull(off, NULL, 0) : 0;
|
||||
environment();
|
||||
}
|
||||
|
||||
// The environment of a plugin's init, the same in the service run and at every start.
|
||||
static void environment(void) {
|
||||
ldr_plugin_import = plugin_import;
|
||||
rt_process_memory = process_memory;
|
||||
rt_plugin_svc = plugin_svc;
|
||||
|
||||
const RtGuestLayout *g = mem_layout();
|
||||
uint64_t main_base = rt_layout[0].base;
|
||||
rtld_size = ldr_nso_size(rt_config()->exefs, "rtld");
|
||||
rtld_base = main_base - align_up(rtld_size, RTLD_ALIGN);
|
||||
for (uint64_t at = align_up(g->alias.base + g->alias.size, POOL_ALIGN); at + POOL_SIZE <= rtld_base && !pool_base;
|
||||
at += POOL_ALIGN)
|
||||
if (plat_mem_reserve(at, POOL_SIZE, 0)) pool_base = at;
|
||||
if (!pool_base) rt_fatal("plugin-init: no free %llu GiB below main for the alias pool", POOL_SIZE / GiB);
|
||||
if (!plat_mem_reserve(rtld_base, align_up(rtld_size, PAGE), 0))
|
||||
rt_fatal("plugin-init: %#llx below main is taken; rtld goes there", (unsigned long long)rtld_base);
|
||||
ldr_map_image(rt_config()->exefs, "rtld", rtld_base, &rtld_text_end, &rtld_data_off);
|
||||
mem_query_hook = query_hook;
|
||||
rt_log("plugin-init: rtld at %#llx+%#llx, alias pool %#llx+%llu GiB, tick offset %#llx", (unsigned long long)rtld_base,
|
||||
(unsigned long long)rtld_size, (unsigned long long)pool_base, POOL_SIZE / GiB,
|
||||
(unsigned long long)rt_svc_tick_offset);
|
||||
}
|
||||
|
||||
// rt_boot, before ldr_load: the plugins' environment when the build has one (nothing otherwise).
|
||||
void plugins_setup(void) {
|
||||
if (!RT_PLUGIN_COUNT || !layout_plugins()) return;
|
||||
play = 1;
|
||||
environment();
|
||||
}
|
||||
|
||||
// From nn::init::Start's module-init callback (boot.c), in place of the game's constructors. rtld initialises the
|
||||
// modules from the last loaded to the first, so on the Switch sdk's DT_INIT runs before a subsdk plugin's; then each
|
||||
// plugin's, the last first. Exits with the record written.
|
||||
void plugin_init_run(Cpu *c, const RtLoaded *sdk) {
|
||||
rt_log("plugin-init: ENTER %s DT_INIT", sdk->name);
|
||||
rt_call(c, sdk->init, 0);
|
||||
pre = calloc((size_t)ldr_count(), sizeof *pre);
|
||||
if (!pre) rt_fatal("plugin-init: out of memory");
|
||||
for (int i = 0; i < ldr_count(); i++) { // the images before the plugins run
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
pre[i] = malloc(m->size);
|
||||
if (!pre[i]) rt_fatal("plugin-init: out of memory for a copy of %s", m->name);
|
||||
memcpy(pre[i], (const void *)(uintptr_t)m->base, m->size);
|
||||
}
|
||||
const char *only = cfg_get("TOTK_PLUGIN_ONLY");
|
||||
ran = calloc((size_t)ldr_count(), 1);
|
||||
if (!ran) rt_fatal("plugin-init: out of memory");
|
||||
for (int i = ldr_count() - 1; i >= 0; i--) {
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
if (!m->plugin) continue;
|
||||
if (only && *only && strcmp(only, m->name)) {
|
||||
rt_log("plugin-init: %s stays loaded, its init does not run (TOTK_PLUGIN_ONLY=%s)", m->name, only);
|
||||
continue;
|
||||
}
|
||||
ran[i] = 1;
|
||||
if (!m->init) refuse("a plugin without DT_INIT");
|
||||
rt_log("plugin-init: ENTER %s DT_INIT at %#llx (%llu init_array entries)", m->name, (unsigned long long)m->init,
|
||||
(unsigned long long)(m->init_array_size / 8));
|
||||
running = 1;
|
||||
rt_trace |= RT_TRACE_PLUGIN;
|
||||
rt_call(c, m->init, 0);
|
||||
rt_trace &= ~(unsigned)RT_TRACE_PLUGIN;
|
||||
running = 0;
|
||||
rt_log("plugin-init: %s DT_INIT returned", m->name);
|
||||
}
|
||||
write_record(NULL);
|
||||
int live = 0;
|
||||
for (int i = 0; i < naliases; i++) live += aliases[i].live;
|
||||
rt_log("plugin-init: record written to %s (%d process-memory maps, %d still mapped)", rt_config()->plugin_init,
|
||||
naliases, live);
|
||||
fflush(NULL);
|
||||
plat_exit(0);
|
||||
}
|
||||
|
||||
// ---- at every start
|
||||
#if RT_PLUGIN_COUNT
|
||||
static const struct { const char *module; int set; const char *sha256; } plugin_tab[] = {
|
||||
#define X(i, mod, set, sha) {mod, set, sha},
|
||||
RT_PLUGINS(X)
|
||||
#undef X
|
||||
};
|
||||
static const struct { int plugin; const char *module; uint64_t off; uint32_t old, now; } write_tab[] = {
|
||||
#define X(p, mod, off, old, now) {p, mod, off, old, now},
|
||||
RT_PLUGIN_WRITES(X)
|
||||
#undef X
|
||||
};
|
||||
#define NWRITES ((int)(sizeof write_tab / sizeof *write_tab))
|
||||
#ifdef RT_PLUGIN_ROOTS
|
||||
static const struct { const char *module; uint64_t off; } root_tab[] = {
|
||||
#define X(mod, off) {mod, off},
|
||||
RT_PLUGIN_ROOTS(X)
|
||||
#undef X
|
||||
};
|
||||
#define NROOTS ((int)(sizeof root_tab / sizeof *root_tab))
|
||||
#else
|
||||
#define NROOTS 0
|
||||
#endif
|
||||
|
||||
static const RtLoaded *loaded(const char *name) {
|
||||
for (int i = 0; i < ldr_count(); i++)
|
||||
if (!strcmp(ldr_module(i)->name, name)) return ldr_module(i);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
// TOTK_PLUGIN_TRACE = 1: the first call of each hook entry (patches.h RT_PLUGIN_ROOTS: callbacks, replace and call
|
||||
// targets) is logged with its count so far; the entries run through count_entry, their dispatch slots rerouted once
|
||||
// at start (dispatch.c rt_dispatch_reroute). The calls inside the plugin are direct and not counted.
|
||||
static struct { uint64_t addr; GuestFn fn; uint64_t calls; } traced[64];
|
||||
static int ntraced;
|
||||
|
||||
static void count_entry(Cpu *c) {
|
||||
for (int i = 0; i < ntraced; i++) {
|
||||
if (traced[i].addr != c->pc) continue;
|
||||
if (__atomic_add_fetch(&traced[i].calls, 1, __ATOMIC_RELAXED) == 1) {
|
||||
char at[128];
|
||||
rt_log("plugins: hook entry %s reached (first call)", ldr_describe(c->pc, at, sizeof at));
|
||||
}
|
||||
__attribute__((musttail)) return traced[i].fn(c);
|
||||
}
|
||||
rt_fatal("plugins: traced entry %#llx without its function", (unsigned long long)c->pc);
|
||||
}
|
||||
|
||||
static void trace_entries(void) {
|
||||
#ifdef RT_PLUGIN_ROOTS
|
||||
for (int i = 0; i < NROOTS && ntraced < 64; i++) {
|
||||
const RtLoaded *m = loaded(root_tab[i].module);
|
||||
if (!m) continue;
|
||||
uint64_t a = m->base + root_tab[i].off;
|
||||
GuestFn f = rt_dispatch_lookup(a);
|
||||
if (!f) continue;
|
||||
traced[ntraced] = (typeof(traced[0])){a, f, 0};
|
||||
if (rt_dispatch_reroute(a, count_entry)) ntraced++;
|
||||
else rt_log("plugins: trace: cannot reroute %#llx", (unsigned long long)a);
|
||||
}
|
||||
#endif
|
||||
rt_log("plugins: trace: %d hook entries counted", ntraced);
|
||||
}
|
||||
|
||||
// Takes back a page an init changed in the sealed part (.text, .rodata) of a game module: main keeps the game's own
|
||||
// bytes, the lifted code being what runs.
|
||||
static void restore_page(const RtLoaded *m, uint64_t off, const uint8_t *was) {
|
||||
if (!plat_mem_protect(m->base + off, PAGE, PLAT_PROT_RW)) rt_fatal("plugins: cannot unseal %s+%#llx", m->name,
|
||||
(unsigned long long)off);
|
||||
memcpy((void *)(uintptr_t)(m->base + off), was, PAGE);
|
||||
plat_mem_protect(m->base + off, PAGE, PLAT_PROT_READ);
|
||||
}
|
||||
|
||||
// After plugin i's init: what it wrote into the game modules, word by word, against the record of the build (its
|
||||
// entries of RT_PLUGIN_WRITES). The first difference goes into why (unless a reason is there already). Then the game
|
||||
// modules are made what they were before the init: the sealed part (.text, .rodata) gets the game's bytes back, a
|
||||
// write elsewhere stays and goes into pre, so the next plugin's init is compared only with what it writes itself.
|
||||
// The aliases this init mapped (from first_alias on) are flushed first; an earlier init's were flushed after it.
|
||||
static void check_writes(int plugin, int first_alias, char *why, size_t cap) {
|
||||
for (int i = first_alias; i < naliases; i++)
|
||||
if (aliases[i].live) flush(&aliases[i]);
|
||||
uint8_t seen[NWRITES > 0 ? NWRITES : 1];
|
||||
memset(seen, 0, sizeof seen);
|
||||
int unexpected = 0, missing = 0;
|
||||
char first[160] = "";
|
||||
for (int i = 0; i < ldr_count(); i++) {
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
if (!pre[i]) continue;
|
||||
uint8_t *now = (uint8_t *)(uintptr_t)m->base;
|
||||
for (uint64_t off = 0; off < m->size; off += PAGE) {
|
||||
if (!memcmp(now + off, pre[i] + off, PAGE)) continue;
|
||||
for (uint64_t w = off; w < off + PAGE; w += 4) {
|
||||
uint32_t a, b;
|
||||
memcpy(&a, pre[i] + w, 4);
|
||||
memcpy(&b, now + w, 4);
|
||||
if (a == b) continue;
|
||||
int k = 0;
|
||||
while (k < NWRITES && !(write_tab[k].plugin == plugin && write_tab[k].off == w &&
|
||||
!strcmp(write_tab[k].module, m->name) && write_tab[k].old == a &&
|
||||
write_tab[k].now == b))
|
||||
k++;
|
||||
if (k < NWRITES) {
|
||||
seen[k] = 1;
|
||||
} else if (!unexpected++) {
|
||||
snprintf(first, sizeof first, "%s+%#llx %08x -> %08x", m->name, (unsigned long long)w, a, b);
|
||||
}
|
||||
}
|
||||
if (off < m->data_off) restore_page(m, off, pre[i] + off);
|
||||
else memcpy(pre[i] + off, now + off, PAGE);
|
||||
}
|
||||
}
|
||||
for (int k = 0; k < NWRITES; k++) missing += write_tab[k].plugin == plugin && !seen[k];
|
||||
if (why[0]) return;
|
||||
if (unexpected)
|
||||
snprintf(why, cap, "its init wrote %d words the build did not record (first %s)", unexpected, first);
|
||||
else if (missing)
|
||||
snprintf(why, cap, "%d of the words the build recorded were not written", missing);
|
||||
}
|
||||
#endif
|
||||
|
||||
int plugins_start(Cpu *c, const RtLoaded *sdk) {
|
||||
#if RT_PLUGIN_COUNT
|
||||
if (!play) return 0;
|
||||
int run[RT_PLUGIN_COUNT] = {0}, nrun = 0;
|
||||
char why[RT_PLUGIN_COUNT][256];
|
||||
for (int i = 0; i < RT_PLUGIN_COUNT; i++) {
|
||||
why[i][0] = 0;
|
||||
const RtLoaded *m = loaded(plugin_tab[i].module);
|
||||
if (!rt_patch_enabled(plugin_tab[i].set)) continue;
|
||||
if (!m) { // loader.c left it out (its file is gone or changed): its set cannot run
|
||||
mods_plugin_off(plugin_tab[i].set, "its file is not the one the build lifted");
|
||||
continue;
|
||||
}
|
||||
if (!m->init) {
|
||||
mods_plugin_off(plugin_tab[i].set, "it has no DT_INIT");
|
||||
continue;
|
||||
}
|
||||
run[i] = 1, nrun++;
|
||||
}
|
||||
if (!nrun) { // every plugin's mod off or gone: the game sees its address space as without plugins
|
||||
mem_query_hook = NULL;
|
||||
rt_process_memory = NULL;
|
||||
return 0;
|
||||
}
|
||||
uint64_t t0 = plat_now_ms();
|
||||
rt_log("boot: ENTER %s DT_INIT first: a mod plugin's init runs before the game's constructors", sdk->name);
|
||||
rt_call(c, sdk->init, 0);
|
||||
pre = calloc((size_t)ldr_count(), sizeof *pre);
|
||||
if (!pre) rt_fatal("plugins: out of memory");
|
||||
for (int i = 0; i < ldr_count(); i++) { // the game modules before the plugins run
|
||||
const RtLoaded *m = ldr_module(i);
|
||||
if (m->plugin) continue;
|
||||
pre[i] = malloc(m->size);
|
||||
if (!pre[i]) rt_fatal("plugins: out of memory for a copy of %s", m->name);
|
||||
memcpy(pre[i], (const void *)(uintptr_t)m->base, m->size);
|
||||
}
|
||||
for (int i = RT_PLUGIN_COUNT - 1; i >= 0; i--) {
|
||||
if (!run[i]) continue;
|
||||
const RtLoaded *m = loaded(plugin_tab[i].module);
|
||||
rt_log("plugins: ENTER %s DT_INIT at %#llx", m->name, (unsigned long long)m->init);
|
||||
failed[0] = 0;
|
||||
int first_alias = naliases;
|
||||
running = 1;
|
||||
rt_trace |= RT_TRACE_PLUGIN;
|
||||
rt_call(c, m->init, 0);
|
||||
rt_trace &= ~(unsigned)RT_TRACE_PLUGIN;
|
||||
running = 0;
|
||||
if (failed[0]) snprintf(why[i], sizeof why[i], "its init asked for what the port does not serve: %s", failed);
|
||||
check_writes(i, first_alias, why[i], sizeof why[i]);
|
||||
char hex[65];
|
||||
sha256_hex((const void *)(uintptr_t)m->base, m->text_end, hex);
|
||||
if (!why[i][0] && strcmp(hex, plugin_tab[i].sha256))
|
||||
snprintf(why[i], sizeof why[i], "its code after the init is not the one the build lifted");
|
||||
}
|
||||
for (int i = 0; i < ldr_count(); i++) free(pre[i]);
|
||||
free(pre);
|
||||
pre = NULL;
|
||||
mem_query_hook = NULL; // the game's own QueryMemory sees its address space as without plugins
|
||||
rt_process_memory = NULL; // a callback that maps process memory stops with a named fatal (svc.c)
|
||||
int on = 0;
|
||||
for (int i = 0; i < RT_PLUGIN_COUNT; i++) {
|
||||
if (!run[i]) continue;
|
||||
if (why[i][0]) {
|
||||
rt_log("plugins: %s stays off: %s", plugin_tab[i].module, why[i]);
|
||||
mods_plugin_off(plugin_tab[i].set, why[i]);
|
||||
continue;
|
||||
}
|
||||
int hooks = 0;
|
||||
for (int k = 0; k < NWRITES; k++) hooks += write_tab[k].plugin == i;
|
||||
rt_log("plugins: %s: init ran, %d hooks on (set %s)", plugin_tab[i].module, hooks,
|
||||
rt_patch_name(plugin_tab[i].set));
|
||||
on++;
|
||||
}
|
||||
int live = 0;
|
||||
for (int i = 0; i < naliases; i++) live += aliases[i].live;
|
||||
rt_log("plugins: %d of %d plugins on in %llu ms (%d process-memory maps, %d kept)", on, nrun,
|
||||
(unsigned long long)(plat_now_ms() - t0), naliases, live);
|
||||
if (on && cfg_bool("TOTK_PLUGIN_TRACE")) trace_entries();
|
||||
return 1;
|
||||
#else
|
||||
(void)c, (void)sdk;
|
||||
return 0;
|
||||
#endif
|
||||
}
|
||||
@@ -625,10 +625,10 @@ int res_wants_wide_overlay(const unsigned out[2]) {
|
||||
return !res_is_16x9(out[0], out[1]) && rt_patch_set("aspect") >= 0;
|
||||
}
|
||||
|
||||
// Field of view (TOTK_FOV, percent of the game's own, 70..140, live): the camera's vertical view angle times the
|
||||
// factor on page +60 (recomp/patches-v143.txt, set fov). The horizontal angle follows through the aspect ratio. 100
|
||||
// switches the set off, so the camera code is the game's own. Runs on a settings change (gfx_present.c
|
||||
// window_settings_live), the first time at the first present.
|
||||
// Field of view (TOTK_FOV, percent of the game's own, 70..140, live): the camera's vertical view angle times the factor
|
||||
// on page +60 (recomp/patches-v110.txt, patches-v121.txt and patches-v143.txt, set fov). The horizontal angle follows
|
||||
// through the aspect ratio. 100 switches the set off, so the camera code is the game's own. Runs on a settings change
|
||||
// (gfx_present.c window_settings_live), the first time at the first present.
|
||||
#define FOV_MIN 70
|
||||
#define FOV_MAX 140
|
||||
void res_fov_live(void) {
|
||||
|
||||
+8
-3
@@ -1146,7 +1146,8 @@ int romfs_shaped_prompts_ready(const char *folder, unsigned w, unsigned h) { ret
|
||||
// The setup writes the shaped folders for the displays connected when it ran (installer/steps.py Shapes). After a
|
||||
// monitor change the menu can start the same generator for the shape of the game resolution: the setup's Python and
|
||||
// tools/make_romfs_patches.py, which build-cache/rebuild.txt names (installer/steps.py write_rebuild_command: line 1 the
|
||||
// Python, pythonw on Windows, so no console opens; line 4 the setup's folder), on the install folder's romfs/. Child
|
||||
// Python, pythonw on Windows, so no console opens; line 4 the setup's folder; both relative to the install folder for
|
||||
// the build kit, installer/layout.py), on the install folder's romfs/. Child
|
||||
// processes the game does not wait for: the generator writes manifest.tsv last, so a folder it has not finished is not
|
||||
// taken, and the next start loads the finished one.
|
||||
static int prepare_tools(char *python, size_t pcap, char *script, size_t scap, char *dir, size_t dcap) {
|
||||
@@ -1166,8 +1167,12 @@ static int prepare_tools(char *python, size_t pcap, char *script, size_t scap, c
|
||||
lines[k++] = p;
|
||||
p = e + 1;
|
||||
}
|
||||
int ok = k >= 4 && !strcmp(lines[1], "-c") && snprintf(python, pcap, "%s", lines[0]) < (int)pcap &&
|
||||
snprintf(script, scap, "%s/tools/make_romfs_patches.py", lines[3]) < (int)scap;
|
||||
int ok = k >= 4 && !strcmp(lines[1], "-c");
|
||||
if (ok) { // the build kit's lines are relative to the install folder
|
||||
const char *at0 = plat_path_absolute(lines[0]) ? "" : dir, *at3 = plat_path_absolute(lines[3]) ? "" : dir;
|
||||
ok = snprintf(python, pcap, "%s%s%s", at0, *at0 ? "/" : "", lines[0]) < (int)pcap &&
|
||||
snprintf(script, scap, "%s%s%s/tools/make_romfs_patches.py", at3, *at3 ? "/" : "", lines[3]) < (int)scap;
|
||||
}
|
||||
free(text);
|
||||
snprintf(path, sizeof path, "%s/romfs/romfs.bin", dir);
|
||||
return ok && plat_file_exists(python) && plat_file_exists(script) && plat_file_exists(path);
|
||||
|
||||
+18
-4
@@ -3,6 +3,8 @@
|
||||
// The SVC entry of the lifted code: preemption checkpoint, trace, then the thread and
|
||||
// synchronization SVCs (kernel.c), IPC (ipc.c), memory (memory.c), Break, OutputDebugString, GetInfo and ExitProcess. The 21 SVCs
|
||||
// TotK has but never reaches on MNG's boot path stop with a named fatal, the caller and the guest backtrace.
|
||||
// GetSystemTick, GetInfo 65001 and Map/UnmapProcessMemory are for mod plugins (exlaunch): the game calls none of them,
|
||||
// and process memory is mapped only in the plugin service run (plugin_init.c).
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
|
||||
@@ -17,9 +19,13 @@
|
||||
#define ST_STACK 0xBu
|
||||
#define PERM_RW 3u
|
||||
#define PROGRAM_ID 0x0100f2c0115b6000ull
|
||||
#define CUR_PROCESS 0xFFFF8001u // Horizon's pseudo handle of the current process
|
||||
|
||||
static uint64_t g_svcs;
|
||||
void (*volatile svc_test_exit_hook)(unsigned code);
|
||||
uint64_t rt_svc_tick_offset;
|
||||
int (*rt_plugin_svc)(Cpu *c, uint32_t imm, uint64_t pc);
|
||||
uint32_t (*rt_process_memory)(int unmap, uint64_t dst, uint32_t process, uint64_t src, uint64_t size);
|
||||
|
||||
RtStats rt_stats(void) {
|
||||
return (RtStats){__atomic_load_n(&g_svcs, __ATOMIC_RELAXED), kern_threads_created(), hostfn_calls()};
|
||||
@@ -121,6 +127,7 @@ static void get_info(Cpu *c, uint64_t pc) {
|
||||
case 18: v = PROGRAM_ID; break;
|
||||
case 20: v = user_exception_context(); break; // the SDK keeps e.g. the dying message at +0x1C0
|
||||
case 23: v = 1; break;
|
||||
case 65001: v = CUR_PROCESS; break; // Mesosphere's current process handle (exlaunch's way to its own memory)
|
||||
default: {
|
||||
char why[64];
|
||||
snprintf(why, sizeof why, "GetInfo id %u (handle %#x, sub id %#llx)", id, (uint32_t)c->x[2],
|
||||
@@ -151,11 +158,13 @@ void rt_svc(Cpu *c, uint32_t imm, uint64_t pc) {
|
||||
|
||||
static void svc_body(Cpu *c, uint32_t imm, uint64_t pc) {
|
||||
__atomic_fetch_add(&g_svcs, 1, __ATOMIC_RELAXED);
|
||||
if (rt_trace & RT_TRACE_SVC) {
|
||||
if (rt_trace & (RT_TRACE_SVC | RT_TRACE_PLUGIN)) {
|
||||
if ((rt_trace & RT_TRACE_PLUGIN) && rt_plugin_svc && rt_plugin_svc(c, imm, pc)) return; // refused, x0 set
|
||||
char at[160];
|
||||
rt_log("svc %#x at %s x0=%#llx x1=%#llx x2=%#llx x3=%#llx", imm, ldr_describe(pc, at, sizeof at),
|
||||
(unsigned long long)c->x[0], (unsigned long long)c->x[1], (unsigned long long)c->x[2],
|
||||
(unsigned long long)c->x[3]);
|
||||
if (rt_trace & RT_TRACE_SVC)
|
||||
rt_log("svc %#x at %s x0=%#llx x1=%#llx x2=%#llx x3=%#llx", imm, ldr_describe(pc, at, sizeof at),
|
||||
(unsigned long long)c->x[0], (unsigned long long)c->x[1], (unsigned long long)c->x[2],
|
||||
(unsigned long long)c->x[3]);
|
||||
}
|
||||
if (kern_svc(c, imm)) return;
|
||||
uint64_t *x = c->x;
|
||||
@@ -210,6 +219,11 @@ static void svc_body(Cpu *c, uint32_t imm, uint64_t pc) {
|
||||
return;
|
||||
}
|
||||
case 0x29: get_info(c, pc); return;
|
||||
case 0x1E: x[0] = rt_ticks() + rt_svc_tick_offset; return; // GetSystemTick (19.2 MHz)
|
||||
case 0x74: case 0x75: // MapProcessMemory / UnmapProcessMemory(x0 dst, w1 process, x2 src, x3 size)
|
||||
if (!rt_process_memory) svc_fatal(c, imm, pc, "process memory is mapped only in the plugin service run");
|
||||
x[0] = rt_process_memory(imm == 0x75, x[0], (uint32_t)x[1], x[2], x[3]);
|
||||
return;
|
||||
case 0x07: { // ExitProcess, as closing the window (gfx_window_closed): pipeline cache, shader records and
|
||||
// profile saved, logs flushed, exit code 0
|
||||
char at[160];
|
||||
|
||||
+19
-3
@@ -1271,7 +1271,13 @@ static void tab_mods(float scale) {
|
||||
if (count) snprintf(cheat, sizeof cheat, menu_text(UI_MENU_MODS_CHEATS), count);
|
||||
else snprintf(cheat, sizeof cheat, menu_text(UI_MENU_MODS_CHEATS_FILE), cheats_file());
|
||||
}
|
||||
char code[512] = "", problem[512] = "", skipped[512] = "", left[512] = "";
|
||||
char code[512] = "", problem[768] = "", skipped[512] = "", left[512] = "", plugin[512] = "";
|
||||
if (m->plugin_off[0]) // its plugin's start differed from the build (plugin_init.c)
|
||||
snprintf(plugin, sizeof plugin, menu_text(UI_MENU_MODS_PLUGIN_OFF), m->plugin_off);
|
||||
else if (m->plugin_hooks && m->problem == MOD_OK)
|
||||
snprintf(plugin, sizeof plugin, menu_text(UI_MENU_MODS_PLUGIN_ON), m->plugin_hooks);
|
||||
else if (m->plugin_note[0] && m->problem != MOD_PLUGIN) // used without its code: the refusal stays visible
|
||||
snprintf(plugin, sizeof plugin, menu_text(UI_MENU_MODS_PLUGIN_REFUSED), m->plugin_file, m->plugin_note);
|
||||
if (m->code[0] && m->problem != MOD_PLUGIN) // used without its code
|
||||
snprintf(skipped, sizeof skipped, menu_text(UI_MENU_MODS_CODE_SKIPPED), m->code);
|
||||
if (m->unsupported && m->problem != MOD_EXEFS_UNSUPPORTED)
|
||||
@@ -1282,7 +1288,16 @@ static void tab_mods(float scale) {
|
||||
snprintf(code, sizeof code, menu_text(UI_MENU_MODS_EXEFS_ON), m->nsets + m->ndata);
|
||||
switch (m->problem) {
|
||||
case MOD_EMPTY: snprintf(problem, sizeof problem, "%s", menu_text(UI_MENU_MODS_EMPTY)); break;
|
||||
case MOD_PLUGIN: snprintf(problem, sizeof problem, "%s", menu_text(UI_MENU_MODS_SUBSDK)); break;
|
||||
case MOD_PLUGIN:
|
||||
if (m->plugin_note[0]) { // the build refused its plugin, and says why
|
||||
int n = snprintf(problem, sizeof problem, menu_text(UI_MENU_MODS_PLUGIN_REFUSED), m->plugin_file,
|
||||
m->plugin_note);
|
||||
if (n >= 0 && n < (int)sizeof problem)
|
||||
snprintf(problem + n, sizeof problem - n, " %s", menu_text(UI_MENU_MODS_PLUGIN_STAYS_OFF));
|
||||
} else {
|
||||
snprintf(problem, sizeof problem, "%s", menu_text(UI_MENU_MODS_SUBSDK));
|
||||
}
|
||||
break;
|
||||
case MOD_EXEFS_LATER: snprintf(problem, sizeof problem, "%s", menu_text(UI_MENU_MODS_EXEFS_LATER)); break;
|
||||
case MOD_EXEFS_UNSUPPORTED:
|
||||
snprintf(problem, sizeof problem, menu_text(UI_MENU_MODS_EXEFS_UNSUPPORTED), m->note);
|
||||
@@ -1292,7 +1307,8 @@ static void tab_mods(float scale) {
|
||||
default: break;
|
||||
}
|
||||
int k = snprintf(desc, sizeof desc, menu_text(UI_MENU_MODS_ROW_DESC), m->replaced, m->added);
|
||||
const char *parts[] = {over, other, cheat, code, skipped, left, problem, menu_text(UI_MENU_MODS_MOVE_HINT)};
|
||||
const char *parts[] = {over, other, cheat, code, plugin, skipped, left, problem,
|
||||
menu_text(UI_MENU_MODS_MOVE_HINT)};
|
||||
for (const char *part : parts)
|
||||
if (*part && k >= 0 && k < (int)sizeof desc) k += snprintf(desc + k, sizeof desc - k, "\n\n%s", part);
|
||||
const char *note = m->enabled && m->problem != MOD_OK ? menu_text(UI_MENU_MODS_UNUSED) : after;
|
||||
|
||||
+27
-5
@@ -86,7 +86,7 @@ static const char *const text_en[UI_TEXT_COUNT] = {
|
||||
"bars, or stretch them over the whole picture. Applies from the next movie.",
|
||||
[UI_MENU_MOVIE_ASPECT_KEEP] = "Black bars", [UI_MENU_MOVIE_ASPECT_STRETCH] = "Stretch",
|
||||
[UI_MENU_FOV] = "Field of view",
|
||||
[UI_MENU_FOV_DESC] = "How much of the world the camera shows, in percent of the game's own. Game versions 1.1.0 and 1.4.3.",
|
||||
[UI_MENU_FOV_DESC] = "How much of the world the camera shows, in percent of the game's own. Game versions 1.1.0, 1.2.1 and 1.4.3.",
|
||||
[UI_MENU_SHADOW_RES] = "Shadow map size",
|
||||
[UI_MENU_SHADOW_RES_DESC] = "Size of the sun's shadow map. The game's own size changes with the view, down to about 400 "
|
||||
"near the ground, so leaf shadows look blocky and jump; a fixed size keeps them still and "
|
||||
@@ -592,6 +592,12 @@ static const char *const text_en[UI_TEXT_COUNT] = {
|
||||
"stays off while this mod is on.",
|
||||
[UI_MENU_MODS_CODE_SKIPPED] = "Its own game code (%s) does not load: the rest of the mod is used. Switch the mod "
|
||||
"off if the game misbehaves with it.",
|
||||
[UI_MENU_MODS_PLUGIN_ON] = "Its plugin runs: %d hooks into the game code.",
|
||||
[UI_MENU_MODS_PLUGIN_REFUSED] = "Its plugin %s cannot run in the port: %s.",
|
||||
[UI_MENU_MODS_PLUGIN_STAYS_OFF] = "The whole mod stays off. Its switch \"Use ... without its code\" below uses the "
|
||||
"rest of it.",
|
||||
[UI_MENU_MODS_PLUGIN_OFF] = "Its plugin stays off this time: it started differently from the build (%s). The game "
|
||||
"rebuilds it at the next start.",
|
||||
[UI_MENU_MODS_WITHOUT_CODE] = "Use %s without its code",
|
||||
[UI_MENU_MODS_WITHOUT_CODE_DESC] = "The mod above brings code the port cannot run: its own game code (a subsdk "
|
||||
"plugin or a replaced main) or code patches (exefs) the port cannot apply. Off: "
|
||||
@@ -663,7 +669,7 @@ static const char *const text_ru[UI_TEXT_COUNT] = {
|
||||
"растянуть их на всю картинку. Действует со следующего ролика.",
|
||||
[UI_MENU_MOVIE_ASPECT_KEEP] = "Чёрные полосы", [UI_MENU_MOVIE_ASPECT_STRETCH] = "Растянуть",
|
||||
[UI_MENU_FOV] = "Поле зрения",
|
||||
[UI_MENU_FOV_DESC] = "Сколько мира показывает камера, в процентах от игрового. Для версий игры 1.1.0 и 1.4.3.",
|
||||
[UI_MENU_FOV_DESC] = "Сколько мира показывает камера, в процентах от игрового. Для версий игры 1.1.0, 1.2.1 и 1.4.3.",
|
||||
[UI_MENU_SHADOW_RES] = "Размер карты теней",
|
||||
[UI_MENU_SHADOW_RES_DESC] = "Размер карты теней от солнца. Собственный размер игры меняется вместе с видом, у земли до "
|
||||
"примерно 400, поэтому тени листвы выглядят квадратными и дёргаются; постоянный размер делает "
|
||||
@@ -1173,6 +1179,12 @@ static const char *const text_ru[UI_TEXT_COUNT] = {
|
||||
"который выключен, пока этот мод включён.",
|
||||
[UI_MENU_MODS_CODE_SKIPPED] = "Его собственный код игры (%s) не загружается: используется остальная часть мода. "
|
||||
"Выключите мод, если игра с ним ведёт себя странно.",
|
||||
[UI_MENU_MODS_PLUGIN_ON] = "Его плагин работает: перехватов в коде игры: %d.",
|
||||
[UI_MENU_MODS_PLUGIN_REFUSED] = "Его плагин %s не может работать в порте: %s.",
|
||||
[UI_MENU_MODS_PLUGIN_STAYS_OFF] = "Мод выключен целиком. Его переключатель «... без его кода» ниже включает "
|
||||
"остальную часть мода.",
|
||||
[UI_MENU_MODS_PLUGIN_OFF] = "Его плагин в этот раз выключен: он запустился не так, как при сборке (%s). Игра "
|
||||
"пересоберёт его при следующем запуске.",
|
||||
[UI_MENU_MODS_WITHOUT_CODE] = "%s без его кода",
|
||||
[UI_MENU_MODS_WITHOUT_CODE_DESC] = "Мод выше приносит код, который порт не может запустить: свой код игры (плагин "
|
||||
"subsdk или заменённый main) или патчи кода (exefs), которые порт не может "
|
||||
@@ -1570,8 +1582,8 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = {
|
||||
[UI_MENU_OV_FMT_FSR] = "AMD FSR 1, nitidez %d%%", [UI_MENU_OV_SCALE_NONE] = "nenhuma (1:1)",
|
||||
[UI_MENU_OV_SCALE_BILINEAR] = "bilinear",
|
||||
[UI_MENU_FOV] = "Campo de visão",
|
||||
[UI_MENU_FOV_DESC] = "Quanto do mundo a câmera mostra, em porcentagem do original do jogo. Nas versões 1.1.0 e 1.4.3 do "
|
||||
"jogo.",
|
||||
[UI_MENU_FOV_DESC] = "Quanto do mundo a câmera mostra, em porcentagem do original do jogo. Nas versões 1.1.0, 1.2.1 e 1.4.3 "
|
||||
"do jogo.",
|
||||
[UI_MENU_SHADOW_RES] = "Tamanho do mapa de sombras",
|
||||
[UI_MENU_SHADOW_RES_DESC] = "Tamanho do mapa de sombras do sol. O tamanho do jogo muda com a vista, até cerca de "
|
||||
"400 perto do chão, então as sombras das folhas ficam quadriculadas e pulam; um "
|
||||
@@ -1788,6 +1800,12 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = {
|
||||
"%s, que fica desligado enquanto este mod está ligado.",
|
||||
[UI_MENU_MODS_CODE_SKIPPED] = "O código próprio do jogo dele (%s) não é carregado: o resto do mod é usado. "
|
||||
"Desligue o mod se o jogo se comportar mal com ele.",
|
||||
[UI_MENU_MODS_PLUGIN_ON] = "O plugin dele está ativo: %d ganchos no código do jogo.",
|
||||
[UI_MENU_MODS_PLUGIN_REFUSED] = "O plugin %s dele não pode rodar no port: %s.",
|
||||
[UI_MENU_MODS_PLUGIN_STAYS_OFF] = "O mod inteiro fica desligado. A chave \"Usar ... sem o código dele\", abaixo, "
|
||||
"usa o resto dele.",
|
||||
[UI_MENU_MODS_PLUGIN_OFF] = "O plugin dele fica desligado desta vez: ele iniciou de forma diferente da compilação "
|
||||
"(%s). O jogo o recompila na próxima inicialização.",
|
||||
[UI_MENU_MODS_WITHOUT_CODE] = "Usar %s sem o código dele",
|
||||
[UI_MENU_MODS_WITHOUT_CODE_DESC] = "O mod acima traz código que o port não consegue rodar: código próprio do jogo "
|
||||
"(um plugin subsdk ou um main substituído) ou patches de código (exefs) que o "
|
||||
@@ -1914,7 +1932,7 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = {
|
||||
[UI_MENU_MOVIE_ASPECT_STRETCH] = "拉伸",
|
||||
[UI_MENU_FOV] = "视野",
|
||||
[UI_MENU_FOV_DESC] = "镜头能看到多大范围,以游戏自身视野的百分比表示。"
|
||||
"支持游戏版本 1.1.0 和 1.4.3。",
|
||||
"支持游戏版本 1.1.0、1.2.1 和 1.4.3。",
|
||||
[UI_MENU_SHARPNESS] = "FSR 锐度",
|
||||
[UI_MENU_SHARPNESS_DESC] = "FSR 缩放后的锐化强度;0 为关闭。",
|
||||
[UI_MENU_COLOR] = "色彩",
|
||||
@@ -2480,6 +2498,10 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = {
|
||||
[UI_MENU_MODS_EXEFS_ON] = "正在使用的代码补丁:%d 个文件。",
|
||||
[UI_MENU_MODS_REPLACES_BUILTIN] = "正在使用的代码补丁:%d 个文件。它们取代本移植自带的补丁 %s,此模组开启时该补丁保持关闭。",
|
||||
[UI_MENU_MODS_CODE_SKIPPED] = "它自带的游戏代码(%s)不会加载:模组的其余部分照常使用。如果游戏因此表现异常,请关闭此模组。",
|
||||
[UI_MENU_MODS_PLUGIN_ON] = "它的插件正在运行:游戏代码中有 %d 处挂钩。",
|
||||
[UI_MENU_MODS_PLUGIN_REFUSED] = "它的插件 %s 无法在移植版中运行:%s。",
|
||||
[UI_MENU_MODS_PLUGIN_STAYS_OFF] = "整个模组保持关闭。打开下方它的“不带代码使用 ...”开关即可使用其余部分。",
|
||||
[UI_MENU_MODS_PLUGIN_OFF] = "它的插件本次保持关闭:启动情况与构建时不同(%s)。游戏会在下次启动时重新构建它。",
|
||||
[UI_MENU_MODS_WITHOUT_CODE] = "不带代码使用 %s",
|
||||
[UI_MENU_MODS_WITHOUT_CODE_DESC] = "上方的模组带有移植版无法运行的代码:它自己的游戏代码(subsdk 插件或替换的 main),或移植版无法应用的代码补丁(exefs)。关闭:整个模组保持关闭。开启:跳过这些代码,使用模组的其余部分:其 RomFS 文件、金手指和可用的代码补丁。它的文件可能依赖缺失的代码;如果游戏表现异常,请再次关闭。下次启动时生效。",
|
||||
[UI_MENU_MODS_WITHOUT_CODE_ALL] = "已对所有模组开启:totk.ini 中 mods_without_code = 1。",
|
||||
|
||||
@@ -92,6 +92,10 @@ typedef enum {
|
||||
UI_MENU_MODS_EXEFS_ON, // printf: its ExeFS patch files in use
|
||||
UI_MENU_MODS_REPLACES_BUILTIN, // printf: its ExeFS patch files in use, the port's own patch sets they turn off
|
||||
UI_MENU_MODS_CODE_SKIPPED, // printf: its code modules in exefs, which do not load (used without its code)
|
||||
UI_MENU_MODS_PLUGIN_ON, // printf: the hooks of its plugin (runtime/plugin_init.c) that run
|
||||
UI_MENU_MODS_PLUGIN_REFUSED, // printf: its plugin file, why the build refused it (English, from the tool)
|
||||
UI_MENU_MODS_PLUGIN_STAYS_OFF, // after UI_MENU_MODS_PLUGIN_REFUSED when the whole mod stays off
|
||||
UI_MENU_MODS_PLUGIN_OFF, // printf: why its plugin's start differed from the build (English)
|
||||
UI_MENU_MODS_WITHOUT_CODE, // printf: the mod's name; its switch "use without its code" (TOTK_MODS_NO_CODE)
|
||||
UI_MENU_MODS_WITHOUT_CODE_DESC,
|
||||
UI_MENU_MODS_WITHOUT_CODE_ALL, // why that switch cannot be changed: TOTK_MODS_WITHOUT_CODE=1
|
||||
|
||||
@@ -0,0 +1,591 @@
|
||||
"""recomp/layout.py with and without mod plugins, and tools/exefs_plugins.py's check of a plugin service run record:
|
||||
a synthetic exlaunch-shaped record with one inline hook that passes, and one bad record per rule; the check of the
|
||||
code the hooks reach; the patch lines of the three hook kinds; and what recomp/patches.py and recomp/gen.py make of
|
||||
those lines (the hook call, the stub's return, the unpatched copy a replace hook's trampoline reaches, the start-up
|
||||
record in patches.h). The start-up comparison itself runs in the game (runtime/plugin_init.c plugins_start); its
|
||||
mods.c side is in totk.exe --probe-mods. No game files.
|
||||
|
||||
Run with .venv (lz4 and numpy, as recomp/): python tests/exefs_plugins_test.py
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT / 'tools'))
|
||||
sys.path.insert(0, str(ROOT / 'recomp'))
|
||||
import exefs_plugins as xp # noqa: E402
|
||||
import gen # noqa: E402
|
||||
import layout # noqa: E402
|
||||
import patches # noqa: E402
|
||||
from nso import Nso # noqa: E402
|
||||
|
||||
|
||||
def nso(size_text, bss=0x1000):
|
||||
"""An uncompressed NSO: .text of size_text bytes (b over MOD0's offset word), .rodata with MOD0 at 0x1000 past
|
||||
.text, .data with an empty .dynamic."""
|
||||
text = bytearray(size_text)
|
||||
struct.pack_into('<II', text, 0, 0x14000002, size_text) # b +8; the MOD0 offset
|
||||
ro = b'MOD0' + struct.pack('<6i', 0x1000, 0, 0, 0, 0, 0) # .dynamic at .rodata + 0x1000 = .data
|
||||
data = bytes(16) # DT_NULL
|
||||
segs = [(0, bytes(text)), (size_text, ro), (size_text + 0x1000, data)]
|
||||
hdr = bytearray(0x100)
|
||||
hdr[0:4] = b'NSO0'
|
||||
off = 0x100
|
||||
for i, (mem, d) in enumerate(segs):
|
||||
struct.pack_into('<IIII', hdr, 0x10 + 0x10 * i, off, mem, len(d), bss if i == 2 else 0)
|
||||
struct.pack_into('<I', hdr, 0x60 + 4 * i, len(d))
|
||||
off += len(d)
|
||||
return bytes(hdr) + b''.join(d for _, d in segs)
|
||||
|
||||
|
||||
def dev_layout_files(exefs, out):
|
||||
"""layout.json and layout.h as recomp/layout.py wrote them before plugins (dev f8505807), the reference for the
|
||||
no-plugin case."""
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
mods, base = [], layout.FIRST_BASE
|
||||
for name in layout.ORDER:
|
||||
if (exefs / name).exists():
|
||||
size = Nso(str(exefs / name)).size
|
||||
mods.append((name, base, size))
|
||||
base = layout.align_up(base + size)
|
||||
(out / 'layout.json').write_text(json.dumps(
|
||||
[{'name': n, 'base': f'{b:#x}', 'size': f'{s:#x}'} for n, b, s in mods], indent=1) + '\n')
|
||||
lines = ['// Module layout chosen by recomp/layout.py. Generated; do not edit.', '#pragma once', '']
|
||||
for n, b, s in mods:
|
||||
lines.append(f'#define MOD_{n.upper()}_BASE {b:#x}ull')
|
||||
lines.append(f'#define MOD_{n.upper()}_SIZE {s:#x}ull')
|
||||
lines += ['', f'#define MOD_COUNT {len(mods)}',
|
||||
'#define MOD_LIST(X) ' + ' '.join(f'X({n}, {n.upper()})' for n, _, _ in mods), '']
|
||||
(out / 'layout.h').write_text('\n'.join(lines))
|
||||
|
||||
|
||||
class Layout(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.tmp = tempfile.TemporaryDirectory()
|
||||
cls.dir = Path(cls.tmp.name)
|
||||
cls.exefs = cls.dir / 'exefs'
|
||||
cls.exefs.mkdir()
|
||||
for name, size in (('main', 0x30000), ('subsdk0', 0x5000), ('sdk', 0x21000)):
|
||||
(cls.exefs / name).write_bytes(nso(size))
|
||||
cls.plugins = [cls.dir / 'mod a' / 'subsdk9', cls.dir / 'mod b' / 'subsdk8']
|
||||
for p, size in zip(cls.plugins, (0x9000, 0x3000)):
|
||||
p.parent.mkdir()
|
||||
p.write_bytes(nso(size))
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.tmp.cleanup()
|
||||
|
||||
def test_no_plugins_identical(self):
|
||||
dev, ours = self.dir / 'dev', self.dir / 'ours'
|
||||
dev_layout_files(self.exefs, dev)
|
||||
layout.write(layout.layout(self.exefs), ours, layout.plugin_files([]))
|
||||
for f in ('layout.json', 'layout.h'):
|
||||
self.assertEqual((dev / f).read_bytes(), (ours / f).read_bytes(), f)
|
||||
|
||||
def test_plugins_after_sdk(self):
|
||||
plain = layout.layout(self.exefs)
|
||||
mods = layout.layout(self.exefs, self.plugins)
|
||||
self.assertEqual(mods[:3], plain) # the game modules keep their bases
|
||||
self.assertEqual([n for n, _, _ in mods], ['main', 'subsdk0', 'sdk', 'plugin0', 'plugin1'])
|
||||
for (_, b0, s0), (_, b1, _) in zip(mods, mods[1:]):
|
||||
self.assertEqual(b1, layout.align_up(b0 + s0))
|
||||
self.assertEqual(mods[3][2], Nso(str(self.plugins[0])).size)
|
||||
out = self.dir / 'with'
|
||||
files = layout.plugin_files(self.plugins)
|
||||
layout.write(mods, out, files)
|
||||
data = json.loads((out / 'layout.json').read_text())
|
||||
self.assertNotIn('file', data[2])
|
||||
self.assertEqual(data[3]['file'], self.plugins[0].resolve().as_posix())
|
||||
self.assertEqual(len(data[3]['sha256']), 64)
|
||||
self.assertEqual(layout.read(out), mods)
|
||||
self.assertIn('#define MOD_PLUGIN1_BASE', (out / 'layout.h').read_text())
|
||||
self.assertIn('#define MOD_COUNT 5', (out / 'layout.h').read_text())
|
||||
|
||||
def test_address_limit(self):
|
||||
old = layout.FIRST_BASE
|
||||
layout.FIRST_BASE = layout.ADDRESS_LIMIT - 0x400000 # the plugins end above 2^39
|
||||
try:
|
||||
with self.assertRaises(ValueError):
|
||||
layout.layout(self.exefs, self.plugins)
|
||||
finally:
|
||||
layout.FIRST_BASE = old
|
||||
|
||||
|
||||
# ---- records
|
||||
MAIN, PLUGIN = 0x7000000000, 0x7005200000
|
||||
SITE_OFF, STUB_OFF, TRAMP_OFF, CB_OFF, IMPL_OFF, THUNK_OFF = 0x1000, 0x4000, 0x2000, 0x66f0, 0x73ec, 0xf00
|
||||
MAIN_TEXT, PLUGIN_TEXT = 0x100000, 0x9000
|
||||
|
||||
|
||||
def b(src, dst, link=False):
|
||||
return (0x94000000 if link else 0x14000000) | ((dst - src) >> 2) & 0x3FFFFFF
|
||||
|
||||
|
||||
def record(original=0x94000010, trampoline_first=None, stub=None, writes=None, **extra):
|
||||
"""A record of one inline hook at main+SITE_OFF whose original is `original` (default bl main+SITE_OFF+0x40)."""
|
||||
site, stub_at, tramp = MAIN + SITE_OFF, PLUGIN + STUB_OFF, PLUGIN + TRAMP_OFF
|
||||
img = bytearray(0x14000)
|
||||
words = stub or [xp.STUR_X30, b(stub_at + 4, PLUGIN + IMPL_OFF, True), xp.LDUR_X30, b(stub_at + 12, tramp)]
|
||||
struct.pack_into('<4IQ', img, STUB_OFF, *words, PLUGIN + CB_OFF)
|
||||
if trampoline_first is None:
|
||||
rel = xp.pc_relative(original, site)
|
||||
trampoline_first = (b(tramp, rel[1], True) if rel and rel[0][0] == 'b' else original)
|
||||
struct.pack_into('<II', img, TRAMP_OFF, trampoline_first, b(tramp + 4, site + 4))
|
||||
rec = {
|
||||
'refused': None, 'svcs': {'0x6': 52, '0x1e': 10, '0x27': 18, '0x29': 17, '0x74': 11, '0x75': 7},
|
||||
'get_info': {'2': 1, '3': 1, '12': 1, '13': 1, '65001': 1},
|
||||
'imports': {'_ZN2nn4util8SNPrintfEPcmPKcz': 20, '_ZN2nn2oe17GetDisplayVersionEPNS0_14DisplayVersionE': 1},
|
||||
'modules': [
|
||||
{'name': 'main', 'base': MAIN, 'size': 0x200000, 'text_end': MAIN_TEXT, 'data_off': 0x180000, 'init': 0,
|
||||
'plugin': False},
|
||||
{'name': 'sdk', 'base': 0x7004400000, 'size': 0x100000, 'text_end': 0x80000, 'data_off': 0xc0000,
|
||||
'init': 0, 'plugin': False},
|
||||
{'name': 'plugin0', 'base': PLUGIN, 'size': 0x14000, 'text_end': PLUGIN_TEXT, 'data_off': 0xd000,
|
||||
'init': PLUGIN + 0x50a0, 'plugin': True, 'functions': [hex(CB_OFF), hex(IMPL_OFF), hex(THUNK_OFF)]}],
|
||||
'writes': writes if writes is not None else [
|
||||
{'module': 'main', 'offset': SITE_OFF, 'old': original, 'new': b(site, stub_at)}],
|
||||
'aliases': [], 'images': {'plugin0': bytes(img)}}
|
||||
rec.update(extra)
|
||||
return rec
|
||||
|
||||
|
||||
IMPLS = {PLUGIN + IMPL_OFF: 'exl_inline_hook_impl'}
|
||||
|
||||
|
||||
class Check(unittest.TestCase):
|
||||
def reasons(self, rec):
|
||||
return xp.check_record(rec, IMPLS)[1]
|
||||
|
||||
def refused(self, rec, text):
|
||||
r = self.reasons(rec)
|
||||
self.assertTrue(r, 'accepted')
|
||||
self.assertTrue(any(text in x for x in r), r)
|
||||
|
||||
def test_good(self):
|
||||
hooks, reasons = xp.check_record(record(), IMPLS)
|
||||
self.assertEqual(reasons, [])
|
||||
self.assertEqual(len(hooks), 1)
|
||||
self.assertEqual(hooks[0]['callback'], f'plugin0+{CB_OFF:#x}')
|
||||
self.assertEqual(hooks[0]['trampoline'], f'plugin0+{TRAMP_OFF:#x}')
|
||||
|
||||
def test_good_plain_original(self):
|
||||
self.assertEqual(self.reasons(record(original=0xAA0003F3)), []) # mov x19, x0, copied as it is
|
||||
|
||||
def test_call_site_rewrite(self): # Quick Weapon Durability: bl straight to a thunk of the plugin
|
||||
site = MAIN + SITE_OFF
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': b(site, PLUGIN + THUNK_OFF, True)}]
|
||||
hooks, reasons = xp.check_record(record(writes=w), IMPLS)
|
||||
self.assertEqual(reasons, [])
|
||||
self.assertEqual(hooks[0]['kind'], 'call')
|
||||
self.assertEqual(hooks[0]['target'], f'plugin0+{THUNK_OFF:#x}')
|
||||
|
||||
def test_replace_hook(self): # Full Inventory Expansion: b from a main function start, Orig trampoline in the pool
|
||||
site, prologue = MAIN + SITE_OFF, 0xA9BA7BFD # stp x29, x30, [sp, #-0x60]!
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': prologue, 'new': b(site, PLUGIN + THUNK_OFF),
|
||||
'fn_start': True}]
|
||||
hooks, reasons = xp.check_record(record(original=prologue, writes=w), IMPLS)
|
||||
self.assertEqual(reasons, [])
|
||||
self.assertEqual(hooks[0]['kind'], 'replace')
|
||||
self.assertEqual(hooks[0]['trampoline'], f'plugin0+{TRAMP_OFF:#x}')
|
||||
|
||||
def test_replace_inside_function(self):
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': b(MAIN + SITE_OFF, PLUGIN + THUNK_OFF),
|
||||
'fn_start': False}]
|
||||
self.refused(record(writes=w), 'from inside a main function')
|
||||
|
||||
def test_call_into_function_middle(self):
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': b(MAIN + SITE_OFF, PLUGIN + 0xf08, True)}]
|
||||
self.refused(record(writes=w), 'neither an inline-hook stub nor a function start')
|
||||
|
||||
def test_bl_to_stub(self): # an inline stub is entered by b; a bl there would skip the saved x30
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': b(MAIN + SITE_OFF, PLUGIN + STUB_OFF, True)}]
|
||||
self.refused(record(writes=w), 'neither an inline-hook stub nor a function start')
|
||||
|
||||
def test_refused_run(self):
|
||||
self.refused(record(refused='svc 0x40 at plugin0+0x76a0'), 'the run was refused')
|
||||
|
||||
def test_svc(self):
|
||||
self.refused(record(svcs={'0x6': 1, '0x40': 1}), 'svc 0x40')
|
||||
|
||||
def test_get_info(self):
|
||||
self.refused(record(get_info={'2': 1, '18': 1}), 'GetInfo id 18')
|
||||
|
||||
def test_import(self):
|
||||
self.refused(record(imports={'_ZN2nn2fs8MountRomEPKcPvm': 1}), 'the import _ZN2nn2fs8MountRom')
|
||||
|
||||
def test_write_main_data(self):
|
||||
w = [{'module': 'main', 'offset': 0x180008, 'old': 0, 'new': 1}]
|
||||
self.refused(record(writes=w), "outside main's .text: main+0x180008")
|
||||
|
||||
def test_write_other_module(self):
|
||||
w = [{'module': 'sdk', 'offset': 0x100, 'old': 0, 'new': 1}]
|
||||
self.refused(record(writes=w), "outside main's .text: sdk+0x100")
|
||||
|
||||
def test_not_a_branch(self):
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': 0xD503201F}]
|
||||
self.refused(record(writes=w), 'is no b')
|
||||
|
||||
def test_branch_elsewhere(self):
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': b(MAIN + SITE_OFF, MAIN + 0x8000)}]
|
||||
self.refused(record(writes=w), "does not reach into a plugin's .text")
|
||||
|
||||
def test_stub_shape(self):
|
||||
stub_at = PLUGIN + STUB_OFF
|
||||
words = [0xD503201F, b(stub_at + 4, PLUGIN + IMPL_OFF, True), xp.LDUR_X30, b(stub_at + 12, PLUGIN + TRAMP_OFF)]
|
||||
self.refused(record(stub=words), 'neither an inline-hook stub nor a function start')
|
||||
|
||||
def test_stub_impl(self):
|
||||
stub_at = PLUGIN + STUB_OFF
|
||||
words = [xp.STUR_X30, b(stub_at + 4, PLUGIN + 0x100, True), xp.LDUR_X30, b(stub_at + 12, PLUGIN + TRAMP_OFF)]
|
||||
self.refused(record(stub=words), 'not exl_inline_hook_impl')
|
||||
|
||||
def test_callback_outside(self):
|
||||
rec = record()
|
||||
img = bytearray(rec['images']['plugin0'])
|
||||
struct.pack_into('<Q', img, STUB_OFF + 16, MAIN + 0x500)
|
||||
rec['images']['plugin0'] = bytes(img)
|
||||
self.refused(rec, "callback 0x7000000500 is outside plugin0's .text")
|
||||
|
||||
def test_trampoline_not_original(self):
|
||||
self.refused(record(original=0xAA0003F3, trampoline_first=0xAA0003F4), 'not the original')
|
||||
|
||||
def test_trampoline_bl_elsewhere(self):
|
||||
tramp = PLUGIN + TRAMP_OFF
|
||||
self.refused(record(trampoline_first=b(tramp, MAIN + SITE_OFF + 0x44, True)), 'not the original')
|
||||
|
||||
def test_trampoline_return(self):
|
||||
rec = record()
|
||||
img = bytearray(rec['images']['plugin0'])
|
||||
struct.pack_into('<I', img, TRAMP_OFF + 4, b(PLUGIN + TRAMP_OFF + 4, MAIN + SITE_OFF + 8))
|
||||
rec['images']['plugin0'] = bytes(img)
|
||||
self.refused(rec, 'no b to the site + 4')
|
||||
|
||||
def test_indirect_original(self):
|
||||
self.refused(record(original=0xD65F03C0), 'is a br, blr or ret')
|
||||
|
||||
|
||||
class Diff(unittest.TestCase):
|
||||
def test_lists_differing_words(self):
|
||||
a, c = record(), record()
|
||||
img = bytearray(c['images']['plugin0'])
|
||||
struct.pack_into('<Q', img, 0x13450, 0x2345000000)
|
||||
c['images']['plugin0'] = bytes(img)
|
||||
a['pool'] = c['pool'] = {'base': '0x2000000000', 'size': '0x1000000000'}
|
||||
lines = xp.diff_records(a, c)
|
||||
self.assertEqual(len(lines), 1)
|
||||
self.assertIn('plugin0+0x13450', lines[0])
|
||||
self.assertIn('alias pool', lines[0])
|
||||
self.assertEqual(xp.diff_records(a, record()), [])
|
||||
|
||||
|
||||
# ---- the code the hooks reach
|
||||
SVC_40, RET_W, NOP = 0xD4000801, 0xD65F03C0, 0xD503201F
|
||||
|
||||
|
||||
def code(words, at=0):
|
||||
img = bytearray(0x200)
|
||||
for i, w in enumerate(words):
|
||||
struct.pack_into('<I', img, at + 4 * i, w)
|
||||
return bytes(img)
|
||||
|
||||
|
||||
class Reach(unittest.TestCase):
|
||||
BASE = 0x7005200000
|
||||
|
||||
def test_follows_calls_and_finds_svc(self):
|
||||
# root: bl +0x40; ret. callee at 0x40: svc 0x40; ret
|
||||
img = bytearray(code([b(self.BASE, self.BASE + 0x40, True), RET_W]))
|
||||
struct.pack_into('<II', img, 0x40, SVC_40, RET_W)
|
||||
svcs, imports, indirect, outside = xp.reach(bytes(img), self.BASE, 0x200, [self.BASE], {})
|
||||
self.assertEqual(svcs, {0x40})
|
||||
|
||||
def test_imports_through_plt(self):
|
||||
img = code([b(self.BASE, self.BASE + 0x80, True), RET_W])
|
||||
svcs, imports, _, _ = xp.reach(img, self.BASE, 0x200, [self.BASE], {self.BASE + 0x80: 'malloc'})
|
||||
self.assertEqual((svcs, imports), (set(), {'malloc': 1}))
|
||||
|
||||
def test_stub_returns_at_its_trampoline_branch(self):
|
||||
# stur; bl impl; ldur; b trampoline (where an svc sits): the b is the stub's return
|
||||
stub = [xp.STUR_X30, b(self.BASE + 4, self.BASE + 0x40, True), xp.LDUR_X30, b(self.BASE + 12, self.BASE + 0x100)]
|
||||
img = bytearray(code(stub))
|
||||
struct.pack_into('<I', img, 0x40, RET_W)
|
||||
struct.pack_into('<I', img, 0x100, SVC_40)
|
||||
self.assertEqual(xp.reach(bytes(img), self.BASE, 0x200, [self.BASE], {}, stubs={self.BASE})[0], set())
|
||||
self.assertEqual(xp.reach(bytes(img), self.BASE, 0x200, [self.BASE], {})[0], {0x40})
|
||||
|
||||
def test_indirect_calls_counted(self):
|
||||
img = code([0xD63F0100, RET_W]) # blr x8
|
||||
self.assertEqual(xp.reach(img, self.BASE, 0x200, [self.BASE], {})[2], 1)
|
||||
|
||||
def test_check_refuses_svc_in_callback(self):
|
||||
rec = record()
|
||||
img = bytearray(rec['images']['plugin0'])
|
||||
struct.pack_into('<II', img, CB_OFF, SVC_40, RET_W)
|
||||
struct.pack_into('<I', img, IMPL_OFF, RET_W)
|
||||
rec['images']['plugin0'] = bytes(img)
|
||||
hooks, _ = xp.check_record(rec, IMPLS)
|
||||
reasons, _ = xp.reach_check(rec, hooks, IMPLS)
|
||||
self.assertTrue(any('svc 0x40' in r for r in reasons), reasons)
|
||||
|
||||
def test_check_allows_output_debug_string(self):
|
||||
rec = record()
|
||||
img = bytearray(rec['images']['plugin0'])
|
||||
struct.pack_into('<II', img, CB_OFF, 0xD40004E1, RET_W) # svc 0x27
|
||||
struct.pack_into('<I', img, IMPL_OFF, RET_W)
|
||||
rec['images']['plugin0'] = bytes(img)
|
||||
hooks, _ = xp.check_record(rec, IMPLS)
|
||||
self.assertEqual(xp.reach_check(rec, hooks, IMPLS)[0], [])
|
||||
|
||||
|
||||
# ---- patch lines of the three hook kinds
|
||||
class Lines(unittest.TestCase):
|
||||
def lines(self, rec):
|
||||
hooks, reasons = xp.check_record(rec, IMPLS)
|
||||
self.assertEqual(reasons, [])
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
(Path(d) / 'plugin0.img').write_bytes(rec['images']['plugin0'])
|
||||
rec['modules'][2]['image'] = 'plugin0.img'
|
||||
return xp.plugin_lines(rec, xp.hooks_of(hooks, 'plugin0'), 'mod_x', d, 'plugin0')
|
||||
|
||||
def test_inline(self):
|
||||
out = self.lines(record(original=0xAA0003F3))
|
||||
self.assertIn(f'site mod_x main {SITE_OFF:x} aa0003f3 @call:plugin0+{STUB_OFF:x} aa0003f3', out)
|
||||
self.assertIn(f'stub plugin0 {STUB_OFF:x}', out)
|
||||
self.assertIn(f'seed plugin0 {CB_OFF:x} {IMPL_OFF:x}', out)
|
||||
self.assertIn(f'expect mod_x main {SITE_OFF:x} aa0003f3 {b(MAIN + SITE_OFF, PLUGIN + STUB_OFF):08x}', out)
|
||||
self.assertTrue(out[0].startswith('plugin mod_x plugin0 '))
|
||||
|
||||
def test_replace(self):
|
||||
site, prologue = MAIN + SITE_OFF, 0xA9BA7BFD
|
||||
new = b(site, PLUGIN + THUNK_OFF)
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': prologue, 'new': new, 'fn_start': True}]
|
||||
out = self.lines(record(original=prologue, writes=w))
|
||||
self.assertIn(f'site mod_x main {SITE_OFF:x} a9ba7bfd {new:08x}', out)
|
||||
self.assertIn(f'orig mod_x plugin0 {TRAMP_OFF:x} main {SITE_OFF:x}', out)
|
||||
self.assertIn(f'seed plugin0 {THUNK_OFF:x}', out)
|
||||
|
||||
def test_call(self):
|
||||
new = b(MAIN + SITE_OFF, PLUGIN + THUNK_OFF, True)
|
||||
w = [{'module': 'main', 'offset': SITE_OFF, 'old': 0x94000010, 'new': new}]
|
||||
out = self.lines(record(writes=w))
|
||||
self.assertIn(f'site mod_x main {SITE_OFF:x} 94000010 {new:08x}', out)
|
||||
self.assertFalse(any(x.startswith(('orig', 'stub')) for x in out))
|
||||
|
||||
|
||||
# ---- one service run per plugin
|
||||
class PerPlugin(unittest.TestCase):
|
||||
"""main() with three plugins in three mods and a stand-in for the service run: plugin0 hooks its site, plugin1's
|
||||
init ends the run without a record (a crash), plugin2's run is refused (an svc). Each gets its own status line;
|
||||
only plugin0's set goes into the patch file."""
|
||||
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.d = Path(tmp.name)
|
||||
self.files = []
|
||||
for name in ('A', 'B', 'C'):
|
||||
f = self.d / 'mods' / name / 'exefs' / 'subsdk9'
|
||||
f.parent.mkdir(parents=True)
|
||||
f.write_bytes(nso(PLUGIN_TEXT) + name.encode())
|
||||
self.files.append(f)
|
||||
(self.d / 'patches.txt').write_text('set fps60 TOTK_FPS60 1\n')
|
||||
self.runs = []
|
||||
saved = xp.run, xp.hook_impls
|
||||
self.real_run = xp.run
|
||||
self.addCleanup(lambda: (setattr(xp, 'run', saved[0]), setattr(xp, 'hook_impls', saved[1])))
|
||||
xp.run, xp.hook_impls = self.fake_run, lambda rec: IMPLS
|
||||
|
||||
def fake_run(self, exe, out, tick_offset, timeout, only):
|
||||
self.runs.append(only)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
if only == 'plugin1':
|
||||
return 0xC0000005, 'its init ended the service run without a record (exit 0xc0000005)'
|
||||
rec = record(writes=[] if only != 'plugin0' else None)
|
||||
rec['modules'][2]['file'] = str(self.files[0])
|
||||
for i in (1, 2):
|
||||
rec['modules'].append(dict(rec['modules'][2], name=f'plugin{i}', base=PLUGIN + i * 0x100000,
|
||||
file=str(self.files[i])))
|
||||
for m in rec['modules'][2:]:
|
||||
m['ran'] = m['name'] == only
|
||||
if only == 'plugin2':
|
||||
rec['refused'] = 'svc 0x1 at plugin2+0x50a0'
|
||||
(out / 'plugin0.img').write_bytes(rec.pop('images')['plugin0'])
|
||||
rec['modules'][2]['image'] = 'plugin0.img'
|
||||
(out / 'record.json').write_text(json.dumps(rec))
|
||||
return 3 if rec['refused'] else 0, None
|
||||
|
||||
def test_each_plugin_alone(self):
|
||||
out, status, patched = self.d / 'rec', self.d / 'plugins.status', self.d / 'patches-out.txt'
|
||||
argv = sys.argv
|
||||
sys.argv = ['exefs_plugins.py', '--exe', 'totk.exe', '--out', str(out), '--mods', str(self.d / 'mods'),
|
||||
'--patches', str(self.d / 'patches.txt'), '--patches-out', str(patched), '--status', str(status)]
|
||||
try:
|
||||
code = xp.main()
|
||||
finally:
|
||||
sys.argv = argv
|
||||
self.assertEqual(code, 1)
|
||||
self.assertEqual(self.runs, ['-', 'plugin0', 'plugin0', 'plugin1', 'plugin2'])
|
||||
lines = {line.split('\t')[0]: line.split('\t') for line in status.read_text().splitlines()}
|
||||
key = {n: xp.fnv(f'{n}/exefs/subsdk9'.encode()) for n in 'ABC'}
|
||||
self.assertEqual(lines[key['A']][2:], ['plugin', '-', 'plugin0, 1 hooks'])
|
||||
self.assertEqual(lines[key['B']][2], 'unsupported')
|
||||
self.assertIn('ended the service run', lines[key['B']][4])
|
||||
self.assertEqual(lines[key['C']][2], 'unsupported')
|
||||
self.assertIn('svc 0x1', lines[key['C']][4])
|
||||
text = patched.read_text()
|
||||
self.assertIn(f'set mod_{key["A"].lower()} ', text)
|
||||
self.assertNotIn(f'mod_{key["B"].lower()}', text)
|
||||
self.assertNotIn(f'mod_{key["C"].lower()}', text)
|
||||
self.assertIn((out / 'plugin0' / 'run1' / 'plugin0.img').resolve().as_posix(), text)
|
||||
|
||||
def test_run_reasons(self):
|
||||
# the real run(): a hang becomes its own reason, a crash names the FATAL line
|
||||
saved = xp.subprocess.run
|
||||
self.addCleanup(setattr, xp.subprocess, 'run', saved)
|
||||
|
||||
def hang(cmd, **kw):
|
||||
raise xp.subprocess.TimeoutExpired(cmd, kw['timeout'], output=b'started', stderr=None)
|
||||
xp.subprocess.run = hang
|
||||
self.assertEqual(self.real_run(None, self.d / 'h', 0, 7, 'plugin0'), (None, 'its init did not return within 7 s'))
|
||||
xp.subprocess.run = lambda cmd, **kw: SimpleNamespace(returncode=0xC0000005, stdout='x\nFATAL: access violation '
|
||||
'(0xc0000005) at host 1\n', stderr='')
|
||||
code, why = self.real_run(None, self.d / 'c', 0, 7, 'plugin0')
|
||||
self.assertEqual(why, 'its init crashed the service run: access violation (0xc0000005) at host 1')
|
||||
|
||||
def test_ran_view_leaves_out_plugins_that_did_not_run(self):
|
||||
rec = record()
|
||||
rec['modules'].append(dict(rec['modules'][2], name='plugin1', base=PLUGIN + 0x100000, ran=False))
|
||||
rec['modules'][2]['ran'] = True
|
||||
rec['images']['plugin1'] = rec['images']['plugin0']
|
||||
view = xp.ran_view(rec)
|
||||
self.assertEqual([m['name'] for m in view['modules'] if m['plugin']], ['plugin0'])
|
||||
self.assertEqual(set(view['images']), {'plugin0'})
|
||||
whole = record() # no "ran": every init ran, the record is its own view
|
||||
self.assertIs(xp.ran_view(whole), whole)
|
||||
|
||||
|
||||
# ---- the generator on those lines
|
||||
def code_nso(words_at, size_text=0x5000):
|
||||
"""An NSO as nso() whose .text holds words_at {offset: word}."""
|
||||
raw = bytearray(nso(size_text))
|
||||
for off, w in words_at.items():
|
||||
struct.pack_into('<I', raw, 0x100 + off, w)
|
||||
return bytes(raw)
|
||||
|
||||
|
||||
FN = 0x100 # the hooked main function: mov x19, x0; mov x0, x19; ret
|
||||
|
||||
|
||||
class Generator(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.d = Path(tmp.name)
|
||||
exefs = self.d / 'exefs'
|
||||
exefs.mkdir()
|
||||
(exefs / 'main').write_bytes(code_nso({FN: 0xAA0003F3, FN + 4: 0xAA1303E0, FN + 8: RET_W}))
|
||||
(exefs / 'sdk').write_bytes(nso(0x1000))
|
||||
stub = PLUGIN_TEXT_STUB = 0x4000
|
||||
self.plugin_file = self.d / 'subsdk9'
|
||||
self.plugin_file.write_bytes(nso(0x5000))
|
||||
self.mods = layout.layout(exefs, [self.plugin_file])
|
||||
self.bases = {n: b_ for n, b_, _ in self.mods}
|
||||
pb = self.bases['plugin0']
|
||||
# the image after the init: a stub at 0x4000 (b to a trampoline at 0x2000), an impl at 0x4100, a callback at
|
||||
# 0x4200
|
||||
img = bytearray(Nso(str(self.plugin_file)).image)
|
||||
struct.pack_into('<4IQ', img, stub, xp.STUR_X30, b(pb + stub + 4, pb + 0x4100, True), xp.LDUR_X30,
|
||||
b(pb + stub + 12, pb + 0x2000), pb + 0x4200)
|
||||
struct.pack_into('<I', img, 0x4100, RET_W)
|
||||
struct.pack_into('<I', img, 0x4200, RET_W)
|
||||
struct.pack_into('<II', img, 0x2000, 0xAA0003F3, b(pb + 0x2004, self.bases['main'] + FN + 4))
|
||||
(self.d / 'plugin0.img').write_bytes(bytes(img))
|
||||
n = Nso(str(self.plugin_file))
|
||||
self.sha = hashlib.sha256(bytes(img[n.text[0]:n.text[0] + n.text[1]])).hexdigest()
|
||||
|
||||
def table(self, site_line, extra=()):
|
||||
text = '\n'.join(['set mod_x TOTK_MODPATCH_X 0 ordered',
|
||||
f'plugin mod_x plugin0 {self.sha} {(self.d / "plugin0.img").as_posix()}',
|
||||
f'expect mod_x main {FN:x} aa0003f3 14000000', site_line, *extra]) + '\n'
|
||||
(self.d / 'p.txt').write_text(text)
|
||||
return patches.load(self.d / 'p.txt', self.mods)
|
||||
|
||||
def emit(self, table, name, entry, end, origs=()):
|
||||
mod = gen.open_module(name, self.d / ('exefs/main' if name == 'main' else 'subsdk9'), self.bases[name], table,
|
||||
plugin=name != 'main')
|
||||
e = gen.Emitter(mod, {entry}, table.for_module(name, self.bases[name]), origs=set(origs))
|
||||
return gen.chunk_text(name, 0, e, [(entry, end)])
|
||||
|
||||
def test_inline_hook_calls_the_stub_then_the_original(self):
|
||||
t = self.table(f'site mod_x main {FN:x} aa0003f3 @call:plugin0+4000 aa0003f3', ['stub plugin0 4000',
|
||||
'seed plugin0 4100 4200'])
|
||||
m = self.bases['main'] + FN
|
||||
c = self.emit(t, 'main', m, m + 12)
|
||||
stub = self.bases['plugin0'] + 0x4000
|
||||
on = c[c.index('if (rt_patch_on[0])'):]
|
||||
self.assertLess(on.index(f'f_{stub:x}(c);'), on.index('} else {')) # the hook, then the original, when on
|
||||
head = on[:on.index(f'f_{stub:x}(c);')]
|
||||
self.assertNotIn('c->x[30] =', head) # no x30 written for the call
|
||||
self.assertIn(f'void f_{stub:x}(Cpu *c);', c)
|
||||
h = patches.header_text(t, 'MARK')
|
||||
self.assertIn('#define RT_PLUGIN_COUNT 1', h)
|
||||
self.assertIn(f'X(0, "main", {FN:#x}, 0xaa0003f3u, 0x14000000u)', h)
|
||||
self.assertIn('X("plugin0", 0x4100) X("plugin0", 0x4200)', h)
|
||||
|
||||
def test_stub_returns_instead_of_branching_to_its_trampoline(self):
|
||||
t = self.table(f'site mod_x main {FN:x} aa0003f3 @call:plugin0+4000 aa0003f3', ['stub plugin0 4000'])
|
||||
stub = self.bases['plugin0'] + 0x4000
|
||||
c = self.emit(t, 'plugin0', stub, stub + 16)
|
||||
body = c[c.index(f'void f_{stub:x}(Cpu *c) {{'):]
|
||||
self.assertIn('return;', body)
|
||||
self.assertNotIn(f'{self.bases["plugin0"] + 0x2000:#x}', body) # the trampoline is not reached
|
||||
|
||||
def test_replace_hook_keeps_an_unpatched_copy(self):
|
||||
pb = self.bases['plugin0']
|
||||
m = self.bases['main'] + FN
|
||||
new = b(m, pb + 0x4200)
|
||||
t = self.table(f'site mod_x main {FN:x} aa0003f3 {new:08x}', [f'orig mod_x plugin0 2000 main {FN:x}',
|
||||
'seed plugin0 4200'])
|
||||
self.assertEqual(t.origs_for('main', self.bases['main']), {m})
|
||||
self.assertEqual(t.orig_entries('plugin0', self.bases), {pb + 0x2000: m})
|
||||
c = self.emit(t, 'main', m, m + 12, origs=[m])
|
||||
copy = c[c.index(f'void fo_{m:x}(Cpu *c) {{'):]
|
||||
self.assertNotIn('rt_patch_on', copy)
|
||||
self.assertIn('rt_patch_on[0]', c[:c.index(f'void fo_{m:x}')])
|
||||
tab = gen.table_text('plugin0', [pb + 0x4200, pb + 0x2000], {pb + 0x2000: gen.orig_name(m)})
|
||||
self.assertIn(f'void fo_{m:x}(Cpu *c);', tab)
|
||||
self.assertIn(f'fo_{m:x}', tab[tab.index('_fn_ptrs'):])
|
||||
|
||||
def test_image_must_match_its_hash(self):
|
||||
t = self.table(f'site mod_x main {FN:x} aa0003f3 @call:plugin0+4000 aa0003f3', ['stub plugin0 4000'])
|
||||
t.plugins['plugin0']['sha256'] = '0' * 64
|
||||
with self.assertRaises(patches.PatchError):
|
||||
gen.open_module('plugin0', self.plugin_file, self.bases['plugin0'], t)
|
||||
|
||||
def test_chunk_key_takes_the_unpatched_copy(self):
|
||||
# the lift cache (recomp/gen.py chunk_key): a chunk whose function gets an unpatched copy is keyed apart, and a
|
||||
# chunk without one keeps the key it had before plugins
|
||||
fns = [(0x100, 0x10c)]
|
||||
plain = gen.chunk_key('m', 0, fns, {}, [], {}, [], None, None)
|
||||
self.assertEqual(gen.chunk_key('m', 0, fns, {}, [], {}, [], None, None, {0x200}), plain)
|
||||
self.assertNotEqual(gen.chunk_key('m', 0, fns, {}, [], {}, [], None, None, {0x100}), plain)
|
||||
site = {0x100: [(0, [('call', 0x7005204000), 0xAA0003F3])]}
|
||||
self.assertNotEqual(gen.chunk_key('m', 0, fns, site, [0x100], {}, [], None, None),
|
||||
gen.chunk_key('m', 0, fns, {0x100: [(0, [('call', 0x7005204010), 0xAA0003F3])]}, [0x100],
|
||||
{}, [], None, None))
|
||||
|
||||
def test_no_plugin_lines_no_plugin_header(self):
|
||||
(self.d / 'q.txt').write_text('set fps60 TOTK_FPS60 0\n')
|
||||
self.assertNotIn('PLUGIN', patches.header_text(patches.load(self.d / 'q.txt', self.mods), 'MARK'))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+215
-3
@@ -18,6 +18,7 @@ ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
from installer import ini, runner, screens, state, steps # noqa: E402
|
||||
from installer.events import Reporter # noqa: E402
|
||||
from installer import layout as layouts # noqa: E402
|
||||
from installer.layout import Layout # noqa: E402
|
||||
|
||||
|
||||
@@ -326,7 +327,7 @@ class LiftRunTest(unittest.TestCase):
|
||||
self.dest, temp = base / 'install', base / 'temp'
|
||||
self.dest.mkdir()
|
||||
temp.mkdir()
|
||||
layout = SimpleNamespace(recomp=ROOT / 'recomp', tools=ROOT / 'tools',
|
||||
layout = SimpleNamespace(dev=True, recomp=ROOT / 'recomp', tools=ROOT / 'tools',
|
||||
patches=lambda version: base / 'patches-v143.txt')
|
||||
self.ctx = SimpleNamespace(dest=self.dest, temp=temp, layout=layout, keys={'lift': 'k'},
|
||||
state={'version': '1.4.3'}, reporter=Reporter(out=io.StringIO()))
|
||||
@@ -383,6 +384,85 @@ class LiftRunTest(unittest.TestCase):
|
||||
self.assertEqual(self.outputs(), [False, False])
|
||||
self.assertIn('warn.exefs_skipped', self.ctx.reporter.out.getvalue())
|
||||
|
||||
def plugin_setup(self, kind, more=()):
|
||||
"""A mod with an exlaunch plugin (SQ), and one more mod per (name, kind) of more; the fake service run writes
|
||||
each plugin's status line of its kind."""
|
||||
kinds = {'SQ': kind, **dict(more)}
|
||||
for name in kinds:
|
||||
(self.dest / 'mods' / name / 'exefs').mkdir(parents=True, exist_ok=True)
|
||||
(self.dest / 'mods' / name / 'exefs' / 'subsdk9').write_bytes(b'NSO0' + name.encode())
|
||||
plugin = self.dest / 'mods' / 'SQ' / 'exefs' / 'subsdk9'
|
||||
dll = self.dest.parent / 'release-bin' / 'SDL3.dll'
|
||||
dll.parent.mkdir(exist_ok=True)
|
||||
dll.write_bytes(b'MZ')
|
||||
(self.ctx.temp / 'build').mkdir(parents=True, exist_ok=True)
|
||||
self.ctx.layout = SimpleNamespace(**vars(self.ctx.layout), llvm='llvm', runtime='runtime', ninja='ninja', dlls=[dll],
|
||||
deps=None, sysroot=None, runtime_lib_for=lambda cpu: None,
|
||||
pgo=lambda version: None, expected=lambda version: 'expected.json')
|
||||
saved = self.run_tool
|
||||
|
||||
def run_tool(ctx, step, script, args, parse=None, cwd=None):
|
||||
code, log = saved(ctx, step, script, args, parse, cwd)
|
||||
args = [str(a) for a in args]
|
||||
if Path(script).name == 'exefs_plugins.py':
|
||||
Path(args[args.index('--patches-out') + 1]).write_text('patches\n')
|
||||
status = Path(args[args.index('--status') + 1])
|
||||
status.parent.mkdir(parents=True, exist_ok=True)
|
||||
lines = []
|
||||
for name, k in kinds.items():
|
||||
h = 0x811C9DC5 # FNV-1a 32 of "<mod>/exefs/subsdk9", as tools/exefs_plugins.py keys it
|
||||
for b in f'{name}/exefs/subsdk9'.encode():
|
||||
h = ((h ^ b) * 0x01000193) & 0xFFFFFFFF
|
||||
lines.append(f'{h:08X}\t00000000\t{k}\t-\tnote\n')
|
||||
status.write_text(''.join(lines))
|
||||
return (1 if 'unsupported' in kinds.values() else 0), log
|
||||
return code, log
|
||||
steps.run_tool = run_tool
|
||||
return plugin
|
||||
|
||||
def test_plugin_stage(self):
|
||||
# the lift with the plugin, a totk.exe of it, the service run, then the lift with its hook sites
|
||||
plugin = self.plugin_setup('plugin')
|
||||
steps.Lift().run(self.ctx)
|
||||
self.assertEqual([c[0] for c in self.calls], ['exefs_patches.py', 'gen.py', 'build_game.py',
|
||||
'exefs_plugins.py', 'gen.py', 'exefs_patches.py --folded'])
|
||||
first, last = self.calls[1][1], self.calls[4][1]
|
||||
self.assertEqual(first[first.index('--plugins') + 1], str(plugin))
|
||||
self.assertEqual(last[last.index('--plugins') + 1], str(plugin))
|
||||
self.assertEqual(last[last.index('--patches') + 1], str(self.ctx.temp / 'patches-plugins.txt'))
|
||||
self.assertTrue(steps.plugins_status(self.ctx).is_file())
|
||||
# the service run's totk.exe finds its DLLs beside it
|
||||
self.assertTrue((self.ctx.temp / 'build' / 'SDL3.dll').is_file())
|
||||
|
||||
def test_refused_plugin_is_not_lifted(self):
|
||||
# a refused plugin: its status line stays (the Mods tab says why), the game is lifted without it
|
||||
self.plugin_setup('unsupported')
|
||||
steps.Lift().run(self.ctx)
|
||||
last = self.calls[4][1]
|
||||
self.assertNotIn('--plugins', last)
|
||||
self.assertTrue(steps.plugins_status(self.ctx).is_file())
|
||||
self.assertNotIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue())
|
||||
|
||||
def test_refused_plugin_stays_in_the_layout_of_a_built_one(self):
|
||||
# one plugin built, one refused: the lift with the hook sites keeps both in the layout (the records hold the
|
||||
# addresses of the service runs' layout); the refused one has no patch set, its status line says why
|
||||
self.plugin_setup('plugin', [('ZZ', 'unsupported')])
|
||||
steps.Lift().run(self.ctx)
|
||||
last = self.calls[4][1]
|
||||
mods = self.dest / 'mods'
|
||||
self.assertEqual(last[last.index('--plugins') + 1],
|
||||
','.join(str(mods / n / 'exefs' / 'subsdk9') for n in ('SQ', 'ZZ')))
|
||||
self.assertIn('unsupported', steps.plugins_status(self.ctx).read_text())
|
||||
self.assertNotIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue())
|
||||
|
||||
def test_failed_plugin_build_warns(self):
|
||||
self.plugin_setup('plugin')
|
||||
self.fail = {'build_game.py'}
|
||||
steps.Lift().run(self.ctx)
|
||||
self.assertNotIn('--plugins', self.calls[-2][1])
|
||||
self.assertFalse(steps.plugins_status(self.ctx).is_file())
|
||||
self.assertIn('warn.plugins_skipped', self.ctx.reporter.out.getvalue())
|
||||
|
||||
|
||||
class CompileLockedTest(unittest.TestCase):
|
||||
"""Compile.run when the linker cannot write its totk.exe: another program holds the file (a game started from the
|
||||
@@ -398,7 +478,7 @@ class CompileLockedTest(unittest.TestCase):
|
||||
dest, temp = base / 'install', base / 'temp'
|
||||
dest.mkdir()
|
||||
temp.mkdir()
|
||||
layout = SimpleNamespace(tools=ROOT / 'tools', llvm=base / 'llvm', deps=None, sysroot=None, runtime=base,
|
||||
layout = SimpleNamespace(dev=True, tools=ROOT / 'tools', llvm=base / 'llvm', deps=None, sysroot=None, runtime=base,
|
||||
ninja=base / 'ninja', dlls=[], runtime_lib_for=lambda cpu: None,
|
||||
pgo=lambda version: None, expected=lambda version: base / 'expected.json')
|
||||
self.ctx = SimpleNamespace(dest=dest, temp=temp, layout=layout, state={'version': '1.4.3'},
|
||||
@@ -555,6 +635,137 @@ class CleanupTest(unittest.TestCase):
|
||||
self.assertEqual(self.run_cleanup(), 'cleanup.romfs_unreadable')
|
||||
|
||||
|
||||
def fake_release(root, version='1.0+abc'):
|
||||
"""A folder release_layout reads as a release: VERSION, the Python file and one file per other kit item."""
|
||||
for name in layouts.KIT_ITEMS:
|
||||
if '.' in name or name == 'VERSION':
|
||||
(root / name).parent.mkdir(parents=True, exist_ok=True)
|
||||
(root / name).write_text(version + '\n' if name == 'VERSION' else name)
|
||||
else:
|
||||
(root / name).mkdir(parents=True, exist_ok=True)
|
||||
(root / name / 'f.txt').write_text(name)
|
||||
layouts.release_python(root).parent.mkdir(parents=True, exist_ok=True)
|
||||
layouts.release_python(root).write_bytes(b'')
|
||||
(root / 'TotK-Setup.exe').write_bytes(b'launcher')
|
||||
(root / 'installer' / '__pycache__').mkdir()
|
||||
(root / 'installer' / '__pycache__' / 'x.pyc').write_bytes(b'')
|
||||
return layouts.release_layout(root)
|
||||
|
||||
|
||||
@unittest.skipUnless(steps.os.name == 'nt', 'the build kit is Windows-only')
|
||||
class KitTest(unittest.TestCase):
|
||||
"""The build kit: the setup's release folder copied into the install folder, the later steps and the rebuild
|
||||
command pointed at it, and an older setup's temp folder taken over."""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
base = Path(self.tmp.name)
|
||||
self.setup, self.dest = base / 'TotK-Setup', base / 'game'
|
||||
self.dest.mkdir()
|
||||
self.layout = fake_release(self.setup)
|
||||
self.missing = mock.patch.object(Layout, 'missing', lambda self, cpu='v3': [])
|
||||
self.missing.start()
|
||||
|
||||
def tearDown(self):
|
||||
self.missing.stop()
|
||||
self.tmp.cleanup()
|
||||
|
||||
def ctx(self, layout):
|
||||
return SimpleNamespace(layout=layout, dest=self.dest, reporter=Reporter(out=io.StringIO()))
|
||||
|
||||
def install(self):
|
||||
ctx = self.ctx(self.layout)
|
||||
kit = steps.Kit()
|
||||
self.assertFalse(kit.current(ctx, {}))
|
||||
kit.run(ctx)
|
||||
kit.settle(ctx)
|
||||
return ctx, kit
|
||||
|
||||
def test_copies_the_release_and_the_next_steps_run_from_it(self):
|
||||
ctx, kit = self.install()
|
||||
folder = self.dest / 'build-kit'
|
||||
self.assertEqual(ctx.layout.root, folder.resolve())
|
||||
self.assertEqual(ctx.layout.release, '1.0+abc')
|
||||
self.assertEqual(sorted(p.name for p in folder.iterdir()), sorted(layouts.KIT_ITEMS))
|
||||
self.assertFalse((folder / 'installer' / '__pycache__').exists())
|
||||
self.assertFalse((self.dest / 'build-kit.new').exists() or (self.dest / 'build-kit.old').exists())
|
||||
self.assertTrue(kit.current(self.ctx(self.layout), {}))
|
||||
|
||||
def test_the_kits_own_rebuild_keeps_it(self):
|
||||
ctx, kit = self.install()
|
||||
(self.dest / 'build-kit' / 'tools' / 'mark').write_text('kept')
|
||||
self.assertTrue(kit.current(ctx, {}))
|
||||
kit.run(ctx)
|
||||
kit.settle(ctx)
|
||||
self.assertEqual((self.dest / 'build-kit' / 'tools' / 'mark').read_text(), 'kept')
|
||||
|
||||
def test_a_newer_setup_replaces_the_kit(self):
|
||||
self.install()
|
||||
(self.dest / 'build-kit' / 'tools' / 'old-only').write_text('old')
|
||||
self.layout = fake_release(Path(self.tmp.name) / 'TotK-Setup-2', '1.1+def')
|
||||
ctx, _ = self.install()
|
||||
self.assertEqual(ctx.layout.release, '1.1+def')
|
||||
self.assertFalse((self.dest / 'build-kit' / 'tools' / 'old-only').exists())
|
||||
|
||||
def test_rebuild_command_is_relative_to_the_install_folder(self):
|
||||
ctx, _ = self.install()
|
||||
cmd = steps.rebuild_command(ctx)
|
||||
self.assertEqual(cmd, [str(Path('build-kit', 'python', 'pythonw.exe' if (
|
||||
ctx.layout.python.with_name('pythonw.exe')).is_file() else 'python.exe')), '-c', steps.REBUILD_BOOT,
|
||||
'build-kit', '--root', 'build-kit', '--rebuild', '.'])
|
||||
self.assertFalse(any(str(self.dest) in a or str(self.setup) in a for a in cmd))
|
||||
|
||||
def test_the_kits_builds_are_portable(self):
|
||||
ctx, _ = self.install()
|
||||
ctx.temp, ctx.state = self.dest / 'build-cache' / 'work', {'version': '1.4.3'}
|
||||
self.assertIn('--portable', steps.build_args(ctx, ctx.temp / 'gen'))
|
||||
ctx.layout = Layout(root=ROOT, dev=True, release='x', python=Path('python'), tools=Path('tools'),
|
||||
recomp=Path('recomp'), llvm=Path('llvm'), deps=None, runtime=Path('rt'),
|
||||
runtime_lib=Path('lib'), ninja=Path('ninja'), hactool=Path('hactool'), xlate=Path('xlate'))
|
||||
self.assertNotIn('--portable', steps.build_args(ctx, ctx.temp / 'gen'))
|
||||
|
||||
def test_dev_mode_has_no_kit(self):
|
||||
ctx = self.ctx(SimpleNamespace(dev=True, release='x', root=ROOT))
|
||||
kit = steps.Kit()
|
||||
self.assertTrue(kit.current(ctx, {}))
|
||||
self.assertEqual(kit.run(ctx), {})
|
||||
kit.settle(ctx)
|
||||
self.assertFalse((self.dest / 'build-kit').exists())
|
||||
|
||||
def test_temp_folder_of_an_older_setup_moves_in(self):
|
||||
old = layouts.temp_folder(self.setup / 'temp', self.dest)
|
||||
(old / 'gen').mkdir(parents=True)
|
||||
(old / 'gen' / '.install-key').write_text('k')
|
||||
(self.dest / 'build-cache').mkdir()
|
||||
(self.dest / 'build-cache' / 'rebuild.txt').write_text(
|
||||
'\n'.join(['py', '-c', 'boot', str(self.setup), '--root', str(self.setup), '--rebuild', str(self.dest)]))
|
||||
temp = layouts.temp_for(self.layout, self.dest)
|
||||
self.assertEqual(temp, self.dest / 'build-cache' / 'work')
|
||||
self.assertIn('moved', steps.adopt_old_temp(self.layout, self.dest, temp))
|
||||
self.assertEqual((temp / 'gen' / '.install-key').read_text(), 'k')
|
||||
self.assertFalse(old.exists())
|
||||
self.assertIsNone(steps.adopt_old_temp(self.layout, self.dest, temp)) # once: the new one exists now
|
||||
|
||||
def test_no_temp_folder_to_take_over(self):
|
||||
temp = layouts.temp_for(self.layout, self.dest)
|
||||
self.assertIsNone(steps.adopt_old_temp(self.layout, self.dest, temp))
|
||||
(self.dest / 'build-cache').mkdir()
|
||||
(self.dest / 'build-cache' / 'rebuild.txt').write_text('build-kit\\python\\pythonw.exe\n-c\nboot\nbuild-kit\n')
|
||||
self.assertIsNone(steps.adopt_old_temp(self.layout, self.dest, temp))
|
||||
|
||||
|
||||
class RuntimeKeyTest(unittest.TestCase):
|
||||
def test_clangs_folder_is_not_in_the_key(self):
|
||||
layout = SimpleNamespace(clang=Path('clang'), tools=Path('tools'), runtime=Path('rt'), deps=None, sysroot=None,
|
||||
dlls=(), runtime_lib_for=lambda cpu: None)
|
||||
keys = []
|
||||
for where in ('C:/setup/llvm/bin', 'D:/game/build-kit/llvm/bin'):
|
||||
out = f'clang version 23\nTarget: x86_64-w64-windows-gnu\nInstalledDir: {where}\nConfiguration file: {where}/x.cfg\n'
|
||||
with mock.patch.object(steps.subprocess, 'run', return_value=SimpleNamespace(stdout=out)):
|
||||
keys.append(steps.runtime_key(SimpleNamespace(layout=layout)))
|
||||
self.assertEqual(keys[0], keys[1])
|
||||
|
||||
|
||||
class FreeSpaceTest(unittest.TestCase):
|
||||
def test_missing_folders_use_the_nearest_existing_parent(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
@@ -727,7 +938,8 @@ class FontsTest(unittest.TestCase):
|
||||
|
||||
def test_all_steps_order(self):
|
||||
self.assertEqual([s.id for s in steps.all_steps()],
|
||||
['check', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss', 'cleanup'])
|
||||
['check', 'kit', 'fonts', 'extract', 'lift', 'compile', 'prepare', 'pc_prompts', 'shapes', 'dlss',
|
||||
'cleanup'])
|
||||
|
||||
def test_extracts_into_the_install_folder(self):
|
||||
self.fake_tool()
|
||||
|
||||
@@ -55,6 +55,7 @@ TESTS = {
|
||||
'ncz_test': ('.venv-py314', None, [], ''),
|
||||
'romfs_prompts_test': ('.venv-py314', None, [], ''),
|
||||
'exefs_patches_test': ('.venv', None, [], ''),
|
||||
'exefs_plugins_test': ('.venv', None, [], ''),
|
||||
'settings_doc_check': ('.venv', None, [], ''),
|
||||
'mouse_direct_check': ('.venv', 'cc', ['--cc', '<cc>'], 'compiles its checks with clang'),
|
||||
'cheats_vm_check': ('.venv', 'cc', ['--cc', '<cc>'], 'compiles runtime/cheats.c into a harness with clang'),
|
||||
|
||||
@@ -45,6 +45,18 @@ def main():
|
||||
problems = [f'{name}: in runtime/cfg.c known[], described in neither totk.ini.example nor docs/settings.md'
|
||||
for name in sorted(known - in_example - in_doc)]
|
||||
problems += [f'{name}: in docs/settings.md, not in runtime/cfg.c known[]' for name in sorted(in_doc - known)]
|
||||
# cfg_value and the cfg_int/cfg_double/cfg_bool over it stop the game when the key is unset and metas[] has no row;
|
||||
# a line that tests cfg_get of the same key first reads it only when it is set
|
||||
metas = set(re.findall(r'\{"([A-Z0-9_]+)", "[^"]*", (?:"[^"]*"|NULL)', cfg))
|
||||
unsafe = set()
|
||||
for path in sorted((ROOT / 'runtime').rglob('*')):
|
||||
if path.suffix in ('.c', '.cpp', '.h'):
|
||||
for line in path.read_text(encoding='utf-8', errors='replace').splitlines():
|
||||
for name in re.findall(r'\bcfg_(?:value|int|double|bool)\("TOTK_([A-Z0-9_]+)"\)', line):
|
||||
if name not in metas and f'cfg_get("TOTK_{name}")' not in line:
|
||||
unsafe.add((name, path.relative_to(ROOT).as_posix()))
|
||||
problems += [f'{name}: {where} reads it through cfg_value, runtime/cfg.c metas[] has no row: the game stops '
|
||||
f'when it is unset' for name, where in sorted(unsafe)]
|
||||
defaults = code_defaults(cfg)
|
||||
for name, written, _ in rows:
|
||||
code = defaults.get(name.upper())
|
||||
|
||||
+347
-97
@@ -23,13 +23,16 @@ them, else build/. tools/romfs_index.py writes build/<version>/romfs-index.bin a
|
||||
directory's index.json (runtime/romfs.c reads the index next to totk.exe). Without that index.json the build stops:
|
||||
the exe would stop at start. --no-romfs builds it anyway.
|
||||
|
||||
Private builds reuse the main build's objects. When --out differs from build/<version>, the main build's generated
|
||||
objects are up to date (checked with a dry run of its build.ninja) and every header they include from runtime/ is
|
||||
byte-identical in --runtime (the list comes from the main build's Ninja deps log), build_game.py hardlinks those
|
||||
objects into <out>/obj and compiles only runtime/*.c. A hardlink is a snapshot: clang and Ninja replace an object by
|
||||
renaming a new file over it, so a later main rebuild leaves the linked copies alone. Otherwise (a header differs, a
|
||||
main object is missing or stale, another --gen) it compiles everything and prints one line saying why. --full-build
|
||||
skips the check.
|
||||
Private builds reuse the main build's objects. When --out differs from build/<version>, build_game.py hardlinks into
|
||||
<out>/obj every generated object of build/<version> whose compile is provably the one ours would run, and compiles the
|
||||
rest (main_objects): the main build.ninja has the same compiler, flags and rule, the main generated objects are up to
|
||||
date (a dry run of its build.ninja), and every file the main compile read (its Ninja deps log: the .c, the runtime/ and
|
||||
generated headers, the system headers) is byte-identical on our side and unchanged since the object was written, and
|
||||
so is the compiler. --gen may name another generated C directory, such as a private lift of a patch: only the chunks
|
||||
that differ from generated/<version> compile. A hardlink is a snapshot: clang and Ninja replace an object by renaming
|
||||
a new file over it, so a later main rebuild leaves the linked copies alone. The build prints `reused N of M generated
|
||||
objects ..., compiling K`, --verbose lists the K with the reason; when nothing can be reused (other flags, a stale main
|
||||
build) it prints one line saying why. --full-build skips the check.
|
||||
|
||||
Object manifest. A flipped bit in an object written by an earlier build stays forever, because Ninja rebuilds an object
|
||||
only when its inputs change, and forks hardlink the object into every private build. So each build dir keeps
|
||||
@@ -44,6 +47,9 @@ big machine does not run out of memory on the largest chunks. Each phase reports
|
||||
of its processes (Windows job object accounting: committed private bytes).
|
||||
|
||||
--ninja PATH names the Ninja to run; without MSYS2 the bin/ninja.exe of the release folder (tools/make_release.py) is used.
|
||||
--portable (the setup's builds, installer/steps.py build_args) writes every path on the tools' drive relative to the
|
||||
folder above tools/, through '..' where needed, so the install folder that holds the build kit, the generated C and the
|
||||
build folder can move and the next build stays incremental (PORTABLE).
|
||||
--toolchain DIR builds with an llvm-mingw tree (DIR/bin/clang.exe) instead of MSYS2 UCRT64 clang. Ninja still comes
|
||||
from MSYS2. The libraries and headers MSYS2 supplies (Vulkan, SDL3, Opus, libvpx, VMA) then come from --deps DIR
|
||||
(default toolchain/deps-mingw: include/, lib/ with the import libraries and libvpx.a, bin/ with the DLLs copied next to
|
||||
@@ -62,6 +68,7 @@ import ctypes.wintypes as wt # the job-object structures below; importing it w
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
@@ -117,6 +124,13 @@ def set_cpu(cpu):
|
||||
# keeps the names equal between the training build and a build from another tree (a worktree, an install).
|
||||
PGO_NAMES = ['-mllvm', '-static-func-full-module-prefix=false']
|
||||
PGO_INPUT = None # the .profdata of --pgo FILE: every compile depends on it, so a new profile rebuilds them
|
||||
# --portable (the setup's builds in an install folder, installer/steps.py build_args): build.ninja names every path on
|
||||
# ROOT's drive relative to ROOT, the compiler too. clang names its own headers by its absolute folder whatever argv[0]
|
||||
# says, so with --toolchain the compiles also get -resource-dir and --sysroot relative to ROOT (the folders clang would
|
||||
# take itself); the Ninja deps log then holds no absolute path. Ninja runs in ROOT, so a folder that holds ROOT, the
|
||||
# generated C and the build folder (the install folder with build-kit/ and build-cache/work/) can move without a full
|
||||
# recompile.
|
||||
PORTABLE = False
|
||||
|
||||
|
||||
def set_pgo(pgo, out):
|
||||
@@ -188,8 +202,10 @@ DEFAULT_DEPS = ROOT / 'toolchain' / 'deps-mingw'
|
||||
# Runtime sources compiled per install, not in the library: romfs.c includes romfs_config.h, patches.c and ov_zstd.c
|
||||
# the patches.h of the player's game version (its patch page is the page after that version's main; ov_zstd.c's host
|
||||
# functions sit on it). mouse_direct.c and ov_ability.c also register host callbacks relative to that version's patch
|
||||
# page. A library source that includes one of PER_VERSION_HEADERS stops the library build (library_sources).
|
||||
INSTALL_SIDE = {'romfs.c', 'patches.c', 'ov_zstd.c', 'mouse_direct.c', 'ov_ability.c'}
|
||||
# page. plugin_init.c reads the install's own plugin records from patches.h (RT_PLUGINS, RT_PLUGIN_WRITES): a mod plugin
|
||||
# the rebuild built in. A library source that includes one of PER_VERSION_HEADERS stops the library build
|
||||
# (library_sources).
|
||||
INSTALL_SIDE = {'romfs.c', 'patches.c', 'ov_zstd.c', 'mouse_direct.c', 'ov_ability.c', 'plugin_init.c'}
|
||||
PER_VERSION_HEADERS = ('"patches.h"', '"romfs_config.h"')
|
||||
LIB_NAME = 'libtotkrt.a'
|
||||
AR = UCRT / 'llvm-ar.exe' if WIN else on_path('llvm-ar')
|
||||
@@ -208,14 +224,27 @@ MEM_BUDGET = 50 * GiB # total for all compile jobs
|
||||
|
||||
def below_root(p):
|
||||
"""Resolved p relative to ROOT, else to MAIN_ROOT (the same relative path through the worktree's junctions), else
|
||||
absolute."""
|
||||
absolute; with --portable relative to ROOT through '..' when it is on ROOT's drive."""
|
||||
p = Path(p).resolve()
|
||||
for root in (ROOT, MAIN_ROOT):
|
||||
if p.is_relative_to(root):
|
||||
return p.relative_to(root)
|
||||
if PORTABLE:
|
||||
try:
|
||||
return Path(os.path.relpath(p, ROOT))
|
||||
except ValueError: # another drive
|
||||
pass
|
||||
return p
|
||||
|
||||
|
||||
def tool_arg(p):
|
||||
"""A program for a Ninja command line: relative like the other paths with --portable (with backslashes on Windows:
|
||||
CreateProcess does not find a relative program named with forward slashes), else absolute."""
|
||||
if not PORTABLE:
|
||||
return cmd_arg(p.as_posix())
|
||||
return cmd_arg(str(below_root(p)) if WIN else below_root(p).as_posix())
|
||||
|
||||
|
||||
def ninja_path(p):
|
||||
"""A path for a build line: relative to the repo root where possible, forward slashes, Ninja-escaped."""
|
||||
return below_root(p).as_posix().replace('$', '$$').replace(' ', '$ ').replace(':', '$:')
|
||||
@@ -242,6 +271,13 @@ def read_layout(gen):
|
||||
return [(m['name'], int(m['base'], 16), int(m['size'], 16)) for m in data]
|
||||
|
||||
|
||||
def read_plugin_files(gen):
|
||||
"""{plugin<N>: (NSO path, SHA-256)} from layout.json: the mod code modules recomp/layout.py placed after sdk. They
|
||||
are no ExeFS files, so the runtime reads each from its path and checks it against this hash."""
|
||||
data = json.loads((gen / 'layout.json').read_text())
|
||||
return {m['name']: (m['file'], m['sha256']) for m in data if 'file' in m}
|
||||
|
||||
|
||||
def chunk_files(gen, mod):
|
||||
"""The C files of one module: the chunks listed in <mod>/manifest.json, then <mod>_table.c."""
|
||||
d = gen / mod
|
||||
@@ -366,9 +402,11 @@ def runtime_object(out, src):
|
||||
return out / 'obj' / 'runtime' / rel.with_suffix('.o')
|
||||
|
||||
|
||||
def module_glue(layout, linked, hashes, paths, version):
|
||||
def module_glue(layout, linked, hashes, paths, version, plugins=None):
|
||||
"""C for build/<version>/modules.c: the layout table, the linked modules' function tables (dispatch.h), the
|
||||
expected ExeFS hashes, the game version and the default game paths (host.h)."""
|
||||
expected ExeFS hashes, the game version and the default game paths (host.h). plugins: read_plugin_files(); a plugin
|
||||
row of the layout table also names its file and hash."""
|
||||
plugins = plugins or {}
|
||||
out = [f'// Module glue written by tools/build_game.py; do not edit. {GENERATED_MARKER}',
|
||||
'#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
|
||||
for name, base, size in layout:
|
||||
@@ -376,7 +414,9 @@ def module_glue(layout, linked, hashes, paths, version):
|
||||
out.append(f'_Static_assert(MOD_{u}_BASE == {base:#x}ull && MOD_{u}_SIZE == {size:#x}ull, '
|
||||
f'"layout.h and layout.json disagree on {name}");')
|
||||
out += ['', 'const RtLayoutEntry rt_layout[] = {']
|
||||
out += [f' {{"{n}", MOD_{n.upper()}_BASE, MOD_{n.upper()}_SIZE}},' for n, _, _ in layout]
|
||||
out += [f' {{"{n}", MOD_{n.upper()}_BASE, MOD_{n.upper()}_SIZE' +
|
||||
(f', {c_string(plugins[n][0])}, {c_string(plugins[n][1])}' if n in plugins else '') + '},'
|
||||
for n, _, _ in layout]
|
||||
out += ['};', f'const int rt_layout_count = {len(layout)};', '']
|
||||
for name in linked:
|
||||
out += [f'extern const uint64_t {name}_fn_addrs[];', f'extern const GuestFn {name}_fn_ptrs[];',
|
||||
@@ -404,25 +444,53 @@ def port_version():
|
||||
return path.read_text(encoding='utf-8').strip() if path.is_file() else 'unknown'
|
||||
|
||||
|
||||
def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=None, make_lib=False, stamp=None):
|
||||
"""build.ninja text, and the object lists per module. With prebuilt the generated objects already exist in
|
||||
out/obj and are link inputs without a compile edge. runtime_lib: a prebuilt library replaces the runtime objects
|
||||
except INSTALL_SIDE. make_lib: build only that library, from the runtime objects except INSTALL_SIDE. stamp: the
|
||||
install_stamp file, an extra input of the runtime objects and of the link."""
|
||||
def cc_line():
|
||||
"""The cc binding of build.ninja. A compiler cache (sccache) may front clang: every lift rewrites the generated
|
||||
chunks with the same contents, so the cached objects save the compile step on the next build."""
|
||||
cache = f'{cmd_arg(os.environ["SCCACHE"])} ' if os.environ.get('SCCACHE') else ''
|
||||
return f'cc = {cache}{tool_arg(CLANG)}'
|
||||
|
||||
|
||||
def genflags_line(rt, gen):
|
||||
"""The genflags binding of build.ninja for runtime directory rt and generated C directory gen."""
|
||||
return f'genflags = {" ".join(GEN_FLAGS)} -I {cmd_path(rt)} -I {cmd_path(gen)}'
|
||||
|
||||
|
||||
# The compile rule of the generated C; main_objects compares the main build's copy of it with this one.
|
||||
CC_GEN_RULE = ['rule cc_gen',
|
||||
' command = $cc $genflags -MD -MF $out.d -c $in -o $out',
|
||||
' deps = gcc',
|
||||
' depfile = $out.d',
|
||||
' description = CC $in']
|
||||
GEN_EDGE_BINDINGS = [' pool = gen'] # the only per-edge binding of a cc_gen edge
|
||||
|
||||
|
||||
def pgo_dep():
|
||||
"""The implicit input every compile edge carries under --pgo FILE: the profile."""
|
||||
return f' | {ninja_path(PGO_INPUT)}' if PGO_INPUT else ''
|
||||
|
||||
|
||||
def gen_object(mod, f):
|
||||
"""The object of generated file f of module mod, relative to the build directory."""
|
||||
return Path('obj') / mod / (f.stem + '.o')
|
||||
|
||||
|
||||
def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=(), runtime_lib=None, make_lib=False, stamp=None):
|
||||
"""build.ninja text, and the object lists per module. prebuilt: generated objects (gen_object paths) that already
|
||||
exist in out/obj and are link inputs without a compile edge. runtime_lib: a prebuilt library replaces the runtime
|
||||
objects except INSTALL_SIDE. make_lib: build only that library, from the runtime objects except INSTALL_SIDE.
|
||||
stamp: the install_stamp file, an extra input of the runtime objects and of the link."""
|
||||
inc = f'-I {cmd_path(RUNTIME)} -I {cmd_path(gen)}'
|
||||
vma = f'{cmd_path(DEPS / "include")} -I {cmd_path(DEPS / "include" / "vma")}' if DEPS else cmd_path(vma_include())
|
||||
imgui = cmd_path(RUNTIME / 'third_party' / 'imgui')
|
||||
rt_inc = f'{inc} -I {cmd_path(out)} -I {vma} -I {imgui}' # romfs_config.h, vk_mem_alloc.h (+ deps headers), imgui.h
|
||||
libs = [f'-L {cmd_path(DEPS / "lib")}'] + (LLVM_LINK_LIBS if WIN else LINK_LIBS) if DEPS else LINK_LIBS
|
||||
# A compiler cache (sccache) may front clang: every lift rewrites the generated chunks with the same contents, so
|
||||
# the cached objects save the compile step on the next build.
|
||||
cache = f'{cmd_arg(os.environ["SCCACHE"])} ' if os.environ.get('SCCACHE') else ''
|
||||
lines = [
|
||||
'# Written by tools/build_game.py; do not edit.',
|
||||
'ninja_required_version = 1.10',
|
||||
f'builddir = {ninja_path(out)}',
|
||||
f'cc = {cache}{cmd_arg(CLANG.as_posix())}',
|
||||
f'genflags = {" ".join(GEN_FLAGS)} {inc}',
|
||||
cc_line(),
|
||||
genflags_line(RUNTIME, gen),
|
||||
f'rtflags = {" ".join(RT_FLAGS)} {rt_inc}',
|
||||
f'ldflags = {" ".join(LINK_FLAGS)}',
|
||||
f'libs = {" ".join(libs)}',
|
||||
@@ -430,11 +498,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
||||
'pool gen',
|
||||
f' depth = {pool_depth}',
|
||||
'',
|
||||
'rule cc_gen',
|
||||
' command = $cc $genflags -MD -MF $out.d -c $in -o $out',
|
||||
' deps = gcc',
|
||||
' depfile = $out.d',
|
||||
' description = CC $in',
|
||||
*CC_GEN_RULE,
|
||||
'',
|
||||
'rule cc_rt',
|
||||
' command = $cc $rtflags -MD -MF $out.d -c $in -o $out',
|
||||
@@ -443,7 +507,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
||||
' description = CC $in',
|
||||
'',
|
||||
'rule rc',
|
||||
f' command = {cmd_arg(WINDRES.as_posix())} -I {cmd_path(RUNTIME)} -O coff $in -o $out',
|
||||
f' command = {tool_arg(WINDRES)} -I {cmd_path(RUNTIME)} -O coff $in -o $out',
|
||||
' description = RC $in',
|
||||
'',
|
||||
'rule link',
|
||||
@@ -454,14 +518,14 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
||||
'',
|
||||
]
|
||||
objs = {}
|
||||
pgo_dep = f' | {ninja_path(PGO_INPUT)}' if PGO_INPUT else ''
|
||||
pgo = pgo_dep()
|
||||
for mod, files in mods.items():
|
||||
objs[mod] = []
|
||||
for f in files:
|
||||
o = out / 'obj' / mod / (f.stem + '.o')
|
||||
if not prebuilt:
|
||||
lines.append(f'build {ninja_path(o)}: cc_gen {ninja_path(f)}{pgo_dep}')
|
||||
lines.append(' pool = gen')
|
||||
o = out / gen_object(mod, f)
|
||||
if gen_object(mod, f) not in prebuilt:
|
||||
lines.append(f'build {ninja_path(o)}: cc_gen {ninja_path(f)}{pgo}')
|
||||
lines += GEN_EDGE_BINDINGS
|
||||
objs[mod].append(o)
|
||||
rt = []
|
||||
sources = runtime_sources(skip)
|
||||
@@ -477,15 +541,15 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
||||
lines.append(f'build {ninja_path(o)}: rc {ninja_path(src)} | {ninja_path(RUNTIME / "totk.manifest")} '
|
||||
f'{ninja_path(RUNTIME / "totk.ico")}')
|
||||
else:
|
||||
stamp_dep = (' | ' if not pgo_dep else ' ') + ninja_path(stamp) if stamp else ''
|
||||
lines.append(f'build {ninja_path(o)}: cc_rt {ninja_path(src)}{pgo_dep}{stamp_dep}')
|
||||
stamp_dep = (' | ' if not pgo else ' ') + ninja_path(stamp) if stamp else ''
|
||||
lines.append(f'build {ninja_path(o)}: cc_rt {ninja_path(src)}{pgo}{stamp_dep}')
|
||||
rt.append(o)
|
||||
objs['runtime'] = rt
|
||||
every = [o for v in objs.values() for o in v]
|
||||
exe = out / EXE
|
||||
if make_lib:
|
||||
exe = out / LIB_NAME
|
||||
lines += ['', 'rule ar', f' command = {cmd_arg(AR.as_posix())} rcs $out $in', ' description = AR $out', '', f'build {ninja_path(exe)}: ar ' + ' '.join(ninja_path(o) for o in every),
|
||||
lines += ['', 'rule ar', f' command = {tool_arg(AR)} rcs $out $in', ' description = AR $out', '', f'build {ninja_path(exe)}: ar ' + ' '.join(ninja_path(o) for o in every),
|
||||
'build objects: phony ' + ' '.join(ninja_path(o) for o in every), f'default {ninja_path(exe)}', '']
|
||||
return '\n'.join(lines), objs, exe
|
||||
rule = 'link'
|
||||
@@ -502,65 +566,218 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
||||
return '\n'.join(lines), objs, exe
|
||||
|
||||
|
||||
# Environment variables through which clang adds include directories or options that build.ninja does not show.
|
||||
CLANG_ENV = ('CPATH', 'C_INCLUDE_PATH', 'CPLUS_INCLUDE_PATH', 'OBJC_INCLUDE_PATH', 'CCC_OVERRIDE_OPTIONS')
|
||||
# Names a preprocessor probe looks up: a new file of that name in our tree could change the compile.
|
||||
HAS_INCLUDE = re.compile(rb'__has_include(?:_next)?\s*\(\s*[<"]([^>"]+)[>"]')
|
||||
REUSED_LIST = 'reused.txt' # obj/reused.txt: the objects the last build hardlinked or copied from the main build
|
||||
|
||||
|
||||
class FileBasicInfo(ctypes.Structure):
|
||||
_fields_ = [('CreationTime', ctypes.c_longlong), ('LastAccessTime', ctypes.c_longlong),
|
||||
('LastWriteTime', ctypes.c_longlong), ('ChangeTime', ctypes.c_longlong), ('FileAttributes', wt.DWORD)]
|
||||
|
||||
|
||||
def change_time_ns(path):
|
||||
"""The last time path's contents or metadata changed, in ns since 1970, or None when it cannot be read. On Windows
|
||||
the NTFS ChangeTime (also the write and creation times, whichever is latest): a package manager restores a file's
|
||||
old mtime on install, and Windows gives a file created under a just-deleted name the old file's creation time, but
|
||||
either write moves ChangeTime. Elsewhere st_ctime, which a write or a replacement moves too."""
|
||||
if not WIN:
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except OSError:
|
||||
return None
|
||||
return max(st.st_ctime_ns, st.st_mtime_ns)
|
||||
k32 = ctypes.WinDLL('kernel32', use_last_error=True)
|
||||
k32.CreateFileW.restype = wt.HANDLE
|
||||
k32.CreateFileW.argtypes = [wt.LPCWSTR, wt.DWORD, wt.DWORD, ctypes.c_void_p, wt.DWORD, wt.DWORD, wt.HANDLE]
|
||||
k32.GetFileInformationByHandleEx.argtypes = [wt.HANDLE, ctypes.c_int, ctypes.c_void_p, wt.DWORD]
|
||||
k32.CloseHandle.argtypes = [wt.HANDLE]
|
||||
# FILE_READ_ATTRIBUTES, share read/write/delete, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS
|
||||
h = k32.CreateFileW(str(path), 0x80, 7, None, 3, 0x02000000, None)
|
||||
if h is None or h == wt.HANDLE(-1).value:
|
||||
return None
|
||||
info = FileBasicInfo()
|
||||
ok = k32.GetFileInformationByHandleEx(h, 0, ctypes.byref(info), ctypes.sizeof(info)) # FileBasicInfo
|
||||
k32.CloseHandle(h)
|
||||
if not ok:
|
||||
return None
|
||||
return (max(info.ChangeTime, info.LastWriteTime, info.CreationTime) - 116444736000000000) * 100
|
||||
|
||||
|
||||
def compiler_files():
|
||||
"""The clang binary and the LLVM libraries beside it (MSYS2 and llvm-mingw: bin/; Linux: lib/) that hold the
|
||||
compiler: a new compiler replaces one of them."""
|
||||
c = CLANG.resolve()
|
||||
files = [c]
|
||||
for d in (c.parent, c.parent.parent / 'lib'):
|
||||
for pat in ('libclang-cpp*', 'libLLVM*'):
|
||||
files += sorted(p for p in d.glob(pat) if p.is_file())
|
||||
return files
|
||||
|
||||
|
||||
def same_bytes(a, b):
|
||||
"""a and b hold the same bytes."""
|
||||
if a.stat().st_size != b.stat().st_size:
|
||||
return False
|
||||
with open(a, 'rb', buffering=0) as fa, open(b, 'rb', buffering=0) as fb:
|
||||
while True:
|
||||
x, y = fa.read(1 << 20), fb.read(1 << 20)
|
||||
if x != y:
|
||||
return False
|
||||
if not x:
|
||||
return True
|
||||
|
||||
|
||||
def new_names(mine, theirs):
|
||||
"""Lower-case base names of the files below mine that have no counterpart below theirs."""
|
||||
if mine == theirs:
|
||||
return set()
|
||||
return {p.name.lower() for p in mine.rglob('*') if p.is_file() and not (theirs / p.relative_to(mine)).exists()}
|
||||
|
||||
|
||||
def main_objects(gen, main_out, rt_dir, mods):
|
||||
"""({relative object path: the main build's object}, '') when the main build's generated objects can stand in for
|
||||
ours, else (None, why). That holds when the main build.ninja compiles the same sources with the same flags, its
|
||||
objects are up to date, and every header they include from runtime/ is byte-identical in rt_dir."""
|
||||
"""Which of the main build's generated objects can stand in for ours. Returns ({gen_object path: the main build's
|
||||
object}, '', [(gen_object path, why it compiles)]), or (None, why, []) when none can.
|
||||
|
||||
An object is reused only when its compile is provably the one the main build ran: the main build.ninja has the
|
||||
same compiler, flags and rule (with the main runtime/ and generated/<version> in place of ours) and a plain edge
|
||||
for the object; the main generated objects are up to date (a dry run of the main build.ninja); and every file the
|
||||
main compile read (its record in the main Ninja deps log: the .c, the runtime and generated headers, the system
|
||||
headers) has the same bytes on our side and has not changed since the object was written, nor have the compiler's
|
||||
binaries. A file new on our side whose name one of those compiles included or probed with __has_include could
|
||||
change what an include finds, so it blocks the reuse too. Anything unknown compiles."""
|
||||
main_ninja = main_out / 'build.ninja'
|
||||
if not main_ninja.is_file():
|
||||
return None, f'{flag_path(main_ninja)} not found'
|
||||
if gen != (ROOT / 'generated' / main_out.name).resolve():
|
||||
return None, 'generated C directory is not the main one'
|
||||
return None, f'{flag_path(main_ninja)} not found', []
|
||||
env_set = [v for v in CLANG_ENV if os.environ.get(v)]
|
||||
if env_set:
|
||||
return None, f'{env_set[0]} is set', []
|
||||
main_gen = (ROOT / 'generated' / main_out.name).resolve()
|
||||
main_rt = (MAIN_ROOT / 'runtime').resolve()
|
||||
lines = main_ninja.read_text(encoding='utf-8').splitlines()
|
||||
sources = dict(l[6:].split(': cc_gen ') for l in lines if l.startswith('build ') and ': cc_gen ' in l)
|
||||
flags = next((l for l in lines if l.startswith('genflags = ')), '')
|
||||
if not flags.startswith('genflags = ' + ' '.join(GEN_FLAGS) + ' -I '):
|
||||
return None, 'main build uses other compile flags'
|
||||
pairs = {}
|
||||
if cc_line() not in lines or genflags_line(main_rt, main_gen) not in lines:
|
||||
return None, 'main build uses another compiler or other compile flags', []
|
||||
i = lines.index(CC_GEN_RULE[0]) if CC_GEN_RULE[0] in lines else -1
|
||||
if i < 0 or lines[i:i + len(CC_GEN_RULE) + 1] != CC_GEN_RULE + ['']:
|
||||
return None, 'main build has another compile rule', []
|
||||
edges = {} # main object -> (inputs, edge bindings)
|
||||
for j, l in enumerate(lines):
|
||||
if l.startswith('build ') and ': cc_gen ' in l:
|
||||
o, src = l[6:].split(': cc_gen ', 1)
|
||||
k = j + 1
|
||||
while k < len(lines) and lines[k].startswith(' '):
|
||||
k += 1
|
||||
edges[o] = (src, lines[j + 1:k])
|
||||
|
||||
candidates, skipped = {}, []
|
||||
for mod, files in mods.items():
|
||||
for f in files:
|
||||
o = Path('obj') / mod / (f.stem + '.o')
|
||||
o = gen_object(mod, f)
|
||||
theirs = main_out / o
|
||||
if sources.get(ninja_path(theirs)) != ninja_path(f):
|
||||
return None, f'main build has no edge for {f.name}'
|
||||
if not theirs.is_file():
|
||||
return None, f'main object {flag_path(theirs)} is missing'
|
||||
pairs[o] = theirs
|
||||
# Ninja runs in the main checkout: the main build.ninja is relative to it, and the worktree's runtime/ files carry newer
|
||||
# mtimes that would make every object look stale. Header equality is checked byte for byte below.
|
||||
try:
|
||||
main_src = main_gen / f.relative_to(gen)
|
||||
except ValueError:
|
||||
skipped.append((o, f'{f} is outside {flag_path(gen)}'))
|
||||
continue
|
||||
if edges.get(ninja_path(theirs)) != (ninja_path(main_src) + pgo_dep(), GEN_EDGE_BINDINGS):
|
||||
skipped.append((o, f'main build has no plain edge for {flag_path(main_src)}'))
|
||||
elif not theirs.is_file():
|
||||
skipped.append((o, f'main object {flag_path(theirs)} is missing'))
|
||||
else:
|
||||
candidates[o] = (theirs, f, main_src)
|
||||
if not candidates:
|
||||
return {}, '', skipped
|
||||
|
||||
# Ninja runs in the main checkout: the main build.ninja is relative to it, and the worktree's runtime/ files carry
|
||||
# newer mtimes that would make every object look stale. Equality of what we compile is checked byte for byte below.
|
||||
env = dict(os.environ, PATH=str(UCRT) + os.pathsep + os.environ.get('PATH', '')) if WIN else dict(os.environ)
|
||||
targets = [ninja_path(t) for t in pairs.values()]
|
||||
targets = [ninja_path(t) for t, _, _ in candidates.values()]
|
||||
for i in range(0, len(targets), 200):
|
||||
r = subprocess.run([str(NINJA), '-f', str(main_ninja), '-n', '-d', 'explain'] + targets[i:i + 200], cwd=MAIN_ROOT,
|
||||
env=env, capture_output=True, text=True)
|
||||
if r.returncode or 'no work to do' not in r.stdout:
|
||||
return None, f'main generated objects are stale ({flag_path(main_out)} needs a build)'
|
||||
return None, f'main generated objects are stale ({flag_path(main_out)} needs a build)', []
|
||||
r = subprocess.run([str(NINJA), '-f', str(main_ninja), '-t', 'deps'], cwd=MAIN_ROOT, env=env, capture_output=True,
|
||||
text=True)
|
||||
ours = {flag_path(t) for t in pairs.values()}
|
||||
headers, cur = set(), False
|
||||
if r.returncode:
|
||||
return None, f'cannot read the Ninja deps log of {flag_path(main_out)}', []
|
||||
deps, cur = {}, None # main object -> ([files its compile read], record valid)
|
||||
for l in r.stdout.splitlines():
|
||||
if l.startswith(' '):
|
||||
if cur:
|
||||
headers.add(l.strip())
|
||||
if cur is not None and l.strip():
|
||||
deps[cur][0].append(l.strip())
|
||||
elif ': #deps ' in l:
|
||||
cur = l.split(': #deps ')[0]
|
||||
deps[cur] = ([], l.rstrip().endswith('(VALID)'))
|
||||
else:
|
||||
cur = l.split(': #deps')[0] in ours
|
||||
main_rt = MAIN_ROOT / 'runtime'
|
||||
for h in sorted(headers):
|
||||
p = Path(h) if Path(h).is_absolute() else MAIN_ROOT / h
|
||||
try:
|
||||
rel = p.resolve().relative_to(main_rt)
|
||||
except ValueError:
|
||||
cur = None
|
||||
|
||||
compiler = [change_time_ns(p) for p in compiler_files()]
|
||||
compiler_ns = None if None in compiler else max(compiler)
|
||||
added = new_names(gen, main_gen) | new_names(rt_dir, main_rt)
|
||||
seen = {} # dep as the deps log names it -> (problem or '', change time, names it includes or probes)
|
||||
|
||||
def check_dep(h):
|
||||
p = (Path(h) if Path(h).is_absolute() else MAIN_ROOT / h).resolve()
|
||||
if p.is_relative_to(main_gen):
|
||||
mine = gen / p.relative_to(main_gen)
|
||||
elif p.is_relative_to(main_rt):
|
||||
mine = rt_dir / p.relative_to(main_rt)
|
||||
elif p.is_relative_to(MAIN_ROOT) or p.is_relative_to(ROOT):
|
||||
return f'{h} is neither generated C nor runtime/', 0, set()
|
||||
else:
|
||||
mine = p # a system header: both compiles read this file
|
||||
ct = change_time_ns(p)
|
||||
if ct is None:
|
||||
return f'cannot read {h}', 0, set()
|
||||
if not mine.is_file() or (mine != p and not same_bytes(mine, p)):
|
||||
return f'{flag_path(mine)} differs from {h}', ct, set()
|
||||
names = {p.name.lower()}
|
||||
if p.suffix.lower() not in ('.c', '.cpp'):
|
||||
names |= {Path(n.decode(errors='replace')).name.lower() for n in HAS_INCLUDE.findall(p.read_bytes())}
|
||||
return '', ct, names
|
||||
|
||||
pairs = {}
|
||||
for o, (theirs, f, main_src) in candidates.items():
|
||||
rec, valid = deps.get(flag_path(theirs), (None, False))
|
||||
if not rec or not valid:
|
||||
skipped.append((o, 'no valid record in the main Ninja deps log'))
|
||||
continue
|
||||
mine = rt_dir / rel
|
||||
if not mine.is_file() or mine.read_bytes() != p.read_bytes():
|
||||
return None, f'header {rel.as_posix()} differs in {flag_path(rt_dir)}'
|
||||
return pairs, ''
|
||||
written = theirs.stat().st_mtime_ns
|
||||
if compiler_ns is None or compiler_ns >= written:
|
||||
skipped.append((o, 'the compiler changed after the main object was written'))
|
||||
continue
|
||||
if (main_src.resolve().as_posix().lower() not in
|
||||
{(Path(h) if Path(h).is_absolute() else MAIN_ROOT / h).resolve().as_posix().lower() for h in rec[:1]}):
|
||||
skipped.append((o, f'the main deps log names another source than {flag_path(main_src)}'))
|
||||
continue
|
||||
why, names = '', set()
|
||||
for h in rec:
|
||||
if h not in seen:
|
||||
seen[h] = check_dep(h)
|
||||
problem, ct, n = seen[h]
|
||||
if problem:
|
||||
why = problem
|
||||
elif ct >= written:
|
||||
why = f'{h} changed after the main object was written'
|
||||
names |= n
|
||||
if why:
|
||||
break
|
||||
if not why and names & added:
|
||||
why = f'new file {sorted(names & added)[0]} could change what an include finds'
|
||||
if why:
|
||||
skipped.append((o, why))
|
||||
else:
|
||||
pairs[o] = theirs
|
||||
return pairs, '', skipped
|
||||
|
||||
|
||||
def link_objects(out, pairs):
|
||||
"""Hardlink the main build's objects into out. Clang and Ninja replace an object by renaming a new file over it,
|
||||
so a link keeps the old bytes after a main rebuild. Copies where linking fails."""
|
||||
so a link keeps the old bytes after a main rebuild. Copies where linking fails. obj/reused.txt lists them, so a
|
||||
later build that compiles one of them deletes it first (a copy is no hardlink unlink_shared would see)."""
|
||||
for o, theirs in pairs.items():
|
||||
dst = out / o
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
@@ -572,15 +789,24 @@ def link_objects(out, pairs):
|
||||
os.link(theirs, dst)
|
||||
except OSError:
|
||||
shutil.copy2(theirs, dst)
|
||||
write_if_changed(out / 'obj' / REUSED_LIST, ''.join(f'{o.as_posix()}\n' for o in sorted(pairs)))
|
||||
|
||||
|
||||
def unlink_shared(out, mods):
|
||||
"""Drop generated objects still hardlinked to another build, so the next compile writes a new file."""
|
||||
def unlink_shared(out, mods, keep=()):
|
||||
"""Drop the generated objects outside keep that are still hardlinked to another build or were copied from one, so
|
||||
the next compile writes a new file."""
|
||||
listed = out / 'obj' / REUSED_LIST
|
||||
reused = set(listed.read_text(encoding='utf-8').split()) if listed.is_file() else set()
|
||||
for mod, files in mods.items():
|
||||
for f in files:
|
||||
o = out / 'obj' / mod / (f.stem + '.o')
|
||||
if o.exists() and o.stat().st_nlink > 1:
|
||||
o.unlink()
|
||||
o = gen_object(mod, f)
|
||||
if o in keep:
|
||||
continue
|
||||
p = out / o
|
||||
if p.exists() and (p.stat().st_nlink > 1 or o.as_posix() in reused):
|
||||
p.unlink()
|
||||
if not keep:
|
||||
listed.unlink(missing_ok=True)
|
||||
|
||||
|
||||
MANIFEST = 'objects.sha256'
|
||||
@@ -638,14 +864,17 @@ def ninja_build(base, targets, env):
|
||||
sys.exit(code)
|
||||
|
||||
|
||||
def verify_objects(out, objs, borrowed, main_out, pairs, base, env):
|
||||
"""Verify every object before the link; see the module docstring. borrowed: the hardlinked generated objects of
|
||||
main_out (checked against its manifest, never recorded); the others belong to out."""
|
||||
def verify_objects(out, objs, main_out, pairs, base, env):
|
||||
"""Verify every object before the link; see the module docstring. pairs: the generated objects hardlinked from
|
||||
main_out (main_objects; checked against its manifest, never recorded); the others belong to out."""
|
||||
t0 = time.monotonic()
|
||||
owned = [o for mod, v in objs.items() for o in v if not (borrowed and mod != 'runtime')]
|
||||
shared = [o for mod, v in objs.items() for o in v if borrowed and mod != 'runtime']
|
||||
pairs = pairs or {}
|
||||
owned = [o for v in objs.values() for o in v if o.relative_to(out) not in pairs]
|
||||
shared = [o for v in objs.values() for o in v if o.relative_to(out) in pairs]
|
||||
man_path = out / 'obj' / MANIFEST
|
||||
man = read_manifest(man_path)
|
||||
for o in shared: # an entry of an object this build compiled before it was reused
|
||||
man.pop(o.relative_to(out / 'obj').as_posix(), None)
|
||||
verified, bad, _ = check_objects(out, owned, man, True)
|
||||
recompiled = 0
|
||||
if bad:
|
||||
@@ -815,6 +1044,8 @@ def main():
|
||||
ap.add_argument('--runtime', type=Path, help='runtime source directory (default runtime)')
|
||||
ap.add_argument('--full-build', action='store_true',
|
||||
help="compile the generated C even where build/<version>'s objects could be reused")
|
||||
ap.add_argument('--verbose', action='store_true',
|
||||
help="list the generated objects that do not reuse build/<version>'s, and why")
|
||||
ap.add_argument('--skip-runtime', default='',
|
||||
help='comma-separated runtime/*.c names to leave out (the link needs another definition of what they '
|
||||
'define)')
|
||||
@@ -849,8 +1080,14 @@ def main():
|
||||
help='compile the generated code at -O2 without a profile (the --pgo level), for A/B builds')
|
||||
ap.add_argument('--expected', type=Path,
|
||||
help='expected hashes (default tools/data/expected-<display version>.json)')
|
||||
ap.add_argument('--portable', action='store_true',
|
||||
help='paths in build.ninja relative to the tools\' folder, also through .. (an install folder that '
|
||||
'moves with its build kit and build folder)')
|
||||
args = ap.parse_args()
|
||||
set_cpu(args.cpu)
|
||||
if args.portable:
|
||||
global PORTABLE
|
||||
PORTABLE = True
|
||||
|
||||
gen = (args.gen or ROOT / 'generated' / args.version).resolve()
|
||||
out = (args.out or ROOT / 'build' / args.version).resolve()
|
||||
@@ -889,6 +1126,12 @@ def main():
|
||||
CLANG = TOOLCHAIN_BIN / f'clang{suffix}'
|
||||
AR = TOOLCHAIN_BIN / f'llvm-ar{suffix}'
|
||||
WINDRES = TOOLCHAIN_BIN / 'llvm-windres.exe'
|
||||
if PORTABLE and WIN:
|
||||
resource = sorted((TOOLCHAIN_BIN.parent / 'lib' / 'clang').glob('*'))
|
||||
if len(resource) != 1:
|
||||
sys.exit(f'--portable: {TOOLCHAIN_BIN.parent / "lib" / "clang"} holds {len(resource)} version folders')
|
||||
for flags in (GEN_FLAGS, RT_FLAGS):
|
||||
flags += ['-resource-dir', cmd_path(resource[0]), f'--sysroot={cmd_path(TOOLCHAIN_BIN.parent)}']
|
||||
if args.toolchain and WIN or args.deps and (args.toolchain or not WIN): # Linux: --deps only, never deps-mingw
|
||||
DEPS = (args.deps or DEFAULT_DEPS).resolve()
|
||||
dlls = sorted((DEPS / 'bin').glob('*.dll')) if WIN else []
|
||||
@@ -923,10 +1166,11 @@ def main():
|
||||
cpus = os.cpu_count() or 1
|
||||
budget = min(MEM_BUDGET, int(total_ram() * 0.8))
|
||||
jobs = args.jobs or max(1, min(cpus, budget // JOB_MEM))
|
||||
missing = [n for n in names if n not in hashes]
|
||||
plugins = read_plugin_files(gen)
|
||||
missing = [n for n in names if n not in hashes and n not in plugins]
|
||||
if missing:
|
||||
sys.exit(f'{expected}: no exefs_sha256 for {missing}')
|
||||
write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths, args.version))
|
||||
write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths, args.version, plugins))
|
||||
if has_romfs:
|
||||
count, size, dt = romfs_index.write(paths['romfs'], out)
|
||||
print(f'romfs index: {count} files, {mib(size)}, {dt:.1f} s')
|
||||
@@ -941,7 +1185,7 @@ def main():
|
||||
sys.exit(f'{stale[0]}: its SPIR-V header no longer matches; run python tools/build_shaders.py')
|
||||
skip = {s for s in args.skip_runtime.split(',') if s}
|
||||
main_out = (ROOT / 'build' / args.version).resolve()
|
||||
pairs = None
|
||||
pairs, skipped = None, []
|
||||
if args.make_runtime_lib:
|
||||
mods, linked, why = {}, [], ''
|
||||
elif args.full_build:
|
||||
@@ -951,15 +1195,22 @@ def main():
|
||||
elif out == main_out:
|
||||
why = ''
|
||||
else:
|
||||
pairs, why = main_objects(gen, main_out, RUNTIME, mods)
|
||||
t0 = time.monotonic()
|
||||
pairs, why, skipped = main_objects(gen, main_out, RUNTIME, mods)
|
||||
dt = time.monotonic() - t0
|
||||
if out != main_out: # forks hold hardlinks to the main build's objects; the main build keeps its own
|
||||
unlink_shared(out, mods, pairs or {})
|
||||
if pairs is not None:
|
||||
link_objects(out, pairs)
|
||||
print(f'build_game: reusing {len(pairs)} generated objects of {flag_path(main_out)} (hardlinks)')
|
||||
else:
|
||||
if out != main_out: # forks hold hardlinks to the main build's objects; the main build keeps its own
|
||||
unlink_shared(out, mods)
|
||||
if why:
|
||||
print(f'build_game: compiling all generated C: {why}')
|
||||
total = sum(len(v) for v in mods.values())
|
||||
print(f'build_game: reused {len(pairs)} of {total} generated objects of {flag_path(main_out)} (hardlinks), '
|
||||
f'compiling {total - len(pairs)} ({dt:.1f} s to check)' +
|
||||
(' (--verbose lists them)' if skipped and not args.verbose else ''))
|
||||
if args.verbose:
|
||||
for o, reason in skipped:
|
||||
print(f' compiles {o.as_posix()}: {reason}')
|
||||
elif why:
|
||||
print(f'build_game: compiling all generated C: {why}')
|
||||
runtime_lib = args.runtime_lib.resolve() if args.runtime_lib else None
|
||||
if runtime_lib and not runtime_lib.is_file():
|
||||
sys.exit(f'{runtime_lib} not found: build it with --make-runtime-lib')
|
||||
@@ -969,8 +1220,7 @@ def main():
|
||||
sys.exit(f'build_game: {", ".join(missing)} missing from {flag_path(RUNTIME)}: the release must ship every '
|
||||
f'INSTALL_SIDE source beside {runtime_lib.name}')
|
||||
stamp = install_stamp(out, runtime_lib) if runtime_lib else None
|
||||
text, objs, exe = ninja_file(gen, out, mods, jobs, skip, pairs is not None, runtime_lib,
|
||||
args.make_runtime_lib, stamp)
|
||||
text, objs, exe = ninja_file(gen, out, mods, jobs, skip, pairs or {}, runtime_lib, args.make_runtime_lib, stamp)
|
||||
write_if_changed(out / 'build.ninja', text)
|
||||
|
||||
env = dict(os.environ)
|
||||
@@ -987,7 +1237,7 @@ def main():
|
||||
for mod, files in objs.items():
|
||||
size = sum(o.stat().st_size for o in files)
|
||||
print(f' {mod:8} {len(files):5} objects {mib(size):>14}')
|
||||
verify_objects(out, objs, pairs is not None, main_out, pairs, base, env)
|
||||
verify_objects(out, objs, main_out, pairs, base, env)
|
||||
if args.cpu == 'v2':
|
||||
# Inline assembly, target attributes and intrinsics pass -march=x86-64-v2 by; one vmovdqa of a YMM register
|
||||
# stops a CPU without AVX (tools/check_isa.py)
|
||||
|
||||
@@ -0,0 +1,667 @@
|
||||
"""Run a mod plugin's init in the plugin service run and check what it did: the build step before the generator emits
|
||||
a plugin's hook sites. After the design in #845 by itaskaev2; the init runs on this port's runtime (totk.exe
|
||||
--plugin-init, runtime/plugin_init.c) instead of under Unicorn, and it runs again at every start of the game in the
|
||||
same environment (plugin_init.c plugins_start), so the plugin keeps real state (its heap blocks, its GOT, its alias
|
||||
addresses). The record says what the build expects that start-up to write.
|
||||
|
||||
python tools/exefs_plugins.py --exe build/<ver>-x/totk.exe --out <dir> # per plugin: two runs, diff, check
|
||||
python tools/exefs_plugins.py --check <dir>/<plugin>/run1 # the check of one record
|
||||
python tools/exefs_plugins.py --exe <exe> --out <dir> --mods <mods folder> --patches <patch file>
|
||||
--patches-out <patch file> --status <file> # and the patch lines for the generator
|
||||
|
||||
With --mods, --patches, --patches-out and --status, an accepted plugin becomes one `ordered` patch set mod_<hash>
|
||||
(recomp/patches.py: a plugin line, its expected writes, stubs, seeds, orig lines and its hook sites) appended to the
|
||||
--patches file into --patches-out, and every plugin gets a line of --status (runtime/mods.c reads it as
|
||||
build-cache/plugins.status), in the format of tools/exefs_patches.py --status:
|
||||
|
||||
<file hash> <content hash> <kind> <built-in sets> <note>
|
||||
|
||||
file hash: FNV-1a 32 of "<mod folder name>/<path of the NSO in the mod>"; content hash: the same over its bytes; kind
|
||||
`plugin` (note: the module and its hook count) or `unsupported` (note: the first reason); built-in sets: the port's
|
||||
own sets with a site the plugin hooks too ("-" none), which the runtime switches off while the mod is on.
|
||||
|
||||
The exe must be built with the plugins in its layout (recomp/gen.py --plugins). Each plugin runs in service runs of
|
||||
its own (TOTK_PLUGIN_ONLY): the other plugins stay loaded, their inits do not run. A plugin that is refused, crashes,
|
||||
hangs or fails the check is refused alone, with its reason; the others are built in. At every start the game runs each
|
||||
init against the game's own bytes as well (runtime/plugin_init.c plugins_start takes a plugin's writes back before the
|
||||
next init), so a plugin's own record is what its start must match. --out gets list/ (a run without inits, which names
|
||||
the plugin modules), and <plugin>/run1 and <plugin>/run2 for each plugin; run2 adds a large offset to every
|
||||
GetSystemTick (TOTK_PLUGIN_TICK_OFFSET), so the init picks other alias addresses. The words that differ between the two
|
||||
records are listed with what they look like, not refused: an init keeps its alias addresses and the regions GetInfo
|
||||
gave it in its .data, and a plugin's callbacks must not use them. A difference in code (a write into a game module,
|
||||
the plugin's .text or .rodata) refuses the plugin: its start could not be compared with the record.
|
||||
|
||||
The check (check_record) accepts a record only when every rule holds; each broken rule is one reason:
|
||||
- the run was not refused (an svc outside the served set, such as CreateSession of exlaunch's process-handle
|
||||
fallback, ends the run with the reason);
|
||||
- the init called only the svcs ALLOWED_SVCS, GetInfo with ALLOWED_INFO, and the imports ALLOWED_IMPORTS;
|
||||
- every write into a game module is a word of main's .text whose original is no br, blr or ret, and it is one of:
|
||||
- an inline hook (MSFix, Full Inventory Expansion): `b <stub>` into a plugin's .text, the stub in exlaunch's
|
||||
shape (stur x30, [sp, #-0x10]; bl exl_inline_hook_impl or the float one; ldur x30, [sp, #-0x10];
|
||||
b <trampoline>; .quad <callback in the plugin's .text>), the trampoline the site's original relocated (the
|
||||
same word, or the same PC-relative target) and `b <site + 4>`;
|
||||
- a call-site rewrite (Quick Weapon Durability): `bl` straight to a function start of the plugin's lift (the
|
||||
record lists them), which the generator can emit as a patch site whose replacement is that branch;
|
||||
- a replace hook (Full Inventory Expansion's Sort Fixer): `b` from a function start of main to a function start of
|
||||
the plugin's lift; the Orig trampoline (the original relocated, b site+4) is listed when the pool holds one.
|
||||
- the code the hooks reach (the stubs, exlaunch's hook impls, the callbacks, the replace and call targets, and what
|
||||
they reach by direct branches and calls, reach()) has no svc but OutputDebugString and calls only the imports
|
||||
REACH_IMPORTS. An indirect call (blr) is not followed: what it reaches cannot be known before the run.
|
||||
A site shared with a built-in patch set turns that set off while the mod is on (the status line names it); a site
|
||||
shared with another mod goes through the run time's pick (the Mods tab order, runtime/patches.c).
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'recomp'))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from cheats import fnv # noqa: E402
|
||||
from exefs_patches import EXEFS_PLACES, builtin_sites # noqa: E402
|
||||
from nso import R_AARCH64_JUMP_SLOT, Nso # noqa: E402
|
||||
|
||||
ALLOWED_SVCS = {0x06: 'QueryMemory', 0x1E: 'GetSystemTick', 0x27: 'OutputDebugString', 0x29: 'GetInfo',
|
||||
0x74: 'MapProcessMemory', 0x75: 'UnmapProcessMemory'}
|
||||
ALLOWED_INFO = {2, 3, 4, 5, 12, 13, 14, 15, 65001}
|
||||
# Reviewed one by one: SNPrintf formats into the caller's buffer; GetDisplayVersion is answered by the runtime with
|
||||
# the build's game version; mersenne_twister_engine<uint64_t, ...>::_M_gen_rand (Full Inventory Expansion's alias
|
||||
# picker) refills the state of the engine object it is given, which lives in the plugin's .bss; memcmp and strlen only
|
||||
# read; memcpy, memmove and memset write the buffers they are given; malloc and free use the game's heap (sdk's musl
|
||||
# allocator), and since the init runs again at every start, a block it keeps exists at play time (Quick Weapon
|
||||
# Durability's init keeps one in its .bss).
|
||||
SAFE_LIBC = {'malloc', 'free', 'memcpy', 'memmove', 'memset', 'memcmp', 'strlen'}
|
||||
MT_GEN_RAND = ('_ZNSt23mersenne_twister_engineImLm64ELm312ELm156ELm31ELm13043109905998158313ELm29ELm6148914691236517205'
|
||||
'ELm17ELm8202884508482404352ELm37ELm18444473444759240704ELm43ELm6364136223846793005EE11_M_gen_randEv')
|
||||
SNPRINTF = '_ZN2nn4util8SNPrintfEPcmPKcz'
|
||||
ALLOWED_IMPORTS = SAFE_LIBC | {SNPRINTF, '_ZN2nn2oe17GetDisplayVersionEPNS0_14DisplayVersionE', MT_GEN_RAND}
|
||||
# The imports the code the hooks reach may call while the game runs (reach()). Each reviewed for what it touches: the
|
||||
# libc ones above (the caller's buffers and the game's heap), SNPrintf (the caller's buffer). Nothing that waits,
|
||||
# talks to a service or changes memory mappings.
|
||||
REACH_IMPORTS = SAFE_LIBC | {SNPRINTF}
|
||||
SVC_OUTPUT_DEBUG_STRING = 0x27
|
||||
HOOK_IMPLS = ('exl_inline_hook_impl', 'exl_inline_float_hook_impl')
|
||||
STUR_X30 = 0xF81F03FE # stur x30, [sp, #-0x10]
|
||||
LDUR_X30 = 0xF85F03FE # ldur x30, [sp, #-0x10]
|
||||
STUB_WORDS = 6 # 4 instructions and the callback's .quad
|
||||
TICK_OFFSET = 0x0123456789AB
|
||||
|
||||
|
||||
def u(v):
|
||||
return int(v, 16) if isinstance(v, str) else v
|
||||
|
||||
|
||||
def ran_view(rec):
|
||||
"""The record as seen from the plugins whose inits ran: a plugin that only stayed loaded (TOTK_PLUGIN_ONLY) counts
|
||||
as a module that is no plugin, so its exports and code are not checked as this record's. A record without "ran"
|
||||
(every init ran) comes back as it is."""
|
||||
if all(m.get('ran', True) for m in rec['modules'] if m['plugin']):
|
||||
return rec
|
||||
view = dict(rec)
|
||||
view['modules'] = [dict(m, plugin=False) if m['plugin'] and not m.get('ran', True) else m for m in rec['modules']]
|
||||
names = {m['name'] for m in view['modules'] if m['plugin']}
|
||||
view['images'] = {k: v for k, v in rec['images'].items() if k in names}
|
||||
return view
|
||||
|
||||
|
||||
# ---- instructions
|
||||
def sext(v, bits):
|
||||
return v - (1 << bits) if v >> (bits - 1) & 1 else v
|
||||
|
||||
|
||||
def branch_target(w, pc):
|
||||
"""Target of b / bl at pc, else None."""
|
||||
if w & 0x7C000000 == 0x14000000:
|
||||
return pc + sext(w & 0x3FFFFFF, 26) * 4
|
||||
return None
|
||||
|
||||
|
||||
def pc_relative(w, pc):
|
||||
"""(kind, absolute value) of a PC-relative instruction at pc, else None: b, bl, b.cond, cbz/cbnz, tbz/tbnz, adr,
|
||||
adrp, ldr (literal). The kind holds the bits besides the offset, so two encodings match when both agree."""
|
||||
if w & 0x7C000000 == 0x14000000:
|
||||
return ('b', w & 0xFC000000), pc + sext(w & 0x3FFFFFF, 26) * 4
|
||||
if w & 0xFF000010 == 0x54000000:
|
||||
return ('b.cond', w & 0xFF00001F), pc + sext(w >> 5 & 0x7FFFF, 19) * 4
|
||||
if w & 0x7E000000 == 0x34000000:
|
||||
return ('cbz', w & 0xFF00001F), pc + sext(w >> 5 & 0x7FFFF, 19) * 4
|
||||
if w & 0x7E000000 == 0x36000000:
|
||||
return ('tbz', w & 0xFFF8001F), pc + sext(w >> 5 & 0x3FFF, 14) * 4
|
||||
if w & 0x9F000000 == 0x10000000:
|
||||
return ('adr', w & 0x1F), pc + sext((w >> 5 & 0x7FFFF) << 2 | w >> 29 & 3, 21)
|
||||
if w & 0x9F000000 == 0x90000000:
|
||||
return ('adrp', w & 0x1F), (pc & ~0xFFF) + (sext((w >> 5 & 0x7FFFF) << 2 | w >> 29 & 3, 21) << 12)
|
||||
if w & 0x3B000000 == 0x18000000:
|
||||
return ('ldr', w & 0xFF00001F), pc + sext(w >> 5 & 0x7FFFF, 19) * 4
|
||||
return None
|
||||
|
||||
|
||||
def is_indirect(w):
|
||||
"""br, blr, ret (and their pointer-authenticated forms): a site exlaunch could hook but the generator cannot."""
|
||||
return w & 0xFE1F0000 == 0xD61F0000 and w >> 21 & 0xF in (0, 1, 2)
|
||||
|
||||
|
||||
def relocated(orig, site, word, at):
|
||||
"""The trampoline word at `at` does what the original at `site` does."""
|
||||
a, b = pc_relative(orig, site), pc_relative(word, at)
|
||||
if a is None:
|
||||
return word == orig
|
||||
return b is not None and a == b
|
||||
|
||||
|
||||
def inline_stub(img, o, p):
|
||||
"""The words at image offset o of plugin p have exlaunch's inline-hook entry shape: stur x30; bl; ldur x30; b;
|
||||
and the .quad after them lies inside the plugin's .text."""
|
||||
if o + 4 * STUB_WORDS > p['text_end']:
|
||||
return False
|
||||
s = struct.unpack_from('<4I', img, o)
|
||||
return s[0] == STUR_X30 and s[2] == LDUR_X30 and s[1] >> 26 == 0x25 and s[3] >> 26 == 0x05
|
||||
|
||||
|
||||
def orig_trampoline(img, p, original, site):
|
||||
"""Image offset of a pair in the plugin's .text that runs the site's original instruction relocated, then b site+4
|
||||
(exlaunch's Orig trampoline of a replace hook), else None."""
|
||||
base = p['base']
|
||||
for o in range(0, p['text_end'] - 4, 4):
|
||||
t1 = struct.unpack_from('<I', img, o + 4)[0]
|
||||
if t1 >> 26 == 0x05 and branch_target(t1, base + o + 4) == site + 4 and \
|
||||
relocated(original, site, struct.unpack_from('<I', img, o)[0], base + o):
|
||||
return o
|
||||
return None
|
||||
|
||||
|
||||
# ---- records
|
||||
def load_record(d):
|
||||
"""record.json of a service run directory, with the images of its plugins as bytes (key 'images')."""
|
||||
d = Path(d)
|
||||
rec = json.loads((d / 'record.json').read_text(encoding='utf-8'))
|
||||
rec['images'] = {}
|
||||
for m in rec['modules']:
|
||||
for k in ('base', 'size', 'text_end', 'data_off', 'init'):
|
||||
m[k] = u(m[k])
|
||||
if m.get('image'):
|
||||
rec['images'][m['name']] = (d / m['image']).read_bytes()
|
||||
for w in rec['writes']:
|
||||
for k in ('offset', 'old', 'new'):
|
||||
w[k] = u(w[k])
|
||||
return rec
|
||||
|
||||
|
||||
def hook_impls(rec):
|
||||
"""{absolute address: name} of exlaunch's hook entry points in each plugin, from the plugin NSO's exports."""
|
||||
out = {}
|
||||
for m in rec['modules']:
|
||||
if m['plugin'] and m.get('file') and Path(m['file']).is_file():
|
||||
for name, _, shndx, value, _ in Nso(m['file']).exports():
|
||||
if name in HOOK_IMPLS:
|
||||
out[m['base'] + value] = name
|
||||
return out
|
||||
|
||||
|
||||
def check_record(rec, impls):
|
||||
"""(hooks, reasons): one dict per hook site, and one line per broken rule (no lines: the record is accepted)."""
|
||||
reasons, hooks = [], []
|
||||
if rec.get('refused'):
|
||||
reasons.append(f'the run was refused: {rec["refused"]}')
|
||||
for imm in rec.get('svcs', {}):
|
||||
if u(imm) not in ALLOWED_SVCS:
|
||||
reasons.append(f'the init called svc {u(imm):#x}')
|
||||
for i in rec.get('get_info', {}):
|
||||
if int(i) not in ALLOWED_INFO:
|
||||
reasons.append(f'the init asked GetInfo id {i}')
|
||||
for name, calls in rec.get('imports', {}).items():
|
||||
if calls and name not in ALLOWED_IMPORTS:
|
||||
reasons.append(f'the init called the import {name}')
|
||||
mods = {m['name']: m for m in rec['modules']}
|
||||
plugins = [m for m in rec['modules'] if m['plugin']]
|
||||
|
||||
def plugin_word(a):
|
||||
"""(plugin, word) at absolute a inside a plugin's .text image, else (None, None)."""
|
||||
for p in plugins:
|
||||
if p['base'] <= a < p['base'] + p['text_end'] and p['name'] in rec['images']:
|
||||
return p, struct.unpack_from('<I', rec['images'][p['name']], a - p['base'])[0]
|
||||
return None, None
|
||||
|
||||
main = mods.get('main')
|
||||
for w in rec['writes']:
|
||||
where = f'{w["module"]}+{w["offset"]:#x}'
|
||||
if w['module'] != 'main' or not main or w['offset'] >= main['text_end'] or w['offset'] & 3:
|
||||
reasons.append(f'a write outside main\'s .text: {where} {w["old"]:#010x} -> {w["new"]:#010x}')
|
||||
continue
|
||||
site = main['base'] + w['offset']
|
||||
stub = branch_target(w['new'], site)
|
||||
link = bool(w['new'] >> 31)
|
||||
if stub is None:
|
||||
reasons.append(f'{where}: the new word {w["new"]:#010x} is no b or bl')
|
||||
continue
|
||||
if is_indirect(w['old']):
|
||||
reasons.append(f'{where}: the original {w["old"]:#010x} is a br, blr or ret')
|
||||
p, _ = plugin_word(stub)
|
||||
if p is None:
|
||||
reasons.append(f"{where}: {'bl' if link else 'b'} {stub:#x} does not reach into a plugin's .text")
|
||||
continue
|
||||
if not (not link and inline_stub(rec['images'][p['name']], stub - p['base'], p)):
|
||||
# A b or bl straight into the plugin's code (CodePatcher-style call-site rewrite): a site whose replacement
|
||||
# is that branch, when the target is a function start of the plugin's lift.
|
||||
# A b replaces the whole main function, so it must sit at a function start of main; exlaunch's replace
|
||||
# hook keeps the function's first instruction relocated + b site+4 in its pool for the callback's Orig.
|
||||
o = stub - p['base']
|
||||
hook = {'site': f'main+{w["offset"]:#x}', 'original': f'{w["old"]:#010x}',
|
||||
'kind': 'call' if link else 'replace', 'target': f'{p["name"]}+{o:#x}'}
|
||||
orig = orig_trampoline(rec['images'][p['name']], p, w['old'], site)
|
||||
if orig is not None:
|
||||
hook['trampoline'] = f'{p["name"]}+{orig:#x}'
|
||||
hooks.append(hook)
|
||||
if o not in {u(x) for x in p.get('functions', [])}:
|
||||
reasons.append(f"{where}: {'bl' if link else 'b'} {p['name']}+{o:#x} is neither an inline-hook stub "
|
||||
f"nor a function start of {p['name']}'s lift")
|
||||
if not link and not w.get('fn_start'):
|
||||
reasons.append(f"{where}: b {p['name']}+{o:#x} from inside a main function (only a function start can "
|
||||
f"be replaced)")
|
||||
continue
|
||||
img, o = rec['images'][p['name']], stub - p['base']
|
||||
s = struct.unpack_from('<4IQ', img, o)
|
||||
impl, tramp, cb = branch_target(s[1], stub + 4), branch_target(s[3], stub + 12), s[4]
|
||||
hook = {'site': f'main+{w["offset"]:#x}', 'original': f'{w["old"]:#010x}', 'kind': 'inline',
|
||||
'stub': f'{p["name"]}+{o:#x}',
|
||||
'impl': impls.get(impl, f'{impl:#x}' if impl is not None else None),
|
||||
'callback': f'{p["name"]}+{cb - p["base"]:#x}' if p['base'] <= cb < p['base'] + p['text_end'] else f'{cb:#x}',
|
||||
'trampoline': f'{p["name"]}+{tramp - p["base"]:#x}' if tramp is not None else None}
|
||||
hooks.append(hook)
|
||||
if impl not in impls:
|
||||
reasons.append(f'{where}: the stub calls {impl:#x}, not {" or ".join(HOOK_IMPLS)}')
|
||||
if not p['base'] <= cb < p['base'] + p['text_end']:
|
||||
reasons.append(f'{where}: the stub\'s callback {cb:#x} is outside {p["name"]}\'s .text')
|
||||
tp, t0 = plugin_word(tramp)
|
||||
_, t1 = plugin_word(tramp + 4)
|
||||
if tp is None or t1 is None:
|
||||
reasons.append(f'{where}: the trampoline {tramp:#x} is outside a plugin\'s .text')
|
||||
continue
|
||||
hook['relocated'] = f'{t0:#010x}'
|
||||
if not relocated(w['old'], site, t0, tramp):
|
||||
reasons.append(f'{where}: the trampoline {hook["trampoline"]} starts with {t0:#010x}, not the original '
|
||||
f'{w["old"]:#010x} relocated')
|
||||
if branch_target(t1, tramp + 4) != site + 4 or t1 >> 31:
|
||||
reasons.append(f'{where}: the trampoline\'s second word {t1:#010x} is no b to the site + 4')
|
||||
return hooks, reasons
|
||||
|
||||
|
||||
# ---- the code the hooks reach
|
||||
def plt_imports(nso, img, base, text_end):
|
||||
"""{absolute address of a PLT stub: the import its GOT slot binds} (adrp x16; ldr x17, [x16, #slot]; add x16, x16,
|
||||
#slot; br x17), from the JUMP_SLOT relocations."""
|
||||
slots = {off: nso.syms[sym][0] for off, typ, sym, _ in nso.relocs if typ == R_AARCH64_JUMP_SLOT}
|
||||
out = {}
|
||||
for o in range(0, text_end - 16, 4):
|
||||
w0, w1, _, w3 = struct.unpack_from('<4I', img, o)
|
||||
if w0 & 0x9F00001F != 0x90000010 or w1 & 0xFFC003FF != 0xF9400211 or w3 != 0xD61F0220:
|
||||
continue
|
||||
imm = sext((w0 >> 29 & 3) | (w0 >> 5 & 0x7FFFF) << 2, 21)
|
||||
slot = ((o & ~0xFFF) + (imm << 12)) + (w1 >> 10 & 0xFFF) * 8
|
||||
if slot in slots:
|
||||
out[base + o] = slots[slot]
|
||||
return out
|
||||
|
||||
|
||||
def reach(img, base, text_end, roots, plt, stubs=()):
|
||||
"""What the code from roots (absolute addresses in the plugin's .text) reaches by falling through, direct branches
|
||||
and direct calls: (svc immediates, {import: calls}, indirect calls, words outside the .text it branched to). A
|
||||
stub's b to its trampoline counts as its return (recomp/patches.py `stub`)."""
|
||||
svcs, imports, indirect, outside = set(), {}, 0, set()
|
||||
seen, work = set(), list(roots)
|
||||
stub_rets = {s + 12 for s in stubs}
|
||||
while work:
|
||||
a = work.pop()
|
||||
while True:
|
||||
if a in seen:
|
||||
break
|
||||
if a in plt:
|
||||
imports[plt[a]] = imports.get(plt[a], 0) + 1
|
||||
break
|
||||
if not base <= a < base + text_end:
|
||||
outside.add(a)
|
||||
break
|
||||
seen.add(a)
|
||||
w = struct.unpack_from('<I', img, a - base)[0]
|
||||
if a in stub_rets:
|
||||
break
|
||||
if w & 0xFFE0001F == 0xD4000001: # svc #imm
|
||||
svcs.add(w >> 5 & 0xFFFF)
|
||||
if w & 0x7C000000 == 0x14000000: # b, bl
|
||||
t = branch_target(w, a)
|
||||
if w >> 31:
|
||||
work.append(t)
|
||||
else:
|
||||
a = t
|
||||
continue
|
||||
elif pc_relative(w, a) and pc_relative(w, a)[0][0] in ('b.cond', 'cbz', 'tbz'):
|
||||
work.append(pc_relative(w, a)[1])
|
||||
elif w & 0xFFFFFC1F == 0xD63F0000: # blr
|
||||
indirect += 1
|
||||
elif is_indirect(w): # br, ret: the path ends
|
||||
break
|
||||
elif w == 0 or w & 0xFFE0001F == 0xD4200000: # udf, brk: ends too
|
||||
break
|
||||
a += 4
|
||||
return svcs, imports, indirect, outside
|
||||
|
||||
|
||||
def reach_check(rec, hooks, impls):
|
||||
"""Reasons the code the hooks reach breaks the rules (svcs, imports), and a summary line."""
|
||||
reasons, notes = [], []
|
||||
for p in (m for m in rec['modules'] if m['plugin'] and m['name'] in rec['images']):
|
||||
img, base = rec['images'][p['name']], p['base']
|
||||
roots, stubs = set(), set()
|
||||
for h in hooks:
|
||||
for k in ('stub', 'callback', 'target'):
|
||||
if h.get(k, '').startswith(p['name'] + '+'):
|
||||
roots.add(base + int(h[k].split('+')[1], 16))
|
||||
if h['kind'] == 'inline' and h['stub'].startswith(p['name'] + '+'):
|
||||
stubs.add(base + int(h['stub'].split('+')[1], 16))
|
||||
roots |= {a for a in impls if base <= a < base + p['text_end']}
|
||||
nso = Nso(p['file']) if p.get('file') and Path(p['file']).is_file() else None
|
||||
plt = plt_imports(nso, img, base, p['text_end']) if nso else {}
|
||||
svcs, imports, indirect, outside = reach(img, base, p['text_end'], sorted(roots), plt, stubs)
|
||||
for imm in sorted(svcs - {SVC_OUTPUT_DEBUG_STRING}):
|
||||
reasons.append(f'the code its hooks reach has svc {imm:#x}')
|
||||
for name in sorted(set(imports) - REACH_IMPORTS):
|
||||
reasons.append(f'the code its hooks reach calls the import {name}')
|
||||
notes.append(f'{p["name"]}: hooks reach {len(roots)} roots, imports {", ".join(sorted(imports)) or "none"}, '
|
||||
f'{indirect} indirect calls not followed' + (f', {len(outside)} branches out of its .text'
|
||||
if outside else ''))
|
||||
return reasons, notes
|
||||
|
||||
|
||||
# ---- patch lines and status for the generator and the runtime
|
||||
def mod_of(nso_path, mods):
|
||||
"""(mod folder name, path of the NSO in the mod) of a plugin file under the mods folder, else None."""
|
||||
try:
|
||||
rel = Path(nso_path).resolve().relative_to(Path(mods).resolve())
|
||||
except ValueError:
|
||||
return None
|
||||
return rel.parts[0], '/'.join(rel.parts[1:])
|
||||
|
||||
|
||||
def hooks_of(hooks, module):
|
||||
"""The hooks whose stub or target lies in the plugin module: its own, when one record holds several plugins."""
|
||||
return [h for h in hooks if (h.get('stub') or h.get('target') or '').split('+')[0] == module]
|
||||
|
||||
|
||||
def plugin_lines(rec, hooks, name, record_dir, module):
|
||||
"""The patch lines of the accepted plugin module (recomp/patches.py) for set name: the plugin line, the expected
|
||||
writes (the words at its hooks' sites), the stubs, the seeds, the orig lines and the hook sites."""
|
||||
lines = []
|
||||
mods = {m['name']: m for m in rec['modules']}
|
||||
p = mods[module]
|
||||
img = rec['images'][p['name']]
|
||||
text_sha = hashlib.sha256(img[:p['text_end']]).hexdigest()
|
||||
lines.append(f'plugin {name} {p["name"]} {text_sha} {(Path(record_dir) / p["image"]).resolve().as_posix()}')
|
||||
own = {int(h['site'].split('+')[1], 16) for h in hooks}
|
||||
for w in rec['writes']:
|
||||
if w['module'] == 'main' and w['offset'] in own:
|
||||
lines.append(f'expect {name} {w["module"]} {w["offset"]:x} {w["old"]:08x} {w["new"]:08x}')
|
||||
seeds, stubs = {}, {}
|
||||
|
||||
def at(ref):
|
||||
mod, off = ref.split('+')
|
||||
return mod, int(off, 16)
|
||||
|
||||
for h in hooks:
|
||||
site = int(h['site'].split('+')[1], 16)
|
||||
if h['kind'] == 'inline':
|
||||
sm, so = at(h['stub'])
|
||||
stubs.setdefault(sm, set()).add(so)
|
||||
cm, co = at(h['callback'])
|
||||
seeds.setdefault(cm, set()).add(co)
|
||||
# the hook impl the stub calls: its bl, as check_record read it
|
||||
stub_at = mods[sm]['base'] + so
|
||||
impl = branch_target(struct.unpack_from('<I', rec['images'][sm], so + 4)[0], stub_at + 4)
|
||||
seeds.setdefault(sm, set()).add(impl - mods[sm]['base'])
|
||||
lines.append(f'site {name} main {site:x} {u(h["original"]):08x} @call:{sm}+{so:x} {u(h["original"]):08x}')
|
||||
else:
|
||||
tm, to = at(h['target'])
|
||||
seeds.setdefault(tm, set()).add(to)
|
||||
new = next(w['new'] for w in rec['writes'] if w['module'] == 'main' and w['offset'] == site)
|
||||
lines.append(f'site {name} main {site:x} {u(h["original"]):08x} {new:08x}')
|
||||
if h['kind'] == 'replace' and h.get('trampoline'):
|
||||
rm, ro = at(h['trampoline'])
|
||||
lines.append(f'orig {name} {rm} {ro:x} main {site:x}')
|
||||
for m in sorted(stubs):
|
||||
lines += [f'stub {m} {o:x}' for o in sorted(stubs[m])]
|
||||
for m in sorted(seeds):
|
||||
lines.append(f'seed {m} ' + ' '.join(f'{o:x}' for o in sorted(seeds[m])))
|
||||
return lines
|
||||
|
||||
|
||||
def plugin_files(mods):
|
||||
"""[(mod folder name, path in the mod, file)] of the exlaunch plugins of the mods folder, by mod name: the subsdk*
|
||||
files of each mod's first ExeFS place (tools/exefs_patches.py EXEFS_PLACES, runtime/mods.c find_root). A replaced
|
||||
main is no plugin."""
|
||||
out = []
|
||||
for mod in sorted((d for d in Path(mods).iterdir() if d.is_dir()), key=lambda d: d.name) if Path(mods).is_dir() else []:
|
||||
exefs = next((mod / w for w in EXEFS_PLACES if (mod / w).is_dir()), None)
|
||||
if exefs:
|
||||
for f in sorted(exefs.iterdir(), key=lambda f: f.name.lower()):
|
||||
if f.is_file() and f.name.lower().startswith('subsdk'):
|
||||
out.append((mod.name, f.relative_to(mod).as_posix(), f))
|
||||
return out
|
||||
|
||||
|
||||
def emit(results, args):
|
||||
"""--patches-out and --status from the checked plugins. results: one (module, its NSO file, its record or None,
|
||||
the record's directory, hooks, reasons) per plugin of the layout; no reasons: accepted."""
|
||||
text = args.patches.read_text() if args.patches else ''
|
||||
builtin = builtin_sites(text)
|
||||
status, sets = [], []
|
||||
files = {str(f.resolve()): (m, rel, f) for m, rel, f in plugin_files(args.mods)}
|
||||
for module, file, rec, record_dir, hooks, reasons in results:
|
||||
hit = files.get(str(Path(file or '').resolve()))
|
||||
if not hit:
|
||||
print(f'plugins: {module}: {file} is not under {args.mods}')
|
||||
continue
|
||||
mod, rel, f = hit
|
||||
key, content = fnv(f'{mod}/{rel}'.encode()), fnv(f.read_bytes())
|
||||
if reasons:
|
||||
note = ' '.join(reasons[0].split())
|
||||
status.append(f'{key}\t{content}\tunsupported\t-\t{note}')
|
||||
print(f'plugins: unsupported {mod}/{rel} ({note})')
|
||||
continue
|
||||
name = f'mod_{key.lower()}'
|
||||
own = hooks_of(hooks, module)
|
||||
sites = [int(h['site'].split('+')[1], 16) for h in own]
|
||||
over = sorted({b for a in sites for b in builtin.get(a, [])})
|
||||
label = f'{mod}/{rel}'.encode('ascii', 'replace').decode()
|
||||
sets.append([f'set {name} TOTK_MODPATCH_{key} 0 ordered # {label}'] + plugin_lines(rec, own, name, record_dir,
|
||||
module))
|
||||
status.append(f'{key}\t{content}\tplugin\t{",".join(over) or "-"}\t{module}, {len(own)} hooks')
|
||||
print(f'plugins: plugin {mod}/{rel} as {module}, {len(own)} hooks' + (f', over {", ".join(over)}'
|
||||
if over else ''))
|
||||
if sets:
|
||||
text += '\n# Mod plugins (tools/exefs_plugins.py)\n' + '\n'.join(line for s in sets for line in s) + '\n'
|
||||
args.patches_out.write_text(text)
|
||||
args.status.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.status.write_text('\n'.join(status) + ('\n' if status else ''), encoding='utf-8')
|
||||
|
||||
|
||||
# ---- the service runs
|
||||
def where_is(v, rec):
|
||||
"""What an address-like value points into, for the diff listing."""
|
||||
for m in rec['modules']:
|
||||
if m['base'] <= v < m['base'] + m['size']:
|
||||
return f'{m["name"]}+{v - m["base"]:#x}'
|
||||
for k in ('rtld', 'pool'):
|
||||
r = rec.get(k)
|
||||
if r and u(r['base']) <= v < u(r['base']) + u(r['size']):
|
||||
return f'the {"alias pool" if k == "pool" else k} (+{v - u(r["base"]):#x})'
|
||||
for a in rec.get('aliases', []):
|
||||
if u(a['dst']) <= v < u(a['dst']) + u(a['size']):
|
||||
return f'an alias of {u(a["src"]):#x}'
|
||||
return None
|
||||
|
||||
|
||||
def section(m, off, nso_path):
|
||||
if off < m['text_end']:
|
||||
return '.text'
|
||||
if off < m['data_off']:
|
||||
return '.rodata'
|
||||
if nso_path and Path(nso_path).is_file():
|
||||
n = Nso(nso_path)
|
||||
if off < n.data[0] + n.data[1]:
|
||||
return '.data'
|
||||
return '.bss'
|
||||
return '.data/.bss'
|
||||
|
||||
|
||||
def diff_records(a, b):
|
||||
"""Lines naming every difference between two records: writes into game modules, plugin image words (8 bytes)."""
|
||||
out = []
|
||||
wa = {(w['module'], w['offset']): w for w in a['writes']}
|
||||
wb = {(w['module'], w['offset']): w for w in b['writes']}
|
||||
for k in sorted(set(wa) | set(wb)):
|
||||
x, y = wa.get(k), wb.get(k)
|
||||
if not x or not y or x['new'] != y['new']:
|
||||
out.append(f'write {k[0]}+{k[1]:#x}: run 1 {x and hex(x["new"])}, run 2 {y and hex(y["new"])}')
|
||||
for m in a['modules']:
|
||||
if not m['plugin']:
|
||||
continue
|
||||
ia, ib = a['images'].get(m['name']), b['images'].get(m['name'])
|
||||
if ia is None or ib is None or len(ia) != len(ib):
|
||||
out.append(f'{m["name"]}: image missing or of another size')
|
||||
continue
|
||||
for off in range(0, len(ia), 8):
|
||||
if ia[off:off + 8] == ib[off:off + 8]:
|
||||
continue
|
||||
va, vb = struct.unpack_from('<Q', ia, off)[0], struct.unpack_from('<Q', ib, off)[0]
|
||||
what = where_is(va, a) or where_is(vb, b)
|
||||
out.append(f'{m["name"]}+{off:#x} ({section(m, off, m.get("file"))}): {va:#x} / {vb:#x}'
|
||||
+ (f', {what}' if what else ''))
|
||||
return out
|
||||
|
||||
|
||||
def run(exe, out, tick_offset, timeout, only):
|
||||
"""One service run into out, only module `only`'s init running ("-": none). (exit code or None, why the run gave
|
||||
no record or None); a refused run writes its record too."""
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
for f in out.iterdir(): # an earlier run's record must not pass for this one's
|
||||
if f.is_file() and (f.name in ('record.json', 'run.log') or f.suffix == '.img'):
|
||||
f.unlink()
|
||||
env = dict(os.environ, TOTK_PLUGIN_TICK_OFFSET=f'{tick_offset:#x}', TOTK_PLUGIN_ONLY=only,
|
||||
TOTK_BACKGROUND_WINDOW='1', TOTK_ASSUME_FOCUS='1')
|
||||
try:
|
||||
r = subprocess.run([str(exe), '--plugin-init', str(out)], env=env, capture_output=True, text=True,
|
||||
errors='replace', timeout=timeout)
|
||||
except subprocess.TimeoutExpired as e:
|
||||
log = b''.join(x if isinstance(x, bytes) else x.encode() for x in (e.stdout, e.stderr) if x)
|
||||
(out / 'run.log').write_text(log.decode(errors='replace'), encoding='utf-8')
|
||||
return None, f'its init did not return within {timeout} s'
|
||||
(out / 'run.log').write_text(r.stdout + r.stderr, encoding='utf-8')
|
||||
if not (out / 'record.json').is_file():
|
||||
# the runtime's crash report names the fault in a FATAL line (an access violation, an abort)
|
||||
fatal = next((line.split('FATAL:', 1)[1].strip() for line in (r.stdout + r.stderr).splitlines()
|
||||
if 'FATAL:' in line), None)
|
||||
return r.returncode, (f'its init crashed the service run: {fatal}' if fatal else
|
||||
f'its init ended the service run without a record (exit {r.returncode & 0xFFFFFFFF:#x})')
|
||||
return r.returncode, None
|
||||
|
||||
|
||||
def report(rec, label):
|
||||
rec = ran_view(rec)
|
||||
impls = hook_impls(rec)
|
||||
hooks, reasons = check_record(rec, impls)
|
||||
more, notes = reach_check(rec, hooks, impls)
|
||||
reasons += more
|
||||
for n in notes:
|
||||
print(f' {n}')
|
||||
print(f'{label}: {len(rec["writes"])} words written into game modules, {len(rec["aliases"])} process-memory maps, '
|
||||
f'svcs {", ".join(f"{ALLOWED_SVCS.get(u(k), hex(u(k)))} x{v}" for k, v in rec["svcs"].items())}; '
|
||||
f'GetInfo {", ".join(f"{k} x{v}" for k, v in rec["get_info"].items())}; '
|
||||
f'imports {", ".join(f"{k} x{v}" for k, v in rec["imports"].items())}')
|
||||
for h in hooks:
|
||||
if h['kind'] == 'inline':
|
||||
print(f' inline hook {h["site"]} (was {h["original"]}) -> stub {h["stub"]} -> {h["impl"]}, callback '
|
||||
f'{h["callback"]}, trampoline {h["trampoline"]} ({h.get("relocated")}; b site+4)')
|
||||
else:
|
||||
print(f' {h["kind"]} {h["site"]} (was {h["original"]}) -> {"bl" if h["kind"] == "call" else "b"} '
|
||||
f'{h["target"]}' + (f', Orig trampoline {h["trampoline"]}' if h.get('trampoline') else ''))
|
||||
for r in reasons:
|
||||
print(f' REFUSED: {r}')
|
||||
if not reasons:
|
||||
print(f' accepted: {len(hooks)} hooks')
|
||||
return hooks, reasons
|
||||
|
||||
|
||||
def plugin_runs(exe, out, module, timeout):
|
||||
"""The two service runs of one plugin (run2 with the tick offset) into out/<module>: (its first record or None,
|
||||
hooks, reasons)."""
|
||||
recs = []
|
||||
for i, offset in enumerate((0, TICK_OFFSET), 1):
|
||||
d = out / module / f'run{i}'
|
||||
code, why = run(exe, d, offset, timeout, module)
|
||||
if why:
|
||||
print(f'{module} run {i}: {why}; see {d / "run.log"}')
|
||||
return None, [], [why]
|
||||
recs.append(load_record(d))
|
||||
print(f'{module} run {i}: exit {code}, tick offset {offset:#x}')
|
||||
if recs[0].get('refused'): # refused: a second run says nothing more
|
||||
hooks, bad = report(recs[0], f'{module} run 1')
|
||||
return recs[0], hooks, bad
|
||||
hooks, bad = report(recs[0], f'{module} run 1')
|
||||
diff = diff_records(ran_view(recs[0]), ran_view(recs[1]))
|
||||
print(f'{module} run 1 against run 2: {len(diff)} differences' + (':' if diff else ''))
|
||||
for line in diff:
|
||||
print(f' {line}')
|
||||
# Words of a game module or of the plugin's .text that differ between the runs: the start-up could not be compared
|
||||
# with the record. Its .data and .bss may differ (alias addresses).
|
||||
unstable = [line for line in diff if line.startswith('write ') or '(.text)' in line or '(.rodata)' in line]
|
||||
if unstable:
|
||||
bad = bad + [f'two runs of the init differ in code: {unstable[0]}']
|
||||
return recs[0], hooks, bad
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--exe', type=Path, help='totk.exe built with the plugins in its layout')
|
||||
ap.add_argument('--out', type=Path, help='directory for list/ and <plugin>/run1, run2')
|
||||
ap.add_argument('--check', type=Path, help='check the record in this directory only')
|
||||
ap.add_argument('--timeout', type=int, default=120, help='seconds a service run may take')
|
||||
ap.add_argument('--mods', type=Path, help='the mods folder the plugins come from')
|
||||
ap.add_argument('--patches', type=Path, help='the patch file so far')
|
||||
ap.add_argument('--patches-out', type=Path, help='the patch file with the plugins\' sets added')
|
||||
ap.add_argument('--status', type=Path, help='the plugin status file (build-cache/plugins.status)')
|
||||
args = ap.parse_args()
|
||||
sys.stdout.reconfigure(errors='replace')
|
||||
emitting = bool(args.patches_out or args.status)
|
||||
if emitting and not (args.mods and args.patches_out and args.status):
|
||||
ap.error('--mods, --patches-out and --status go together')
|
||||
if args.check:
|
||||
rec = load_record(args.check)
|
||||
hooks, bad = report(rec, str(args.check))
|
||||
if emitting:
|
||||
emit([(m['name'], m.get('file'), rec, args.check, hooks, bad) for m in ran_view(rec)['modules']
|
||||
if m['plugin']], args)
|
||||
return 1 if bad else 0
|
||||
if not args.exe or not args.out:
|
||||
ap.error('--exe and --out, or --check')
|
||||
# A run without any init names the plugin modules of the exe's layout and checks that the service run works at all:
|
||||
# without it no plugin can be built.
|
||||
code, why = run(args.exe, args.out / 'list', 0, args.timeout, '-')
|
||||
if why:
|
||||
print(f'the service run without inits gave no record: {why}; see {args.out / "list" / "run.log"}')
|
||||
return 2
|
||||
listing = load_record(args.out / 'list')
|
||||
results = []
|
||||
for m in (m for m in listing['modules'] if m['plugin']):
|
||||
rec, hooks, bad = plugin_runs(args.exe, args.out, m['name'], args.timeout)
|
||||
print(f'{m["name"]}: ' + (f'refused: {bad[0]}' if bad else f'accepted, {len(hooks)} hooks'))
|
||||
results.append((m['name'], m.get('file'), rec, args.out / m['name'] / 'run1', hooks, bad))
|
||||
if emitting:
|
||||
emit(results, args)
|
||||
return 1 if any(r[5] for r in results) else 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -19,13 +19,15 @@ cached in toolchain/release-dl/ and checked against the sha256 in PINS; a mismat
|
||||
install compiles that update's code and the runtime libraries with it
|
||||
VERSION the repository's VERSION plus the commit, 0.1.0+973c626; the installer's release string
|
||||
TotK-Setup.exe installer/launcher/launcher.c built with llvm/ (-mwindows): starts python/pythonw.exe -m installer.gui
|
||||
Everything else is the build kit the setup copies into the install folder (installer/layout.py KIT_ITEMS)
|
||||
|
||||
--prove compiles the generated C and links build/release-check/totk.exe with nothing but the folder's python/, llvm/,
|
||||
runtime/ and bin/ (PATH holds the Windows system directory and those), then runs the four probes there.
|
||||
|
||||
Both need one game version extracted and lifted first (tools/extract.py, recomp/gen.py): generated/v121 or, without
|
||||
it, generated/v143, and for --prove its build/extracted/<version> (they are read, nothing is written to them). The
|
||||
runtime library reads only what both versions' layout.h share (MOD_COUNT, MOD_MAIN_BASE), so either serves.
|
||||
runtime library reads only what both versions' layout.h share (MOD_MAIN_BASE; its module tables hold RT_MAX_MODULES, so an
|
||||
install's layout may add mod plugins), so either serves.
|
||||
|
||||
A release also needs build/xlate built from the current sources (tools/build_xlate.py); the script stops first if not.
|
||||
|
||||
@@ -115,7 +117,7 @@ LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'}
|
||||
TOOLS_FILES = ['build_game.py', 'build_shaders.py', 'button_icons.py', 'check_content.py', 'check_isa.py', 'extract.py',
|
||||
'hostmem.py', 'make_romfs_patches.py', 'ncz.py', 'pack_shader_cache.py', 'prepare_shaders.py',
|
||||
'romfs_index.py', 'shader_census.py', 'shader_list.py', 'ultrawide_layout.py', 'cheats.py',
|
||||
'exefs_patches.py']
|
||||
'exefs_patches.py', 'exefs_plugins.py']
|
||||
TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes
|
||||
RUNTIME_SUFFIXES = {'.h', '.inc', '.ico'} # .ico: the installer window icon
|
||||
|
||||
|
||||
+4
-1
@@ -28,6 +28,9 @@
|
||||
# highfps_fix = 1
|
||||
# Climbing walls and ladders at full speed above 60 fps; 30 and 60 fps are unchanged. Needs realtime = 1.
|
||||
# climb_fix = 1
|
||||
# The game's audio thread takes every queued sound command list each 5 ms instead of one, so the queue does not fill
|
||||
# above 200 fps, where the game would switch its sound off for good. 0 = the game's own code.
|
||||
# audio_drain_fix = 1
|
||||
|
||||
[performance]
|
||||
# The keys of this section and of [diagnostics] are for testing and troubleshooting, and take effect at the next start.
|
||||
@@ -229,7 +232,7 @@
|
||||
# does. Read when a movie starts, so a change takes effect at the next movie.
|
||||
# movie_aspect = keep
|
||||
# Field of view, in percent of the game's own: 70 to 140, 100 (the default) leaves the camera as it is. Higher shows
|
||||
# more of the world around Link. Game versions 1.1.0 and 1.4.3. Changes while the game runs take effect at once.
|
||||
# more of the world around Link. Game versions 1.1.0, 1.2.1 and 1.4.3. Changes while the game runs take effect at once.
|
||||
# fov = 100
|
||||
# Anisotropic filtering of surface textures: game (the level the game asks for, as on the Switch) or 2, 4, 8, 16. Keeps
|
||||
# ground, roads and walls seen at a shallow angle sharp into the distance. Only filtered, mipmapped textures that tile
|
||||
|
||||
Reference in new issue
Block a user