mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-20 12:45:57 +00:00
Build the game code with an execution profile when the release has one
build_game.py --pgo gen builds an instrumented totk.exe (llvm-mingw: MSYS2's clang has no profile runtime) that adds its counts to <out>/pgo/*.profraw every minute and at exit; plat_exit ends the process with ExitProcess, which skips compiler-rt's atexit write. tools/pgo.py merges them into build/pgo/pgo-<ver>.profdata with a record of the recomp/ tree it fits. build_game.py --pgo FILE compiles the generated code at -O2 and the runtime with that profile. make_release.py ships build/pgo's profiles in data/ and builds the runtime libraries with them; the installer passes the update's profile to build_game.py, and builds at -O1 without one. Profiles name lifted functions by address and hold branch counts and control-flow hashes of the generated C, no bytes of the game. 1.4.3 on machine A, mean frame by world clock: the fight2 bench 11.49 -> 10.06 ms (-12.5%), a fight outside the training 13.00 -> 12.24 ms (-5.8%); every window CPU-bound. The profile was trained on scripted runs and 24 minutes of play. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0f65b4ee07
commit
282130a128
9 files changed
+255
-19
No files matched your search
@@ -150,6 +150,33 @@ your machines without MSYS2, Python or .NET. `romfs.bin` stays out unless you pa
|
|||||||
`romfs\` next to the exe or point `TOTK_ROMFS` at its folder. Run `totk.exe --prepare-shaders` once in the copied
|
`romfs\` next to the exe or point `TOTK_ROMFS` at its folder. Run `totk.exe --prepare-shaders` once in the copied
|
||||||
folder. The folder holds code and data derived from your game: for your own machines only.
|
folder. The folder holds code and data derived from your game: for your own machines only.
|
||||||
|
|
||||||
|
### Profile-guided build
|
||||||
|
|
||||||
|
`build_game.py --pgo <file>.profdata` compiles the generated code at -O2 with an execution profile, and the runtime
|
||||||
|
with it too: hot paths laid out straight, hot calls inlined, code that never ran compiled for size. On 1.4.3 the fight
|
||||||
|
bench ran 12-15% faster per frame and a fight outside the training about 6%. The installer builds with
|
||||||
|
`data/pgo-<ver>.profdata` of the release when the update has one, else at -O1.
|
||||||
|
|
||||||
|
A profile only fits the generated C it was trained on: functions are matched by name and control-flow hash, so a
|
||||||
|
function whose C changes loses its counts without a warning. Retrain after a lifter or generator change, from a clean
|
||||||
|
`recomp/`. The instrumented build needs llvm-mingw, as MSYS2's clang has no profile runtime:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$llvm = "toolchain\llvm-mingw-<date>\llvm-mingw-<date>-ucrt-x86_64"
|
||||||
|
.venv\Scripts\python tools\build_game.py --version v143 --out build\v143-pgogen --toolchain $llvm --pgo gen
|
||||||
|
```
|
||||||
|
|
||||||
|
Play `build\v143-pgogen\totk.exe` through varied scenes (fights, towns, riding, building, the Depths, menus) for 20-30
|
||||||
|
minutes; it is slower than a normal build. It writes `build\v143-pgogen\pgo\*.profraw` every minute and at exit.
|
||||||
|
Scripted runs from saves (`tools/run_game.py`) add to the same file. Then:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.venv\Scripts\python tools\pgo.py merge build\v143-pgogen --version v143
|
||||||
|
```
|
||||||
|
|
||||||
|
writes `build\pgo\pgo-v143.profdata` and its record (`.json`: the `recomp/` tree it fits). `make_release.py` ships every
|
||||||
|
profile of `build\pgo` in `data\` and warns when a record names another `recomp/` tree than the one released.
|
||||||
|
|
||||||
## Linux (in progress)
|
## Linux (in progress)
|
||||||
|
|
||||||
The runtime builds and runs on x86-64 Linux with a window, input and menu on SDL3 (`*_sdl.c`) in place of the Win32
|
The runtime builds and runs on x86-64 Linux with a window, input and menu on SDL3 (`*_sdl.c`) in place of the Win32
|
||||||
|
|||||||
+5
-3
@@ -147,15 +147,17 @@ llvm/ llvm-mingw, x86_64-only subset (Apache-2.0 with LLVM excep
|
|||||||
runtime/ libtotkrt.a, libtotkrt-v2.a and the headers the generated code and modules.c/romfs.c/patches.c
|
runtime/ libtotkrt.a, libtotkrt-v2.a and the headers the generated code and modules.c/romfs.c/patches.c
|
||||||
include
|
include
|
||||||
bin/ totk_xlate.exe, hactool.exe, SDL3.dll, libopus-0.dll, ninja.exe
|
bin/ totk_xlate.exe, hactool.exe, SDL3.dll, libopus-0.dll, ninja.exe
|
||||||
data/ shader_list.tsv (MacNativeGames, MIT)
|
data/ shader_list.tsv (MacNativeGames, MIT); pgo-<ver>.profdata + .json, the execution profile an
|
||||||
|
update's code and the runtime libraries compile with (building.md, "Profile-guided build")
|
||||||
THIRD_PARTY_NOTICES.md
|
THIRD_PARTY_NOTICES.md
|
||||||
totk.ini.example, VERSION the default ini; the release string (the repository's VERSION plus the commit)
|
totk.ini.example, VERSION the default ini; the release string (the repository's VERSION plus the commit)
|
||||||
```
|
```
|
||||||
|
|
||||||
- SDL3 comes from the upstream release, not MSYS2: MSYS2's DLL imports the LGPL libiconv.
|
- SDL3 comes from the upstream release, not MSYS2: MSYS2's DLL imports the LGPL libiconv.
|
||||||
- Every bundled third-party piece gets a row in `THIRD_PARTY_NOTICES.md`.
|
- Every bundled third-party piece gets a row in `THIRD_PARTY_NOTICES.md`.
|
||||||
- The release contains nothing game-derived; `make_release.py` runs `tools/check_content.py` over the folder before it
|
- The release contains no code, data, text or assets of the game; `make_release.py` runs `tools/check_content.py` over
|
||||||
finishes.
|
the folder before it finishes. The execution profiles name lifted functions by address (`f_<address>`) and hold
|
||||||
|
branch counts and control-flow hashes of the generated C.
|
||||||
|
|
||||||
## Switch system fonts
|
## Switch system fonts
|
||||||
|
|
||||||
|
|||||||
+13
-2
@@ -12,6 +12,8 @@ Release (docs/installer.md "Release layout"), root = the unpacked folder:
|
|||||||
include/ + lib/ of Vulkan, SDL3, Opus, libvpx and VMA (build_game.py --deps)
|
include/ + lib/ of Vulkan, SDL3, Opus, libvpx and VMA (build_game.py --deps)
|
||||||
bin/ totk_xlate.exe (+ totk_xlate.id), hactool.exe, ninja.exe, SDL3.dll, libopus-0.dll
|
bin/ totk_xlate.exe (+ totk_xlate.id), hactool.exe, ninja.exe, SDL3.dll, libopus-0.dll
|
||||||
data/shader_list.tsv the MacNativeGames shader list
|
data/shader_list.tsv the MacNativeGames shader list
|
||||||
|
data/pgo-<v>.profdata the execution profile the update's code is compiled with (tools/pgo.py; Windows only);
|
||||||
|
without one the game is built at -O1
|
||||||
totk.ini.example
|
totk.ini.example
|
||||||
VERSION the release version, one line
|
VERSION the release version, one line
|
||||||
temp/ the default temp folder (created)
|
temp/ the default temp folder (created)
|
||||||
@@ -24,7 +26,7 @@ Dev mode, root = a repository checkout (no python/python.exe): build/py314-embed
|
|||||||
Python the release ships, with the same packages), else this Python; toolchain/llvm-mingw-*/ and toolchain/deps-mingw;
|
Python the release ships, with the same packages), else this Python; toolchain/llvm-mingw-*/ and toolchain/deps-mingw;
|
||||||
runtime/ as the sources with build/install-rtlib/libtotkrt.a and libtotkrt-v2.a (tools/build_game.py --make-runtime-lib
|
runtime/ as the sources with build/install-rtlib/libtotkrt.a and libtotkrt-v2.a (tools/build_game.py --make-runtime-lib
|
||||||
--out build/install-rtlib, once per --cpu); MSYS2 UCRT64 ninja.exe; toolchain/hactool; build/xlate/totk_xlate.exe; the shader list from
|
--out build/install-rtlib, once per --cpu); MSYS2 UCRT64 ninja.exe; toolchain/hactool; build/xlate/totk_xlate.exe; the shader list from
|
||||||
refs/MacNativeGames; build/install-temp. The version is VERSION with -dev and the commit: 0.1.0-dev+973c626. On Linux,
|
refs/MacNativeGames; build/pgo/pgo-<v>.profdata when there is one; build/install-temp. The version is VERSION with -dev and the commit: 0.1.0-dev+973c626. On Linux,
|
||||||
dev mode takes clang, lld and ninja from PATH and the system's libraries (no deps folder, no SDL3 copy).
|
dev mode takes clang, lld and ninja from PATH and the system's libraries (no deps folder, no SDL3 copy).
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
@@ -66,6 +68,7 @@ class Layout:
|
|||||||
temp: Path | None = None
|
temp: Path | None = None
|
||||||
sysroot: Path | None = None # Linux release: what the game links against, instead of the host's
|
sysroot: Path | None = None # Linux release: what the game links against, instead of the host's
|
||||||
runtime_lib_v2: Path | None = None # libtotkrt-v2.a: the runtime for CPUs without AVX2 (build_game.py --cpu v2)
|
runtime_lib_v2: Path | None = None # libtotkrt-v2.a: the runtime for CPUs without AVX2 (build_game.py --cpu v2)
|
||||||
|
pgo_dir: Path | None = None # pgo-<v>.profdata (tools/pgo.py); None: no profiles (Linux)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def clang(self):
|
def clang(self):
|
||||||
@@ -79,6 +82,12 @@ class Layout:
|
|||||||
return lib
|
return lib
|
||||||
return None if self.dev else lib
|
return None if self.dev else lib
|
||||||
|
|
||||||
|
def pgo(self, version):
|
||||||
|
"""The execution profile build_game.py --pgo compiles the update's code with, None without one (the game is
|
||||||
|
then built at -O1)."""
|
||||||
|
p = self.pgo_dir / f'pgo-{version_tag(version)}.profdata' if self.pgo_dir else None
|
||||||
|
return p if p and p.is_file() else None
|
||||||
|
|
||||||
def patches(self, version):
|
def patches(self, version):
|
||||||
"""The code patch table for recomp/gen.py."""
|
"""The code patch table for recomp/gen.py."""
|
||||||
return self.recomp / f'patches-{version_tag(version)}.txt'
|
return self.recomp / f'patches-{version_tag(version)}.txt'
|
||||||
@@ -129,6 +138,7 @@ def release_layout(root):
|
|||||||
xlate=root / 'bin' / ('totk_xlate' + EXE),
|
xlate=root / 'bin' / ('totk_xlate' + EXE),
|
||||||
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll') if WIN else (root / 'bin' / 'libSDL3.so.0',),
|
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll') if WIN else (root / 'bin' / 'libSDL3.so.0',),
|
||||||
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example',
|
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example',
|
||||||
|
pgo_dir=root / 'data' if WIN else None,
|
||||||
# Linux: None, the install folder's .setup-temp (installer/cli.py): the AppImage is mounted read-only, and on
|
# Linux: None, the install folder's .setup-temp (installer/cli.py): the AppImage is mounted read-only, and on
|
||||||
# a Steam Deck /home is often too full for the generated C while the install folder is on the SD card
|
# a Steam Deck /home is often too full for the generated C while the install folder is on the SD card
|
||||||
temp=root / 'temp' if WIN else None,
|
temp=root / 'temp' if WIN else None,
|
||||||
@@ -154,7 +164,8 @@ def dev_layout(root):
|
|||||||
ninja=UCRT / ('ninja' + EXE), hactool=root / 'toolchain' / 'hactool' / ('hactool' + EXE),
|
ninja=UCRT / ('ninja' + EXE), hactool=root / 'toolchain' / 'hactool' / ('hactool' + EXE),
|
||||||
xlate=root / 'build' / 'xlate' / ('totk_xlate' + EXE),
|
xlate=root / 'build' / 'xlate' / ('totk_xlate' + EXE),
|
||||||
dlls=tuple(sorted((deps / 'bin').glob('*.dll'))),
|
dlls=tuple(sorted((deps / 'bin').glob('*.dll'))),
|
||||||
shader_list=None, ini_example=root / 'totk.ini.example', temp=root / 'build' / 'install-temp')
|
shader_list=None, ini_example=root / 'totk.ini.example', temp=root / 'build' / 'install-temp',
|
||||||
|
pgo_dir=root / 'build' / 'pgo')
|
||||||
|
|
||||||
|
|
||||||
def dev_commit(root):
|
def dev_commit(root):
|
||||||
|
|||||||
+5
-2
@@ -944,7 +944,8 @@ class Compile(Step):
|
|||||||
id = 'compile'
|
id = 'compile'
|
||||||
|
|
||||||
def key(self, ctx):
|
def key(self, ctx):
|
||||||
return fingerprint('compile-3', ctx.keys['lift'], runtime_key(ctx))
|
return fingerprint('compile-3', ctx.keys['lift'], runtime_key(ctx),
|
||||||
|
ctx.layout.pgo(game_version(ctx)) or 'no profile')
|
||||||
|
|
||||||
def current(self, ctx, record):
|
def current(self, ctx, record):
|
||||||
exe = ctx.dest / ('totk' + EXE)
|
exe = ctx.dest / ('totk' + EXE)
|
||||||
@@ -970,10 +971,12 @@ class Compile(Step):
|
|||||||
rep.progress('compile', 'compile', force=True)
|
rep.progress('compile', 'compile', force=True)
|
||||||
cpu = getattr(ctx, 'cpu', 'v3')
|
cpu = getattr(ctx, 'cpu', 'v3')
|
||||||
lib = layout.runtime_lib_for(cpu)
|
lib = layout.runtime_lib_for(cpu)
|
||||||
|
pgo = layout.pgo(game_version(ctx))
|
||||||
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
|
code, log = run_tool(ctx, 'compile', layout.tools / 'build_game.py', [
|
||||||
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--cpu', cpu,
|
'--gen', ctx.temp / 'gen', '--out', build, '--toolchain', layout.llvm, '--cpu', cpu,
|
||||||
*(['--deps', layout.deps] if layout.deps else []), *(['--sysroot', layout.sysroot] if layout.sysroot else []),
|
*(['--deps', layout.deps] if layout.deps else []), *(['--sysroot', layout.sysroot] if layout.sysroot else []),
|
||||||
'--runtime', layout.runtime, *(['--runtime-lib', lib] if lib else ()), '--ninja', layout.ninja,
|
'--runtime', layout.runtime, *(['--runtime-lib', lib] if lib else ()), *(['--pgo', pgo] if pgo else ()),
|
||||||
|
'--ninja', layout.ninja,
|
||||||
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
|
'--version', version_tag(game_version(ctx)), '--expected', layout.expected(game_version(ctx)),
|
||||||
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
'--exefs', dest / 'exefs', '--romfs', dest / 'romfs',
|
||||||
'--profile', dest / 'profile'], parse)
|
'--profile', dest / 'profile'], parse)
|
||||||
|
|||||||
@@ -1275,7 +1275,16 @@ int plat_stderr_path(char *out, size_t cap) {
|
|||||||
|
|
||||||
// ---- process
|
// ---- process
|
||||||
|
|
||||||
|
#ifdef TOTK_PGO_GEN
|
||||||
|
// compiler-rt's profile runtime (an instrumented build, tools/build_game.py --pgo gen). It writes its counters from an
|
||||||
|
// atexit handler, which _exit skips; __llvm_profile_dump writes them once and makes later writes no-ops.
|
||||||
|
int __llvm_profile_dump(void);
|
||||||
|
#endif
|
||||||
|
|
||||||
void plat_exit(int code) {
|
void plat_exit(int code) {
|
||||||
|
#ifdef TOTK_PGO_GEN
|
||||||
|
__llvm_profile_dump();
|
||||||
|
#endif
|
||||||
fflush(stdout);
|
fflush(stdout);
|
||||||
fflush(stderr);
|
fflush(stderr);
|
||||||
_exit(code);
|
_exit(code);
|
||||||
|
|||||||
@@ -791,7 +791,33 @@ void plat_aligned_free(void *p) { _aligned_free(p); }
|
|||||||
|
|
||||||
// ---- process
|
// ---- process
|
||||||
|
|
||||||
|
#ifdef TOTK_PGO_GEN
|
||||||
|
// compiler-rt's profile runtime (an instrumented build, tools/build_game.py --pgo gen). It writes its counters from an
|
||||||
|
// atexit handler, which ExitProcess skips; __llvm_profile_dump writes them once and makes later writes no-ops.
|
||||||
|
int __llvm_profile_dump(void);
|
||||||
|
int __llvm_profile_write_file(void);
|
||||||
|
void __llvm_profile_reset_counters(void);
|
||||||
|
|
||||||
|
// A long played session must not lose its counts to a crash or a reboot: every minute the counters are added to the
|
||||||
|
// .profraw (the file merges, -fprofile-generate's %m) and zeroed. Counts taken between the write and the reset are lost.
|
||||||
|
static DWORD WINAPI pgo_flush_thread(void *arg) {
|
||||||
|
(void)arg;
|
||||||
|
for (;;) {
|
||||||
|
Sleep(60 * 1000);
|
||||||
|
if (__llvm_profile_write_file() == 0) __llvm_profile_reset_counters();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__attribute__((constructor)) static void pgo_flush_start(void) {
|
||||||
|
HANDLE h = CreateThread(NULL, 0, pgo_flush_thread, NULL, 0, NULL);
|
||||||
|
if (h) CloseHandle(h);
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
void plat_exit(int code) {
|
void plat_exit(int code) {
|
||||||
|
#ifdef TOTK_PGO_GEN
|
||||||
|
__llvm_profile_dump();
|
||||||
|
#endif
|
||||||
fflush(stdout);
|
fflush(stdout);
|
||||||
fflush(stderr);
|
fflush(stderr);
|
||||||
ExitProcess((UINT)code);
|
ExitProcess((UINT)code);
|
||||||
|
|||||||
+42
-2
@@ -109,6 +109,36 @@ def set_cpu(cpu):
|
|||||||
GEN_FLAGS[GEN_FLAGS.index('-march=x86-64-v3')] = march
|
GEN_FLAGS[GEN_FLAGS.index('-march=x86-64-v3')] = march
|
||||||
RT_FLAGS[RT_FLAGS.index('-march=x86-64-v3')] = march
|
RT_FLAGS[RT_FLAGS.index('-march=x86-64-v3')] = march
|
||||||
RT_FLAGS.append(f'-DRT_CPU_LEVEL={3 if cpu == "v3" else 2}')
|
RT_FLAGS.append(f'-DRT_CPU_LEVEL={3 if cpu == "v3" else 2}')
|
||||||
|
|
||||||
|
|
||||||
|
# Profile-guided optimisation (--pgo). Both modes compile the generated code at -O2, the level the profile's inlining and
|
||||||
|
# hot/cold layout work at. A static function's profile name carries its source path by default; the base name alone
|
||||||
|
# keeps the names equal between the training build and a build from another tree (a worktree, an install).
|
||||||
|
PGO_NAMES = ['-mllvm', '-static-func-full-module-prefix=false']
|
||||||
|
PGO_INPUT = None # the .profdata of --pgo FILE: every compile depends on it, so a new profile rebuilds them
|
||||||
|
|
||||||
|
|
||||||
|
def set_pgo(pgo, out):
|
||||||
|
"""--pgo gen: instrument the generated code and the runtime; totk.exe adds its counts to
|
||||||
|
out/pgo/default_<id>.profraw every minute and at exit (runtime/platform win32.c), across runs. --pgo FILE: compile
|
||||||
|
with that merged profile (tools/pgo.py merge). Functions the profile never saw run are optimised for size."""
|
||||||
|
global PGO_INPUT
|
||||||
|
GEN_FLAGS[GEN_FLAGS.index('-O1')] = '-O2'
|
||||||
|
if pgo == 'gen':
|
||||||
|
# absolute: the instrumented exe resolves the directory against its own working directory
|
||||||
|
flags = [f'-fprofile-generate={cmd_arg((out / "pgo").resolve().as_posix())}']
|
||||||
|
RT_FLAGS.append('-DTOTK_PGO_GEN')
|
||||||
|
LINK_FLAGS.append('-fprofile-generate') # links clang_rt.profile
|
||||||
|
else:
|
||||||
|
PGO_INPUT = Path(pgo).resolve()
|
||||||
|
if not PGO_INPUT.is_file():
|
||||||
|
sys.exit(f'--pgo {pgo}: no such profile')
|
||||||
|
# unprofiled: chunks the training never entered; out-of-date and backend-plugin: functions whose code changed
|
||||||
|
# since the training, which then compile without a profile
|
||||||
|
flags = [f'-fprofile-use={cmd_path(PGO_INPUT)}', '-Wno-profile-instr-unprofiled',
|
||||||
|
'-Wno-profile-instr-out-of-date', '-Wno-backend-plugin']
|
||||||
|
GEN_FLAGS.extend(flags + PGO_NAMES)
|
||||||
|
RT_FLAGS.extend(flags + PGO_NAMES)
|
||||||
if WIN:
|
if WIN:
|
||||||
# _WIN32_WINNT: Windows 10, the port's minimum. MSYS2's headers default to it, llvm-mingw's (the release toolchain)
|
# _WIN32_WINNT: Windows 10, the port's minimum. MSYS2's headers default to it, llvm-mingw's (the release toolchain)
|
||||||
# to Windows 7, which hides the per-monitor DPI functions gfx_window_win.c and dpi_probe_win.c call.
|
# to Windows 7, which hides the per-monitor DPI functions gfx_window_win.c and dpi_probe_win.c call.
|
||||||
@@ -384,12 +414,13 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
|||||||
'',
|
'',
|
||||||
]
|
]
|
||||||
objs = {}
|
objs = {}
|
||||||
|
pgo_dep = f' | {ninja_path(PGO_INPUT)}' if PGO_INPUT else ''
|
||||||
for mod, files in mods.items():
|
for mod, files in mods.items():
|
||||||
objs[mod] = []
|
objs[mod] = []
|
||||||
for f in files:
|
for f in files:
|
||||||
o = out / 'obj' / mod / (f.stem + '.o')
|
o = out / 'obj' / mod / (f.stem + '.o')
|
||||||
if not prebuilt:
|
if not prebuilt:
|
||||||
lines.append(f'build {ninja_path(o)}: cc_gen {ninja_path(f)}')
|
lines.append(f'build {ninja_path(o)}: cc_gen {ninja_path(f)}{pgo_dep}')
|
||||||
lines.append(' pool = gen')
|
lines.append(' pool = gen')
|
||||||
objs[mod].append(o)
|
objs[mod].append(o)
|
||||||
rt = []
|
rt = []
|
||||||
@@ -406,7 +437,7 @@ def ninja_file(gen, out, mods, pool_depth, skip=(), prebuilt=False, runtime_lib=
|
|||||||
lines.append(f'build {ninja_path(o)}: rc {ninja_path(src)} | {ninja_path(RUNTIME / "totk.manifest")} '
|
lines.append(f'build {ninja_path(o)}: rc {ninja_path(src)} | {ninja_path(RUNTIME / "totk.manifest")} '
|
||||||
f'{ninja_path(RUNTIME / "totk.ico")}')
|
f'{ninja_path(RUNTIME / "totk.ico")}')
|
||||||
else:
|
else:
|
||||||
lines.append(f'build {ninja_path(o)}: cc_rt {ninja_path(src)}')
|
lines.append(f'build {ninja_path(o)}: cc_rt {ninja_path(src)}{pgo_dep}')
|
||||||
rt.append(o)
|
rt.append(o)
|
||||||
objs['runtime'] = rt
|
objs['runtime'] = rt
|
||||||
every = [o for v in objs.values() for o in v]
|
every = [o for v in objs.values() for o in v]
|
||||||
@@ -769,6 +800,10 @@ def main():
|
|||||||
help=f'build only <out>/{LIB_NAME} from runtime/ (no generated C, no totk.exe)')
|
help=f'build only <out>/{LIB_NAME} from runtime/ (no generated C, no totk.exe)')
|
||||||
ap.add_argument('--runtime-lib', type=Path, help=f'link against this {LIB_NAME} instead of compiling runtime/ '
|
ap.add_argument('--runtime-lib', type=Path, help=f'link against this {LIB_NAME} instead of compiling runtime/ '
|
||||||
f'({", ".join(sorted(INSTALL_SIDE))} and modules.c still compile)')
|
f'({", ".join(sorted(INSTALL_SIDE))} and modules.c still compile)')
|
||||||
|
ap.add_argument('--pgo', metavar='gen|FILE',
|
||||||
|
help='profile-guided optimisation, generated code at -O2: gen builds an instrumented totk.exe '
|
||||||
|
'(needs --toolchain on Windows: MSYS2 clang ships no profile runtime); FILE builds with that '
|
||||||
|
'.profdata (tools/pgo.py merge)')
|
||||||
ap.add_argument('--expected', type=Path,
|
ap.add_argument('--expected', type=Path,
|
||||||
help='expected hashes (default tools/data/expected-<display version>.json)')
|
help='expected hashes (default tools/data/expected-<display version>.json)')
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
@@ -776,6 +811,11 @@ def main():
|
|||||||
|
|
||||||
gen = (args.gen or ROOT / 'generated' / args.version).resolve()
|
gen = (args.gen or ROOT / 'generated' / args.version).resolve()
|
||||||
out = (args.out or ROOT / 'build' / args.version).resolve()
|
out = (args.out or ROOT / 'build' / args.version).resolve()
|
||||||
|
if args.pgo:
|
||||||
|
if args.pgo == 'gen' and WIN and not args.toolchain:
|
||||||
|
sys.exit('--pgo gen: MSYS2 UCRT64 clang ships no profile runtime (clang_rt.profile); pass --toolchain with '
|
||||||
|
'an llvm-mingw directory')
|
||||||
|
set_pgo(args.pgo, out)
|
||||||
extracted = None if args.exefs and args.romfs else work_dir() / 'extracted' / args.version
|
extracted = None if args.exefs and args.romfs else work_dir() / 'extracted' / args.version
|
||||||
paths = {'exefs': (args.exefs or extracted / 'exefs').resolve(),
|
paths = {'exefs': (args.exefs or extracted / 'exefs').resolve(),
|
||||||
'romfs': (args.romfs or extracted / 'romfs').resolve(),
|
'romfs': (args.romfs or extracted / 'romfs').resolve(),
|
||||||
|
|||||||
+44
-10
@@ -15,6 +15,8 @@ cached in toolchain/release-dl/ and checked against the sha256 in PINS; a mismat
|
|||||||
libraries of SDL3, Opus, libvpx, Vulkan and VMA)
|
libraries of SDL3, Opus, libvpx, Vulkan and VMA)
|
||||||
bin/ totk_xlate.exe (+ .id), hactool.exe, SDL3.dll (upstream), libopus-0.dll, ninja.exe
|
bin/ totk_xlate.exe (+ .id), hactool.exe, SDL3.dll (upstream), libopus-0.dll, ninja.exe
|
||||||
recomp/, tools/, data/, installer/, licenses/, THIRD_PARTY_NOTICES.md, totk.ini.example
|
recomp/, tools/, data/, installer/, licenses/, THIRD_PARTY_NOTICES.md, totk.ini.example
|
||||||
|
data/pgo-<v>.profdata build/pgo's execution profile of an update (tools/pgo.py) and its .json record: the
|
||||||
|
install compiles that update's code and the runtime libraries with it
|
||||||
VERSION the repository's VERSION plus the commit, 0.1.0+973c626; the installer's release string
|
VERSION the repository's VERSION plus the commit, 0.1.0+973c626; the installer's release string
|
||||||
TotK-Setup.exe installer/launcher/launcher.c built with llvm/ (-mwindows): starts python/pythonw.exe -m installer.gui
|
TotK-Setup.exe installer/launcher/launcher.c built with llvm/ (-mwindows): starts python/pythonw.exe -m installer.gui
|
||||||
|
|
||||||
@@ -27,9 +29,11 @@ runtime library reads only what both versions' layout.h share (MOD_COUNT, MOD_MA
|
|||||||
|
|
||||||
A release also needs build/xlate built from the current sources (tools/build_xlate.py); the script stops first if not.
|
A release also needs build/xlate built from the current sources (tools/build_xlate.py); the script stops first if not.
|
||||||
|
|
||||||
The folder contains nothing derived from the game: check_content runs over it before the script finishes. Then the
|
The folder contains no code, data, text or assets of the game: check_content runs over it before the script finishes.
|
||||||
folder's own python runs a check-only installer pass (no game files: it must get through the release, toolchain and host
|
The execution profiles name the lifted functions by their address (f_<address>) and hold branch counts and
|
||||||
checks and stop at the missing game files) and the window's --selftest.
|
control-flow hashes of the generated C, nothing of the game's own bytes. Then the folder's own python runs a check-only
|
||||||
|
installer pass (no game files: it must get through the release, toolchain and host checks and stop at the missing game
|
||||||
|
files) and the window's --selftest.
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import hashlib
|
import hashlib
|
||||||
@@ -55,6 +59,7 @@ ROOT = Path(__file__).resolve().parent.parent
|
|||||||
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/, generated/, toolchain/ and refs/ are junctions in a worktree
|
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/, generated/, toolchain/ and refs/ are junctions in a worktree
|
||||||
DL = MAIN_ROOT / 'toolchain' / 'release-dl'
|
DL = MAIN_ROOT / 'toolchain' / 'release-dl'
|
||||||
WORK = MAIN_ROOT / 'toolchain' / 'release-work' # unpacked downloads
|
WORK = MAIN_ROOT / 'toolchain' / 'release-work' # unpacked downloads
|
||||||
|
PGO_DIR = MAIN_ROOT / 'build' / 'pgo' # pgo-<version>.profdata + .json (tools/pgo.py merge)
|
||||||
|
|
||||||
MSYS2 = 'https://repo.msys2.org/mingw/ucrt64/'
|
MSYS2 = 'https://repo.msys2.org/mingw/ucrt64/'
|
||||||
SDL_VERSION = '3.4.18'
|
SDL_VERSION = '3.4.18'
|
||||||
@@ -378,17 +383,39 @@ def lifted_version():
|
|||||||
'(tools/extract.py, recomp/gen.py; docs/building.md)')
|
'(tools/extract.py, recomp/gen.py; docs/building.md)')
|
||||||
|
|
||||||
|
|
||||||
def runtime_lib(out, llvm):
|
def profiles():
|
||||||
|
"""{version tag: .profdata} of build/pgo (tools/pgo.py merge). A profile trained on another recomp/ than the one
|
||||||
|
shipped still works but misses the functions whose generated C changed since: warn, so the next release retrains."""
|
||||||
|
tree = subprocess.run(['git', '-C', str(ROOT), 'rev-parse', 'HEAD:recomp'], capture_output=True, text=True).stdout.strip()
|
||||||
|
dirty = subprocess.run(['git', '-C', str(ROOT), 'status', '--porcelain', '--', 'recomp'], capture_output=True,
|
||||||
|
text=True).stdout.strip()
|
||||||
|
found = {}
|
||||||
|
for p in sorted(PGO_DIR.glob('pgo-*.profdata')):
|
||||||
|
rec = p.with_suffix('.json')
|
||||||
|
record = json.loads(rec.read_text(encoding='utf-8')) if rec.is_file() else {}
|
||||||
|
if record.get('recomp_tree') != tree or dirty:
|
||||||
|
print(f'make_release: warning: {p.name} was trained on recomp tree {record.get("recomp_tree", "unknown")[:12]}, '
|
||||||
|
f'this is {tree[:12]}{" with local changes" if dirty else ""}: functions whose generated C changed '
|
||||||
|
'build without a profile; retrain (docs/building.md, "Profile-guided build")', file=sys.stderr)
|
||||||
|
found[p.stem.removeprefix('pgo-')] = p
|
||||||
|
if not found:
|
||||||
|
print(f'make_release: no execution profile in {PGO_DIR}: installs build the game at -O1', file=sys.stderr)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def runtime_lib(out, llvm, pgo):
|
||||||
"""Build libtotkrt.a (x86-64-v3) and libtotkrt-v2.a (x86-64-v2) with the folder's llvm/, and copy them, the headers
|
"""Build libtotkrt.a (x86-64-v3) and libtotkrt-v2.a (x86-64-v2) with the folder's llvm/, and copy them, the headers
|
||||||
and the install-side sources into runtime/."""
|
and the install-side sources into runtime/. pgo: profiles() of the release."""
|
||||||
version = lifted_version()
|
version = lifted_version()
|
||||||
gen = MAIN_ROOT / 'generated' / version
|
gen = MAIN_ROOT / 'generated' / version
|
||||||
rt = out / 'runtime'
|
rt = out / 'runtime'
|
||||||
|
# the runtime is the same for every update: any update's profile covers it (the newest update's, as players run it)
|
||||||
|
newest = sorted(pgo.items())[-1:]
|
||||||
for cpu, name in (('v3', 'libtotkrt.a'), ('v2', 'libtotkrt-v2.a')):
|
for cpu, name in (('v3', 'libtotkrt.a'), ('v2', 'libtotkrt-v2.a')):
|
||||||
build = MAIN_ROOT / 'build' / 'release-check' / f'rtlib-{cpu}'
|
build = MAIN_ROOT / 'build' / 'release-check' / f'rtlib-{cpu}'
|
||||||
run([sys.executable, ROOT / 'tools' / 'build_game.py', '--make-runtime-lib', '--toolchain', llvm, '--cpu', cpu,
|
run([sys.executable, ROOT / 'tools' / 'build_game.py', '--make-runtime-lib', '--toolchain', llvm, '--cpu', cpu,
|
||||||
'--version', version, '--deps', out / 'runtime' / 'deps', '--ninja', out / 'bin' / 'ninja.exe',
|
'--version', version, '--deps', out / 'runtime' / 'deps', '--ninja', out / 'bin' / 'ninja.exe',
|
||||||
'--gen', gen, '--out', build],
|
'--gen', gen, '--out', build, *(['--pgo', newest[0][1]] if newest else [])],
|
||||||
env=clean_env(llvm / 'bin', out / 'bin'))
|
env=clean_env(llvm / 'bin', out / 'bin'))
|
||||||
shutil.copy2(build / 'libtotkrt.a', rt / name)
|
shutil.copy2(build / 'libtotkrt.a', rt / name)
|
||||||
for p in sorted((ROOT / 'runtime').rglob('*')):
|
for p in sorted((ROOT / 'runtime').rglob('*')):
|
||||||
@@ -399,7 +426,7 @@ def runtime_lib(out, llvm):
|
|||||||
shutil.copy2(ROOT / 'runtime' / name, rt / name)
|
shutil.copy2(ROOT / 'runtime' / name, rt / name)
|
||||||
|
|
||||||
|
|
||||||
def build_sources(out):
|
def build_sources(out, pgo):
|
||||||
copy_tree(ROOT / 'recomp', out / 'recomp', lambda r: not r.endswith(('.py', '.txt')) or '__pycache__' in r)
|
copy_tree(ROOT / 'recomp', out / 'recomp', lambda r: not r.endswith(('.py', '.txt')) or '__pycache__' in r)
|
||||||
for f in TOOLS_FILES:
|
for f in TOOLS_FILES:
|
||||||
(out / 'tools').mkdir(exist_ok=True)
|
(out / 'tools').mkdir(exist_ok=True)
|
||||||
@@ -409,6 +436,10 @@ def build_sources(out):
|
|||||||
(out / 'data').mkdir(exist_ok=True)
|
(out / 'data').mkdir(exist_ok=True)
|
||||||
text = shader_list.history(MAIN_ROOT / 'refs' / 'MacNativeGames', MNG_COMMIT)
|
text = shader_list.history(MAIN_ROOT / 'refs' / 'MacNativeGames', MNG_COMMIT)
|
||||||
(out / 'data' / 'shader_list.tsv').write_text(text, encoding='utf-8', newline='\n')
|
(out / 'data' / 'shader_list.tsv').write_text(text, encoding='utf-8', newline='\n')
|
||||||
|
for p in pgo.values():
|
||||||
|
for f in (p, p.with_suffix('.json')):
|
||||||
|
if f.is_file():
|
||||||
|
shutil.copy2(f, out / 'data' / f.name)
|
||||||
copy_tree(ROOT / 'installer', out / 'installer', lambda r: '__pycache__' in r or r.startswith('launcher/'))
|
copy_tree(ROOT / 'installer', out / 'installer', lambda r: '__pycache__' in r or r.startswith('launcher/'))
|
||||||
shutil.copy2(ROOT / 'totk.ini.example', out / 'totk.ini.example')
|
shutil.copy2(ROOT / 'totk.ini.example', out / 'totk.ini.example')
|
||||||
shutil.copy2(ROOT / 'THIRD_PARTY_NOTICES.md', out / 'THIRD_PARTY_NOTICES.md')
|
shutil.copy2(ROOT / 'THIRD_PARTY_NOTICES.md', out / 'THIRD_PARTY_NOTICES.md')
|
||||||
@@ -527,12 +558,14 @@ def prove(out, cpu='v3'):
|
|||||||
ext = MAIN_ROOT / 'build' / 'extracted' / version
|
ext = MAIN_ROOT / 'build' / 'extracted' / version
|
||||||
chk = MAIN_ROOT / 'build' / 'release-check'
|
chk = MAIN_ROOT / 'build' / 'release-check'
|
||||||
env = clean_env(out / 'python', out / 'llvm' / 'bin', out / 'bin')
|
env = clean_env(out / 'python', out / 'llvm' / 'bin', out / 'bin')
|
||||||
|
pgo = out / 'data' / f'pgo-{version}.profdata'
|
||||||
t0 = time.time()
|
t0 = time.time()
|
||||||
run([out / 'python' / 'python.exe', out / 'tools' / 'build_game.py', '--toolchain', out / 'llvm', '--cpu', cpu,
|
run([out / 'python' / 'python.exe', out / 'tools' / 'build_game.py', '--toolchain', out / 'llvm', '--cpu', cpu,
|
||||||
'--deps', out / 'runtime' / 'deps', '--runtime', out / 'runtime',
|
'--deps', out / 'runtime' / 'deps', '--runtime', out / 'runtime',
|
||||||
'--runtime-lib', out / 'runtime' / ('libtotkrt-v2.a' if cpu == 'v2' else 'libtotkrt.a'),
|
'--runtime-lib', out / 'runtime' / ('libtotkrt-v2.a' if cpu == 'v2' else 'libtotkrt.a'),
|
||||||
'--ninja', out / 'bin' / 'ninja.exe', '--version', version, '--gen', gen, '--out', chk, '--exefs', ext / 'exefs',
|
'--ninja', out / 'bin' / 'ninja.exe', '--version', version, '--gen', gen, '--out', chk, '--exefs', ext / 'exefs',
|
||||||
'--romfs', ext / 'romfs', '--profile', chk / 'profile'], env=env)
|
'--romfs', ext / 'romfs', '--profile', chk / 'profile', *(['--pgo', pgo] if pgo.is_file() else [])],
|
||||||
|
env=env)
|
||||||
print(f'prove: build {time.time() - t0:.0f} s')
|
print(f'prove: build {time.time() - t0:.0f} s')
|
||||||
for dll in ('SDL3.dll', 'libopus-0.dll'):
|
for dll in ('SDL3.dll', 'libopus-0.dll'):
|
||||||
shutil.copy2(out / 'bin' / dll, chk / dll)
|
shutil.copy2(out / 'bin' / dll, chk / dll)
|
||||||
@@ -568,8 +601,9 @@ def main():
|
|||||||
llvm = build_llvm(out)
|
llvm = build_llvm(out)
|
||||||
build_bin(out)
|
build_bin(out)
|
||||||
build_deps(out)
|
build_deps(out)
|
||||||
runtime_lib(out, llvm)
|
pgo = profiles()
|
||||||
build_sources(out)
|
runtime_lib(out, llvm, pgo)
|
||||||
|
build_sources(out, pgo)
|
||||||
build_launcher(out, llvm)
|
build_launcher(out, llvm)
|
||||||
collect_licenses(out)
|
collect_licenses(out)
|
||||||
check_folder(out)
|
check_folder(out)
|
||||||
|
|||||||
@@ -0,0 +1,84 @@
|
|||||||
|
"""Merge the raw profiles of an instrumented build into the .profdata that build_game.py --pgo FILE reads.
|
||||||
|
|
||||||
|
python tools/pgo.py merge build/v143-pgogen --version v143 [--out build/pgo/pgo-v143.profdata]
|
||||||
|
|
||||||
|
An instrumented build (build_game.py --pgo gen) writes <build>/pgo/default_<id>.profraw while totk.exe runs (every
|
||||||
|
minute and at exit); successive runs of the same exe add to that file. llvm-profdata comes from the llvm-mingw toolchain
|
||||||
|
the instrumented build used: its raw format is tied to the compiler version.
|
||||||
|
|
||||||
|
Beside the profile goes <out>.json, the training record: the game version and the git tree of recomp/ at the merge.
|
||||||
|
The profile matches the generated C of that generator; tools/make_release.py ships it only while recomp/ is unchanged,
|
||||||
|
since llvm-profdata matches functions by name and control-flow hash and a regenerated function silently loses its
|
||||||
|
counts. Merge right after the training runs, from the commit they were built at.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import datetime
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
EXE = '.exe' if sys.platform == 'win32' else ''
|
||||||
|
DEFAULT_DIR = ROOT / 'build' / 'pgo' # tools/make_release.py takes pgo-<version>.profdata from here
|
||||||
|
|
||||||
|
|
||||||
|
def llvm_profdata(toolchain):
|
||||||
|
if toolchain:
|
||||||
|
tool = toolchain / 'bin' / f'llvm-profdata{EXE}'
|
||||||
|
else:
|
||||||
|
found = sorted((ROOT / 'toolchain').glob(f'llvm-mingw-*/llvm-mingw-*/bin/llvm-profdata{EXE}'))
|
||||||
|
tool = found[-1] if found else Path(f'llvm-profdata{EXE}')
|
||||||
|
if not tool.is_file():
|
||||||
|
sys.exit(f'{tool} not found: pass --toolchain')
|
||||||
|
return tool
|
||||||
|
|
||||||
|
|
||||||
|
def recomp_tree():
|
||||||
|
"""The git tree id of recomp/ at HEAD; None when recomp/ has uncommitted changes."""
|
||||||
|
dirty = subprocess.run(['git', '-C', str(ROOT), 'status', '--porcelain', '--', 'recomp'], capture_output=True,
|
||||||
|
text=True).stdout.strip()
|
||||||
|
if dirty:
|
||||||
|
return None
|
||||||
|
return subprocess.run(['git', '-C', str(ROOT), 'rev-parse', 'HEAD:recomp'], capture_output=True,
|
||||||
|
text=True).stdout.strip() or None
|
||||||
|
|
||||||
|
|
||||||
|
def merge(build, out, version, toolchain):
|
||||||
|
raw = sorted((build / 'pgo').glob('*.profraw'))
|
||||||
|
if not raw:
|
||||||
|
sys.exit(f'no .profraw in {build / "pgo"}: run the instrumented totk.exe first')
|
||||||
|
tree = recomp_tree()
|
||||||
|
if not tree:
|
||||||
|
sys.exit('recomp/ has uncommitted changes: the record could not name the generator the profile matches')
|
||||||
|
out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp = out.with_suffix('.profdata.tmp')
|
||||||
|
cmd = [str(llvm_profdata(toolchain)), 'merge', '--failure-mode=all', '-o', str(tmp), *map(str, raw)]
|
||||||
|
p = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if p.returncode:
|
||||||
|
tmp.unlink(missing_ok=True)
|
||||||
|
sys.exit(f'llvm-profdata merge failed: {(p.stdout + p.stderr).strip()[-2000:]}')
|
||||||
|
tmp.replace(out)
|
||||||
|
record = {'version': version, 'recomp_tree': tree, 'merged': datetime.date.today().isoformat(),
|
||||||
|
'raw_files': len(raw)}
|
||||||
|
out.with_suffix('.json').write_text(json.dumps(record, indent=1) + '\n', encoding='utf-8')
|
||||||
|
size = sum(f.stat().st_size for f in raw)
|
||||||
|
print(f'{out}: {len(raw)} raw files ({size / 2**20:.1f} MiB) -> {out.stat().st_size / 2**20:.1f} MiB; '
|
||||||
|
f'recomp tree {tree[:12]}')
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
sub = ap.add_subparsers(dest='cmd', required=True)
|
||||||
|
m = sub.add_parser('merge', help='merge <build>/pgo/*.profraw into one .profdata and its training record')
|
||||||
|
m.add_argument('build', type=Path, help='the instrumented build directory')
|
||||||
|
m.add_argument('--version', required=True, help='game version the build was made for (v121, v143)')
|
||||||
|
m.add_argument('--out', type=Path, help=f'default {DEFAULT_DIR.relative_to(ROOT).as_posix()}/pgo-<version>.profdata')
|
||||||
|
m.add_argument('--toolchain', type=Path, help='llvm-mingw directory (default: the newest under toolchain/)')
|
||||||
|
a = ap.parse_args()
|
||||||
|
out = (a.out or DEFAULT_DIR / f'pgo-{a.version}.profdata').resolve()
|
||||||
|
merge(a.build.resolve(), out, a.version, a.toolchain)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Reference in new issue
Block a user