mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-22 13:05:57 +00:00
Keep the Linux setup's temp files and Mesa's shader cache with the game, and stop the setup with one message on a glibc older than 2.34.
A Steam Deck tester had 1.2 GB free in /home, where the setup put its 1.3 GB of generated C and objects (~/.cache/totk-setup) and RADV its shader cache (up to 1 GB, about 35 MB a minute at the title screen), while the game goes to the SD card. The setup's temp is now .setup-temp in the install folder (--temp changes it; the empty-folder check leaves it out), and totk points MESA_SHADER_CACHE_DIR at mesa-cache beside itself unless it is set. A scan of every ELF file in the release folder showed the bundled LLVM and the cryptography wheel need glibc 2.34, not the 2.31 the docs gave: make_release_linux.py now checks that floor (check_glibc), the setup's check stops below it (check.glibc_old) instead of failing on a compiler that cannot start, and the docs give both floors. The Linux notes also record the testers' results, that the Deck has the host-import extension too, and its software ASTC decoding.
This commit is contained in:
1 parent
48d1a6d3a0
commit
b7baf43332
9 files changed
+101
-21
No files matched your search
+8
-3
@@ -122,8 +122,9 @@ folder. The folder holds code and data derived from your game: for your own mach
|
||||
## Linux (in progress)
|
||||
|
||||
The runtime builds and runs on x86-64 Linux with a window, input and menu on SDL3 (`*_sdl.c`) in place of the Win32
|
||||
files (`*_win.c`); `runtime/platform/linux.c` implements `platform.h`. Status: the selftest and the probes pass, and
|
||||
1.2.1 plays at 60 fps on RADV (RX 7800 XT); 1.4.3 and long play sessions are not verified yet.
|
||||
files (`*_win.c`); `runtime/platform/linux.c` implements `platform.h`. Status: the selftest and the probes pass; 1.2.1
|
||||
plays at 60 fps on RADV (RX 7800 XT) and NVIDIA (RTX 3060); 1.4.3 builds, probes and plays on RADV (RX 6600, and the
|
||||
Vega 7 of a Ryzen 5 4600G at about 34 fps). Long play sessions and the Steam Deck itself are not verified yet.
|
||||
|
||||
Requirements:
|
||||
|
||||
@@ -154,7 +155,11 @@ re-protecting imported memory invalidates the driver's userptr buffer on every s
|
||||
amdgpu), and importing pins the pages, which drops their write protection. Each CPU-visible pool therefore gets a
|
||||
host-visible copy (shadow storage, `runtime/vk_memory.c`): the pages the CPU wrote go in before each submit, and the GPU's
|
||||
writes come back at its fence. In play that costs about 0.2 ms a frame and doubles the memory of those pools (about
|
||||
1.1 GB). `TOTK_SHADOW=1` turns it on under Windows too.
|
||||
1.1 GB). `TOTK_SHADOW=1` turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as
|
||||
well: the shadow copies are chosen for the measured cost above, not for a missing extension.
|
||||
|
||||
The Deck's GPU (Van Gogh) has no native ASTC, so its textures go through the compute decoder (`runtime/vk_astc.c`,
|
||||
`TOTK_ASTC`) as on desktop AMD GPUs without it; frame times measured there include it.
|
||||
|
||||
## Checks
|
||||
|
||||
|
||||
+14
-7
@@ -125,15 +125,22 @@ runtime/ libtotkrt.a, headers, the install-side sources, totk.png;
|
||||
bin/ totk_xlate, hactool, ninja, libSDL3.so.0
|
||||
```
|
||||
|
||||
- Everything that runs on the player's machine is built against the sniper sysroot (glibc 2.31): the game, hostcheck,
|
||||
hactool and the translator need no newer glibc, so they run on SteamOS and current distributions. The game links
|
||||
Opus, libvpx, zstd and libstdc++ statically and needs only glibc, the system's Vulkan loader and the SDL3 beside it.
|
||||
- The setup's temp folder is `$XDG_CACHE_HOME/totk-setup` (default `~/.cache/totk-setup`): the AppImage is mounted
|
||||
read-only.
|
||||
- What we compile is built against the sniper sysroot (glibc 2.31): the game, hostcheck, hactool and the translator
|
||||
need no newer glibc. The game links Opus, libvpx, zstd and libstdc++ statically and needs only glibc, the system's
|
||||
Vulkan loader and the SDL3 beside it. The setup itself needs glibc 2.34 (Ubuntu 22.04, Debian 12, Fedora 35,
|
||||
SteamOS 3): the official LLVM binaries and the `cryptography` wheel are built against it. `make_release_linux.py`
|
||||
checks every ELF file in the folder against that floor (`check_glibc`), and the setup's check step stops with one
|
||||
message on an older system.
|
||||
- The setup's temp folder is `.setup-temp` inside the install folder (`--temp` changes it): the AppImage is mounted
|
||||
read-only, and a Steam Deck's `/home` is often too full for the generated C while the game goes to the SD card.
|
||||
- The game keeps Mesa's shader cache in `mesa-cache` beside `totk` unless `MESA_SHADER_CACHE_DIR` is set, for the
|
||||
same reason: RADV fills its default folder in `~/.cache` up to 1 GB.
|
||||
- The window follows `LANG` for English or Russian, cancels through SIGTERM to the CLI's process group (the CLI handles
|
||||
it as Ctrl+Break on Windows) and opens the finished folder with `xdg-open`.
|
||||
- Requirements beyond Windows': Linux 6.7 or later (the write watch, `docs/building.md`), and enough space in the cache
|
||||
folder's file system for the generated C (about 1.3 GB).
|
||||
- Requirements beyond Windows': Linux 6.7 or later (the write watch, `docs/building.md`) and glibc 2.34.
|
||||
- Building the release needs one game version extracted and lifted (`tools/extract.py`, `recomp/gen.py`, either
|
||||
version): the runtime library includes the generated `layout.h`, whose shared part (`MOD_COUNT`, `MOD_MAIN_BASE`)
|
||||
is the same for both. `tools/make_release.py` takes `generated/v121`, else `generated/v143`.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
+9
-3
@@ -28,6 +28,11 @@ class Parser(argparse.ArgumentParser):
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
# The temp folder inside the install folder when the layout names none (the Linux release); the check step does not
|
||||
# count it when it requires an empty install folder
|
||||
TEMP_IN_DEST = '.setup-temp'
|
||||
|
||||
|
||||
def temp_folder(base, dest):
|
||||
"""One temp folder per install folder, so two installs never share generated files."""
|
||||
tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8]
|
||||
@@ -46,8 +51,8 @@ def main(argv=None):
|
||||
ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)')
|
||||
ap.add_argument('--dest', type=Path, required=True, help='install folder')
|
||||
ap.add_argument('--root', type=Path, help='release folder or repository (default: the folder above installer/)')
|
||||
ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release, build/install-temp in a '
|
||||
'repository)')
|
||||
ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release on Windows, .setup-temp in '
|
||||
'the install folder on Linux, build/install-temp in a repository)')
|
||||
ap.add_argument('--verify', action='store_true', help='hash every kept file, also those this release made')
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
@@ -58,7 +63,8 @@ def main(argv=None):
|
||||
signal.signal(getattr(signal, name), cancel)
|
||||
layout = layouts.find(args.root)
|
||||
dest = args.dest.resolve()
|
||||
temp = temp_folder((args.temp or layout.temp).resolve(), dest)
|
||||
base = args.temp or layout.temp
|
||||
temp = temp_folder(base.resolve(), dest) if base else dest / TEMP_IN_DEST
|
||||
log = temp / 'logs' / time.strftime('install-%Y%m%d-%H%M%S.log')
|
||||
rep = Reporter(log_path=log)
|
||||
t0 = time.monotonic()
|
||||
|
||||
+3
-7
@@ -90,12 +90,6 @@ class Layout:
|
||||
return [str(p) for p in need if not p.exists()]
|
||||
|
||||
|
||||
def cache_dir():
|
||||
"""$XDG_CACHE_HOME, else ~/.cache: the Linux release runs from a read-only AppImage mount, so its temp goes here."""
|
||||
xdg = os.environ.get('XDG_CACHE_HOME')
|
||||
return Path(xdg) if xdg and Path(xdg).is_absolute() else Path.home() / '.cache'
|
||||
|
||||
|
||||
def release_python(root):
|
||||
return root / 'python' / ('python.exe' if WIN else 'bin/python3')
|
||||
|
||||
@@ -117,7 +111,9 @@ def release_layout(root):
|
||||
xlate=root / 'bin' / ('totk_xlate' + EXE),
|
||||
dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll') if WIN else (root / 'bin' / 'libSDL3.so.0',),
|
||||
shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example',
|
||||
temp=root / 'temp' if WIN else cache_dir() / 'totk-setup',
|
||||
# Linux: None, the install folder's .setup-temp (installer/cli.py): the AppImage is mounted read-only, and on
|
||||
# a Steam Deck /home is often too full for the generated C while the install folder is on the SD card
|
||||
temp=root / 'temp' if WIN else None,
|
||||
sysroot=None if WIN else root / 'llvm' / 'sysroot')
|
||||
|
||||
|
||||
|
||||
+21
-1
@@ -102,6 +102,21 @@ def kill_tree(proc):
|
||||
proc.kill()
|
||||
|
||||
|
||||
def linux_glibc():
|
||||
"""(major, minor) of the C library this Linux process runs on, None elsewhere or when it is not glibc."""
|
||||
try:
|
||||
name = os.confstr('CS_GNU_LIBC_VERSION') if sys.platform.startswith('linux') else None
|
||||
except (ValueError, OSError):
|
||||
return None
|
||||
m = re.match(r'glibc (\d+)\.(\d+)', name or '')
|
||||
return (int(m.group(1)), int(m.group(2))) if m else None
|
||||
|
||||
|
||||
# The bundled clang, lld and the cryptography wheel are built against glibc 2.34 (tools/make_release_linux.py
|
||||
# GLIBC_FLOOR); on an older one they cannot start
|
||||
GLIBC_NEEDED = (2, 34)
|
||||
|
||||
|
||||
def linux_kernel():
|
||||
"""(major, minor) of the running Linux kernel, None elsewhere."""
|
||||
if not sys.platform.startswith('linux'):
|
||||
@@ -214,7 +229,8 @@ class Check(Step):
|
||||
raise InstallError('check.release_incomplete', paths=missing[:5])
|
||||
if dest.exists() and not dest.is_dir():
|
||||
raise InstallError('check.dest_not_folder', path=str(dest))
|
||||
if ctx.mode == 'install' and dest.is_dir() and any(dest.iterdir()):
|
||||
own = (ctx.temp, *ctx.temp.parents) # a temp folder inside the install folder (installer/cli.py) is ours
|
||||
if ctx.mode == 'install' and dest.is_dir() and any(p not in own for p in dest.iterdir()):
|
||||
raise InstallError('check.dest_not_empty', path=str(dest))
|
||||
lang = ctx.options.lang or ctx.state.get('lang')
|
||||
if not lang:
|
||||
@@ -242,6 +258,10 @@ class Check(Step):
|
||||
|
||||
def host(self, ctx):
|
||||
layout, work = ctx.layout, ctx.temp / 'check'
|
||||
glibc = linux_glibc()
|
||||
if glibc and glibc < GLIBC_NEEDED and not layout.dev:
|
||||
raise InstallError('check.glibc_old', version='.'.join(map(str, glibc)),
|
||||
need='.'.join(map(str, GLIBC_NEEDED)))
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
exe = work / ('hostcheck' + EXE)
|
||||
command = [str(layout.clang), '-O1', str(HOSTCHECK), '-o', str(exe)]
|
||||
|
||||
@@ -41,6 +41,7 @@
|
||||
"check.lang_missing": "Choose the game language.",
|
||||
"check.lang_unknown": "Unknown language '{lang}'. Known: {known}.",
|
||||
"check.toolchain_broken": "The bundled compiler does not work: {detail}",
|
||||
"check.glibc_old": "This system's C library is glibc {version}; the setup's compiler needs {need} or later (Ubuntu 22.04, Debian 12, Fedora 35, SteamOS 3 and newer have it). Updating the system fixes this.",
|
||||
"check.host_probe_failed": "The computer check did not run (exit code {code}).",
|
||||
"check.cpu_no_avx2": "This processor lacks AVX2 (x86-64-v3), which the game needs. Intel Haswell (2013) or AMD Excavator / Zen and newer have it.",
|
||||
"check.vulkan_missing": "No Vulkan driver is installed. Install the latest driver for your graphics card.",
|
||||
|
||||
@@ -41,6 +41,7 @@
|
||||
"check.lang_missing": "Выберите язык игры.",
|
||||
"check.lang_unknown": "Неизвестный язык «{lang}». Доступные: {known}.",
|
||||
"check.toolchain_broken": "Встроенный компилятор не работает: {detail}",
|
||||
"check.glibc_old": "Библиотека C этой системы — glibc {version}; компилятору установщика нужна {need} или новее (она есть в Ubuntu 22.04, Debian 12, Fedora 35, SteamOS 3 и новее). Обновление системы это исправит.",
|
||||
"check.host_probe_failed": "Проверка компьютера не выполнилась (код выхода {code}).",
|
||||
"check.cpu_no_avx2": "Этому процессору не хватает AVX2 (x86-64-v3), который нужен игре. AVX2 есть у Intel Haswell (2013) и новее, у AMD Excavator и Zen и новее.",
|
||||
"check.vulkan_missing": "Драйвер Vulkan не установлен. Установите свежий драйвер видеокарты.",
|
||||
|
||||
@@ -41,7 +41,21 @@ static int layer_present(const char *name) {
|
||||
return found;
|
||||
}
|
||||
|
||||
// Mesa's drivers (RADV on Linux and the Steam Deck) keep a shader cache of their own beside our pipeline cache, by
|
||||
// default in ~/.cache and up to 1 GB: about 35 MB in a minute of the title screen. A Deck's /home is small and the game
|
||||
// usually sits on the SD card, so unless MESA_SHADER_CACHE_DIR names a folder, it goes to mesa-cache beside the exe.
|
||||
static void mesa_cache_beside_exe(void) {
|
||||
char dir[PLAT_PATH_MAX * 2];
|
||||
if (plat_getenv("MESA_SHADER_CACHE_DIR", dir, sizeof dir)) return;
|
||||
int len = plat_exe_path(dir, sizeof dir - 16);
|
||||
char *cut = len ? plat_last_sep(dir) : NULL;
|
||||
if (!cut) return;
|
||||
strcpy(cut + 1, "mesa-cache");
|
||||
plat_setenv("MESA_SHADER_CACHE_DIR", dir);
|
||||
}
|
||||
|
||||
static int create_instance(void) {
|
||||
mesa_cache_beside_exe();
|
||||
const char *ext[8];
|
||||
uint32_t n_ext = (uint32_t)gfx_window_vk_extensions(ext, 7);
|
||||
const char *layers[] = {"VK_LAYER_KHRONOS_validation"};
|
||||
|
||||
@@ -404,6 +404,35 @@ def write_appimage(out, path):
|
||||
|
||||
# ---- checks -------------------------------------------------------------------------------------------------------------
|
||||
|
||||
# The newest glibc any program or library in the folder may need: the official LLVM binaries and the cryptography
|
||||
# wheel are built against 2.34 (Ubuntu 22.04, Debian 12, SteamOS 3 have it); the game itself links the sysroot's 2.31.
|
||||
GLIBC_FLOOR = (2, 34)
|
||||
|
||||
|
||||
def check_glibc(out):
|
||||
"""Every ELF file in the folder (llvm/sysroot aside: glibc itself, read by the link) needs glibc GLIBC_FLOOR or
|
||||
older; prints which files set the floor."""
|
||||
newest, by = (0, 0), []
|
||||
for p in sorted(out.rglob('*')):
|
||||
if not p.is_file() or p.is_symlink() or out / 'llvm' / 'sysroot' in p.parents:
|
||||
continue
|
||||
with open(p, 'rb') as f:
|
||||
if f.read(4) != b'\x7fELF':
|
||||
continue
|
||||
text = subprocess.run(['readelf', '-V', str(p)], capture_output=True, text=True).stdout
|
||||
need = max((tuple(map(int, v.split('.'))) for v in re.findall(r'GLIBC_(\d+\.\d+(?:\.\d+)?)', text)),
|
||||
default=(0, 0))
|
||||
if need > newest:
|
||||
newest, by = need, []
|
||||
if need == newest:
|
||||
by.append(str(p.relative_to(out)))
|
||||
floor = '.'.join(map(str, GLIBC_FLOOR))
|
||||
if newest > GLIBC_FLOOR:
|
||||
sys.exit(f'glibc check: {", ".join(by[:8])} need glibc {".".join(map(str, newest))}, above {floor}')
|
||||
print(f'glibc check: the folder needs glibc {".".join(map(str, newest))} (floor {floor}), set by '
|
||||
f'{", ".join(by[:6])}{" and others" if len(by) > 6 else ""}')
|
||||
|
||||
|
||||
def check_bundle(out):
|
||||
"""The folder's own python runs the installer without game files and the window's selftest."""
|
||||
import json
|
||||
@@ -487,6 +516,7 @@ def main():
|
||||
collect_licenses(out, sysroot)
|
||||
build_entry(out)
|
||||
mr.check_folder(out)
|
||||
check_glibc(out)
|
||||
check_bundle(out)
|
||||
mr.sizes(out)
|
||||
if args.appimage:
|
||||
|
||||
Reference in new issue
Block a user