diff --git a/docs/building.md b/docs/building.md index ffc8439e..780b6f1b 100644 --- a/docs/building.md +++ b/docs/building.md @@ -122,8 +122,9 @@ folder. The folder holds code and data derived from your game: for your own mach ## Linux (in progress) The runtime builds and runs on x86-64 Linux with a window, input and menu on SDL3 (`*_sdl.c`) in place of the Win32 -files (`*_win.c`); `runtime/platform/linux.c` implements `platform.h`. Status: the selftest and the probes pass, and -1.2.1 plays at 60 fps on RADV (RX 7800 XT); 1.4.3 and long play sessions are not verified yet. +files (`*_win.c`); `runtime/platform/linux.c` implements `platform.h`. Status: the selftest and the probes pass; 1.2.1 +plays at 60 fps on RADV (RX 7800 XT) and NVIDIA (RTX 3060); 1.4.3 builds, probes and plays on RADV (RX 6600, and the +Vega 7 of a Ryzen 5 4600G at about 34 fps). Long play sessions and the Steam Deck itself are not verified yet. Requirements: @@ -154,7 +155,11 @@ re-protecting imported memory invalidates the driver's userptr buffer on every s amdgpu), and importing pins the pages, which drops their write protection. Each CPU-visible pool therefore gets a host-visible copy (shadow storage, `runtime/vk_memory.c`): the pages the CPU wrote go in before each submit, and the GPU's writes come back at its fence. In play that costs about 0.2 ms a frame and doubles the memory of those pools (about -1.1 GB). `TOTK_SHADOW=1` turns it on under Windows too. +1.1 GB). `TOTK_SHADOW=1` turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as +well: the shadow copies are chosen for the measured cost above, not for a missing extension. + +The Deck's GPU (Van Gogh) has no native ASTC, so its textures go through the compute decoder (`runtime/vk_astc.c`, +`TOTK_ASTC`) as on desktop AMD GPUs without it; frame times measured there include it. ## Checks diff --git a/docs/installer.md b/docs/installer.md index a991f935..0d83cd85 100644 --- a/docs/installer.md +++ b/docs/installer.md @@ -125,15 +125,22 @@ runtime/ libtotkrt.a, headers, the install-side sources, totk.png; bin/ totk_xlate, hactool, ninja, libSDL3.so.0 ``` -- Everything that runs on the player's machine is built against the sniper sysroot (glibc 2.31): the game, hostcheck, - hactool and the translator need no newer glibc, so they run on SteamOS and current distributions. The game links - Opus, libvpx, zstd and libstdc++ statically and needs only glibc, the system's Vulkan loader and the SDL3 beside it. -- The setup's temp folder is `$XDG_CACHE_HOME/totk-setup` (default `~/.cache/totk-setup`): the AppImage is mounted - read-only. +- What we compile is built against the sniper sysroot (glibc 2.31): the game, hostcheck, hactool and the translator + need no newer glibc. The game links Opus, libvpx, zstd and libstdc++ statically and needs only glibc, the system's + Vulkan loader and the SDL3 beside it. The setup itself needs glibc 2.34 (Ubuntu 22.04, Debian 12, Fedora 35, + SteamOS 3): the official LLVM binaries and the `cryptography` wheel are built against it. `make_release_linux.py` + checks every ELF file in the folder against that floor (`check_glibc`), and the setup's check step stops with one + message on an older system. +- The setup's temp folder is `.setup-temp` inside the install folder (`--temp` changes it): the AppImage is mounted + read-only, and a Steam Deck's `/home` is often too full for the generated C while the game goes to the SD card. +- The game keeps Mesa's shader cache in `mesa-cache` beside `totk` unless `MESA_SHADER_CACHE_DIR` is set, for the + same reason: RADV fills its default folder in `~/.cache` up to 1 GB. - The window follows `LANG` for English or Russian, cancels through SIGTERM to the CLI's process group (the CLI handles it as Ctrl+Break on Windows) and opens the finished folder with `xdg-open`. -- Requirements beyond Windows': Linux 6.7 or later (the write watch, `docs/building.md`), and enough space in the cache - folder's file system for the generated C (about 1.3 GB). +- Requirements beyond Windows': Linux 6.7 or later (the write watch, `docs/building.md`) and glibc 2.34. +- Building the release needs one game version extracted and lifted (`tools/extract.py`, `recomp/gen.py`, either + version): the runtime library includes the generated `layout.h`, whose shared part (`MOD_COUNT`, `MOD_MAIN_BASE`) + is the same for both. `tools/make_release.py` takes `generated/v121`, else `generated/v143`. ## Constraints diff --git a/installer/cli.py b/installer/cli.py index a754c427..b6d4238a 100644 --- a/installer/cli.py +++ b/installer/cli.py @@ -28,6 +28,11 @@ class Parser(argparse.ArgumentParser): sys.exit(2) +# The temp folder inside the install folder when the layout names none (the Linux release); the check step does not +# count it when it requires an empty install folder +TEMP_IN_DEST = '.setup-temp' + + def temp_folder(base, dest): """One temp folder per install folder, so two installs never share generated files.""" tag = hashlib.sha256(str(dest).lower().encode()).hexdigest()[:8] @@ -46,8 +51,8 @@ def main(argv=None): ap.add_argument('--lang', help='game language, a Switch language code (en-US, en-GB, ja, de, ru, ...)') ap.add_argument('--dest', type=Path, required=True, help='install folder') ap.add_argument('--root', type=Path, help='release folder or repository (default: the folder above installer/)') - ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release, build/install-temp in a ' - 'repository)') + ap.add_argument('--temp', type=Path, help='temp folder (default: temp/ of the release on Windows, .setup-temp in ' + 'the install folder on Linux, build/install-temp in a repository)') ap.add_argument('--verify', action='store_true', help='hash every kept file, also those this release made') args = ap.parse_args(argv) @@ -58,7 +63,8 @@ def main(argv=None): signal.signal(getattr(signal, name), cancel) layout = layouts.find(args.root) dest = args.dest.resolve() - temp = temp_folder((args.temp or layout.temp).resolve(), dest) + base = args.temp or layout.temp + temp = temp_folder(base.resolve(), dest) if base else dest / TEMP_IN_DEST log = temp / 'logs' / time.strftime('install-%Y%m%d-%H%M%S.log') rep = Reporter(log_path=log) t0 = time.monotonic() diff --git a/installer/layout.py b/installer/layout.py index 4eef537a..2a83a034 100644 --- a/installer/layout.py +++ b/installer/layout.py @@ -90,12 +90,6 @@ class Layout: return [str(p) for p in need if not p.exists()] -def cache_dir(): - """$XDG_CACHE_HOME, else ~/.cache: the Linux release runs from a read-only AppImage mount, so its temp goes here.""" - xdg = os.environ.get('XDG_CACHE_HOME') - return Path(xdg) if xdg and Path(xdg).is_absolute() else Path.home() / '.cache' - - def release_python(root): return root / 'python' / ('python.exe' if WIN else 'bin/python3') @@ -117,7 +111,9 @@ def release_layout(root): xlate=root / 'bin' / ('totk_xlate' + EXE), dlls=(root / 'bin' / 'SDL3.dll', root / 'bin' / 'libopus-0.dll') if WIN else (root / 'bin' / 'libSDL3.so.0',), shader_list=root / 'data' / 'shader_list.tsv', ini_example=root / 'totk.ini.example', - temp=root / 'temp' if WIN else cache_dir() / 'totk-setup', + # Linux: None, the install folder's .setup-temp (installer/cli.py): the AppImage is mounted read-only, and on + # a Steam Deck /home is often too full for the generated C while the install folder is on the SD card + temp=root / 'temp' if WIN else None, sysroot=None if WIN else root / 'llvm' / 'sysroot') diff --git a/installer/steps.py b/installer/steps.py index fc7c1114..ea9e0548 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -102,6 +102,21 @@ def kill_tree(proc): proc.kill() +def linux_glibc(): + """(major, minor) of the C library this Linux process runs on, None elsewhere or when it is not glibc.""" + try: + name = os.confstr('CS_GNU_LIBC_VERSION') if sys.platform.startswith('linux') else None + except (ValueError, OSError): + return None + m = re.match(r'glibc (\d+)\.(\d+)', name or '') + return (int(m.group(1)), int(m.group(2))) if m else None + + +# The bundled clang, lld and the cryptography wheel are built against glibc 2.34 (tools/make_release_linux.py +# GLIBC_FLOOR); on an older one they cannot start +GLIBC_NEEDED = (2, 34) + + def linux_kernel(): """(major, minor) of the running Linux kernel, None elsewhere.""" if not sys.platform.startswith('linux'): @@ -214,7 +229,8 @@ class Check(Step): raise InstallError('check.release_incomplete', paths=missing[:5]) if dest.exists() and not dest.is_dir(): raise InstallError('check.dest_not_folder', path=str(dest)) - if ctx.mode == 'install' and dest.is_dir() and any(dest.iterdir()): + own = (ctx.temp, *ctx.temp.parents) # a temp folder inside the install folder (installer/cli.py) is ours + if ctx.mode == 'install' and dest.is_dir() and any(p not in own for p in dest.iterdir()): raise InstallError('check.dest_not_empty', path=str(dest)) lang = ctx.options.lang or ctx.state.get('lang') if not lang: @@ -242,6 +258,10 @@ class Check(Step): def host(self, ctx): layout, work = ctx.layout, ctx.temp / 'check' + glibc = linux_glibc() + if glibc and glibc < GLIBC_NEEDED and not layout.dev: + raise InstallError('check.glibc_old', version='.'.join(map(str, glibc)), + need='.'.join(map(str, GLIBC_NEEDED))) work.mkdir(parents=True, exist_ok=True) exe = work / ('hostcheck' + EXE) command = [str(layout.clang), '-O1', str(HOSTCHECK), '-o', str(exe)] diff --git a/installer/text/en.json b/installer/text/en.json index 502a06d6..a0a9fd69 100644 --- a/installer/text/en.json +++ b/installer/text/en.json @@ -41,6 +41,7 @@ "check.lang_missing": "Choose the game language.", "check.lang_unknown": "Unknown language '{lang}'. Known: {known}.", "check.toolchain_broken": "The bundled compiler does not work: {detail}", + "check.glibc_old": "This system's C library is glibc {version}; the setup's compiler needs {need} or later (Ubuntu 22.04, Debian 12, Fedora 35, SteamOS 3 and newer have it). Updating the system fixes this.", "check.host_probe_failed": "The computer check did not run (exit code {code}).", "check.cpu_no_avx2": "This processor lacks AVX2 (x86-64-v3), which the game needs. Intel Haswell (2013) or AMD Excavator / Zen and newer have it.", "check.vulkan_missing": "No Vulkan driver is installed. Install the latest driver for your graphics card.", diff --git a/installer/text/ru.json b/installer/text/ru.json index 3e4b91e1..1c3488b5 100644 --- a/installer/text/ru.json +++ b/installer/text/ru.json @@ -41,6 +41,7 @@ "check.lang_missing": "Выберите язык игры.", "check.lang_unknown": "Неизвестный язык «{lang}». Доступные: {known}.", "check.toolchain_broken": "Встроенный компилятор не работает: {detail}", + "check.glibc_old": "Библиотека C этой системы — glibc {version}; компилятору установщика нужна {need} или новее (она есть в Ubuntu 22.04, Debian 12, Fedora 35, SteamOS 3 и новее). Обновление системы это исправит.", "check.host_probe_failed": "Проверка компьютера не выполнилась (код выхода {code}).", "check.cpu_no_avx2": "Этому процессору не хватает AVX2 (x86-64-v3), который нужен игре. AVX2 есть у Intel Haswell (2013) и новее, у AMD Excavator и Zen и новее.", "check.vulkan_missing": "Драйвер Vulkan не установлен. Установите свежий драйвер видеокарты.", diff --git a/runtime/vk_device.c b/runtime/vk_device.c index 4c204906..bfacbbe2 100644 --- a/runtime/vk_device.c +++ b/runtime/vk_device.c @@ -41,7 +41,21 @@ static int layer_present(const char *name) { return found; } +// Mesa's drivers (RADV on Linux and the Steam Deck) keep a shader cache of their own beside our pipeline cache, by +// default in ~/.cache and up to 1 GB: about 35 MB in a minute of the title screen. A Deck's /home is small and the game +// usually sits on the SD card, so unless MESA_SHADER_CACHE_DIR names a folder, it goes to mesa-cache beside the exe. +static void mesa_cache_beside_exe(void) { + char dir[PLAT_PATH_MAX * 2]; + if (plat_getenv("MESA_SHADER_CACHE_DIR", dir, sizeof dir)) return; + int len = plat_exe_path(dir, sizeof dir - 16); + char *cut = len ? plat_last_sep(dir) : NULL; + if (!cut) return; + strcpy(cut + 1, "mesa-cache"); + plat_setenv("MESA_SHADER_CACHE_DIR", dir); +} + static int create_instance(void) { + mesa_cache_beside_exe(); const char *ext[8]; uint32_t n_ext = (uint32_t)gfx_window_vk_extensions(ext, 7); const char *layers[] = {"VK_LAYER_KHRONOS_validation"}; diff --git a/tools/make_release_linux.py b/tools/make_release_linux.py index a4a39501..30073602 100644 --- a/tools/make_release_linux.py +++ b/tools/make_release_linux.py @@ -404,6 +404,35 @@ def write_appimage(out, path): # ---- checks ------------------------------------------------------------------------------------------------------------- +# The newest glibc any program or library in the folder may need: the official LLVM binaries and the cryptography +# wheel are built against 2.34 (Ubuntu 22.04, Debian 12, SteamOS 3 have it); the game itself links the sysroot's 2.31. +GLIBC_FLOOR = (2, 34) + + +def check_glibc(out): + """Every ELF file in the folder (llvm/sysroot aside: glibc itself, read by the link) needs glibc GLIBC_FLOOR or + older; prints which files set the floor.""" + newest, by = (0, 0), [] + for p in sorted(out.rglob('*')): + if not p.is_file() or p.is_symlink() or out / 'llvm' / 'sysroot' in p.parents: + continue + with open(p, 'rb') as f: + if f.read(4) != b'\x7fELF': + continue + text = subprocess.run(['readelf', '-V', str(p)], capture_output=True, text=True).stdout + need = max((tuple(map(int, v.split('.'))) for v in re.findall(r'GLIBC_(\d+\.\d+(?:\.\d+)?)', text)), + default=(0, 0)) + if need > newest: + newest, by = need, [] + if need == newest: + by.append(str(p.relative_to(out))) + floor = '.'.join(map(str, GLIBC_FLOOR)) + if newest > GLIBC_FLOOR: + sys.exit(f'glibc check: {", ".join(by[:8])} need glibc {".".join(map(str, newest))}, above {floor}') + print(f'glibc check: the folder needs glibc {".".join(map(str, newest))} (floor {floor}), set by ' + f'{", ".join(by[:6])}{" and others" if len(by) > 6 else ""}') + + def check_bundle(out): """The folder's own python runs the installer without game files and the window's selftest.""" import json @@ -487,6 +516,7 @@ def main(): collect_licenses(out, sysroot) build_entry(out) mr.check_folder(out) + check_glibc(out) check_bundle(out) mr.sizes(out) if args.appimage: