diff --git a/README.md b/README.md index cd851018..01f98617 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,22 @@ # TotK Recomp -**TotK Recomp** is a static recompilation of *The Legend of Zelda: Tears of the Kingdom* (Nintendo Switch, 2023) into a native Windows x86-64 application ([docs/architecture.md](docs/architecture.md)). +**TotK Recomp** is a static recompilation of *The Legend of Zelda: Tears of the Kingdom* (Nintendo Switch, 2023) into a native x86-64 application for Windows and Linux ([docs/architecture.md](docs/architecture.md)). > [!IMPORTANT] -> **No game files are included.** You need your own legally obtained copy of *Tears of the Kingdom*, dumped from your own console: the base game and the 1.2.1 or 1.4.3 update. Instructions for obtaining those files will not be provided. +> **No game files are included.** You need your own legally obtained copy of *Tears of the Kingdom*, dumped from your own console: the base game and the 1.1.1, 1.2.1 or 1.4.3 update. Instructions for obtaining those files will not be provided. > > The game is built from your files on your machine, so the result contains the game. It is for your own use. Do not share it. ## Status -Early development. The game is playable on Windows; visual bugs and crashes are still expected. +Early development. The game is playable on Windows and Linux; visual bugs and crashes are still expected. Report problems as issues, following [CONTRIBUTING.md](CONTRIBUTING.md). ## Features - Native code, not emulation -- Game updates 1.2.1 and 1.4.3, with saves that carry over between them +- Game updates 1.1.1, 1.2.1 and 1.4.3; 1.2.1 and 1.4.3 saves carry over between them - 60 fps by default; up to 144 fps or the monitor's refresh rate (experimental) - Custom resolutions (1080p, 2k...), supersampling and AMD FSR 1 upscaling - 16:10 (Steam Deck, handhelds), 21:9 and 32:9 (ultrawide) game resolutions, with the camera and the interface in that shape @@ -27,12 +27,13 @@ Report problems as issues, following [CONTRIBUTING.md](CONTRIBUTING.md). ## Planned -- Linux and the Steam Deck - Mods made for emulators: unpack a mod into a `mods` folder - VRR ## Installing +### Windows + Requirements: Windows 10 or later, an x86-64 CPU with AVX2, a GPU with Vulkan 1.3 and `VK_EXT_external_memory_host`, about 36 GiB free during setup (20 GiB after). 1. Download `TotK-Setup--windows.zip` from [Releases](../../releases/latest) and unpack it. @@ -40,7 +41,16 @@ Requirements: Windows 10 or later, an x86-64 CPU with AVX2, a GPU with Vulkan 1. 3. Pick your game files, the language and an install folder, and press *Start*. The setup builds the game once. 4. Play `totk.exe` from the install folder. -To update, run a newer `TotK-Setup.exe` on the same folder; saves and settings are kept. Settings are in the in-game menu and in `totk.ini` ([totk.ini.example](totk.ini.example) lists every key). Controls: [docs/controls.md](docs/controls.md). +### Linux + +Requirements: as on Windows, plus Linux 6.7 or later and glibc 2.34 or later (Ubuntu 22.04, Debian 12, Fedora 35, SteamOS 3). + +1. Download `TotK-Setup-x86_64.AppImage` from [Releases](../../releases/latest), or build it ([docs/installer.md](docs/installer.md), "Linux release"). +2. Make it executable (`chmod +x TotK-Setup-x86_64.AppImage`) and start it. On a Steam Deck, do this in Desktop Mode. +3. Pick your game files, the language and an install folder, and press *Start*. +4. Play `totk` from the install folder. + +To update, run a newer setup on the same folder; saves and settings are kept. Settings are in the in-game menu and in `totk.ini` ([totk.ini.example](totk.ini.example) lists every key). Controls: [docs/controls.md](docs/controls.md). ## Credits diff --git a/docs/building.md b/docs/building.md index f457b81a..0c881d0c 100644 --- a/docs/building.md +++ b/docs/building.md @@ -69,7 +69,8 @@ of them. keys file. `config.local.toml` is ignored by git. The tools read the keys file themselves. 3. Extract and verify. This writes `build\extracted\v0` (1.0.0) and `build\extracted\` for each update in - `updates`, identified by its hash; a hash mismatch stops the run: + `updates`, identified by its hash; a hash mismatch stops the run. With another `[work] dir` in `config.local.toml`, + `extracted` goes there instead, and `build_game.py` reads the ExeFS and RomFS from there: ```powershell .venv-py314\Scripts\python tools\extract.py @@ -115,6 +116,11 @@ of them. Without this step the first launch compiles shaders and pipelines during play and stutters until the caches fill. The AMD driver keeps its pipeline cache per exe path and holds a limited amount: preparing another build folder can evict this one's. `totk.exe --warm-shaders` rebuilds it from the shader cache. + When the driver's shader compiler crashes on a variant (NVIDIA 615.78.08 on an RTX 5080 does, issue #30), + `prepare_shaders.py` runs the driver step again with `--warm-shaders`, up to three times, without the variants the + driver was building at the crash. `shader-prepare-skip.txt` next to the exe lists them with the GPU and driver + version; they build at their first draw instead, another driver version builds them again, and so does deleting the + file. 7. Run. The game keeps running when its window loses the focus (`assume_focus = 1` in `totk.ini`, the default; the Game tab switches it). With 0 it pauses, as it does on the Switch when Home is pressed. diff --git a/docs/installer.md b/docs/installer.md index 304e239c..792f7453 100644 --- a/docs/installer.md +++ b/docs/installer.md @@ -9,7 +9,7 @@ holds only the game; the installer and its tools stay outside it. 1. Downloads `TotK-Setup--windows.zip` from GitHub Releases (none is published yet), unpacks it anywhere and starts `TotK-Setup.exe`. 2. Picks: - - the base game (XCI or NSP) and the 1.2.1 or 1.4.3 update (NSP); + - the base game (XCI or NSP) and the 1.1.1, 1.2.1 or 1.4.3 update (NSP); - their `prod.keys`; - the game language and the install folder. 3. Waits for the setup, one time. The window shows each step and its progress. @@ -34,12 +34,13 @@ The player runs a newer installer on the same folder. It reads `totk-install.jso `profile/` and `totk.ini` after checking their hashes, rebuilds `totk.exe` and re-prepares the shaders. A newer release that changes nothing game-derived skips the rebuild. -Given an update file other than the recorded one (1.2.1, then 1.4.3), the installer extracts and builds again from it and -keeps `profile/`. Saves carry over both ways: 1.4.3 converts a 1.2.1 save on load, and 1.2.1 loads the saves 1.4.3 wrote. +Given an update file other than the recorded one (1.1.1, then 1.2.1, then 1.4.3), the installer extracts and builds again from it +and keeps `profile/`. Saves carry over both ways between 1.2.1 and 1.4.3: 1.4.3 converts a 1.2.1 save on load, and 1.2.1 +loads the saves 1.4.3 wrote. ## Setup steps -1. **Check** before writing anything: the files open with the keys, the update is 1.2.1 or 1.4.3 (by hash, as +1. **Check** before writing anything: the files open with the keys, the update is 1.1.1, 1.2.1 or 1.4.3 (by hash, as `tools/extract.py`), the keys are new enough for the update, enough space is free on the target disk (about 36 GiB at the peak of a first install, 20 GiB after it), the CPU has AVX2 (`-march=x86-64-v3`), a Vulkan 1.3 GPU exists. Each failure is one clear message. @@ -55,6 +56,9 @@ keeps `profile/`. Saves carry over both ways: 1.4.3 converts a 1.2.1 save on loa writes the translations it makes in play as loose files; running the tool again with the game closed folds those in. The patches are written twice: `romfs-patches` and `romfs-patches-xbox` (`--xbox-prompts`, the A/B and X/Y prompts swapped for `pad_layout = xbox_prompts`). + When the graphics driver crashes while building the shaders, `prepare_shaders.py` runs that part again without the + variants it was building ([building.md](building.md), step 6), and the setup finishes with a warning that names how + many were left out. 6. **Clean up** the temp folder (generated C and objects, about 1.3 GB). A cancelled or failed setup resumes from the last finished step (`totk-install.json`). @@ -79,9 +83,10 @@ python -m installer --base FILE --update FILE --keys FILE --lang CODE --dest DIR the merged RomFS (15.9 GiB) before it deletes them. The check computes the exact figure. - Window (`python -m installer.gui`, `installer/gui.py` draws, `installer/guilogic.py` decides): form, step list with progress, space line, collapsible log, Cancel (Ctrl+Break to the CLI's process group, then it waits for the exit), - finish screen with Open folder. English or Russian, chosen from the Windows display language, switchable in the - window; every sentence is an id in `installer/text/.json`. A folder with `totk-install.json` needs only the - folder. `--selftest [--shots DIR]` builds every screen in both languages and closes itself. + finish screen with Open folder. English, Russian or Brazilian Portuguese, chosen from the Windows display language, + switchable in the window; every sentence is an id in `installer/text/.json`. A folder with + `totk-install.json` needs only the folder. `--selftest [--shots DIR]` builds every screen in every language and + closes itself. - `tests/installer_gui_test.py`: event model, form, catalogs, cancel, the self-closing window and one end-to-end run of the CLI in `--verify` mode (needs `build/install-test` and `build/extracted/v121`; see its docstring). - `tests/installer_test.py`: bookkeeping with fake steps. `tests/installer_e2e.py`: a dev-mode run with the files of diff --git a/docs/maxwell.md b/docs/maxwell.md index dc71a041..472360f7 100644 --- a/docs/maxwell.md +++ b/docs/maxwell.md @@ -142,6 +142,7 @@ Source shorthands: | TotK | "detective mode" Link on AMD | AMD EDS3 colour-blend driver regression | YZ-2306 PR 10946 | | BotW | shadow artifacts at object edges (AMD) | gather rounding | yuzu commit 780240e (2.10) | | TotK | dashed dark lines along wall and floor edges, changing with the camera angle (AMD, RADV) | gather rounding | gather bias (2.10) | +| TotK | blocky foliage shadows that jump instead of sliding | not a GPU difference: the engine sizes its sun shadow map to the area it covers, down to about 400 near the ground, and each change of size moves the shadow texels | fixed size, `shadow_res` (the shadow set of `recomp/patches-v121.txt`; the community mods force 1024 the same way) | | BotW | black textures on GCN | HTILE (depth compression) | `AMD_DEBUG=nohyperz` wiki (1.9) | | BotW | 3D scene black | deferred texture handles | NXN | | BotW | albedo scattered squares | duplicate colour attachments | NXN | diff --git a/docs/tester-guide.md b/docs/tester-guide.md index 67a9fc70..a1cab09e 100644 --- a/docs/tester-guide.md +++ b/docs/tester-guide.md @@ -8,7 +8,7 @@ from it may be shared. - Windows 10 or later, an x86-64 CPU with AVX2, about 60 GB free. - A GPU with Vulkan 1.3 and `VK_EXT_external_memory_host`, with a current driver. -- *Tears of the Kingdom* base game (1.0.0) and the 1.2.1 or 1.4.3 update, dumped from your own Switch as NSP or XCI. +- *Tears of the Kingdom* base game (1.0.0) and the 1.1.1, 1.2.1 or 1.4.3 update, dumped from your own Switch as NSP or XCI. - `prod.keys` dumped from your own Switch. Never send it to anyone, and never paste it into a log or an issue. ## Build diff --git a/installer/gui.py b/installer/gui.py index cd94fcde..646bc8f4 100644 --- a/installer/gui.py +++ b/installer/gui.py @@ -2,10 +2,10 @@ A front end over the CLI (docs/installer.md): it collects the game files, the keys file path, the game language and the install folder, runs `python -m installer ...` as a subprocess and shows its JSON events. Decisions live in guilogic.py; -this file only draws. The window text is in installer/text/.json under "ui.*" and "launcher.*" ids. +this file only draws. The window text is in installer/text/.json under "ui.*" and "launcher.*" ids. - --selftest [--shots DIR] build every screen in both languages and close by itself (PNG per screen with --shots) - --lang en|ru window language (default: the Windows display language) + --selftest [--shots DIR] build every screen in every language and close by itself (PNG per screen with --shots) + --lang en|ru|pt-BR window language (default: the Windows display language) """ import argparse import json @@ -534,7 +534,7 @@ def grab(root, path): def selftest(layout, shots=None): - """Every screen in both languages; the window closes itself. Returns 0 when nothing raised.""" + """Every screen in every language; the window closes itself. Returns 0 when nothing raised.""" fix_tcl_paths() root = tk.Tk() root.geometry('+60+40') diff --git a/installer/guilogic.py b/installer/guilogic.py index 8d6c4e91..2056e86c 100644 --- a/installer/guilogic.py +++ b/installer/guilogic.py @@ -24,11 +24,12 @@ from . import state as statefile from .events import render TEXT = Path(__file__).resolve().parent / 'text' -UI_LANGUAGES = ('en', 'ru') +UI_LANGUAGES = ('en', 'ru', 'pt-BR') STEP_ORDER = ('check', 'extract', 'lift', 'compile', 'prepare', 'cleanup') # message parameter -> catalog id prefix of the names that replace its raw values PARAM_NAMES = {'lang': 'lang.', 'known': 'lang.', 'step': 'step.', 'input': 'param.input.'} LANG_RU = 0x19 # primary language id of Russian in a Windows LANGID +LANG_PT_BR = 0x0416 # Portuguese (Brazil), the whole LANGID: Portugal's Portuguese stays English def load_texts(lang): @@ -76,15 +77,16 @@ def flat(value): def windows_ui_language(): - """'ru' when the display language (Windows; elsewhere LC_ALL, LC_MESSAGES or LANG) is Russian, else 'en'.""" + """'ru' when the display language (Windows; elsewhere LC_ALL, LC_MESSAGES or LANG) is Russian, 'pt-BR' when it is + Brazilian Portuguese (LANGID 0x0416, or a pt_BR locale), else 'en'.""" if os.name != 'nt': - loc = os.environ.get('LC_ALL') or os.environ.get('LC_MESSAGES') or os.environ.get('LANG') or '' - return 'ru' if loc.lower().startswith('ru') else 'en' + loc = (os.environ.get('LC_ALL') or os.environ.get('LC_MESSAGES') or os.environ.get('LANG') or '').lower() + return 'ru' if loc.startswith('ru') else 'pt-BR' if loc.startswith(('pt_br', 'pt-br')) else 'en' try: langid = ctypes.windll.kernel32.GetUserDefaultUILanguage() except (AttributeError, OSError): return 'en' - return 'ru' if langid & 0x3ff == LANG_RU else 'en' + return 'ru' if langid & 0x3ff == LANG_RU else 'pt-BR' if langid == LANG_PT_BR else 'en' def default_game_language(ui): diff --git a/installer/launcher/launcher.c b/installer/launcher/launcher.c index 11994cd9..ce3a2d2c 100644 --- a/installer/launcher/launcher.c +++ b/installer/launcher/launcher.c @@ -9,14 +9,19 @@ #include "launcher_text.h" -#define LANGID_RUSSIAN 0x19 +#define LANGID_RUSSIAN 0x19 /* primary language id */ +#define LANGID_PT_BR 0x0416 /* the whole LANGID: Portugal's Portuguese stays English */ #define PATH_CHARS (MAX_PATH * 2) static void fail(void) { - int russian = (GetUserDefaultUILanguage() & 0x3ff) == LANGID_RUSSIAN; - MessageBoxW(NULL, russian ? LAUNCHER_PYTHON_MISSING_RU : LAUNCHER_PYTHON_MISSING_EN, - russian ? LAUNCHER_TITLE_RU : LAUNCHER_TITLE_EN, MB_OK | MB_ICONERROR); + LANGID id = GetUserDefaultUILanguage(); + int russian = (id & 0x3ff) == LANGID_RUSSIAN, brazilian = id == LANGID_PT_BR; + MessageBoxW(NULL, + russian ? LAUNCHER_PYTHON_MISSING_RU : brazilian ? LAUNCHER_PYTHON_MISSING_PT_BR + : LAUNCHER_PYTHON_MISSING_EN, + russian ? LAUNCHER_TITLE_RU : brazilian ? LAUNCHER_TITLE_PT_BR : LAUNCHER_TITLE_EN, + MB_OK | MB_ICONERROR); } int WINAPI WinMain(HINSTANCE instance, HINSTANCE previous, LPSTR command_line, int show) diff --git a/installer/steps.py b/installer/steps.py index c24cb1d0..5ef4874d 100644 --- a/installer/steps.py +++ b/installer/steps.py @@ -644,6 +644,9 @@ class Compile(Step): # ---- 5. prepare TRANSLATED = re.compile(r'gfx_shader: prepare: (\d+) of (\d+) pairs') +# runtime/vk_pipeline.c lb_warm_prepared: variants the driver crashed on in an earlier run (tools/prepare_shaders.py +# runs the driver step again after such a crash) +SKIPPED = re.compile(r'prepare-shaders: (\d+) variants skipped: the graphics driver crashed') # The game resolutions of another shape than 16:9 that runtime/ui/menu.cpp row_output_res lists: 21:9 and 32:9 monitors, @@ -715,10 +718,14 @@ class Prepare(Step): shader_list = ctx.temp / 'shader_list.tsv' shader_list.write_text(shader_list_text(layout), encoding='utf-8', newline='\n') + skipped = [0] + def parse(line): m = TRANSLATED.search(line) if m: rep.progress('prepare', 'translate', int(m.group(1)), int(m.group(2)), 'shaders') + elif SKIPPED.search(line): + skipped[0] = int(SKIPPED.search(line).group(1)) elif 'listed variants' in line: rep.progress('prepare', 'translate', force=True) elif 'gfx_shader: prepare:' in line and 'translated' in line and 'programs' in line: @@ -730,6 +737,8 @@ class Prepare(Step): dest / 'exefs', '--list', shader_list], parse) if code or not (dest / 'shader-prepared.bin').is_file(): raise InstallError('prepare.shaders_failed', code=code, log=str(log)) + if skipped[0]: + rep.warning('warn.shaders_skipped', count=skipped[0], log=str(log)) # One shader-cache.pack instead of four files per shader (~117,000): totk.exe reads the pack first. rep.progress('prepare', 'pack', force=True) diff --git a/installer/text/en.json b/installer/text/en.json index 16625c72..167cdcfa 100644 --- a/installer/text/en.json +++ b/installer/text/en.json @@ -86,6 +86,7 @@ "warn.lang_needs_fonts": "Text in {lang} needs the Switch system fonts, which the installer does not extract; some text may not show.", "warn.npdm_differs": "main.npdm differs from the reference {version} dump; only the code modules are used.", "warn.xbox_prompts_failed": "The Xbox button prompts could not be prepared (exit code {code}); the game keeps the Switch prompts. Details: {log}", + "warn.shaders_skipped": "The graphics driver crashed while building some shaders, so {count} were left out of the preparation. The game builds them when it first needs them, and the driver may crash there too; a driver update may fix it. Details: {log}", "warn.kernel_old": "Linux {version} lacks the write tracking the game relies on (6.7 or later); the game may run very slowly. Updating the system fixes this.", "ui.title": "TotK Setup", diff --git a/installer/text/pt-BR.json b/installer/text/pt-BR.json new file mode 100644 index 00000000..0f9791a8 --- /dev/null +++ b/installer/text/pt-BR.json @@ -0,0 +1,180 @@ +{ + "step.check": "Verificando o computador e os arquivos do jogo", + "step.extract": "Extraindo o jogo", + "step.lift": "Traduzindo o código do jogo para C", + "step.compile": "Compilando o {game_exe}", + "step.prepare": "Preparando os shaders", + "step.cleanup": "Finalizando", + + "phase.check.host": "Verificando o processador e a placa de vídeo", + "phase.check.verify": "Verificando os arquivos já instalados", + "phase.check.verify_romfs": "Verificando os dados do jogo instalados", + "phase.check.inputs": "Abrindo os arquivos do jogo com as chaves", + "phase.check.versions": "Verificando as versões do jogo", + "phase.extract.hash_base": "Calculando o checksum do jogo base", + "phase.extract.hash_update": "Calculando o checksum da atualização", + "phase.extract.exefs": "Extraindo o código do jogo", + "phase.extract.copy": "Copiando os dados do jogo", + "phase.extract.merge": "Aplicando a atualização aos dados do jogo", + "phase.extract.diagnose": "Procurando o arquivo do jogo danificado", + "phase.extract.index": "Verificando os dados do jogo", + "phase.lift.discover": "Encontrando as funções", + "phase.lift.emit": "Gerando o código C", + "phase.compile.compile": "Compilando", + "phase.compile.link": "Ligando o {game_exe}", + "phase.prepare.romfs_patches": "Gravando os patches de 60 fps nos dados do jogo", + "phase.prepare.seed": "Lendo os shaders dos dados do jogo", + "phase.prepare.translate": "Traduzindo os shaders", + "phase.prepare.libraries": "Montando os shaders para o driver de vídeo", + "phase.prepare.pack": "Juntando o cache de shaders num único arquivo", + "phase.cleanup.ini": "Gravando o totk.ini", + "phase.cleanup.temp": "Removendo os arquivos temporários", + + "cli.usage": "Linha de comando inválida: {detail}", + "internal": "Erro inesperado ({exception}): {detail}", + "state.unreadable": "Não foi possível ler {path}. Escolha outra pasta ou apague esse arquivo.", + "state.newer_format": "{path} foi gravado por um instalador mais novo. Use esse instalador ou outra pasta.", + "state.output_changed": "Os arquivos instalados da etapa '{step}' mudaram desde que foram criados; essa etapa será executada de novo.", + + "check.release_incomplete": "A pasta do instalador está incompleta; faltam: {paths}. Extraia o pacote de novo.", + "check.dest_not_folder": "O local de instalação {path} é um arquivo, não uma pasta.", + "check.dest_not_empty": "A pasta de instalação {path} não está vazia e não tem uma instalação anterior. Escolha uma pasta vazia.", + "check.lang_missing": "Escolha o idioma do jogo.", + "check.lang_unknown": "Idioma desconhecido '{lang}'. Conhecidos: {known}.", + "check.toolchain_broken": "O compilador incluído não funciona: {detail}", + "check.glibc_old": "A biblioteca C deste sistema é a glibc {version}; o compilador da instalação precisa da {need} ou mais nova (Ubuntu 22.04, Debian 12, Fedora 35, SteamOS 3 e mais novos têm). Atualizar o sistema resolve.", + "check.host_probe_failed": "A verificação do computador não rodou (código de saída {code}).", + "check.cpu_no_avx2": "Este processador não tem AVX2 (x86-64-v3), que o jogo exige. Intel Haswell (2013) ou AMD Excavator / Zen e mais novos têm.", + "check.vulkan_missing": "Nenhum driver Vulkan está instalado. Instale o driver mais recente da sua placa de vídeo.", + "check.vulkan_no_device": "O Vulkan não encontra nenhuma placa de vídeo. Instale o driver mais recente da sua placa de vídeo.", + "check.vulkan_too_old": "O jogo precisa de uma placa de vídeo com Vulkan 1.3; encontrado: {devices}. Atualize o driver de vídeo.", + "check.inputs_needed": "Os arquivos do jogo são necessários: escolha o jogo base, a atualização 1.2.1 ou 1.4.3 e o prod.keys.", + "check.input_missing": "Arquivo não encontrado ({input}): {path}", + "check.keys_unreadable": "Não foi possível ler o arquivo de chaves. Escolha um arquivo prod.keys.", + "check.keys_no_header_key": "O arquivo de chaves não tem header_key. Extraia o prod.keys do seu console de novo.", + "check.keys_wrong": "As chaves não abrem o arquivo de {input}. Use o prod.keys do seu console.", + "check.keys_too_old": "O arquivo de chaves é antigo demais para o arquivo de {input} (precisa da geração de chaves {generation}). Extraia o prod.keys de um console com firmware mais novo.", + "check.not_game_file": "O arquivo de {input} não é um arquivo de jogo XCI ou NSP.", + "check.file_unreadable": "Não foi possível ler o arquivo de {input}: {detail}", + "check.ticket_personalized": "O arquivo de {input} tem um ticket personalizado; é preciso um ticket comum.", + "check.ticket_missing": "O arquivo de {input} não tem o ticket de que o conteúdo precisa.", + "check.base_not_totk": "O arquivo do jogo base não é The Legend of Zelda: Tears of the Kingdom ({detail}).", + "check.base_is_update": "O arquivo do jogo base é uma atualização. Escolha aqui o NSP ou XCI do jogo base (versão 1.0.0) e este arquivo como a atualização.", + "check.base_combined": "O arquivo do jogo base contém vários pacotes (jogo base, atualização ou DLC em um só arquivo). Escolha um dump do jogo base que contenha só o jogo.", + "check.update_not_nsp": "A atualização precisa ser um arquivo NSP.", + "check.update_is_base": "O arquivo de atualização contém um jogo base. Escolha o NSP da atualização 1.2.1 ou 1.4.3.", + "check.update_not_totk": "O arquivo de atualização não é uma atualização de Tears of the Kingdom ({detail}).", + "check.exefs_failed": "O código do jogo no arquivo de {input} falhou nas verificações; veja {log}.", + "check.input_damaged": "O arquivo de {input} está danificado ou incompleto: {path} ({detail}). Copie esse arquivo de novo da origem, ou extraia-o de novo, e então execute a instalação outra vez.", + "check.base_wrong_version": "O jogo base não é a versão 1.0.0 ({modules} diferem).", + "check.update_wrong_version": "A atualização é a versão {found}; esta versão do instalador aceita {wanted}.", + "check.disk_space": "Espaço livre insuficiente em {path}: são necessários {need_gb} GB, há {free_gb} GB livres.", + + "extract.exefs_mismatch": "O código do jogo extraído não corresponde à versão {version} ({modules}).", + "extract.romfs_merge_failed": "Falha ao aplicar a atualização aos dados do jogo: {detail}. Detalhes: {log}", + "extract.romfs_mismatch": "Os dados do jogo extraídos não correspondem à versão {version} (tamanho {size}, checksum {sha256}...).", + "lift.failed": "Falha ao traduzir o código do jogo (código de saída {code}). Detalhes: {log}", + "compile.failed": "Falha ao compilar o {game_exe} (código de saída {code}). Detalhes: {log}", + "compile.exe_in_use": "{path} está em uso. Feche o jogo e execute o instalador de novo.", + "prepare.shader_list_missing": "A lista de shaders está faltando.", + "prepare.romfs_patches_failed": "Falha ao gravar os patches nos dados do jogo (código de saída {code}). Detalhes: {log}", + "prepare.shaders_failed": "Falha ao preparar os shaders (código de saída {code}). Detalhes: {log}", + "cleanup.romfs_unreadable": "Os dados do jogo em {path} estão faltando ou não podem ser lidos depois da instalação. Reinicie o Windows e execute o instalador de novo na mesma pasta: ele extrai os dados do jogo outra vez e mantém o resto.", + "prepare.pack_failed": "Falha ao juntar o cache de shaders (código de saída {code}). Detalhes: {log}", + "cleanup.ini_failed": "Não foi possível gravar o totk.ini: {detail}", + + "warn.lang_needs_fonts": "Textos em {lang} precisam das fontes de sistema do Switch, que o instalador não extrai; parte do texto pode não aparecer.", + "warn.npdm_differs": "O main.npdm difere do dump de referência da versão {version}; só os módulos de código são usados.", + "warn.xbox_prompts_failed": "Não foi possível preparar os ícones de botão do Xbox (código de saída {code}); o jogo mantém os ícones do Switch. Detalhes: {log}", + "warn.shaders_skipped": "O driver gráfico travou ao compilar alguns shaders, então {count} ficaram fora da preparação. O jogo os compila quando precisar deles pela primeira vez, e o driver pode travar ali também; uma atualização do driver pode resolver. Detalhes: {log}", + "warn.kernel_old": "O Linux {version} não tem o rastreamento de escrita de que o jogo depende (6.7 ou mais novo); o jogo pode rodar muito devagar. Atualizar o sistema resolve.", + + "ui.title": "Instalação do TotK", + "ui.version": "versão {version}", + "ui.intro": "Monta uma pasta jogável de Tears of the Kingdom a partir dos seus próprios arquivos do jogo. Isso demora um pouco e só precisa ser feito uma vez.", + "ui.notice": "A pasta que esta instalação monta contém o jogo. Ela é só para seu uso pessoal; não compartilhe nem envie para a internet.", + "ui.keys_private": "O arquivo de chaves é lido no lugar. O conteúdo dele nunca é mostrado, copiado nem guardado.", + "ui.base": "Jogo base (NSP ou XCI)", + "ui.update": "Atualização 1.2.1 ou 1.4.3 (NSP)", + "ui.keys": "prod.keys", + "ui.game_language": "Idioma do jogo", + "ui.dest": "Pasta de instalação", + "ui.browse": "Procurar...", + "ui.pick_game": "Escolha o arquivo do jogo base", + "ui.pick_update": "Escolha o arquivo da atualização 1.2.1 ou 1.4.3", + "ui.pick_keys": "Escolha o prod.keys", + "ui.pick_dest": "Escolha a pasta de instalação", + "ui.type_game": "Arquivos de jogo (NSP, XCI)", + "ui.type_nsp": "Arquivos NSP", + "ui.type_keys": "Arquivos de chaves (*.keys)", + "ui.type_all": "Todos os arquivos", + "ui.start": "Iniciar", + "ui.cancel": "Cancelar", + "ui.cancelling": "Cancelando...", + "ui.show_log": "Mostrar log", + "ui.hide_log": "Ocultar log", + "ui.close": "Fechar", + "ui.back": "Voltar", + "ui.open_folder": "Abrir pasta", + "ui.update_mode": "Esta pasta tem uma instalação anterior. A instalação vai continuar ou atualizá-la; os arquivos do jogo só são pedidos se forem necessários de novo.", + "ui.need_dest": "Escolha a pasta de instalação.", + "ui.need_base": "Escolha o arquivo do jogo base.", + "ui.need_update": "Escolha o arquivo da atualização 1.2.1 ou 1.4.3.", + "ui.need_keys": "Escolha o arquivo prod.keys.", + "ui.need_lang": "Escolha o idioma do jogo.", + "ui.missing_base": "O arquivo do jogo base não existe.", + "ui.missing_update": "O arquivo de atualização não existe.", + "ui.missing_keys": "O arquivo de chaves não existe.", + "ui.running_install": "Instalando", + "ui.running_resume": "Continuando a instalação anterior", + "ui.running_update": "Atualizando a instalação", + "ui.state_started": "iniciada", + "ui.state_done": "concluída", + "ui.state_skipped": "pulada", + "ui.state_pending": "aguardando", + "ui.state_running": "em andamento", + "ui.state_failed": "falhou", + "ui.log_warning": "Aviso", + "ui.log_error": "Erro", + "ui.log_space": "Espaço necessário: {need} GB, livre: {free} GB", + "ui.log_end": "Concluído", + "ui.result_ok": "O jogo está pronto", + "ui.result_failed": "A instalação falhou", + "ui.result_cancelled": "A instalação foi cancelada", + "ui.done_text": "Abra o {game_exe} em {dest}. O programa de instalação e os arquivos do jogo que você escolheu não são mais necessários.", + "ui.cancelled_text": "As etapas concluídas são mantidas. Inicie a instalação de novo com a mesma pasta para continuar.", + "ui.log_file": "Arquivo de log: {path}", + "ui.process_died": "A instalação parou de forma inesperada (código de saída {code}).", + "ui.confirm_cancel": "A instalação está em andamento. Cancelar e fechar a janela?", + "ui.space_one": "Espaço livre necessário em {path}: {need} GB (livre: {free} GB).", + "ui.space_two": "Espaço livre necessário em {path}: {need} GB (livre: {free} GB); em {temp}: {temp_need} GB (livre: {temp_free} GB).", + "ui.amount_bytes": "{done} / {total} GB", + "ui.amount_chunks": "{done} / {total} partes", + "ui.amount_objects": "{done} / {total} arquivos", + "ui.amount_shaders": "{done} / {total} shaders", + "ui.amount_other": "{done} / {total} {unit}", + "ui.seconds": "{n} s", + "param.input.base": "jogo base", + "param.input.update": "atualização", + "param.input.keys": "chaves", + + "launcher.title": "Instalação do TotK", + "launcher.python_missing": "Os arquivos de instalação estão incompletos: a pasta python ao lado do TotK-Setup.exe não existe. Extraia o arquivo compactado inteiro de novo e abra o TotK-Setup.exe a partir da pasta extraída.", + + "lang.ja": "Japonês", + "lang.en-US": "Inglês (EUA)", + "lang.en-GB": "Inglês (Reino Unido)", + "lang.fr": "Francês", + "lang.fr-CA": "Francês (Canadá)", + "lang.de": "Alemão", + "lang.it": "Italiano", + "lang.es": "Espanhol", + "lang.es-419": "Espanhol (América Latina)", + "lang.nl": "Holandês", + "lang.pt": "Português", + "lang.pt-BR": "Português (Brasil)", + "lang.ru": "Russo", + "lang.ko": "Coreano", + "lang.zh-Hans": "Chinês (Simplificado)", + "lang.zh-Hant": "Chinês (Tradicional)" +} diff --git a/installer/text/ru.json b/installer/text/ru.json index 3e843b4a..f7778dee 100644 --- a/installer/text/ru.json +++ b/installer/text/ru.json @@ -86,6 +86,7 @@ "warn.lang_needs_fonts": "Для языка «{lang}» нужны системные шрифты Switch, которые установка не извлекает; часть текста может не отображаться.", "warn.npdm_differs": "main.npdm отличается от эталонного дампа {version}; используются только модули с кодом.", "warn.xbox_prompts_failed": "Не удалось подготовить подсказки кнопок Xbox (код выхода {code}); игра оставит подсказки Switch. Подробности: {log}", + "warn.shaders_skipped": "Графический драйвер аварийно завершился при сборке части шейдеров, поэтому {count} из них пропущены при подготовке. Игра соберёт их, когда они впервые понадобятся, и драйвер может упасть и там; обновление драйвера может это исправить. Подробности: {log}", "warn.kernel_old": "В Linux {version} нет отслеживания записи, на которое опирается игра (нужна версия 6.7 или новее); игра может работать очень медленно. Обновление системы это исправит.", "ui.title": "Установка TotK", diff --git a/recomp/patches-v121.txt b/recomp/patches-v121.txt index 7dbb63ed..a68eb80a 100644 --- a/recomp/patches-v121.txt +++ b/recomp/patches-v121.txt @@ -9,6 +9,7 @@ set cloth TOTK_CLOTH_REALTIME 1 set bounce TOTK_HIGHFPS_FIX 1 set output TOTK_PATCH_OUTPUT 0 set scene TOTK_PATCH_SCENE 0 +set shadow TOTK_PATCH_SHADOW 0 # 60 fps: the adaptive present-interval controller (main+0x8f26c8, 1.0.0 0x8f6bec) gets the minimum interval 1. site fps60 main 8f273c 6b08015f 7100055f # clamp: cmp w10, #1 (was cmp w10, w8 = minimum) @@ -947,3 +948,12 @@ spin sdk 23a590 23a5d8 # 19 instructions; loads spin sdk 23a658 23a6a0 # 19 instructions; loads spin sdk 23a7fc 23a918 # 19 instructions; loads spin sdk 3d3250 3d3270 # 9 instructions; loads + +# Sun shadow map size (runtime/resolution.c shadow_init switches the set on). Each frame the engine sizes its shadow +# cascades (a D16 4-layer array) in main+0xcebdf4: a base size (960 in the field), interpolated down by how close the +# cover is, rounded up to 8 at main+0xcec22c and stored as the width (+0x3d8 of the cascade set) and, through the packed +# store at +0xe4, the height (+0x3f8). Near the ground it falls to about 400, which draws foliage shadows blocky, and +# every change of size moves the shadow texels, so leaf shadows jump instead of sliding. The page's fixed size (+92) +# goes in just before the rounding, as the community's shadow resolution mods do (1.1.0: mov w11, #1024 over the +# computation). Found by a write watch on the size field (the writer was this function, and no other). +site shadow main cec22c 11001d6b @page:x11 b9405d6b 11001d6b # add w11, w11, #7 -> adrp x11, page; ldr w11, [x11, #92]; add w11, w11, #7 diff --git a/recomp/patches-v143.txt b/recomp/patches-v143.txt index c1a839fe..696f94b1 100644 --- a/recomp/patches-v143.txt +++ b/recomp/patches-v143.txt @@ -10,6 +10,7 @@ set cloth TOTK_CLOTH_REALTIME 1 set bounce TOTK_HIGHFPS_FIX 1 set output TOTK_PATCH_OUTPUT 0 set scene TOTK_PATCH_SCENE 0 +set shadow TOTK_PATCH_SHADOW 0 # 60 fps: the adaptive present-interval controller (main+0x2db174, 1.2.1 0x8f26c8) gets the minimum interval 1. site fps60 main 2db204 6b08015f 7100055f # clamp: cmp w10, #1 (was cmp w10, w8 = minimum) @@ -976,3 +977,8 @@ spin subsdk0 1c4e54 1c4e60 # 4 instructions; loads spin sdk 2d3654 2d3668 # 6 instructions; loads spin sdk 443864 443884 # 9 instructions; loads spin sdk 541e54 541f98 # 32 instructions; loads + +# Sun shadow map size (patches-v121.txt explains the set and the page). 1.4.3 interpolates the size between a minimum +# and the base and rounds it up to 8 at main+0x683604; the rounded value is the width (+0x3d8) and, through the packed +# store at +0xf4, the height (+0x3f8). The page's size goes in just before the rounding. +site shadow main 683604 11001d6b @page:x11 b9405d6b 11001d6b # add w11, w11, #7 -> adrp x11, page; ldr w11, [x11, #92]; add w11, w11, #7 diff --git a/recomp/patches.py b/recomp/patches.py index ecc12e7e..72ef2f74 100644 --- a/recomp/patches.py +++ b/recomp/patches.py @@ -14,8 +14,8 @@ encoded for the site's address. The patch page is the first 4 KiB page after mai of the 2 MiB-aligned module slot free); the runtime commits it and owns its contents. Words in use, by byte offset: +0..15 the frame-rate step (runtime/frame_rate.c), +16..55 the output, swapchain and scene sizes (runtime/resolution.c res_init), +56..59 the camera's aspect ratio (runtime/resolution.c res_init, set aspect), +60..63 -the field-of-view factor (runtime/resolution.c res_fov_live, set fov). A new -set takes words outside these. +the field-of-view factor (runtime/resolution.c res_fov_live, set fov), +92..95 the sun shadow map size +(runtime/resolution.c shadow_init, set shadow). A new set takes words outside these. The replacement words are lifted at the site's address in place of the original instruction, and the generated C reads `if (rt_patch_on[]) { replacement } else { original }`: one byte load and a branch per executed site. Replacement diff --git a/runtime/cfg.c b/runtime/cfg.c index d9c4cd61..4bc12865 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -47,10 +47,11 @@ static const char *const known[] = { "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHECK", "RESOLVE_CHECK", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", - "SHADER_MEMO", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "SWKBD_ANSWER", + "SHADER_MEMO", "SHADOW_RES", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "SWKBD_ANSWER", "SWKBD_TEST", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "UBO_SNAPSHOT", "UPSCALER", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64", "VK_PRESENT", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VS_OUTS", - "WB_EXACT", "WB_FENCE_LIMIT", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_PROCS", "ZSTD_CHECK", "ZSTD_NATIVE", + "WALK_KEY", "WALK_SCALE", "WB_EXACT", "WB_FENCE_LIMIT", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_PROCS", + "ZSTD_CHECK", "ZSTD_NATIVE", }; // The totk.ini section a key cfg_set may write goes to when the file lacks it (totk.ini.example's sections). @@ -61,12 +62,13 @@ static const struct { const char *key, *section; } sections[] = { {"GPU_INDEX", "display"}, {"OUTPUT_RES", "display"}, {"RENDER_SCALE", "display"}, {"UPSCALER", "display"}, {"SHARPNESS", "display"}, {"ASPECT", "display"}, {"FOV", "display"}, {"COLOR_PRESET", "display"}, {"COLOR_BRIGHTNESS", "display"}, {"COLOR_CONTRAST", "display"}, {"COLOR_SATURATION", "display"}, - {"COLOR_GAMMA", "display"}, + {"COLOR_GAMMA", "display"}, {"SHADOW_RES", "display"}, {"DEVICE_PROFILE", "display"}, {"ASSUME_FOCUS", "display"}, {"MENU_LANG", "language"}, {"LANG", "language"}, {"OPERATION_MODE", "language"}, {"VOLUME", "audio"}, {"MOTION", "input"}, {"RUMBLE_STRENGTH", "input"}, {"RUMBLE_CURVE", "input"}, {"RUMBLE_TRIGGERS", "input"}, - {"PAD_LAYOUT", "input"}, {"STICK_DEADZONE", "input"}, {"MOUSE_SENS", "input"}, {"INPUT_RECORD", "input"}, + {"PAD_LAYOUT", "input"}, {"STICK_DEADZONE", "input"}, {"MOUSE_SENS", "input"}, + {"WALK_KEY", "input"}, {"WALK_SCALE", "input"}, {"INPUT_RECORD", "input"}, {"BIND_A", "input"}, {"BIND_B", "input"}, {"BIND_X", "input"}, {"BIND_Y", "input"}, {"BIND_LCLICK", "input"}, {"BIND_RCLICK", "input"}, {"BIND_L", "input"}, {"BIND_R", "input"}, {"BIND_ZL", "input"}, {"BIND_ZR", "input"}, {"BIND_PLUS", "input"}, {"BIND_MINUS", "input"}, {"BIND_LEFT", "input"}, {"BIND_UP", "input"}, @@ -103,6 +105,7 @@ static const CfgMeta metas[] = { {"COLOR_CONTRAST", "100", "70..150", LIVE}, {"COLOR_SATURATION", "100", "50..200", LIVE}, {"COLOR_GAMMA", "1", "0.8..1.2", LIVE}, {"DEVICE_PROFILE", "auto", "auto|desktop|deck", RESTART}, + {"SHADOW_RES", "game", "game|512|768|1024|1536|2048", RESTART}, // Game {"FPS_LIMIT", "60", "30|40|45|60|90|120|144|display", LIVE}, {"FPS60", "1", "0|1", LIVE}, {"REALTIME", "1", "0|1", RESTART}, {"CLOTH_REALTIME", "1", "0|1", RESTART}, @@ -115,6 +118,7 @@ static const CfgMeta metas[] = { {"MOTION", "1", "0|1", LIVE}, {"RUMBLE_STRENGTH", "130", "0..300", LIVE}, {"RUMBLE_CURVE", "0.6", "0.2..1.5", LIVE}, {"RUMBLE_TRIGGERS", "0", "0|1", LIVE}, {"PAD_LAYOUT", "auto", "auto|nintendo|xbox|xbox_prompts", LIVE}, {"STICK_DEADZONE", "8", "0..50", LIVE}, {"MOUSE_SENS", "1", "0..10", LIVE}, + {"WALK_KEY", "none", NULL, LIVE}, {"WALK_SCALE", "0.51", "0.2..0.9", LIVE}, {"MENU_LANG", "", "en|ru", LIVE}, // Advanced: scheduling {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, diff --git a/runtime/host.h b/runtime/host.h index fd7bc1ef..b0eb23e4 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -57,6 +57,9 @@ typedef struct { int have_gpu, have_vram, vram_exact; uint64_t gpu_ns, gpu_parts, gpu_missed; // busy time, submission parts timed, parts left untimed uint64_t vram_used, vram_budget; + int have_idle; // calibrated timestamps: the idle split below is filled + uint64_t idle_ns, starved_ns, queued_ns; // GPU idle between timed parts; of it, before the part was submitted; submit to start + uint64_t idle_gaps, max_gap_ns; // gaps over 0.5 ms; the longest } PerfGpu; extern void (*perf_gpu_stats)(PerfGpu *out); // The performance overlay (perfmon.c): mode 0 off, 1 fps, 2 full. While it is on, perf_present hands every frame to diff --git a/runtime/input.c b/runtime/input.c index 86391739..108b967b 100644 --- a/runtime/input.c +++ b/runtime/input.c @@ -236,6 +236,52 @@ static float mouse_sens_get(void) { } +// ---- walk key: the keyboard's stick samples scaled down while a key is held (TOTK_WALK_KEY, TOTK_WALK_SCALE) +// A keyboard key is all or nothing, so every keyboard-driven stick sample is +/-32767 and the game only ever sees a +// full tilt. Holding this key scales those two samples, which gives the half tilt a gamepad gets from its analog range, +// so the keyboard can walk as well as run. yuzu and eden do the same with their modifier and modifier_scale settings. +// Only the keyboard is scaled: the gamepad's axes and the mouse camera sample their own values. +static int walk_key = KEY_NONE; // TOTK_WALK_KEY as a KEY_*, KEY_NONE = off; live (cfg_version) +static float walk_scale = 0.51f; // TOTK_WALK_SCALE, 0.2 to 0.9; live (cfg_version) +static unsigned walk_seen = ~0u; + +// The ini takes the key names the Controls tab writes; a hand-written line may spell them in any case. +static int name_matches(const char *name, const char *key) { + for (; *name && *key; name++, key++) { + unsigned char a = (unsigned char)*name, b = (unsigned char)*key; + if (a >= 'A' && a <= 'Z') a = (unsigned char)(a - 'A' + 'a'); + if (b >= 'A' && b <= 'Z') b = (unsigned char)(b - 'A' + 'a'); + if (a != b) return 0; + } + return *name == *key; +} + +static void walk_get(void) { + unsigned ver = cfg_version(); + if (ver == walk_seen) return; + walk_seen = ver; + const char *name = cfg_value("TOTK_WALK_KEY"); // "none" (the default) turns the key off + int key = KEY_NONE; + if (*name && !name_matches(name, "none")) { + for (int k = KEY_A; k < KEY_COUNT && key == KEY_NONE; k++) + if (input_key_name(k) && name_matches(name, input_key_name(k))) key = k; + if (key == KEY_NONE) rt_log("input: walk_key: unknown key '%s', the keyboard sticks keep their full tilt", name); + } + // 0.51 rather than 0.5: a keyboard sample is 32767, and 32767 * 0.5 = 16383 falls one short of stick_bits' + // STICK_DIR (16384), which would leave the game's stick-direction bits clear while the walk key is held. The value + // in the settings table carries that default; this code only rejects what the table's range does not allow. + const char *sv = cfg_value("TOTK_WALK_SCALE"); + char *end; + double d = strtod(sv, &end); + double s = end == sv || *end || !(d >= 0.2 && d <= 0.9) ? strtod(cfg_meta("TOTK_WALK_SCALE")->def, NULL) : d; + if (key != walk_key || (float)s != walk_scale) { + walk_key = key; + walk_scale = (float)s; + rt_log("input: walk key %s, keyboard sticks at %.0f%% of their tilt (mouse and gamepad unchanged)", + key != KEY_NONE ? input_key_name(key) : "off", s * 100); + } +} + // The mouse camera as a right-stick sample, held for the whole presented frame. Under `lock`. static void mouse_camera(int32_t out[2]) { static uint64_t frame = ~0ull, t0; @@ -456,9 +502,13 @@ static void poll_merged(uint64_t *buttons, int32_t *lx, int32_t *ly, int32_t *rx was_captured = cap; b |= input_map_buttons(down); int32_t k[2]; + walk_get(); + int walk = walk_key != KEY_NONE && down[walk_key]; input_map_stick(down, 0, k); + if (walk) k[0] = (int32_t)(k[0] * walk_scale), k[1] = (int32_t)(k[1] * walk_scale); pick(&sl[0], &sl[1], k[0], k[1]); input_map_stick(down, 1, k); + if (walk) k[0] = (int32_t)(k[0] * walk_scale), k[1] = (int32_t)(k[1] * walk_scale); pick(&sr[0], &sr[1], k[0], k[1]); mouse_camera(k); pick(&sr[0], &sr[1], k[0], k[1]); @@ -505,15 +555,20 @@ void input_menu(int open) { ikb_cursor_update(); } -// 1 while flag (g_block, g_menu) blocks the game's input; state 2 ends at the first poll with nothing held. -static int blocked(volatile int32_t *flag, uint64_t buttons) { +// 1 while flag (g_block, g_menu) blocks the game's input; state 2 ends at the first poll with nothing held. A stick +// axis counts as held as well: with the walk key the keyboard's sticks move part way without setting a direction bit, so +// a gate that watched the buttons alone would let that movement through the moment the menu or the text entry closes. +static int blocked(volatile int32_t *flag, uint64_t buttons, const int32_t *axes) { int32_t k = *flag; - return k && !(k == 2 && !buttons && plat_cas32(flag, 0, 2) == 2); + int held = buttons != 0; + for (int i = 0; i < 4; i++) held |= axes[i] != 0; + return k && !(k == 2 && !held && plat_cas32(flag, 0, 2) == 2); } void input_poll(uint64_t *buttons, int32_t *lx, int32_t *ly, int32_t *rx, int32_t *ry) { poll_merged(buttons, lx, ly, rx, ry); - int text = blocked(&g_block, *buttons), menu = blocked(&g_menu, *buttons); + const int32_t axes[4] = {*lx, *ly, *rx, *ry}; + int text = blocked(&g_block, *buttons, axes), menu = blocked(&g_menu, *buttons, axes); if (!text && !menu) return; *buttons = 0; *lx = *ly = *rx = *ry = 0; } diff --git a/runtime/nvn_api_obj.c b/runtime/nvn_api_obj.c index 5e3b9b72..6d3c3fcf 100644 --- a/runtime/nvn_api_obj.c +++ b/runtime/nvn_api_obj.c @@ -323,8 +323,11 @@ NVN_API(nvnTextureReadTexelsStrided) { read_texels(c, 1); } NVN_API(nvnTextureWriteTexels) { write_texels(c, 0); } NVN_API(nvnTextureWriteTexelsStrided) { write_texels(c, 1); } // void nvnTexture{Flush,Invalidate}Texels(const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r): texel -// access is coherent -NVN_API(nvnTextureFlushTexels) { (void)c; } +// access is coherent; flush marks the storage as written for re-upload +NVN_API(nvnTextureFlushTexels) { + const tex_t *t = arg_p(c, 0); + if (t && t->mem && !nvn_gpu_only((uintptr_t)t->mem)) nvn_guest_written((uintptr_t)t->mem, t->L.size); +} NVN_API(nvnTextureInvalidateTexels) { (void)c; } // ---------- Samplers diff --git a/runtime/perf.c b/runtime/perf.c index 3db4b6ed..ef04ede5 100644 --- a/runtime/perf.c +++ b/runtime/perf.c @@ -183,6 +183,9 @@ static uint64_t ft_frame(uint64_t frame, uint64_t now, uint64_t d) { // process memory, and which side holds the frame back. pace: the frame waited for pacing (vsync, the frame limiter) at // least 15% of the window; gpu: the GPU was busy for 80% of the frame interval or more (without timestamps: threads // waited for fences for half the interval or more); cpu otherwise. +// With calibrated timestamps a third line splits the GPU's idle time between timed parts: the part was submitted only +// after the previous one had ended (the GPU waited for the CPU's replay or the game), or it was already queued (untimed +// work such as uploads and presents ran there, or a wait held it). static void perf_gpu_line(uint64_t frame, double interval_ms, double pace_share, uint64_t n) { PerfGpu g; memset(&g, 0, sizeof g); @@ -205,6 +208,11 @@ static void perf_gpu_line(uint64_t frame, double interval_ms, double pace_share, rt_log("perf: frame %llu: %s; fence wait per frame: render thread %.2f ms, other threads %.2f ms; %s; ram working set " "%.0f MiB, private %.0f MiB; bound: %s", (unsigned long long)frame, gs, rw, ow, vs, ws, priv, bound); + if (g.have_gpu && g.have_idle) + rt_log("perf: frame %llu: gpu idle per frame %.2f ms between timed parts: %.2f ms waiting for the next submission, " + "%.2f ms with it queued (untimed work or waits); %.1f gaps over 0.5 ms, longest %.2f ms; submit to start avg %.2f ms", + (unsigned long long)frame, g.idle_ns / 1e6 / n, g.starved_ns / 1e6 / n, (g.idle_ns - g.starved_ns) / 1e6 / n, + (double)g.idle_gaps / n, g.max_gap_ns / 1e6, g.gpu_parts ? g.queued_ns / 1e6 / g.gpu_parts : 0.0); } int rt_ab(void) { diff --git a/runtime/platform/linux.c b/runtime/platform/linux.c index 23a9f7fd..136b43c4 100644 --- a/runtime/platform/linux.c +++ b/runtime/platform/linux.c @@ -1247,6 +1247,8 @@ int plat_os_version(char *out, size_t cap) { return snprintf(out, cap, "%s (Linux %s)", name, u.release) < (int)cap; } +int plat_is_wine(void) { return 0; } + int plat_stderr_path(char *out, size_t cap) { struct stat st; if (fstat(2, &st) || !S_ISREG(st.st_mode)) return 0; diff --git a/runtime/platform/platform.h b/runtime/platform/platform.h index a6f4a5be..66ac1870 100644 --- a/runtime/platform/platform.h +++ b/runtime/platform/platform.h @@ -367,6 +367,7 @@ uint32_t plat_thread_id(void); // id of the calling t int plat_font_path(const char *lang, char *out, size_t cap); // The operating system and its build ("Windows 10.0.28000") into out; 0 when unknown. int plat_os_version(char *out, size_t cap); +int plat_is_wine(void); // 1 when the Windows build runs under Wine; 0 elsewhere // The file the process's stderr (the log) goes to, as a UTF-8 path; 0 when stderr is not a file (a console, a pipe). int plat_stderr_path(char *out, size_t cap); diff --git a/runtime/platform/win32.c b/runtime/platform/win32.c index bbd8aa56..b32c644d 100644 --- a/runtime/platform/win32.c +++ b/runtime/platform/win32.c @@ -1191,6 +1191,11 @@ int plat_os_version(char *out, size_t cap) { return 1; } +int plat_is_wine(void) { + HMODULE nt = GetModuleHandleW(L"ntdll.dll"); + return nt && GetProcAddress(nt, "wine_get_version") != NULL; +} + int plat_stderr_path(char *out, size_t cap) { HANDLE h = GetStdHandle(STD_ERROR_HANDLE); wchar_t w[WPATH]; diff --git a/runtime/resolution.c b/runtime/resolution.c index 1d8e2355..d042ef8b 100644 --- a/runtime/resolution.c +++ b/runtime/resolution.c @@ -43,6 +43,7 @@ #define SCENE_H 900 #define DTA_KIB 113664u // DynamicTextureAllocatorMem, shipped #define SCENE_MAX 4096 // per side, as MacNativeGames' TOTK_RENDER_RES +#define SHADOW_BASE 960 // the sun shadow map's base size (its cascades: a D16 array of 4 layers) static struct { int decided; @@ -50,6 +51,7 @@ static struct { int scene_set; // the scene differs from the game's 5/6 of the output int heapdef_ok; // romfs.c serves the grown HeapDef int on; // res_init switched the patch sets on + unsigned shadow; // fixed sun shadow map size (TOTK_SHADOW_RES), 0: the game's own } R; static void decide(void) { @@ -83,19 +85,31 @@ static void decide(void) { if (fit < 1.0) R.scene_set = 1; R.sw = (unsigned)(sw * fit + 0.5); R.sh = (unsigned)(sh * fit + 0.5); + const char *sr = cfg_value("TOTK_SHADOW_RES"); + if (strcmp(sr, "game")) { + unsigned n = (unsigned)atoi(sr); + if (n >= 256 && n <= 4096 && !(n & 7)) R.shadow = n; + else rt_log("resolution: TOTK_SHADOW_RES=%s is not game or a size 256..4096 (a multiple of 8); game", sr); + } +} + +// Texture heap for a sun shadow map above the base size: the four layers of 2 bytes, three maps in flight (the engine +// makes the array anew each frame). +static uint64_t shadow_kib(void) { + return R.shadow > SHADOW_BASE ? ((uint64_t)R.shadow * R.shadow - SHADOW_BASE * SHADOW_BASE) * 8 * 3 / 1024 : 0; } static int bigger(void) { decide(); - return R.ow != BASE_W || R.oh != BASE_H || R.scene_set; + return R.ow != BASE_W || R.oh != BASE_H || R.scene_set || shadow_kib(); } uint64_t res_heap_extra_kib(void) { if (!bigger()) return 0; double grow = fmax(0.0, (double)R.ow * R.oh / (BASE_W * BASE_H) - 1.0) + fmax(0.0, (double)R.sw * R.sh / (SCENE_W * SCENE_H) - 1.0); - if (grow <= 0) return 0; - return (uint64_t)ceil(DTA_KIB * (1.0 + grow) / 1024.0) * 1024 - DTA_KIB; + uint64_t kib = grow > 0 ? (uint64_t)ceil(DTA_KIB * (1.0 + grow) / 1024.0) * 1024 - DTA_KIB : 0; + return kib + ((shadow_kib() + 1023) & ~1023ull); // whole MiB } uint64_t res_heap_extra_bytes(void) { return ((res_heap_extra_kib() << 10) + 0x1fffff) & ~0x1fffffull; } @@ -190,12 +204,32 @@ int res_heapdef_build(const void *in, size_t n, void **out, size_t *out_n) { void res_heapdef_served(void) { R.heapdef_ok = 1; } +// ---- sun shadow map size: each frame the engine sizes its shadow cascades to the area they cover, down to about 400 near +// the ground, which draws foliage shadows blocky, and each change of size moves the shadow texels, so leaf shadows jump. +// A fixed size (the shadow set, recomp/patches-v121.txt) keeps them still and sharper. Page +92. +static void shadow_init(uint8_t *page) { + if (!R.shadow) { + rt_log("resolution: sun shadow map sized by the game (TOTK_SHADOW_RES=game)"); + return; + } + int set = rt_patch_set("shadow"); + if (set < 0) { + rt_log("resolution: the generated C has no shadow patch set: the game sizes its shadow map"); + return; + } + int32_t n = (int32_t)R.shadow; + memcpy(page + 92, &n, 4); + rt_patch_switch(set, 1); + rt_log("resolution: sun shadow map %ux%u (TOTK_SHADOW_RES)", R.shadow, R.shadow); +} + // ---- start: the patch page and the sets static uint64_t pack(unsigned w, unsigned h) { return (uint64_t)h << 32 | w; } void res_init(void) { if (!bigger()) { rt_log("resolution: the game's own (output 1920x1080, scene 1600x900)"); + shadow_init(rt_patch_page()); return; } int output = rt_patch_set("output"), scene = rt_patch_set("scene"); @@ -238,6 +272,7 @@ void res_init(void) { } else { rt_patch_switch(aspect, 0); // the page holds no ratio } + shadow_init(page); rt_log("resolution: output %ux%u, scene %ux%u%s; texture heap +%llu KiB", R.ow, R.oh, R.sw, R.sh, scene_on ? "" : " (the game's own 5/6 of the output)", (unsigned long long)res_heap_extra_kib()); } diff --git a/runtime/ui/menu.cpp b/runtime/ui/menu.cpp index f3e39b11..77f9d4b6 100644 --- a/runtime/ui/menu.cpp +++ b/runtime/ui/menu.cpp @@ -505,6 +505,15 @@ static void row_render_scale(void) { opt_choice_list("TOTK_RENDER_SCALE", UI_MENU_RENDER_SCALE, vals, names, N, UI_MENU_RENDER_SCALE_DESC); } +static void row_shadow_res(void) { + static const char *const vals[] = {"game", "512", "768", "1024", "1536", "2048"}; + enum { N = sizeof vals / sizeof *vals }; + const char *names[N]; + names[0] = menu_text(UI_MENU_SHADOW_RES_GAME); + for (int i = 1; i < N; i++) names[i] = vals[i]; + opt_choice_list("TOTK_SHADOW_RES", UI_MENU_SHADOW_RES, vals, names, N, UI_MENU_SHADOW_RES_DESC); +} + static void row_gpu(void) { const char *vals[MAX_GPUS + 1], *names[MAX_GPUS + 1]; char idx[MAX_GPUS][4]; @@ -623,6 +632,7 @@ static void tab_screen(float scale) { opt_group(UI_MENU_GRAPHICS); row_output_res(); row_render_scale(); + row_shadow_res(); static const UiTextId up[] = {UI_MENU_UPSCALER_FSR, UI_MENU_UPSCALER_BILINEAR}; opt_choice("TOTK_UPSCALER", UI_MENU_UPSCALER, up, 2, UI_MENU_UPSCALER_DESC); opt_slider("TOTK_SHARPNESS", UI_MENU_SHARPNESS, 0, 100, UI_MENU_SHARPNESS_DESC, 5, "%.0f %%"); @@ -668,6 +678,41 @@ static const char *pad_layout_note(void) { return note; } +// The action a key presses, or -1: the walk key is a key of its own, but Shift is B in the default bindings, and a +// walk key that also presses something walks and does that something at the same time. The bindings come from a +// snapshot, and every key of an action is looked at: binds_refresh rebuilds the live table on the polling thread while +// the menu draws, and an action may hold more than the two keys its cells show. +static int key_action(const InputMap *m, int key) { + for (int a = 0; a < ACTION_COUNT; a++) { + uint8_t keys[INPUT_BINDING_MAX]; + int n = input_map_keys_of(m, a, keys, INPUT_BINDING_MAX); + for (int i = 0; i < n; i++) + if (keys[i] == key) return a; + } + return -1; +} + +// The walk key as the handful of keys a player is likely to pick. "none" rather than an empty value closes it: an +// unset key falls back to the default in the settings table, the same as a fresh installation. totk.ini takes any name +// input_map.c knows (walks of the keyboard sticks, not of the gamepad). A key the current bindings also use says which +// action it presses. +static void row_walk_key(void) { + static const char *const keys[] = {"none", "lshift", "lalt", "lctrl", "tab", "capslock"}; + const char *names[sizeof keys / sizeof *keys] = {menu_text(UI_MENU_OFF), "Shift", "Alt", "Ctrl", "Tab", "Caps Lock"}; + const char *cur = cfg_value("TOTK_WALK_KEY"); + static char note[64]; + const char *n = nullptr; + InputMap m; + input_map_snapshot(&m); + for (int k = KEY_A; k < KEY_COUNT && !n; k++) { + if (!input_key_name(k) || _stricmp(input_key_name(k), cur)) continue; + int a = key_action(&m, k); + if (a >= 0 && input_action_name(a)) snprintf(note, sizeof note, menu_text(UI_MENU_WALK_KEY_TAKEN), input_action_name(a)), n = note; + } + opt_choice_list("TOTK_WALK_KEY", UI_MENU_WALK_KEY, keys, names, sizeof keys / sizeof *keys, UI_MENU_WALK_KEY_DESC, + -1, n); +} + static void tab_controls(float scale) { opt_begin("##controls", scale); static const UiTextId layout[] = {UI_MENU_PAD_LAYOUT_AUTO, UI_MENU_PAD_LAYOUT_NINTENDO, UI_MENU_PAD_LAYOUT_XBOX, @@ -679,6 +724,8 @@ static void tab_controls(float scale) { opt_toggle("TOTK_RUMBLE_TRIGGERS", UI_MENU_RUMBLE_TRIGGERS, UI_MENU_RUMBLE_TRIGGERS_DESC); opt_slider("TOTK_STICK_DEADZONE", UI_MENU_STICK_DEADZONE, 0, 50, UI_MENU_STICK_DEADZONE_DESC, 1, "%.0f %%"); opt_slider("TOTK_MOUSE_SENS", UI_MENU_MOUSE_SENS, 0, 10, UI_MENU_MOUSE_SENS_DESC, 0.1f, "%.1f"); + row_walk_key(); + opt_slider("TOTK_WALK_SCALE", UI_MENU_WALK_SCALE, 0.2f, 0.9f, UI_MENU_WALK_SCALE_DESC, 0.01f, "%.2f"); binds_rows(scale); opt_end(); binds_after_page(); diff --git a/runtime/ui_text.c b/runtime/ui_text.c index 35a436a2..2ca5a1e9 100644 --- a/runtime/ui_text.c +++ b/runtime/ui_text.c @@ -69,6 +69,11 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_MENU_ASPECT_FIT] = "Black bars", [UI_MENU_ASPECT_STRETCH] = "Stretch", [UI_MENU_FOV] = "Field of view", [UI_MENU_FOV_DESC] = "How much of the world the camera shows, in percent of the game's own. Game version 1.4.3 only.", + [UI_MENU_SHADOW_RES] = "Shadow map size", + [UI_MENU_SHADOW_RES_DESC] = "Size of the sun's shadow map. The game's own size changes with the view, down to about 400 " + "near the ground, so leaf shadows look blocky and jump; a fixed size keeps them still and " + "sharper. Above 1024 the shadows cost more GPU time and video memory.", + [UI_MENU_SHADOW_RES_GAME] = "Game (changes)", [UI_MENU_SHARPNESS] = "FSR sharpness", [UI_MENU_SHARPNESS_DESC] = "Sharpening after FSR scaling; 0 turns it off.", [UI_MENU_COLOR] = "Colour", @@ -136,6 +141,13 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_MENU_STICK_DEADZONE_DESC] = "How far a gamepad stick moves before the game sees it. Raise it if the character or the camera moves on its own.", [UI_MENU_MOUSE_SENS] = "Mouse camera sensitivity", [UI_MENU_MOUSE_SENS_DESC] = "How fast the mouse turns the camera.", + [UI_MENU_WALK_KEY] = "Walk key", + [UI_MENU_WALK_KEY_DESC] = "Hold it to tilt the keyboard's sticks part of the way, so the keyboard walks instead of " + "runs. The walk scale sets how far. A controller and the mouse are not affected.", + [UI_MENU_WALK_KEY_TAKEN] = "also %s", + [UI_MENU_WALK_SCALE] = "Walk scale", + [UI_MENU_WALK_SCALE_DESC] = "The tilt the keyboard's sticks get while the walk key is held. Half is a walk, the " + "full tilt a run.", [UI_BINDS_HELP] = "Select a cell and press A or Enter, then the key, mouse button or controller button. Delete or " "X clears a cell. A key in use moves here; a controller button swaps with the one it replaces.", @@ -330,6 +342,11 @@ static const char *const text_ru[UI_TEXT_COUNT] = { [UI_MENU_ASPECT_FIT] = "Чёрные полосы", [UI_MENU_ASPECT_STRETCH] = "Растянуть", [UI_MENU_FOV] = "Поле зрения", [UI_MENU_FOV_DESC] = "Сколько мира показывает камера, в процентах от игрового. Только для версии игры 1.4.3.", + [UI_MENU_SHADOW_RES] = "Размер карты теней", + [UI_MENU_SHADOW_RES_DESC] = "Размер карты теней от солнца. Собственный размер игры меняется вместе с видом, у земли до " + "примерно 400, поэтому тени листвы выглядят квадратными и дёргаются; постоянный размер делает " + "их ровными и чётче. Выше 1024 тени требуют больше времени GPU и видеопамяти.", + [UI_MENU_SHADOW_RES_GAME] = "Как в игре (меняется)", [UI_MENU_SHARPNESS] = "Резкость FSR", [UI_MENU_SHARPNESS_DESC] = "Повышение резкости после масштабирования FSR; 0 выключает его.", [UI_MENU_COLOR] = "Цвет", @@ -398,6 +415,13 @@ static const char *const text_ru[UI_TEXT_COUNT] = { [UI_MENU_STICK_DEADZONE_DESC] = "Насколько нужно отклонить стик геймпада, чтобы игра это заметила. Увеличьте, если персонаж или камера двигаются сами.", [UI_MENU_MOUSE_SENS] = "Чувствительность мыши", [UI_MENU_MOUSE_SENS_DESC] = "Как быстро мышь поворачивает камеру.", + [UI_MENU_WALK_KEY] = "Клавиша шага", + [UI_MENU_WALK_KEY_DESC] = "Удерживайте: стики клавиатуры отклоняются частично, и персонаж идёт, а не бежит. " + "Насколько — задаёт сила шага. Контроллер и мышь не затрагиваются.", + [UI_MENU_WALK_KEY_TAKEN] = "также %s", + [UI_MENU_WALK_SCALE] = "Сила шага", + [UI_MENU_WALK_SCALE_DESC] = "Отклонение стиков клавиатуры при удержании клавиши шага. Половина — шаг, полное " + "отклонение — бег.", [UI_BINDS_HELP] = "Выберите ячейку и нажмите A или Enter, затем клавишу, кнопку мыши или кнопку контроллера. " "Delete или X очищает ячейку. Занятая клавиша переходит сюда; кнопка контроллера меняется " diff --git a/runtime/ui_text.h b/runtime/ui_text.h index 2d5670f1..41502d88 100644 --- a/runtime/ui_text.h +++ b/runtime/ui_text.h @@ -37,6 +37,7 @@ typedef enum { UI_MENU_UPSCALER, UI_MENU_UPSCALER_DESC, UI_MENU_UPSCALER_FSR, UI_MENU_UPSCALER_BILINEAR, UI_MENU_ASPECT, UI_MENU_ASPECT_DESC, UI_MENU_ASPECT_FIT, UI_MENU_ASPECT_STRETCH, UI_MENU_FOV, UI_MENU_FOV_DESC, + UI_MENU_SHADOW_RES, UI_MENU_SHADOW_RES_DESC, UI_MENU_SHADOW_RES_GAME, UI_MENU_SHARPNESS, UI_MENU_SHARPNESS_DESC, UI_MENU_COLOR, // group heading over the colour rows UI_MENU_COLOR_PRESET, UI_MENU_COLOR_PRESET_DESC, UI_MENU_COLOR_DEFAULT, UI_MENU_COLOR_VIVID, UI_MENU_COLOR_CLARITY, @@ -68,6 +69,7 @@ typedef enum { UI_MENU_PAD_LAYOUT_PROMPTS_MISSING, UI_MENU_PAD_LAYOUT_PROMPTS_RESTART, UI_MENU_STICK_DEADZONE, UI_MENU_STICK_DEADZONE_DESC, UI_MENU_MOUSE_SENS, UI_MENU_MOUSE_SENS_DESC, + UI_MENU_WALK_KEY, UI_MENU_WALK_KEY_DESC, UI_MENU_WALK_KEY_TAKEN, UI_MENU_WALK_SCALE, UI_MENU_WALK_SCALE_DESC, // Controls tab: the remapping rows (ui/binds_page.cpp) UI_BINDS_HELP, UI_BINDS_COL_KEY1, UI_BINDS_COL_KEY2, UI_BINDS_COL_PAD, diff --git a/runtime/vk.h b/runtime/vk.h index 51b66104..b45692d8 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -33,7 +33,8 @@ typedef struct VkCtx { int has_custom_border_color, has_eds, has_eds2, has_eds3, has_vertex_input_dynamic_state, has_conditional_rendering, has_transform_feedback, has_provoking_vertex, has_device_fault, has_memory_budget, has_robustness2, has_buffer_device_address, has_fragment_shader_interlock, has_buffer_marker, - has_astc_ldr; // textureCompressionASTC_LDR (enabled when reported) + has_astc_ldr, // textureCompressionASTC_LDR (enabled when reported) + has_calibrated; // VK_KHR/EXT_calibrated_timestamps (GPU idle split in the perf line) // Feedback loops (maxwell.md 1.9, vk_exec_draw.c feedback_prepare): a draw samples a target of its open pass that // an earlier draw of the pass wrote. TOTK_FEEDBACK=0: VK_FEEDBACK_OFF (sampled in GENERAL without a barrier, the // old behaviour); VK_FEEDBACK_LAYOUT with VK_EXT_attachment_feedback_loop_layout and dynamicRenderingLocalRead diff --git a/runtime/vk_device.c b/runtime/vk_device.c index e92e6de0..e523f033 100644 --- a/runtime/vk_device.c +++ b/runtime/vk_device.c @@ -247,6 +247,9 @@ static int create_device(void) { CHAIN(lr, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DYNAMIC_RENDERING_LOCAL_READ_FEATURES_KHR); } if (has_ext(ep, ne, VK_EXT_MEMORY_BUDGET_EXTENSION_NAME)) { g_vk.has_memory_budget = 1; ext[n_ext++] = VK_EXT_MEMORY_BUDGET_EXTENSION_NAME; } + // GPU timestamps on the CPU clock (vk_sync.c): the perf line splits GPU idle time into waiting for a submission or not. + if (has_ext(ep, ne, VK_KHR_CALIBRATED_TIMESTAMPS_EXTENSION_NAME)) { g_vk.has_calibrated = 2; ext[n_ext++] = VK_KHR_CALIBRATED_TIMESTAMPS_EXTENSION_NAME; } + else if (has_ext(ep, ne, VK_EXT_CALIBRATED_TIMESTAMPS_EXTENSION_NAME)) { g_vk.has_calibrated = 1; ext[n_ext++] = VK_EXT_CALIBRATED_TIMESTAMPS_EXTENSION_NAME; } // GPU breadcrumbs for a lost device (vk_crumbs.c): AMD's drivers and RADV report it. if (has_ext(ep, ne, VK_AMD_BUFFER_MARKER_EXTENSION_NAME)) { g_vk.has_buffer_marker = 1; ext[n_ext++] = VK_AMD_BUFFER_MARKER_EXTENSION_NAME; } // Graphics pipeline library: chained last so it can be dropped again when vk_caps_gpl declines it. diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index fba5677a..39ec43e3 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -827,7 +827,7 @@ static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) { // - GPU -> guest: the replay notes each GPU write into CPU-visible memory at record time (vk_mem_gpu_pending) and // reports it at the fence (vk_gpu_written), which copies those bytes back into guest memory. Until then the CPU // copies skip them: a page the CPU and the GPU both write keeps the GPU's bytes. -// TOTK_SHADOW=1/0 forces the mode on or off; the default is on for Linux. +// TOTK_SHADOW=1/0 forces the mode on or off; the default is on for Linux and Wine. typedef struct { uint64_t lo, hi; } SRange; static PlatMutex shadow_lock = PLAT_MUTEX_INIT; // after pool_lock; before gen_lock static PoolRec **stouch; // shadow_lock: pools resolved since the last sync @@ -1187,7 +1187,7 @@ static void mirror_mode(int discrete) { if (freeze_idle < 0) freeze_idle = 0; const char *sh = cfg_get("TOTK_SHADOW"); #ifdef _WIN32 - shadow_on = sh && *sh && strcmp(sh, "0"); + shadow_on = sh && *sh ? strcmp(sh, "0") != 0 : plat_is_wine(); #else shadow_on = !sh || !*sh || strcmp(sh, "0"); #endif diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index feba5764..b0953383 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -2569,7 +2569,8 @@ static struct { uint8_t clamp, poly, minus_w, samples; // library state (GKey fields) double t0, ms[3]; // ms: build time per kind (vertex, fragment, compute) // warmed: driver cache only; nolib: vertex and fragment variants skipped with the graphics pipeline library off - uint64_t built[3], have[3], fail[3], warmed[3], noshader, nolib, done, cs_draw; + // skipped: variants of the skip list (SK below) + uint64_t built[3], have[3], fail[3], warmed[3], noshader, nolib, skipped, done, cs_draw; double warm_ms[3]; volatile int32_t kept; // libraries entered in libs[] so far int32_t keep_n; @@ -2577,6 +2578,134 @@ static struct { PlatMutex stat; } LB = {.stat = PLAT_MUTEX_INIT}; +// ---- preparation that outlives a driver crash. A bug in the driver's shader compiler can kill the process inside +// vkCreate*Pipelines: NVIDIA 615.78.08 on an RTX 5080 crashed on the same vertex library at the same point of every +// preparation, natively and under Wine, so preparation never finished (issue #30 and a report on PR #60). +// While lb_warm_prepared builds, slot k of /shader-prepare-journal.bin holds the variant its worker k has handed to +// the driver (input id and stage; zeros between variants). The writes go straight to the OS, so they outlast the +// process, and a run that finishes deletes the file. The next preparation or boot (skip_init) moves the variants a +// leftover journal names into /shader-prepare-skip.txt with this GPU and driver version, and preparation and the +// boot prebuild (lb_worker) skip what that file lists for them. The variants the other workers were building at the +// crash are skipped along with the culprit; like it, they build at their first draw, where the driver can still crash. +// Another driver version builds them again, and so does deleting the file. tools/prepare_shaders.py runs the build +// again (--warm-shaders) after a run that left a journal. +#define JR_SLOT 64 // bytes per journal slot: input id, stage, zeros +#define JR_SLOTS 64 // lb_warm_prepared's thread cap + +static struct { + PlatFile *journal; // while lb_warm_prepared builds + uint8_t (*iid)[32]; // the skip list's variants for this GPU and driver + size_t n; + int loaded; +} SK; + +static int cache_dir(char *dir, size_t n) { // the folder of pipeline-cache.bin and shader-prepared.bin + snprintf(dir, n, "%s", P.cache_path); + char *cut = plat_last_sep(dir); + if (!cut) return 0; + *cut = 0; + return 1; +} + +static void gpu_key(char *out, size_t n) { // vendor:device:driver version, as the skip list names a GPU and driver + const VkPhysicalDeviceProperties *p = &vk_ctx()->props; + snprintf(out, n, "%04x:%04x:%08x", p->vendorID, p->deviceID, p->driverVersion); +} + +static const char *stage_name(int st) { + return st == GFX_STAGE_VS ? "vertex" : st == GFX_STAGE_FS ? "fragment" : st == GFX_STAGE_CS ? "compute" : "other"; +} + +static int hex_digit(int c) { + return c >= '0' && c <= '9' ? c - '0' : c >= 'a' && c <= 'f' ? c - 'a' + 10 : c >= 'A' && c <= 'F' ? c - 'A' + 10 : -1; +} + +// A journal a crashed run left into the skip list, then the skip list's variants for this GPU and driver into SK. +static void skip_init(const char *dir) { + if (SK.loaded) return; + SK.loaded = 1; + char jp[PLAT_PATH_MAX * 2 + 48], sp[PLAT_PATH_MAX * 2 + 48], key[32]; + snprintf(jp, sizeof jp, "%s/shader-prepare-journal.bin", dir); + snprintf(sp, sizeof sp, "%s/shader-prepare-skip.txt", dir); + gpu_key(key, sizeof key); + uint64_t jn = 0, sn = 0; + uint8_t *j = plat_file_read(jp, &jn); + char *s = (char *)plat_file_read(sp, &sn); + int keep_journal = 0; + if (j) { + static const char head[] = + "# Shader variants the graphics driver was building when the process died during preparation\n" + "# (runtime/vk_pipeline.c skip_init): input id, stage, GPU vendor:device:driver version. Preparation skips\n" + "# them on that GPU and driver. Delete this file to build them again.\n"; + size_t cap = (s ? sn : sizeof head) + (jn / JR_SLOT + 1) * 128, len = 0; + char *out = malloc(cap); + int added = 0; + if (out) { + len = s ? sn : sizeof head - 1; + memcpy(out, s ? s : head, len); + if (len && out[len - 1] != '\n') out[len++] = '\n'; + for (uint64_t o = 0; o + 33 <= jn; o += JR_SLOT) { + int set = 0; + for (int b = 0; b < 32; b++) set |= j[o + b]; + if (!set) continue; + char hex[65]; + for (int b = 0; b < 32; b++) snprintf(hex + 2 * b, 3, "%02x", j[o + b]); + len += (size_t)snprintf(out + len, cap - len, "%s %s %s\n", hex, stage_name(j[o + 32]), key); + added++; + } + out[len] = 0; // parsed below as the list's text + } + if (added && plat_file_write_atomic(sp, out, len)) { + rt_log("vk_pipeline: the last shader preparation died while the driver built %d variants; %s now lists them " + "and preparation skips them on this GPU and driver (%s)", added, sp, key); + free(s), s = out, sn = len, out = NULL; + } else if (added) { + rt_log("vk_pipeline: cannot write %s; %s stays for the next run", sp, jp); + keep_journal = 1; + } + free(out); + free(j); + } + if (!keep_journal) plat_file_delete(jp); + for (char *line = s, *next; line && *line; line = next) { + next = strchr(line, '\n'); + if (next) *next++ = 0; + char hex[65], st[16], k[32]; + uint8_t iid[32]; + int ok = line[0] != '#' && sscanf(line, "%64s %15s %31s", hex, st, k) == 3 && strlen(hex) == 64 && + !strcmp(k, key); + for (int b = 0; ok && b < 32; b++) { + int hi = hex_digit(hex[2 * b]), lo = hex_digit(hex[2 * b + 1]); + if (hi < 0 || lo < 0) ok = 0; + else iid[b] = (uint8_t)(hi << 4 | lo); + } + uint8_t (*grown)[32] = ok ? realloc(SK.iid, (SK.n + 1) * sizeof *SK.iid) : NULL; + if (grown) memcpy(grown[SK.n++], iid, 32), SK.iid = grown; + } + free(s); + if (SK.n) rt_log("vk_pipeline: %zu shader variants skipped ahead of play on this GPU and driver (%s)", SK.n, sp); +} + +static int skip_listed(const uint8_t iid[32]) { + for (size_t i = 0; i < SK.n; i++) + if (!memcmp(SK.iid[i], iid, 32)) return 1; + return 0; +} + +static void skip_count(void) { + plat_mutex_lock(&LB.stat); + LB.skipped++; + plat_mutex_unlock(&LB.stat); +} + +// Journal slot `slot`: the variant e its worker hands to the driver next, or zeros (e NULL) when the build returned. +static void journal_mark(int slot, const GfxWarmEnt *e) { + if (!SK.journal || slot < 0 || slot >= JR_SLOTS) return; + uint8_t rec[JR_SLOT] = {0}; + if (e) memcpy(rec, e->iid, 32), rec[32] = e->stage; + plat_file_pwrite(SK.journal, rec, sizeof rec, (uint64_t)slot * JR_SLOT); +} + VkPipeline vk_pipeline_compute(const GfxShader *cs, const VkPipeInfo **info) { if (info) *info = NULL; if (!cs || !ensure_init()) return VK_NULL_HANDLE; @@ -2749,7 +2878,8 @@ static void lb_report(const char *how) { rt_log("vk_pipeline: stage library prebuild %s: %llu of %zu variants in %.0f ms; kept: vertex %llu built (avg %.1f ms), " "%llu there; fragment %llu built (avg %.1f ms), %llu there; compute %llu built (avg %.1f ms), %llu there; driver " "cache only: vertex %llu (avg %.1f ms), fragment %llu (avg %.1f ms), compute %llu (avg %.1f ms); %llu failed, %llu not in the shader cache, " - "%llu vertex/fragment skipped (graphics pipeline library off); process working set %.0f MB", how, (unsigned long long)LB.done, LB.n, + "%llu vertex/fragment skipped (graphics pipeline library off), %llu skipped (the driver crashed on them before, " + "shader-prepare-skip.txt); process working set %.0f MB", how, (unsigned long long)LB.done, LB.n, now_ms() - LB.t0, (unsigned long long)LB.built[0], LB.built[0] ? LB.ms[0] / LB.built[0] : 0.0, (unsigned long long)LB.have[0], (unsigned long long)LB.built[1], LB.built[1] ? LB.ms[1] / LB.built[1] : 0.0, (unsigned long long)LB.have[1], (unsigned long long)LB.built[2], LB.built[2] ? LB.ms[2] / LB.built[2] : 0.0, @@ -2757,7 +2887,7 @@ static void lb_report(const char *how) { (unsigned long long)LB.warmed[1], LB.warmed[1] ? LB.warm_ms[1] / LB.warmed[1] : 0.0, (unsigned long long)LB.warmed[2], LB.warmed[2] ? LB.warm_ms[2] / LB.warmed[2] : 0.0, (unsigned long long)(LB.fail[0] + LB.fail[1] + LB.fail[2]), - (unsigned long long)LB.noshader, (unsigned long long)LB.nolib, (double)mem_proc); + (unsigned long long)LB.noshader, (unsigned long long)LB.nolib, (unsigned long long)LB.skipped, (double)mem_proc); plat_mutex_unlock_shared(&LB.stat); } @@ -2768,6 +2898,13 @@ static void lb_worker(void *arg) { for (;;) { size_t i = (size_t)plat_inc64(&LB.next) - 1; if (i >= LB.n || LB.stop) break; + if (skip_listed(LB.e[i].iid)) { + skip_count(); + plat_mutex_lock(&LB.stat); + LB.done++; + plat_mutex_unlock(&LB.stat); + continue; + } plat_thread_priority(PLAT_PRIO_NORMAL); // gfx_shader's cache lock, disk reads const GfxShader *sh = gfx_shader_load_input(LB.e[i].iid, 0); plat_thread_priority(PLAT_PRIO_IDLE); @@ -2805,6 +2942,8 @@ static void lb_start(const uint32_t pr_hist[16], const uint32_t fs_hist[4]) { const char *e = cfg_value("TOTK_LIB_PREBUILD"), *ke = cfg_get("TOTK_LIB_PREBUILD_KEEP"); const char *te = cfg_get("TOTK_LIB_PREBUILD_THREADS"); // no fixed default: a quarter of the CPUs when unset if (!A.on || !cfg_bool("TOTK_LIB_PREBUILD")) return; + char dir[PLAT_PATH_MAX * 2]; + if (cache_dir(dir, sizeof dir)) skip_init(dir); GfxWarmEnt *all = NULL; size_t total = gfx_shader_warm_list(&all), n = 0, n_cs = 0; LB.keep_n = ke && *ke ? atoi(ke) : 0; @@ -2847,16 +2986,20 @@ static void lb_stop(void) { LB.threads = 0; } -// Preparation: one prepared variant at a time into the driver's disk cache (private shader copy, nothing kept). +// Preparation: one prepared variant at a time into the driver's disk cache (private shader copy, nothing kept). arg is +// the worker's journal slot. static void lb_warm_worker(void *arg) { - (void)arg; + int slot = (int)(intptr_t)arg; for (;;) { size_t i = (size_t)plat_inc64(&LB.next) - 1; if (i >= LB.n) return; - int owned = 0; - const GfxShader *sh = gfx_shader_load_private(LB.e[i].iid, &owned); + int owned = 0, skip = skip_listed(LB.e[i].iid); + const GfxShader *sh = skip ? NULL : gfx_shader_load_private(LB.e[i].iid, &owned); int st = sh ? gfx_shader_stage(sh) : -1; - if (!sh) { + if (sh) journal_mark(slot, &LB.e[i]); // from here to its clearing below, a crash names this variant + if (skip) { + skip_count(); + } else if (!sh) { plat_mutex_lock(&LB.stat); LB.noshader++; plat_mutex_unlock(&LB.stat); @@ -2874,6 +3017,7 @@ static void lb_warm_worker(void *arg) { if (P.gpl) lb_library(sh, st == GFX_STAGE_VS, 0); else lb_skip_lib(); } + if (sh) journal_mark(slot, NULL); if (owned) gfx_shader_release_private(sh); plat_mutex_lock(&LB.stat); uint64_t d = ++LB.done; @@ -2894,23 +3038,36 @@ static void lb_warm_prepared(const char *dir) { memcpy(LB.e[i].iid, f + 16 + 36 * i + 4, 32), LB.e[i].stage = f[16 + 36 * i], LB.e[i].prio = f[16 + 36 * i + 1]; free(f); if (!LB.e) return; + skip_init(dir); + char jp[PLAT_PATH_MAX * 2 + 48]; + snprintf(jp, sizeof jp, "%s/shader-prepare-journal.bin", dir); + SK.journal = plat_fs_create(jp, NULL); // skip_init removed a leftover one + if (!SK.journal) rt_log("vk_pipeline: cannot create %s: a driver crash in this run will not be recorded", jp); LB.n = n, LB.idle = 0, LB.t0 = now_ms(); static const uint32_t no_pr[16], no_fs[4]; lb_state(no_pr, no_fs); int cpus = (int)plat_cpu_usable(), k = 0; - PlatThread *t[64]; - for (int i = 0; i < cpus && i < 64; i++) - if ((t[k] = plat_thread_new(lb_warm_worker, NULL, "lib warm"))) k++; - if (!k) lb_warm_worker(NULL); + PlatThread *t[JR_SLOTS]; + for (int i = 0; i < cpus && i < JR_SLOTS; i++) + if ((t[k] = plat_thread_new(lb_warm_worker, (void *)(intptr_t)k, "lib warm"))) k++; + if (!k) lb_warm_worker((void *)(intptr_t)0); for (int i = 0; i < k; i++) plat_thread_join(t[i]); + if (SK.journal) plat_file_close(SK.journal), SK.journal = NULL, plat_file_delete(jp); // every build returned lb_report("preparation done"); // with gpl off only compute variants are built: say so, rather than count the skipped ones as built + unsigned long long skipped = LB.skipped; + size_t tried = n - (size_t)skipped; if (LB.nolib) printf("prepare-shaders: %zu variants on %d threads in %.0f ms, %llu vertex/fragment variants skipped (graphics " - "pipeline library off, they build at their first draw)\n", n, k, now_ms() - LB.t0, (unsigned long long)LB.nolib); + "pipeline library off, they build at their first draw)\n", tried, k, now_ms() - LB.t0, + (unsigned long long)LB.nolib); else - printf("prepare-shaders: %zu variants built into the driver cache on %d threads in %.0f ms\n", n, k, now_ms() - LB.t0); - free(LB.e), LB.e = NULL, LB.n = 0, LB.next = 0, LB.done = 0; + printf("prepare-shaders: %zu variants built into the driver cache on %d threads in %.0f ms\n", tried, k, + now_ms() - LB.t0); + if (skipped) // installer/steps.py Prepare reads this line + printf("prepare-shaders: %llu variants skipped: the graphics driver crashed while building them in an earlier " + "run (shader-prepare-skip.txt); they build at their first draw\n", skipped); + free(LB.e), LB.e = NULL, LB.n = 0, LB.next = 0, LB.done = 0, LB.skipped = 0; } // totk.exe --prepare-shaders (tools/prepare_shaders.py): the device sets the translator's host answers (they are in every @@ -2922,10 +3079,7 @@ int vk_pipeline_prepare_shaders(void) { return 1; } char dir[PLAT_PATH_MAX * 2]; - snprintf(dir, sizeof dir, "%s", P.cache_path); - char *cut = plat_last_sep(dir); - if (!cut) return 1; - *cut = 0; + if (!cache_dir(dir, sizeof dir)) return 1; int cpus = (int)plat_cpu_usable(); const char *pe = cfg_get("TOTK_PREWARM_XLATE"); // no fixed default: half the CPUs, at most 16, when unset int procs = pe && *pe ? atoi(pe) : cpus / 2 > 16 ? 16 : cpus / 2; @@ -2944,10 +3098,7 @@ int vk_pipeline_warm_shaders(void) { return 1; } char dir[PLAT_PATH_MAX * 2]; - snprintf(dir, sizeof dir, "%s", P.cache_path); - char *cut = plat_last_sep(dir); - if (!cut) return 1; - *cut = 0; + if (!cache_dir(dir, sizeof dir)) return 1; char prp[PLAT_PATH_MAX * 2 + 32]; snprintf(prp, sizeof prp, "%s/shader-prepared.bin", dir); if (!plat_file_exists(prp)) { diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index 8f16f154..8a4acdcf 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -61,6 +61,14 @@ static volatile uint8_t ts_busy[TS_PAIRS]; // written until the completion threa static int ts_cur = -1; // the pair of the part being recorded, -1 untimed static uint64_t ts_last_end; // completion thread: end of the latest part read (ticks) static volatile int64_t ts_busy_ns, ts_parts, ts_missed; +// GPU idle (with calibrated timestamps): the gap before each timed part, split by whether the part was submitted after +// the previous one had ended (the GPU waited for the CPU: starved) or before (it waited for untimed work or a wait). +static uint64_t ts_sub_ns[TS_PAIRS]; // CPU time the part was handed to vkQueueSubmit2 +static int ts_unsent[4], ts_n_unsent; // pairs closed and not yet submitted (sub_lock) +static uint64_t ts_last_end_ns; // completion thread: ts_last_end on the CPU clock +static volatile int64_t ts_idle_ns, ts_starved_ns, ts_gaps, ts_max_gap_ns, ts_queued_ns; +static int64_t cal_tick, cal_ns, cal_at; // completion thread: a calibrated (GPU tick, CPU ns) pair and when it was taken +static PFN_vkGetCalibratedTimestampsKHR cal_fn; // On the first submission that wants timing (perf_gpu_timing: TOTK_TRACE=perf or the full performance overlay). static void ts_setup(VkCtx *c) { @@ -91,6 +99,49 @@ static void ts_setup(VkCtx *c) { typedef struct { uint32_t pair; } TsRead; +// Completion thread: refresh the GPU tick / CPU clock pair every 250 ms; 0 when there is no calibration. The host +// domain is the clock plat_now_ns reads (QPC on Windows, CLOCK_MONOTONIC elsewhere), plat_counter_hz ticks a second. +static int cal_refresh(VkCtx *c) { + if (!c->has_calibrated) return 0; + if (!cal_fn) { + cal_fn = (PFN_vkGetCalibratedTimestampsKHR)vkGetDeviceProcAddr(c->device, c->has_calibrated == 2 ? "vkGetCalibratedTimestampsKHR" + : "vkGetCalibratedTimestampsEXT"); + if (!cal_fn) { c->has_calibrated = 0; return 0; } + } + int64_t now = (int64_t)plat_now_ns(); + if (cal_at && now - cal_at < 250000000) return 1; + VkCalibratedTimestampInfoKHR ci[2] = {{.sType = VK_STRUCTURE_TYPE_CALIBRATED_TIMESTAMP_INFO_KHR, .timeDomain = VK_TIME_DOMAIN_DEVICE_KHR}, +#ifdef _WIN32 + {.sType = VK_STRUCTURE_TYPE_CALIBRATED_TIMESTAMP_INFO_KHR, .timeDomain = VK_TIME_DOMAIN_QUERY_PERFORMANCE_COUNTER_KHR}}; +#else + {.sType = VK_STRUCTURE_TYPE_CALIBRATED_TIMESTAMP_INFO_KHR, .timeDomain = VK_TIME_DOMAIN_CLOCK_MONOTONIC_KHR}}; +#endif + uint64_t v[2], dev; + if (cal_fn(c->device, 2, ci, v, &dev) != VK_SUCCESS) { + if (!cal_at) { // the domain pair is not supported: no split + rt_log("vk: calibrated timestamps failed: the perf line shows no GPU idle split"); + c->has_calibrated = 0; + } + return cal_at != 0; + } + cal_tick = (int64_t)(v[0] & ts_mask), cal_ns = (int64_t)((double)v[1] * 1e9 / (double)plat_counter_hz()), cal_at = now; + return 1; +} + +static int64_t tick_ns(uint64_t t) { return cal_ns + (int64_t)((double)((int64_t)t - cal_tick) * ts_period); } + +static void max64(volatile int64_t *m, int64_t v) { + int64_t o = __atomic_load_n(m, __ATOMIC_RELAXED); + while (v > o && !__atomic_compare_exchange_n(m, &o, v, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED)) {} +} + +// Just before vkQueueSubmit2 (sub_lock held): the parts closed since the last submit leave the CPU now. +static void ts_submitted(void) { + uint64_t now = plat_now_ns(); + for (int i = 0; i < ts_n_unsent; i++) ts_sub_ns[ts_unsent[i]] = now; + ts_n_unsent = 0; +} + static void ts_read_cb(void *arg) { // completion thread, in submission order TsRead *t = arg; VkCtx *c = vk_ctx(); @@ -98,6 +149,18 @@ static void ts_read_cb(void *arg) { // completion thread, in submission order if (vkGetQueryPoolResults(c->device, ts_pool, 2 * t->pair, 2, sizeof v, v, sizeof v[0], VK_QUERY_RESULT_64_BIT) == VK_SUCCESS) { uint64_t s = v[0] & ts_mask, e = v[1] & ts_mask; if (e >= s) { + if (cal_refresh(c)) { + int64_t sn = tick_ns(s), en = tick_ns(e), sub = (int64_t)ts_sub_ns[t->pair]; + if (ts_last_end_ns && sn > ts_last_end_ns) { + int64_t gap = sn - ts_last_end_ns, late = sub - ts_last_end_ns; + plat_add64(&ts_idle_ns, gap); + if (late > 0) plat_add64(&ts_starved_ns, late < gap ? late : gap); + if (gap > 500000) plat_inc64(&ts_gaps); + max64(&ts_max_gap_ns, gap); + } + if (sub && sn > sub) plat_add64(&ts_queued_ns, sn - sub); + if (en > ts_last_end_ns) ts_last_end_ns = en; + } uint64_t from = s > ts_last_end ? s : ts_last_end; if (e > from) { int64_t ns = (int64_t)((double)(e - from) * ts_period); @@ -132,6 +195,8 @@ static int ts_open(VkCommandBuffer cb) { static void ts_close(VkCommandBuffer cb, int pair, uint64_t fence) { if (pair < 0) return; vkCmdWriteTimestamp(cb, VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, ts_pool, 2 * (uint32_t)pair + 1); + ts_sub_ns[pair] = 0; + if (ts_n_unsent < 4) ts_unsent[ts_n_unsent++] = pair; TsRead *t = malloc(sizeof *t); if (!t) rt_fatal("vk: out of memory"); t->pair = (uint32_t)pair; @@ -159,6 +224,12 @@ static void gpu_stats(PerfGpu *g) { g->gpu_parts = (uint64_t)plat_xchg64(&ts_parts, 0); g->gpu_missed = (uint64_t)plat_xchg64(&ts_missed, 0); g->have_vram = vk_mem_vram(&g->vram_used, &g->vram_budget, &g->vram_exact); + g->have_idle = vk_ctx()->has_calibrated != 0 && cal_at != 0; + g->idle_ns = (uint64_t)plat_xchg64(&ts_idle_ns, 0); + g->starved_ns = (uint64_t)plat_xchg64(&ts_starved_ns, 0); + g->queued_ns = (uint64_t)plat_xchg64(&ts_queued_ns, 0); + g->idle_gaps = (uint64_t)plat_xchg64(&ts_gaps, 0); + g->max_gap_ns = (uint64_t)plat_xchg64(&ts_max_gap_ns, 0); } // Weak stand-in for vk_exec.c's replay: only empty recordings ({OP_BEGIN, OP_END}) can be submitted. @@ -303,6 +374,7 @@ VkCommandBuffer vk_sync_split(void) { {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO, .commandBuffer = cb}}; VkSubmitInfo2 si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO_2, .commandBufferInfoCount = pro ? 2 : 1, .pCommandBufferInfos = pro ? cbi : cbi + 1}; + ts_submitted(); vk_queue_lock(); VkResult r = vkQueueSubmit2(c->gfx_queue, 1, &si, VK_NULL_HANDLE); vk_queue_unlock(); @@ -366,6 +438,7 @@ uint64_t vk_sync_submit(const uint64_t *handles, int n) { .commandBufferInfoCount = (pro ? 1u : 0u) + (recorded ? 1u : 0u), .pCommandBufferInfos = pro ? cbi : cbi + 1, .signalSemaphoreInfoCount = 1, .pSignalSemaphoreInfos = &sig}; + ts_submitted(); vk_queue_lock(); VkResult r = vkQueueSubmit2(c->gfx_queue, 1, &si, VK_NULL_HANDLE); vk_queue_unlock(); diff --git a/tests/installer_gui_test.py b/tests/installer_gui_test.py index 4eb9c228..6eb17786 100644 --- a/tests/installer_gui_test.py +++ b/tests/installer_gui_test.py @@ -31,8 +31,8 @@ from installer import state as statefile # noqa: E402 from installer.events import Reporter # noqa: E402 from installer.runner import Context # noqa: E402 from installer.steps import all_steps # noqa: E402 -from installer.guilogic import (DONE, FAILED, PENDING, RUNNING, SKIPPED, Catalog, Form, Model, Process, # noqa: E402 - amount, describe_event, parse_line, windows_ui_language) +from installer.guilogic import (DONE, FAILED, PENDING, RUNNING, SKIPPED, UI_LANGUAGES, Catalog, Form, Model, # noqa: E402 + Process, amount, describe_event, parse_line, windows_ui_language) INSTALLED = ROOT / 'build' / 'install-test' FIXTURE = ROOT / 'build' / 'inst-gui' / 'e2e-install' @@ -233,12 +233,16 @@ class FormTest(unittest.TestCase): class CatalogTest(unittest.TestCase): - def test_russian_covers_english_with_the_same_placeholders(self): - en, ru = Catalog('en').english, Catalog('ru').texts - self.assertEqual(sorted(set(en) - set(ru)), []) - self.assertEqual(sorted(set(ru) - set(en)), []) - holes = [k for k in en if sorted(re.findall(r'\{\w+\}', en[k])) != sorted(re.findall(r'\{\w+\}', ru[k]))] - self.assertEqual(holes, []) + def test_every_language_covers_english_with_the_same_placeholders(self): + en = Catalog('en').english + for lang in UI_LANGUAGES[1:]: + with self.subTest(lang=lang): + other = Catalog(lang).texts + self.assertEqual(sorted(set(en) - set(other)), []) + self.assertEqual(sorted(set(other) - set(en)), []) + holes = [k for k in en + if sorted(re.findall(r'\{\w+\}', en[k])) != sorted(re.findall(r'\{\w+\}', other[k]))] + self.assertEqual(holes, []) def test_every_id_the_window_uses_exists(self): cat = Catalog('en') @@ -246,12 +250,12 @@ class CatalogTest(unittest.TestCase): 'process_died', 'result_ok', 'result_failed', 'result_cancelled')] ids += [f'step.{s}' for s in ('check', 'extract', 'lift', 'compile', 'prepare', 'cleanup')] ids += ['ui.state_' + n for n in ('started', 'done', 'skipped', 'pending', 'running', 'failed')] - for lang in ('en', 'ru'): + for lang in UI_LANGUAGES: cat.set_language(lang) self.assertEqual([i for i in ids if not cat.has(i)], []) def test_codes_become_names(self): - for lang, korean in (('en', 'Korean'), ('ru', 'Корейский')): + for lang, korean in (('en', 'Korean'), ('ru', 'Корейский'), ('pt-BR', 'Coreano')): cat = Catalog(lang) self.assertIn(korean, cat('warn.lang_needs_fonts', lang='ko')) self.assertNotIn(' ko ', cat('warn.lang_needs_fonts', lang='ko')) @@ -280,7 +284,7 @@ class CatalogTest(unittest.TestCase): self.assertEqual(describe_event(cat, {'type': 'start'}), '') def test_windows_language_is_a_known_value(self): - self.assertIn(windows_ui_language(), ('en', 'ru')) + self.assertIn(windows_ui_language(), UI_LANGUAGES) BREAK_CHILD = 'import time\nprint("ready", flush=True)\ntime.sleep(60)\n' diff --git a/tools/build_game.py b/tools/build_game.py index 65a2062f..99df700a 100644 --- a/tools/build_game.py +++ b/tools/build_game.py @@ -17,9 +17,11 @@ first (rpath $ORIGIN). docs/building.md lists the packages. Every runtime/*.c is compiled. build//modules.c carries the layout, the linked function tables, the expected SHA-256 of each ExeFS file (tools/data/expected-.json) and the default game paths -(--exefs, --romfs, --profile), which totk.exe uses unless its own command line names others. tools/romfs_index.py -writes build//romfs-index.bin and romfs_config.h from the RomFS directory's index.json (runtime/romfs.c reads -the index next to totk.exe). +(--exefs, --romfs, --profile), which totk.exe uses unless its own command line names others. --exefs and --romfs +default to extracted/ in the work folder: the [work] dir of config.local.toml, where tools/extract.py writes +them, else build/. tools/romfs_index.py writes build//romfs-index.bin and romfs_config.h from the RomFS +directory's index.json (runtime/romfs.c reads the index next to totk.exe). Without that index.json the build stops: +the exe would stop at start. --no-romfs builds it anyway. Private builds reuse the main build's objects. When --out differs from build/, the main build's generated objects are up to date (checked with a dry run of its build.ninja) and every header they include from runtime/ is @@ -241,6 +243,16 @@ def copy_romfs_patches(version, out): print(f'romfs patches: copied {name} from {src}') +def work_dir(): + """The work folder: the [work] dir of config.local.toml (relative to the checkout), where tools/extract.py writes + extracted/; build/ without the file or the key.""" + cfg = ROOT / 'config.local.toml' + if not cfg.is_file(): + return ROOT / 'build' + import tomllib + return ROOT / tomllib.loads(cfg.read_text(encoding='utf-8')).get('work', {}).get('dir', 'build') + + def c_string(s): return '"' + s.replace('\\', '\\\\').replace('"', '\\"') + '"' @@ -710,9 +722,12 @@ def main(): help='comma-separated runtime/*.c names to leave out (the link needs another definition of what they ' 'define)') ap.add_argument('--exefs', type=Path, - help="totk.exe's default ExeFS directory (default build/extracted//exefs)") + help="totk.exe's default ExeFS directory (default /extracted//exefs, the work " + "dir being the [work] dir of config.local.toml, else build)") ap.add_argument('--romfs', type=Path, help="totk.exe's default RomFS directory with romfs.bin and index.json " - '(default build/extracted//romfs)') + '(default /extracted//romfs)') + ap.add_argument('--no-romfs', action='store_true', + help='build even when the RomFS directory has no index.json (the exe then stops at start)') ap.add_argument('--profile', type=Path, help="totk.exe's default profile directory (default build/profile for v121, " "build/profile- otherwise, so trying another version leaves " "the main profile alone; saves load on 1.2.1 and 1.4.3 alike)") @@ -735,11 +750,16 @@ def main(): gen = (args.gen or ROOT / 'generated' / args.version).resolve() out = (args.out or ROOT / 'build' / args.version).resolve() - extracted = ROOT / 'build' / 'extracted' / args.version + extracted = None if args.exefs and args.romfs else work_dir() / 'extracted' / args.version paths = {'exefs': (args.exefs or extracted / 'exefs').resolve(), 'romfs': (args.romfs or extracted / 'romfs').resolve(), 'profile': (args.profile or ROOT / 'build' / ('profile' if args.version == 'v121' else f'profile-{args.version}')).resolve()} + has_romfs = (paths['romfs'] / 'index.json').is_file() + if not (has_romfs or args.no_romfs or args.make_runtime_lib or args.compile_only): + sys.exit(f'{paths["romfs"] / "index.json"} not found: without the RomFS the game stops at start. Run ' + 'tools/extract.py (it writes extracted/ into the [work] dir of config.local.toml), pass ' + '--romfs DIR, or --no-romfs to build without it') expected = args.expected if not expected: if args.version not in DISPLAY_VERSIONS: @@ -795,7 +815,7 @@ def main(): if missing: sys.exit(f'{expected}: no exefs_sha256 for {missing}') write_if_changed(out / 'modules.c', module_glue(layout, linked, hashes, paths, args.version)) - if (paths['romfs'] / 'index.json').is_file(): + if has_romfs: count, size, dt = romfs_index.write(paths['romfs'], out) print(f'romfs index: {count} files, {mib(size)}, {dt:.1f} s') else: diff --git a/tools/make_release.py b/tools/make_release.py index 11bf7e0f..36b055ec 100644 --- a/tools/make_release.py +++ b/tools/make_release.py @@ -424,12 +424,14 @@ def c_wide(text): def launcher_header(path): - """launcher_text.h: the launcher's two messages in English and Russian, taken from the installer's catalog.""" - lines = ['/* generated by tools/make_release.py from installer/text/{en,ru}.json */'] - for lang in ('en', 'ru'): + """launcher_text.h: the launcher's two messages in English, Russian and Brazilian Portuguese, taken from the + installer's catalog (LAUNCHER_TITLE_PT_BR for pt-BR.json).""" + lines = ['/* generated by tools/make_release.py from installer/text/{en,ru,pt-BR}.json */'] + for lang in ('en', 'ru', 'pt-BR'): texts = json.loads((ROOT / 'installer' / 'text' / f'{lang}.json').read_text(encoding='utf-8')) for key in ('title', 'python_missing'): - lines.append(f'#define LAUNCHER_{key.upper()}_{lang.upper()} {c_wide(texts["launcher." + key])}') + macro = f'LAUNCHER_{key.upper()}_{lang.upper().replace("-", "_")}' + lines.append(f'#define {macro} {c_wide(texts["launcher." + key])}') path.parent.mkdir(parents=True, exist_ok=True) path.write_text('\n'.join(lines) + '\n', encoding='ascii') diff --git a/tools/prepare_shaders.py b/tools/prepare_shaders.py index f26aa30d..f7351e99 100644 --- a/tools/prepare_shaders.py +++ b/tools/prepare_shaders.py @@ -18,6 +18,11 @@ shader-prepared.bin (one record per variant, in priority order) and builds every pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "stage library prebuild"). The seed files can be deleted afterwards. +A driver whose shader compiler crashes on a variant crashes on it in every run (NVIDIA 615.78.08 on an RTX 5080, issue +#30). The translations and shader-prepared.bin are written by then, and the journal the run leaves names the variants +the driver was building (vk_pipeline.c skip_init), so only the driver step runs again (`totk.exe --warm-shaders`), up to +RETRIES times, and skips them. + Game-derived output stays in the build directory (ignored). The list is read at preparation time from tools/data/shader_list.tsv when that file exists (the installer archive bundles it; it stays out of the repository), else from refs/ through git with the rows its history dropped (tools/shader_list.py); nothing of it is copied into the repository. Needs Python 3.14 (compression.zstd), so run it in .venv-py314 or the embeddable 3.14: @@ -54,6 +59,8 @@ TYPES = {'float': 0, 'sint': 1, 'uint': 2} DEFAULT = {'tex': {str(slot): '2d' for slot in range(32)}} # answers outside a shader's use mask are dropped CONTROL = 0x880 ANS_REC = 4 + 32 + 32 + 147 + 1 +JOURNAL = 'shader-prepare-journal.bin' # vk_pipeline.c skip_init: a 64-byte slot per worker, input id first +RETRIES = 3 BUNDLED_LIST = ROOT / 'tools' / 'data' / 'shader_list.tsv' # the installer archive carries a copy (no git, no refs/) @@ -128,6 +135,13 @@ def code_hash(stage, control, code, length, c1): struct.pack('