mirror of
https://github.com/valentderah/totk-pc-recomp.git
synced 2026-11-21 04:55:56 +00:00
Merge dev into task/bindinc: the incremental binding keeps its logic and takes dev's inline compares and copies (TOTK_MEMX)
The ten conflicts in runtime/vk_exec_draw.c were dev's memcmp/memcpy -> MX_EQ/MX_NE/MX_CPY swap on lines task/bindinc had rewritten. Each keeps the task/bindinc code with the swap applied to its hot-path compares and copies (push and set signature compares, the shader memo's pipe compare, the pipeline memo and recorded-state compares, the vertex attribute and stream lists). The check-mode compares that call bi_mismatch keep memcmp as the reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
d52c16e651
21 files changed
+750
-103
No files matched your search
@@ -90,7 +90,7 @@ Every guest thread is a host thread. TotK assumes Horizon's scheduler: three app
|
||||
- Memory pools. Apple's unified memory let MacNativeGames use the game's pools zero-copy. The back end imports the pools as host memory (`VK_EXT_external_memory_host`). On a discrete GPU, pools bound as buffers are mirrored in device-local memory and the pages the CPU wrote (write watch, above) are uploaded before a replay; idle pool regions are made read-only so the write-watch queries skip them (`vk_memory.c`). GPU-rendered textures are written back when the CPU reads them: on a pool invalidate, on a CPU copy, and for small targets once the CPU waits for a fence (`vk_texture.c`).
|
||||
- Replay thread. `TOTK_REPLAY_ASYNC=1` moves the replay off the submitting thread onto a queue at most one frame behind the game; reads of GPU results wait for the newest writer instead of draining the queue (`vk_backend.c`). It is off by default. Where the game waits for the GPU it overlaps the replay with the game threads (a GTX 1650 in the Depths, issue #88), a gain measured only there; while it was the default, a GTX 1660 Ti at the 60 fps cap with the GPU at 26 % reported uneven motion (#659). It adds up to a frame of latency; `replay_async = 1` turns it on.
|
||||
- Chunked replay. A large submission is recorded in chunks by several threads (`TOTK_REPLAY_WORKERS`, 0 by default: the sequential replay; 4 is a good value from 8 logical processors) and the chunks go to the queue in order (`vk_exec.c`). A sequential scan keeps a copy of the queue state where each chunk starts and lists what a chunk would otherwise learn from the chunks before it: inline uniform updates, conditions, contact shadow marches, uniform drops. It also makes render targets live in replay order and notes the op, since a chunk can sample a storage before the chunk that renders it got there or after the scan went past a later target there: a chunk's texture slots resolve as of its own op, and the chunks after a texture with no entry yet wait for the chunk that makes it. DLSS's decisions (the G-buffer, the light target, the end of the scene, the scaling pass) are the scan's too, and its copy of the lit scene and its evaluation are records of the chunk that holds the op; a chunk's draws are jittered while they come before the scan's end of the scene. The evaluation waits until the chunks before it are recorded, as it reads what their draws noted (the camera block, the jitter, the per-object motion list, which chunks add to under one lock), and the chunks after it wait for it, as they sample its output. `TOTK_REPLAY_STATE_CHECK=1` compares the state after the last chunk with the scan's.
|
||||
- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `TOTK_SHADOW=1` on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.2.1 and 1.4.3); results the game reads back from the GPU after it can be a frame old.
|
||||
- Frame ahead. `TOTK_SYNC_AHEAD=1` (`sync_ahead`, off by default) makes the one `nvnSyncWait` per frame that waits for the frame presented last wait for the frame before it, so the game builds the next frame while the GPU still runs the previous one (`nvn_api_core.c`, `nvnSyncWait`). The game then writes memory that in-flight work still reads. This is safe only with shadow storage (`vk_memory.c`, the CPU-visible pools are host-visible copies of guest memory, on by default on Linux and Wine, `shadow = 1` (`TOTK_SHADOW=1`) on native Windows; without it the switch is ignored): a CPU write reaches the GPU only through the shadow copy made before each submit (`vk_mem_shadow_sync`), and a copy that would overwrite bytes an unfinished submission reads is queued as a GPU copy in the prologue command buffer of the part submitted next (`vk_mem_shadow_prologue`, `TOTK_SHADOW_DEFER=1`, the default with the feature) or, with `=0`, waited for on the replay thread (`shadow_reader_wait`). Only that one wait lags (identified by its call site, known for 1.2.1 and 1.4.3); results the game reads back from the GPU after it can be a frame old.
|
||||
- Recorder thread. The Vulkan calls the replay makes into its command buffer are made by a thread of their own (`vk_rec.c`, after Savage Wind Recomp and LAS Recomp): each `vkCmd*` copies its arguments into a packet of a ring and the recorder makes the calls in order; calls into other command buffers are made at once; the recorder also ends the command buffer, and the submit waits until it has. In the town on an RX 9070 the replay takes 17% less time on the submitting thread. It is off by default on AMD GPUs, where the driver loses the device or hangs the GPU at startup with it (`TOTK_VK_RECORDER=1` forces it on, `=0` makes every call on the submitting thread).
|
||||
- Uniform snapshot. The game rewrites per-object uniform blocks on other threads soon after it submits the frame
|
||||
that reads them, while the replay still records it. At the start of a submission's replay every uniform range it
|
||||
|
||||
+8
-3
@@ -100,7 +100,7 @@ of them.
|
||||
objects before it links: any AVX, BMI or other instruction above x86-64-v2 outside a function listed there as
|
||||
dispatched by a runtime CPU check stops the build.
|
||||
|
||||
`gen.py` sizes its emit pool from the CPU count and the free memory; `--jobs N` sets it, and so does the environment variable `TOTK_LIFT_JOBS=N`, which the installer's lift step honours too (it has no `--jobs`). When a worker runs out of memory, the pool restarts with half the workers (`emit: out of memory with N workers, retrying with M`), down to one; only an out-of-memory error with one worker fails the step.
|
||||
`gen.py` sizes its emit pool from the CPU count and the free memory; `--jobs N` sets it, and so does the environment variable `TOTK_LIFT_JOBS=N`, which the installer's lift step honours too (it has no `--jobs`). When a worker runs out of memory, the pool restarts with half the workers (`emit: out of memory with N workers, retrying with M`), down to one; only an out-of-memory error with one worker fails the step. A later lift into the same `--out` lifts again only the chunks whose inputs changed (`<mod>/lift_cache.json`, see the top of `recomp/gen.py`); `--no-cache`, or `TOTK_LIFT_CACHE=0` (which the installer's lift step honours too), lifts every chunk.
|
||||
|
||||
Lifting takes about 15 s, a clean compile about 200 s on 24 threads; later builds recompile only what changed.
|
||||
A compiler cache in front of clang shortens it further: with `SCCACHE` set to an sccache executable, `build_game.py`
|
||||
@@ -200,7 +200,7 @@ re-protecting imported memory invalidates the driver's userptr buffer on every s
|
||||
amdgpu), and importing pins the pages, which drops their write protection. Each CPU-visible pool therefore gets a
|
||||
host-visible copy (shadow storage, `runtime/vk_memory.c`): the pages the CPU wrote go in before each submit, and the GPU's
|
||||
writes come back at its fence. In play that costs about 0.2 ms a frame and doubles the memory of those pools (about
|
||||
1.1 GB). `TOTK_SHADOW=1` turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as
|
||||
1.1 GB). `shadow = 1` in `totk.ini` (or `TOTK_SHADOW=1`) turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as
|
||||
well: the shadow copies are chosen for the measured cost above, not for a missing extension.
|
||||
|
||||
The Deck's GPU (Van Gogh) has no native ASTC, so its textures go through the compute decoder (`runtime/vk_astc.c`,
|
||||
@@ -222,7 +222,12 @@ when a setting is described neither in `totk.ini.example` nor in [settings.md](s
|
||||
`.venv\Scripts\python tests\exefs_patches_test.py`, which makes its own modules and runs `gen.py` on one with a jump
|
||||
table (no game files), and `totk.exe --probe-mods`. A change to the cheat VM or the cheat opcode tables
|
||||
(`runtime/cheats.c`, `tools/cheats.py`) passes `.venv\Scripts\python tests\cheats_vm_check.py`, which runs both on
|
||||
cheats of its own with clang from PATH (`--cc` names another).
|
||||
cheats of its own with clang from PATH (`--cc` names another). A change to what a generated chunk depends on, or to
|
||||
the lift cache of `recomp/gen.py` (`<mod>/lift_cache.json`), passes `.venv\Scripts\python tests\lift_cache_test.py`,
|
||||
which lifts a module it makes itself with and without the cache and compares the output (no game files); a new input
|
||||
of the chunk text goes into `chunk_key` with a case there. On game files, a lift with the cache into a used output
|
||||
folder and a `--no-cache` lift into an empty one give the same files apart from `manifest.json` and
|
||||
`lift_cache.json`.
|
||||
|
||||
The lifter (`recomp/`) and the CPU headers (`runtime/cpu.h`, `runtime/simd.h`) are checked against Unicorn. Every lifter
|
||||
change passes these, all run with `.venv`:
|
||||
|
||||
+13
-5
@@ -106,9 +106,13 @@ A cancelled or failed setup resumes from the last finished step (`totk-install.j
|
||||
|
||||
## Rebuilding for the mods
|
||||
|
||||
The generated C and the objects stay in the temp folder, so the next lift rewrites only the chunks whose C changed and
|
||||
Ninja recompiles only those: a new or edited code cheat costs the lift (under a minute on 24 cores, a few minutes on 4)
|
||||
and a few dozen chunks, not a full compile, and the shaders stay prepared.
|
||||
The generated C and the objects stay in the temp folder. The next lift lifts again only the chunks whose inputs
|
||||
changed (the lift cache of `recomp/gen.py`: the patch sites inside a chunk, with their set indices) and rewrites only
|
||||
those whose C changed, and Ninja recompiles only those. A new or edited code cheat costs a few seconds of lift
|
||||
(discovery and a hash of the kept chunks; 7 s for one changed cheat word on 12 cores, against 102 s for the whole lift)
|
||||
and the compile of the chunks that hold its sites, not a full compile, and the shaders stay prepared. A cheat set added
|
||||
or removed before others in the patch file moves their set indices, so the chunks of their sites are lifted too. A
|
||||
change to `recomp/*.py` lifts everything.
|
||||
|
||||
`build-cache/rebuild.txt` holds the command that opens the setup window in its rebuild mode on the install folder
|
||||
(`installer/gui.py --rebuild <install folder>`, started through `python -c` so the release folder is on `sys.path`
|
||||
@@ -116,8 +120,12 @@ whatever the working folder), one argument per line. When an enabled mod has cod
|
||||
lacks, `totk.exe` runs it at start and exits (`runtime/cheats.c` `rebuild_if_needed`); the
|
||||
window runs the setup on the folder at once and starts the game again when it succeeds. A failed or cancelled rebuild
|
||||
offers to start the game with the code it has; `build-cache/rebuild.skip` then keeps the same files from starting it
|
||||
again, until "Rebuild the game now" on the Cheats or Mods tab. `[mods] mods_rebuild = off` turns it off. The rebuild
|
||||
needs the release folder the setup ran from: a moved or deleted release folder means running a setup again.
|
||||
again, until "Rebuild the game now" on the Cheats or Mods tab. A command that does not start at all leaves no
|
||||
`rebuild.skip`, so the next start tries it again. `[mods] mods_rebuild = off` turns it off. The rebuild needs the
|
||||
release folder the setup ran from: a moved or deleted release folder means running a setup again. On Linux the command
|
||||
starts the AppImage file itself (`$APPIMAGE --rebuild <install folder>`), because the AppImage's mounted folder is gone
|
||||
once the setup exits; a moved or deleted AppImage means running a setup again. Installs set up by an AppImage before
|
||||
this change hold a command into that mount: running the new setup once on the install folder writes the new one.
|
||||
|
||||
## Command line
|
||||
|
||||
|
||||
+15
-6
@@ -14,9 +14,10 @@ screen shapes and the prompts style this player needs, installer/screens.py), to
|
||||
shader state of totk.exe --prepare-shaders, totk.ini, totk-install.json.
|
||||
Temp folder (<temp>/<dest name>-<hash>): check/ (ExeFS of both inputs, the host check), unpacked/ (the NSP of an NSZ
|
||||
or XCZ input until the extraction is done), gen/ (C), build/ (objects), logs/. The cleanup keeps gen/, build/ and
|
||||
logs/: the next run lifts into the same gen/, where recomp/gen.py rewrites only the chunks whose C changed, and Ninja
|
||||
in build/ recompiles only those. A change of the mods' code cheats then costs the lift (about a minute) and the chunks
|
||||
it touches, not a full compile.
|
||||
logs/: the next run lifts into the same gen/, where recomp/gen.py lifts again only the chunks whose inputs changed (its
|
||||
lift cache, gen/<mod>/lift_cache.json) and rewrites only those whose C changed, and Ninja in build/ recompiles only
|
||||
those. A change of the mods' code cheats then costs a few seconds of lift and the chunks holding its sites, not a full
|
||||
lift or compile.
|
||||
|
||||
Install folder build-cache/: cheats.status (tools/cheats.py), exefs.status and exefs.data (tools/exefs_patches.py) and
|
||||
rebuild.txt, the command that starts this setup in its rebuild mode on the install folder (installer/gui.py --rebuild),
|
||||
@@ -900,8 +901,9 @@ class Lift(Step):
|
||||
return (gen / 'layout.json').is_file() and stamp.is_file() and stamp.read_text().strip() == ctx.keys['lift']
|
||||
|
||||
def run(self, ctx):
|
||||
# gen/ of the last run stays: gen.py rewrites only the chunks whose C changed (and removes the ones a smaller
|
||||
# partition no longer has), so Ninja recompiles only those
|
||||
# gen/ of the last run stays: gen.py lifts again only the chunks whose inputs changed
|
||||
# (gen/<mod>/lift_cache.json; TOTK_LIFT_CACHE=0 lifts all), rewrites only those whose C changed (and removes
|
||||
# the ones a smaller partition no longer has), so Ninja recompiles only those
|
||||
layout, rep, gen = ctx.layout, ctx.reporter, ctx.temp / 'gen'
|
||||
(gen / STAMP).unlink(missing_ok=True)
|
||||
rep.progress('lift', 'discover', force=True)
|
||||
@@ -1372,8 +1374,15 @@ REBUILD_BOOT = ('import runpy, sys; sys.path.insert(0, sys.argv.pop(1)); '
|
||||
def rebuild_command(ctx):
|
||||
"""The command that starts the setup window in its rebuild mode on this install folder (installer/gui.py
|
||||
--rebuild): pythonw beside the setup's Python on Windows, so no console opens. In a repository that is the Python
|
||||
running this setup: the tools' embeddable Python there has no tkinter."""
|
||||
running this setup: the tools' embeddable Python there has no tkinter.
|
||||
In the Linux AppImage the release folder is the AppImage's mount (/tmp/.mount_*), which goes away when the setup
|
||||
exits, so a path into it would fail to start later and the game would keep its old code. The command starts the
|
||||
AppImage file itself instead ($APPIMAGE, set by the AppImage runtime); its AppRun passes the arguments to
|
||||
installer.gui."""
|
||||
layout = ctx.layout
|
||||
appimage = os.environ.get('APPIMAGE', '')
|
||||
if not layout.dev and appimage and Path(appimage).is_file():
|
||||
return [str(Path(appimage).resolve()), '--rebuild', str(ctx.dest)]
|
||||
python = Path(sys.executable) if layout.dev else layout.python
|
||||
if os.name == 'nt' and python.with_name('pythonw.exe').is_file():
|
||||
python = python.with_name('pythonw.exe')
|
||||
|
||||
+186
-14
@@ -3,12 +3,14 @@
|
||||
|
||||
Usage: gen.py --exefs build/extracted/v121/exefs --out generated/v121 [--modules main,subsdk0,sdk] [--jobs N]
|
||||
[--limit N] [--chunk-fns N] [--chunk-bytes B] [--big-fn B] [--patches FILE] [--v-locals K [--v-trust]]
|
||||
[--no-cache]
|
||||
|
||||
Output, under --out:
|
||||
layout.h, layout.json module bases (recomp/layout.py)
|
||||
<mod>/<mod>_NNNN.c one `void f_<addr>(Cpu *c)` per guest function, about --chunk-bytes of C per file
|
||||
<mod>/<mod>_table.c sorted <mod>_fn_addrs[] / <mod>_fn_ptrs[] / <mod>_fn_count for rt_dispatch
|
||||
<mod>/manifest.json chunk list, function count, statistics and timings
|
||||
<mod>/lift_cache.json per chunk: the key of its inputs, its file's SHA-256, size and mtime, and its results
|
||||
<mod>/folded.txt the module's bytes the generated code took as constants: what the jump tables read at
|
||||
lift time, as module offsets [start, end); tools/exefs_patches.py keeps a mod's data
|
||||
writes off them
|
||||
@@ -24,8 +26,20 @@ Emission runs in worker processes, one chunk per task. Each worker loads the mod
|
||||
time and resolves its jump tables (recomp/jumptables.py), so memory per worker does not grow with the module.
|
||||
Chunks are sized from a per-instruction estimate of the C they produce, so their names and contents only depend on
|
||||
the input; a chunk whose text did not change is not rewritten and keeps its mtime.
|
||||
|
||||
A chunk is not lifted again when its inputs did not change. Its text depends only on the generator (recomp/*.py,
|
||||
GENERATED_MARKER, the Emitter options), the module (its file, its base, and the function starts discovered from it),
|
||||
the chunk's number and functions, and the patch sites and spin waits inside the chunk's address range: the per-module
|
||||
caches of recomp/jumptables.py are derived from the module's bytes and base, and its per-function state is reset for
|
||||
each function. The SHA-256 of all of these is the chunk's key. <mod>/lift_cache.json records the key of each chunk
|
||||
with the SHA-256, size and mtime of the file it wrote and the results the manifest and folded.txt need, under a
|
||||
digest of the entry. A chunk whose entry is intact, whose key matches and whose file still has that size, mtime and
|
||||
hash is reused as it is; any other chunk is lifted. So a rebuild after a change to one cheat lifts only the chunks
|
||||
holding its sites. --no-cache, or TOTK_LIFT_CACHE=0 in the environment, lifts every chunk.
|
||||
"""
|
||||
import argparse
|
||||
import bisect
|
||||
import hashlib
|
||||
import json
|
||||
import multiprocessing as mp
|
||||
import os
|
||||
@@ -33,6 +47,7 @@ import re
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
@@ -210,7 +225,8 @@ def partition(est, max_fns, max_bytes, big_fn):
|
||||
# ------------------------------------------------------------------------------------------------ emission
|
||||
|
||||
class Emitter:
|
||||
"""C for the functions of one module. starts is the set of every function start of the module."""
|
||||
"""C for the functions of one module. starts is the set of every function start of the module. A new input of
|
||||
the C (an argument here, or anything else it reads) must go into the lift cache's key: chunk_key."""
|
||||
|
||||
def __init__(self, mod, starts, patched=None, reg_locals=regs.LOCALS, callee_saved=True, spins=None,
|
||||
v_locals=vlocals.LOCALS, v_trust=False):
|
||||
@@ -532,9 +548,10 @@ def chunk_text(name, number, emitter, fns):
|
||||
+ '\n' + '\n\n'.join(body) + '\n')
|
||||
|
||||
|
||||
def write_if_changed(path, text):
|
||||
def write_if_changed(path, text, wait=True):
|
||||
"""Returns True if the file was written. An unchanged file keeps its mtime, so the build skips it. A changed one
|
||||
is written under a temporary name and renamed into place, so a reader never sees it half written."""
|
||||
is written under a temporary name and renamed into place, so a reader never sees it half written. wait: retry
|
||||
the rename while another process holds the file (_retry); without it the first failure raises."""
|
||||
data = text.encode()
|
||||
try:
|
||||
if path.stat().st_size == len(data) and path.read_bytes() == data:
|
||||
@@ -544,7 +561,10 @@ def write_if_changed(path, text):
|
||||
tmp = path.with_name(f'.{path.name}.{os.getpid()}.tmp')
|
||||
tmp.write_bytes(data)
|
||||
try:
|
||||
_retry(os.replace, tmp, path)
|
||||
if wait:
|
||||
_retry(os.replace, tmp, path)
|
||||
else:
|
||||
os.replace(tmp, path)
|
||||
except OSError:
|
||||
tmp.unlink(missing_ok=True)
|
||||
raise
|
||||
@@ -610,8 +630,8 @@ def merge_ranges(ranges):
|
||||
|
||||
|
||||
def _emit_chunk(task):
|
||||
"""Worker: write one chunk. Returns its statistics."""
|
||||
name, number, path, fns = task
|
||||
"""Worker: write one chunk. Returns its statistics, and the SHA-256, size and mtime of its file for the cache."""
|
||||
name, number, path, fns, key = task
|
||||
t0 = time.perf_counter()
|
||||
# Test hook: a pool wider than this raises the MemoryError a busy machine does, so the retry can be exercised.
|
||||
limit = os.environ.get('TOTK_LIFT_TEST_OOM_ABOVE')
|
||||
@@ -621,17 +641,20 @@ def _emit_chunk(task):
|
||||
e.stats.clear()
|
||||
e.unsupported.clear()
|
||||
text = chunk_text(name, number, e, fns)
|
||||
error = None
|
||||
error, st = None, None
|
||||
try:
|
||||
written = write_if_changed(Path(path), text)
|
||||
st = os.stat(path)
|
||||
except OSError as ex:
|
||||
written, error = False, f'{path}: {ex}'
|
||||
reads = _W['reads'].get(name, [])
|
||||
folded = merge_ranges(reads)
|
||||
reads.clear()
|
||||
return {'module': name, 'number': number, 'file': Path(path).name, 'functions': len(fns), 'bytes': len(text),
|
||||
'written': written, 'error': error, 'seconds': time.perf_counter() - t0, 'stats': dict(e.stats),
|
||||
'unsupported': dict(e.unsupported), 'folded': folded}
|
||||
'written': written, 'reused': False, 'error': error, 'seconds': time.perf_counter() - t0,
|
||||
'stats': dict(e.stats), 'unsupported': dict(e.unsupported), 'folded': folded, 'key': key,
|
||||
'sha256': _sha256(text.encode()), 'size': st.st_size if st else None,
|
||||
'mtime_ns': st.st_mtime_ns if st else None}
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------ driver
|
||||
@@ -670,6 +693,120 @@ def remove_stale(moddir, name, nchunks, errors):
|
||||
return removed
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------ lift cache
|
||||
|
||||
CACHE_FILE = 'lift_cache.json'
|
||||
CACHE_VERSION = 1 # raise when the cache's format or the meaning of its key changes
|
||||
# Per-chunk results the cache keeps beside the key: what the manifest, folded.txt and the reuse check need.
|
||||
CACHED = ('functions', 'bytes', 'stats', 'unsupported', 'folded', 'sha256', 'size', 'mtime_ns')
|
||||
|
||||
|
||||
def _sha256(data):
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def _key(*parts):
|
||||
"""SHA-256 of JSON-encodable parts."""
|
||||
return _sha256(json.dumps(parts, separators=(',', ':')).encode())
|
||||
|
||||
|
||||
def generator_key(opts):
|
||||
"""Key of everything the chunk text depends on besides the module and the patches: the generator's sources
|
||||
(recomp/*.py, not the patch tables beside them), the marker in each header, the Emitter options, the Python
|
||||
version that runs them."""
|
||||
srcs = [[p.name, _sha256(p.read_bytes())] for p in sorted((ROOT / 'recomp').glob('*.py'))]
|
||||
return _key('gen', CACHE_VERSION, list(sys.version_info[:2]), GENERATED_MARKER, sorted(opts.items()), srcs)
|
||||
|
||||
|
||||
def module_key(gen_key, name, path, base, abs_starts):
|
||||
"""Key of one module's inputs: the generator, the module's file and base, and the function starts discovered
|
||||
from them (they decide which branches become direct calls)."""
|
||||
return _key(gen_key, name, _sha256(Path(path).read_bytes()), base, _sha256(abs_starts.tobytes()))
|
||||
|
||||
|
||||
def in_span(addrs, lo, hi):
|
||||
"""The addresses of the sorted list addrs in [lo, hi)."""
|
||||
return addrs[bisect.bisect_left(addrs, lo):bisect.bisect_left(addrs, hi)]
|
||||
|
||||
|
||||
def chunk_key(mod_key, number, fns, patched, site_addrs, spins, spin_addrs):
|
||||
"""Key of one chunk: its module, number (in its header) and functions, and the patch sites (set index and
|
||||
replacement words, @page resolved, so the patch page is in them) and spin waits inside its address range. Every
|
||||
input of the chunk text that is not in generator_key or module_key must be added here, with a case in
|
||||
tests/lift_cache_test.py."""
|
||||
lo, hi = fns[0][0], fns[-1][1]
|
||||
sites = [[a, [[k, words] for k, words in patched[a]]] for a in in_span(site_addrs, lo, hi)]
|
||||
return _key(mod_key, number, fns, sites, [[b, spins[b]] for b in in_span(spin_addrs, lo, hi)])
|
||||
|
||||
|
||||
def _entry_check(file, entry):
|
||||
"""Digest of a cache entry: its chunk file name, key and results. The results go into the manifest and folded.txt
|
||||
as they are, so a changed value in the cache file must not be taken (tools/exefs_patches.py --folded keeps a
|
||||
mod's data writes off the folded bytes)."""
|
||||
return _key(file, entry['key'], [entry[k] for k in CACHED])
|
||||
|
||||
|
||||
def load_cache(moddir):
|
||||
"""{chunk file name: entry} from <mod>/lift_cache.json: the entries whose digest matches; empty when the file is
|
||||
missing or unreadable."""
|
||||
try:
|
||||
data = json.loads((moddir / CACHE_FILE).read_text())
|
||||
if data.get('version') != CACHE_VERSION or not isinstance(data.get('chunks'), dict):
|
||||
return {}
|
||||
out = {}
|
||||
for file, e in data['chunks'].items():
|
||||
try:
|
||||
if e['check'] == _entry_check(file, e):
|
||||
out[file] = e
|
||||
except (KeyError, TypeError):
|
||||
pass
|
||||
return out
|
||||
except (OSError, ValueError, AttributeError):
|
||||
return {}
|
||||
|
||||
|
||||
def _intact(path, entry):
|
||||
"""True when the file at path still is the one the entry recorded: same size and mtime, then same SHA-256 (a bit
|
||||
flipped in a kept file would otherwise stay forever, as tools/build_game.py check_objects says of objects)."""
|
||||
try:
|
||||
st = path.stat()
|
||||
if st.st_size != entry['size'] or st.st_mtime_ns != entry['mtime_ns']:
|
||||
return False
|
||||
return _sha256(path.read_bytes()) == entry['sha256']
|
||||
except (OSError, KeyError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
def reuse(tasks, caches):
|
||||
"""The results of the tasks whose chunk can be reused: its recorded key equals the task's and its file is intact.
|
||||
The files are hashed on threads (hashlib releases the GIL), about 1 GB in all for the three modules."""
|
||||
cands = []
|
||||
for name, number, path, fns, key in tasks:
|
||||
e = caches.get(name, {}).get(Path(path).name)
|
||||
if e is not None and e['key'] == key:
|
||||
cands.append((name, number, Path(path), key, e))
|
||||
with ThreadPoolExecutor(min(16, os.cpu_count() or 4)) as ex:
|
||||
ok = list(ex.map(lambda c: _intact(c[2], c[4]), cands))
|
||||
out = []
|
||||
for (name, number, path, key, e), good in zip(cands, ok):
|
||||
if good:
|
||||
out.append({'module': name, 'number': number, 'file': path.name, 'written': False, 'reused': True,
|
||||
'error': None, 'seconds': 0.0, 'key': key, **{k: e[k] for k in CACHED}})
|
||||
return out
|
||||
|
||||
|
||||
def cache_text(name, results):
|
||||
"""<mod>/lift_cache.json for this run's chunks; a chunk that could not be written gets no entry. Keys keep their
|
||||
order (no sort_keys), so a reused chunk's `unsupported` merges into the census as it did when it was lifted."""
|
||||
chunks = {}
|
||||
for r in results:
|
||||
if not r['error'] and r['size'] is not None:
|
||||
e = {'key': r['key'], **{k: r[k] for k in CACHED}}
|
||||
chunks[r['file']] = {**e, 'check': _entry_check(r['file'], e)}
|
||||
return json.dumps({'module': name, 'marker': GENERATED_MARKER, 'version': CACHE_VERSION, 'chunks': chunks},
|
||||
separators=(',', ':')) + '\n'
|
||||
|
||||
|
||||
# Peak memory of one emit worker on 1.4.3 is 1.3 GiB (1.5 GiB with margin); the parent holds about 0.7 GiB.
|
||||
WORKER_MEM = 1536 << 20
|
||||
MEM_SHARE = 0.8 # of the free memory the workers may take, the rest is for the system and other programs
|
||||
@@ -718,9 +855,16 @@ def main():
|
||||
ap.add_argument('--v-trust', action='store_true',
|
||||
help='with --v-locals: do not reload V8-V15 after a call (AAPCS64: the callee preserves their low '
|
||||
'64 bits)')
|
||||
ap.add_argument('--no-cache', action='store_true',
|
||||
help=f'lift every chunk, also those whose inputs match <mod>/{CACHE_FILE} (it is written anew); '
|
||||
'so does the environment variable TOTK_LIFT_CACHE=0')
|
||||
args = ap.parse_args()
|
||||
# The installer's lift step passes no --no-cache; the environment reaches it, as TOTK_LIFT_JOBS does (pick_jobs)
|
||||
no_cache = ('--no-cache' if args.no_cache else
|
||||
'TOTK_LIFT_CACHE=0' if os.environ.get('TOTK_LIFT_CACHE', '').strip() == '0' else None)
|
||||
opts = {'reg_locals': args.reg_locals, 'callee_saved': not args.no_callee_saved_trust, 'v_locals': args.v_locals,
|
||||
'v_trust': args.v_trust}
|
||||
gen_key = generator_key(opts)
|
||||
|
||||
t0 = time.perf_counter()
|
||||
mods = layout(args.exefs)
|
||||
@@ -748,9 +892,13 @@ def main():
|
||||
chunks = partition(est[:n], args.chunk_fns, args.chunk_bytes, args.big_fn)
|
||||
moddir = args.out / name
|
||||
moddir.mkdir(parents=True, exist_ok=True)
|
||||
mkey = module_key(gen_key, name, path, base, abs_starts)
|
||||
patched, spins = table.for_module(name, base), table.spins_for(name, base)
|
||||
site_addrs, spin_addrs = sorted(patched), sorted(spins)
|
||||
for number, (i0, i1) in enumerate(chunks):
|
||||
fns = list(zip(abs_starts[i0:i1].tolist(), ends[i0:i1].tolist()))
|
||||
tasks.append((int(est[i0:i1].sum()), (name, number, str(moddir / f'{name}_{number:04d}.c'), fns)))
|
||||
key = chunk_key(mkey, number, fns, patched, site_addrs, spins, spin_addrs)
|
||||
tasks.append((int(est[i0:i1].sum()), (name, number, str(moddir / f'{name}_{number:04d}.c'), fns, key)))
|
||||
paths[name] = (str(path), base)
|
||||
starts_abs[name] = abs_starts
|
||||
info[name] = {'base': base, 'lo': mod.lo, 'hi': mod.hi, 'emitted': abs_starts[:n].tolist(),
|
||||
@@ -769,9 +917,23 @@ def main():
|
||||
ctx = mp.get_context('spawn')
|
||||
jobs, why = pick_jobs(args.jobs)
|
||||
pending = [t for _, t in tasks]
|
||||
emitted = 0
|
||||
while True:
|
||||
print(f'emitting with {jobs} workers ({why})', file=sys.stderr, flush=True)
|
||||
reused = [] if no_cache else reuse(pending, {name: load_cache(d['moddir']) for name, d in info.items()})
|
||||
for r in reused:
|
||||
results[r['module']].append(r)
|
||||
folded[r['module']] += r['folded']
|
||||
skip = {(r['module'], r['number']) for r in reused}
|
||||
pending = [t for t in pending if (t[0], t[1]) not in skip]
|
||||
emitted = len(reused)
|
||||
print(f'reusing {len(reused)} of {len(tasks)} chunks (inputs unchanged), {time.perf_counter() - t_emit:.1f}s'
|
||||
if not no_cache else f'lifting every chunk ({no_cache})', file=sys.stderr, flush=True)
|
||||
if reused or not pending:
|
||||
# installer/steps.py EMITTED reads these lines for its progress bar: the reused chunks count as done, and a
|
||||
# run with nothing to lift still reaches M/M
|
||||
print(f'emitted {emitted}/{len(tasks)} chunks, 0s', file=sys.stderr, flush=True)
|
||||
if len(pending) < jobs:
|
||||
jobs, why = max(1, len(pending)), f'{len(pending)} chunk{"s" if len(pending) != 1 else ""} to lift'
|
||||
while pending:
|
||||
print(f'emitting with {jobs} worker{"s" if jobs != 1 else ""} ({why})', file=sys.stderr, flush=True)
|
||||
try:
|
||||
# Chunk writes are atomic and idempotent, so a chunk lost with a failed pool is simply emitted again.
|
||||
# Its folded bytes come with its result: a chunk emitted again reads them again in its new worker.
|
||||
@@ -841,6 +1003,7 @@ def main():
|
||||
'c_bytes': sum(r['bytes'] for r in rs),
|
||||
'largest_chunk_bytes': max((r['bytes'] for r in rs), default=0),
|
||||
'chunks_written': sum(r['written'] for r in rs) + table_written,
|
||||
'chunks_reused': sum(r['reused'] for r in rs),
|
||||
'stale_chunks_removed': removed,
|
||||
**{k: v for k, v in sorted(stats.items()) if k.startswith('v_')}, # --v-locals only
|
||||
},
|
||||
@@ -853,8 +1016,17 @@ def main():
|
||||
},
|
||||
}
|
||||
write_if_changed(moddir / 'manifest.json', json.dumps(manifest, indent=1) + '\n')
|
||||
# The cache goes last, after every output it describes. It only saves time, so a failed write warns and the
|
||||
# lift still succeeds, without waiting for a holder of the file: the previous cache stays, and its entries
|
||||
# for the chunks this run rewrote no longer match their files' size, mtime and hash. A run stopped before
|
||||
# here leaves it the same way.
|
||||
try:
|
||||
write_if_changed(moddir / CACHE_FILE, cache_text(name, rs), wait=False)
|
||||
except OSError as ex:
|
||||
print(f'{name}: {CACHE_FILE} not written, the next lift reuses less: {ex}', file=sys.stderr, flush=True)
|
||||
s = manifest['stats']
|
||||
print(f'{name}: {s["emitted_functions"]} functions, {len(rs)} chunks, {s["c_bytes"] / 1e6:.0f} MB of C, '
|
||||
print(f'{name}: {s["emitted_functions"]} functions, {len(rs)} chunks ({s["chunks_reused"]} reused), '
|
||||
f'{s["c_bytes"] / 1e6:.0f} MB of C, '
|
||||
f'{s["jump_tables"]} jump tables, {s["all_label_fallbacks"]} all-label fallbacks, '
|
||||
f'{s["br_dispatch"]} br via dispatch, {s["traps"]} traps ({s["unsupported"]} unsupported)',
|
||||
file=sys.stderr)
|
||||
|
||||
+2
-1
@@ -49,7 +49,8 @@ def adrp(pc, target, rd):
|
||||
|
||||
class Table:
|
||||
"""sets: [(name, env, default)]; sites: {module: {offset: [(set index, [word | ('page', reg)], original)]}} (the
|
||||
alternatives of a site in file order); page."""
|
||||
alternatives of a site in file order); page. What the generated C takes from here (for_module, spins_for) is in
|
||||
the lift cache's key (recomp/gen.py chunk_key); a new kind of line the Emitter reads must go there too."""
|
||||
|
||||
def __init__(self, sets=(), sites=None, page=0, spins=None):
|
||||
self.sets = list(sets)
|
||||
|
||||
+4
-4
@@ -51,7 +51,7 @@ static const char *const known[] = {
|
||||
"GAMEPLAY_WEAPON_WHEEL",
|
||||
"GPL", "GPL_DRAW_BUILD", "GPU", "GPU_INDEX", "GPU_NAME", "GRASS_RANGE", "HDR", "HDR_GAME_NITS", "HDR_HUE",
|
||||
"HDR_PEAK_NITS", "HDR_UI_NITS", "HIGHFPS_FIX", "INPUT", "INPUT_RECORD", "INPUT_SCRIPT", "KSPIN_US", "LANG", "LIB_PREBUILD",
|
||||
"LIB_PREBUILD_KEEP", "LIB_PREBUILD_THREADS", "LOCALE", "LOD_GUARD", "LSTICK_DEADZONE", "MAP_CLICK", "MEMORY", "MENU_LANG", "MENU_TEST", "MENU_FONT_FONTCONFIG",
|
||||
"LIB_PREBUILD_KEEP", "LIB_PREBUILD_THREADS", "LOCALE", "LOD_GUARD", "LSTICK_DEADZONE", "MAP_CLICK", "MEMORY", "MEMX", "MENU_LANG", "MENU_TEST", "MENU_FONT_FONTCONFIG",
|
||||
"MEM_RESERVE_MIB", "MENU_CURSOR", "MENU_CURSOR_LOG", "MENU_TEST_DIR", "MENU_TEST_SET", "MINIMAL_LAYOUT", "MIRROR_GAP", "MODS_DIR", "MODS_DISABLED", "MODS_ORDER", "MODS_REBUILD", "MOTION", "MOUSE_AIM_SCALE", "MOUSE_MODE", "MOUSE_DIRECT_PATCH", "MOUSE_SENS", "MOVIE_ASPECT", "MOVIE_HOST", "MOVIE_PACE",
|
||||
"MVOBJ_ALL", "MVOBJ_BANKS", "MVOBJ_EQUAL", "MVOBJ_EXACT", "MVOBJ_EXEC", "MVOBJ_NOC4", "MVOBJ_ONLY", "MVOBJ_TIMES", "MVPREV_CHECK", "MVPREV_GROW", "MVPREV_SKIP", "MVPREV_TRUST", "MV_CHECK",
|
||||
"MV_SURVEY",
|
||||
@@ -61,7 +61,7 @@ static const char *const known[] = {
|
||||
"RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_CHUNKS", "REPLAY_CHUNK_DRAWS", "REPLAY_SKEW_US", "REPLAY_STATE_CHECK", "REPLAY_WORKERS",
|
||||
"REPLAY_CHECK", "REPORT_GUARD", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES",
|
||||
"ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE",
|
||||
"SHADER_MEMO", "SHADOW_DEFER", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHADOW_SNAP", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR",
|
||||
"SHADER_MEMO", "SHADOW", "SHADOW_DEFER", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHADOW_SNAP", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR",
|
||||
"SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_US", "STICK_DEADZONE", "STICK_SENS",
|
||||
"SWKBD_ANSWER", "SYNC_AHEAD",
|
||||
"SWKBD_TEST", "TEST_OOM", "TEST_PAD", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_BLOCK_MIB", "TEX_HEAP_EXTRA_MB", "TEX_MEMO", "TEX_RELEASED_MIB", "TEX_VRAM_PRESSURE", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UB_OVERLAY", "UB_OVERLAY_OUT", "UPSCALER", "VB_USED", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NO_FLOAT64",
|
||||
@@ -106,7 +106,7 @@ static const struct { const char *key, *section; } sections[] = {
|
||||
{"LIB_PREBUILD", "performance"}, {"LIB_PREBUILD_THREADS", "performance"}, {"PIPE_PREBUILD", "performance"},
|
||||
{"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"},
|
||||
{"TEX_ALIAS", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"TEX_RELEASED_MIB", "performance"}, {"TEX_BLOCK_MIB", "performance"}, {"VMA_BLOCK_MIB", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"},
|
||||
{"BIND_SKIP", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"TEX_MEMO", "performance"}, {"VB_USED", "performance"}, {"DESC_TEMPLATE", "performance"},
|
||||
{"BIND_SKIP", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"MEMX", "performance"}, {"TEX_MEMO", "performance"}, {"VB_USED", "performance"}, {"DESC_TEMPLATE", "performance"},
|
||||
{"SYNC_AHEAD", "performance"}, {"SHADOW_DEFER", "performance"}, {"PUSH_SKIP", "performance"}, {"SPLIT_DRAWS", "performance"}, {"SPLIT_US", "performance"}, {"VK_VALIDATION", "performance"},
|
||||
{"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"},
|
||||
{"OPTREF_GUARD", "performance"}, {"ABILITY_GUARD", "performance"}, {"REPORT_GUARD", "performance"},
|
||||
@@ -209,7 +209,7 @@ static const CfgMeta metas[] = {
|
||||
{"OPTREF_GUARD", "1", "0|1", RESTART}, {"ABILITY_GUARD", "1", "0|1", RESTART}, {"REPORT_GUARD", "1", "0|1", RESTART},
|
||||
// render fixes
|
||||
{"LOD_GUARD", "1", "0|1", RESTART}, {"FIX_C0", "1", "0|1", RESTART}, {"FIX_ZERO", "1", "0|1", RESTART},
|
||||
{"BIND_SKIP", "1", "0|1", RESTART}, {"SHADER_MEMO", "1", "0|1", RESTART}, {"PIPE_MEMO", "1", "0|1", RESTART}, {"TEX_MEMO", "1", "0|1", RESTART},
|
||||
{"BIND_SKIP", "1", "0|1", RESTART}, {"SHADER_MEMO", "1", "0|1", RESTART}, {"PIPE_MEMO", "1", "0|1", RESTART}, {"MEMX", "1", "0|1|check", RESTART}, {"TEX_MEMO", "1", "0|1", RESTART},
|
||||
{"VB_USED", "1", "0|1", RESTART},
|
||||
{"DESC_TEMPLATE", "1", "0|1", RESTART},
|
||||
{"PUSH_SKIP", "1", "0|1", RESTART}, {"SPLIT_DRAWS", "64", "0..1000", RESTART},
|
||||
|
||||
+4
-1
@@ -575,7 +575,10 @@ static void rebuild_if_needed(const char *dir) {
|
||||
int ok = plat_spawn(argv, dir);
|
||||
free(cmd);
|
||||
if (!ok) {
|
||||
rt_log("cheats: the rebuild did not start; the game starts with the code it has");
|
||||
// The skip key is for a rebuild that ran and failed or was cancelled. One that never started (a command whose
|
||||
// program is gone) leaves no key, so the next start and "Rebuild the game now" try it again.
|
||||
plat_fs_unlink(skip_path);
|
||||
rt_log("cheats: the rebuild did not start (%s); the game starts with the code it has", cmd_path);
|
||||
return;
|
||||
}
|
||||
prof_write(); // plat_exit runs no atexit handlers
|
||||
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
// Inline equality compares and copies for the replay's per-draw paths (TOTK_MEMX=0|1|check, default 1).
|
||||
// ucrtbase's memcmp walks 8 bytes a step with a call around it, and the replay compares and copies a few hundred
|
||||
// bytes at a time thousands of times a frame (pipeline states, shader answers, descriptor signatures): MX_EQ and
|
||||
// MX_CPY do the same work with 32-byte loads in the caller, so a constant size unrolls and a variable one loops
|
||||
// without a call. MX_EQ answers only equal or not, never an order. MX_CPY is memcpy: source and destination never
|
||||
// overlap. 1: the inline forms. 0: memcmp/memcpy as before. check: memcmp/memcpy, and every MX_EQ also runs the
|
||||
// inline compare and logs a different answer (vk_memx_bad). The mode is read once per process (vk_memx_init); calls
|
||||
// before that take the off path.
|
||||
#pragma once
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
#include <string.h>
|
||||
|
||||
extern int vk_memx_mode; // -1 before vk_memx_init, 0 off, 1 on, 2 check
|
||||
void vk_memx_init(void);
|
||||
void vk_memx_bad(const char *file, int line, size_t n);
|
||||
|
||||
typedef uint64_t mx_v4 __attribute__((vector_size(32), aligned(1)));
|
||||
typedef uint64_t mx_v2 __attribute__((vector_size(16), aligned(1)));
|
||||
|
||||
static inline __attribute__((always_inline)) mx_v4 mx_ld4(const uint8_t *p) {
|
||||
mx_v4 v;
|
||||
__builtin_memcpy(&v, p, 32);
|
||||
return v;
|
||||
}
|
||||
|
||||
static inline __attribute__((always_inline)) mx_v2 mx_ld2(const uint8_t *p) {
|
||||
mx_v2 v;
|
||||
__builtin_memcpy(&v, p, 16);
|
||||
return v;
|
||||
}
|
||||
|
||||
static inline __attribute__((always_inline)) uint64_t mx_ld1(const uint8_t *p) {
|
||||
uint64_t v;
|
||||
__builtin_memcpy(&v, p, 8);
|
||||
return v;
|
||||
}
|
||||
|
||||
static inline __attribute__((always_inline)) uint32_t mx_ldw(const uint8_t *p) {
|
||||
uint32_t v;
|
||||
__builtin_memcpy(&v, p, 4);
|
||||
return v;
|
||||
}
|
||||
|
||||
// 1 when the n bytes at a and b are equal. Sizes of 32 and up compare in 128-byte steps with one branch each, the
|
||||
// last partial step as an overlapping 32-byte load from the end; smaller sizes take two overlapping loads.
|
||||
static inline __attribute__((always_inline)) int mx_eq(const void *a, const void *b, size_t n) {
|
||||
const uint8_t *p = a, *q = b;
|
||||
if (n >= 32) {
|
||||
size_t i = 0;
|
||||
for (; i + 128 <= n; i += 128) {
|
||||
mx_v4 d = (mx_ld4(p + i) ^ mx_ld4(q + i)) | (mx_ld4(p + i + 32) ^ mx_ld4(q + i + 32)) |
|
||||
(mx_ld4(p + i + 64) ^ mx_ld4(q + i + 64)) | (mx_ld4(p + i + 96) ^ mx_ld4(q + i + 96));
|
||||
if (d[0] | d[1] | d[2] | d[3]) return 0;
|
||||
}
|
||||
mx_v4 d = {0, 0, 0, 0};
|
||||
for (; i + 32 <= n; i += 32) d |= mx_ld4(p + i) ^ mx_ld4(q + i);
|
||||
if (i < n) d |= mx_ld4(p + n - 32) ^ mx_ld4(q + n - 32);
|
||||
return !(d[0] | d[1] | d[2] | d[3]);
|
||||
}
|
||||
if (n >= 16) {
|
||||
mx_v2 d = (mx_ld2(p) ^ mx_ld2(q)) | (mx_ld2(p + n - 16) ^ mx_ld2(q + n - 16));
|
||||
return !(d[0] | d[1]);
|
||||
}
|
||||
if (n >= 8) return !((mx_ld1(p) ^ mx_ld1(q)) | (mx_ld1(p + n - 8) ^ mx_ld1(q + n - 8)));
|
||||
if (n >= 4) return !((mx_ldw(p) ^ mx_ldw(q)) | (mx_ldw(p + n - 4) ^ mx_ldw(q + n - 4)));
|
||||
for (size_t i = 0; i < n; i++)
|
||||
if (p[i] != q[i]) return 0;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// memcpy of n bytes between buffers that do not overlap: 32-byte moves, the last partial one overlapping the one
|
||||
// before it (it rewrites bytes it already wrote with the same values).
|
||||
static inline __attribute__((always_inline)) void mx_cpy(void *d, const void *s, size_t n) {
|
||||
uint8_t *p = d;
|
||||
const uint8_t *q = s;
|
||||
if (n >= 32) {
|
||||
mx_v4 last = mx_ld4(q + n - 32);
|
||||
for (size_t i = 0; i + 32 <= n; i += 32) {
|
||||
__asm__("" : "+r"(p)); // opaque to the loop-idiom pass, which would turn the loop back into a memcpy call
|
||||
mx_v4 v = mx_ld4(q + i);
|
||||
__builtin_memcpy(p + i, &v, 32);
|
||||
}
|
||||
__builtin_memcpy(p + n - 32, &last, 32);
|
||||
return;
|
||||
}
|
||||
if (n >= 16) {
|
||||
mx_v2 x = mx_ld2(q), y = mx_ld2(q + n - 16);
|
||||
__builtin_memcpy(p, &x, 16), __builtin_memcpy(p + n - 16, &y, 16);
|
||||
return;
|
||||
}
|
||||
if (n >= 8) {
|
||||
uint64_t x = mx_ld1(q), y = mx_ld1(q + n - 8);
|
||||
__builtin_memcpy(p, &x, 8), __builtin_memcpy(p + n - 8, &y, 8);
|
||||
return;
|
||||
}
|
||||
if (n >= 4) {
|
||||
uint32_t x = mx_ldw(q), y = mx_ldw(q + n - 4);
|
||||
__builtin_memcpy(p, &x, 4), __builtin_memcpy(p + n - 4, &y, 4);
|
||||
return;
|
||||
}
|
||||
for (size_t i = 0; i < n; i++) p[i] = q[i];
|
||||
}
|
||||
|
||||
static inline __attribute__((always_inline)) int mx_eq_at(const void *a, const void *b, size_t n, const char *file,
|
||||
int line) {
|
||||
if (vk_memx_mode == 1) return mx_eq(a, b, n);
|
||||
int r = !memcmp(a, b, n);
|
||||
if (vk_memx_mode == 2 && mx_eq(a, b, n) != r) vk_memx_bad(file, line, n);
|
||||
return r;
|
||||
}
|
||||
|
||||
static inline __attribute__((always_inline)) void mx_cpy_sw(void *d, const void *s, size_t n) {
|
||||
if (vk_memx_mode == 1) mx_cpy(d, s, n);
|
||||
else memcpy(d, s, n);
|
||||
}
|
||||
|
||||
#define MX_EQ(a, b, n) mx_eq_at((a), (b), (n), __FILE__, __LINE__)
|
||||
#define MX_NE(a, b, n) (!MX_EQ(a, b, n))
|
||||
#define MX_CPY(d, s, n) mx_cpy_sw((d), (s), (n))
|
||||
@@ -494,7 +494,7 @@ static int sync_ahead(void) {
|
||||
// that. The Vulkan back end sets the mode while it starts, before the game's first present or wait; the null
|
||||
// back end never does.
|
||||
if (on && !vk_mem_shadow_on()) {
|
||||
rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage (TOTK_SHADOW=1 on native Windows); "
|
||||
rt_log("nvn: TOTK_SYNC_AHEAD needs shadow storage (shadow = 1 in totk.ini or TOTK_SHADOW=1 on native Windows); "
|
||||
"the game waits for the GPU as before");
|
||||
on = 0;
|
||||
}
|
||||
|
||||
+5
-2
@@ -1,5 +1,6 @@
|
||||
// Ported from MacNativeGames games/totk/third_party/d2r_gfx/gfx/nvn/nvn_hmap.h @ b75be8cf (MIT, Copyright (c) 2026
|
||||
// jhaizhou; origin d2r-mobile); Chinese duplicate comments dropped, code unchanged.
|
||||
// jhaizhou; origin d2r-mobile); Chinese duplicate comments dropped, code unchanged but for
|
||||
// nvn_hm_get's key compare (MX_EQ, memx.h).
|
||||
//
|
||||
// Open-addressing hash table from fixed-length byte keys -> pointers (linear probing + backward shift on deletion, no
|
||||
// tombstones). The replay hot path uses it instead of a boxed dictionary: keys are not boxed, no allocation, a lookup
|
||||
@@ -11,6 +12,8 @@
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "memx.h"
|
||||
|
||||
typedef struct { uint8_t *keys; void **vals; uint32_t klen, cap, n; } nvn_hmap; // vals[i] == NULL means an empty slot
|
||||
|
||||
static inline uint64_t nvn_hm_hash(const void *k, uint32_t len) {
|
||||
@@ -25,7 +28,7 @@ static inline void *nvn_hm_get(const nvn_hmap *m, const void *k) {
|
||||
if (!m->n) return NULL;
|
||||
for (uint32_t i = (uint32_t)nvn_hm_hash(k, m->klen) & (m->cap - 1);; i = (i + 1) & (m->cap - 1)) {
|
||||
if (!m->vals[i]) return NULL;
|
||||
if (!memcmp(m->keys + (size_t)i * m->klen, k, m->klen)) return m->vals[i];
|
||||
if (MX_EQ(m->keys + (size_t)i * m->klen, k, m->klen)) return m->vals[i];
|
||||
}
|
||||
}
|
||||
static inline void *nvn_hm_put(nvn_hmap *m, const void *k, void *v);
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
|
||||
#include "cfg.h"
|
||||
#include "host.h"
|
||||
#include "memx.h"
|
||||
#include "vk.h"
|
||||
#include "platform/platform.h"
|
||||
#include "vk_dlaa.h"
|
||||
@@ -497,6 +498,7 @@ long vk_validation_errors_total(void) { return g_errors_total + (g_ready ? g_vk.
|
||||
|
||||
int vk_init(void) {
|
||||
plat_mutex_lock(&g_lock);
|
||||
vk_memx_init();
|
||||
if (!g_ready) {
|
||||
memset(&g_vk, 0, sizeof g_vk);
|
||||
if (create_instance() && pick_device() && create_device())
|
||||
|
||||
+50
-9
@@ -942,6 +942,18 @@ static int perf_quality(uint32_t w, uint32_t ow) {
|
||||
return k >= 0.66 ? 2 : k >= 0.57 ? 1 : k >= 0.49 ? 0 : 3;
|
||||
}
|
||||
|
||||
// A target of the scene's shape (resolution.c's scene size, within 5 %; the dynamic resolution tiers keep it). In the
|
||||
// inventory the game renders the items' icons as the slots fill in, each through a G-buffer and a light target of its
|
||||
// own with a D32F depth, square (432x432 or 426x426 at a 3840x2160 scene) and with no wider G-buffer before it in the
|
||||
// frame; DLSS armed on that, scaled it to the output and wrote it over the menu, so a weapon filled the screen on the
|
||||
// frames an icon was drawn.
|
||||
static int scene_shaped(const VkExecTarget *c) {
|
||||
unsigned o[2], s[2];
|
||||
res_sizes(o, s);
|
||||
uint64_t a = (uint64_t)c->width * s[1], b = (uint64_t)c->height * s[0];
|
||||
return a * 20 >= b * 19 && b * 20 >= a * 19;
|
||||
}
|
||||
|
||||
// dst: the target DLSS's picture goes to, NULL for the light target itself (DLAA in place).
|
||||
static void evaluate(VkExec *ex, const VkExecTarget *dst) {
|
||||
const VkExecTarget *t = &D.color, *z = &D.depth;
|
||||
@@ -1081,10 +1093,40 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) {
|
||||
(unsigned long long)D.why[2], (unsigned long long)R.made, (unsigned long long)R.missed);
|
||||
}
|
||||
|
||||
// More than half the scene size resolution.c set, on each side (the dynamic resolution tiers go down to 80 %). Around
|
||||
// the quick weapon menu the only G-buffer before the scene's end is at times the half-size one (1920x1080 at a
|
||||
// 3840x2160 scene), and the half-size RGBA16F effects target with its D32F depth armed; DLSS scaled the effects to the
|
||||
// output in place of the scene, which showed dark and blurred for that frame.
|
||||
static int scene_large(const VkExecTarget *c) {
|
||||
unsigned o[2], s[2];
|
||||
res_sizes(o, s);
|
||||
return c->width * 2 > s[0] && c->height * 2 > s[1];
|
||||
}
|
||||
|
||||
static int scene_color(VkFormat f) {
|
||||
return f == VK_FORMAT_B10G11R11_UFLOAT_PACK32 || f == VK_FORMAT_R16G16B16A16_SFLOAT;
|
||||
}
|
||||
|
||||
// The bloom chain's first pass: at most a quarter of the scene's width, into level 0 of a scene colour target. At some
|
||||
// views (seen in the yard of Lookout Landing) the game draws a full-screen pass without depth into the light target,
|
||||
// then a half-size scene colour target and its mips, and only then the light target with the scene depth. Level 1 of
|
||||
// those mips is a quarter of the scene's width; taken for the bloom it ended the scene before anything was armed, DLSS
|
||||
// skipped those frames, and the picture flickered between anti-aliased frames and plain ones as the view changed.
|
||||
// The pass also has the scene's shape (resolution.c's scene size, within 5 %) and no depth. In rain by a waterfall the
|
||||
// game drew two transparents into the light target with the scene depth before its sky lookups (25x25 and 256x256
|
||||
// RGBA16F) and its quarter-size particles (960x540 RGBA16F with a depth of their own), which then ended the armed
|
||||
// scene: DLSS ran before the mist and most transparents, which were drawn over its picture unprocessed, on runs of
|
||||
// frames, and the mist in the distance darkened or brightened for a frame wherever a frame in between came out whole.
|
||||
static int bloom_start(const VkExec *ex, uint32_t scene_w) {
|
||||
const VkExecTarget *c = &ex->color[0];
|
||||
if (!ex->pass_w || ex->pass_w * 4 > scene_w || !c->image || !scene_color(c->format) || c->level || ex->depth.image)
|
||||
return 0;
|
||||
unsigned o[2], s[2];
|
||||
res_sizes(o, s);
|
||||
uint64_t a = (uint64_t)ex->pass_w * s[1], b = (uint64_t)ex->pass_h * s[0];
|
||||
return a * 20 >= b * 19 && b * 20 >= a * 19;
|
||||
}
|
||||
|
||||
int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) {
|
||||
const VkExecTarget *c = &ex->color[0], *z = &ex->depth;
|
||||
if (!dl_active() || D.done) return 0;
|
||||
@@ -1116,8 +1158,7 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) {
|
||||
if (D.gbuf && !D.armed && !D.ended && !z->image && ex->pass_ncolor == 1 && c->image && scene_color(c->format) &&
|
||||
c->width >= D.gbuf_w && !c->level)
|
||||
D.nodepth = c->image;
|
||||
if (D.nodepth && !D.armed && !D.ended && ex->pass_w && ex->pass_w * 4 <= D.gbuf_w && c->image &&
|
||||
scene_color(c->format)) {
|
||||
if (D.nodepth && !D.armed && !D.ended && bloom_start(ex, D.gbuf_w)) {
|
||||
D.end_fence = ex->fence, D.end_pos = ex->pos;
|
||||
__atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE);
|
||||
}
|
||||
@@ -1130,9 +1171,9 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) {
|
||||
// first pair to arm; DLSS then scaled the effects into the light target, which filled the lens with garbage in the
|
||||
// DLAA mode.
|
||||
if (D.gbuf && !D.ended && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && z->image && z->format == VK_FORMAT_D32_SFLOAT &&
|
||||
c->width == z->width && c->height == z->height && c->width >= DLAA_SCENE_MIN_W && c->width >= D.gbuf_w &&
|
||||
c->width == z->width && c->height == z->height && c->width >= DLAA_SCENE_MIN_W && c->width >= D.gbuf_w && scene_shaped(c) &&
|
||||
(!D.armed || (c->width >= D.color.width && c->height >= D.color.height)) &&
|
||||
c->samples <= 1 && !c->is3d) {
|
||||
c->samples <= 1 && !c->is3d && scene_large(c)) {
|
||||
int first = !D.armed;
|
||||
D.color = *c, D.depth = *z, D.armed = 1, D.nodepth = VK_NULL_HANDLE;
|
||||
if (!first) return 0;
|
||||
@@ -1141,11 +1182,11 @@ int vk_dlaa_plan(const VkExec *ex, VkExecJob *j) {
|
||||
}
|
||||
if (!D.armed) return 0;
|
||||
if (!D.ended) {
|
||||
// The bloom chain's first pass ends the scene: a pass at most a quarter of the scene's width into a scene
|
||||
// colour target. With the camera rune or the scope open the game draws a 160-wide depth-only pass and 160x90
|
||||
// RG16F and RGBA8 passes right after the lighting, then more geometry and the transparents; DLSS ended the
|
||||
// scene at those, the next G-buffer pass disarmed it and the scaling pass found nothing to evaluate.
|
||||
if (!ex->pass_w || ex->pass_w * 4 > D.color.width || !c->image || !scene_color(c->format)) return 0;
|
||||
// The bloom chain's first pass ends the scene (bloom_start). With the camera rune or the scope open the game
|
||||
// draws a 160-wide depth-only pass and 160x90 RG16F and RGBA8 passes right after the lighting, then more
|
||||
// geometry and the transparents; DLSS ended the scene at those, the next G-buffer pass disarmed it and the
|
||||
// scaling pass found nothing to evaluate.
|
||||
if (!bloom_start(ex, D.color.width)) return 0;
|
||||
D.end_fence = ex->fence, D.end_pos = ex->pos;
|
||||
__atomic_store_n(&D.ended, 1, __ATOMIC_RELEASE);
|
||||
unsigned o[2], s[2];
|
||||
|
||||
+60
-38
@@ -30,6 +30,7 @@
|
||||
#include "vk_exec.h"
|
||||
#include "platform/platform.h"
|
||||
#include "vk_dlaa.h"
|
||||
#include "memx.h"
|
||||
|
||||
#define MAX_BINDINGS 128
|
||||
#define MAX_DESC 1024
|
||||
@@ -227,7 +228,7 @@ typedef struct { uint32_t n, bytes; uint8_t sup, c0, ok, np; RsPtr p[RS_PTRS]; B
|
||||
// the 2048-slot direct-mapped table this replaced missed 3% of them on conflicts.
|
||||
#define SHADER_SETS 2048
|
||||
#define SHADER_WAYS 4
|
||||
typedef struct { uint64_t serial; int stage; int used; GfxShaderAnswers a; const GfxShader *s; } ShaderEnt;
|
||||
typedef struct { uint64_t serial, h; int stage; int used; GfxShaderAnswers a; const GfxShader *s; } ShaderEnt; // h: hash of a
|
||||
|
||||
// Which shader of a program serves each NVN stage, and its code length, per program serial (new at every
|
||||
// nvnProgramSetShaders): the scan reads two words of every stage's control block, cold guest cache lines, and runs
|
||||
@@ -1055,12 +1056,13 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin
|
||||
answers_for(ex, st, ctl, slot_mask_of(D, serial, st), &a);
|
||||
a.mvprev = (uint8_t)(sf_mvprev && st == NVN_STAGE_VS);
|
||||
ShaderEnt *set = NULL;
|
||||
uint64_t h = 0;
|
||||
if (serial) {
|
||||
uint64_t h = (serial * 0x9E3779B97F4A7C15ull ^ (uint64_t)st) ^ nvn_hm_hash(&a, sizeof a); // a is zero-padded
|
||||
h = (serial * 0x9E3779B97F4A7C15ull ^ (uint64_t)st) ^ nvn_hm_hash(&a, sizeof a); // a is zero-padded
|
||||
set = D->shader_tab[(h ^ h >> 29) & (SHADER_SETS - 1)];
|
||||
for (int w = 0; w < SHADER_WAYS && set[w].used; w++) {
|
||||
ShaderEnt *e = &set[w];
|
||||
if (e->serial != serial || e->stage != st || memcmp(&e->a, &a, sizeof a)) continue;
|
||||
if (e->serial != serial || e->stage != st || (vk_memx_mode == 1 && e->h != h) || MX_NE(&e->a, &a, sizeof a)) continue;
|
||||
if (w) { // to the front
|
||||
ShaderEnt hit = *e;
|
||||
memmove(&set[1], &set[0], w * sizeof *set);
|
||||
@@ -1081,7 +1083,7 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin
|
||||
(unsigned long long)serial);
|
||||
else if (set) { // in front, the least recently used way drops out
|
||||
memmove(&set[1], &set[0], (SHADER_WAYS - 1) * sizeof *set);
|
||||
set[0] = (ShaderEnt){.serial = serial, .stage = st, .used = 1, .a = a, .s = s};
|
||||
set[0] = (ShaderEnt){.serial = serial, .h = h, .stage = st, .used = 1, .a = a, .s = s};
|
||||
}
|
||||
return s;
|
||||
}
|
||||
@@ -1202,14 +1204,14 @@ static VkDescriptorBufferInfo c0_info(VkExec *ex, int st) {
|
||||
memcpy(in.ssbo, ex->ssbo[st], sizeof in.ssbo);
|
||||
if (st == NVN_STAGE_CS) memcpy(in.ubo, ex->ubo[st], sizeof in.ubo), memcpy(in.dim, D->c0_dim, sizeof in.dim);
|
||||
else in.binst = D->c0_binst;
|
||||
if (D->B.c0_valid[k] && !memcmp(&in, &D->B.c0_in[k], sizeof in))
|
||||
if (D->B.c0_valid[k] && MX_EQ(&in, &D->B.c0_in[k], sizeof in))
|
||||
return (VkDescriptorBufferInfo){D->B.c0_buf[k], D->B.c0_off[k], VK_C0_BYTES};
|
||||
D->B.c0_in[k] = in;
|
||||
uint8_t tmp[VK_C0_BYTES];
|
||||
vk_c0_fill(tmp, st, ex->ssbo[st], ex->ubo[st], D->c0_dim, D->c0_binst);
|
||||
if (!D->B.c0_valid[k] || memcmp(tmp, D->B.c0[k], VK_C0_BYTES)) {
|
||||
memcpy(scratch(ex, VK_C0_BYTES, &D->B.c0_buf[k], &D->B.c0_off[k]), tmp, VK_C0_BYTES);
|
||||
memcpy(D->B.c0[k], tmp, VK_C0_BYTES);
|
||||
if (!D->B.c0_valid[k] || MX_NE(tmp, D->B.c0[k], VK_C0_BYTES)) {
|
||||
MX_CPY(scratch(ex, VK_C0_BYTES, &D->B.c0_buf[k], &D->B.c0_off[k]), tmp, VK_C0_BYTES);
|
||||
MX_CPY(D->B.c0[k], tmp, VK_C0_BYTES);
|
||||
D->B.c0_valid[k] = 1;
|
||||
}
|
||||
return (VkDescriptorBufferInfo){D->B.c0_buf[k], D->B.c0_off[k], VK_C0_BYTES};
|
||||
@@ -1314,7 +1316,7 @@ static int upd_read(const VkExec *ex, uint64_t a, void *out, size_t n) {
|
||||
const VkExecUpd *e = &u[i & mask];
|
||||
if (a >= e->hi || a + n <= e->lo) continue;
|
||||
if (a < e->lo || a + n > e->hi) return 0; // partly covered: memory
|
||||
memcpy(out, e->data + (a - e->lo), n);
|
||||
MX_CPY(out, e->data + (a - e->lo), n);
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
@@ -1327,7 +1329,7 @@ static void upd_apply(const VkExec *ex, uint64_t a, uint8_t *out, size_t n) {
|
||||
for (uint32_t i = U->n > UPD_N ? U->n - UPD_N : 0; i < U->n; i++) {
|
||||
const VkExecUpd *e = &U->u[i % UPD_N];
|
||||
uint64_t lo = e->lo > a ? e->lo : a, hi = e->hi < a + n ? e->hi : a + n;
|
||||
if (lo < hi) memcpy(out + (lo - a), e->data + (lo - e->lo), hi - lo);
|
||||
if (lo < hi) MX_CPY(out + (lo - a), e->data + (lo - e->lo), hi - lo);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2444,15 +2446,15 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info, const uint64_t *b
|
||||
if (b->kind == VK_PIPE_SUPPORT) {
|
||||
float in[3];
|
||||
support_in(ex, in);
|
||||
if (!D->B.sup_valid || memcmp(in, D->B.sup_in, sizeof in)) { // support_fill reads only these
|
||||
if (!D->B.sup_valid || MX_NE(in, D->B.sup_in, sizeof in)) { // support_fill reads only these
|
||||
uint8_t tmp[SUPPORT_BYTES];
|
||||
support_fill(tmp, in[0], in[1], in[2]);
|
||||
memcpy(D->B.sup_in, in, sizeof in);
|
||||
if (!D->B.sup_valid || memcmp(tmp, D->B.support, SUPPORT_BYTES)) { // a new buffer only when the contents changed
|
||||
MX_CPY(D->B.sup_in, in, sizeof in);
|
||||
if (!D->B.sup_valid || MX_NE(tmp, D->B.support, SUPPORT_BYTES)) { // a new buffer only when the contents changed
|
||||
VkBuffer sb;
|
||||
VkDeviceSize so;
|
||||
memcpy(scratch(ex, SUPPORT_BYTES, &sb, &so), tmp, SUPPORT_BYTES);
|
||||
memcpy(D->B.support, tmp, SUPPORT_BYTES);
|
||||
MX_CPY(scratch(ex, SUPPORT_BYTES, &sb, &so), tmp, SUPPORT_BYTES);
|
||||
MX_CPY(D->B.support, tmp, SUPPORT_BYTES);
|
||||
D->B.sup_buf = sb, D->B.sup_off = so, D->B.sup_valid = 1;
|
||||
}
|
||||
}
|
||||
@@ -2552,7 +2554,7 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info, const uint64_t *b
|
||||
D->BS.push_skip++;
|
||||
continue;
|
||||
}
|
||||
if (n && D->B.ds_pushed[s] && D->DS.pn[s] == n && !memcmp(D->DS.psig[s], ds, n)) {
|
||||
if (n && D->B.ds_pushed[s] && D->DS.pn[s] == n && MX_EQ(D->DS.psig[s], ds, n)) {
|
||||
D->BS.push_skip++;
|
||||
continue;
|
||||
}
|
||||
@@ -2562,7 +2564,7 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info, const uint64_t *b
|
||||
D->RS.cmds++;
|
||||
if (gfx) {
|
||||
D->B.ds_bound[s] = VK_NULL_HANDLE, D->B.ds_pushed[s] = n != 0;
|
||||
if (n) D->DS.pn[s] = n, memcpy(D->DS.psig[s], ds, n);
|
||||
if (n) D->DS.pn[s] = n, MX_CPY(D->DS.psig[s], ds, n);
|
||||
}
|
||||
} else if (!gfx || s > 1) {
|
||||
VkDescriptorSet set = alloc_set(ex, s ? info->set_layout1 : info->set_layout);
|
||||
@@ -2572,11 +2574,11 @@ static void bind_resources(VkExec *ex, const VkPipeInfo *info, const uint64_t *b
|
||||
VkDescriptorSetLayout sl = s ? info->set_layout1 : info->set_layout;
|
||||
VkDescriptorSet set = D->B.ds_last[s];
|
||||
int kept = inc && bim == 1 && !changed[s] && set && D->B.ds_last_layout[s] == sl && D->DS.n[s] == dbytes;
|
||||
if (!kept && (!set || D->B.ds_last_layout[s] != sl || D->DS.n[s] != dbytes || memcmp(D->DS.sig[s], ds, dbytes))) {
|
||||
if (!kept && (!set || D->B.ds_last_layout[s] != sl || D->DS.n[s] != dbytes || MX_NE(D->DS.sig[s], ds, dbytes))) {
|
||||
set = bim == 1 && !ex->chunked ? alloc_set_batched(ex, sl) : alloc_set(ex, sl);
|
||||
write_set(info, s, set, ds, w, i0, i1);
|
||||
D->B.ds_last[s] = set, D->B.ds_last_layout[s] = sl, D->DS.n[s] = dbytes;
|
||||
memcpy(D->DS.sig[s], ds, dbytes);
|
||||
MX_CPY(D->DS.sig[s], ds, dbytes);
|
||||
D->BS.set_new++;
|
||||
D->RS.cmds++;
|
||||
} else {
|
||||
@@ -2788,7 +2790,7 @@ static void memo_key(const VkExec *ex, uint32_t gen, MemoKey *k) {
|
||||
k->pipe_gen = ex->pipe_gen;
|
||||
memcpy(k->tex[0], ex->tex[NVN_STAGE_VS], sizeof k->tex[0]), memcpy(k->tex[1], ex->tex[NVN_STAGE_FS], sizeof k->tex[1]);
|
||||
memcpy(k->img[0], ex->img[NVN_STAGE_VS], sizeof k->img[0]), memcpy(k->img[1], ex->img[NVN_STAGE_FS], sizeof k->img[1]);
|
||||
memcpy(&k->pipe, &ex->pipe, sizeof k->pipe);
|
||||
MX_CPY(&k->pipe, &ex->pipe, sizeof k->pipe);
|
||||
}
|
||||
|
||||
static int bind_inc_mode(void);
|
||||
@@ -2799,14 +2801,14 @@ static void bi_mismatch(VkExec *ex, int kind, const char *fmt, ...);
|
||||
static int memo_match(VkExec *ex, uint32_t gen, MemoKey *k) {
|
||||
if (k->prog != ex->prog || k->serial != ex->prog->serial || k->gen != gen) return 0;
|
||||
int bim = bind_inc_mode(), same_gen = k->pipe_gen == ex->pipe_gen;
|
||||
if (!(bim == 1 && same_gen && !ex->chunked) && memcmp(&k->pipe, &ex->pipe, sizeof k->pipe)) {
|
||||
if (!(bim == 1 && same_gen && !ex->chunked) && MX_NE(&k->pipe, &ex->pipe, sizeof k->pipe)) {
|
||||
if (bim == 2 && same_gen && !ex->chunked) bi_mismatch(ex, BIM_PIPE, "shader memo: bound state changed without its generation");
|
||||
return 0;
|
||||
}
|
||||
k->pipe_gen = ex->pipe_gen;
|
||||
if (k->tex_serial == ex->tex_serial) return 1;
|
||||
if (memcmp(k->tex[0], ex->tex[NVN_STAGE_VS], sizeof k->tex[0]) || memcmp(k->tex[1], ex->tex[NVN_STAGE_FS], sizeof k->tex[1]) ||
|
||||
memcmp(k->img[0], ex->img[NVN_STAGE_VS], sizeof k->img[0]) || memcmp(k->img[1], ex->img[NVN_STAGE_FS], sizeof k->img[1]))
|
||||
if (MX_NE(k->tex[0], ex->tex[NVN_STAGE_VS], sizeof k->tex[0]) || MX_NE(k->tex[1], ex->tex[NVN_STAGE_FS], sizeof k->tex[1]) ||
|
||||
MX_NE(k->img[0], ex->img[NVN_STAGE_VS], sizeof k->img[0]) || MX_NE(k->img[1], ex->img[NVN_STAGE_FS], sizeof k->img[1]))
|
||||
return 0;
|
||||
k->tex_serial = ex->tex_serial;
|
||||
return 1;
|
||||
@@ -2821,6 +2823,26 @@ static int shader_memo_on(void) {
|
||||
return on;
|
||||
}
|
||||
|
||||
// ---- inline compares and copies (memx.h, TOTK_MEMX)
|
||||
int vk_memx_mode = -1;
|
||||
|
||||
void vk_memx_init(void) {
|
||||
if (vk_memx_mode >= 0) return;
|
||||
const char *e = cfg_value("TOTK_MEMX");
|
||||
int m = !strcmp(e, "check") ? 2 : strcmp(e, "0") && strcmp(e, "off") ? 1 : 0;
|
||||
rt_log("vk: inline compares and copies (TOTK_MEMX) %s", m == 2 ? "check" : m ? "on" : "off");
|
||||
__atomic_store_n(&vk_memx_mode, m, __ATOMIC_RELAXED);
|
||||
}
|
||||
|
||||
void vk_memx_bad(const char *file, int line, size_t n) {
|
||||
static int logged;
|
||||
static long total;
|
||||
long t = __atomic_add_fetch(&total, 1, __ATOMIC_RELAXED);
|
||||
if (__atomic_fetch_add(&logged, 1, __ATOMIC_RELAXED) < 20 || t % 100000 == 0)
|
||||
rt_log("vk: TOTK_MEMX check: the inline compare of %zu bytes at %s:%d differs from memcmp (%ld so far)", n, file,
|
||||
line, t);
|
||||
}
|
||||
|
||||
static int pipe_memo_on(void) {
|
||||
static int on = -1;
|
||||
if (on < 0) on = cfg_bool("TOTK_PIPE_MEMO");
|
||||
@@ -3115,7 +3137,7 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) {
|
||||
if (pg_hit && pg == 1) {
|
||||
sp = &D->PM.s;
|
||||
} else {
|
||||
s = ex->pipe;
|
||||
MX_CPY(&s, &ex->pipe, sizeof s);
|
||||
s.vs = vs, s.fs = fs;
|
||||
s.feedback = fb;
|
||||
s.prim = (uint8_t)prim;
|
||||
@@ -3151,15 +3173,15 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) {
|
||||
uint64_t w0 = 0;
|
||||
PipeCache *pc = NULL;
|
||||
int pg_take = pg_hit && pg == 1; // the memo's pipeline without building or comparing the state
|
||||
if (!pg_take && pipe_memo_on() && !(D->PM.valid && !memcmp(&D->PM.s, &s, sizeof s)))
|
||||
if (!pg_take && pipe_memo_on() && !(D->PM.valid && MX_EQ(&D->PM.s, &s, sizeof s)))
|
||||
pc = &D->PC[nvn_hm_hash(&s, sizeof s) & (PC_N - 1)];
|
||||
if (pg_take) {
|
||||
info = D->PM.info, pending = 0;
|
||||
p = __atomic_load_n(&info->pipeline, __ATOMIC_ACQUIRE);
|
||||
} else if (pipe_memo_on() && (!pc || (pc->info && !memcmp(&pc->s, &s, sizeof s)))) {
|
||||
} else if (pipe_memo_on() && (!pc || (pc->info && MX_EQ(&pc->s, &s, sizeof s)))) {
|
||||
info = pc ? pc->info : D->PM.info, pending = 0;
|
||||
p = __atomic_load_n(&info->pipeline, __ATOMIC_ACQUIRE);
|
||||
if (pc) D->PM.valid = 1, D->PM.s = s, D->PM.info = info, D->B.st_pm = 0;
|
||||
if (pc) D->PM.valid = 1, MX_CPY(&D->PM.s, &s, sizeof s), D->PM.info = info, D->B.st_pm = 0;
|
||||
D->PM.gen = pgen; // s equals PM.s: built from pipe as it is now
|
||||
} else {
|
||||
w0 = t0 ? t0 : wait_ms ? kern_now_ns() : 0; // the clock is read only for draws that may wait
|
||||
@@ -3176,8 +3198,8 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) {
|
||||
p = vk_pipeline_graphics_async(&s, &info, &pending, ONESHOT_WAIT_MS);
|
||||
}
|
||||
D->PM.valid = !pending && p && info, D->B.st_pm = 0;
|
||||
if (D->PM.valid) D->PM.s = s, D->PM.info = info, D->PM.gen = pgen;
|
||||
if (D->PM.valid && pc) pc->s = s, pc->info = info;
|
||||
if (D->PM.valid) MX_CPY(&D->PM.s, &s, sizeof s), D->PM.info = info, D->PM.gen = pgen;
|
||||
if (D->PM.valid && pc) MX_CPY(&pc->s, &s, sizeof s), pc->info = info;
|
||||
}
|
||||
uint64_t waited = w0 ? kern_now_ns() - w0 : 0;
|
||||
static int wait_logs;
|
||||
@@ -3193,7 +3215,7 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) {
|
||||
if (!p || !info) return warn_once(W_PIPELINE, "a graphics pipeline could not be created: draws with it are skipped"), 0;
|
||||
bind_pipeline(ex, VK_PIPELINE_BIND_POINT_GRAPHICS, p);
|
||||
// On a generation hit the state recorded last is the memo's when st_pm says so (both were the last draw's).
|
||||
int st_same = D->B.st_valid && (pg_take && D->B.st_pm ? 1 : !memcmp(&D->B.st, sp, sizeof *sp));
|
||||
int st_same = D->B.st_valid && (pg_take && D->B.st_pm ? 1 : MX_EQ(&D->B.st, sp, sizeof *sp));
|
||||
if (pg_hit && pg == 2 && D->B.st_valid && D->B.st_pm && !st_same)
|
||||
bi_mismatch(ex, BIM_DYN, "the recorded dynamic state differs from the pipeline memo's");
|
||||
if (!st_same) {
|
||||
@@ -3213,7 +3235,7 @@ static int prepare_draw(VkExec *ex, int prim, int32_t count, int restart) {
|
||||
__atomic_fetch_add(&DD.calls, 1, __ATOMIC_RELAXED), __atomic_fetch_add(&DD.cmds, cmds, __ATOMIC_RELAXED);
|
||||
if (!cmds) __atomic_fetch_add(&DD.empty, 1, __ATOMIC_RELAXED);
|
||||
ex->recorded += cmds != 0;
|
||||
D->B.st = *sp, D->B.st_valid = 1;
|
||||
MX_CPY(&D->B.st, sp, sizeof *sp), D->B.st_valid = 1;
|
||||
if (ex->mark && ex->fr.draws == ex->mark_draws0) ex->mark->first[0] = vs, ex->mark->first[1] = fs;
|
||||
}
|
||||
D->B.st_pm = D->PM.valid; // st is this draw's state, which PM holds while valid
|
||||
@@ -4010,7 +4032,7 @@ int vk_mvobj_execute(VkExec *ex, uint64_t cam, VkImageView mv, VkImageView depth
|
||||
static void bind_index(VkExec *ex, const void *data, uint64_t bytes, VkIndexType type) {
|
||||
VkBuffer b;
|
||||
VkDeviceSize o;
|
||||
memcpy(scratch(ex, bytes, &b, &o), data, bytes);
|
||||
MX_CPY(scratch(ex, bytes, &b, &o), data, bytes);
|
||||
vkCmdBindIndexBuffer(ex->cb, b, o, type);
|
||||
ex->recorded++;
|
||||
}
|
||||
@@ -4389,8 +4411,8 @@ static int ub_bind(VkExec *ex, uint64_t addr, uint64_t n, uint64_t align, VkDesc
|
||||
if (ex->draw->BI.cur) ex->draw->BI.cur->vol = 1;
|
||||
const uint8_t *base = vk_usnap_bytes(addr, n);
|
||||
VkMemSpan sp;
|
||||
if (base) memcpy(dst, base, n);
|
||||
else if (vk_mem_resolve(addr, n, &sp)) memcpy(dst, (const void *)(uintptr_t)addr, n);
|
||||
if (base) MX_CPY(dst, base, n);
|
||||
else if (vk_mem_resolve(addr, n, &sp)) MX_CPY(dst, (const void *)(uintptr_t)addr, n);
|
||||
else memset(dst, 0, n);
|
||||
if (ov_foreign_hits(addr, hi)) OV.c_mixed++;
|
||||
upd_apply(ex, addr, dst, n);
|
||||
@@ -4427,7 +4449,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) {
|
||||
int here = ex->in_pass || ov_out_on();
|
||||
if (here && OV.n < OV_N && OV.np < OV_N && !ov_foreign_hits(u->dst, u->dst + u->size)) {
|
||||
uint8_t *d = scratch(ex, u->size, &b, &o);
|
||||
memcpy(d, u->data, u->size);
|
||||
MX_CPY(d, u->data, u->size);
|
||||
VK_MEM_PICK(sp);
|
||||
ov_add(u->dst, u->dst + u->size, b, o, &sp);
|
||||
vk_mem_gpu_wrote(&sp, u->size);
|
||||
@@ -4444,7 +4466,7 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) {
|
||||
ov_retire(u->dst, u->dst + u->size);
|
||||
}
|
||||
vk_exec_barrier(ex); // earlier draws finished with the old contents; overlaid updates' copies go first
|
||||
memcpy(scratch(ex, u->size, &b, &o), u->data, u->size);
|
||||
MX_CPY(scratch(ex, u->size, &b, &o), u->data, u->size);
|
||||
VK_MEM_PICK(sp);
|
||||
VkBufferCopy cp = {o, sp.offset, u->size};
|
||||
vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp);
|
||||
@@ -4525,7 +4547,7 @@ int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload) {
|
||||
const p_bind_vattr *b = payload;
|
||||
if (b->first < 0 || b->n < 0 || b->first + b->n > 16) break;
|
||||
uint32_t v[16] = {0}; // MNG: the new list replaces the old one
|
||||
memcpy(&v[b->first], b->st, sizeof(uint32_t) * (size_t)b->n);
|
||||
MX_CPY(&v[b->first], b->st, sizeof(uint32_t) * (size_t)b->n);
|
||||
pipe_put(ex, ex->pipe.vattr, v, sizeof v);
|
||||
break;
|
||||
}
|
||||
@@ -4534,7 +4556,7 @@ int vk_exec_bind(VkExec *ex, uint32_t op, const void *payload) {
|
||||
if (b->first < 0 || b->n < 0 || b->first + b->n > 16) break;
|
||||
vs_st v[16];
|
||||
memset(v, 0, sizeof v);
|
||||
memcpy(&v[b->first], b->st, sizeof(vs_st) * (size_t)b->n);
|
||||
MX_CPY(&v[b->first], b->st, sizeof(vs_st) * (size_t)b->n);
|
||||
pipe_put(ex, ex->pipe.vstream, v, sizeof v);
|
||||
break;
|
||||
}
|
||||
|
||||
+1
-1
@@ -991,7 +991,7 @@ static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) {
|
||||
// - GPU -> guest: the replay notes each GPU write into CPU-visible memory at record time (vk_mem_gpu_pending) and
|
||||
// reports it at the fence (vk_gpu_written), which copies those bytes back into guest memory. Until then the CPU
|
||||
// copies skip them: a page the CPU and the GPU both write keeps the GPU's bytes.
|
||||
// TOTK_SHADOW=1/0 forces the mode on or off; the default is on for Linux and Wine.
|
||||
// TOTK_SHADOW=1/0 (shadow in totk.ini) forces the mode on or off; the default is on for Linux and Wine.
|
||||
typedef struct { uint64_t lo, hi; } SRange;
|
||||
static PlatMutex shadow_lock = PLAT_MUTEX_INIT; // after pool_lock; before gen_lock
|
||||
static PoolRec **stouch; // shadow_lock: pools resolved since the last sync
|
||||
|
||||
@@ -49,6 +49,7 @@
|
||||
#include "cfg.h"
|
||||
#include "gfx.h"
|
||||
#include "host.h"
|
||||
#include "memx.h"
|
||||
#include "nvn_hmap.h"
|
||||
#include "platform/platform.h"
|
||||
#include "tex.h"
|
||||
@@ -1187,7 +1188,7 @@ static uint32_t vs_outs(const GfxShader *vs, const GfxShader *fs, GfxShaderIo io
|
||||
|
||||
static Pipe *pipe_find(const void *key, size_t size, uint64_t h) {
|
||||
for (Pipe *p = pipes[h % PIPE_BUCKETS]; p; p = p->next)
|
||||
if (p->hash == h && p->key_size == size && !memcmp(p->key, key, size)) return p;
|
||||
if (p->hash == h && p->key_size == size && MX_EQ(p->key, key, size)) return p;
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -3278,7 +3279,7 @@ uint32_t vk_pipeline_set_dynamic_delta(VkCommandBuffer cb, const VkPipeState *s,
|
||||
if (v.restart != o->restart) vkCmdSetPrimitiveRestartEnable(cb, v.restart), k++;
|
||||
// The vertex input is made from the vertex shader's inputs and the attribute and stream states alone
|
||||
// (vertex_input): a draw that keeps all three keeps it, and the bindings it reads.
|
||||
if (s->vs != DC.vs || memcmp(s->vattr, DC.vattr, sizeof DC.vattr) || memcmp(s->vstream, DC.vstream, sizeof DC.vstream)) {
|
||||
if (s->vs != DC.vs || MX_NE(s->vattr, DC.vattr, sizeof DC.vattr) || MX_NE(s->vstream, DC.vstream, sizeof DC.vstream)) {
|
||||
uint32_t streams = VK_PIPE_ALL_BINDINGS;
|
||||
if (s->vs) {
|
||||
VtxInput vi;
|
||||
@@ -3288,7 +3289,7 @@ uint32_t vk_pipeline_set_dynamic_delta(VkCommandBuffer cb, const VkPipeState *s,
|
||||
for (uint32_t i = 0; i < vi.n_bind; i++) streams |= 1u << vi.b[i].binding;
|
||||
}
|
||||
DC.vs = s->vs, DC.streams = streams;
|
||||
memcpy(DC.vattr, s->vattr, sizeof DC.vattr), memcpy(DC.vstream, s->vstream, sizeof DC.vstream);
|
||||
MX_CPY(DC.vattr, s->vattr, sizeof DC.vattr), MX_CPY(DC.vstream, s->vstream, sizeof DC.vstream);
|
||||
}
|
||||
if (P.dyn_clamp && v.clamp != o->clamp) rec_bool(P.set_clamp, cb, v.clamp), k++;
|
||||
if (P.dyn_poly && v.poly != o->poly) rec_polygon_mode(P.set_poly, cb, v.poly), k++;
|
||||
@@ -3297,10 +3298,10 @@ uint32_t vk_pipeline_set_dynamic_delta(VkCommandBuffer cb, const VkPipeState *s,
|
||||
// One with fewer leaves the ones past its count as they were, which no draw of it reads.
|
||||
uint32_t n = (uint32_t)v.n;
|
||||
int all = v.n > o->n;
|
||||
if (n && P.dyn_blend_en && (all || memcmp(v.en, o->en, n * sizeof *v.en))) rec_blend_enable(P.set_blend_en, cb, 0, n, v.en), k++;
|
||||
if (n && P.dyn_blend_eq && (all || memcmp(v.eq, o->eq, n * sizeof *v.eq))) rec_blend_equation(P.set_blend_eq, cb, 0, n, v.eq), k++;
|
||||
if (n && P.dyn_wmask && (all || memcmp(v.wm, o->wm, n * sizeof *v.wm))) rec_write_mask(P.set_wmask, cb, 0, n, v.wm), k++;
|
||||
DC.v = v; // the invariant now covers [0, v.n) only
|
||||
if (n && P.dyn_blend_en && (all || MX_NE(v.en, o->en, n * sizeof *v.en))) rec_blend_enable(P.set_blend_en, cb, 0, n, v.en), k++;
|
||||
if (n && P.dyn_blend_eq && (all || MX_NE(v.eq, o->eq, n * sizeof *v.eq))) rec_blend_equation(P.set_blend_eq, cb, 0, n, v.eq), k++;
|
||||
if (n && P.dyn_wmask && (all || MX_NE(v.wm, o->wm, n * sizeof *v.wm))) rec_write_mask(P.set_wmask, cb, 0, n, v.wm), k++;
|
||||
MX_CPY(&DC.v, &v, sizeof v); // the invariant now covers [0, v.n) only
|
||||
*cmds = k;
|
||||
return DC.streams;
|
||||
}
|
||||
|
||||
+8
-2
@@ -14,6 +14,7 @@
|
||||
|
||||
#include "cfg.h"
|
||||
#include "host.h"
|
||||
#include "memx.h"
|
||||
#include "vk.h"
|
||||
#include "platform/platform.h"
|
||||
|
||||
@@ -175,11 +176,16 @@ static void *put(uint32_t op, size_t bytes) {
|
||||
// cost the producer an event set and the recorder a wake-up. Savage Wind Recomp measured that at 6.7% of its render
|
||||
// thread's busy time on 4 cores and moved to 64 KiB there (runtime/nvn/vulkan/record.c rec_init, publish_every).
|
||||
// TOTK_REC_PUB=<words> sets the batch; =ab: 64 words in the A blocks of TOTK_AB, 8192 in the B blocks.
|
||||
// The wake goes through wake(), whose full fence orders the head store before the read of Q.sleeping. rec_main sets
|
||||
// Q.sleeping and then reads head; without the fence x86 may read Q.sleeping before the head store is visible, both
|
||||
// sides miss each other and the recorder sleeps out its 2 ms timeout with the packets waiting. Since rec_submit returns
|
||||
// without a drain (TOTK_REC_ASYNC=1), its publish is the last one of a submission, and every queue user then waited in
|
||||
// rec_queue_wait for that timeout (issue #1174: 1.6 to 3.4 ms a wait on an RX 9070 XT).
|
||||
static int pub_ab;
|
||||
static void pub_now(void) {
|
||||
Q.pub_h = Q.put_h;
|
||||
__atomic_store_n(&Q.head, Q.put_h, __ATOMIC_RELEASE);
|
||||
if (__atomic_load_n(&Q.sleeping, __ATOMIC_RELAXED)) plat_event_set(&wake_ev), RS_.wakes++;
|
||||
wake();
|
||||
}
|
||||
|
||||
static void pub(void) {
|
||||
@@ -473,7 +479,7 @@ int rec_owns(VkCommandBuffer cb) { return OWNED(cb); }
|
||||
static void *tail_copy(uint8_t **at, const void *arr, size_t bytes) {
|
||||
if (!arr || !bytes) return NULL;
|
||||
void *d = *at;
|
||||
memcpy(d, arr, bytes);
|
||||
MX_CPY(d, arr, bytes);
|
||||
*at += (bytes + 7) & ~(size_t)7;
|
||||
return d;
|
||||
}
|
||||
|
||||
+20
-6
@@ -321,6 +321,24 @@ static void init_image(Tex *e, VkCommandBuffer cb) {
|
||||
}
|
||||
}
|
||||
|
||||
// The first transition of a new image, in the upload batch, which runs ahead of the submission being recorded. A chunk
|
||||
// of a chunked replay records it into its own command buffer at its op instead when the image is bound into pool
|
||||
// memory: the chunks before it may be on the GPU already, and a clear of the new image (its depth) there would land on
|
||||
// what they rendered into the same storage for the chunks after them to read. The inventory and the quick weapon menu
|
||||
// render the items' icons into new targets at the storage the frame's other targets share, and a black strip showed
|
||||
// right of the quick menu for a frame where its new depth targets were cleared.
|
||||
static void init_new(Tex *e) {
|
||||
int reopen = 0;
|
||||
VkCommandBuffer cb = e->bound_to && vk_exec_replay_pos() != UINT32_MAX ? vk_exec_inline_begin(&reopen) : VK_NULL_HANDLE;
|
||||
if (cb) {
|
||||
init_image(e, cb);
|
||||
vk_exec_inline_end(reopen);
|
||||
return;
|
||||
}
|
||||
init_image(e, vk_upload_begin());
|
||||
vk_upload_end();
|
||||
}
|
||||
|
||||
// TOTK_SHADOW_SCALE (1 to 4): the game's shadow maps are made this many times larger on each side. TotK renders its
|
||||
// sun cascades into a 960x960 DEPTH16 2D array of 4 layers and its static shadows into 1440x1440 and 720x720 arrays of 2,
|
||||
// whatever the output resolution, and shrinks the cascades under load (872, 848, 800, 480 and so on). Those are made
|
||||
@@ -505,10 +523,7 @@ static Tex *create_tex(const tex_t *t, const TexKey *k, int defer_init) {
|
||||
p->fb_usage = (uint8_t)fb_usage;
|
||||
p->scale = (uint8_t)scale;
|
||||
|
||||
if (!defer_init) {
|
||||
init_image(e, vk_upload_begin());
|
||||
vk_upload_end();
|
||||
}
|
||||
if (!defer_init) init_new(e);
|
||||
return e;
|
||||
}
|
||||
|
||||
@@ -1435,8 +1450,7 @@ static Tex *entry_of(const tex_t *t) {
|
||||
e->pub.gpu_owned = 1;
|
||||
}
|
||||
} else {
|
||||
init_image(e, vk_upload_begin());
|
||||
vk_upload_end();
|
||||
init_new(e);
|
||||
if (alias) {
|
||||
if (e->pub.mirrored) n_alias_upload++;
|
||||
else n_alias_uninit++;
|
||||
|
||||
@@ -526,6 +526,21 @@ class CleanupTest(unittest.TestCase):
|
||||
self.assertEqual(lines[1:], ['-c', steps.REBUILD_BOOT, str(self.ctx.layout.root), '--root',
|
||||
str(self.ctx.layout.root), '--rebuild', str(self.dest)])
|
||||
|
||||
def test_appimage_rebuild_command_starts_the_appimage(self):
|
||||
# The AppImage's mount, where its Python lives, is gone once the setup exits (#560, #403)
|
||||
appimage = Path(self.tmp.name) / 'TotK-Setup-x86_64.AppImage'
|
||||
appimage.write_bytes(b'')
|
||||
with mock.patch.dict('os.environ', {'APPIMAGE': str(appimage)}):
|
||||
self.assertEqual(self.run_cleanup(), 'ok')
|
||||
lines = (self.dest / 'build-cache' / 'rebuild.txt').read_text(encoding='utf-8').splitlines()
|
||||
self.assertEqual(lines, [str(appimage.resolve()), '--rebuild', str(self.dest)])
|
||||
|
||||
def test_appimage_variable_without_the_file_keeps_the_python_command(self):
|
||||
with mock.patch.dict('os.environ', {'APPIMAGE': str(Path(self.tmp.name) / 'gone.AppImage')}):
|
||||
self.assertEqual(self.run_cleanup(), 'ok')
|
||||
lines = (self.dest / 'build-cache' / 'rebuild.txt').read_text(encoding='utf-8').splitlines()
|
||||
self.assertEqual(lines[1:3], ['-c', steps.REBUILD_BOOT])
|
||||
|
||||
def test_emptied_romfs_fails(self):
|
||||
for p in (self.dest / 'romfs').iterdir():
|
||||
p.unlink()
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
"""recomp/gen.py's lift cache (<mod>/lift_cache.json) on a module it makes itself: a run with unchanged inputs reuses
|
||||
every chunk, a changed patch site or spin wait lifts only the chunk that holds it, a chunk file or cache entry changed
|
||||
behind the cache's back is not taken, a cache that cannot be written only warns, TOTK_LIFT_CACHE=0 and a change to
|
||||
the generator's sources or options lift everything, and the output always equals a --no-cache lift's, folded.txt
|
||||
included. No game files.
|
||||
|
||||
Run with .venv (lz4 and numpy, as recomp/): python tests/lift_cache_test.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
sys.path.insert(0, str(ROOT / 'tools'))
|
||||
from check_content import GENERATED_MARKER # noqa: E402
|
||||
# The jump table module and the NSO writer of tests/exefs_patches_test.py; a change there changes this test's module,
|
||||
# and test_reuse_and_invalidation checks that it still has its jump table (folded.txt 1020 1024).
|
||||
from exefs_patches_test import JT_DATA, JT_RO, JT_TEXT, MOV_W8_100, MOV_W8_200, RET, nso # noqa: E402
|
||||
|
||||
MOV_W8_300 = 0x52806008
|
||||
|
||||
|
||||
def bl(pc, target):
|
||||
return 0x94000000 | (((target - pc) >> 2) & 0x3FFFFFF)
|
||||
|
||||
|
||||
# The jump table module of tests/exefs_patches_test.py (fn at 0x8, its table in .rodata) and two more functions after
|
||||
# it that call each other, so both are function starts: g: bl h; ret, h: mov w8, #0x100; bl g; ret. With --chunk-fns 1
|
||||
# each of fn, g and h is a chunk of its own; h's mov is the patch site.
|
||||
G = len(JT_TEXT)
|
||||
H = G + 8
|
||||
TEXT = JT_TEXT + struct.pack('<5I', bl(G, H), RET, MOV_W8_100, bl(H + 4, G), RET)
|
||||
# fn's `b.hi Ldefault` at 0xc, as a spin wait back-edge to Ldefault (0x48): patches.check accepts it, the lift gates it
|
||||
SPIN = 'spin main 48 c'
|
||||
|
||||
|
||||
def patch_file(*replacements, first=None, spin=False):
|
||||
"""A patch file with set a patching h's mov with each replacement (sets a0, a1, ...); first: a set defined before
|
||||
them, which moves their indices; spin: SPIN too."""
|
||||
lines = [f'set {first} TEST_{first.upper()} 0'] if first else []
|
||||
for i, w in enumerate(replacements):
|
||||
lines += [f'set a{i} TEST_A{i} 0', f'site a{i} main {H:x} {MOV_W8_100:08x} {w:08x}']
|
||||
if spin:
|
||||
lines.append(SPIN)
|
||||
return '\n'.join(lines) + '\n'
|
||||
|
||||
|
||||
class LiftCache(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.d = Path(tempfile.mkdtemp())
|
||||
(self.d / 'exefs').mkdir()
|
||||
(self.d / 'exefs' / 'main').write_bytes(nso(TEXT, JT_RO, JT_DATA))
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.d, ignore_errors=True)
|
||||
|
||||
def gen(self, patches, out='gen', *extra, env=None):
|
||||
p = self.d / 'patches.txt'
|
||||
p.write_text(patches)
|
||||
env = {**os.environ, 'TOTK_LIFT_CACHE': '', **(env or {})}
|
||||
r = subprocess.run([sys.executable, str(ROOT / 'recomp' / 'gen.py'), '--exefs', str(self.d / 'exefs'),
|
||||
'--out', str(self.d / out), '--modules', 'main', '--jobs', '1', '--chunk-fns', '1',
|
||||
'--patches', str(p), *extra], capture_output=True, text=True, env=env)
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
self.stderr = r.stderr
|
||||
return json.loads((self.d / out / 'main' / 'manifest.json').read_text())['stats']
|
||||
|
||||
def files(self, out='gen'):
|
||||
"""{name: bytes} of everything gen.py wrote except the manifest (timings) and the cache."""
|
||||
root = self.d / out
|
||||
return {p.relative_to(root).as_posix(): p.read_bytes() for p in sorted(root.rglob('*'))
|
||||
if p.is_file() and p.name not in ('manifest.json', 'lift_cache.json')}
|
||||
|
||||
def assert_like_fresh(self, patches):
|
||||
"""The output under gen/ equals a --no-cache lift of the same inputs into an empty directory."""
|
||||
stats = self.gen(patches, 'fresh', '--no-cache')
|
||||
self.assertEqual(stats['chunks_reused'], 0)
|
||||
self.assertEqual(self.files(), self.files('fresh'))
|
||||
shutil.rmtree(self.d / 'fresh')
|
||||
|
||||
def chunk(self, n):
|
||||
return self.d / 'gen' / 'main' / f'main_{n:04d}.c'
|
||||
|
||||
def test_reuse_and_invalidation(self):
|
||||
stats = self.gen(patch_file(MOV_W8_200))
|
||||
self.assertEqual((stats['emitted_functions'], stats['chunks_reused']), (3, 0))
|
||||
cache = json.loads((self.d / 'gen' / 'main' / 'lift_cache.json').read_text())
|
||||
self.assertEqual(cache['marker'], GENERATED_MARKER)
|
||||
self.assertEqual(sorted(cache['chunks']), ['main_0000.c', 'main_0001.c', 'main_0002.c'])
|
||||
self.assertIn('rt_patch_on[0]', self.chunk(2).read_text())
|
||||
folded = (self.d / 'gen' / 'main' / 'folded.txt').read_text()
|
||||
self.assertIn('1020 1024', folded)
|
||||
|
||||
# Same inputs: nothing lifted, nothing written, the jump table's bytes still in folded.txt (replayed from the
|
||||
# cache, not read again) and the statistics unchanged.
|
||||
mtimes = [self.chunk(n).stat().st_mtime_ns for n in range(3)]
|
||||
again = self.gen(patch_file(MOV_W8_200))
|
||||
self.assertEqual((again['chunks_reused'], again['chunks_written']), (3, 0))
|
||||
self.assertEqual({k: v for k, v in again.items() if not k.startswith('chunks_')},
|
||||
{k: v for k, v in stats.items() if not k.startswith('chunks_')})
|
||||
self.assertEqual([self.chunk(n).stat().st_mtime_ns for n in range(3)], mtimes)
|
||||
self.assertEqual((self.d / 'gen' / 'main' / 'folded.txt').read_text(), folded)
|
||||
self.assert_like_fresh(patch_file(MOV_W8_200))
|
||||
|
||||
# Another replacement word: only h's chunk is lifted.
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_300))['chunks_reused'], 2)
|
||||
self.assert_like_fresh(patch_file(MOV_W8_300))
|
||||
|
||||
# A set defined before the site's moves its index (rt_patch_on[1]): h's chunk again.
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_300, first='z'))['chunks_reused'], 2)
|
||||
self.assertIn('rt_patch_on[1]', self.chunk(2).read_text())
|
||||
self.assert_like_fresh(patch_file(MOV_W8_300, first='z'))
|
||||
|
||||
# A second alternative at the site.
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_300, MOV_W8_200))['chunks_reused'], 2)
|
||||
self.assert_like_fresh(patch_file(MOV_W8_300, MOV_W8_200))
|
||||
|
||||
def test_changed_file_is_lifted_again(self):
|
||||
"""A chunk whose bytes changed while its size and mtime stayed (a flipped bit) is lifted and rewritten."""
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
p = self.chunk(0)
|
||||
st = p.stat()
|
||||
data = bytearray(p.read_bytes())
|
||||
data[-2] ^= 0x01
|
||||
p.write_bytes(bytes(data))
|
||||
os.utime(p, ns=(st.st_atime_ns, st.st_mtime_ns))
|
||||
stats = self.gen(patch_file(MOV_W8_200))
|
||||
self.assertEqual((stats['chunks_reused'], stats['chunks_written']), (2, 1))
|
||||
self.assert_like_fresh(patch_file(MOV_W8_200))
|
||||
|
||||
def test_deleted_file_and_cache(self):
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
self.chunk(1).unlink()
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200))['chunks_reused'], 2)
|
||||
(self.d / 'gen' / 'main' / 'lift_cache.json').write_text('{not json')
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200))['chunks_reused'], 0)
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200), 'gen', '--no-cache')['chunks_reused'], 0)
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200))['chunks_reused'], 3)
|
||||
self.assert_like_fresh(patch_file(MOV_W8_200))
|
||||
|
||||
def test_spin_wait_is_in_the_key(self):
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200, spin=True))['chunks_reused'], 2)
|
||||
self.assertIn('rt_spin_gate', self.chunk(0).read_text())
|
||||
self.assert_like_fresh(patch_file(MOV_W8_200, spin=True))
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200))['chunks_reused'], 2)
|
||||
self.assertNotIn('rt_spin_gate', self.chunk(0).read_text())
|
||||
|
||||
def test_changed_cache_entry_is_not_taken(self):
|
||||
"""A changed result in lift_cache.json (a folded range, here) fails the entry's digest: the chunk is lifted
|
||||
and folded.txt keeps the jump table's real bytes."""
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
p = self.d / 'gen' / 'main' / 'lift_cache.json'
|
||||
data = json.loads(p.read_text())
|
||||
self.assertEqual(data['chunks']['main_0000.c']['folded'], [[0x1020, 0x1024]])
|
||||
data['chunks']['main_0000.c']['folded'] = [[0x1060, 0x1064]]
|
||||
p.write_text(json.dumps(data))
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200))['chunks_reused'], 2)
|
||||
self.assertIn('1020 1024', (self.d / 'gen' / 'main' / 'folded.txt').read_text())
|
||||
self.assert_like_fresh(patch_file(MOV_W8_200))
|
||||
|
||||
def test_unwritable_cache_only_warns(self):
|
||||
"""A cache that cannot be replaced leaves the lift successful, its manifest written; the old cache's entry for
|
||||
the chunk the run rewrote no longer matches it."""
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
p = self.d / 'gen' / 'main' / 'lift_cache.json'
|
||||
os.chmod(p, stat.S_IREAD)
|
||||
try:
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_300))['chunks_reused'], 2)
|
||||
self.assertIn('lift_cache.json not written', self.stderr)
|
||||
finally:
|
||||
os.chmod(p, stat.S_IREAD | stat.S_IWRITE)
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_300))['chunks_reused'], 2)
|
||||
self.assert_like_fresh(patch_file(MOV_W8_300))
|
||||
|
||||
def test_environment_switch(self):
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200), env={'TOTK_LIFT_CACHE': '0'})['chunks_reused'], 0)
|
||||
self.assertIn('lifting every chunk (TOTK_LIFT_CACHE=0)', self.stderr)
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200))['chunks_reused'], 3)
|
||||
|
||||
def test_emitter_options_are_in_the_key(self):
|
||||
self.gen(patch_file(MOV_W8_200))
|
||||
self.assertEqual(self.gen(patch_file(MOV_W8_200), 'gen', '--reg-locals', '0')['chunks_reused'], 0)
|
||||
|
||||
def test_generator_sources_are_in_the_key(self):
|
||||
"""A copy of the generator lifts everything again after a comment in its lifter.py changes, and nothing
|
||||
after a patch table beside it (recomp/patches-*.txt) changes."""
|
||||
copy = self.d / 'repo'
|
||||
shutil.copytree(ROOT / 'recomp', copy / 'recomp', ignore=shutil.ignore_patterns('__pycache__'))
|
||||
(copy / 'tools').mkdir()
|
||||
for name in ('check_content.py', 'hostmem.py'):
|
||||
shutil.copy2(ROOT / 'tools' / name, copy / 'tools' / name)
|
||||
|
||||
def run():
|
||||
r = subprocess.run([sys.executable, str(copy / 'recomp' / 'gen.py'), '--exefs', str(self.d / 'exefs'),
|
||||
'--out', str(self.d / 'gen'), '--modules', 'main', '--jobs', '1', '--chunk-fns', '1'],
|
||||
capture_output=True, text=True, env={**os.environ, 'TOTK_LIFT_CACHE': ''})
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
return json.loads((self.d / 'gen' / 'main' / 'manifest.json').read_text())['stats']['chunks_reused']
|
||||
|
||||
self.assertEqual(run(), 0)
|
||||
self.assertEqual(run(), 3)
|
||||
table = next((copy / 'recomp').glob('patches-*.txt'))
|
||||
table.write_text(table.read_text() + '# another line\n')
|
||||
self.assertEqual(run(), 3)
|
||||
lifter = copy / 'recomp' / 'lifter.py'
|
||||
lifter.write_text(lifter.read_text() + '# another comment\n')
|
||||
self.assertEqual(run(), 0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+6
-2
@@ -57,9 +57,13 @@
|
||||
# game waits once for the GPU to finish the frame it presented last; with 1 that wait covers the frame before it
|
||||
# instead. On Great Sky Island (Linux, RX 9070 XT) with shadow_defer = 1 this took about 73 to 100 fps. Costs one more
|
||||
# frame of input delay, and frame times vary more from frame to frame than with 0. GPU results the game reads back after that wait, for example occlusion queries, can be a frame old.
|
||||
# Works on 1.2.1 and 1.4.3; 1.1.0 ignores it. Needs shadow storage, which is on by default on Linux and Wine and needs
|
||||
# the environment variable TOTK_SHADOW=1 on native Windows; without it the setting is ignored and the log says so.
|
||||
# Works on 1.2.1 and 1.4.3; 1.1.0 ignores it. Needs shadow storage (shadow below), which is on by default on Linux and
|
||||
# Wine and needs shadow = 1 on native Windows; without it the setting is ignored and the log says so.
|
||||
# sync_ahead = 0
|
||||
# Shadow storage, 0 or 1: each CPU-visible memory pool of the game gets a host-visible copy that the GPU reads, in
|
||||
# place of importing the game's memory. Default: on under Linux and Wine, off on native Windows. sync_ahead needs it.
|
||||
# Costs about 0.2 ms a frame and doubles the memory of those pools (about 1.1 GB; docs/building.md). Read at start.
|
||||
# shadow = 0
|
||||
# With sync_ahead = 1: when the game changes memory the GPU still reads, queue the copy into the GPU's shadow of that
|
||||
# memory for the GPU to run in order, 0 or 1. Default: on. With 0 the replay thread waits for the GPU to finish
|
||||
# reading instead, which costs part of the gain.
|
||||
|
||||
Reference in new issue
Block a user