diff --git a/docs/development/american-nightmare-pixel-debug.md b/docs/development/american-nightmare-pixel-debug.md new file mode 100644 index 000000000..ef5992dd0 --- /dev/null +++ b/docs/development/american-nightmare-pixel-debug.md @@ -0,0 +1,197 @@ +# American Nightmare on Pixel: rendering workaround and remaining limits + +Local engineering results from 2026-10-11/12. Steam game 202750 was tested on a +Pixel 11 Pro Fold using the Pixel Lab runtime. These observations do not establish +compatibility across the Pixel family or a fix for the system resets. + +## Rendering result + +The bright dotted triangle edges over textured rocks, boards and the player are +not an application wireframe setting: traced input and driver pipelines all used +`VK_POLYGON_MODE_FILL`, with no dynamic polygon mode. The original affected path +used 2x MSAA, RGBA16F color and D24S8 depth. The game's minimum UI AA setting and +`d3d11.disableMsaa` did not disable its D3D9 multisampling. + +A real single-sample D3D9 comparison removed the visible bright triangle edges. +Trial 16 verified 114 input and 114 driver pipelines, all single-sample. Trial 17 +retained correct meshes/textures through movement and turning, with 119 pipelines +in each phase, all single-sample. This isolates a useful workaround to the MSAA +path; it does not identify the precise faulty operation or prove a vendor bug. +The override also changes effective surface descriptors, so the application may +select different AA-aware shader/rendering paths. + +`tools/dxvk/patches/0004-d3d9-force-single-sample.patch` adds an explicit per-game +`d3d9.forceSingleSample` option, default **False**. After validating the original +sample request, D3D9 surface normalization selects `D3DMULTISAMPLE_NONE` and quality +0. GetDesc, image creation, resolves and sample-dependent state use this effective +description. This intentionally disables MSAA, per-sample masks and +alpha-to-coverage. It is a compatibility override, not transparent MSAA emulation +or a universal PowerVR default. Existing low32, BC-format, present synchronization +and tiler workarounds must remain intact. + +Swapchain backbuffers have a separate creation path. With the override enabled, +that path now validates a copy of the original requested descriptor and transfers +the effective sample type/quality before allocation, so automatic depth surfaces +and backbuffers agree. Validation uses the application's lockable-backbuffer flag; +the internal lockability required for GDI fallback is retained on the allocated +backbuffer. With the override disabled the original backbuffer path is unchanged. +This correction addresses official PR602 review 5486295088. Its CPU harness +compiles the actual DXVK sample decoder, surface normalizer and backbuffer method, +plus the automatic-depth descriptor and framebuffer depth-binding excerpts. +Fifteen cases cover on/off, NONMASKABLE quality, matching depth, no-depth operation, +unsupported/invalid/lockable requests, allocation cleanup and recreation. Backend +images and adapter capabilities are mocked; these are not GPU or gameplay tests. + +## Independent HUD correction + +The diagnostic frametimes HUD bound an R8_UINT storage texel buffer to a shader +whose SPIR-V image format was Unknown. This required formatless buffer writes +which the device did not report for that format, producing VUID 07029. +`0003-hud-explicit-r8-texel-format.patch` declares `r8ui` to match the actual host +view. No capabilities are fabricated. The build now compiles and validates the +actual shader, and checks binding 3 for DimBuffer/storage/R8ui. The original shader +fails that contract; the patched shader passes. Real-game trial 15 used the new +DLLs, removed 07029, and retained the original triangle edges. This HUD bug is +separate from the rendering workaround, and is not a proven reset cause. + +VUID 06479 for color-format comparison descriptors remains under investigation. +The D3D9 shader uses a conditional comparison path; static validation output alone +does not prove that the offending branch executes. No warning suppression or +false format-feature claims are proposed. Trial 14's LOG+FAIL with a duplicate +limit blocked only the first message of each kind, not every invalid API call; +subsequent trials used logging only. Do not describe these runs as fully blocked +validation or GPU-assisted validation. + +## Responsiveness, performance and stability + +Paired screenshots prove movement and camera changes, rather than relying on an +input receipt. Trial 17 completed eight samples without a black viewport. Its +cold sample was 3.2 FPS; later recognized samples were 19.1-29.9 FPS, with roughly +2.54 CPU-core equivalents. Camera/workload, capture overhead and thermals were +not controlled; this is not a reliable A/B benchmark. + +The planned eight-minute trial 17 stopped at a focus/session guard after the last +confirmed sample at 315.6 seconds. Steam owned focus; the target game was still +available for a single client quit. Its temporary settings were successfully +restored, and the phone boot counter remained 12. The focus change's cause is +unknown. The run is **incomplete**, not an eight-minute pass or a demonstrated +game crash. Controls, focus ownership and sustained stability still need broader +coverage beyond these short tests. + +### Final production configuration and user confirmation + +The clean production configuration was persisted only for game 202750 through +the application's per-game environment UI. It selects the tested DXVK component +and sets `d3d9.forceSingleSample = True`, retaining the existing low32, BC, +presentation-sync and `dxvk.tilerMode = False` workarounds. Shared configuration +and other games were unchanged. Shader audit layers, validation and the +frametimes HUD are not enabled in this production configuration. + +A final automated production trial loaded the existing saved game, verified +turning/movement with five paired non-black screenshots, and observed 115 seconds +without the bright triangle edges or another phone reset. It then quit normally; +the saved per-game environment remained unchanged. This is bounded observation, +not a long-duration stability or controlled performance benchmark. + +On 2026-10-11 (America/Toronto), the device owner separately confirmed that this +stage worked and authorized publication. No duration, frame rate or full-game +completion was supplied with that confirmation. It is additional user validation +of the stage, not a claim that all Pixel devices or the GNOC resets are fixed. + +## Native guards, resets and diagnostic controls + +The Pixel base is e589e3ae21959c9c74c490d917cebf256acc8ac8. PR561's 9e6fe314 and +3c363910 guards were integrated in an isolated worktree; 26 selected production-C +regressions and a full APK/runtime verification passed. Pixel AHB support remains +present. The existing Lab app was updated without uninstalling or resetting its +Steam account. This APK preceded a second system reset, so the native guards are +not a verified reset fix. + +Two resets changed the boot counter 10 -> 11 -> 12. After personal unlock, narrow +kernel/firmware projections confirmed GNOC/access-control details for the first, +and `reset,warm,gnoc_error` with a fresh GNOC numeric record for the second. The +specific faulty component/operation and any relation to rendering are unknown. +`tools/pixel_reset_diagnostics.py` exports only fixed fault categories/numeric +fields, lock state, boot counter and entry times; never arbitrary log text or +account data. Boot-property and kernel last-boot reasons use the same exact +allowlist of canonical categories and the observed `reset,warm,gnoc_error` value; +unknown reasons and custom suffixes produce `unknown`, never their original text. +Primary-user states are likewise limited to recognized lifecycle categories. +Output files are created exclusively with mode `0600` before any bytes are +written (POSIX), and existing files/symlinks are refused. Host regressions cover +these boundaries as well as freshness, unlock gating and narrow output. +Dropbox's offset-free local timestamp is not assigned a guessed instant during +a DST transition: repeated hours report `ambiguous_device_local_time`, skipped +hours report `nonexistent_device_local_time`. Neither is attributed to the trial +or used to project kernel-body categories until its time can be established. + +Private opt-in diagnostics captured maintenance5 inline modules, which the older +shim path missed. Materialization passed nine actual-C ASan/UBSan cases; benign +pipeline-chain planning passed twelve. Real-game NoContraction, scaled vertex +conversion and force-clip rewrites were actually applied and did not remove the +edges. None is a production default. Sixty-two valid bounded, owned offscreen GPU +cases passed exact pixel readback, fences and validation, covering interpolation, +seams, perspective meshes, interface widening, scaled fetch, HDR/depth prepasses, +real clipping and nonconstant depth. Invalid initial fixtures are excluded. These +passing controls do not exonerate every driver/rendering path. + +## Reproducibility and recovery + +The runtime artifact supplies both architectures, exact DLL hashes, public patched +DXVK sources, all four patches, wrapper and build recipe. It contains no game +assets, bytecode, saves, account files, screenshots or raw device logs. Source +patches apply cleanly to pinned DXVK revision 25ca63f17f34bdc05a39873ee63907d3cbbfa030. +All five runtime DLL targets were built for both architectures. + +Stop further load tests after a full reset. Wait for personal unlock and inspect a +fresh narrow projection before another launch. Temporary profile restoration checks +baseline/current hashes and unchanged shared/other-game settings; it never rolls +back saved game progress. Persistent selection must affect only 202750 and leave +private shader audit layers, validation settings and frametimes HUD disabled. +Routine Heartbeat must not launch games or automatically resume this engineering. + +## Submission scope + +These opt-in runtime tools are submitted directly against official DroidDeck +`main`. The standalone DXVK build recipe, source pin, BC-format prerequisite and +presentation-semaphore patch/check are imported verbatim from the PowerVR proposal + at +`d0f7264ea8e402d30c868779198dc2fd74e7431c`. That proposal's attribution and existing +source/license provenance are retained. The semaphore correction originated in +. No Android app changes, managed pack +or the experimental fork's development history are imported with those recipes. + +The source tools can be reviewed and built independently. Automatic PowerVR +integration and delivery in DroidDeck require the separately reviewed #570 +foundation and a new managed-runtime release. This submission does not change +managed-pack manifests/checksums or globally enable any override. The earlier +fork-only stage is superseded by the +official submission; its review history is preserved. + +For standalone verification, run +`python3 -m unittest discover -s tools/tests -p test_pixel_reset_diagnostics.py` +on the host. On native 64-bit Linux/glibc, +`tools/hybris/low32/build.sh /absolute/output-dir` builds and runs the CPU integrity +and independent mmap/mmap64 preload-chain regressions (both library orders in +off, trace and low32 modes). Optional Vulkan headers produce `gpu-copy-native` +for the selected native host architecture, not a hardcoded ARM64 filename. +`python3 -m unittest discover -s tools/tests -p test_dxvk_build.py` verifies output +isolation with a synthetic toolchain, without fetching or compiling DXVK. +The DXVK component recipe is `tools/dxvk/build.sh /absolute/output-dir`; +it requires the pinned DXVK build prerequisites, including Meson/Ninja, MinGW +cross compilers, glslangValidator and SPIR-V Tools. It retains both architectures, +source/license provenance and downstream patches, without editing an installed +runtime or APK. Its output directory must be new or empty (including hidden +entries). Prior or partially built outputs are refused before fetching sources, +left untouched and never included in a new DLL manifest. Use a fresh directory +after a failed build. +The recipe also requires a native C++17 compiler for +`python3 tools/dxvk/check_d3d9_single_sample.py /path/to/patched-dxvk` before DLL +compilation. Add `--sanitize` on supported hosts to run its ASan/UBSan controls. + +The compositor ownership safeguards are independently proposed in +. Those native changes were present +in the integrated Pixel Lab APK used for the game experiments, but are not part +of this runtime PR's diff. The original BC emulation is a prerequisite supplied by +the separately proposed PowerVR runtime. This submission supplies source and reproducibility tools, +not an APK, proprietary driver, game data or private diagnostic archive. diff --git a/tools/dxvk/build.sh b/tools/dxvk/build.sh new file mode 100755 index 000000000..d403aa050 --- /dev/null +++ b/tools/dxvk/build.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# CI component build; this does not change the APK or an installed Proton depot. +set -euo pipefail +root=$(cd "$(dirname "$0")/../.." && pwd) +. "$root/tools/dxvk/source.env" +mkdir -p "${1:?output directory required}" +out=$(cd "$1" && pwd) +# Refuse to mix a pinned build with prior DLLs or an incomplete previous build. +# Include dotfiles in the check and leave all existing contents untouched. +shopt -s nullglob dotglob +existing=("$out"/*) +if ((${#existing[@]})); then + printf '%s\n' 'output directory must be empty; use a fresh build directory' >&2 + exit 1 +fi +shopt -u nullglob dotglob +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT +git init -q "$work/source" +git -C "$work/source" fetch --depth=1 https://github.com/doitsujin/dxvk.git "$DXVK_REVISION" +git -C "$work/source" checkout -q --detach FETCH_HEAD +test "$(git -C "$work/source" rev-parse HEAD)" = "$DXVK_REVISION" +git -C "$work/source" submodule update --init --recursive --depth=1 +for patch in "$root"/tools/dxvk/patches/*.patch; do + git -C "$work/source" apply --check "$patch" + git -C "$work/source" apply "$patch" +done +python3 "$root/tools/dxvk/check_present_sync.py" "$work/source/src/dxvk/dxvk_presenter.cpp" +python3 "$root/tools/dxvk/check_hud_storage_format.py" "$work/source/src/dxvk/hud/shaders/hud_frame_time_eval.comp" +python3 "$root/tools/dxvk/check_d3d9_single_sample.py" "$work/source" +for arch in 32 64; do + meson setup "$work/build$arch" "$work/source" --cross-file "$work/source/build-win$arch.txt" \ + --buildtype release --prefix "$out" --bindir "x$arch" --libdir "x$arch" -Db_ndebug=if-release --strip + meson compile -C "$work/build$arch" -j 4 + meson install -C "$work/build$arch" +done +cp "$work/source/LICENSE" "$out/LICENSE" +printf '%s\n' "$DXVK_REVISION" > "$out/source-commit" +git -C "$work/source" submodule status --recursive > "$out/submodules.txt" +git -C "$work/source" diff > "$out/downstream.patch" +tar --exclude-vcs --zstd -cf "$out/source.tar.zst" -C "$work/source" . +(cd "$out" && find x32 x64 -name '*.dll' -type f -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS) diff --git a/tools/dxvk/check_d3d9_single_sample.py b/tools/dxvk/check_d3d9_single_sample.py new file mode 100644 index 000000000..8f73a3f3a --- /dev/null +++ b/tools/dxvk/check_d3d9_single_sample.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""Compile pinned DXVK's actual descriptor paths against a controlled backend. + +This checks CPU control flow, not GPU rendering. No Wine/game/device is started. +""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import subprocess +import sys +import tempfile + +CASES = ('forced_msaa_depth', 'default_msaa_depth', 'forced_nonmaskable', + 'default_nonmaskable', 'forced_already_single', 'forced_no_depth', + 'forced_unsupported', 'forced_invalid_type', 'forced_lockable_msaa', + 'forced_lockable_single', 'forced_allocation_failure', + 'default_allocation_failure', 'forced_recreate', 'ordinary_surface', + 'ordinary_invalid_lockable') + + +def function(text, signature): + start = text.index(signature) + opening = text.index('{', start) + # Ignore braces in comments and strings while preserving source positions. + stripped = re.sub(r'//[^\n]*|/\*.*?\*/|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'', + lambda m: ' ' * len(m.group()), text, flags=re.S) + depth = 0 + for index in range(opening, len(stripped)): + depth += (stripped[index] == '{') - (stripped[index] == '}') + if depth == 0: + return text[start:index + 1] + raise ValueError('Unterminated source function: ' + signature) + + +def production(source): + base = source / 'src/d3d9' + common = (base / 'd3d9_common_texture.cpp').read_text() + header = (base / 'd3d9_common_texture.h').read_text() + swap = (base / 'd3d9_swapchain.cpp').read_text() + device = (base / 'd3d9_device.cpp').read_text() + util = (base / 'd3d9_util.cpp').read_text() + pieces = { + 'decode': function(util, 'HRESULT DecodeMultiSampleType('), + 'normalize': function(common, 'HRESULT D3D9CommonTexture::NormalizeTextureProperties('), + 'backbuffers': function(swap, 'HRESULT D3D9SwapChainEx::CreateBackBuffers('), + 'uses_image': function(header, 'static bool TextureUsesImage('), + } + reset = function(device, 'HRESULT D3D9DeviceEx::ResetSwapChain(') + begin = reset.index('D3D9_COMMON_TEXTURE_DESC desc;', reset.index('if (pPresentationParameters->EnableAutoDepthStencil)')) + end = reset.index('m_autoDepthStencil =', begin) + pieces['auto_depth'] = reset[begin:end].rstrip() + framebuffer = function(device, 'void D3D9DeviceEx::BindFramebuffer(') + begin = framebuffer.index('const bool mismatch =') + end = framebuffer.index(';', framebuffer.index('const bool bindDS =', begin)) + 1 + pieces['depth_binding'] = framebuffer[begin:end] + return pieces + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('source', type=Path, help='Pinned DXVK tree with downstream patches applied') + parser.add_argument('--cxx', default=os.environ.get('CXX', 'c++')) + parser.add_argument('--sanitize', action='store_true', help='Enable ASan/UBSan on the CPU harness') + args = parser.parse_args() + pieces = production(args.source) + harness = (Path(__file__).parent / 'tests/d3d9_single_sample.cpp').read_text() + harness = harness.replace('/* PRODUCTION_USES_IMAGE */', pieces['uses_image']) + with tempfile.TemporaryDirectory(prefix='dxvk-d3d9-') as directory: + out = Path(directory) + (out / 'harness.cpp').write_text(harness) + include = '\n\n'.join(pieces[key] for key in ('decode', 'normalize', 'backbuffers')) + include += '\nHRESULT auto_depth(D3D9DeviceEx* device, D3DPRESENT_PARAMETERS* pPresentationParameters, D3D9_COMMON_TEXTURE_DESC* output) {\n' + include += 'auto* self = device;\n' + pieces['auto_depth'].replace('(this,', '(self,') + include += '\n*output = desc; return D3D_OK;\n}\n' + include += '\nbool depth_binds(const ImageInfo& color, const ImageInfo& dsInfo) {\n' + include += 'auto sampleCount = color.sampleCount; auto renderArea = color.extent;\n' + include += 'const bool noRT0Bound = false, ignoreRT0 = false;\n' + include += pieces['depth_binding'] + '\nreturn bindDS;\n}\n' + (out / 'd3d9_production.inc').write_text(include) + command = [args.cxx, '-std=c++17', '-Wall', '-Wextra', '-Werror', + '-Wno-unused-parameter', str(out / 'harness.cpp'), '-o', str(out / 'check')] + # The original CreateBackBuffers does not use Flags; that known baseline + # warning must not turn a negative control into a compiler-fixture error. + if args.sanitize: + command[1:1] = ['-fsanitize=address,undefined', '-fno-omit-frame-pointer'] + compilation = subprocess.run(command, capture_output=True, text=True) + if compilation.returncode: + raise SystemExit(compilation.stderr) + results = [] + for case in CASES: + process = subprocess.run([str(out / 'check'), case], capture_output=True, + text=True, timeout=10, cwd=out, + env={'ASAN_OPTIONS': 'detect_leaks=' + + ('0' if sys.platform == 'darwin' else '1')}) + results.append({'case': case, 'passed': process.returncode == 0, + 'exit': process.returncode, 'detail': process.stderr.strip()[-800:]}) + result = {'cases': results, 'passed': sum(row['passed'] for row in results), + 'failed': sum(not row['passed'] for row in results), + 'production_sha256': {key: hashlib.sha256(value.encode()).hexdigest() + for key, value in pieces.items()}, + 'backend_and_adapter_mocked': True, 'GPU_test': False, + 'sanitizers': args.sanitize, + 'leak_detection': args.sanitize and sys.platform != 'darwin'} + print(json.dumps(result, indent=2)) + return bool(result['failed']) + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tools/dxvk/check_hud_storage_format.py b/tools/dxvk/check_hud_storage_format.py new file mode 100644 index 000000000..f376cad6b --- /dev/null +++ b/tools/dxvk/check_hud_storage_format.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""Compile the actual HUD shader and verify its binding-3 R8_UINT contract. + +The frame-time HUD binds an R8_UINT texel-buffer view. An Unknown SPIR-V image +format requires STORAGE_WRITE_WITHOUT_FORMAT, which PowerVR does not report +for that buffer format. Static spirv-val alone accepts both declarations; +this check validates the shader against the known host-side binding contract. +""" +import argparse +from pathlib import Path +import shutil +import struct +import subprocess +import tempfile + + +def check(shader): + compiler = shutil.which('glslangValidator') + validator = shutil.which('spirv-val') + if not compiler or not validator: + raise RuntimeError('glslangValidator and spirv-val required') + with tempfile.TemporaryDirectory(prefix='dxvk-hud-contract-') as directory: + output = Path(directory) / 'hud.spv' + for command in ([compiler, '-V', '--target-env', 'vulkan1.3', str(shader), '-o', str(output)], + [validator, '--target-env', 'vulkan1.3', str(output)]): + process = subprocess.run(command, capture_output=True, text=True, timeout=30) + if process.returncode: + raise RuntimeError('HUD shader compilation/SPIR-V validation failed') + raw = output.read_bytes() + if len(raw) < 20 or len(raw) % 4: + raise RuntimeError('Malformed SPIR-V') + words = struct.unpack('<' + str(len(raw) // 4) + 'I', raw) + if words[0] != 0x07230203: + raise RuntimeError('Unexpected SPIR-V magic') + bindings, sets, variables, pointers, images = {}, {}, {}, {}, {} + offset = 5 + while offset < len(words): + size, opcode = words[offset] >> 16, words[offset] & 0xffff + if not size or size > len(words) - offset: + raise RuntimeError('Malformed SPIR-V instruction') + instruction = words[offset:offset + size] + if opcode == 71 and size == 4: # OpDecorate + if instruction[2] == 33: + bindings[instruction[1]] = instruction[3] + elif instruction[2] == 34: + sets[instruction[1]] = instruction[3] + elif opcode == 59 and size >= 4: # OpVariable + variables[instruction[2]] = instruction[1] + elif opcode == 32 and size == 4: # OpTypePointer + pointers[instruction[1]] = instruction[3] + elif opcode == 25 and size >= 9: # OpTypeImage + images[instruction[1]] = (instruction[3], instruction[7], instruction[8]) + offset += size + targets = [images.get(pointers.get(variables.get(identifier))) + for identifier, binding in bindings.items() + if binding == 3 and sets.get(identifier) == 0] + if targets != [(5, 2, 39)]: # DimBuffer, storage image, ImageFormatR8ui + raise RuntimeError('HUD binding 3 must compile to a storage texel buffer with ImageFormatR8ui') + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('shader', type=Path) + args = parser.parse_args() + try: + check(args.shader) + except RuntimeError as error: + parser.exit(1, str(error) + '\n') + print('HUD binding 3: SPIR-V valid, explicit R8ui storage texel-buffer format') + + +if __name__ == '__main__': + main() diff --git a/tools/dxvk/check_present_sync.py b/tools/dxvk/check_present_sync.py new file mode 100644 index 000000000..ad47a8ade --- /dev/null +++ b/tools/dxvk/check_present_sync.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +"""Replay asynchronous WSI ownership using the pinned presenter's actual selections. + +This checks semaphore selection, not the full Vulkan presenter. Real Vulkan +validation and device testing remain separate gates. Requires a native C++ compiler. +""" +import argparse +import os +from pathlib import Path +import re +import subprocess +import tempfile + + +def selection_code(source): + match = re.search( + r"// Return relevant Vulkan objects for the acquired image\s*" + r"(?P.*?)\s*m_presentPending = true;", source, re.S) + wait = re.search(r"info\.pWaitSemaphores\s*=.*?;", source, re.S) + if not match or not wait: + raise ValueError("unsupported pinned presenter layout") + return match["acquire"], wait[0] + + +HARNESS = r''' +#include +#include +#include +#include +using VkSemaphore = uint64_t; +struct Sync { VkSemaphore acquire, present; }; +struct Info { const VkSemaphore* pWaitSemaphores; }; + +struct Presenter { + std::vector m_semaphores; + std::vector m_images = {0, 1, 2, 3}; + unsigned m_frameIndex = 0, m_imageIndex = 0; + bool m_hasSwapchainMaintenance1; + explicit Presenter(bool maintenance): m_hasSwapchainMaintenance1(maintenance) { + for (unsigned i = 0; i < (maintenance ? 4u : 8u); ++i) + m_semaphores.push_back({100 + i, 1 + i}); + } + Sync acquire() { + Sync sync{}; unsigned image; + ACQUIRE_SELECTION + (void)image; + return sync; + } + VkSemaphore present() { + auto& currSync = m_semaphores.at(m_frameIndex); + Info info{}; + PRESENT_SELECTION + return *info.pWaitSemaphores; + } +}; + +int replay(bool maintenance) { + Presenter p(maintenance); + std::array outstandingByImage{}; + std::array outstandingByFence{}; + // Deliberately delay reacquisition of image 2 beyond a full frame-ring cycle. + const std::array imageOrder = { + 2,0,3,0,1,3,0,1,1,2,3,1,0,3,2,2,0,1,3,0,2 + }; + for (unsigned iteration = 0; iteration < 200; ++iteration) { + p.m_imageIndex = imageOrder[iteration % imageOrder.size()]; + if (maintenance) { + // The existing fence path waits this frame's previous present fence. + const auto completed = outstandingByFence.at(p.m_frameIndex); + for (auto& pending : outstandingByImage) + if (pending == completed) pending = 0; + } + // Acquisition and its GPU wait complete the last present of THIS image. + outstandingByImage.at(p.m_imageIndex) = 0; + const auto signal = p.acquire().present; + const auto wait = p.present(); + if (signal != wait) { + std::fprintf(stderr, "FAIL: render signal differs from present wait\n"); + return 2; + } + for (auto pending : outstandingByImage) { + if (pending == signal) { + std::fprintf(stderr, "FAIL: semaphore reused before presentation completion at frame %u\n", iteration); + return 2; + } + } + outstandingByImage.at(p.m_imageIndex) = signal; + if (maintenance) outstandingByFence.at(p.m_frameIndex) = signal; + p.m_frameIndex = (p.m_frameIndex + 1) % p.m_semaphores.size(); + } + std::printf("PASS: %s, 200 out-of-order acquisitions\n", maintenance ? "present-fence path" : "no-present-fence path"); + return 0; +} +int main() { + if (int status = replay(false)) return status; + return replay(true); +} +''' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("source", type=Path, help="pinned dxvk_presenter.cpp, after patching") + args = parser.parse_args() + acquire, present = selection_code(args.source.read_text()) + code = HARNESS.replace("ACQUIRE_SELECTION", acquire).replace("PRESENT_SELECTION", present) + with tempfile.TemporaryDirectory(prefix="dxvk-present-sync-") as directory: + cpp = Path(directory) / "replay.cpp" + exe = Path(directory) / "replay" + cpp.write_text(code) + subprocess.run([os.environ.get("HOST_CXX", "c++"), "-std=c++17", "-Wall", "-Wextra", str(cpp), "-o", str(exe)], check=True) + return subprocess.run([str(exe)]).returncode + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/dxvk/patches/0001-check-direct3d-bc-formats.patch b/tools/dxvk/patches/0001-check-direct3d-bc-formats.patch new file mode 100644 index 000000000..db327f038 --- /dev/null +++ b/tools/dxvk/patches/0001-check-direct3d-bc-formats.patch @@ -0,0 +1,53 @@ +diff --git a/src/dxvk/dxvk_device_info.cpp b/src/dxvk/dxvk_device_info.cpp +index db09fba..8e9a76e 100644 +--- a/src/dxvk/dxvk_device_info.cpp ++++ b/src/dxvk/dxvk_device_info.cpp +@@ -118,6 +118,17 @@ namespace dxvk { + initQueueProperties(instance, adapter, deviceInfo); + initMemoryProperties(instance, adapter); + ++ m_hasBcFormats = true; ++ for (uint32_t value = VK_FORMAT_BC1_RGBA_UNORM_BLOCK; value <= VK_FORMAT_BC7_SRGB_BLOCK; value++) { ++ VkFormatProperties properties = { }; ++ vk->vkGetPhysicalDeviceFormatProperties(adapter, VkFormat(value), &properties); ++ constexpr VkFormatFeatureFlags required = VK_FORMAT_FEATURE_SAMPLED_IMAGE_BIT ++ | VK_FORMAT_FEATURE_SAMPLED_IMAGE_FILTER_LINEAR_BIT ++ | VK_FORMAT_FEATURE_TRANSFER_SRC_BIT | VK_FORMAT_FEATURE_TRANSFER_DST_BIT; ++ if ((properties.optimalTilingFeatures & required) != required) ++ m_hasBcFormats = false; ++ } ++ + disableUnusedFeatures(instance, safeMode); + + enableFeaturesAndExtensions(); +@@ -785,6 +796,9 @@ namespace dxvk { + } + } + ++ if (!m_hasBcFormats) ++ return std::string("Device does not support the BC formats required by Direct3D"); ++ + if (!m_featuresEnabled.extDescriptorHeap.descriptorHeap && m_properties.core.properties.limits.maxPushConstantsSize < MaxTotalPushDataSize) + return str::format("Device does not support ", MaxTotalPushDataSize, " of push data"); + +@@ -886,7 +900,7 @@ namespace dxvk { + ENABLE_FEATURE(core.features, shaderResourceResidency, false), + ENABLE_FEATURE(core.features, shaderResourceMinLod, false), + ENABLE_FEATURE(core.features, tessellationShader, false), +- ENABLE_FEATURE(core.features, textureCompressionBC, true), ++ ENABLE_FEATURE(core.features, textureCompressionBC, false), + ENABLE_FEATURE(core.features, variableMultisampleRate, false), + ENABLE_FEATURE(core.features, vertexPipelineStoresAndAtomics, false), + ENABLE_FEATURE(core.features, wideLines, false), +diff --git a/src/dxvk/dxvk_device_info.h b/src/dxvk/dxvk_device_info.h +index 0a55f90..8bbc770 100644 +--- a/src/dxvk/dxvk_device_info.h ++++ b/src/dxvk/dxvk_device_info.h +@@ -394,6 +394,7 @@ namespace dxvk { + + DxvkDeviceQueueMapping m_queueMapping = { }; + ++ bool m_hasBcFormats = false; + bool m_hasMeshShader = false; + bool m_hasFmask = false; + diff --git a/tools/dxvk/patches/0002-present-semaphore-per-swapchain-image.patch b/tools/dxvk/patches/0002-present-semaphore-per-swapchain-image.patch new file mode 100644 index 000000000..bd22bdc52 --- /dev/null +++ b/tools/dxvk/patches/0002-present-semaphore-per-swapchain-image.patch @@ -0,0 +1,26 @@ +diff --git a/src/dxvk/dxvk_presenter.cpp b/src/dxvk/dxvk_presenter.cpp +index 643f7c5..4dc8c4d 100644 +--- a/src/dxvk/dxvk_presenter.cpp ++++ b/src/dxvk/dxvk_presenter.cpp +@@ -161,6 +161,10 @@ namespace dxvk { + + // Return relevant Vulkan objects for the acquired image + sync = m_semaphores.at(m_frameIndex); ++ // Without a present fence, reacquiring this image is the only guarantee ++ // that its previous presentation has consumed the wait semaphore. ++ if (!m_hasSwapchainMaintenance1) ++ sync.present = m_semaphores.at(m_imageIndex).present; + image = m_images.at(m_imageIndex); + + m_presentPending = true; +@@ -239,7 +243,9 @@ namespace dxvk { + + VkPresentInfoKHR info = { VK_STRUCTURE_TYPE_PRESENT_INFO_KHR }; + info.waitSemaphoreCount = 1; +- info.pWaitSemaphores = &currSync.present; ++ info.pWaitSemaphores = m_hasSwapchainMaintenance1 ++ ? &currSync.present ++ : &m_semaphores.at(m_imageIndex).present; + info.swapchainCount = 1; + info.pSwapchains = &m_swapchain; + info.pImageIndices = &m_imageIndex; diff --git a/tools/dxvk/patches/0003-hud-explicit-r8-texel-format.patch b/tools/dxvk/patches/0003-hud-explicit-r8-texel-format.patch new file mode 100644 index 000000000..85ed529e8 --- /dev/null +++ b/tools/dxvk/patches/0003-hud-explicit-r8-texel-format.patch @@ -0,0 +1,12 @@ +--- a/src/dxvk/hud/shaders/hud_frame_time_eval.comp ++++ b/src/dxvk/hud/shaders/hud_frame_time_eval.comp +@@ -41,7 +41,8 @@ + draw_info_t draw_infos[]; + }; + +-layout(binding = 3) ++// The bound text view is VK_FORMAT_R8_UINT; do not require formatless writes. ++layout(binding = 3, r8ui) + uniform writeonly uimageBuffer text_buffer; + + layout(push_constant) diff --git a/tools/dxvk/patches/0004-d3d9-force-single-sample.patch b/tools/dxvk/patches/0004-d3d9-force-single-sample.patch new file mode 100644 index 000000000..6ae7343c1 --- /dev/null +++ b/tools/dxvk/patches/0004-d3d9-force-single-sample.patch @@ -0,0 +1,86 @@ +diff --git a/dxvk.conf b/dxvk.conf +index ba4bc20..26d3127 100644 +--- a/dxvk.conf ++++ b/dxvk.conf +@@ -336,6 +336,15 @@ + + # d3d11.disableMsaa = False + ++# Forces D3D9 surfaces to single-sample after validating the original request. ++# The effective surface description reports D3DMULTISAMPLE_NONE; MSAA quality, ++# alpha-to-coverage and per-sample masks are consequently disabled. This is an ++# opt-in compatibility override for games with broken MSAA rendering. ++# ++# Supported values: True, False ++ ++# d3d9.forceSingleSample = False ++ + + # Forces insertion of memory and control barriers after writes to group-shared + # memory in compute shaders. This is only intended to be used as a workaround +diff --git a/src/d3d9/d3d9_common_texture.cpp b/src/d3d9/d3d9_common_texture.cpp +index 9d5fe38..629fea3 100644 +--- a/src/d3d9/d3d9_common_texture.cpp ++++ b/src/d3d9/d3d9_common_texture.cpp +@@ -212,6 +212,15 @@ namespace dxvk { + return D3DERR_INVALIDCALL; + } + ++ // Keep original request validation above. This opt-in override changes ++ // the effective surface description as well as the backing image. ++ // All surface descriptors and resolves then see the actual sample count. ++ if (ResourceType == D3DRTYPE_SURFACE && options->forceSingleSample ++ && sampleCount != VK_SAMPLE_COUNT_1_BIT) { ++ pDesc->MultiSample = D3DMULTISAMPLE_NONE; ++ pDesc->MultisampleQuality = 0; ++ } ++ + // Using MANAGED pool with DYNAMIC usage is illegal + if (IsPoolManaged(pDesc->Pool) && (pDesc->Usage & D3DUSAGE_DYNAMIC)) + return D3DERR_INVALIDCALL; +diff --git a/src/d3d9/d3d9_options.cpp b/src/d3d9/d3d9_options.cpp +index e46864e..0cab7dc 100644 +--- a/src/d3d9/d3d9_options.cpp ++++ b/src/d3d9/d3d9_options.cpp +@@ -59,6 +59,7 @@ namespace dxvk { + this->memoryTrackTest = config.getOption ("d3d9.memoryTrackTest", false); + this->forceSamplerTypeSpecConstants = config.getOption ("d3d9.forceSamplerTypeSpecConstants", false); + this->forceSampleRateShading = config.getOption ("d3d9.forceSampleRateShading", false); ++ this->forceSingleSample = config.getOption ("d3d9.forceSingleSample", false); + this->forceAspectRatio = config.getOption ("d3d9.forceAspectRatio", ""); + this->forceRefreshRate = config.getOption ("d3d9.forceRefreshRate", 0u); + this->modeCountCompatibility = config.getOption ("d3d9.modeCountCompatibility", false); +diff --git a/src/d3d9/d3d9_options.h b/src/d3d9/d3d9_options.h +index e69a39a..4478420 100644 +--- a/src/d3d9/d3d9_options.h ++++ b/src/d3d9/d3d9_options.h +@@ -113,6 +113,9 @@ namespace dxvk { + /// Forces sample rate shading + bool forceSampleRateShading; + ++ /// Optional compatibility override: report and render surfaces as single-sample. ++ bool forceSingleSample; ++ + /// Allow D3DLOCK_DISCARD + bool allowDiscard; + +diff --git a/src/d3d9/d3d9_swapchain.cpp b/src/d3d9/d3d9_swapchain.cpp +--- a/src/d3d9/d3d9_swapchain.cpp ++++ b/src/d3d9/d3d9_swapchain.cpp +@@ -1055,1 +1055,16 @@ namespace dxvk { ++ if (m_parent->GetOptions()->forceSingleSample) { ++ // Backbuffers bypass the normal surface-creation entry points. Validate ++ // the application's request, then apply the same effective sample count. ++ // IsLockable above is an internal GDI fallback requirement, not the ++ // application's lockability request used by MSAA validation. ++ auto requestedDesc = desc; ++ requestedDesc.IsLockable = (Flags & D3DPRESENTFLAG_LOCKABLE_BACKBUFFER) != 0; ++ HRESULT hr = D3D9CommonTexture::NormalizeTextureProperties( ++ m_parent, D3DRTYPE_SURFACE, &requestedDesc); ++ if (FAILED(hr)) ++ return hr; ++ desc.MultiSample = requestedDesc.MultiSample; ++ desc.MultisampleQuality = requestedDesc.MultisampleQuality; ++ } ++ + const bool isExtended = m_parent->IsD3DCompatibile(D3DCompatibility::D3D9Ex); diff --git a/tools/dxvk/source.env b/tools/dxvk/source.env new file mode 100644 index 000000000..7aac170c6 --- /dev/null +++ b/tools/dxvk/source.env @@ -0,0 +1 @@ +DXVK_REVISION=25ca63f17f34bdc05a39873ee63907d3cbbfa030 diff --git a/tools/dxvk/tests/d3d9_single_sample.cpp b/tools/dxvk/tests/d3d9_single_sample.cpp new file mode 100644 index 000000000..2c9782f92 --- /dev/null +++ b/tools/dxvk/tests/d3d9_single_sample.cpp @@ -0,0 +1,214 @@ +// Backend/adapter fixture for the verbatim DXVK functions in d3d9_production.inc. +#include +#include +#include +#include +#include +#include +#include + +using HRESULT = int32_t; +using DWORD = uint32_t; +using UINT = uint32_t; +using VkSampleCountFlagBits = uint32_t; +constexpr HRESULT D3D_OK = 0, D3DERR_INVALIDCALL = -1, + D3DERR_NOTAVAILABLE = -2, D3DERR_OUTOFVIDEOMEMORY = -3; +constexpr uint32_t VK_SAMPLE_COUNT_1_BIT = 1, VK_IMAGE_ASPECT_COLOR_BIT = 1; +constexpr bool FALSE = false, TRUE = true; +#define FAILED(x) ((x) < 0) +#define unlikely(x) (x) +enum D3DMULTISAMPLE_TYPE { D3DMULTISAMPLE_NONE, D3DMULTISAMPLE_NONMASKABLE, + D3DMULTISAMPLE_16_SAMPLES = 16 }; +enum D3DRESOURCETYPE { D3DRTYPE_SURFACE, D3DRTYPE_CUBETEXTURE, D3DRTYPE_VOLUMETEXTURE }; +enum D3DPOOL { D3DPOOL_DEFAULT, D3DPOOL_MANAGED, D3DPOOL_SYSTEMMEM, D3DPOOL_SCRATCH }; +enum class D3DCompatibility { D3D9Ex }; +enum class D3D9Format { X8R8G8B8, D24S8, A8, NULL_FORMAT, ATI2, + D32_LOCKABLE, D32F_LOCKABLE, D16_LOCKABLE, S8_LOCKABLE }; +constexpr DWORD D3DUSAGE_RENDERTARGET = 1, D3DUSAGE_DEPTHSTENCIL = 2, + D3DUSAGE_DYNAMIC = 4, D3DUSAGE_WRITEONLY = 8, D3DUSAGE_AUTOGENMIPMAP = 16; +constexpr DWORD D3DPRESENTFLAG_LOCKABLE_BACKBUFFER = 1, D3DPRESENTFLAG_DISCARD_DEPTHSTENCIL = 2; +constexpr int D3DDEVTYPE_HAL = 1, VK_IMAGE_LAYOUT_UNDEFINED = 0; +struct Extent { uint32_t width, height, depth = 1; }; +namespace util { +uint32_t computeMipLevelCount(Extent e) { + uint32_t n = std::max({e.width, e.height, e.depth}), levels = 0; + do { ++levels; n >>= 1; } while (n); + return levels; +} +} +bool IsPoolManaged(D3DPOOL pool) { return pool == D3DPOOL_MANAGED; } +bool IsDXTFormat(D3D9Format) { return false; } +bool IsDepthStencilFormat(D3D9Format format) { return format == D3D9Format::D24S8; } +bool IsLockableDepthStencilFormat(D3D9Format) { return false; } +struct D3D9_FORMAT_BLOCK_SIZE { uint32_t Width, Height; }; +D3D9_FORMAT_BLOCK_SIZE GetFormatAlignedBlockSize(D3D9Format) { return {1, 1}; } +D3D9Format EnumerateFormat(D3D9Format format) { return format; } +struct D3D9_COMMON_TEXTURE_DESC { + uint32_t Width = 1280, Height = 720, Depth = 1, MipLevels = 1, ArraySize = 1; + D3D9Format Format = D3D9Format::X8R8G8B8; + D3DMULTISAMPLE_TYPE MultiSample = D3DMULTISAMPLE_NONE; + DWORD MultisampleQuality = 0; + D3DPOOL Pool = D3DPOOL_DEFAULT; + DWORD Usage = D3DUSAGE_RENDERTARGET; + bool Discard = false, IsBackBuffer = false, IsAttachmentOnly = false, IsLockable = false; +}; +struct D3DPRESENT_PARAMETERS { + uint32_t BackBufferWidth = 1280, BackBufferHeight = 720; + D3D9Format BackBufferFormat = D3D9Format::X8R8G8B8; + D3DMULTISAMPLE_TYPE MultiSampleType = D3DMULTISAMPLE_TYPE(4); + DWORD MultiSampleQuality = 0, Flags = 0; + D3D9Format AutoDepthStencilFormat = D3D9Format::D24S8; +}; +struct Options { + bool forceSingleSample = false, extraFrontbuffer = false, + disableA8RT = false, supportCubeDepthFormats = false; +}; +struct FormatMapping { + uint32_t Aspect; + bool IsValid() const { return true; } +}; +struct Adapter { + bool supported = true; + std::vector checked; + HRESULT CheckDeviceMultiSampleType(int, D3D9Format, bool, D3DMULTISAMPLE_TYPE samples, void*) { + checked.push_back(samples); + return supported ? D3D_OK : D3DERR_NOTAVAILABLE; + } +}; +struct ImageInfo { Extent extent; VkSampleCountFlagBits sampleCount; }; +struct DxvkImage { ImageInfo info; }; +template using Rc = std::shared_ptr; +template using small_vector = std::vector; +struct DxvkContext { + unsigned initialized = 0; + void initImage(const Rc&, int) { ++initialized; } +}; +struct D3D9DeviceEx { + Options options; + Adapter adapter; + DxvkContext context; + unsigned attempts = 0, failAt = 0, losable = 0, live = 0, injected = 0; + std::vector created; + Options* GetOptions() { return &options; } + Adapter* GetAdapter() { return &adapter; } + FormatMapping LookupFormat(D3D9Format format) { return {IsDepthStencilFormat(format) ? 2u : 1u}; } + struct Unsupported { unsigned elementSize = 0; }; + Unsupported unsupported; + const Unsupported* UnsupportedFormatInfo(D3D9Format) { return &unsupported; } + bool IsD3DCompatibile(D3DCompatibility) { return false; } + void IncrementLosableCounter() { ++losable; } + template void InjectCs(F function) { ++injected; function(&context); } +}; +HRESULT DecodeMultiSampleType(D3DMULTISAMPLE_TYPE, DWORD, VkSampleCountFlagBits*); +struct D3D9CommonTexture { + Rc image; + const Rc& GetImage() const { return image; } + static HRESULT NormalizeTextureProperties(D3D9DeviceEx*, D3DRESOURCETYPE, D3D9_COMMON_TEXTURE_DESC*); + /* PRODUCTION_USES_IMAGE */ +}; +class DxvkError : public std::runtime_error { +public: + using std::runtime_error::runtime_error; + std::string message() const { return what(); } +}; +struct Logger { static void err(const std::string&) {} }; +struct D3D9SwapChainEx; +struct D3D9Surface { + D3D9CommonTexture texture; + D3D9DeviceEx* device; + D3D9Surface(D3D9DeviceEx* d, D3D9_COMMON_TEXTURE_DESC* desc, bool, D3D9SwapChainEx*, void*) : device(d) { + if (++d->attempts == d->failAt) throw DxvkError("controlled allocation failure"); + VkSampleCountFlagBits samples; + if (FAILED(DecodeMultiSampleType(desc->MultiSample, desc->MultisampleQuality, &samples))) + throw DxvkError("invalid backend sample count"); + texture.image = std::make_shared(DxvkImage{{{desc->Width, desc->Height}, samples}}); + d->created.push_back(*desc); ++d->live; + } + ~D3D9Surface() { --device->live; } + D3D9CommonTexture* GetCommonTexture() { return &texture; } +}; +struct D3D9SwapChainEx { + D3D9DeviceEx* m_parent; + D3DPRESENT_PARAMETERS m_presentParams; + std::vector> m_backBuffers; + bool flip = false; + unsigned destroys = 0; + bool SwapWithFrontBuffer() { return flip; } + void DestroyBackBuffers() { ++destroys; m_backBuffers.clear(); } + HRESULT CreateBackBuffers(uint32_t, DWORD); +}; +#include "d3d9_production.inc" + +void paired(bool enabled, D3DMULTISAMPLE_TYPE type, DWORD quality) { + D3D9DeviceEx device; device.options.forceSingleSample = enabled; + device.options.extraFrontbuffer = true; + D3D9SwapChainEx chain{&device, {}, {}}; + chain.m_presentParams.MultiSampleType = type; chain.m_presentParams.MultiSampleQuality = quality; + assert(chain.CreateBackBuffers(2, 0) == D3D_OK); + D3D9_COMMON_TEXTURE_DESC depth; + assert(auto_depth(&device, &chain.m_presentParams, &depth) == D3D_OK); + VkSampleCountFlagBits requested, dsSamples; + assert(DecodeMultiSampleType(type, quality, &requested) == D3D_OK); + assert(DecodeMultiSampleType(depth.MultiSample, depth.MultisampleQuality, &dsSamples) == D3D_OK); + assert(chain.m_backBuffers.size() == 3 && device.context.initialized == 3); + for (const auto& surface : chain.m_backBuffers) { + const auto& image = surface->GetCommonTexture()->GetImage()->info; + assert(image.sampleCount == (enabled ? 1u : requested)); + assert(depth_binds(image, {{depth.Width, depth.Height}, dsSamples})); + } + for (const auto& desc : device.created) { + assert(desc.IsLockable && desc.IsBackBuffer && !desc.IsAttachmentOnly); + assert(desc.MultiSample == (enabled ? D3DMULTISAMPLE_NONE : type)); + assert(desc.MultisampleQuality == (enabled ? 0u : quality)); + } + // Original presentation parameters remain available for reset/auto-depth validation. + assert(chain.m_presentParams.MultiSampleType == type && chain.m_presentParams.MultiSampleQuality == quality); + if (enabled && requested > 1) assert(device.adapter.checked.front() == D3DMULTISAMPLE_TYPE(requested)); +} + +int main(int argc, char** argv) { + assert(argc == 2); std::string name = argv[1]; + if (name == "forced_msaa_depth") paired(true, D3DMULTISAMPLE_TYPE(4), 0); + else if (name == "default_msaa_depth") paired(false, D3DMULTISAMPLE_TYPE(4), 0); + else if (name == "forced_nonmaskable") paired(true, D3DMULTISAMPLE_NONMASKABLE, 2); + else if (name == "default_nonmaskable") paired(false, D3DMULTISAMPLE_NONMASKABLE, 2); + else if (name == "forced_already_single") paired(true, D3DMULTISAMPLE_NONE, 0); + else if (name == "forced_no_depth" || name == "forced_lockable_single") { + D3D9DeviceEx d; d.options.forceSingleSample = true; + D3D9SwapChainEx c{&d, {}, {}}; + if (name == "forced_lockable_single") c.m_presentParams.MultiSampleType = D3DMULTISAMPLE_NONE; + assert(c.CreateBackBuffers(1, name == "forced_lockable_single" ? D3DPRESENTFLAG_LOCKABLE_BACKBUFFER : 0) == D3D_OK); + assert(c.m_backBuffers.size() == 1 && d.created[0].MultiSample == D3DMULTISAMPLE_NONE); + assert(d.created[0].IsLockable && d.context.initialized == 1); + } else if (name == "forced_unsupported" || name == "forced_invalid_type" || name == "forced_lockable_msaa") { + D3D9DeviceEx d; d.options.forceSingleSample = true; + D3D9SwapChainEx c{&d, {}, {}}; + if (name == "forced_unsupported") d.adapter.supported = false; + if (name == "forced_invalid_type") c.m_presentParams.MultiSampleType = D3DMULTISAMPLE_TYPE(17); + HRESULT expected = name == "forced_unsupported" ? D3DERR_NOTAVAILABLE : D3DERR_INVALIDCALL; + assert(c.CreateBackBuffers(2, name == "forced_lockable_msaa" ? D3DPRESENTFLAG_LOCKABLE_BACKBUFFER : 0) == expected); + assert(d.attempts == 0 && d.losable == 0 && d.injected == 0 && c.m_backBuffers.empty()); + } else if (name == "forced_allocation_failure" || name == "default_allocation_failure") { + D3D9DeviceEx d; d.options.forceSingleSample = name == "forced_allocation_failure"; d.failAt = 2; + D3D9SwapChainEx c{&d, {}, {}}; + assert(c.CreateBackBuffers(3, 0) == D3DERR_OUTOFVIDEOMEMORY); + assert(d.attempts == 2 && d.losable == 1 && d.live == 0 && d.injected == 0 && c.m_backBuffers.empty()); + } else if (name == "forced_recreate") { + D3D9DeviceEx d; d.options.forceSingleSample = true; + D3D9SwapChainEx c{&d, {}, {}}; + assert(c.CreateBackBuffers(3, 0) == D3D_OK); + c.m_presentParams.MultiSampleType = D3DMULTISAMPLE_TYPE(8); + assert(c.CreateBackBuffers(1, 0) == D3D_OK); + assert(d.live == 1 && c.m_backBuffers.size() == 1 && c.destroys == 2); + assert(d.adapter.checked.size() == 2 && d.adapter.checked.back() == D3DMULTISAMPLE_TYPE(8)); + assert(c.m_backBuffers[0]->GetCommonTexture()->GetImage()->info.sampleCount == 1); + } else if (name == "ordinary_surface" || name == "ordinary_invalid_lockable") { + D3D9DeviceEx d; d.options.forceSingleSample = true; + D3D9_COMMON_TEXTURE_DESC desc; desc.MultiSample = D3DMULTISAMPLE_TYPE(4); + desc.IsLockable = name == "ordinary_invalid_lockable"; + assert(D3D9CommonTexture::NormalizeTextureProperties(&d, D3DRTYPE_SURFACE, &desc) + == (desc.IsLockable ? D3DERR_INVALIDCALL : D3D_OK)); + if (!desc.IsLockable) assert(desc.MultiSample == D3DMULTISAMPLE_NONE && desc.MultisampleQuality == 0); + } else assert(false); + return 0; +} diff --git a/tools/hybris/low32/README.md b/tools/hybris/low32/README.md new file mode 100644 index 000000000..31d5305b6 --- /dev/null +++ b/tools/hybris/low32/README.md @@ -0,0 +1,99 @@ +# Opt-in device-backed low mappings for Wine WoW64 + +On the tested Pixel PowerVR stack, Vulkan maps host-visible GPU allocations +above 4 GiB. Wine's 32-bit WoW64 Vulkan bridge rejects those pointers. This +caused American Nightmare (202750) to stop at D3D9 vertex-buffer creation even +after the separate PowerVR BC texture profile passed device initialization. + +`low32.c` is an experimental native Linux/glibc preload adapter. It is off by +default. It applies only to anonymous-address, non-executable `MAP_SHARED` +mappings of one explicitly selected character device. It retains the original +device FD, offset, protection, flags, and actual GPU backing, and searches low +address space with `MAP_FIXED_NOREPLACE`. It never truncates a pointer, copies +into shadow storage, replaces existing mappings, or disables Wine's guard. +Caller-specified/fixed mappings, regular files, private/anonymous memory and +other devices retain their original behavior. Failure to find a suitable low +hole fails with ENOMEM. The mmap and mmap64 preload chains are independently +resolved and called through their respective RTLD_NEXT symbols; +only resolver bootstrap uses the raw syscall. A compact atomic cursor avoids +wasting large address slots on thousands of small driver control allocations. +If the compact probe collides, a bounded address-range walk of +/proc/self/maps finds page-sized holes and uses NOREPLACE again to handle races. +The walk runs before coarse fallback, so small allocations do not fragment +large available ranges. A game load had 2.3 GB free below 4 GiB but no hole as large as 4 MiB; the +coarse-only version falsely failed even 4 KiB mappings. Mapping failures are +reported even after the ordinary 256-event trace limit. + +## Build and verify + +Use a native 64-bit Linux/glibc compiler, not the Android NDK or macOS compiler: + +```sh +tools/hybris/low32/build.sh /absolute/build-dir /path/to/Vulkan-Headers/include +``` + +The native library and unit test are always built. GPU probes are optional when +headers are provided; `gpu-copy-native` uses the selected native host compiler +and has the host's architecture, while the Windows x86 probe also needs +i686-w64-mingw32-gcc. Copy a native probe to a device only when their architectures match. +The unit test uses /dev/zero and proves CPU data integrity, collision handling, +concurrency, 2000 simultaneous small mappings, invalid offsets, forwarding and +opt-out. A fragmented-address-space regression reserves the low region, opens +64 off-grid page holes, and verifies all 64 mappings and their data. The old +coarse adapter fails that regression; the address-range fallback passes. An +additional density regression fills 64 existing page holes while preserving a +separate contiguous 16 MiB interval for a later complete data-readback map; the +prior gap-after-coarse implementation fails and the gap-first strategy passes. These +CPU tests do not by themselves prove GPU correctness. +The build also tests both preload orders with a second library that hooks mmap +and mmap64 independently. Opt-out, tracing and adaptation must each preserve +the selected symbol's chain, invalid-offset errno and file offsets above 4 GiB. + +`gpu-copy.c` tests actual Vulkan transfer operations on vendor 0x1010. It maps +two 32 MiB device allocations, runs four copies of full deterministic patterns, +including offset/unmap/remap, uses flush/invalidate and barriers/fences, and +compares every word. It can require each full CPU range to fit below 4 GiB: + +```sh +./gpu-copy-native 32 --require-low32 +# Through a separate Proton ARM64/WoW64 test prefix: +proton run gpu-copy-x86.exe 32 --require-low32 'Z:\path\probe-output.txt' +``` + +The Windows output file avoids Wine's detached console hiding probe results. +Run with Khronos synchronization validation. Keep raw Steam/account logs local; +export only controlled PROBE results, validation IDs and counts. The tested +memory type was device-local, host-visible and coherent; noncoherent physical +memory is not covered by that run. + +## Activation scope + +Set only the target game's overrides, after identifying the render device's +`st_rdev` and validating the installed library's hash: + +```text +LD_PRELOAD=/absolute/guest/path/libdroiddeck-low32.so +DROIDDECK_LOW32_MODE=low32 +DROIDDECK_LOW32_RDEV= +``` + +Preserve any existing target-game preload rather than replacing it. The tested +Pixel's /dev/dri/renderD128 was character device major226/minor128, st_rdev57984. +Do not hardcode that value for other machines. BC emulation and the repaired +DXVK presentation component remain separate prerequisites; this adapter does +not implement them. No global Steam profile, managed graphics pack, Android +security setting or unrelated game's overrides should change. + +Before activating, back up the exact game-environment bytes and only the target +prefix's graphics DLLs, preserving symlinks. Rollback must check for concurrent +edits. Keep original backups in the app; do not copy account files or an entire +Steam prefix to a report. Removing these three overrides disables the adapter. + +## Compatibility limits + +This initial binary was built on Ubuntu24.04 ARM64 and needs glibc2.38. The +tested guest supports it. This is not yet a universal runtime component; other +glibc versions, page sizes, drivers, GPU families, native32-bit processes and +general game stability require their own tests. Bounded native and Windows GPU +copies prove data integrity for those workloads, not unrestricted gameplay. +Never enable globally based on successful testing of one game. diff --git a/tools/hybris/low32/build.sh b/tools/hybris/low32/build.sh new file mode 100755 index 000000000..f92307066 --- /dev/null +++ b/tools/hybris/low32/build.sh @@ -0,0 +1,27 @@ +#!/bin/sh +set -eu +# Run on native 64-bit Linux/glibc. This is not an Android/Bionic library. +base=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +output=${1:?usage: build.sh /absolute/build-directory [Vulkan-header-directory]} +mkdir -p "$output" +cc=${CC:-cc} +"$cc" -std=gnu11 -Wall -Wextra -Werror -O2 -fPIC -shared -s \ + "$base/low32.c" -ldl -o "$output/libdroiddeck-low32.so" +"$cc" -std=gnu11 -Wall -Wextra -Werror -O2 "$base/test-low32.c" -pthread -o "$output/test-low32" +LD_PRELOAD="$output/libdroiddeck-low32.so" "$output/test-low32" +"$cc" -std=gnu11 -Wall -Wextra -Werror -O2 -fPIC -shared \ + "$base/test-chain-hook.c" -ldl -o "$output/test-chain-hook.so" +"$cc" -std=gnu11 -Wall -Wextra -Werror -O2 "$base/test-preload-chain.c" -ldl -o "$output/test-preload-chain" +for preloads in "$output/libdroiddeck-low32.so:$output/test-chain-hook.so" \ + "$output/test-chain-hook.so:$output/libdroiddeck-low32.so"; do + for mode in off trace low32; do + LD_PRELOAD="$preloads" "$output/test-preload-chain" "$mode" + done +done +if [ "$#" -ge 2 ]; then + "$cc" -std=gnu11 -Wall -Wextra -Werror -O2 -s -I"$2" "$base/gpu-copy.c" -ldl -o "$output/gpu-copy-native" + if command -v i686-w64-mingw32-gcc >/dev/null 2>&1; then + i686-w64-mingw32-gcc -std=gnu11 -Wall -Wextra -Werror -O2 -s -I"$2" \ + "$base/gpu-copy.c" -o "$output/gpu-copy-x86.exe" + fi +fi diff --git a/tools/hybris/low32/gpu-copy.c b/tools/hybris/low32/gpu-copy.c new file mode 100644 index 000000000..d59dbf677 --- /dev/null +++ b/tools/hybris/low32/gpu-copy.c @@ -0,0 +1,166 @@ +/* Same standalone GPU-copy probe built as Linux AArch64 and Windows x86. + * Tests actual Vulkan buffers, complete data comparison, noncoherent cache + * operations, remapping with an offset and queue completion. No Steam API. + */ +#define VK_NO_PROTOTYPES +#ifdef _WIN32 +#include +#else +#include +#endif +#include +#include +#include +#include +#include +#include + +#define IFNS(X) X(DestroyInstance) X(EnumeratePhysicalDevices) X(GetPhysicalDeviceProperties) \ + X(GetPhysicalDeviceMemoryProperties) X(GetPhysicalDeviceQueueFamilyProperties) X(CreateDevice) X(GetDeviceProcAddr) +#define DFNS(X) X(DestroyDevice) X(GetDeviceQueue) X(CreateBuffer) X(DestroyBuffer) \ + X(GetBufferMemoryRequirements) X(AllocateMemory) X(FreeMemory) X(BindBufferMemory) \ + X(MapMemory) X(UnmapMemory) X(FlushMappedMemoryRanges) X(InvalidateMappedMemoryRanges) \ + X(CreateCommandPool) X(DestroyCommandPool) X(AllocateCommandBuffers) X(ResetCommandBuffer) \ + X(BeginCommandBuffer) X(EndCommandBuffer) X(CmdCopyBuffer) X(CmdPipelineBarrier) \ + X(CreateFence) X(DestroyFence) X(ResetFences) X(QueueSubmit) X(WaitForFences) +#define DECL(n) static PFN_vk##n p##n; +IFNS(DECL) +DFNS(DECL) +#undef DECL +#define CHECK(x) do { VkResult rc=(x); if(rc!=VK_SUCCESS){printf("PROBE_FAIL stage=%s vk_result=%d\n",#x,rc);goto cleanup;} }while(0) +#define REQUIRE(x,msg) do {if(!(x)){puts("PROBE_FAIL " msg);goto cleanup;}}while(0) + +int main(int argc,char **argv) +{ +#ifdef _WIN32 + /* Wine/Proton can detach the Windows console from the Linux pipe. */ + if(argc>3 && !freopen(argv[3],"w",stdout)) return 4; +#endif + setvbuf(stdout,NULL,_IONBF,0); + puts("PROBE_STAGE main"); + unsigned mib=argc>1?(unsigned)strtoul(argv[1],NULL,10):32; + int require_low=argc>2&&!strcmp(argv[2],"--require-low32"); + if(mib<1||mib>256)return 2; + const VkDeviceSize bytes=(VkDeviceSize)mib*1024*1024; + VkInstance instance=VK_NULL_HANDLE;VkDevice device=VK_NULL_HANDLE; + VkBuffer buffers[2]={0};VkDeviceMemory memories[2]={0}; + VkDeviceSize allocated[2]={0};void *mapped[2]={0}; + VkCommandPool pool=VK_NULL_HANDLE;VkFence fence=VK_NULL_HANDLE; + int success=0,pending=0;uint64_t mismatches=0;unsigned iterations=0; + int low_all=1;uint32_t memory_flags[2]={0}; + PFN_vkGetInstanceProcAddr gipa; +#ifdef _WIN32 + HMODULE loader=LoadLibraryA("vulkan-1.dll"); + REQUIRE(loader,"load_vulkan_failed"); + gipa=(PFN_vkGetInstanceProcAddr)(void *)GetProcAddress(loader,"vkGetInstanceProcAddr"); +#else + void *loader=dlopen("libvulkan.so.1",RTLD_NOW|RTLD_LOCAL); + REQUIRE(loader,"load_vulkan_failed"); + gipa=(PFN_vkGetInstanceProcAddr)dlsym(loader,"vkGetInstanceProcAddr"); +#endif + REQUIRE(gipa,"missing_gipa"); + puts("PROBE_STAGE vulkan_loader"); + PFN_vkCreateInstance create=(PFN_vkCreateInstance)gipa(NULL,"vkCreateInstance"); + REQUIRE(create,"missing_create_instance"); + VkApplicationInfo app={.sType=VK_STRUCTURE_TYPE_APPLICATION_INFO, + .pApplicationName="DroidDeck low32 buffer integrity probe",.apiVersion=VK_API_VERSION_1_1}; + VkInstanceCreateInfo ici={.sType=VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO,.pApplicationInfo=&app}; + CHECK(create(&ici,NULL,&instance)); + puts("PROBE_STAGE instance_created"); +#define LOADI(n) p##n=(PFN_vk##n)gipa(instance,"vk" #n);REQUIRE(p##n,"missing_" #n); + IFNS(LOADI) +#undef LOADI + uint32_t count=0;CHECK(pEnumeratePhysicalDevices(instance,&count,NULL)); + puts("PROBE_STAGE physical_device_enumeration"); + REQUIRE(count&&count<=16,"physical_device_count"); + VkPhysicalDevice physicals[16],physical=VK_NULL_HANDLE; + CHECK(pEnumeratePhysicalDevices(instance,&count,physicals)); + VkPhysicalDeviceProperties properties; + for(uint32_t i=0;i +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef MAP_FIXED_NOREPLACE +#define MAP_FIXED_NOREPLACE 0x100000 +#endif + +static _Atomic uint64_t cursor; +static _Atomic unsigned logged; +typedef void *(*map_function)(void *,size_t,int,int,int,off_t); +typedef void *(*map64_function)(void *,size_t,int,int,int,off64_t); +static _Atomic(map_function) next_map; +static _Atomic(map64_function) next_map64; +_Static_assert(sizeof(off_t) == sizeof(off64_t) && sizeof(off_t) == 8, + "64-bit host required"); +static __thread int resolving __attribute__((tls_model("initial-exec"))); +static __thread int active __attribute__((tls_model("initial-exec"))); +static void *native_map(void *addr, size_t length, int prot, int flags, int fd, off_t offset) +{ + map_function next=atomic_load(&next_map); + if (!next && !resolving) { + resolving=1; + next=(map_function)dlsym(RTLD_NEXT,"mmap"); + atomic_store(&next_map,next); + resolving=0; + } + /* Bootstrap only while resolving; otherwise preserve existing preload hooks. */ + if (next) return next(addr,length,prot,flags,fd,offset); + return (void *)syscall(SYS_mmap, addr, length, prot, flags, fd, offset); +} + +static void *native_map64(void *addr, size_t length, int prot, int flags, int fd, off_t offset) +{ + map64_function next=atomic_load(&next_map64); + if (!next && !resolving) { + resolving=1; + next=(map64_function)dlsym(RTLD_NEXT,"mmap64"); + atomic_store(&next_map64,next); + resolving=0; + } + if (next) return next(addr,length,prot,flags,fd,(off64_t)offset); + return (void *)syscall(SYS_mmap, addr, length, prot, flags, fd, offset); +} + +static void trace_map(const char *kind, size_t n, int prot, int flags, int fd, + off_t off, const struct stat *st, void *p, int error) +{ + int saved = errno; + unsigned count = atomic_fetch_add(&logged, 1); + if (count < 256 || error != 0) { + char text[320]; + int nbytes = snprintf(text, sizeof(text), + "DD_LOW32 %s pid=%ld len=%zu prot=%x flags=%x fd=%d off=%jd mode=%o rdev=%ju ptr=%p errno=%d\n", + kind, (long)getpid(), n, prot, flags, fd, (intmax_t)off, st ? st->st_mode : 0, + st ? (uintmax_t)st->st_rdev : 0, p, error); + if (nbytes > 0) { + ssize_t written = write(STDERR_FILENO, text, + (size_t)nbytes < sizeof(text) ? (size_t)nbytes : sizeof(text)-1); + (void)written; + } + } + errno = saved; +} + +static int target_device(int fd, struct stat *st) +{ + const char *value = getenv("DROIDDECK_LOW32_RDEV"); + char *end = NULL; + if (!value || !*value || fd < 0) return 0; + int saved = errno; + errno = 0; + uintmax_t expected = strtoumax(value, &end, 10); + int valid = !errno && end && !*end && expected > 0 && + fstat(fd, st) == 0 && S_ISCHR(st->st_mode) && + (uintmax_t)st->st_rdev == expected; + errno = saved; + return valid; +} + +static void *adapted_map(map_function forward, void *addr, size_t length, int prot, + int flags, int fd, off_t offset) +{ + const char *mode = getenv("DROIDDECK_LOW32_MODE"); + struct stat st = {0}; + if (mode && !strcmp(mode, "trace")) { + void *p = forward(addr, length, prot, flags, fd, offset); + if (fd >= 0 && (prot & PROT_WRITE) && !(prot & PROT_EXEC) && + (flags & MAP_TYPE) == MAP_SHARED) { + int saved=errno; + (void)fstat(fd,&st); + errno=saved; + trace_map("trace", length, prot, flags, fd, offset, &st, p, + p == MAP_FAILED ? errno : 0); + } + return p; + } + if (!mode || strcmp(mode, "low32") || addr || !length || + (flags & (MAP_ANONYMOUS | MAP_FIXED | MAP_FIXED_NOREPLACE)) || + (prot & PROT_EXEC) || (flags & MAP_TYPE) != MAP_SHARED || + !target_device(fd, &st)) + return forward(addr, length, prot, flags, fd, offset); + + const uintptr_t lower = 0x10000000u, upper = 0xffff0000u; + const uintptr_t step = 0x00400000u; + if (length > upper-lower) { + errno = ENOMEM; + trace_map("too_large", length, prot, flags, fd, offset, &st, MAP_FAILED, errno); + return MAP_FAILED; + } + long page_value=sysconf(_SC_PAGESIZE); + uintptr_t page=page_value>0?(uintptr_t)page_value:4096; + uintptr_t rounded=(length+page-1)&~(page-1); + uintptr_t span=upper-lower; + uintptr_t compact=(uintptr_t)(atomic_fetch_add(&cursor,rounded)%span); + if (rounded>span-compact) compact=0; + unsigned slots = (unsigned)((span-length)/step)+1; + unsigned start = (unsigned)(compact/step)%slots; + int original_errno = errno; + for (unsigned i = 0; i < 1; i++) { + /* Compact first: hundreds of small driver mappings must not waste a + * 4MiB address slot each. Coarse search is only for existing Wine holes. */ + uintptr_t hint = lower + (i==0?compact:(uintptr_t)((start+i-1)%slots)*step); + void *p = forward((void *)hint, length, prot, + flags|MAP_FIXED_NOREPLACE, fd, offset); + if (p != MAP_FAILED) { + uintptr_t first = (uintptr_t)p; + if (first >= lower && first < upper && length <= upper-first) { + errno = original_errno; + trace_map("mapped", length, prot, flags, fd, offset, &st, p, 0); + return p; + } + /* Older kernels can ignore NOREPLACE; never return a high mapping. */ + int saved = errno; + (void)syscall(SYS_munmap, p, length); + errno = saved; + } else if (errno != EEXIST && errno != ENOMEM && errno != EAGAIN) { + trace_map("rejected", length, prot, flags, fd, offset, &st, p, errno); + return p; + } + } + /* Pack small allocations into address holes before coarse probing can + * fragment large holes. Walk + * this process's current mappings, then use NOREPLACE to tolerate races. + * No pointer paths or contents are read; only address ranges are used. */ + for (unsigned pass = 0; pass < 2; ++pass) { + FILE *maps = fopen("/proc/self/maps", "r"); + if (!maps) break; + char line[512]; + uintptr_t gap_start = lower; + unsigned lines = 0; + while (fgets(line, sizeof(line), maps) && ++lines < 65536) { + uintptr_t first, last; + if (sscanf(line, "%" SCNxPTR "-%" SCNxPTR, &first, &last) != 2) continue; + if (last <= gap_start) continue; + uintptr_t gap_end = first < upper ? first : upper; + if (gap_end > gap_start && rounded <= gap_end - gap_start) { + void *p = forward((void *)gap_start, length, prot, + flags | MAP_FIXED_NOREPLACE, fd, offset); + if (p != MAP_FAILED) { + uintptr_t address = (uintptr_t)p; + if (address >= lower && address < upper && length <= upper - address) { + fclose(maps); + errno = original_errno; + trace_map("mapped_gap", length, prot, flags, fd, offset, &st, p, 0); + return p; + } + (void)syscall(SYS_munmap, p, length); + } else if (errno != EEXIST && errno != ENOMEM && errno != EAGAIN) { + int saved = errno; + fclose(maps); + errno = saved; + trace_map("rejected", length, prot, flags, fd, offset, &st, MAP_FAILED, errno); + return MAP_FAILED; + } + } + if (last > gap_start) gap_start = last; + if (gap_start >= upper) break; + } + if (gap_start < upper && rounded <= upper - gap_start) { + void *p = forward((void *)gap_start, length, prot, + flags | MAP_FIXED_NOREPLACE, fd, offset); + if (p != MAP_FAILED) { + uintptr_t address = (uintptr_t)p; + if (address >= lower && address < upper && length <= upper - address) { + fclose(maps); + errno = original_errno; + trace_map("mapped_gap", length, prot, flags, fd, offset, &st, p, 0); + return p; + } + (void)syscall(SYS_munmap, p, length); + } + } + fclose(maps); + } + for (unsigned i = 1; i <= slots; i++) { + /* Last resort if address walking was unavailable or raced. */ + uintptr_t hint = lower + (i==0?compact:(uintptr_t)((start+i-1)%slots)*step); + void *p = forward((void *)hint, length, prot, + flags|MAP_FIXED_NOREPLACE, fd, offset); + if (p != MAP_FAILED) { + uintptr_t first = (uintptr_t)p; + if (first >= lower && first < upper && length <= upper-first) { + errno = original_errno; + trace_map("mapped", length, prot, flags, fd, offset, &st, p, 0); + return p; + } + /* Older kernels can ignore NOREPLACE; never return a high mapping. */ + int saved = errno; + (void)syscall(SYS_munmap, p, length); + errno = saved; + } else if (errno != EEXIST && errno != ENOMEM && errno != EAGAIN) { + trace_map("rejected", length, prot, flags, fd, offset, &st, p, errno); + return p; + } + } + errno = ENOMEM; + trace_map("no_low_hole", length, prot, flags, fd, offset, &st, MAP_FAILED, errno); + return MAP_FAILED; +} + +static void *intercept_map(map_function forward, void *addr, size_t length, + int prot, int flags, int fd, off_t offset) +{ + if (active) return forward(addr,length,prot,flags,fd,offset); + active=1; + void *p=adapted_map(forward,addr,length,prot,flags,fd,offset); + active=0; + return p; +} + +void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset) +{ + return intercept_map(native_map,addr,length,prot,flags,fd,offset); +} + +void *mmap64(void *addr, size_t length, int prot, int flags, int fd, off64_t offset) +{ + return intercept_map(native_map64,addr,length,prot,flags,fd,(off_t)offset); +} diff --git a/tools/hybris/low32/test-chain-hook.c b/tools/hybris/low32/test-chain-hook.c new file mode 100644 index 000000000..b5e68a1e6 --- /dev/null +++ b/tools/hybris/low32/test-chain-hook.c @@ -0,0 +1,37 @@ +/* Controlled second preload: distinct mmap/mmap64 counters and 64-bit offsets. */ +#define _GNU_SOURCE +#define _LARGEFILE64_SOURCE +#include +#include +#include +#include + +static int selected_fd = -2; +static unsigned calls[2]; +static int64_t offsets[2]; + +void chain_clear(int fd) +{ + selected_fd = fd; + calls[0] = calls[1] = 0; + offsets[0] = offsets[1] = -1; +} + +unsigned chain_calls(unsigned channel) { return calls[channel]; } +int64_t chain_offset(unsigned channel) { return offsets[channel]; } + +void *mmap(void *addr, size_t len, int prot, int flags, int fd, off_t off) +{ + static void *(*next)(void *, size_t, int, int, int, off_t); + if (!next) next = dlsym(RTLD_NEXT, "mmap"); + if (fd == selected_fd) { ++calls[0]; offsets[0] = off; } + return next(addr, len, prot, flags, fd, off); +} + +void *mmap64(void *addr, size_t len, int prot, int flags, int fd, off64_t off) +{ + static void *(*next)(void *, size_t, int, int, int, off64_t); + if (!next) next = dlsym(RTLD_NEXT, "mmap64"); + if (fd == selected_fd) { ++calls[1]; offsets[1] = off; } + return next(addr, len, prot, flags, fd, off); +} diff --git a/tools/hybris/low32/test-low32.c b/tools/hybris/low32/test-low32.c new file mode 100644 index 000000000..88c2b7130 --- /dev/null +++ b/tools/hybris/low32/test-low32.c @@ -0,0 +1,119 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define N (32u*1024u*1024u) +static int zero_fd; +static void *thread_test(void *arg) +{ + unsigned seed = (unsigned)(uintptr_t)arg; + uint32_t *p = mmap(NULL,N,PROT_READ|PROT_WRITE,MAP_SHARED,zero_fd,0); + assert(p != MAP_FAILED && (uintptr_t)p+N <= 0x100000000ull); + for (size_t i=0;i=0); + struct stat st; assert(!fstat(zero_fd,&st)); char value[48]; + snprintf(value,sizeof(value),"%ju",(uintmax_t)st.st_rdev); + assert(!setenv("DROIDDECK_LOW32_RDEV",value,1)); + assert(!setenv("DROIDDECK_LOW32_MODE","low32",1)); + /* Existing low-address memory must survive device mappings. */ + uint32_t *guard=mmap((void *)0x10000000,128u*1024u*1024u,PROT_READ|PROT_WRITE, + MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED_NOREPLACE,-1,0); + assert(guard==(void *)0x10000000);guard[0]=0xdeadbeef;guard[32767]=0xcafef00d; + pthread_t threads[8]; + for (uintptr_t i=0;i<8;i++) assert(!pthread_create(&threads[i],NULL,thread_test,(void *)i)); + for (unsigned i=0;i<8;i++) assert(!pthread_join(threads[i],NULL)); + assert(guard[0]==0xdeadbeef && guard[32767]==0xcafef00d); + puts("PASS concurrent 32MiB mappings, full pattern readback and occupied-memory preservation"); + assert(!munmap(guard,128u*1024u*1024u)); + /* Caller-specified address and MAP_FIXED semantics are preserved. */ + void *explicit=mmap((void *)0x30000000,4096,PROT_READ|PROT_WRITE, + MAP_SHARED|MAP_FIXED_NOREPLACE,zero_fd,0); + assert(explicit==(void *)0x30000000); assert(!munmap(explicit,4096)); + int mem=memfd_create("low32-nontarget",0);assert(mem>=0);assert(!ftruncate(mem,N)); + uint32_t *q=mmap((void *)0x500000000ull,N,PROT_READ|PROT_WRITE,MAP_SHARED,mem,0); + assert(q==(void *)0x500000000ull);q[0]=123;assert(q[0]==123);assert(!munmap(q,N));close(mem); + puts("PASS unrelated regular-file mapping and explicit-address forwarding"); + errno=0; + void *bad=mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_SHARED,zero_fd,1); + assert(bad==MAP_FAILED && errno==EINVAL); + puts("PASS invalid device offset retains EINVAL"); + /* Real game initialization creates many small control/buffer mappings. */ + void *small[2000]; + for(unsigned i=0;i<2000;i++) { + small[i]=mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_SHARED,zero_fd,0); + assert(small[i]!=MAP_FAILED && (uintptr_t)small[i]+4096<=0x100000000ull); + *(uint32_t *)small[i]=i; + } + for(unsigned i=0;i<2000;i++){ + assert(*(uint32_t *)small[i]==i);assert(!munmap(small[i],4096)); + } + puts("PASS 2000 simultaneous small mappings without address-slot exhaustion"); + /* A fragmented Wine address space has usable holes away from every + * 4MiB probe location. The mapping walk must find them without clobbering. */ + const uintptr_t lo=0x10000000u, hi=0xffff0000u; + void *reservation=mmap((void *)lo,hi-lo,PROT_NONE, + MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE|MAP_FIXED_NOREPLACE,-1,0); + assert(reservation==(void *)lo); + void *holes[64], *fragmented[64]; + for (unsigned i=0;i<64;i++) { + holes[i]=(void *)(lo+0x12345000u+(uintptr_t)i*0x10000u); + assert(!munmap(holes[i],4096)); + } + for (unsigned i=0;i<64;i++) { + fragmented[i]=mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_SHARED,zero_fd,0); + assert(fragmented[i]!=MAP_FAILED); + int found=0;for(unsigned j=0;j<64;j++)found|=fragmented[i]==holes[j];assert(found); + *(uint32_t *)fragmented[i]=i+77; + } + for (unsigned i=0;i<64;i++) assert(*(uint32_t *)fragmented[i]==i+77); + for (unsigned i=0;i<64;i++) assert(!munmap(fragmented[i],4096)); + assert(!munmap(reservation,hi-lo)); + puts("PASS fragmented low32 mappings find 64 page holes beyond coarse probes"); + /* Small allocations must consume existing page holes rather than punch + * coarse-grid holes into the only large interval needed by a later map. */ + reservation=mmap((void *)lo,hi-lo,PROT_NONE, + MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE|MAP_FIXED_NOREPLACE,-1,0); + assert(reservation==(void *)lo); + for (unsigned i=0;i<64;i++) assert(!munmap(holes[i],4096)); + const uintptr_t large_address=0x70000000u; + const size_t large_size=16u*1024u*1024u; + assert(!munmap((void *)large_address,large_size)); + for (unsigned i=0;i<64;i++) { + fragmented[i]=mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_SHARED,zero_fd,0); + assert(fragmented[i]!=MAP_FAILED); + int found=0;for(unsigned j=0;j<64;j++)found|=fragmented[i]==holes[j];assert(found); + *(uint32_t *)fragmented[i]=i+99; + } + uint32_t *large=mmap(NULL,large_size,PROT_READ|PROT_WRITE,MAP_SHARED,zero_fd,0); + assert(large==(void *)large_address); + for(size_t i=0;i +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +int main(int argc, char **argv) +{ + assert(argc == 2); + void (*clear)(int) = dlsym(RTLD_DEFAULT, "chain_clear"); + unsigned (*calls)(unsigned) = dlsym(RTLD_DEFAULT, "chain_calls"); + int64_t (*offset)(unsigned) = dlsym(RTLD_DEFAULT, "chain_offset"); + assert(clear && calls && offset); + if (strcmp(argv[1], "off")) assert(!setenv("DROIDDECK_LOW32_MODE", argv[1], 1)); + else assert(!unsetenv("DROIDDECK_LOW32_MODE")); + int fd = open("/dev/zero", O_RDWR | O_CLOEXEC); + assert(fd >= 0); + struct stat st; + assert(!fstat(fd, &st)); + char rdev[48]; + snprintf(rdev, sizeof(rdev), "%ju", (uintmax_t)st.st_rdev); + assert(!setenv("DROIDDECK_LOW32_RDEV", rdev, 1)); + for (unsigned channel = 0; channel < 2; ++channel) { + clear(fd); + void *p = channel ? mmap64(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0) + : mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + assert(p != MAP_FAILED); + assert(calls(channel) == 1 && calls(1 - channel) == 0 && offset(channel) == 0); + if (!strcmp(argv[1], "low32")) assert((uintptr_t)p + 4096 <= UINT64_C(0x100000000)); + *(unsigned *)p = 0xcafef00d; + assert(*(unsigned *)p == 0xcafef00d && !munmap(p, 4096)); + clear(fd); + errno = 0; + p = channel ? mmap64(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 1) + : mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 1); + assert(p == MAP_FAILED && errno == EINVAL); + assert(calls(channel) == 1 && calls(1 - channel) == 0 && offset(channel) == 1); + } + close(fd); + /* Regular-file forwarding must preserve an mmap64 offset above 4 GiB. */ + fd = memfd_create("preload-chain-test", 0); + const off64_t large_offset = INT64_C(0x200001000); + assert(fd >= 0 && !ftruncate64(fd, large_offset + 4096)); + unsigned value = 0x1234abcd; + assert(pwrite64(fd, &value, sizeof(value), large_offset) == (ssize_t)sizeof(value)); + clear(fd); + unsigned *p = mmap64(NULL, 4096, PROT_READ, MAP_SHARED, fd, large_offset); + assert(p != MAP_FAILED && *p == value); + assert(calls(1) == 1 && calls(0) == 0 && offset(1) == large_offset); + assert(!munmap(p, 4096)); + close(fd); + printf("PASS separate mmap/mmap64 chains (%s), EINVAL and >4GiB offset readback\n", argv[1]); + return 0; +} diff --git a/tools/pixel_reset_diagnostics.py b/tools/pixel_reset_diagnostics.py new file mode 100644 index 000000000..462b9d460 --- /dev/null +++ b/tools/pixel_reset_diagnostics.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""Read-only reboot projection. Never writes raw Android logs or account data. + +This is evidence collection, not a GPU fault injector or a root-cause oracle. +The optional kernel projection reads only SYSTEM_LAST_KMSG after user unlock. +It emits fixed categories and numeric hardware fields, never arbitrary log lines. +""" +import argparse +from datetime import datetime, timezone +import json +import os +from pathlib import Path +import re +import subprocess +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +ENTRY = re.compile(r'^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(SYSTEM_LAST_KMSG)\b', re.M) +CATEGORIES = ('GNOC', 'PVRSRV', 'watchdog', 'IOMMU', 'S2MPU', 'access control', 'Kernel panic', 'HWR', 'GPU fault') +PRIMARY_USER_STATES = frozenset(('RUNNING_LOCKED', 'RUNNING_UNLOCKING', 'RUNNING_UNLOCKED')) +# Exact canonical reasons plus the vendor reason observed in the bounded Pixel +# tests. Unrecognized/custom suffixes are not diagnostic output categories. +BOOT_REASONS = frozenset(('cold', 'hard', 'warm', 'reboot', 'shutdown', + 'recovery', 'bootloader', 'kernel_panic', 'watchdog', + 'reset,warm,gnoc_error')) + + +def primary_state(text): + match = re.search(r'Started users state: \[0=([A-Z_]+)(?=[,\]])', text) + return match.group(1) if match and match.group(1) in PRIMARY_USER_STATES else 'unknown' + + +def safe_reason(text): + value = text.strip() + return value if value in BOOT_REASONS else 'unknown' + + +def kernel_projection(text, device_timezone, after=None): + entries = list(ENTRY.finditer(text)) + result = {'metadata_dates': [m.group(1) for m in entries], 'raw_body_saved': False, + 'root_cause_proven': False} + if not entries: + result['status'] = 'no_accessible_entry' + return result + latest = max(enumerate(entries), key=lambda row: row[1].group(1)) + index, match = latest + body = text[match.end():entries[index + 1].start() if index + 1 < len(entries) else len(text)] + result['entry_date_device_local'] = match.group(1) + try: + naive = datetime.strptime(match.group(1), '%Y-%m-%d %H:%M:%S') + zone = ZoneInfo(device_timezone) + except (ValueError, ZoneInfoNotFoundError): + result['status'] = 'unknown_device_timezone' + return result + # Dropbox supplies a local wall clock without an offset. Round-trip both + # folds: a repeated hour has two possible instants; a skipped hour has none. + instants = {local.astimezone(timezone.utc) + for fold in (0, 1) + for local in (naive.replace(tzinfo=zone, fold=fold),) + if local.astimezone(timezone.utc).astimezone(zone).replace(tzinfo=None) == naive} + if len(instants) != 1: + result['status'] = ('ambiguous_device_local_time' if instants + else 'nonexistent_device_local_time') + return result + local = instants.pop() + result['entry_date_utc'] = local.isoformat() + if after is not None and local < after: + result['status'] = 'stale_entry_not_attributed_to_current_trial' + return result + result['status'] = 'projected_entry_not_a_root_cause' + result['categories'] = {key: len(re.findall(re.escape(key), body, re.I)) for key in CATEGORIES} + patterns = { + 'gnoc_addresses': r'Error from GNOC at addr (0x[0-9a-fA-F]+)', + 'gnoc_error_types': r'GNOC error type (0x[0-9a-fA-F]+)', + 'firmware_reboot_codes': r'Reboot reason:\s+(0x[0-9a-fA-F]+)', + 'access_control_addresses': r'Access Control Error \((0x[0-9a-fA-F]+)', + } + result['hardware_fields'] = {key: sorted(set(re.findall(pattern, body))) for key, pattern in patterns.items()} + # Read the complete value, not just a recognized-looking prefix. Apply the + # same fixed projection as getprop; no arbitrary kernel reason text escapes. + reasons = re.findall(r'Last boot reason:[ \t]*([^\r\n]*)', body) + result['last_boot_reasons'] = sorted({safe_reason(reason) for reason in reasons}) + return result + + +def collect(adb, serial, include_kernel=False, after=None): + def shell(*args): + try: + process = subprocess.run([adb, '-s', serial, 'shell', *args], capture_output=True, + text=True, timeout=20) + except (OSError, subprocess.TimeoutExpired): + return None, '' + # No command stderr is copied to output: it may contain unrelated data. + return process.returncode, process.stdout if process.returncode == 0 else '' + + result = {'at': datetime.now(timezone.utc).isoformat(), 'read_only': True, + 'raw_logs_saved': False, 'credentials_collected': False} + code, value = shell('settings', 'get', 'global', 'boot_count') + result['boot_count'] = int(value.strip()) if code == 0 and value.strip().isdigit() else None + code, value = shell('getprop', 'ro.boot.bootreason') + result['bootloader_reason'] = safe_reason(value) + code, value = shell('getprop', 'sys.boot.reason') + result['system_boot_reason'] = safe_reason(value) + code, value = shell('dumpsys', 'user') + result['primary_user_state'] = primary_state(value) + code, value = shell('getprop', 'persist.sys.timezone') + device_timezone = value.strip() + try: + ZoneInfo(device_timezone) + result['device_timezone'] = device_timezone + except (ValueError, ZoneInfoNotFoundError): + result['device_timezone'] = 'unknown' + code, value = shell('dumpsys', 'dropbox', 'SYSTEM_LAST_KMSG') + result['kernel_metadata_dates'] = [m.group(1) for m in ENTRY.finditer(value)] + if include_kernel: + if result['primary_user_state'] != 'RUNNING_UNLOCKED': + result['kernel'] = {'status': 'requires_user_unlock', 'raw_body_saved': False} + else: + code, value = shell('dumpsys', 'dropbox', '--print', 'SYSTEM_LAST_KMSG') + result['kernel'] = kernel_projection(value, result['device_timezone'], after) + result['kernel']['read_exit_code'] = code + return result + + +def write_private_output(path, output): + # Exclusive creation also rejects existing symlinks. Set the private mode + # before any bytes are written, rather than chmod after exposing the data. + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_NOFOLLOW', 0) + descriptor = os.open(path, flags, 0o600) + with os.fdopen(descriptor, 'w', encoding='utf-8') as stream: + stream.write(output) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--adb', default='adb') + parser.add_argument('--serial', required=True) + parser.add_argument('--kernel', action='store_true') + parser.add_argument('--after', help='Earliest relevant UTC ISO timestamp; excludes stale kernel entries') + parser.add_argument('--output', type=Path, help='New JSON file; refuses to overwrite') + args = parser.parse_args() + after = datetime.fromisoformat(args.after.replace('Z', '+00:00')) if args.after else None + if after is not None and after.tzinfo is None: + parser.error('--after must include a timezone') + result = collect(args.adb, args.serial, args.kernel, after) + output = json.dumps(result, indent=2) + '\n' + if args.output: + write_private_output(args.output, output) + print(output, end='') + + +if __name__ == '__main__': + main() diff --git a/tools/tests/test_dxvk_build.py b/tools/tests/test_dxvk_build.py new file mode 100644 index 000000000..0038d0dc9 --- /dev/null +++ b/tools/tests/test_dxvk_build.py @@ -0,0 +1,121 @@ +"""Exercise output isolation with a synthetic toolchain; no network/DLL build.""" +import hashlib +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] + +# Substitute only external source/compiler/archive commands. The real build.sh +# owns output checks, architecture iteration and manifest inputs. +FAKE_TOOL = r''' +import hashlib, json, os +from pathlib import Path +import sys +name = Path(sys.argv[0]).name +args = sys.argv[1:] +with open(os.environ['TOOL_CALLS'], 'a') as stream: + stream.write(name + '\n') +if name == 'git': + if args[0] == 'init': + Path(args[-1]).mkdir() + (Path(args[-1]) / 'LICENSE').write_text('synthetic public license') + elif 'rev-parse' in args: + print(os.environ['TEST_REVISION']) +elif name == 'meson': + if args[0] == 'setup': + p = Path(args[1]); p.mkdir() + p.joinpath('config.json').write_text(json.dumps({ + 'out': args[args.index('--prefix') + 1], + 'arch': args[args.index('--bindir') + 1]})) + elif args[0] == 'install': + config = json.loads(Path(args[2], 'config.json').read_text()) + out = Path(config['out'], config['arch']); out.mkdir(exist_ok=True) + out.joinpath('current.dll').write_bytes(b'synthetic current pinned DLL') + if config['arch'] == os.environ.get('FAIL_INSTALL_ARCH'): + sys.exit(5) +elif name == 'tar': + Path(args[args.index('-cf') + 1]).write_bytes(b'synthetic source archive') +elif name == 'sha256sum': + for arg in args: + print(hashlib.sha256(Path(arg).read_bytes()).hexdigest() + ' ' + arg) +elif name != 'python3': + raise AssertionError(name) +''' + + +class DxvkBuildTest(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.base = Path(self.temporary.name) + self.bin = self.base / 'bin'; self.bin.mkdir() + executable = self.bin / 'fake-tool' + executable.write_text('#!' + sys.executable + '\n' + FAKE_TOOL) + executable.chmod(0o755) + for name in ('git', 'meson', 'tar', 'sha256sum', 'python3'): + (self.bin / name).symlink_to(executable) + self.calls = self.base / 'calls' + self.env = dict(os.environ, PATH=str(self.bin) + os.pathsep + os.environ['PATH'], + TOOL_CALLS=str(self.calls)) + source_env = ROOT / 'tools/dxvk/source.env' + self.env['TEST_REVISION'] = next(line.split('=', 1)[1].strip('"\'') + for line in source_env.read_text().splitlines() + if line.startswith('DXVK_REVISION=')) + self.output = self.base / 'component' + + def build(self): + return subprocess.run(['bash', str(ROOT / 'tools/dxvk/build.sh'), str(self.output)], + env=self.env, capture_output=True, text=True, timeout=10) + + def test_stale_dll_is_rejected_untouched_before_fetch(self): + stale = self.output / 'x32/stale.dll'; stale.parent.mkdir(parents=True) + stale.write_bytes(b'prior pinned build') + result = self.build() + self.assertNotEqual(result.returncode, 0) + self.assertIn('output directory must be empty', result.stderr) + self.assertEqual(stale.read_bytes(), b'prior pinned build') + self.assertFalse(self.calls.exists()) + self.assertFalse((self.output / 'SHA256SUMS').exists()) + + def test_hidden_contents_are_also_rejected(self): + self.output.mkdir(); (self.output / '.previous').write_text('retained') + self.assertNotEqual(self.build().returncode, 0) + self.assertFalse(self.calls.exists()) + self.assertEqual((self.output / '.previous').read_text(), 'retained') + + def test_empty_or_new_output_contains_only_current_dlls(self): + for existing in (False, True): + with self.subTest(existing=existing): + self.output = self.base / ('existing' if existing else 'new') + if existing: self.output.mkdir() + result = self.build() + self.assertEqual(result.returncode, 0, result.stderr) + rows = (self.output / 'SHA256SUMS').read_text().splitlines() + self.assertEqual([row.split(' ')[1] for row in rows], + ['x32/current.dll', 'x64/current.dll']) + for row in rows: + digest, path = row.split(' ') + self.assertEqual(digest, hashlib.sha256((self.output / path).read_bytes()).hexdigest()) + + def test_partial_build_is_not_reused_or_deleted(self): + self.env['FAIL_INSTALL_ARCH'] = 'x64' + result = self.build() + self.assertEqual(result.returncode, 5, result.stderr) + before = {str(p.relative_to(self.output)): p.read_bytes() + for p in self.output.rglob('*') if p.is_file()} + self.calls.unlink(); del self.env['FAIL_INSTALL_ARCH'] + result = self.build() + self.assertNotEqual(result.returncode, 0) + self.assertFalse(self.calls.exists()) + after = {str(p.relative_to(self.output)): p.read_bytes() + for p in self.output.rglob('*') if p.is_file()} + self.assertEqual(after, before) + + +if __name__ == '__main__': + unittest.main() diff --git a/tools/tests/test_pixel_reset_diagnostics.py b/tools/tests/test_pixel_reset_diagnostics.py new file mode 100644 index 000000000..8cb5ba1a0 --- /dev/null +++ b/tools/tests/test_pixel_reset_diagnostics.py @@ -0,0 +1,215 @@ +"""Test freshness, narrow output, and lock handling without accessing a device.""" +from datetime import datetime, timezone +import importlib.util +import json +import io +import os +from pathlib import Path +import stat +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +spec = importlib.util.spec_from_file_location('reset_projection', Path(__file__).resolve().parents[1] / 'pixel_reset_diagnostics.py') +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class ResetProjectionTest(unittest.TestCase): + def test_only_primary_state(self): + self.assertEqual(module.primary_state('Private space secret\nStarted users state: [0=RUNNING_UNLOCKED, 10=STOPPED]'), 'RUNNING_UNLOCKED') + self.assertEqual(module.primary_state('State: RUNNING_UNLOCKED'), 'unknown') + + def test_stale_body_not_used(self): + raw = '2026-10-10 20:10:09 SYSTEM_LAST_KMSG (text, 1024 bytes)\nGNOC error type 0x2000' + result = module.kernel_projection(raw, 'America/Toronto', datetime(2026, 10, 11, 21, tzinfo=timezone.utc)) + self.assertEqual(result['status'], 'stale_entry_not_attributed_to_current_trial') + self.assertNotIn('categories', result) + + def test_repeated_dst_hour_is_ambiguous_before_freshness(self): + raw = '2026-11-01 01:30:00 SYSTEM_LAST_KMSG\nGNOC error type 0x2000' + for after in (None, datetime(2026, 11, 1, 6, tzinfo=timezone.utc)): + with self.subTest(after=after): + result = module.kernel_projection(raw, 'America/Toronto', after) + self.assertEqual(result['status'], 'ambiguous_device_local_time') + self.assertNotIn('entry_date_utc', result) + self.assertNotIn('categories', result) + self.assertNotIn('hardware_fields', result) + + def test_skipped_dst_hour_is_not_a_real_instant(self): + raw = '2026-03-08 02:30:00 SYSTEM_LAST_KMSG\nGNOC error type 0x2000' + result = module.kernel_projection(raw, 'America/Toronto') + self.assertEqual(result['status'], 'nonexistent_device_local_time') + self.assertNotIn('entry_date_utc', result) + self.assertNotIn('categories', result) + + def test_unambiguous_times_keep_correct_offsets_and_cutoff(self): + for stamp, expected in (('2026-11-01 00:30:00', '2026-11-01T04:30:00+00:00'), + ('2026-11-01 02:30:00', '2026-11-01T07:30:00+00:00'), + ('2026-03-08 03:30:00', '2026-03-08T07:30:00+00:00')): + with self.subTest(stamp=stamp): + raw = stamp + ' SYSTEM_LAST_KMSG\nGNOC' + result = module.kernel_projection(raw, 'America/Toronto') + self.assertEqual(result['entry_date_utc'], expected) + self.assertEqual(result['status'], 'projected_entry_not_a_root_cause') + cutoff = datetime.fromisoformat(expected) + self.assertEqual(module.kernel_projection(raw, 'America/Toronto', cutoff)['status'], + 'projected_entry_not_a_root_cause') + + def test_latest_entry_and_no_arbitrary_body(self): + raw = ('2026-10-10 20:10:09 SYSTEM_LAST_KMSG (text)\nGNOC error type 0x1111\n' + '2026-10-11 17:35:01 SYSTEM_LAST_KMSG (text)\n' + 'PRIVATE_TEST_VALUE=a_secret_that_must_not_be_emitted\n' + 'Error from GNOC at addr 0x28220000\nGNOC error type 0x2000\n' + 'Reboot reason: 0xc005 - Secret arbitrary description\nLast boot reason: reset,warm,gnoc_error\n') + result = module.kernel_projection(raw, 'America/Toronto', datetime(2026, 10, 11, 21, tzinfo=timezone.utc)) + self.assertEqual(result['entry_date_utc'], '2026-10-11T21:35:01+00:00') + self.assertEqual(result['hardware_fields']['gnoc_error_types'], ['0x2000']) + self.assertNotIn('0x1111', json.dumps(result)) + self.assertNotIn('a_secret', json.dumps(result)) + self.assertNotIn('arbitrary description', json.dumps(result)) + self.assertFalse(result['root_cause_proven']) + + def test_missing_entry_and_timezone(self): + self.assertEqual(module.kernel_projection('Permission denied', 'America/Toronto')['status'], 'no_accessible_entry') + self.assertEqual(module.kernel_projection('2026-10-11 17:00:00 SYSTEM_LAST_KMSG\nGNOC', 'Unknown/Region')['status'], 'unknown_device_timezone') + + def test_reason_rejects_paths_and_urls(self): + self.assertEqual(module.safe_reason('reset,warm,gnoc_error\n'), 'reset,warm,gnoc_error') + self.assertEqual(module.safe_reason('https://private.invalid/login?secret=abc'), 'unknown') + self.assertEqual(module.safe_reason('/data/private'), 'unknown') + + def test_reason_only_recognized_exact_values(self): + for reason in ('cold', 'hard', 'warm', 'reboot', 'shutdown', 'recovery', + 'bootloader', 'kernel_panic', 'watchdog', 'reset,warm,gnoc_error'): + with self.subTest(reason=reason): + self.assertEqual(module.safe_reason(reason + '\n'), reason) + for reason in ('PRIVATE TEST VALUE', 'private_test_value', '123456', + 'reboot,private_test_value', 'reset,warm,gnoc_error,private_test_value', + 'reset,warm,gnoc_error trailing_private_test_value', ''): + with self.subTest(reason=reason): + self.assertEqual(module.safe_reason(reason), 'unknown') + + def test_kernel_reason_uses_same_fixed_projection(self): + raw = ('2026-10-11 17:35:01 SYSTEM_LAST_KMSG (text)\n' + 'Last boot reason: private_test_value\n' + 'Last boot reason: reset,warm,gnoc_error trailing_private_test_value\n' + 'Last boot reason: reset,warm,gnoc_error\n' + 'Last boot reason:\nPRIVATE TEST VALUE\n') + result = module.kernel_projection(raw, 'America/Toronto') + self.assertEqual(result['last_boot_reasons'], ['reset,warm,gnoc_error', 'unknown']) + self.assertNotIn('private_test_value', json.dumps(result)) + self.assertNotIn('PRIVATE TEST VALUE', json.dumps(result)) + + def test_primary_state_uses_fixed_categories(self): + for state in ('RUNNING_LOCKED', 'RUNNING_UNLOCKING', 'RUNNING_UNLOCKED'): + self.assertEqual(module.primary_state('Started users state: [0=' + state + ']'), state) + for state in ('RUNNING_PRIVATE_TEST_VALUE', 'RUNNING_UNLOCKED_PRIVATE', + 'RUNNING_UNLOCKED=private_test_value'): + self.assertEqual(module.primary_state('Started users state: [0=' + state + ']'), 'unknown') + + def test_property_reason_never_exports_custom_text(self): + def run(argv, **kwargs): + if argv[-2:] == ['getprop', 'ro.boot.bootreason']: + text = 'PRIVATE TEST VALUE' + elif argv[-2:] == ['getprop', 'sys.boot.reason']: + text = 'reset,warm,gnoc_error,private_test_value' + elif argv[-1] == 'persist.sys.timezone': + text = 'America/Toronto' + else: + text = '' + return subprocess.CompletedProcess(argv, 0, text, '') + with patch.object(module.subprocess, 'run', side_effect=run): + result = module.collect('adb', 'test-device') + self.assertEqual(result['bootloader_reason'], 'unknown') + self.assertEqual(result['system_boot_reason'], 'unknown') + self.assertNotIn('PRIVATE TEST VALUE', json.dumps(result)) + self.assertNotIn('private_test_value', json.dumps(result)) + + @unittest.skipUnless(os.name == 'posix', 'POSIX private file modes') + def test_cli_output_is_private_before_writing(self): + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / 'diagnostics.json' + saved_umask = os.umask(0) + original_fdopen = os.fdopen + original_path_open = Path.open + creation_modes = [] + def inspect_stream(stream): + try: + creation_modes.append(stat.S_IMODE(os.fstat(stream.fileno()).st_mode)) + self.assertEqual(os.fstat(stream.fileno()).st_size, 0) + self.assertEqual(creation_modes[-1], 0o600) + except BaseException: + stream.close() + raise + return stream + def inspect_fd(fd, *args, **kwargs): + return inspect_stream(original_fdopen(fd, *args, **kwargs)) + def inspect_path(path, *args, **kwargs): + stream = original_path_open(path, *args, **kwargs) + return inspect_stream(stream) if path == output else stream + try: + with patch.object(sys, 'argv', ['diagnostics', '--serial', 'test-device', + '--output', str(output)]), \ + patch.object(module, 'collect', return_value={'boot_count': 12}), \ + patch.object(os, 'fdopen', side_effect=inspect_fd), \ + patch.object(Path, 'open', new=inspect_path), \ + patch('sys.stdout', new_callable=io.StringIO): + module.main() + finally: + os.umask(saved_umask) + self.assertEqual(creation_modes, [0o600]) + self.assertEqual(stat.S_IMODE(output.stat().st_mode), 0o600) + self.assertEqual(json.loads(output.read_text()), {'boot_count': 12}) + + def test_output_refuses_existing_file(self): + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / 'diagnostics.json' + output.write_text('existing controlled test data') + with self.assertRaises(FileExistsError): + module.write_private_output(output, '{"replacement":true}\n') + self.assertEqual(output.read_text(), 'existing controlled test data') + + @unittest.skipUnless(os.name == 'posix', 'POSIX symlink behavior') + def test_output_refuses_symlink_without_touching_target(self): + with tempfile.TemporaryDirectory() as directory: + target = Path(directory) / 'target.json' + target.write_text('existing controlled test data') + output = Path(directory) / 'diagnostics.json' + output.symlink_to(target) + with self.assertRaises(FileExistsError): + module.write_private_output(output, '{"replacement":true}\n') + self.assertTrue(output.is_symlink()) + self.assertEqual(target.read_text(), 'existing controlled test data') + + def test_locked_skips_kernel_body(self): + commands = [] + def run(argv, **kwargs): + commands.append(argv) + if argv[-2:] == ['dumpsys', 'user']: + text = 'Started users state: [0=RUNNING_LOCKED]' + elif argv[-1] == 'persist.sys.timezone': + text = 'America/Toronto' + elif argv[-3:] == ['get', 'global', 'boot_count']: + text = '12' + else: + text = '' + return subprocess.CompletedProcess(argv, 0, text, 'PRIVATE_STDERR') + with patch.object(module.subprocess, 'run', side_effect=run): + result = module.collect('adb', 'test-device', True) + self.assertEqual(result['kernel']['status'], 'requires_user_unlock') + self.assertFalse(any('--print' in command for command in commands)) + self.assertNotIn('PRIVATE_STDERR', json.dumps(result)) + + def test_command_failure_does_not_export_stderr(self): + with patch.object(module.subprocess, 'run', return_value=subprocess.CompletedProcess([], 1, 'UNRELATED_DATA', 'PRIVATE_STDERR')): + result = module.collect('adb', 'test-device', True) + self.assertIsNone(result['boot_count']) + self.assertNotIn('UNRELATED_DATA', json.dumps(result)) + self.assertNotIn('PRIVATE_STDERR', json.dumps(result)) + + +if __name__ == '__main__': + unittest.main()