Shader translator: Maxwell results where SPIR-V leaves bitfields, funnel shifts and unwritten registers undefined

- BFE / BFI (patch 0012): a field past bit 31 is cut there, one that starts past it extracts 0 (signed: bit 31
  repeated) and inserts nothing, as PTX bfe / bfi. Clamped only where the field is not a constant inside the word.
  1.4.3's Depths terrain vertex shaders (42 variants, 134 extracts) compute the position as ((2y + x) * 8) & 0xff,
  up to 248; AMD's bitfield instructions take it modulo 32. UltraTears saw the Depths walls and ground missing on
  RADV (Steam Deck, RX 570).
- SHF by 0 (patch 0013) gives the unshifted word instead of shifting the other one by 32. Every register SHF of
  1.4.3 pairs its word with 0, so 1.4.3 results do not change.
- Function-local variables start at 0 (patch 0014): a register read on a path that never wrote it is no longer an
  undefined value Mesa may fold.

All 12,454 1.4.3 shaders translate and pass spirv-val. RX 9070, Depths save: the same picture before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/maxwell.md

@@ -86,6 +86,9 @@ Source shorthands:
 | 3.7 | **Integer division** does not exist (compiled to `XMAD`/`IMAD` sequences); MUFU.RCP(0) = +Inf, LG2(0) = -Inf. | Same IEEE edge cases through `1/x`, `Log2`. | Low risk. | — |
 | 3.8 | **FP64** in some TotK cutscene shaders. | No FP64 on some hosts. | yuzu FP64 → FP32 (YZ-2306, PR 10693); ours: `softfloat64` job flag (`tools/xlate/Program.cs` `QueryHostSupportsShaderFloat64`). | — |
 | 3.9 | **Integer vertex inputs read as raw bits** (bone indices `R8UI`). | Float-declared inputs convert integers. | NXN Mario: stretched characters; re-translated with `uint` inputs. Ours: attribute types passed to `QueryGraphicsState` (`Program.cs`). | NXN |
+| 3.10 | **BFE / BFI fields past bit 31** (as PTX `bfe` / `bfi`): a field that runs past bit 31 is cut there; one that starts past it extracts 0 (signed: bit 31 repeated) and inserts nothing. TotK 1.4.3's Depths terrain vertex shaders (42 variants) take the position from a register, `((2y + x) * 8) & 0xff` with 5-bit x and y, so up to 248. | `OpBitField*Extract` / `OpBitFieldInsert` undefined when offset or offset + count exceeds 32; AMD's bitfield instructions take offset and width modulo 32. | Ryujinx passes the fields through. UltraTears saw the Depths walls and ground missing on the Steam Deck and a Linux RX 570 (RADV). Ours: clamped where the field is not a constant inside the word (translator patch 0012). RX 9070 on Windows drew the Depths the same before and after. | PTX ISA `bfe`, `bfi` |
+| 3.11 | **SHF by 0** gives the unshifted word (C for `SHF.L`, A for `SHF.R`). | Ryujinx's funnel shifts the other word by 32: undefined in SPIR-V; AMD shifts take the low 5 bits and give A \| C. | yuzu fixed Depths terrain gaps in its funnel shift (PR 10459). Ours: a select on a zero amount (translator patch 0013). In 1.4.3 every register SHF pairs its word with 0, so the result was right already. | — |
+| 3.12 | **Registers read before any write** on a path (paths the structurizer adds included) have no defined value in the game's code; UltraTears reports 0 from the NVIDIA driver. | An `OpVariable` without an initializer loads an undefined value; Mesa folds undefined values freely. | Ours: every function-local variable starts at 0 (`OpConstantNull`, translator patch 0014). | — |
 
 ## 4. Memory model
 

tools/build_xlate.py

@@ -1,7 +1,7 @@
 """Build the shader translator: build/xlate/totk_xlate.exe (Ryujinx 1.3.2 + patches, NativeAOT, SPIR-V for Vulkan).
 
 Copies tools/vendor/ryujinx to build/xlate/ryujinx, applies tools/xlate/patches/*.patch there (tools/vendor stays
-byte-identical; 0001-0004 are MNG's, 0005-0011 are ours), then runs `dotnet publish`. Needs the .NET 10 SDK, the MSVC build
+byte-identical; 0001-0004 are MNG's, 0005-0014 are ours), then runs `dotnet publish`. Needs the .NET 10 SDK, the MSVC build
 tools and vswhere.exe (docs/building.md). Run with .venv. On Linux it builds build/xlate/totk_xlate for linux-x64, which
 NativeAOT links with the system's clang (no MSVC); NativeAOT builds for the OS it runs on only.
 
@@ -99,8 +99,8 @@ def prepare_sources() -> Path:
     for junk in list(src.glob("src/*/obj")) + list(src.glob("src/*/bin")):
         shutil.rmtree(junk)
     patches = sorted((XLATE / "patches").glob("*.patch"))
-    if len(patches) != 11:
-        sys.exit(f"expected 11 patches, found {len(patches)}")
+    if len(patches) != 14:
+        sys.exit(f"expected 14 patches, found {len(patches)}")
     for patch in patches:
         # git apply works outside a repository when given a directory; --check first so a failure names the patch.
         for extra in (["--check"], []):

tools/xlate/patches/0012-bitfield-fields-past-bit-31.patch

@@ -0,0 +1,64 @@
+--- a/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBitfield.cs
++++ b/src/Ryujinx.Graphics.Shader/Instructions/InstEmitBitfield.cs
+@@ -142,13 +142,46 @@ namespace Ryujinx.Graphics.Shader.Instructions
+ 
+             Operand size = context.BitfieldExtractU32(srcB, Const(8), Const(8));
+ 
++            // TOTK: Maxwell's BFE (as PTX bfe) cuts a field that runs past bit 31 there, and a field that starts past it
++            // gives 0, or bit 31 repeated when signed and the size is not 0. SPIR-V leaves an offset or offset + count over
++            // 32 undefined, and AMD's bitfield instructions take offset and width modulo 32. TOTK's Depths terrain vertex
++            // shaders take position and size from a register (BFE.R), so they are clamped into the defined range here;
++            // constant fields inside the word are emitted as before.
++            bool constantInRange = srcB.Type == OperandType.Constant && (srcB.Value & 0xff) + ((srcB.Value >> 8) & 0xff) <= 32;
++            Operand positionIn = position;
++            Operand sizeIn = size;
++
++            if (!constantInRange)
++            {
++                ClampBitfield(context, ref position, ref size);
++            }
++
+             Operand res = isSigned
+                 ? context.BitfieldExtractS32(srcA, position, size)
+                 : context.BitfieldExtractU32(srcA, position, size);
+ 
++            if (!constantInRange && isSigned)
++            {
++                // The clamped extract gave 0 for a field that starts past bit 31; a signed one is bit 31 repeated.
++                Operand signBits = context.ConditionalSelect(context.ICompareEqual(sizeIn, Const(0)), Const(0), context.ShiftRightS32(srcA, Const(31)));
++
++                res = context.ConditionalSelect(context.ICompareGreaterOrEqualUnsigned(positionIn, Const(32)), signBits, res);
++            }
++
+             context.Copy(GetDest(rd), res);
+ 
+-            // TODO: CC, X, corner cases.
++            // TODO: CC, X.
++        }
++
++        // TOTK: position and size of a bitfield cut to the 32-bit word: the size shrinks to what is left of the word
++        // after the position (0 for a position of 32 or more) and the position stays at most 31, so offset and
++        // offset + count stay inside the range SPIR-V defines.
++        private static void ClampBitfield(EmitterContext context, ref Operand position, ref Operand size)
++        {
++            Operand left = context.ISubtract(Const(32), context.IMinimumU32(position, Const(32)));
++
++            size = context.IMinimumU32(size, left);
++            position = context.IMinimumU32(position, Const(31));
+         }
+ 
+         private static void EmitBfi(EmitterContext context, Operand srcA, Operand srcB, Operand srcC, int rd)
+@@ -157,6 +190,13 @@ namespace Ryujinx.Graphics.Shader.Instructions
+ 
+             Operand size = context.BitfieldExtractU32(srcB, Const(8), Const(8));
+ 
++            // TOTK: as in EmitBfe, a field past bit 31 is cut there; one that starts past it inserts nothing (size 0
++            // after the clamp), so C comes back unchanged, as on Maxwell.
++            if (!(srcB.Type == OperandType.Constant && (srcB.Value & 0xff) + ((srcB.Value >> 8) & 0xff) <= 32))
++            {
++                ClampBitfield(context, ref position, ref size);
++            }
++
+             Operand res = context.BitfieldInsert(srcC, srcA, position, size);
+ 
+             context.Copy(GetDest(rd), res);

tools/xlate/patches/0013-funnel-shift-by-zero.patch

@@ -0,0 +1,32 @@
+--- a/src/Ryujinx.Graphics.Shader/Instructions/InstEmitShift.cs
++++ b/src/Ryujinx.Graphics.Shader/Instructions/InstEmitShift.cs
+@@ -119,6 +119,10 @@ namespace Ryujinx.Graphics.Shader.Instructions
+             bool signedShift = maxShift == MaxShift.S64;
+             int maxShiftConst = isLongShift ? 64 : 32;
+ 
++            // TOTK: the shift amount as a constant, when it is one (taken before the mask makes it an operation).
++            bool constantShift = srcB.Type == OperandType.Constant;
++            int shiftValue = constantShift ? (mask ? srcB.Value & (maxShiftConst - 1) : srcB.Value) : 0;
++
+             if (mask)
+             {
+                 srcB = context.BitwiseAnd(srcB, Const(maxShiftConst - 1));
+@@ -159,6 +163,18 @@ namespace Ryujinx.Graphics.Shader.Instructions
+                 }
+             }
+ 
++            // TOTK: a shift by 0 shifts the other word by 32, which SPIR-V leaves undefined; AMD's shift instructions take
++            // the amount's low 5 bits and give A | C. Maxwell gives the unshifted word: C for SHF.L, A for SHF.R, 32- and
++            // 64-bit alike. TOTK's Depths terrain vertex shaders shift by registers (SHF.L.R, SHF.R.R).
++            if (!constantShift)
++            {
++                res = context.ConditionalSelect(context.ICompareEqual(srcB, Const(0)), left ? srcC : srcA, res);
++            }
++            else if (shiftValue == 0)
++            {
++                res = left ? srcC : srcA;
++            }
++
+             if (!mask)
+             {
+                 // Clamped shift value.

tools/xlate/patches/0014-zero-initialised-locals.patch

@@ -0,0 +1,18 @@
+--- a/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Declarations.cs
++++ b/src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Declarations.cs
+@@ -33,8 +33,13 @@ namespace Ryujinx.Graphics.Shader.CodeGen.Spirv
+         {
+             foreach (AstOperand local in function.Locals)
+             {
+-                SpvInstruction localPointerType = context.TypePointer(StorageClass.Function, context.GetType(local.VarType));
+-                SpvInstruction spvLocal = context.Variable(localPointerType, StorageClass.Function);
++                SpvInstruction localType = context.GetType(local.VarType);
++                SpvInstruction localPointerType = context.TypePointer(StorageClass.Function, localType);
++
++                // TOTK: a register read on a path that did not write it reads 0. Without an initializer the load is
++                // undefined, and a compiler that folds undefined values (Mesa's, on the Steam Deck) may compute anything
++                // from it.
++                SpvInstruction spvLocal = context.Variable(localPointerType, StorageClass.Function, context.ConstantNull(localType));
+ 
+                 context.AddLocalVariable(spvLocal);
+                 context.DeclareLocal(local, spvLocal);