DroidDeck/docs/cardboard-vr.md

# Cardboard VR (experimental)

Cardboard mode turns the session screen into a phone VR viewer (Google Cardboard and similar): the
panel is split into a left and a right eye, each drawn through a barrel distortion that the viewer's
lenses undo. Two sources can feed it:

- **Virtual screen** (any game, Steam, the desktop): the session hangs as a flat screen in front of
  you, with real stereo depth from the eye distance. With head tracking on, the phone's rotation
  sensor keeps the screen fixed in the room while your head turns (3DoF).
- **Side-by-side 3D**: the frame already holds a left and a right image. This is what SteamVR draws
  here (below), and what SBS 3D video or a stereo shader like SuperDepth3D produces. Only the lens is
  added.

The settings live in the session drawer, **Display > Cardboard VR**: mode, picture, viewer (lens
profile), eye distance, screen size and distance, head tracking and recenter, and SteamVR. Everything
applies live. `tools/droiddeckctl prefs set session cardboard '{"enabled":true,...}'` drives the same
settings (`gpu/Cardboard.kt` has the keys).

## How the picture is made

```
game / Steam -> gamescope -> DroidDeck compositor -> effects -> frame generation
             -> Cardboard pass (cardboard.frag) -> 1:1 blit -> panel
```

`app/src/main/cpp/waylandcomp/src/cardboard.c` is the last stage of the compositor pass. It forces the
copy path (no zero-copy layer: the pass draws the whole panel) and renders `cardboard.frag` into a
panel-sized target:

- each output pixel picks its eye, finds its offset from that eye's lens centre in tangent space, and
  applies the viewer's distortion polynomial `r(1 + k1 r² + k2 r⁴)` (sampling outwards draws the image
  barrel-shaped; the lens pincushions it back), optionally per colour channel for chromatic
  aberration;
- **virtual screen**: the ray from the eye (offset by half the IPD), rotated by the head pose,
  meets a plane `screenDistance` away; outside it is a dark backdrop;
- **side-by-side**: the eye samples its own half of the frame over the same field of view.

The lens centres sit inset towards the nose so they line up with the viewer's lenses on the panel
actually used (the panel's physical size from DisplayMetrics). The field of view is the panel's height
seen from the lens's focal distance (`Cardboard.fovDeg`). Viewer profiles are in `Cardboard.Viewer`.

Head tracking (`gpu/HeadTracker.kt`) reads the game rotation vector (gyro and accelerometer, no compass)
on a thread of its own and hands a row-major head-to-world matrix straight to native code. Recenter
resets the heading only, so the horizon stays level. When the view is more than 60° from level (lying
back, or the device flat on a table) it takes the whole orientation as straight ahead. A new pose
redraws even a static scene (`vkp_cardboard_pose_changed` on the vsync tick).

The pose is also published to the guest: a 128-byte seqlocked file, `headpose_shm.h`, at
`/run/droiddeck/droiddeck-headpose`. It carries the rotation, the IPD, and the frustum the side-by-side
pass expects of a source (`tan_half_fov`, `eye_aspect`, eye size), so a renderer in the guest can match
the lens exactly.

## SteamVR

Valve's ARM64 SteamVR, the build the Steam Frame runs, comes from Valve's public SteamOS package
server (`deckard-steamvr`, pinned by version and SHA-256 in `droiddeck-steamvr`; the same hash
[steam-frame-nixos](https://github.com/y0usaf/steam-frame-nixos) pins). `droiddeck-steamvr install`
unpacks it to `/opt/steamvr`. The drawer's **Install** button runs that (571 MB). **Start** then
switches the picture to side-by-side and runs SteamVR with three DroidDeck pieces (`tools/vr`):

1. **`driver_droiddeck.so`**, an OpenVR driver with one HMD:
   - pose: the shared head pose file (3DoF, standing height);
   - projection: exactly the frustum the Cardboard pass samples, with identity distortion (the app does
     the lens). A new frustum mid-run (the glasses' SteamVR field of view) goes out with
     `SetDisplayProjectionRaw` and `VREvent_LensDistortionChanged`;
   - a proximity sensor that reads "worn" while Cardboard mode is on. Without it SteamVR takes a still
     headset (a phone on a table) for an unused one, enters standby after a few seconds, and the
     compositor stops submitting frames;
   - frames: `IVRDriverDirectModeComponent` with a free-running 60 Hz vsync announcer, as ALVR does on
     Linux. SteamVR renders each eye into textures the driver allocates through its IPC resource
     manager (dmabufs);
   - `presenter.cpp` imports those eye textures into a Vulkan device of its own and blits them side
     by side into a fullscreen window on DroidDeck's own compositor (`wayland-0`, the app_id
     `droiddeck-cardboard-vr` keeps it above gamescope's window), sized to the Cardboard panel so
     nothing scales it down. SteamVR hands the textures over with no fence the driver can wait on, so
     the presenter gives the compositor's GPU work a few milliseconds first, as ALVR does.
     `DROIDDECK_VR_PRESENT=x11` uses the older path instead: an X11 window on gamescope's Xwayland marked
     `STEAM_OVERLAY` (the external overlay slot is mangoapp's), at the session's resolution;
2. **`VK_LAYER_DROIDDECK_vr`**, a Vulkan layer, enabled explicitly in SteamVR's processes and implicit
   for the rest of the session (Steam's own web helper shares its in-VR overlay with SteamVR too, and
   crashes without it until Steam quits). Implicitly it does nothing in an ordinary session:
   - the display stand-in below happens only while SteamVR runs (`droiddeck-steamvr`'s `active`
     marker, which a new session clears) or where the layer is enabled by name. Gamescope and games
     asking for display extensions outside SteamVR get them untouched;
   - the semaphore bridge runs in those same processes, and in Steam's web helper once SteamVR is
     installed;
   - with SteamVR not installed, every process gets the driver's functions unchanged.

   What it does in SteamVR's processes:
   - `vrcompositor` always asks for `VK_KHR_display` and keeps a direct-mode window even with a
     driver-direct HMD. Turnip on KGSL refuses to enumerate any GPU when `VK_KHR_display` is enabled
     ("I can't KHR_display"). The layer takes the display extensions off the instance and stands in
     for one display with one mode and one plane, whose surface is a headless one
     (`VK_EXT_headless_surface`, sized from the mode because headless reports none). The vblank side
     of `VK_EXT_display_control` is a 60 Hz clock;
   - SteamVR shares a timeline semaphore with every shared image, exported as an opaque fd, which
     Turnip/KGSL cannot export. The layer creates it unshared and hands out a memfd holding a 64-bit
     counter instead. Every process with the semaphore keeps a real local one, and a poller thread
     carries values both ways (reached locally: up into the counter; seen in the counter: signalled
     locally from the host), so GPU waits work as written, a poll period late. An imported binary
     semaphore cannot be bridged that way, and its import is refused. With the semaphores dropped
     outright (the first version) the dashboard showed torn, half-drawn tiles;
   - Turnip on KGSL does not defer a submit that waits on a timeline value nobody has signalled yet:
     `vkQueueSubmit` blocks until it is. SteamVR's client waits on a compositor value that the
     compositor signals only after the client has submitted and said so, which deadlocked every app
     before its first frame (`helloxr`'s swapchain acquire, DXVK's submit thread in a Proton game).
     A submit's wait on an imported shared value that is neither reached nor queued in that process
     gets 3 ms for the other process to reach it, then is dropped. On a counter that has not moved for
     100 ms it is dropped at once: Angry Birds VR waits on two such values a frame, and the grace cost
     its submit thread 6 ms a frame and held it at 60 fps. Keeping these waits stops a Proton game
     before its first frame (Angry Birds VR stayed on SteamVR's loading card). While a Quest
     session's frame socket (`/run/droiddeck/droiddeck-xr-frames`) exists, every wait is kept, as the
     Quest path was built and tested with them;
   - Steam's web helper has the semaphore bridge whenever SteamVR is installed: its GPU process
     outlives SteamVR runs, and one started without the layer crashed when SteamVR showed Steam's
     overlays, until Steam quit;
   - `DROIDDECK_VR_CRASHTRACE=1` (or a `crashtrace` file beside the `active` marker, for processes like
     Steam whose environment is not ours) prints a backtrace on a fatal signal. `vrcompositor`'s
     launcher drops `LD_PRELOAD`, but the layer is loaded there. It only takes signals nobody handles
     yet: Wine and FEX handle `SIGSEGV` as part of running a Proton game, and a replaced handler
     crashed The Lab;
   - a `semtrace` file beside the marker logs, every 2 s, each bridged semaphore's local and shared
     value and the memfd inode that names it in every process, plus the first waits and signals
     (values up to 3) submitted on them.
3. **`droiddeck-steamvr`**, the guest helper (`status | install | start | stop`), which the app drives
   through the guest agent's fixed `vr` request (no agent commands needed). It also supplies an `xset`
   stand-in, because the dashboard's web helper checks the X server with it. Before every start it
   removes SteamVR's crash marker (`config/vrserver_crash_timestamp.txt`, written when vrserver
   crashes) and clears `driver_droiddeck.blocked_by_safe_mode`: after a crash early in a run SteamVR
   starts in safe mode and blocks every add-on driver, so a single crash left the glasses blank on
   every later launch (`HmdNotFound`). The driver also stops its pose, vsync and presenter threads
   if vrserver exits without deactivating it, since a running `std::thread` destroyed at exit aborts
   the process.

Imported eye textures carry the same format list SteamVR's mutable sRGB images do. With it Turnip keeps
UBWC compression on, and an import that disagreed read compressed data as garbage.

Two more fixes outside `tools/vr` were needed for the dashboard (Steam's library, in VR):

- **SysV semaphores** (`tools/linuxfs/preload/sysv.c`): the session emulates them, and `IPC_STAT`
  reported `sem_otime` 0 forever. SteamVR's web helper does the usual race-free init (wait until
  `sem_otime` is non-zero), so its zygotes slept in a `semctl`/`usleep` loop and the dashboard never
  appeared. `semop` now stamps `sem_otime`, `SETVAL`/`SETALL`/`IPC_SET` stamp `sem_ctime`;
  `test_sysv_preload.py` covers it.
- `vrwebhelper` gets the same tracer treatment as Steam's `steamwebhelper` (`tracer.c`).

Verified on the AYN Thor (SD 8 Gen 2, Turnip): SteamVR 2.17.10 starts with the DroidDeck HMD active,
the compositor runs driver-direct at about 60 fps, the dashboard (Steam's library in VR, with the
controller's laser) shows cleanly in both eyes at the panel's resolution while Steam keeps running, and
a pose written to the shared file turns SteamVR's view. Started from the drawer button through the agent.
Once a headset (VITURE, RayNeo or XREAL glasses) is connected and under DroidDeck's control, the
Steam page shows **Play SteamVR** at the top left, with a **Settings** badge that drops that headset's
settings from the top; before that neither appears. Play SteamVR starts Steam and then SteamVR
(installing it first if needed) once the glasses show a picture, and SteamVR stops when the glasses
are unplugged. **Play Steam** never starts it: while SteamVR runs, Steam keeps its own window in front
of flat games. Through a VITURE Mobile Dock the glasses are only a display: the dock does not pass
their controls or head tracking to the phone, so the Steam page shows neither, and only the session
drawer's glasses page (output and **Game on glasses**) applies.

SteamVR defaults to 2D output: a single eye fills the glasses while head tracking remains active.
Enable **3D (side-by-side)** manually or opt into **3D for VR games** in the glasses controls for stereo. Automatic
switching waits until SteamVR reports that it is running; starting or installing it does not change
the display mode. A saved **3D for VR games** preference is retained.

While SteamVR runs, its side-by-side source takes priority over the glasses' saved picture mode,
including **3DoF virtual screen**. SteamVR already applies head tracking; projecting its combined
eye pair onto a virtual screen would show both images in each eye and apply head rotation again.
The glasses still supply poses to SteamVR. The saved picture mode resumes when SteamVR stops.

With **Game on glasses** off, SteamVR keeps the glasses' picture settings and IMU while Android
mirrors the phone output to the glasses. Turning off the dedicated display must not turn off the
renderer/HMD proximity flag: SteamVR otherwise reports the headset as taken off and enters standby,
freezing the view on both displays. Without a glasses display or an enabled phone VR viewer, SteamVR
uses a single-eye phone preview with the phone's tracking preference. These effective rendering
settings last only while SteamVR runs; the saved Cardboard setting is unchanged.
Verified on the VITURE Pro 2 with Game on glasses off: live glasses IMU, roughly 60 fps beyond
the previous standby point, and user-confirmed updating images and correct head tracking on both
screens. The rendering regression tests cover mirrored/dedicated parity and phone fallback.

A separate freeze occurred after swapping Pro 2 for GT Max and restarting SteamVR in the same
DroidDeck process (2026-10-14, session 52): GT Max board `0x41` kept streaming poses and the HMD
stayed active, but `vrcompositor` reported `vkerror=-4` (`VK_ERROR_DEVICE_LOST`) from 17:18:53.
Restarting the Steam session alone reproduced it (sessions 53/54). Restarting DroidDeck itself
cleared it; session 55 continued rendering with live GT Max tracking for over a minute without
new device-loss errors. This is a recovery workaround, not a confirmed permanent fix; retained
graphics state across restarts is a hypothesis. Frame counters alone do not prove recovery:
the presenter can keep presenting an old SteamVR image while the compositor reports device loss.

### Joy-Con controllers

**Glasses → Tracking → Experimental → Luma Joy-Con tracking** is off by default. Enable
positional tracking first, then this toggle. It uses the Luma Ultra primary stereo camera pair
to locate each hand holding a Joy-Con, with the existing calibrated, capture-time head transform.
Joy-Con IMU orientation, buttons, sticks, battery reporting and haptics remain in use. This mode
also enables Joy-Con takeover for SteamVR even if the ordinary controller preference is off.
Android 12+ and controller IMU support are still required.

Stereo positions and the 6DoF head share the same recentered floor origin. If a hand is hidden,
invalid, or its observation is older than 200 ms, that side blends to the arm model anchored to
the current head position. Reacquisition also blends (180 ms time constant). No usable calibration
or visible palm means arm-model-only positioning. Loss of head tracking beyond 100 ms invalidates
the controllers and releases their inputs. Reconnects, origin changes and Joy-Con recentering discard
old positional corrections. Point the Joy-Cons forward when connecting or recentering; camera
positions do not correct IMU yaw drift. Position extrapolation is disabled in this mode.

The camera observes the gripping **hand**, not the Joy-Con shell. Occluded palm landmarks, crossed
hands, and handedness ambiguity can reduce tracking availability. This is an experimental hand
position proxy with estimated fallback, not marker-based controller tracking. Controller grip offset,
aiming, motion latency, and sustained performance still need physical validation with held Joy-Cons.

Luma Joy-Con tracking and stereo bare-hand controllers are mutually exclusive. Either selected
Luma controller mode pauses phone-camera inference. External VR controllers retain priority.
Disable both Luma controller toggles to return to the upstream Joy-Con/device-camera modes,
which retain their existing fixed head-position anchor. Positional tracking, stereo controllers,
and phone-camera controls are grouped at the bottom of the glasses settings under **Experimental**.

Nintendo Joy-Cons paired over Bluetooth become SteamVR hands while SteamVR runs, and stay an ordinary
game pad otherwise. It needs Android 12 or later (motion sensors on an `InputDevice`) and a kernel whose
`hid-nintendo` driver reports the IMU (Android common kernel `android14-6.1` or newer; older 5.x GKI
builds expose the buttons only). A Joy-Con without motion sensors stays a game pad.
**Controller settings > Joy-Cons as VR controllers** (on by default) turns it off.

`input/JoyConVr.kt` takes a Joy-Con once SteamVR reports that it is running. It reads the Joy-Con's
gyroscope and accelerometer on a thread of its own. A Mahony filter turns them into an orientation,
and the gyro bias is learned while the Joy-Con rests. Each IMU report carries three samples every
10 ms. On the tested kernel they arrive newest first and about four in ten reports repeat the
previous report's newest sample, while the kernel's timestamps run about twice as fast in an uneven
3.3/3.3/13.3 ms pattern. `JoyConReports` collects a report's samples, learns their order from the
repeats, drops the repeated one and times the rest from the report's arrival (report interval divided
by new samples). That cut the gyro's error from 13 to 48 degrees per movement to about 2 to 3.

The accelerometer also measures movement. Linux removes the gyro's offset but not the
accelerometer's (one Joy-Con read 10.08 m/s² lying still, the other 9.51), so `JoyConAccelCalibration`
learns offset and scale by fitting a sphere to the readings of a Joy-Con resting in at least five
different positions; it happens on its own while you play and is saved per Joy-Con. `JoyConMotion`
takes gravity away in the world frame, integrates what is left into a velocity and a displacement,
sets the velocity to zero whenever the Joy-Con is still, and leaks it away quickly once a movement
lasts longer than a second, so the displacement holds for short lifts, pushes and punches and cannot
run away. It also keeps a second velocity that is never zeroed and only leaks over 30 s, the plain
integral of the acceleration, for camera tracking. The pose, displacement, both velocities, buttons,
stick and battery go to a 224-byte seqlocked file, `joycon_shm.h`, at
`/run/droiddeck/droiddeck-joycon`. The game pad no longer sees that Joy-Con;
it gets it back when SteamVR stops, the Joy-Con disconnects or the setting is turned off.

The driver adds a left and a right Index controller (`knuckles`, Valve's Index input profile and
models) when the first Joy-Con appears. The pad hand on that side switches off. A Joy-Con cannot
measure where it is, only how it turns and how it speeds up. Without the camera, Google VR's arm model
places the hand from its orientation and the head, and the measured displacement moves the hand on
top of that before it settles back over 0.6 s (1.5 s while the trigger or grip is held, at most
35 cm away), so a quick lift or punch shows but a slow sideways drift does not. With camera hand
tracking (below) the camera places the hand instead. Its pointing direction is the Index controller's
aim pose. Its heading is matched to the head when it connects. Holding Home (right) or Capture
(left) for 0.8 s matches it again, with a short buzz; point the Joy-Con where you look first. A short
press is the system button. A hand that sends nothing for 500 ms disconnects.

| Index        | Left Joy-Con     | Right Joy-Con  |
|--------------|------------------|----------------|
| Trigger      | ZL               | ZR             |
| Grip         | L, SL, SR        | R, SL, SR      |
| A            | down, right      | A, B           |
| B            | up, left, minus  | X, Y, plus     |
| Thumbstick   | stick, click     | stick, click   |
| System       | Capture (press)  | Home (press)   |

The fingers follow the buttons on a simulated hand: the index finger curls with the trigger, the
other three with the grip, the thumb when it is on the stick or a face button. SteamVR's haptic
events go to that hand's Joy-Con through `@droiddeck-rumble` (slots `0x100` and `0x101`) and respect
the rumble setting.

Outside SteamVR two Joy-Cons are combined into one player: left stick, D-pad, L, ZL, minus and Capture
from the left one, the face buttons, right stick, R, ZR, plus and Home from the right. Face buttons
keep their positions (Nintendo X is the top button, Y on Xbox). SL and SR are shoulder buttons on a lone
Joy-Con and are ignored while two are combined. A combined pair splits rumble the way a pad does,
strong to the left and weak to the right. A lone Joy-Con plays the stronger of the two.

#### Camera hand tracking

The device's camera can watch the player so each Joy-Con hand sits where the real hand is, including
lifting, lowering, reaching and moving sideways. With no controller connected it tracks bare hands
instead (below). It needs Android 12 or later. The first time a headset connects, DroidDeck asks whether to use the camera. On a device
with cameras on both sides it offers the back one (usually sharper and wider) or the front one; with
cameras on one side only it uses those without asking. Android's camera permission is requested
right after. **Headset sheet > Head tracking > Hand tracking** turns it on or off, switches the
camera, asks for the permission again and opens the placement check.

When camera tracking is on, **Play SteamVR** first shows the placement check: the live picture with
the detected body drawn on it, what to fix, and **Skip**. It reads the device's gravity sensor and
asks to stand a device up when it lies flat face up or face down, or to stand it more upright when
the camera points more than 45 degrees up or down. Then it asks to step into view, to step back or
come closer (the shoulders should take 12 to 26 percent of the picture's width, about 0.75 to 1.6 m
from a 90 degree camera), to step back when an elbow or wrist is near the edge of the picture, and
to hold both hands in view. Once
everything has been fine for 1.2 s SteamVR starts on its own. With the back camera the screen faces
away from the player, so the advice is also spoken aloud.

`handtracking/HandCamera.kt` runs while SteamVR runs and either a Joy-Con is taken or no game
controller is connected at all. It picks the widest
camera on the chosen side (up to 100 degrees, zooming a logical camera out when it can), holds it at
its fastest fixed frame rate up to 30 per second so exposures stay short and moving hands do not
blur, takes 960x720 frames from CameraX, turns them upright by gravity, and runs MediaPipe's Pose
Landmarker (Lite) on the CPU in video mode, which follows the body from frame to frame and smooths it,
15 times a second (10 and then 6 when the device reports it is getting hot). The pose model's depth
cannot tell a hand at the chest from one reaching out (it read about 25 cm in front of the shoulders
for both), so each hand is measured on its own: `HandModel` runs MediaPipe's Hand Landmarker on a
square around the hand. To find a hand it looks for up to two hands in a square around the pose's
wrist and keeps the one whose wrist is nearest, taking turns between a square 4 shoulder widths across
centred on the wrist and one 2.5 shoulder widths across moved toward the fingers. The palm detector
misses many hands in a tight square: with the player about 0.65 m away the 2.5-wide square found 73
percent of the hands in view (a 1.4-wide one 49, the whole picture 68), and with the player lying close
to the camera, where hands look large, the wide square found 77 percent of the hands the tracker had
seen and the two in turn 82 (the 2.5-wide one alone 54). After that it follows the hand in a square 2.8
hands across, and finds it again in the same frame when it is lost. A hand whose wrist is nearer the other arm's wrist than its own is dropped.
The hand's 21 landmarks come in metres and in pixels, so a least-squares fit gives the hand's
distance from the size it appears (`BodyPose.handFix`); the middle of the palm is the point used.
The head is fitted the same way from the pose model's face landmarks. Each hand is given relative to
the eyes in a frame levelled by gravity, along with the face's heading from the nose and ears. When
the hand model misses a hand the pose still sees, the hand keeps its last distance for half a second
and is marked as estimated. The frame's exposure time, moved onto the monotonic clock, the hands, each hand-model hand's 21
landmarks relative to the middle of its palm (in metres, in the same levelled frame), the face heading
and whether bare hands are wanted go to a 568-byte seqlocked file, `handcam_shm.h`, at
`/run/droiddeck/droiddeck-handcam`.
No picture is kept or sent.

The driver turns the camera's frame to SteamVR's by comparing the face's heading with the headset's
heading at the same moment, averaged over about 3 s and restarted when they disagree for half a
second (after a recenter). Each hand then runs a Kalman filter (`HandKalman` in `hand_motion.h`) with
position and velocity on each axis. The Joy-Con's plain velocity integral moves it forward between
camera frames; its reports are timed 45 ms earlier than they arrive, which lines their movement up
with the camera's. The Joy-Con is trusted less while it moves hard, since a small misalignment then
turns into a large error. The filter's speed fades with a 0.3 s time constant (0.1 s once the camera
has not seen the hand for 0.2 s, since a small tilt error then pushes the hand steadily), and while the Joy-Con
has been still for 0.1 s (turning under 0.3 rad/s and accelerating under 0.6 m/s²) it is told the hand
is not moving, so a hand the camera loses settles where it was instead of sliding away, and a resting
hand does not shake (on a recording, 3.6 to 4.3 cm of shake fell to 1.0 to 1.5 cm). A camera sighting
is applied at the moment the picture was taken, 60 to 140 ms in the past, and the Joy-Con movement
since then is replayed on top, so the hand is where it is now, not where it was. A hand-model sighting
counts as about 3 cm of camera noise, an estimated one as 6 cm.
One sighting far from the filter (the model swapping left and right) is ignored unless three in a row
agree. When the camera has not seen a hand for 0.75 s, the arm model takes over from where the hand
was, and changes between the two blend over a quarter of a second.

#### Bare hands

With camera tracking on and no game controller or Joy-Con connected (checked again whenever an input
device comes or goes), the app sets a flag in the camera file and the driver gives SteamVR two hands
from the camera alone, the way a Quest's hand tracking does; the pad hands switch off. Each hand is
still an Index controller to SteamVR, so games' Index bindings and the dashboard work, but its render
model becomes a white glove with a light grey band at the wrist, like the plain hands a Quest draws
(`{droiddeck}droiddeck_hand_left` and `_right`, built at build time by `tools/vr/make_hand_models.cpp`
from the same skeleton). The glove is a smooth surface: tapered capsules along each finger bone, a
domed palm and the band are blended into one distance field and meshed with surface nets (a 2.6 mm
grid, up to about 10,000 vertices a part). SteamVR draws render models almost unlit, so the shading
is baked in: ambient occlusion sampled from the distance field, and a soft light on the back of the
hand, pick a column of a 64x64 texture whose rows run from the glove's white to the band's grey. It
has two parts with two shapes each: the palm with the middle, ring and little fingers (open, or
curled into a fist while `/input/grip/touch` is on, at half closed), and the thumb with the index
finger (open, or pinching while `/input/trigger/click` is on). The thumb part carries a copy of the
palm 0.3 mm inside the other part's, so the two blend without a seam. SteamVR only routes a part to
its input when the part also names a legacy button (`controller_button`, as the Index model's buttons
do; vrserver then logs `inputRouting=1` for it), and a hand whose laser points at the dashboard sends
its trigger to the dashboard, so that hand clicks without drawing the pinch. Its pointer and grip poses are the Index
controller's, so the laser leaves along the fingers. Games that draw hands from the
skeleton get the camera's finger curls. Joy-Cons connected later take over; their hands go back to the
Index model.

| Gesture | SteamVR input |
| --- | --- |
| Thumb and index finger together | Trigger: it starts pulling 6 cm apart, clicks under 3.5 cm and lets go over 4.5 cm |
| Middle, ring and little fingers closed | Grip, clicking when fully closed |
| Thumb to middle finger | B |
| Thumb to ring finger | A |
| Right palm toward the face, then pinch | System (the SteamVR dashboard) |
| Left palm toward the face, then pinch | B (most games' menu) |

A gesture has to hold for 50 ms to press and be gone for 100 ms to release, and a palm has to face the
headset for 0.3 s before its pinch becomes the menu. A and B only count with the hand open, take the
fingertips within 2.5 cm (released over 3.5 cm), and while the palm faces the headset the trigger stays
released. The distances were checked on 504 photos from the public HaGRID gesture set (50 each of
eleven gestures) through the same hand model: MediaPipe puts touching thumb and index tips a median
3.4 cm apart, so a 2.5 cm pinch caught only 27 percent of the "ok" signs; at 3.5 cm it catches 51
percent while open hands, pointing, "stop", "four", "three" and thumbs-up misfire in 4 percent or
fewer. A fist pinches in a quarter of them, which games read as grabbing with the trigger held. The
grip read closed for 92 percent of fists and 96 percent of pointing hands and for none of the open
hands.

`CameraHand` in `hand_pose.h` does the work. Each camera frame's landmarks are turned by the camera's
yaw. The finger curls come from each finger's joint angles, looked up in a table of the skeleton's own
angles. The skeleton's palm at those curls (wrist and four knuckles) is fitted to the camera's with
Horn's closed-form quaternion method, and the root is placed so the skeleton's palm lands on the palm
position. Only sightings with hand-model landmarks count; the pose-only estimates are left out. The
position goes through a 1€ filter (Casiez, Roussel and Vogel, 2012): a low-pass at 1 Hz while the
hand is still that opens up the further and faster the hand gets ahead of it. On a recording of bare hands it cut
the wobble of a still hand from the 9 to 11 mm the Joy-Con Kalman filter left to 2.5 to 3 mm, and the
largest jump between frames from 10 to 17 cm to about 1.3 cm. The turn goes through the same kind of
filter (1 Hz when still, 0.2 Hz more per radian per second), and the curls follow within 80 ms. A fit
more than 2 cm off is skipped, and a fit that turns the hand more than 45 degrees at once is held back
until the next one agrees within 25 degrees (the hand model now and then flips a hand around; the
largest turn between frames fell from 14 to 7.5 degrees). When the camera has not seen a hand's landmarks for half a second,
the hand reports itself out of range and lets go of every input; it reappears wherever it is seen
next. There is no stick or trackpad, so moving around needs a game's own hand tracking or pointing
to teleport.

On MediaPipe's own example pictures (a raised index finger, a thumbs-up) the right-hand skeleton fits
the palm within 7 mm, the curls read as the pose looks, and the palm direction points at the camera
when the palm faces it.

Depth from one camera is approximate: the model expects one person, up to about 4 m away, with the
head in view.

Joy-Con orientation and buttons were tested on a REDMAGIC tablet, and the movement code was tuned on
recordings from its Joy-Cons. Camera tracking was recorded on the same tablet with the front camera:
the camera holding 30 frames per second and video mode took the hands' frame-to-frame jitter from
about 5 cm to 1 to 2 cm; the hand model and the filter were then tuned on that recording. For a
diagnosis, an empty file named `handcam-record` in the app's files folder makes the next sessions save
up to 6 minutes of small camera pictures with every landmark and timing to the app's cache
(`handcam-rec/`), and the driver writes its camera, Joy-Con and filter values to
`/run/droiddeck/droiddeck-handtrace.csv` while they run. `tools/vr/build.sh` runs `test_joycon` (arm
model, aim pose, hand skeleton, movement, the filter's latency, noise and outlier handling, yaw,
both file layouts, a lost hand settling, the palm fit, curls, gestures, bare camera hands, the turn
filter and the hand render model). `JoyConTest`,
`JoyConOrientationTest`, `JoyConMotionTest`, `JoyConShmTest`, `JoyConPadTest`, `RumbleComponentTest`,
`BodyPoseTest`, `TrackingCamerasTest`, `PlacementAdviceTest`, `HandCameraShmTest`, `HandTrackingTest`
and `HandCameraPromptTest` cover the app side.

### Device-loss recovery

During a SteamVR session, DroidDeck reads new `vrcompositor.txt` output every two seconds and detects
`VK_ERROR_DEVICE_LOST` or `vkerror=-4`. Existing errors at session start, ordinary acquire timeouts,
and unrelated failures do not trigger recovery. Reads are bounded to the newest 128 KiB and handle
log replacement, truncation, and partial lines.

A detected fault shows an eight-second cancellable countdown while the session is foreground.
Touches outside the prompt do not dismiss it; the user can explicitly choose Keep open.
Recovery stops the guest, gives teardown and diagnostic collection up to 15 seconds, then uses a
private `:vr_recovery` activity to kill only DroidDeck's main process and reopen SteamVR. That process
does not initialize the glasses SDK or guest runtime. It recreates the native graphics context; an
Activity recreation or SteamVR-only restart does not. Running games close and unsaved progress cannot
be restored. A persisted ten-minute cooldown disables another automatic attempt, while still
offering a manual restart. Session events record detection and recovery requests. This mitigates
the freeze; it does not fix the underlying Vulkan device loss.

Validated on the phone with a clearly marked `SIMULATED_RECOVERY_TEST: VK_ERROR_DEVICE_LOST`
log entry in session 58. Detection and the eight-second countdown led to a new main-process PID
and session 59 automatically reached READY and requested SteamVR. The
cooldown timestamp survived the process restart. This was a recovery-flow test, not a new GPU
failure. Subsequent user feedback and `vrserver.txt` showed that SteamVR had blocked `driver_droiddeck`
after a previous crash: the server ran, but reported `HmdNotFound` and never produced VR output.
Recovery now clears only `driver_droiddeck.blocked_by_safe_mode`, atomically after stopping the old
process and before launching the new session. It preserves global safe-mode preferences and other
drivers' blocks. A running server or resumed sensor stream alone is not proof of successful recovery.
The follow-up marked test in session 63 exercised the manual retry during the cooldown. Recovery
logged the driver unblock, restarted into session 64, and SteamVR recognized the DroidDeck HMD.
Fresh compositor output and a 3840x1080 `DroidDeck Cardboard VR` presenter appeared, with subsequent
display intervals at 56–59 fps. The user confirmed the GT Max VR view and tracking work after this
recovery. The 19 SteamVR unit tests and APK runtime-input verification passed.
The glasses/SteamVR suite includes stale/rotated/split log handling, countdown,
cancellation, foreground-only behavior, and cooldown checks.

Stereo also requires a physical side-by-side display mode. On the tested Astra 2
firmware the Pro XR command succeeds but DisplayPort falls back to 640×480;
direct-connection stereo remains unresolved. DroidDeck remembers such a fallback for that glasses
model and stops asking for 3D on its own; **Retry** in the glasses settings tries again. See the
[Astra/VITURE investigation](development/astra-viture-stereo.md) for measured
results, app recovery behavior, and the unverified powered-adapter design.

### Not done yet

- **Controllers**: the driver adds two hands driven by the session's game pad. It reads
  the absolute-state snapshot the app already publishes in fake input ring 0
  (`FAKE_EVDEV_MEMFD_PATHS`), so the app needs no new plumbing. **Play SteamVR → Settings → Controllers**
  chooses what they are; the app writes the choice to `/run/droiddeck/droiddeck-vr-controllers`
  (`emulate=index|touch|vive|off`, `sticks=game|hands`). The same choice is in the session menu's glasses page
  and on the phone while the picture is on the glasses. The driver checks the file every second and switches
  while SteamVR runs: SteamVR fixes a device's input profile when it activates, so each type gets its own pair
  of devices, added the first time it is chosen, and the pair it replaces is disconnected:
  - **Valve Index** (default) and **Meta Quest 3** (Touch Plus) are thumbstick controllers with the
    stock input profiles, so games use their own Index or Touch bindings. Triggers are triggers, bumpers
    grip, A/B and X/Y are each hand's face buttons, stick clicks are thumbstick clicks and Guide is system.
    Quest's left menu button is Start or Select.
  - **HTC Vive wands** keep the older layout: the sticks are trackpads, A/X or a stick click presses them,
    Start and Select are menu.
  - **Regular game pad** adds no hands, for games that play with a game pad.

  The hands rest low in front of each side and point where the head looks. By default the sticks move you
  in the game; pressing both sticks in together switches them to moving the hands and back. Each stick then
  moves its hand sideways (40 cm), up (50 cm) and down (30 cm) and turns it as if from an elbow 45 cm
  behind, so pointers and aim follow it too; the hand eases back when the stick is let go. The d-pad's up
  and down move both hands in and out; while a hand's trigger or grip is held they move only that hand
  (a slingshot or bow pulled back), and it eases back on release. Both hands activate in SteamVR and their
  laser shows on the dashboard. They switch off while another driver's controller is connected. A Joy-Con
  hand replaces the pad hand on its side, and bare camera hands replace both while no controller is
  connected (above).
- **Layers**: only the first layer of each frame is drawn. SteamVR's compositor composites the
  dashboard into it, so this has not mattered yet.
- **Timing**: no reprojection. The presenter blits the frame submitted last once `vrcompositor`'s GPU
  work for it is done: the layer fences that process's submits and counts issued and finished ones in
  `present-sync` beside the `active` marker, and the presenter sleeps on the count (a futex). Without
  the file it falls back to a guard delay after the frame arrived (`DROIDDECK_VR_PRESENT_GUARD_MS`,
  4 ms). It hands its own picture to Wayland only once its blit is done, because KGSL does not order
  the DroidDeck compositor's copy after it. The chain is SteamVR → presenter → DroidDeck compositor →
  lens pass.
- **Cross-process GPU sync** is a poll away from exact (see the semaphore note above); binary
  semaphores are not bridged.
- **Apps and games**: Windows VR games run through Proton ARM64 (FEX for the x86-64 game, DXVK, and
  Proton's `vrclient_x64` bridging to SteamVR's ARM64 `vrclient.so`). Tested on the Thor:
  - **Thumper** (356400, its own D3D11 engine), started with its VR launch option: menus and levels
    at 60 Hz, about 7 ms CPU and 13 ms GPU a frame, 23 dropped frames in 6485;
  - **The Thrill of the Fight** (494150, Unity): loads into its gym, 60 Hz with 6.5 ms CPU and 8.5 ms
    GPU a frame;
  - **The Lab** (6.6 GB) loads, but Android's low-memory killer took the whole app while it did, next
    to Steam and SteamVR. Two VR games at once do the same, so quit one (`droiddeckctl quit <appid>`)
    before starting the next;
  - **SUPERHOT VR** stops at its EULA, which SteamVR's dashboard shows in VR;
  - **SteamVR Home** is not in the ARM64 package. Steam's own Windows SteamVR (250820) has it
    (`tools/steamvr_environments`, `steamtours.exe`), and it starts under Proton, but Proton's
    `vrclient_x64` implements only `IVROverlayView_003` and Home asks for `_002`, so it stops at
    "SteamVR initialization failed!";
  - SteamVR's ARM64 OpenXR sample, `helloxr -g Vulkan2`, reaches a focused session and submits frames
    (it was stuck before the first one until the wait fix above), but its picture shows up as torn
    tiles: next to chase. `hellovr_sdl` (OpenGL) needs GLEW, which the runtime lacks.

  SteamVR opens its game menu over a newly started game; `vrcmd --hidedashboard` (in
  `/opt/steamvr/bin/linuxarm64`) closes it.
- **Quest 3**: unrelated to this; on the Quest DroidDeck stays a 2D panel (`docs/quest.md`).

## Building and testing

- `tools/vr/build.sh [OUT]` builds the driver and the layer in the local cross image, or
  `--native OUT` with aarch64 compilers on PATH (CI). `tools/build_local.sh` and CI put them in the
  APK under `assets/linuxfs/usr/local/lib/droiddeck-vr`; `SessionFiles` stages them, and
  `droiddeck-steamvr`, into the rootfs.
- Shader: `cardboard_frag.h` is generated from `cardboard.frag`
  (`glslangValidator -V --vn cardboard_frag_code -o cardboard_frag.h cardboard.frag`).
  `tools/tests/test_cardboard.py` checks it is current, and checks the push-constant packing on the
  host.
- `CardboardTest` covers the settings, lens inset and field of view, the head-pose maths (quaternions,
  display-rotation remaps, recentring) and the SteamVR status. `test_droiddeck_agent.py` covers the
  `vr` request.
- Logs: the compositor's `[cardboard]` lines; SteamVR's in `~/.local/share/Steam/logs`
  (`vrserver.txt`, `vrcompositor.txt`) and `/run/droiddeck/droiddeck-steamvr/vrstartup.log`. The
  driver's lines start with `droiddeck:`.

Lessons from bringing this up on the Thor, and how to debug a live session: `docs/development/cardboard-vr-learnings.md`.