DroidDeck/docs/development/luma-spatial-tracking.md

# Luma Ultra spatial tracking

For supported glasses across brands, feature levels, and app-wide setup, see the
[smart-glasses support matrix](../smart-glasses-support.md).

This is an experimental implementation of native Luma Ultra head tracking, positional virtual screens, and optical SteamVR controllers. Positional tracking, bare-hand controllers, and Luma Joy-Con tracking all default to off and are grouped under Experimental. Combined release acceptance remains pending until the calibration, accuracy, latency, gameplay, and sustained performance targets below have been measured on hardware.

## Build and setup

Luma Ultra now includes a shared starting calibration exported from the user's verified Fold7 profile on 2026-10-16. The bundled `calibration/luma-ultra-640x480.json` contains the original lens, stereo, and camera-to-head geometry without the source glasses' serial identifier. It applies only to the SDK-confirmed Luma Ultra primary `left0/right0` pair at 640 by 480. Matching saved device calibration takes precedence, including an unfinished personal fit that still requires distance validation. Missing, corrupt, or foreign profiles fall back to the model base; unknown serials can also use the base.

The base enables experimental hand tracking without requiring every user to repeat board collection. It is not marked individually verified on other units. Tracking and setup show **Shared Luma Ultra calibration** while it is in use; diagnostics distinguish `calibrationSource` (`luma-ultra-base`, `device`, or `none`) from `calibrationVerified`. The existing `calibrated` availability field is true for either the base or a verified personal profile. The following board steps are optional refinement for a new device using the base; a newly calculated personal profile still needs the unchanged distance check before use. A successful distance check saves the profile for the connected identified glasses. Source validation does not establish manufacturing consistency or aiming accuracy across other units.

Use the existing optional `droiddeck.vitureSdk` local property with the external VITURE Android SDK 2.4.0. No vendor headers or libraries are checked into the repository. `-PvitureSdkDir=` explicitly builds without the SDK. The ordinary runtime freshness checks remain enabled. Rebuild the ARM64 VR components with `tools/vr/build_windows.py`; its manifest includes `spatialpose_shm.h`.

The hand pipeline pins MediaPipe Tasks Vision 1.0.0, OpenCV 4.13.0, and the model and SHA-256 recorded in `tools/packs/tracking/hand_landmarker.provenance.json`. The model, OpenCV and MediaPipe native libraries are downloaded on first use as the tracking pack (`tools/packs`); the pack build and the inference worker verify the model checksum. CPU inference, the full model, concurrent camera inference, scheduled hand discovery, and automatic input-rate adjustment are the defaults, including after app restarts, reconnects, and provider restarts. **Show diagnostics → GPU hand inference** temporarily enables GPU until the next tracking restart; GPU remains experimental because SteamVR restarts and Vulkan device loss were observed during testing. Diagnostics report the selection as `handGpuEnabled` and the initialized detectors as `handInferenceBackend`; `not initialized` means detector startup has not completed. Sustained performance acceptance remains unresolved.

Hand input starts by copying 500 permille of stereo callbacks, approximately 12.5 pairs per second at the SDK's documented 25 Hz camera rate. On CPU, automatic adjustment can raise this to 600 or 800 permille (about 15 or 20 Hz) only after eight consecutive valid observations have sufficient processing and freshness headroom. Promotion requires average processing at most 55 ms and observation age at most 110 ms for 15 Hz, then at most 40 ms and 90 ms for 20 Hz. A moving average allows occasional scheduled discovery frames while preserving a sustained processing budget. An average above the selected period reduces the rate; any single processing cost of 80 ms or observation age of 140 ms immediately returns it to 500 permille. Tracking failure and detector resets also return it to 500. This policy requests more input only when processing can accommodate it; it does not guarantee that many completed or valid hand updates.

Skipping happens before copying camera pixels into DroidDeck, so skipped pairs also avoid JNI transfer, preprocessing, and inference. The SDK exposes no tracking-camera rate setter: sensor capture, vendor VIO, pose callbacks, and head polling retain their normal rates. Calibration collection, distance checks, explicit diagnostic capture, and preview-only operation use full-rate input. Diagnostics keep incoming `cameraHz` distinct from `cameraCopyPermille`, `rateLimitedFrames` (intentional skips), and `dropped` (eligible pairs replaced while the worker is busy). The compatibility field `cameraFrameDivisor` is 2 at 500 permille, 1 at 1000, and 0 for intermediate fractions. The bounded latest-pair queue and original capture timestamps remain intact. **Automatic hand input rate** can be disabled to expose the manual half/full-rate switch; both choices reset with tracking. GPU uses that manual preference and does not adapt. Thermal, latency, and gameplay benefits require hardware comparison.

1. Connect a Luma Ultra and grant its existing USB permission. Spatial controls appear only after the SDK identifies the device as Carina.
2. Enable **Positional tracking — experimental**. This restarts the provider in 6DoF mode. Switching picture modes does not restart it.
3. Enter the current seated or standing eye height (shown in centimeters and feet/inches) and choose **Set floor from current height**. Normal VR recentering then preserves that floor, including while crouched. After reconnecting or explicitly restarting tracking, accept the new origin through setup/recenter. A recenter request during lost/stale tracking or recovery remains pending until 250 ms of fresh, steady poses arrive; repeated taps are unnecessary. The pending request is canceled on disconnect.
4. Open **Stereo camera setup and diagnostics → Open setup**. The page uses the selected DroidDeck theme and app scale, with a single compact Back/title row so the controls have room to scroll in landscape. Check head tracking and camera synchronization first; capture controls become available when both are ready. Expand **Show diagnostics** for frame rate, frame age, dropped frames, processing time, and recovery tools.
5. Print the 5 by 7 ChArUco board at actual size. Measure a square: it must be 30 mm (approximately 1.181 inches). Keep the board stationary and capture at least 24 sharp views by moving and rotating the glasses, covering the image center and edges. Include both yaw and pitch, roll, and varied distances. If the underlying tracking frame changes, start a new capture set; software recentering in an unchanged frame preserves the set.
6. Calculate calibration. Accepted captures confirm board detection, not alignment quality. The fitter compares pinhole and fisheye models on held-out views. It rejects poor stereo reprojection, an implausible baseline, insufficient rotational diversity, and inconsistent stationary-board alignment. Turning and tilting the glasses is required; pause before each capture. If the board stayed fixed but alignment varies beyond 5 cm (about 2 inches), the head-tracking frame may have shifted during collection. Wait for steady tracking and start a new set; do not remove the quality gate to accept the old one.
7. Start with **12 in (30.48 cm)** or **30 cm (11.81 in)** under **Measured distance**; **20 in** and **50 cm** remain available. The closer settings make the small printed markers easier to resolve in the 640 by 480 tracking cameras. Show the complete board with its center exactly the selected distance from the left camera, then choose **Validate distance**. You can hold or support the glasses without wearing them. Progress and the result appear beside this button. The count refers to detected checkerboard intersections, not the paper's edges; a fully visible but small or blurred board can still have too few detections. Validation detects corners in the original images, then corrects those coordinates for lens distortion before triangulating, avoiding marker blur from resampling the images. Rejected stereo geometry reports that depth could not be measured; a valid reconstruction shows measured and selected distances in both units. Each imperial or metric choice uses its exact distance. The complete 24-corner board and physical-distance check within 5 cm (about 2 inches) are still required before a newly fitted personal profile becomes usable. Profiles are keyed to the SDK board-serial SHA-256 identifier (falling back to a nonblank Android USB serial), dimensions, and format version. Verified profiles reload automatically after an app restart or reconnect to the same identified glasses. If neither identifier is available, profiles remain connection-local.
8. Enable **Hand controllers — experimental**. Present an open hand first so inputs can arm. Aim the virtual controller at the target, pinch thumb and index finger together to press the trigger/click, then separate them to release. Moving the SteamVR dashboard by its handle uses this same pinch-and-release action. Keep the other fingers open when only clicking; curling the middle, ring, and little fingers activates the separate grip input, which the installed Vive dashboard binding maps to Back. Games can use grip for holding objects. After tracking loss, open the hand again before another press. Gamepad sticks supply locomotion and turning; buttons retain the Vive-compatible mapping. External VR controllers retain precedence.

The diagnostic capture button saves one stereo pair and metadata to the app-private `files/spatial/captures` directory. Merely opening diagnostics does not save images. Calibration profiles stay in `files/spatial`. A brief missed hand detection holds the last observed controller transform only until its original observation reaches 350 ms old. Input freshness remains limited to 200 ms. Pinch and grip release immediately on a detected miss and require an observed release before rearming. The hold does not extrapolate motion, refresh timestamps, or substitute a gamepad pose. Head tracking loss, origin/connection changes, disconnects, and controller-mode changes clear it. Gamepad menus and locomotion remain available during optical hand loss. Switch hand controllers off explicitly to return to gamepad-positioned controllers.

Accepted board landmarks and their capture-time head poses are saved separately in app-private `files/spatial/*-captures.json`, without camera images. A matching SDK or USB device identity and image geometry allow automatic recovery for recalculation. Restored observations cannot be appended to a new tracking frame; start a new set to capture again. Devices without either stable identifier keep a session-labelled diagnostic backup and require explicit developer recovery onto the same physical device. They do not automatically reuse another connection's unidentified capture set or personal calibration; they can still use the bundled model base.

The vendor SDK working directory is `noBackupFilesDir/viture`. Android can evict `cacheDir` during a live session, so active tracking files must not depend on that location. The SDK initializes its own files in the new directory on the next provider start; existing printed-board calibration profiles remain in `files/spatial`.

## Coordinates and lifecycle

`SpatialPose` uses meters and right/up/back OpenGL coordinates. Quaternions are normalized xyzw internally; SDK arrays are position followed by wxyz. Legacy orientation conversion and the 128-byte head-pose channel remain intact. The session coordinator survives `HeadTracker` recreation and owns separate full-pose screen and yaw/floor VR origins. The camera worker uses a generation guard, a bounded latest-frame queue, and no SDK lifecycle operations in callbacks.

Head polling is approximately every 4 ms with prediction disabled; this is not a claim of a 250 Hz sensor. Camera-rate callbacks provide a liveness check. Stale or unstable poses retain the last rendered transform and produce an invalid SteamVR pose. While a positional session is active, DroidDeck covers the resulting grey view with a themed recovery screen on the phone and glasses, including one readable copy per eye in SBS. It distinguishes tracking loss, USB disconnection, required recentering, and stabilization after recentering. Tracking-loss instructions ask the user to look around slowly in a well-lit room with visible detail, keep the cameras clear, and open the setup diagnostics to restart tracking if recovery stalls. Phone controls remain accessible, and the screen closes automatically when tracking is valid. Freshness is checked even if pose callbacks stop. Repeated USB timeouts require connection recovery rather than a synthetic valid pose. Large pose discontinuities latch a setup requirement. Provider shutdown invalidates queued work before closing the vendor provider and Java USB connection.

The 176-byte `droiddeck-spatialpose` snapshot uses protocol version 4 and sequence-counter publication. The driver validates file length, version, finite normalized poses, and monotonic age. Head and both hands use one snapshot and one VR origin. Hand pose flags independently identify valid palm tracking (bit 0), pinch (bit 1), and grip (bit 2); head flags use only bit 0. The formerly reserved 32-bit word at byte offset 36 is now `input_epochs`, containing independent 8-bit loss counters for left trigger, left grip, right trigger, and right grip, from least to most significant byte. The layout size and pose offsets are unchanged, but versions 1 through 3 are rejected; app and driver must be rebuilt together.

Snapshot flags select positional tracking (bit 0), bare-hand controllers (bit 1), or Luma Joy-Con
controllers (bit 2). The latter two selections are mutually exclusive; either enables stereo inference.
The Joy-Con mode consumes only wrist landmark 0 and the original capture timestamp; rotation and inputs
come from the paired Joy-Con. `luma_joycon.h` selects fresh stereo positions or a 6DoF-head-anchored arm
model independently per hand, rejects observations over 1.5 m from the head, and blends source switches
with a 180 ms time constant. It never integrates accelerometer position or uses the phone-camera
correction in this mode. Stale head poses invalidate output and release inputs. Selected mode flags
survive disconnect so a lost Luma cannot silently switch to a fixed-origin controller. Origin/connection
changes reset both positional blending and controller yaw alignment. This is hand-assisted tracking,
not a detector for the controller shell; see [Joy-Con setup and limits](../cardboard-vr.md#joy-con-controllers).

Each counter advances when its gesture becomes ineligible, including invalid finger data, hand/head loss, and observation expiry. It survives later valid snapshots, allowing the driver to notice a short loss even if it never read the intervening invalid snapshot. Counters wrap independently within their bytes. Connection/origin changes reset driver input state, and a gap beyond 200 ms from its last accepted observation also releases inputs. Every hand retains its original camera observation timestamp: experimental hand poses and inputs expire after 200 ms, while head poses expire after 100 ms. A changed loss counter disarms only its affected gesture. Duplicate or older observations cannot rearm an input or replay a press; a new, valid released observation is required after loss before another press. Missing finger data never counts as a release gesture. Head updates cannot extend a hand observation's lifetime. Diagnostics expose the hand limit as `handMaxAgeMs`.

The driver submits trigger scalar, trigger click, and grip updates before submitting the controller pose, including when the pose is becoming invalid. Disconnect and deactivation also clear inputs first. Changes in input state, loss counters, or API error codes produce concise driver log records with raw gestures, output values, armed state, pose/input validity, observation age, connection, and origin. Successful input API calls confirm submission to SteamVR; they do not prove that the dashboard received its mouse-up event. The installed dashboard grab handler retains a hand attachment until mouse-up and has no tracking-loss cleanup in that JavaScript component. Recovery of the reported stuck dashboard drag therefore still requires hardware validation; the implementation does not manufacture another press or a valid pose to force release.

Controller display smoothing has no stationary anchors or dead zones. Small intentional motion remains visible; smoothing and prediction do not change the measured pose used for gesture validation.

The displayed controller converges toward each measured palm pose on every driver update with one 15 ms correction time constant and normalized, hemisphere-corrected quaternions. There is no additional per-observation smoothing stage. After three consecutive observations with consistent motion, translation advances from the result's receipt time for at most 80 ms and 2 cm; capture timestamps still determine velocity and freshness. Receipt-relative prediction can continue between delayed inference results instead of consuming its entire horizon before a result arrives. Prediction is disabled above 1.5 m/s or 6 m/s², on a missed/invalid observation, head loss, or changed origin/connection. A missed observation freezes the last displayed transform through duplicate old samples until a newer observation arrives or the original 350 ms visual-hold expiry is reached. Fast motion reaches the 2 cm bound sooner, and correction adds a small settling delay. Stabilization and prediction do not alter gesture values or observation timestamps and cannot replace measured latency validation.

The screen shader puts translation in `extra.yzw`, rotates the eye offset, and rejects intersections behind the eye. SBS and flat branches ignore translation. The push-constant allocation stays unchanged. A screen reference can include pitch and roll, preserving placement while reclining.

## Stereo reconstruction and limitations

Skipped camera pairs do not require clock warmup again when the source/host offset remains consistent. A source gap above 200 ms checks that offset against the existing 20 ms bound instead of unconditionally resetting synchronization. Nonmonotonic or invalid timestamps and inconsistent long gaps require 16 fresh observations; normal short-gap jitter retains the existing rolling-minimum and p90 policy. This preserves synchronization through a processing stall without publishing expired hand observations.

Palm orientation uses an explicit hand side. The lateral axis reverses for the right hand so mirrored palms have the same normal and finger-direction convention; the existing left-hand mapping is retained. This corrects the mathematical right-hand inversion, while physical controller aiming still needs confirmation.

The stock two-hand video graph can repeat whole-image palm detection when fewer than two hands were tracked previously. With both hands tracked it can skip that search; see the [MediaPipe graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259). CPU scheduling maintains two persistent detector banks, each containing one detector per camera: one bank permits one hand, and the other permits two. After three observations agree on exactly one confident hand in both cameras, the one-hand bank handles intervening frames, and a two-hand search becomes due after 240 ms of camera time, running on the next processed frame. Both hands, a miss, handedness disagreement, nonmonotonic time, or a gap above 400 ms return selection to the two-hand bank. Second-hand discovery therefore remains enabled but can wait for the next scheduled search and processing. Banks retain their video tracking state between selections; they are not rebuilt per frame. GPU retains the ordinary two-hand bank. Diagnostics expose the active bank as `inferenceHandLimit`, alongside raw and filtered hand counts and per-camera timing. These choices reduce unnecessary searches by design; their benefit and extra model-memory cost require device measurement.

The primary `left0/right0` images are copied only for diagnostics, calibration, or enabled hands. At most one pair is pending while the worker processes another. Automatic exposure is retained because the cameras also serve native VIO. SDK 2.4.0 documents the tracking-camera rate as 25 Hz and provides no stereo frame-rate setter. Polling more frequently does not create additional camera frames. The camera-rate readout accounts for skipped source-frame sequence numbers instead of treating processed-pair throughput as sensor rate. The selected video-mode Hand Landmarker pair receives matching timestamps. CPU inference runs the cameras concurrently with separate image buffers; each bank's right detector is created, used, and closed on one dedicated worker. Both calls must finish before pixels or detectors can be reused, including after failure or shutdown interruption. GPU inference remains serial on its owning thread. The pipeline detects landmarks in the original grayscale images replicated into RGB channels, without image normalization or fisheye remapping, corrects landmark coordinates for stereo reconstruction, matches unambiguous same-handed candidates, rejects excessive reprojection or implausible depth, and derives the palm pose from triangulated landmarks. Gesture recognition uses agreement between the two image-space finger estimates. MediaPipe world landmarks are not treated as headset-relative positions.

The public header does not document the coordinate basis of the camera-rate pose callback. Its pose values are therefore not used as canonical head transforms. A bounded history of polled OpenGL poses supplies the capture-time transform. Source-to-host clock offsets must first stabilize over at least 16 observations; callback timestamp agreement and a nearby canonical head sample are required. Hand inference starts without waiting for the matching VIO callback, then checks for that exact camera timestamp before reconstruction can publish a valid pose. Missing confirmation, changed connections/tracking frames, invalid head tracking, and hand observations older than 200 ms still discard the result. Calibration and preview-only acquisition retain their bounded pose wait. Receipt-based offset estimation cannot establish absolute camera latency. External motion-to-visible-controller measurement is still required. Increasing expiry can reduce disappearing controllers at the cost of displaying older poses; it does not resolve GPU device loss or establish the latency/performance gates.

Stereo validation retains the original 3-pixel reprojection, positive-depth, and 15–150 cm depth checks per landmark. Bare-hand controller position and orientation require a valid triangulated wrist and all four palm knuckles (0, 5, 9, 13, 17). Joy-Con mode reconstructs wrist landmark 0 only and takes orientation and inputs from the paired controller. Live pinch and grip no longer require trustworthy fingertip depth: each image estimates palm-normalized pinch and middle/ring/little finger curl, then both views must agree. Grip combines fingertip reach and finger-chain bending, with separate press/release thresholds and three curled fingers to press versus two to retain a press. Invalid, degenerate, or conflicting image evidence releases only its affected gesture and requires an observed release before rearming. A valid 3D palm is never sufficient by itself to invent a grab or click. Calibration and distance validation still require every board point to pass. Conservative stereo association rejects ambiguous hands rather than inventing depth. Crossed hands, edge-on palms, and occlusion may reduce availability. The 2D gesture consensus and aiming still require physical testing through grabs, pulls, and releases; they do not establish an accuracy or latency pass. No skeleton output, desktop gestures, fast-action guarantee, or hands-only locomotion is provided.

Diagnostics include automatic or manual half/full input rates and a Lite-model comparison. Choices reset with the tracking session; the default is full-model parallel CPU with scheduled discovery and an adaptive rate starting at 500 permille. The Lite bundle uses Google's Lite palm detector and Lite landmark model with matching Tasks metadata. Both bundles are checksum-pinned in Gradle and at runtime; `tools/build_hand_lite.py` reproduces the candidate from pinned inputs. Saved-pair CPU replay compares full-model two-hand inference, scheduled full, scheduled lite, then full-model two-hand inference again, all with concurrent cameras. Rows identify `parallel` and `scheduled`; samples identify `inferenceHandLimit`. Its timings exclude model construction and cannot establish live detection coverage, external latency, second-hand discovery delay, or game performance.

## Combined release gates

| Area | Required evidence |
|---|---|
| Translation | Measured right/up/back movement and approximately 1:1 meter scale; lean and crouch; no double application |
| Origins | Crouched recenter preserves floor; picture switching preserves independent screen and VR references |
| Screens | Anchoring while moving sideways, toward and away; stable output after crossing the screen plane |
| Hands | At least 95% valid slow unobstructed observations at 30–80 cm; position error at most 5 cm; stationary jitter at most 2 cm |
| Latency | External physical-motion-to-visible-controller p95 below 100 ms |
| Recovery | Immediate release on detected loss; observation expiry extended to 200 ms by user request; clear-hand reacquisition within 500 ms |
| Performance | At most 10% regression in p95 game frame time over 20 minutes, with baseline frame rate maintained |
| Gameplay | Dashboard selection and Angry Birds VR aim, grip, pull, release |
| Robustness | Crossed hands, partial occlusion, head motion, lighting, USB reconnect, thermal load |

Record results in `luma-spatial-evidence.md`. A build or head-tracking pass alone does not satisfy this combined gate.