Merge remote-tracking branch 'upstream/main' into wallentx/game-debug-logging
No files matched your search
@@ -185,6 +185,145 @@ there stays behind the client's loading screen however healthy its window looks
|
||||
server's windows with map state, override-redirect, app id, `WM_STATE`, pid, class, opacity and
|
||||
`_NET_WM_STATE`.
|
||||
|
||||
## Hand tracking diagnostics
|
||||
|
||||
With debugging tools enabled, the glasses provider accepts `spatial-hand-options` with optional
|
||||
boolean `lite`, `halfRate`, `parallel`, `adaptive`, and `scheduled` fields. These are temporary
|
||||
tracking-session controls and do not change the CPU/GPU delegate. `parallel` selects concurrent
|
||||
camera inference on CPU; GPU remains serial on its owning thread. `scheduled` enables persistent
|
||||
one-hand and two-hand detector banks on CPU, using the one-hand bank after stable agreement
|
||||
and periodically searching for another hand. GPU retains ordinary two-hand inference.
|
||||
`adaptive` lets CPU input rise from 500 permille to 600 or 800 only after sustained processing
|
||||
and observation-age headroom. Slow processing lowers the rate, and tracking loss resets it.
|
||||
Explicit `halfRate` selects manual input and disables adaptation; if both fields are supplied,
|
||||
`adaptive` is applied afterward. This allows restoring the manual preference and automatic mode
|
||||
in one request. CPU, full model, parallel inference, scheduled discovery, and automatic input
|
||||
starting at 500 permille are the defaults.
|
||||
|
||||
`handModel`, `handCpuParallel`, `handAdaptiveInput`, and `handScheduledTracking` identify the live
|
||||
selection. `handInputRatePermille` reports the current requested frame fraction, while
|
||||
`cameraCopyPermille` records the fraction used for the latest reported camera processing sample.
|
||||
At a 25 Hz source, 500/600/800 correspond to input limits of about 12.5/15/20 Hz, not completed
|
||||
tracking throughput. Calibration and explicit capture use 1000. `cameraFrameDivisor` remains
|
||||
2 for 500, 1 for 1000, and 0 for intermediate fractions; use `cameraCopyPermille` for rate calculations.
|
||||
`cameraHz` still describes the source cameras; `rateLimitedFrames` counts deliberate skips and
|
||||
`dropped` counts eligible pairs replaced while processing is busy. These controls do not change
|
||||
the glasses' sensor rate or native head tracking.
|
||||
|
||||
`spatial-hand-capture`, with debugging tools enabled, requests one local stereo pair in
|
||||
`files/spatial/captures`. Use only for an explicitly requested diagnostic capture; it includes
|
||||
the wearer's surroundings. It does not upload images.
|
||||
|
||||
`spatial-rgb-capture` requests one local JPEG from the Luma Ultra's separate UVC RGB camera.
|
||||
It requires Android camera permission, USB permission for that camera, and debugging tools.
|
||||
If USB permission is requested, approve it and explicitly repeat the capture request; permission
|
||||
approval alone does not save an image. Poll `spatial-rgb-status` for completion and the local
|
||||
`files/spatial/rgb-captures` path. The isolated worker stops the stream after the requested
|
||||
diagnostic window or a timeout, then exits. The default `transport: "sdk"` uses the vendor's
|
||||
fixed 1920×1080/30-fps stream. With the optional UVC build present, `transport: "uvc"` selects
|
||||
the independent eight-buffer transport; `width`, `height`, `fps`, and `durationMs` default to
|
||||
640, 480, 30, and 5000. Duration is bounded to 1000–30000 ms and camera modes must be supported
|
||||
by the device. This remains a manual debugging command, not a source of live controller positions.
|
||||
Both paths save `usb.log` and `usb-descriptors.bin`, including when acquisition fails.
|
||||
Direct UVC reports `snapshotAcquired` separately from `streamHealthy`. Health requires the
|
||||
full requested observation window, at least 80% of requested FPS, and no frame gap exceeding
|
||||
the larger of 250 ms or three frame periods. `observedFps`, `maxFrameGapMs`, and
|
||||
`tailFrameAgeMs` include silence at the end of the window. Status remains busy with
|
||||
`phase: "finishing"` until the capture worker exits, so a new request cannot race its teardown.
|
||||
`receiptNs` is the host callback time, not camera exposure time; the SDK timestamp can be zero.
|
||||
This diagnostic does not supply controller positions or replace the stereo tracking stream.
|
||||
|
||||
`spatial-hand-benchmark` takes a numeric `capture` directory name from the app's existing
|
||||
`files/spatial/captures` folder. It replays that saved stereo pair on CPU with full-model two-hand
|
||||
inference, scheduled full, scheduled lite, then full-model two-hand inference again, all with
|
||||
concurrent cameras. It clears live hand input during the comparison and never captures new images.
|
||||
Poll `glasses.spatial.handBenchmark` for completion, timing, and geometric results. Every replay
|
||||
row reports `parallel` and `scheduled`. `wristOnly` identifies the selected controller mode: bare-hand
|
||||
replay estimates `hands`, while Joy-Con replay estimates `joyCons` from wrist 0. Both report
|
||||
`wrists` after stereo geometry checks. Image preprocessing is raw grayscale replicated into RGB;
|
||||
lens correction applies once to landmark coordinates before triangulation. `wrists` reports landmark 0 after the existing stereo depth
|
||||
and reprojection checks, with the same unambiguous handedness pairing. Live Joy-Con tracking
|
||||
also uses wrist 0 directly, with motion and freshness checks; its `positionEstimator` is
|
||||
`joycon-wrist`. Samples identify the selected bank as `inferenceHandLimit`
|
||||
(1 or 2). Timings exclude model construction. Saved-pair results are not live accuracy,
|
||||
second-hand discovery delay, game performance, or motion-to-display latency measurements.
|
||||
|
||||
The live baseline is Full palm + Full landmarks. `handImagePreprocessing` is `raw-grayscale-rgb`;
|
||||
`handLensCorrection` reports `pinhole-points-only`, `fisheye-points-only`, or `none`.
|
||||
`steadyHandAim` is true only when bare hands and the optional shoulder-based aim setting are active.
|
||||
|
||||
`handTracking` includes `leftRawHands`/`rightRawHands` before handedness-confidence filtering,
|
||||
`previousLeftRawHands`/`previousRightRawHands` from the previous inference on those detectors
|
||||
(-1 after detector creation), and `leftInferenceMs`/`rightInferenceMs`. Existing
|
||||
`leftCameraHands`/`rightCameraHands` count confident handedness results. Compare camera timing
|
||||
against its previous raw count: the stock two-hand graph may repeat palm search when fewer
|
||||
than two hand regions were tracked. `inferenceHandLimit` identifies the bank that produced these
|
||||
counts; previous counts belong to that same persistent bank, not necessarily the preceding camera
|
||||
frame. These counts approximate graph conditions; they do not directly expose internal graph
|
||||
execution. Replay samples also include raw counts and camera timings. Accepted palms still
|
||||
require stereo depth, while pinch and grip validity now use agreement between image-space
|
||||
finger estimates from both cameras. Palm acceptance alone does not establish valid gesture input.
|
||||
|
||||
`handTracking.experimental` is always true. `observationTimeNs` identifies the processed camera
|
||||
observation; `observationAgeMs` is its age when diagnostics were published. Per-side `left` and
|
||||
`right` records expose `outcome`, `reconstructedPalmLandmarks`, raw trigger/grip estimates,
|
||||
geometry validity, and final input validity. `observations` contains connection-lifetime
|
||||
`processedPairs`, `inferencePairs`, and independent left/right acceptance, rejection-cause,
|
||||
tracking-loss, and gesture-loss counters. Each processed camera sequence is counted once;
|
||||
repeated status polling does not inflate these counters. Setup/off/clock/head failures count
|
||||
as processed pairs but not completed inference pairs. Skipped callbacks are not fabricated as
|
||||
observations. Sample deltas over a controlled hand-visible interval and inspect rejection
|
||||
causes before changing thresholds; these counters do not prove external latency or accuracy.
|
||||
|
||||
`spatial.calibrationSource` is `device`, `luma-ultra-base`, or `none`.
|
||||
`calibrationVerified` indicates an individual distance-validated profile; `calibrated` indicates
|
||||
geometry is available for experimental tracking, including the bundled model base. A shared base
|
||||
does not claim validation on every Luma Ultra. Matching local profiles take precedence.
|
||||
|
||||
`glasses set joyConTracking true` enables experimental Luma stereo-assisted Joy-Cons after
|
||||
positional tracking is enabled. It defaults off, excludes bare-hand controllers, and appears in
|
||||
both saved settings and spatial diagnostics. Driver logs identify each hand's transition between
|
||||
`stereo hand` and `6DoF arm estimate`; camera positions are hand proxies, while Joy-Con sensors and
|
||||
buttons supply orientation and input.
|
||||
|
||||
The shared-memory channel is protocol **4**, still 176 bytes. Snapshot flag bit 2 selects Luma
|
||||
Joy-Con tracking; bit 1 selects bare-hand controllers. Its word at byte offset 36 is
|
||||
`input_epochs`: four unsigned bytes in left-trigger, left-grip, right-trigger, right-grip order.
|
||||
Each byte increments modulo 256 when that gesture becomes ineligible. Inspect these counters
|
||||
when a loss may have happened between two reader samples: a later valid pose does not erase
|
||||
the intervening loss. Versions 1 through 3 are rejected, so a decoder, app, and driver must agree
|
||||
on version 4. Hand observation age remains limited to 200 ms, head age to 100 ms. For bare-hand
|
||||
controllers, a gap beyond
|
||||
200 ms from the driver's last accepted observation releases input state even when the next
|
||||
read contains a fresh replacement. Duplicate observations cannot rearm an input; recovery
|
||||
requires a newer valid released gesture before a subsequent press.
|
||||
|
||||
Driver lines containing `left optical input` or `right optical input` report state changes,
|
||||
changed connection/origin/loss counters, and changed input API errors, rather than every frame.
|
||||
`pose` is the displayed pose's validity and can remain true during its short held-pose window;
|
||||
`input` is the current observation's eligibility for input processing. Read `flags` for independent
|
||||
gesture validity. `raw` contains trigger strength/grip state; `out` contains submitted trigger
|
||||
strength/click/grip; `armed` and `epochs` are trigger/grip pairs. `error` is ordered **grip, trigger
|
||||
click, trigger scalar**, using OpenVR's `EVRInputError` codes (0 means `VRInputError_None`).
|
||||
The line also includes `ageMs`, `connection`, and `origin`. `controller inputs cleared: error=...`
|
||||
records neutralization on mode changes, disconnect, or deactivation; a nonzero value is the last
|
||||
failed component update in that clearing pass.
|
||||
|
||||
The driver publishes these input updates before the controller pose, including invalid poses,
|
||||
and clears components before deactivation. A zero API error establishes successful submission,
|
||||
not delivery of a dashboard mouse-up event. The installed Vive compositor binding maps trigger
|
||||
to dashboard `LeftClick` (0.65 activation, 0.60 release) and grip to `Back`. Its grab-handle
|
||||
JavaScript attaches to the initiating hand on mouse-down and clears that attachment on mouse-up;
|
||||
that component has no tracking-invalid cleanup. A stuck dashboard drag therefore needs the
|
||||
driver submission and actual dashboard behavior checked separately. Do not synthesize a press
|
||||
or a valid pose as a recovery shortcut.
|
||||
|
||||
Controller display smoothing is independent of these gesture logs: there is no stationary
|
||||
dead zone. Poses converge with one 15 ms correction and bounded receipt-relative translation
|
||||
prediction (80 ms / 2 cm). A missed observation holds the last pose for up to 350 ms from capture;
|
||||
input freshness remains limited to 200 ms, and a detected miss releases inputs immediately.
|
||||
Neither stabilization nor prediction refreshes camera timestamps or creates gesture input.
|
||||
|
||||
## Events
|
||||
|
||||
Each session writes `events.jsonl` beside its other artifacts: the lifecycle (`session.created`,
|
||||
|
||||
@@ -162,6 +162,11 @@ Enable **3D (side-by-side)** manually or opt into **3D for VR games** in the gla
|
||||
switching waits until SteamVR reports that it is running; starting or installing it does not change
|
||||
the display mode. A saved **3D for VR games** preference is retained.
|
||||
|
||||
While SteamVR runs, its side-by-side source takes priority over the glasses' saved picture mode,
|
||||
including **3DoF virtual screen**. SteamVR already applies head tracking; projecting its combined
|
||||
eye pair onto a virtual screen would show both images in each eye and apply head rotation again.
|
||||
The glasses still supply poses to SteamVR. The saved picture mode resumes when SteamVR stops.
|
||||
|
||||
With **Game on glasses** off, SteamVR keeps the glasses' picture settings and IMU while Android
|
||||
mirrors the phone output to the glasses. Turning off the dedicated display must not turn off the
|
||||
renderer/HMD proximity flag: SteamVR otherwise reports the headset as taken off and enters standby,
|
||||
@@ -183,6 +188,32 @@ the presenter can keep presenting an old SteamVR image while the compositor repo
|
||||
|
||||
### Joy-Con controllers
|
||||
|
||||
**Glasses → Tracking → Experimental → Luma Joy-Con tracking** is off by default. Enable
|
||||
positional tracking first, then this toggle. It uses the Luma Ultra primary stereo camera pair
|
||||
to locate each hand holding a Joy-Con, with the existing calibrated, capture-time head transform.
|
||||
Joy-Con IMU orientation, buttons, sticks, battery reporting and haptics remain in use. This mode
|
||||
also enables Joy-Con takeover for SteamVR even if the ordinary controller preference is off.
|
||||
Android 12+ and controller IMU support are still required.
|
||||
|
||||
Stereo positions and the 6DoF head share the same recentered floor origin. If a hand is hidden,
|
||||
invalid, or its observation is older than 200 ms, that side blends to the arm model anchored to
|
||||
the current head position. Reacquisition also blends (180 ms time constant). No usable calibration
|
||||
or visible palm means arm-model-only positioning. Loss of head tracking beyond 100 ms invalidates
|
||||
the controllers and releases their inputs. Reconnects, origin changes and Joy-Con recentering discard
|
||||
old positional corrections. Point the Joy-Cons forward when connecting or recentering; camera
|
||||
positions do not correct IMU yaw drift. Position extrapolation is disabled in this mode.
|
||||
|
||||
The camera observes the gripping **hand**, not the Joy-Con shell. Occluded palm landmarks, crossed
|
||||
hands, and handedness ambiguity can reduce tracking availability. This is an experimental hand
|
||||
position proxy with estimated fallback, not marker-based controller tracking. Controller grip offset,
|
||||
aiming, motion latency, and sustained performance still need physical validation with held Joy-Cons.
|
||||
|
||||
Luma Joy-Con tracking and stereo bare-hand controllers are mutually exclusive. Either selected
|
||||
Luma controller mode pauses phone-camera inference. External VR controllers retain priority.
|
||||
Disable both Luma controller toggles to return to the upstream Joy-Con/device-camera modes,
|
||||
which retain their existing fixed head-position anchor. Positional tracking, stereo controllers,
|
||||
and phone-camera controls are grouped at the bottom of the glasses settings under **Experimental**.
|
||||
|
||||
Nintendo Joy-Cons paired over Bluetooth become SteamVR hands while SteamVR runs, and stay an ordinary
|
||||
game pad otherwise. It needs Android 12 or later (motion sensors on an `InputDevice`) and a kernel whose
|
||||
`hid-nintendo` driver reports the IMU (Android common kernel `android14-6.1` or newer; older 5.x GKI
|
||||
|
||||
@@ -87,6 +87,41 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- **Memory.** Steam, SteamVR and one VR game fit in the Thor's 12 GB. A large game such as The Lab, or
|
||||
two games at once, push the device into critical pressure, and the low-memory killer takes
|
||||
DroidDeck itself (`ended-without-teardown.txt` in the session logs, `lowmemorykiller` in logcat).
|
||||
- **Adreno 840 forced sysmem.** On the Fold with VITURE Luma Ultra, launching SteamVR from the library
|
||||
inherited `TU_DEBUG=sysmem` and produced `vkerror=-4`, triggering recovery. Adding `noubwc` still
|
||||
failed; replacing the flags with `noubwc` or an empty value worked. The shared launch helper removes
|
||||
only `sysmem` on `Adreno840*`, leaving other flags, the parent Steam session and other GPUs unchanged.
|
||||
The user confirmed improved landing-area brightness and colors with this configuration and asked
|
||||
to preserve them.
|
||||
- **Adreno 840 with DD-Turnip v0.2.0.** SteamVR reproduced `vkerror=-4` with `sysmem` removed,
|
||||
including on the previous app build and with the presenter disabled. After a phone reboot,
|
||||
the unchanged `fcf8c588` baseline rendered for 109 seconds without observed device loss.
|
||||
`TU_DEBUG=flushall` initially appeared promising: a global run rendered for over a minute,
|
||||
and a SteamVR-only run rendered for about 32 seconds before background suspension. The
|
||||
packaged SteamVR-only change then reproduced device loss with `flushall` confirmed in
|
||||
`vrcompositor`, so the automatic flag was removed. These bounded runs do not establish the
|
||||
underlying cause or a reliable workaround.
|
||||
- **Live files are not cache.** Android's `installd` evicted `cache/shm/u10528-*` while SteamVR was
|
||||
running, followed by Chromium shared-memory failures and lost IPC. Shared memory, emulated device
|
||||
files, sandbox trees and PRoot temporary files now use `no_backup/guest-runtime`, retaining their
|
||||
existing cleanup lifetimes. Session state must survive cache eviction.
|
||||
- **One host signal owner.** The submit-wait path must not mirror a shared timeline value itself:
|
||||
the poller can signal it between the submit thread's counter read and signal, producing an invalid
|
||||
duplicate signal. The poller now owns mirroring; a forced-interleaving test covers this race.
|
||||
- **Lost hand tracking is not a disconnect.** Publishing a disconnected pose and then a connected,
|
||||
untracked pose on every optical update floods SteamVR with device-change notifications. With
|
||||
uncalibrated hands this grew its web buffer to about 34 MB and triggered the Web thread watchdog.
|
||||
Release inputs and publish only the connected, untracked pose; reserve disconnection for an
|
||||
actual controller removal. The controller regression exercises repeated loss and reacquisition.
|
||||
- **Failed realtime scheduling can still install a CPU limit.** On the Fold, SteamVR's SDL RTKit
|
||||
fallback set `RLIMIT_RTTIME` to 100/200 ms despite realtime priority being refused. The render
|
||||
thread later received kernel `SIGXCPU` during `IOCTL_KGSL_GPU_COMMAND`, with policy `SCHED_OTHER`
|
||||
and the soft limit advanced to 1100 ms, confirming the realtime watchdog fired. The VR scheduling
|
||||
preload rejects lowering this limit only in `vrcompositor` when `RLIMIT_RTPRIO` is zero and
|
||||
`CAP_SYS_NICE` is absent. It preserves inherited limits, other processes and other resources;
|
||||
it does not intercept signals. SDL's scheduling hint cannot cover SteamVR's direct
|
||||
`SDL_LinuxSetThreadPriorityAndPolicy` call. A native preload regression covers rejection and
|
||||
forwarding. The exact Android kernel boost/accounting path was not established.
|
||||
- **Frames shown out of order.** Video from the glasses jumped back a few frames in bursts while the
|
||||
head turned. Two causes, each found by stamping a frame counter into the presenter's output:
|
||||
- the presenter copied SteamVR's eye texture a fixed 4 ms after `Present`, with no fence. Under
|
||||
@@ -106,6 +141,20 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
device x. Getting this wrong made a turn at the hips tilt the view. Recentring now keeps the world
|
||||
vertical and resets the heading only.
|
||||
|
||||
## Library launch validation (2026-10-15)
|
||||
|
||||
Two hands-off launches through the Steam library's Launch button completed 308 seconds and
|
||||
188 seconds, with an intentional stop between runs. Each retained its compositor process and
|
||||
session, sustained 60 fps, showed dashboard/world transitions, and kept USB control and valid
|
||||
6DoF head tracking. Neither run reported device loss, a watchdog abort, recovery, or a session
|
||||
restart. Hand controllers remained enabled but uncalibrated. Brightness stayed at 8; no shader
|
||||
color changes or temporary graphics overrides were used.
|
||||
|
||||
These timed runs used APK SHA-256
|
||||
`4ce7a623ceb6b1a100c0ed1184530ece2ea8e13a1cb9ee6c0ab24666a8249004`. A newer calibration/recenter
|
||||
APK was subsequently installed and verified; see `luma-spatial-evidence.md`. The timed results
|
||||
do not establish sustained gameplay or calibrated-hand acceptance for that newer build.
|
||||
|
||||
## Debugging a live session
|
||||
|
||||
- **Logs:** `~/.local/share/Steam/logs/{vrserver,vrcompositor,xrclient_*}.txt` and
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# Tracked DroidDeck menus
|
||||
|
||||
When glasses output and the tracked virtual screen are enabled, DroidDeck mirrors its own foreground menus into a glasses `Presentation`. Phone controls remain on the phone. The menu renderer does not start the guest compositor, request MediaProjection, or capture another application's windows.
|
||||
|
||||
`GlassesMenuCapture` draws the activity's visible view roots into a hardware Canvas at up to 30 fps. `GlassesMenuSurface` consumes that buffer as an external-OES texture and redraws the screen at the glasses refresh rate. The plane geometry follows the existing `cardboard.frag`: row-major head-to-anchor rotation, metres for translation, +Y up, -Z forward, and a top-left source image. Mono uses a central eye; SBS renders the full UI to both eyes with 64 mm eye separation.
|
||||
|
||||
The pose adapter keeps its 3DoF reference across menu pages and uses the shared spatial **screen** origin for 6DoF. Menu recenter does not change the VR floor origin. Stale or invalid tracking holds the last safe view. Explicitly changing between 3DoF and 6DoF can reanchor the screen because those modes have separate origins; recenter restores the chosen mode's view.
|
||||
|
||||
On Android 29 and later, capture includes visible app dialog/popup roots exposed by `WindowInspector`. Android 26–28 captures the activity decor only, so separate dialog and popup windows are not mirrored. System keyboards, system permission prompts, other apps, and separately composited `SurfaceView` content are outside this capture path. Secure windows are excluded. Phone interaction remains available for these cases.
|
||||
|
||||
## Shader regression check
|
||||
|
||||
The opt-in Python check compiles the actual embedded GLES shaders, imports an asymmetric texture through EGLImage/external-OES, renders it, and checks readback pixels. It covers image orientation, both SBS eyes, translation, yaw, and views behind the screen. It does not require Python packages beyond the standard library and is not part of default pytest discovery.
|
||||
|
||||
On Debian/Ubuntu with Python 3:
|
||||
|
||||
```sh
|
||||
sudo apt-get install libegl1 libegl-mesa0 libgles2 libgl1-mesa-dri
|
||||
python3 tools/check_glasses_menu_shader.py
|
||||
```
|
||||
|
||||
Run from any working directory by specifying the script's path. `--output-dir PATH` overrides the default `build/glasses-menu-shader`. The check defaults to headless Mesa software rendering and GLES 2.0. It writes `menu-shader-validation.json` with the shader hashes, GL version, and pixel comparisons, plus `menu-shader-mono.png` and `.ppm`.
|
||||
|
||||
The test supplies the Android consumer's vertical-flip texture transform. It verifies the real shader under that transform; it does not replace checking Android's actual Canvas → SurfaceTexture producer/consumer integration or display orientation on the phone.
|
||||
|
||||
## Phone and glasses smoke test
|
||||
|
||||
1. Connect supported glasses, open DroidDeck's setup page, and enable glasses output and the tracked virtual screen. Confirm the phone remains touchable and text in the glasses is upright with the same left/right ordering.
|
||||
2. Switch among Steam and non-Steam launcher pages without launching a game. Scroll, open menus and app dialogs, and change a setting from the phone. Check that the glasses follow the current page and retain the screen's head-tracked anchor.
|
||||
3. In 3DoF, turn and tilt the glasses, then use Recenter. Test mono and SBS. Each SBS eye must contain the full UI with matching vertical alignment.
|
||||
4. Enable 6DoF on supported hardware and complete its setup. Translate left/right, up/down, and toward/away from the plane. Recenter the menu and verify that VR floor height has not changed. Briefly obscure tracking; the held menu must remain finite and stable until tracking recovers.
|
||||
5. Launch a game, open and close the session menu, then return to the launcher. Confirm clean ownership handoff between the game output and menu presentation and that phone-only operation still works.
|
||||
6. Background/foreground DroidDeck, rotate the phone, change the glasses display mode, and unplug/replug the glasses. Check for stale frames, black output that persists after reopening, crashes, or a reused orientation from the previous device connection.
|
||||
|
||||
For evidence, `tools/droiddeckctl displays` lists logical display IDs and `tools/droiddeckctl screenshot ./glasses.png --display <id>` captures the selected output. Inspect `GlassesMenu` log messages for capture or renderer failures. A display screenshot verifies image content; head movement and stereo comfort still require the physical glasses.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Hybrid hand inference benchmark
|
||||
|
||||
Measured on the connected Samsung SM-F976U on 2026-10-17. This sparse-correspondence
|
||||
prototype did not improve useful hand throughput. It was slower than scheduled stereo
|
||||
on the central open-hand capture and accepted palms only on its dual-detector recovery
|
||||
frames. The earlier estimated 20–60% throughput improvement is not supported by this test.
|
||||
|
||||
Turning image lens correction off did not improve accepted-palm counts. The saved
|
||||
calibration uses the **pinhole** distortion model, not OpenCV's fisheye model. The benchmark did not change its diagnostic build's live algorithm or input-rate policy,
|
||||
and restored the prior settings after each sweep. That build ended with Lite selected.
|
||||
The shipping baseline was subsequently set to Full with raw grayscale-to-RGB input.
|
||||
The failed hybrid prototype and its runners have been removed; these results are historical.
|
||||
|
||||
## Central open-hand result
|
||||
|
||||
The table pools the initial and final central-scene runs in the lens on/off sweep.
|
||||
Each stereo entry combines the baselines before and after the hybrids. Times include
|
||||
per-frame image preparation, inference, matching when applicable, reconstruction, and
|
||||
palm/gesture estimation. Acceptance means at least one geometrically accepted palm in
|
||||
a repeated saved image, not live detection accuracy. This scene contains two open hands;
|
||||
Full consistently recovered only one.
|
||||
|
||||
| Full-model path | Image lens correction | Median / p95 processing | Frames with a palm | Sparse frames with a palm |
|
||||
|---|---|---:|---:|---:|
|
||||
| Scheduled stereo | On | 30.15 / 61.60 ms | 160 / 160 | — |
|
||||
| Left-led hybrid | On | 62.59 / 84.06 ms | 48 / 80 | 0 / 32 |
|
||||
| Right-led hybrid | On | 40.58 / 80.21 ms | 48 / 80 | 0 / 32 |
|
||||
| Scheduled stereo | Off | 30.51 / 61.18 ms | 160 / 160 | — |
|
||||
| Left-led hybrid | Off | 62.57 / 79.82 ms | 48 / 80 | 0 / 32 |
|
||||
| Right-led hybrid | Off | 61.75 / 83.05 ms | 48 / 80 | 0 / 32 |
|
||||
|
||||
All successful hybrid frames came from ordinary stereo inference. Correspondence failed
|
||||
to reconstruct the five required palm anchors; the original palm gates were retained.
|
||||
The first corrected-image trial measured about 19–21 ms for rectification and matching
|
||||
alone. Repeated failures also prevented the hybrid from sustaining the cheaper one-hand
|
||||
tracker schedule. The live input limit should not be raised on the strength of this candidate.
|
||||
|
||||
Lite accepted no palms with either lens setting or either architecture on the lens-sweep
|
||||
captures. The peripheral open hand and representative controller grip also yielded no
|
||||
accepted palms. The no-hand control remained negative. The earlier corrected-image sweep
|
||||
covered all nine saved pairs, including six controller grips, with no extra palms recovered
|
||||
by the hybrid. These are a small set of static scenes, not independent accuracy trials.
|
||||
|
||||
## What was measured
|
||||
|
||||
`HybridHandBenchmark` was reachable only through the saved-capture diagnostic. Both models
|
||||
use CPU VIDEO inference, the existing 0.6 thresholds, and the existing one-/two-hand
|
||||
schedule. Stereo uses concurrent camera inference. The hybrid runs one detector, rectifies
|
||||
both images, and searches for each landmark along its epipolar line using 11 × 11 normalized
|
||||
correlation patches. It requires correlation at least 0.80, a 0.04 uniqueness margin outside
|
||||
three pixels, patch standard deviation at least four, and forward/backward agreement within
|
||||
1.5 pixels. Search depth spans 0.15–1.5 m. Existing triangulation and palm/gesture checks
|
||||
remain in force. This is one concrete hybrid candidate, not a test of every stereo matcher.
|
||||
|
||||
Both detectors run every fifth frame and on the frame after failed sparse palm estimation.
|
||||
The hybrid's recovery costs and failures are included in the results. Each configuration
|
||||
has a separate cold frame, five warmups, and forty measured frames. Synthetic VIDEO
|
||||
timestamps advance by 50 ms, but processing runs unpaced; the experiment did not measure
|
||||
a live 20 Hz stream. Initialization, file decoding, camera transfer, head synchronization,
|
||||
pose publication, and rendering are excluded. Replayed poses are never published.
|
||||
|
||||
Contrast normalization and grayscale-to-RGB conversion stay enabled in both lens cases.
|
||||
With input lens correction off, detection runs on raw images; landmarks are still corrected
|
||||
using calibration for triangulation. The hybrid always uses corrected, rectified images
|
||||
for correspondence and transforms matched points back to the detector's image coordinates.
|
||||
This comparison does not disable the geometric calibration required for metric depth.
|
||||
|
||||
The lens sweep tested the central scene, peripheral scene, no-hand control, one grip scene,
|
||||
and then the central scene again. Full and Lite each ran stereo-before, left-led hybrid,
|
||||
right-led hybrid, and stereo-after for both lens settings: 3,200 measured replays. The
|
||||
separate initial corrected-image corpus sweep also recorded 3,200 replays across nine
|
||||
unique pairs, repeating the central scene last. Repetition is for timing, not extra accuracy
|
||||
samples. SteamVR was idle and live tracking was temporarily disabled during these runs.
|
||||
|
||||
## Validation and limits
|
||||
|
||||
The Android build and 33 focused JVM tests passed. On the phone, the matcher passed fifty
|
||||
synthetic correspondences in both directions and rejected textureless synthetic images
|
||||
before each capture benchmark. These checks establish basic coordinate/search behavior;
|
||||
they do not establish matching accuracy on real hands.
|
||||
|
||||
Android thermal status was 0–1 during the corrected-image sweep and 1 during the lens
|
||||
sweep. Timings varied substantially within the fixed configuration order, so the on/off
|
||||
timing difference cannot be attributed solely to lens correction. Full stereo medians in
|
||||
the first lens-sweep central trial ranged from 16.7 to 30.8 ms; the final trial was slower
|
||||
with correction enabled. Baselines bracket the hybrid cases, but the run is not randomized
|
||||
or temperature-controlled.
|
||||
|
||||
The initial corpus's final Full/stereo-before trial included a 2,611.74 ms inference stall.
|
||||
It remains in the raw samples and means; p95 alone hides that isolated stall. Geometric
|
||||
acceptance does not apply live capture-age gates, so that observation would be stale in
|
||||
live tracking. The later central lens-sweep samples had no processing times above 200 ms.
|
||||
No moving-hand, pinch/release, sustained SteamVR throughput, external latency, or independent
|
||||
position-accuracy pass is claimed. The present matcher needs better correspondence and
|
||||
lower cost before a live increased-input-rate trial is justified.
|
||||
|
||||
## Reproduction and evidence
|
||||
|
||||
The temporary runner and matcher are no longer shipped. Local evidence from those runs:
|
||||
|
||||
- Final lens comparison and raw samples: `build/hand-hybrid/lens/` and its `summary.json`.
|
||||
- Initial full corpus: `build/hand-hybrid/corpus/` and its `summary.json`.
|
||||
- First prototype trial: `build/hand-hybrid/first/`.
|
||||
- Build and focused tests: `build/hand-hybrid-validation.log`.
|
||||
- Lens comparison build: `build/hand-hybrid-lens-build.log`.
|
||||
- Installed lens-benchmark APK SHA-256: `d976329b96e6aaceec62ef38085e203942cf763c51102adfeb50c4e5614891f1`.
|
||||
- Calibration SHA-256: `3983438843877a321d3098f73496dfd9256f0088f095aa8d591602cf0b2cb213`.
|
||||
|
||||
Each result also includes both capture-image hashes and the model hashes. Build-directory
|
||||
evidence is local and may not be present in a fresh checkout.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Full versus Lite hand models and image preprocessing
|
||||
|
||||
The saved-image comparison favors the Full palm detector and Full landmark model.
|
||||
Neither contrast normalization nor lens correction improved usable bare-hand detections
|
||||
on the two available bare-hand scenes. This is a small static corpus, not a live tracking pass.
|
||||
|
||||
## Method
|
||||
|
||||
The same saved 640 × 480 grayscale stereo pairs were evaluated with eight configurations:
|
||||
Full or Lite palm/landmark bundles, each with raw input, calibrated lens correction,
|
||||
1st/99th-percentile contrast normalization, or both corrections. Grayscale-to-RGB replication
|
||||
and MediaPipe's model-required tensor scaling were enabled in every configuration.
|
||||
|
||||
The main comparison ran on the Samsung SM-F976U using the app's Android MediaPipe Tasks
|
||||
1.0.0 and OpenCV 4.13.0. Each configuration created fresh CPU VIDEO detectors, with two
|
||||
hands per camera, parallel camera inference, and all detection/presence/tracking thresholds
|
||||
at 0.6. Adaptive input and scheduled one-hand inference were excluded. Each pair received
|
||||
one cold inference, four warmups, and twelve measured repetitions at synthetic 80 ms
|
||||
timestamp intervals. Bare-hand mode was verified; wrist-only mode was false.
|
||||
|
||||
This was an offline replay with the game idle and tracking cameras disconnected. It loaded
|
||||
the phone's saved verified calibration and never initialized cameras or published replay
|
||||
poses. Raw-image landmarks were undistorted during triangulation; corrected-image landmarks
|
||||
were triangulated without applying distortion correction again. Both paths retained the
|
||||
existing handedness, depth, reprojection, palm geometry, and motion checks.
|
||||
|
||||
The corpus contains two bare-hand scenes, one no-hand control, and six controller-grip
|
||||
scenes. The central bare-hand scene was also repeated at the end of the phone run. Replaying
|
||||
one pair twelve times does not create twelve independent accuracy samples. Desktop IMAGE
|
||||
and VIDEO tests using MediaPipe 0.10.21/OpenCV 4.11.0 provide a separate cross-check.
|
||||
|
||||
## Bare-hand result
|
||||
|
||||
For capture `64390821246320`, two open hands are visible in both cameras:
|
||||
|
||||
| Image preprocessing | Full: usable stereo palms per replay | Lite: usable stereo palms per replay |
|
||||
|---|---:|---:|
|
||||
| Raw | 1 of 2 | 0 of 2 |
|
||||
| Lens correction only | 1 of 2 | 0 of 2 |
|
||||
| Contrast normalization only | 1 of 2 | 0 of 2 |
|
||||
| Lens correction + contrast normalization | 1 of 2 | 0 of 2 |
|
||||
|
||||
The cold frame and all twelve measured frames agreed. Full found one hand in each eye;
|
||||
Lite found no hands with raw, lens-only, or contrast-only input. With both corrections,
|
||||
Lite found one hand in the left camera and none in the right, so it could not reconstruct
|
||||
a stereo position.
|
||||
|
||||
Capture `72745789186883` contains one open hand, peripheral in the left image. Every
|
||||
configuration missed it in both cameras throughout cold and measured replay. The desktop
|
||||
IMAGE and VIDEO tests agreed with these bare-hand outcomes. The no-hand control
|
||||
`59405518228482` returned zero hands across all configurations.
|
||||
|
||||
The final repeat of the central scene reproduced the same cold and measured counts in
|
||||
every configuration. All six controller-grip pairs (`110854218431877`, `110858058677761`,
|
||||
`110862058272915`, `110935538464606`, `110939418512521`, `110943418461791`) produced
|
||||
zero accepted stereo palms and zero accepted wrists in all measured phone replays. Some
|
||||
Full configurations produced individual camera detections, but these did not become valid
|
||||
stereo positions. Grip scenes are reported separately from the bare-hand comparison.
|
||||
|
||||
## Phone timing
|
||||
|
||||
These are median milliseconds for a parallel stereo pair on the central scene. They include
|
||||
RGB preparation, inference, and result extraction, but exclude decoding, precomputed lens
|
||||
correction, model initialization, and stereo reconstruction. They are not end-to-end latency.
|
||||
|
||||
| Model | Preprocessing | First run | Final repeat |
|
||||
|---|---|---:|---:|
|
||||
| Full | Raw | 46.10 | 82.98 |
|
||||
| Full | Lens only | 75.17 | 84.46 |
|
||||
| Full | Contrast only | 85.64 | 82.99 |
|
||||
| Full | Lens + contrast | 84.10 | 80.49 |
|
||||
| Lite | Raw | 43.07 | 40.06 |
|
||||
| Lite | Lens only | 37.29 | 39.70 |
|
||||
| Lite | Contrast only | 38.73 | 42.54 |
|
||||
| Lite | Lens + contrast | 78.78 | 75.34 |
|
||||
|
||||
Android reported thermal status 2 throughout the saved-pair runs. Configuration order was
|
||||
fixed, and inference work differs when a hand is found. Full/raw's first-versus-last variation
|
||||
also prevents attributing the timing differences to preprocessing alone. Lite's lower timings
|
||||
mostly accompany zero returned hands; its two-camera usable-hand result remains zero.
|
||||
|
||||
## What normalization changes
|
||||
|
||||
The saved calibration uses pinhole radial distortion coefficients (`fisheye: false`). The
|
||||
comparison tests that calibrated lens correction, rather than forcing a fisheye model onto
|
||||
the coefficients. Keeping the original camera matrix and output dimensions narrows the
|
||||
field of view: the left output samples approximately raw x=96–544 and y=56–437, and the
|
||||
right approximately x=93–547 and y=57–440. Some peripheral fingers near raw x=600 disappear
|
||||
from the corrected left image. This is a concrete limitation of the current mapping, although
|
||||
the raw version also failed to detect that hand; the crop alone does not explain all failures.
|
||||
|
||||
Model tensor normalization was already provided by MediaPipe. The additional normalization
|
||||
tested here changes image contrast. Replicating grayscale into RGB satisfies the input channel
|
||||
contract but cannot recover missing color information.
|
||||
|
||||
## Recommendation and limits
|
||||
|
||||
Use Full with raw grayscale replicated into RGB as the next live baseline, retaining MediaPipe's
|
||||
tensor scaling and calibrated point correction for stereo geometry. Full has the stronger
|
||||
detection result here; extra contrast/lens preprocessing has no demonstrated acceptance gain
|
||||
on these scenes. Broader motion testing is still needed before calling any configuration best
|
||||
overall. Full itself still misses one of two central hands and the peripheral hand.
|
||||
|
||||
The comparison used a temporary diagnostic build with opt-in preprocessing and offline replay.
|
||||
That comparison code has been removed after selecting the Full/raw baseline; the live path
|
||||
keeps grayscale-to-RGB conversion and calibrated point correction. Build, twenty focused JVM
|
||||
tests, packaged runtime verification, and the installed Android replay were checked for the experiment.
|
||||
|
||||
Local evidence is under `build/hand-comparison/`: `phone/` contains full per-frame results,
|
||||
thermal snapshots and before/after status; `phone-summary.json` contains compact counts;
|
||||
`desktop/` and `desktop-video/` contain the independent comparisons. The scripts are
|
||||
`compare.py`, `compare_video.py`, `run_phone.py`, and `report_phone.py` in that directory.
|
||||
`input-manifest.json` records all capture, calibration, and model hashes plus runtime versions.
|
||||
@@ -0,0 +1,466 @@
|
||||
# Luma Ultra hand tracking investigation
|
||||
|
||||
Date: 2026-10-17. Status: **experimental; gameplay usability has not passed**.
|
||||
|
||||
## Current Full/raw baseline and hand rendering
|
||||
|
||||
The stereo pipeline defaults to Full palm detection plus Full hand landmarks following the
|
||||
[saved-image comparison](hand-model-comparison-2026-10-16.md). The optional Lite bundle contains
|
||||
Lite palm and Lite landmarks; earlier historical timings describe a different Full-palm/Lite-landmark
|
||||
bundle. `tools/build_hand_lite.py` pins both official Lite source models,
|
||||
checks their input/output tensor contracts, and copies Tasks metadata from the Full bundle.
|
||||
Both the outer task ZIP and the embedded label ZIPs have deterministic timestamps.
|
||||
|
||||
Live inference consumes original grayscale camera bytes, replicated into reusable RGB buffers.
|
||||
It skips full-frame lens remapping, contrast normalization, and the grayscale-to-RGBA-to-Bitmap
|
||||
round trip. Preview bitmaps are created only at preview cadence; grayscale Mats are populated
|
||||
only for calibration, validation, explicit capture, preview, or Joy-Con optical flow.
|
||||
|
||||
Detection, preview overlays, and Joy-Con flow share raw image coordinates. Calibration corrects
|
||||
the sparse landmark coordinates once before triangulation. The saved `fisheye` flag selects
|
||||
fisheye or pinhole correction; the bundled Luma profile uses pinhole coefficients. Calibration
|
||||
fitting, validation, and diagnostic captures retain their original raw images. The unsuccessful hybrid matcher and preprocessing-comparison code were removed
|
||||
after baseline selection. The existing saved-pair diagnostic uses the same raw RGB
|
||||
preprocessing and point correction as live tracking.
|
||||
|
||||
Each grayscale byte is copied into R, G, and B without changing exposure or image coordinates.
|
||||
[MediaPipe Tasks](https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker/android)
|
||||
then performs model-specific tensor normalization from metadata; the app does not divide these
|
||||
image bytes by 255 a second time.
|
||||
|
||||
Joy-Con mode retains only landmark 0 (the wrist), triangulates and follows that position, and
|
||||
does not estimate palm orientation, pinch, or grip. Replay follows the selected controller mode.
|
||||
The landmark network still predicts all 21 landmarks internally. Physical Joy-Con
|
||||
buttons and IMU orientation remain the controller input sources.
|
||||
|
||||
The user reported that the hand smoothing and dead zones made tracking feel worse. The
|
||||
common bare-hand `OpticalPose` path therefore removes the 3 mm position and 1-degree rotation
|
||||
anchors and restores the earlier fixed 15 ms rendering interpolation, replacing the added
|
||||
adaptive smoothing. Small accepted movements can change the rendered pose immediately.
|
||||
Position prediction still requires consecutive velocities to agree in direction, preventing
|
||||
alternating measurement noise from being extrapolated. The normal mode keeps measured palm
|
||||
orientation; only the opt-in steady-aim setting uses `ShoulderAim`. Joy-Con filtering, gesture
|
||||
release, observation freshness, and calibration gates are unchanged.
|
||||
|
||||
SteamVR startup defers Luma Ultra 6DoF until two seconds after the first confirmed RUNNING
|
||||
status. The launch gate applies before SDK initialization, with display controls available
|
||||
through a nonspatial connection during the wait. Repeated RUNNING polls do not extend the
|
||||
deadline. Stop, failed launch, and session cleanup cancel the pending start; reopening always
|
||||
uses the current device and saved tracking settings. This changes startup sequencing, not the
|
||||
tracking preference or hand-model configuration.
|
||||
Automatic restoration on the same device, including a cancelled launch or early stop,
|
||||
requests the existing stable-pose recenter. Status polls from superseded requests or ended
|
||||
sessions cannot restart the pending tracking transition.
|
||||
|
||||
Bare-hand controller rendering holds the last accepted pose for up to 350 ms from its original
|
||||
capture timestamp, extending the previous 200 ms visual limit. Missing, invalid, or stale
|
||||
observations freeze the rendered pose and stop prediction; repeated packets cannot extend the
|
||||
hold. New observations still require the existing 200 ms freshness check, and trigger/grip
|
||||
release immediately when their observations become invalid or stale. Head loss, origin or
|
||||
connection changes, and controller-mode changes clear the visual cache. This bridges short
|
||||
hand-detection gaps without adding a delay to fresh frames; it does not improve detection or
|
||||
preserve a gesture through tracking loss. Both Full and Lite predict the 21 landmarks used by
|
||||
the grab estimator; wrist-only Joy-Con mode uses physical buttons instead.
|
||||
|
||||
Before this rollback, synthetic 12.5 Hz alternating-noise replay reduced position RMS from 4.354 to 1.772 mm and
|
||||
rotation RMS from 1.722° to 1.398°. At 25 Hz, the respective changes were 1.242 to 0.988 mm
|
||||
and 1.058° to 0.800°. The tradeoff in the 12.5 Hz slow-motion case was slightly more lag:
|
||||
maximum position error changed from 0.423 to 0.593 mm and angular error from 0.489° to
|
||||
0.632°. These results describe the reverted adaptive filter, not the current behavior or
|
||||
measured live tracking improvements; the user's subsequent experience favored removing it.
|
||||
The 80 focused JVM tests and native spatial, controller, adaptive-filter, and Joy-Con tests
|
||||
passed. Build and installation evidence is recorded under `build/hand-baseline/`.
|
||||
|
||||
The visual-hold follow-up passed the full native suite, including capture-age boundaries,
|
||||
frozen rendering, stale-packet rejection, immediate gesture release, fresh reacquisition, and
|
||||
cache invalidation. Only the VR driver changed among the 150 staged runtime outputs.
|
||||
Build and installation evidence is recorded under `build/hand-visibility/`.
|
||||
|
||||
The subsequent dead-zone/adaptive-filter rollback passed the full native suite. Regression
|
||||
tests exercise 0.2 mm translation and 0.05-degree rotation immediately after settling, sustained
|
||||
small motion, reversal, and the retained visibility/input protections. Synthetic 95% step
|
||||
response was 44 ms after receipt at both 12.5 and 25 Hz observation rates. This measures the
|
||||
render filter, not end-to-end camera latency. Only the VR driver changed among the 150 runtime
|
||||
outputs; rollback evidence is recorded under `build/hand-filter-revert/`.
|
||||
|
||||
The reviewed startup-delay follow-up passed 129 focused JVM tests, and an isolated snapshot
|
||||
containing only the startup changes passed its 128 focused tests. Earlier phone logs recorded the spatial SDK
|
||||
open attempt 3.348 seconds after confirmed SteamVR RUNNING: the two-second delay plus SDK
|
||||
connection teardown. The resumed session reported valid 6DoF with no pending or required
|
||||
recenter. The native driver and hand configuration were unchanged. Evidence is recorded under
|
||||
`build/luma-vr-start-delay/`. The review fixes were rebuilt and installed, but the phone
|
||||
reported no glasses connected during the follow-up, so hardware cancellation recovery
|
||||
was not rechecked.
|
||||
|
||||
The installed APK SHA-256 is `fac0973a21e41db842d88617bb11f16672987bd0b955736ad510abceca46b00f`.
|
||||
Earlier connected-phone diagnostics confirmed `FULL`, `raw-grayscale-rgb`,
|
||||
`pinhole-points-only`, concurrent CPU inference, scheduled discovery, adaptive input,
|
||||
and `steadyHandAim=false`. Camera/head processing was active with the verified device
|
||||
calibration. The launch check reached READY and staged VR driver SHA-256
|
||||
`e79525c144e934761f08cdc1d3170d058df89bf55c8458ddc955b58c90c9b128`, matching the rebuilt
|
||||
payload. This verifies deployment and live processing, not moving-hand accuracy or comfort.
|
||||
|
||||
The optional Lite bundle can be rebuilt with `mediapipe==0.10.21` using:
|
||||
|
||||
```sh
|
||||
python tools/build_hand_lite.py --palm palm_detection_lite.tflite --lite hand_landmark_lite.tflite
|
||||
```
|
||||
|
||||
Source URLs and hashes are in `app/src/main/assets/models/hand_landmarker_lite.provenance.json`.
|
||||
These changes still need live moving-hand, closed-fist, gesture-continuity, and latency validation.
|
||||
Earlier Lite/normalized-input validation passed 52 focused JVM tests, and bundle rebuild and label-archive reproducibility
|
||||
checks passed. Desktop MediaPipe 0.10.21 inference succeeded on color hand fixtures. All six
|
||||
saved left-camera grayscale grip images still returned zero detections after correction and
|
||||
contrast normalization at the existing 0.6 confidence thresholds. This is not a tracking-accuracy
|
||||
pass. Synthetic projection/undistortion/triangulation checks stayed below 0.12 mm reconstruction
|
||||
error for the bundled pinhole profile and near numerical precision for a synthetic fisheye profile.
|
||||
Local reports for that implementation check are in `build/hand-lite-models/`; Android hardware
|
||||
inference was not run at that stage.
|
||||
|
||||
The subsequent [Full/Lite preprocessing comparison](hand-model-comparison-2026-10-16.md)
|
||||
ran all eight model/preprocessing combinations on the phone using nine saved stereo pairs.
|
||||
Full recovered one of two central open hands; Lite recovered no usable stereo hand. Neither
|
||||
model found the peripheral hand, and contrast/lens correction added no accepted bare hands.
|
||||
Full with raw grayscale-to-RGB input and calibrated point correction is now the live baseline;
|
||||
the earlier comparison diagnostic itself did not change the defaults.
|
||||
|
||||
There is room to improve the current implementation, but the available evidence does not support calling another smoothing adjustment a gameplay fix. The strongest current blockers are rejected palm reconstruction, interrupted gesture continuity, and processing latency under SteamVR load. The next useful milestone is a repeatable open → pinch → release interaction with a stable aiming pose, followed by sustained in-game testing.
|
||||
|
||||
## What the existing evidence establishes
|
||||
|
||||
The latest recorded post-installation interaction is `build/spatial-live-hands-sticky-grab-updated.json`, summarized by `build/sticky-grab-updated.log`. It belongs to Android process 17867 and SteamVR session `2026-10-16-60-steam`, using the stability candidate before the fork migration. It is historical evidence, not a measurement of the changes made during this investigation.
|
||||
|
||||
| Finding | Evidence | Interpretation and limits |
|
||||
|---|---|---|
|
||||
| Head tracking remained available while hands failed | All 2,823 coherent channel samples had fresh head tracking; right hand was fresh in 1,446 samples (51.2%) | The observed interruption is in the hand path. This is sampled channel availability, not accepted-camera-observation rate; the user's exact hand visibility was not recorded. |
|
||||
| Palm acceptance is a major failure point | Five of eight status snapshots reported matched/reconstructed hands but zero accepted palms | Detection alone is insufficient. The old message grouped missing palm anchors, implausible width, degenerate orientation, and temporal outliers together. It does not identify which gate caused each rejection. |
|
||||
| Gesture continuity can prevent clicks entirely | The right-hand trace contained valid open and pressed pinch values, separated by loss transitions | Requiring a fresh observed release after loss is correct, but unreliable tracking can prevent a usable press sequence. See the replay below. |
|
||||
| Processing already consumes most of the latency target | Eight reported processing times ranged from 63.3 to 114.0 ms; observation ages from 63.6 to 155.7 ms | These are internal status samples, not physical-motion-to-display latency. Some exceed the complete external p95 target of 100 ms before rendering is counted. |
|
||||
| The run was thermally constrained | Android thermal status was 4; HAL skin temperature increased from 53.9 to 54.9 °C | This run cannot establish cold performance or isolate thermal throttling from game workload. |
|
||||
| Latest stability changes did not establish usable interaction | The user reported similar stationary jitter and inability to start a dashboard drag | The previous candidate's automated tests and successful installation did not establish a hardware interaction pass. |
|
||||
|
||||
The trace used CPU, full model, concurrent stereo inference, automatic input rate, and scheduled tracking. All eight diagnostic snapshots selected the two-hand search bank and a 500-permille camera copy rate. Approximately 25 Hz camera delivery therefore supplied at most about 12.5 eligible stereo pairs per second before processing and rejection. The left-hand channel had only 14 fresh samples; this was not a controlled two-hand visibility test, so that number is not left-hand detection accuracy.
|
||||
|
||||
### Why a detected pinch may produce no click
|
||||
|
||||
The right-hand transition list spans 15.885 seconds and contains 142 rows. Applying the current trigger rules from `tools/vr/src/spatial_tracking.h` to those sampled rows gives:
|
||||
|
||||
| Trace quantity | Count |
|
||||
|---|---:|
|
||||
| Fresh, valid new observations below the 0.25 release threshold | 31 |
|
||||
| Fresh, valid new observations above the 0.65 press threshold | 12 |
|
||||
| Changes in the right trigger's loss-counter byte | 16 |
|
||||
| Valid-palm transition rows with invalid pinch geometry | 58 |
|
||||
| Above-threshold observations while the replayed trigger remained armed | 0 |
|
||||
|
||||
Replay starts disarmed because the first sampled hand is invalid. It compares the low byte of the right hand's `epochs`, releases on invalid/stale hand or pinch data, ignores duplicate observations for arming, checks the 200 ms gap against the previous accepted hand timestamp, and arms only on a fresh valid trigger value below 0.25. A valid palm with invalid pinch still advances the hand observation timestamp, as in the driver.
|
||||
|
||||
This is an inference from a sampled shared-memory trace. Sampling may omit intermediate observations, and this replay does not record what SteamVR or its dashboard consumed. It nevertheless gives a concrete explanation consistent with the reported inability to start a drag: the recognizer produces high pinch values, but continuity is lost before those values can form an armed press. Removing the release requirement would reintroduce presses on reacquisition and would not repair the observation stream.
|
||||
|
||||
## Prioritized work
|
||||
|
||||
1. **Measure each rejection before changing acceptance.** Count processed observations, valid palm/gesture observations, and rejection reasons separately for each hand. Distinguish no matching pair, missing triangulated palm anchors, palm width, degenerate orientation, implausible motion, synchronization, and age. Keep cumulative counters separate from sparse UI snapshots. These counters need no camera image recording. A broad reduction in stereo-quality thresholds could trade disappearing hands for incorrect depth and jumps.
|
||||
|
||||
2. **Establish one reliable interaction sequence.** With a single visible hand at 30, 50, then 80 cm, repeat open → pinch → hold → release while collecting the observation counters and existing driver input-transition logs. Repeat for the other hand, then with both visible. Record whether failure happens before palm acceptance, at pinch validation, at arming, at OpenVR publication, or inside the dashboard. The dashboard uses trigger for click/drag; the installed Vive binding maps grip to Back. Keep the other fingers open during the pinch test so the two actions remain distinguishable.
|
||||
|
||||
3. **Reduce measured processing cost and discovery spikes.** Run temperature-matched comparisons of full/scheduled CPU, lite/scheduled CPU, and input-rate settings, retaining the same game scene and deliberate motions. Measure completed-observation rate and accepted-observation rate, not camera rate or repeated channel reads. Existing saved-open-palm benchmarks show potential: under SteamVR, scheduled full inference had median/p95 46.92/94.71 ms versus full-search 91.48/99.26 ms; scheduled lite gave 40.26/86.77 ms. Those static-image results cannot establish moving-hand, pinch, or crossed-hand accuracy. The full model should remain the baseline until those comparisons pass. GPU requires a separate stability check because earlier evidence records graphics recovery/device-loss concerns.
|
||||
|
||||
4. **Tune pose and aiming with motion evidence.** Record stationary raw and rendered pose distributions and a repeatable slow aiming path. At the time of this trace, driver stabilization held position within 3 mm and orientation within 1 degree around a stationary anchor, applied a 15 ms rendering correction, and bounded prediction to 2 cm/80 ms. The anchors have since been removed at the user's request. Increasing smoothing without measuring raw noise and lag risks making aiming less responsive. Test motion-aware filtering or a more stable aim direction only against the same recorded motions and external latency measurement. Aiming direction and a game controller's grip pose may need separate treatment.
|
||||
|
||||
5. **Validate game compatibility after continuity.** Optical controllers currently expose Vive-compatible trigger/grip/trackpad inputs, with gamepad sticks and buttons supplying locomotion and menus; there is no skeletal hand output. `CardboardController::UpdateOptical` initializes `DriverPose_t` to zero and populates position and rotation, leaving linear and angular velocity zero. This is a source-confirmed omission to investigate for throwing and other velocity-dependent games, not an established cause of the dashboard problem. A velocity implementation needs capture-time derivatives, outlier/loss resets, a defined pose timestamp, and a game-level test before enabling it.
|
||||
|
||||
Startup should be measured separately from steady state. Earlier logs contained shader-pipeline creation well after the dashboard appeared, while both hand detectors were already initialized. Correlate observation latency and CPU/GPU workload through launch and the following two minutes; model warmup alone does not explain the reported minute-long startup delay.
|
||||
|
||||
The exported calibration also helps explain why a 3 mm stationary hold may have little visible effect. Its focal lengths are approximately 226–241 pixels and its stereo baseline is approximately 14 cm. The ideal rectified-stereo approximation `depth change ≈ depth² / (focal length × baseline) × disparity change` gives roughly 8 mm per disparity pixel at 50 cm and 20 mm at 80 cm. These are sensitivity estimates, not measured hand errors; the actual cameras have lens distortion and relative rotation. Independent landmark jitter can therefore exceed the driver's hold radius. If the new counters confirm palm reconstruction dominates, a robust fit across the available palm landmarks and temporal stereo correspondence are stronger candidates than simply enlarging that hold radius. Such a fit still needs explicit outlier rejection, side identity, capture-time freshness, and measured motion/gesture tests.
|
||||
|
||||
## Vendor runtime options
|
||||
|
||||
The public [VITURE Glasses C SDK documentation](https://www.viture.com/developer/glasses-sdk/glasses) describes USB management, head tracking, display control, and camera streaming, including Android support. It does not document a phone hand-tracking runtime that can replace this application's MediaPipe reconstruction directly. That is a statement about the published API reviewed on this date, not proof that VITURE has no private or forthcoming implementation.
|
||||
|
||||
The [VITURE Unity SDK documentation](https://www.viture.com/developer/unity-sdk/unity) does describe hand tracking, responsive/stable filter modes, and arm/wrist contributions to aiming. Its stated hardware requirement and deployment instructions target the Pro Neckband. Those controls provide useful design references, but the documentation does not establish that their runtime can be redistributed inside this phone app. Confirm phone runtime availability, platform support, and licensing with VITURE before planning that replacement.
|
||||
|
||||
## Calibration and completion criteria
|
||||
|
||||
The phone's current calibration can be shipped as the Luma Ultra starting profile. That reduces setup friction; it does not fix inference cost, hand occlusion, gesture continuity, or jitter by itself. Preserve each device's own calibration as the higher-priority choice, restrict the base profile to its matching model and camera geometry, and identify it as a shared starting calibration. A calibration validated on the source glasses has not been independently measured on every Luma Ultra unit.
|
||||
|
||||
Retain the existing acceptance gates in `luma-spatial-tracking.md`: at least 95% accepted slow unobstructed hand observations at 30–80 cm, no more than 5 cm position error and 2 cm stationary jitter, external p95 latency below 100 ms, reliable release and reacquisition, and no more than 10% regression in p95 game frame time over a 20-minute run. Dashboard selection and Angry Birds VR aim/grip/pull/release still require deliberate physical validation. Keep the feature visibly experimental until these tests pass.
|
||||
|
||||
Supporting records: `luma-spatial-evidence.md`, `build/spatial-live-hands-sticky-grab-updated.json`, `build/hand-reliability-benchmark-smoothness-steamvr.json`, and `build/hand-startup-review.json`. The build-directory records are local historical artifacts and may not accompany a fresh checkout.
|
||||
|
||||
## Joy-Con grip and arm extension investigation (2026-10-16)
|
||||
|
||||
The target is a closed fist holding a controller, with the wrist and forearm visible from the
|
||||
wearer's viewpoint. Open-palm performance is not sufficient evidence for this use case. In a
|
||||
requested 16-second extension exercise, 5 of 80 sampled left positions and 1 of 80 right positions
|
||||
were valid while head tracking stayed valid. These are channel availability samples, not 80
|
||||
independent detector results. The earlier driver also restarted its transition to the arm model
|
||||
on each invalid hand frame, preventing intermittent camera positions from converging.
|
||||
|
||||
The initial grip estimator required the wrist, middle base knuckle, and one supporting base
|
||||
knuckle, rather than all five palm landmarks and a hand-derived orientation. The later
|
||||
wrist-only update below removes the remaining knuckle dependency. It never generates hand
|
||||
gestures; Joy-Con IMU and buttons remain authoritative. Stereo optical flow can carry the
|
||||
identified wrist through brief model misses, with forward/backward consistency, photometric,
|
||||
stereo reprojection and motion checks. Tracks expire 750 ms after the last detector seed; frame
|
||||
gaps over 200 ms discard them. The driver retains a camera observation only until 200 ms after
|
||||
its original capture time. Neither mechanism turns stale coordinates into fresh measurements.
|
||||
|
||||
### Model assessment
|
||||
|
||||
- **MediaPipe Full (historical baseline):** an existing Android pipeline with separate palm
|
||||
detection and landmark regression. Curling fingers does not intrinsically exclude a hand,
|
||||
but occlusion and the egocentric view must be tested with actual grips. Its published
|
||||
monocular world landmarks are not used as metric camera distance; calibrated stereo supplies
|
||||
distance. [Official task documentation](https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker).
|
||||
- **Earlier bundled Lite:** that bundle changed the landmark regressor while retaining the Full
|
||||
palm detector (see `tools/build_hand_lite.py`). It can reduce processing cost, but cannot be
|
||||
assumed to fix the initial fist-detection problem. Earlier saved open-palm replay under
|
||||
SteamVR measured scheduled Full median/p95 46.92/94.71 ms and Lite 40.26/86.77 ms. Those figures
|
||||
do not establish closed-fist accuracy.
|
||||
- **RTMDet-nano hand detector plus RTMPose-m Hand5:** a concrete alternative detector and
|
||||
landmark pipeline worth evaluating against grip frames. The official demo uses separate
|
||||
hand boxes and landmarks. It is not a replacement `.task` asset: Android inference,
|
||||
preprocessing, handedness/identity association, and device timings need integration and
|
||||
verification. No claim of superior Joy-Con performance is established yet.
|
||||
[Official hand demo](https://github.com/open-mmlab/mmpose/blob/main/demo/docs/en/2d_hand_demo.md).
|
||||
- **HaMeR:** useful as an offline reference for occluded and egocentric hands, with published
|
||||
evaluation procedures. Its reference implementation uses PyTorch, ViTPose, and MANO assets;
|
||||
it is not an established real-time Android replacement for this app. Do not substitute its
|
||||
inferred mesh depth for measured stereo depth.
|
||||
[Official implementation](https://github.com/geopavlakos/hamer).
|
||||
- **Whole-body arm pose:** not the first candidate for cameras on the wearer's glasses,
|
||||
where the wearer's shoulders and torso are commonly outside view. An egocentric fist/wrist
|
||||
detector or forearm segmentation model is the more directly matched development target.
|
||||
This is an engineering assessment, not a measured comparison of those models on this device.
|
||||
|
||||
The third camera is a separately exposed UVC pass-through camera, not an unused member of the
|
||||
stereo callback. It may provide better RGB recognition, but requires its own capture/decode and
|
||||
camera-to-head calibration. It does not independently provide metric depth. The preferred use
|
||||
would be recognition or association assistance while stereo supplies distance; size-based
|
||||
monocular distance would need separate calibration and uncertainty handling.
|
||||
[VITURE SDK camera support](https://www.viture.com/developer/glasses-sdk/glasses).
|
||||
|
||||
Saved-grip replay reports `joyCons` acceptance separately from bare-hand `hands`, for both Full
|
||||
and Lite. Repeating one image measures model/geometry behavior on that image, not motion
|
||||
continuity. Live tests must still check reach response, left/right identity, reacquisition,
|
||||
camera loss, and head motion before considering the experiment reliable.
|
||||
|
||||
### Closed-fist model comparison
|
||||
|
||||
Three requested grip pairs were saved locally as captures `110935538464606`, `110939418512521`,
|
||||
and `110943418461791`. In on-phone saved-frame replay of the middle and last pair, Full detected
|
||||
one hand in the left image and none in the right image in all 20 measured repetitions per
|
||||
configuration; Lite detected none in either image. Neither produced a valid grip position.
|
||||
On the middle pair, scheduled Full median/p95 was 56.05/62.96 ms and Lite 46.00/47.99 ms. On
|
||||
the last pair, it was 21.17/23.33 ms and 18.08/20.06 ms respectively. Timing differences between
|
||||
frames are not accuracy improvements. Repeated static frames are not independent observations.
|
||||
|
||||
An isolated Windows evaluator tested the official RTMDet-nano hand / RTMPose-m Hand5 ONNX
|
||||
exports using `rtmlib` 0.0.16 and ONNX Runtime 1.31.0. Across the six raw images, hand-box counts
|
||||
were 0, 0, 1, 0, 2, 1. Lens-corrected images gave 0, 1, 1, 0, 2, 1. Neither variant found both
|
||||
hands in both cameras for any pair. These are box counts, not validated stereo positions or
|
||||
ground-truth accuracy. Desktop timings cannot predict Android cost. Model downloads and hashes
|
||||
are in `build/joycon-alternative-models/provenance.json`; results and overlays are local.
|
||||
|
||||
MediaPipe desktop preprocessing comparisons used the same Full model asset with Python Tasks
|
||||
1.1.1, versus Android Tasks 1.0.0. Overlapping 400-pixel crops recovered some missing detections;
|
||||
lens correction and lower thresholds also changed detections and sometimes handedness. Only
|
||||
isolated stereo matches passed the existing geometry checks. This does not justify enabling
|
||||
either preprocessing change in the live pipeline yet, or replacing Full with Lite/RTMPose.
|
||||
|
||||
The post-install shared-memory sample arrays in `build/joycon-reach-after.json` and
|
||||
`build/joycon-reach-grip.json` are not valid live-pose measurements: the SteamVR session had
|
||||
not restarted, so its pose publisher was inactive. Their before/after provider diagnostics and
|
||||
requested image captures remain useful. During the actual closed-fist interval, all 253
|
||||
processed pairs reported invalid head tracking, so no live reach claim can be made. The local
|
||||
measurement script now requires a running session, valid head tracking, and advancing head
|
||||
timestamps to reject this situation. The image comparisons above do not depend on a live head
|
||||
pose. A new live test is still required after head tracking and the session recover.
|
||||
|
||||
### Third-camera grip comparison
|
||||
|
||||
The separately exposed RGB camera delivered 1920 by 1080 JPEGs after Android Camera and USB
|
||||
permissions were granted. SDK 2.4.0 crashed in `xr_camera_provider_start` when reopening the
|
||||
provider in the main app process. RGB diagnostics now run in a dedicated process that skips
|
||||
normal app initialization. Four initial successful captures preserved the main app PID.
|
||||
Subsequent one-shot attempts intermittently returned zero callbacks despite a successful SDK
|
||||
start. Keeping a stream open between requests did not resolve this: four successive startup
|
||||
attempts returned zero callbacks. USB reconnection and a separate attempt with positional
|
||||
tracking temporarily disabled also returned no frames. The user's decision was to stop the RGB
|
||||
route and compare the models on stereo. The persistent-stream and RGB/stereo calibration-buffer
|
||||
prototypes were removed; the isolated one-shot command remains for manual diagnostics only.
|
||||
|
||||
Three requested normal-grip RGB captures (`112581486574499`, `112588468819444`, and
|
||||
`112595950634337`) were evaluated with the same local model assets and thresholds:
|
||||
|
||||
| Pipeline, full image | First | Second | Third |
|
||||
|---|---:|---:|---:|
|
||||
| MediaPipe Full | 0 hands | 0 hands | 2 hands |
|
||||
| MediaPipe Lite | 0 hands | 0 hands | 1 hand |
|
||||
| RTMDet-nano + RTMPose-m Hand5 | 2 hands | 2 hands | 2 hands |
|
||||
|
||||
Overlapping square crops recovered one grip in the first image with Full and Lite; neither
|
||||
recovered a hand in the second. The no-hands control image produced zero detections for all
|
||||
three pipelines. Visual inspection of the alternative model's overlays found plausible wrist
|
||||
locations on these grips, including the closer, partly cropped second view. Three selected
|
||||
images are not an accuracy dataset or evidence of continuous tracking, identity preservation,
|
||||
or measured reach. RGB results and overlays are in `build/joycon-rgb-evaluation`.
|
||||
|
||||
The unchanged official ONNX models were also replayed on the Fold7 using Android ONNX Runtime
|
||||
1.31.0 and OpenCV 4.13.0 preprocessing. Four warm-ups and ten timed iterations per image gave
|
||||
two-thread median times of 119.90, 119.88, and 129.24 ms; the largest sample was 151.04 ms.
|
||||
Four threads gave medians of 120.45, 120.79, and 121.05 ms. Timings include preprocessing,
|
||||
detection, and two hand landmark passes, but exclude JPEG decoding, camera acquisition,
|
||||
stereo depth, head fusion, rendering, and concurrent SteamVR load. Thus they already exceed
|
||||
the 100 ms end-to-end target. The app's OpenCV DNN importer alone rejected the detector's TopK
|
||||
operator; these measurements used a separate native benchmark, not a new production model
|
||||
dependency. [ONNX Runtime mobile support](https://onnxruntime.ai/docs/get-started/with-mobile.html).
|
||||
|
||||
The RGB SDK timestamps were zero on this device. Diagnostics record host callback receipt
|
||||
separately. A temporary prototype successfully collected one nearby stereo pair with surrounding
|
||||
frames, but the subsequent board attempt received no RGB frame. No RGB calibration was fitted.
|
||||
Existing stereo calibration does not supply RGB lens parameters or the RGB-to-stereo transform.
|
||||
RGB has never been used to publish controller positions in this experiment.
|
||||
|
||||
### Stereo-only follow-up and wrist landmark 0
|
||||
|
||||
The alternative RTMDet/RTMPose pipeline was replayed on the Fold7 against the same three raw
|
||||
stereo grip pairs, without RGB. With live Joy-Con inference temporarily paused, counts were
|
||||
0/0, 1/0, and 2/1 hands in the left/right images. Per-image medians were 45.63/45.42,
|
||||
129.35/45.03, and 199.50/123.13 ms. These are sequential single-image benchmark measurements,
|
||||
not concurrent stereo latency or end-to-end tracking. Both candidate pairings in the last
|
||||
capture failed the current geometry gates. The plausible wrist pair projected with 4.17 px
|
||||
and 3.27 px error, exceeding the 3 px limit; the other pairing reconstructed behind the camera.
|
||||
Neither a valid grip nor a valid isolated wrist was established. The log and geometry report
|
||||
are `build/joycon-stereo-phone-ort-controlled.log` and
|
||||
`build/joycon-stereo-phone-ort-geometry.json`.
|
||||
|
||||
“Full” in this project is the full MediaPipe Hands landmark model, not a full-body model.
|
||||
At the time of the model comparison, the live Joy-Con estimator used the midpoint of wrist 0
|
||||
and middle-finger base 9, validated with an additional base knuckle. The saved-pair diagnostic
|
||||
added `wrists`, testing landmark 0 without those palm anatomy requirements. This cannot
|
||||
recover a wrist when the upstream hand detector misses the grip in either camera.
|
||||
|
||||
The existing desktop Full outputs at the production 0.6 thresholds, using raw uncropped
|
||||
images, gave confident left/right counts of 0/0, 1/0, and 0/0. Thus none offered a wrist
|
||||
pair even before palm geometry checks. Desktop Tasks 1.1.1 static-image evaluation differs
|
||||
from Android Tasks 1.0.0 VIDEO replay and must not be presented as an Android measurement.
|
||||
The local report is `build/joycon-mediapipe-wrist-evaluation.json`. The updated Android
|
||||
wrist-only diagnostic was built and installed, but its replay is pending the glasses USB
|
||||
approval after installation. The last phone settings check showed positional tracking on,
|
||||
Joy-Con tracking off, and hand controllers off; there was no active stereo engine.
|
||||
|
||||
Keep MediaPipe Full as the live baseline. The alternative's better RGB detections did not
|
||||
transfer to dependable stereo grip positions on these captures. No new physical arm-extension
|
||||
validation has been completed.
|
||||
|
||||
### Live wrist-only Joy-Con position
|
||||
|
||||
Following the user's correction, Joy-Con mode triangulates only landmark 0 and publishes its
|
||||
position transformed through the camera-to-head and captured head poses. Knuckle availability,
|
||||
palm proportions, and the wrist/knuckle midpoint no longer affect Joy-Con acceptance or position.
|
||||
The estimator is `JoyConWristPosition`, and diagnostics report `joycon-wrist` with wrist-specific
|
||||
missing-landmark and motion outcomes. Bare-hand gesture tracking still uses its palm geometry.
|
||||
|
||||
Optical flow now seeds only the wrist and accepts a single tracked point per camera; the
|
||||
previous three-point minimum is removed. Forward/backward and photometric checks, stereo
|
||||
depth and reprojection limits, motion rejection, head synchronization, freshness, and track
|
||||
expiry remain in place. Controller rotation and buttons still come from the Joy-Con, and
|
||||
the driver still falls back to the arm model when fresh camera positions are unavailable.
|
||||
This removes the unnecessary knuckle dependency, but does not bypass MediaPipe's hand
|
||||
detector or establish continuous wrist visibility in the saved grip images.
|
||||
|
||||
### RGB transport retry: hidden USB submission failure
|
||||
|
||||
After the wrist-only test did not meet the user's needs, the RGB investigation resumed.
|
||||
Capture `116048611549114` added stderr collection with `LIBUSB_DEBUG=4`, separate from the
|
||||
vendor log hook. The SDK returned start status 0 and streaming status 1, but libusb reported
|
||||
`submit_iso_transfer: submiturb failed, errno=12` (`ENOMEM`) and subsequent isochronous packet
|
||||
errors. No RGB callbacks arrived. GET_INTERFACE showed alternate setting 5; the negotiated
|
||||
UVC payload was 2400 bytes per microframe. This establishes USB allocation/submission failure,
|
||||
not its exact resource limit: Android denied reading the USB memory-limit sysfs attribute.
|
||||
|
||||
Binary inspection also traced the earlier native crash to `libusb_cancel_transfer(NULL)`
|
||||
while cleaning up a failed start. Reopening alone was therefore too specific an explanation.
|
||||
The embedded SDK's roughly 100-transfer queue is a plausible source of pressure. Its public
|
||||
API cannot change the queue or the fixed 1080p30 mode.
|
||||
|
||||
An optional independent diagnostic transport now uses libusb 1.0.30 and libuvc 0.0.8 with eight
|
||||
queued transfers. The pinned libuvc source was patched to preserve submission errors and safely
|
||||
drain partial queues instead of reporting success. A hardware-free executable passed 60 starts
|
||||
on the phone covering failure before any submission, failure after three submissions, and all
|
||||
eight submitted. This validates failure cleanup, not camera capture. The reproducible recipe,
|
||||
source hashes, and licenses are documented in `docs/third-party/uvc/README.md`.
|
||||
|
||||
Raw USB descriptors advertise MJPEG 640×480, 1280×720, and 1920×1080 among other sizes, with
|
||||
30/25/20/15/10/5 fps intervals. The direct path can compare these modes while head/stereo tracking
|
||||
continues. It saves a local JPEG plus transfer diagnostics and stream statistics; it is not yet
|
||||
connected to controller tracking. RGB lens/extrinsic calibration and measured reach remain
|
||||
separate unfinished requirements.
|
||||
|
||||
### Direct UVC follow-up after the firmware update
|
||||
|
||||
The glasses now report firmware `12.0.02.002_20260901`, previously
|
||||
`12.0.01.100_20260326`. Direct UVC acquired short bursts, but no sustained stream:
|
||||
capture `140416689286120` delivered 22 callbacks over 708 ms, followed by 4.3 seconds
|
||||
without another callback. The main app stayed alive. These are transport observations,
|
||||
not arm-extension measurements.
|
||||
|
||||
The rejection logs exposed a separate parsing bug: valid JPEG end markers were followed
|
||||
by one to seven zero bytes, with buffers aligned to eight bytes. The diagnostic now trims
|
||||
up to seven trailing zeros before checking the end marker and saves the trimmed payload.
|
||||
A native test on the phone covers padding, truncation, garbage, null input, and size bounds.
|
||||
This fixes rejection of padded frames; it cannot recover missing USB callbacks. JPEG
|
||||
boundary validation also does not establish successful decoding.
|
||||
|
||||
A 32-transfer queue with eight packets per transfer was compared with the original
|
||||
eight transfers of 32 packets. Both fault-injection builds passed 60 simulated starts.
|
||||
The smaller transfers started 1080p30 without the earlier submission error, but received
|
||||
zero frames (`140593211685272`). At 640p the smaller transfers still stalled after seven
|
||||
callbacks (`140493747047289`). Neither configuration established reliable acquisition.
|
||||
|
||||
Explicit USB device resets returned success but produced no frames at 640×480/30 or
|
||||
320×240/5. GET_INTERFACE retained the selected alternate setting. Android's external
|
||||
camera provider reported zero devices. A physical reconnect with positional tracking
|
||||
disabled before attachment also yielded zero callbacks (`141363168667894`): capture
|
||||
waited five seconds for the first frame and never entered its requested ten-second
|
||||
observation window. The original positional and Joy-Con settings were restored afterward;
|
||||
bare-hand controllers remained off. This failure occurs without the stereo/VIO stream.
|
||||
|
||||
Capture evidence is under `build/joycon-rgb-captures/<id>`, including `usb.log`,
|
||||
`frame.json`, descriptors, and per-worker `libuvc-logcat.txt`. Teardown cancellation
|
||||
messages are not evidence that transfers were cancelled during acquisition. RGB remains
|
||||
diagnostic-only, with no RGB depth calibration or controller-position publication.
|
||||
|
||||
Detailed USB logging in capture `141627121555242` recorded 224 successful isochronous
|
||||
transfer completions over approximately 223 ms, each totaling 96 bytes, followed by no
|
||||
successful completions until capture teardown. These small transfers did not produce a
|
||||
decoded image or frame callback. A follow-up (`141844371790836`) read the UVC streaming
|
||||
error code before and after the stall: both returned code 0. Android denied reading the
|
||||
USB power-management sysfs attributes, so autosuspend was not verified as a cause.
|
||||
|
||||
The user confirmed a direct VITURE cable connection. With the same glasses and cable
|
||||
connected to Windows, FFmpeg/DirectShow decoded 300 frames in each of three ten-second
|
||||
runs: 640×480/30, 1920×1080/30, then 640×480/30 after reopening. Outputs used passthrough
|
||||
frame timing and frame hashes, with no frame-rate duplication or saved video. All exited
|
||||
successfully without timeout. Evidence is in `build/joycon-rgb-windows`, produced by
|
||||
`build/test-luma-rgb-windows.py`, which selects only USB VID:PID `0c45:636b`. This supports
|
||||
investigating the Android host/USB path further; it does not prove the camera will be
|
||||
reliable on every host or establish the exact Android fault.
|
||||
|
||||
A controlled APK replacement of only `libusb.so` with official libusb 1.0.27 also
|
||||
returned zero callbacks (`142296418051861`). Its source archive SHA-256 is
|
||||
`ffaa41d741a8a3bee244ac8e54a72ea05bf2879663c098c82fc5757853441575`; provenance and the
|
||||
one-entry APK comparison are under `build/rgb-libusb-1.0.27`. The production dependency
|
||||
pin remains 1.0.30. An isolated YUYV 320×240/5 diagnostic negotiated format 2 but likewise
|
||||
received zero raw frame callbacks (`142407601079475`), so JPEG validation was not its
|
||||
failure point. Neither comparison changed the production camera format.
|
||||
|
||||
Finally, 64 queued transfers of 32 packets increased the high-speed queue coverage from
|
||||
approximately 32 ms to 256 ms. Its fault-injection executable passed all 60 starts.
|
||||
Capture `142557405834522` recovered 18 valid JPEGs over 577 ms, then stalled for the
|
||||
remaining 9.42 seconds of its ten-second observation. All 18 passed the padding fix;
|
||||
the saved final JPEG also decoded at 640×480 in OpenCV. This is not sustained tracking
|
||||
and neither alternate queue was adopted. Temporary USB reset, queue configuration, and
|
||||
verbose logging experiments were removed. The retained UVC recipe uses eight transfers
|
||||
of at most 32 packets, and retains the JPEG-padding fix.
|
||||
@@ -0,0 +1,445 @@
|
||||
# Luma Ultra implementation evidence
|
||||
|
||||
Date: 2026-10-15. Status: experimental prototype, combined release gate not passed. Native 6DoF and synchronized stereo camera delivery were observed on hardware. Camera calibration and optical-hand acceptance were deferred at the user's request.
|
||||
|
||||
## Device and build
|
||||
|
||||
| Item | Observed value |
|
||||
|---|---|
|
||||
| Android host | Samsung Galaxy Z Fold7, SM-F976U |
|
||||
| Android version | 17 |
|
||||
| Glasses | SDK-confirmed VITURE Luma Ultra / Carina |
|
||||
| Glasses firmware | `12.0.01.100_20260326` |
|
||||
| Vendor SDK | External Android C SDK 2.4.0 |
|
||||
| Test application | `com.droiddeck.launcher.xr`, debug experimental build |
|
||||
| Camera pair | Primary left0/right0, both 640 by 480 |
|
||||
| Hand runtime | MediaPipe Tasks Vision 1.0.0; OpenCV 4.13.0; pinned model SHA-256 in the model provenance asset |
|
||||
| CPU or GPU inference benchmark | Not run; no accepted calibration |
|
||||
|
||||
Existing picture, refresh-rate, and graphics work was preserved. The test temporarily enabled positional tracking and in-memory camera diagnostics. Diagnostics were stopped and the positional preference restored to its previous off setting afterward. Hand controllers were never enabled. No camera image recordings were saved. Subsequent builds are local artifacts; the phone was left available for a separate active SteamVR test.
|
||||
|
||||
## Hardware observations
|
||||
|
||||
Initial VIO output was unstable. After the user wore/moved the glasses and the software origin was recentered, valid 6DoF output was observed. Changing position values confirms positional data is available, but does not establish correct physical axes or 1:1 scale without a measured reference.
|
||||
|
||||
Camera frames initially arrived before their matching VIO callback. A bounded wait of up to 35 ms in the latest-frame reader allowed matching timestamps without building a queue. The public callback's undocumented pose basis is not used; capture transforms come from canonical OpenGL pose history.
|
||||
|
||||
A subsequent diagnostic interval produced these results:
|
||||
|
||||
| Measurement | Result and scope |
|
||||
|---|---|
|
||||
| Sampled head validity | 30 of 30 status snapshots valid |
|
||||
| Sampled camera/head synchronization | 30 of 30 status snapshots synchronized |
|
||||
| Processed stereo pairs | Counter increased by 1,048 |
|
||||
| Dropped stereo pairs | Counter increased by 9 |
|
||||
| Median reported camera rate | 24.998 Hz |
|
||||
| Median estimated frame age | 24.73 ms |
|
||||
| Observed estimated frame-age range | 21.44–45.96 ms across the 30 snapshots |
|
||||
| Median diagnostic processing time | 1.62 ms; no hand inference or calibrated undistortion |
|
||||
| Host receipt age of latest head poll | 0.09–4.40 ms across sampled snapshots; not sensor latency |
|
||||
|
||||
These are sparse status samples in an uncontrolled workload, not a frame-by-frame latency benchmark. Receipt-based source-clock mapping does not measure absolute camera exposure delay. None of these values establishes physical-motion-to-visible-controller latency, hand availability, or game-frame-time performance.
|
||||
|
||||
Local raw metadata is in `build/spatial-hardware-observations.json`. The earlier startup/recenter observations are in `build/spatial-after-movement.json`, `build/spatial-recenter-state.json`, and `build/spatial-sync-state.json`. Those files contain pose/status metadata, not images.
|
||||
|
||||
## Automated and build validation
|
||||
|
||||
- SDK-enabled and explicitly SDK-free debug APK builds pass with runtime freshness checks enabled.
|
||||
- All 305 affected Kotlin glasses, head-pose, Cardboard, and glasses UI tests pass with zero failures or skips. New cases exercise canonical coordinates, normalization, floor anchoring, crouched recenter, independent origins, stale/discontinuous samples, generation rejection, pose history, camera-clock warmup/reset, gesture geometry, and inference-buffer ownership.
|
||||
- ARM64 VR driver and presentation-layer builds pass. Native tests cover ABI offsets and size, empty/truncated/version-mismatched or incoherent reads, independent hand timestamps, finite poses, stale tracking, trigger hysteresis, lost-input release and released-gesture rearming. Existing fence, timeline, and render-target tests also pass.
|
||||
- Native Cardboard parameter tests cover all three translation values, zero-translation equivalence, and unchanged SBS/flat parameters. The GLSL compiles; the committed SPIR-V matches regenerated glslang output, and push constants remain within the guaranteed range.
|
||||
- VR build manifests include the new shared header. Packaged runtime and APK payload checks reject mismatched source/output checksums. Both external vendor files and transient dependency-inspection files remain outside tracked source.
|
||||
- Packaging regressions pass: 8 Windows VR build tests, 13 Windows preparation tests, and 11 runtime-input tests. Runtime-input tests ran in the Linux container because the Windows host has no standalone zstd executable.
|
||||
|
||||
Build logs are `build/spatial-sdkfree.log`, `build/spatial-final-build.log`, `build/spatial-vr-build.log`, `build/spatial-shader.log`, and `build/spatial-runtime-evidence.json`. Test reports are under `app/build/reports/tests/testDebugUnitTest`.
|
||||
|
||||
The preserved SDK-enabled snapshot is `build/DroidDeck-luma-spatial-experimental.apk`; the SDK-free counterpart is `build/DroidDeck-spatial-sdkfree.apk`. SHA-256 hashes and final payload checks are recorded in `build/spatial-final-validation.json`. The final local snapshot includes additional buffer-ownership and conservative tracking-loss fixes after the hardware smoke test. It was not installed over the phone being used for the separate SteamVR tests. Integrate these sources into that chat's next build before later calibration; do not replace its newer runtime-storage fixes with this older snapshot.
|
||||
|
||||
A subsequent report from the parallel SteamVR test identified Android eviction of the live `cache/viture` directory. The SDK working-directory source was changed to `noBackupFilesDir/viture` afterward. The preserved APK snapshot and the hardware observations above predate this storage change. No additional phone restart or installation was performed here; the next shared-workspace build should include it.
|
||||
|
||||
## Calibration usability follow-up (2026-10-15)
|
||||
|
||||
The calibration screen now uses DroidDeck's saved palette, app scale, settings groups, and buttons, with separate tracking, capture, validation, and diagnostic sections. Eye height, virtual-screen dimensions, board instructions, and validation messages include imperial measurements. The distance check accepts either 20 inches (0.508 m) or 50 cm (0.5 m), using the selected value in reconstruction validation.
|
||||
|
||||
Recenter requests made while the newest pose is stale or invalid were previously ignored. Requests now remain pending and are acknowledged immediately in the UI; recovery waits for 250 ms of fresh, steady tracking. Normal recentering preserves floor height, new connections cancel old requests, and captures cannot mix different VIO coordinate frames. Looking straight upward also retains a stable heading instead of deriving yaw from an almost-zero forward projection. Camera diagnostics resume when the provider restarts while setup remains open.
|
||||
|
||||
Validation: 247 glasses and calibration/settings UI tests pass with zero failures or skips. The SDK-enabled debug APK builds, runtime input verification passes, and native Robolectric renders of the recovery and distance screens were visually checked. The APK's DEX files were checked for the new measurement controls. Logs are in `build/spatial-setup-build.log`; previews are in `app/build/reports/spatial-setup/`. The phone was not reinstalled or reconfigured during the other chat's SteamVR stability run. The user's reported repeated-recenter behavior still needs confirmation on the glasses after installing this build.
|
||||
|
||||
After those stability tests finished, the combined calibration/recenter APK was installed at 17:09 MDT on October 15. Its SHA-256 is `80bf99374a31676b675856c782298aecc1e1c649ad2fc5c1ce3b18b981979e3e`; hashing the installed `base.apk` confirmed an exact match. The app cold launch and agent status checks passed. The helper, VR driver, and scheduling preload match the preceding freeze-tested build. This install confirms delivery of the latest calibration changes, not hardware acceptance of recentering or calibrated hands.
|
||||
|
||||
## Capture counter fix (2026-10-15)
|
||||
|
||||
Board detection checked head freshness against the worker's start time. A newer head poll arriving during detection made that timestamp precede the latest pose and rejected an otherwise valid capture. Completion now checks freshness under the session lock using the current monotonic time, while still rejecting stale poses, changed connections, changed tracking frames, and pending recentering. The same timing check was corrected around hand inference. Capture progress now exposes pending work and shows success or the rejection reason beside the counter, including the number of common board corners when detection fails.
|
||||
|
||||
Validation: 24 targeted spatial and setup UI tests pass with no failures or skips, including the head-poll/detection timing regression and stale/disconnected-frame rejection. The SDK-enabled APK builds with runtime freshness verification. Build log: `build/spatial-capture-fix.log`. APK: `build/DroidDeck-spatial-capture-fix.apk`, SHA-256 `bf362bb606a3ec72665102b66e4a122c8fad9f5f7333f5a9a3ea3fef9a3bc9d5`. The installed Fold7 `base.apk` hash matches.
|
||||
|
||||
With USB permission granted, valid native tracking, and synchronized stereo images at approximately 25 Hz, a live Capture board view action advanced the counter from 0 to 1 of 24 and reported success. The accepted view remains in the open calibration session. Status-only evidence is in `build/spatial-capture-fix-device.json`; no diagnostic camera recording was saved. This confirms capture acceptance and counter progression, not a completed calibration or optical-hand acceptance.
|
||||
|
||||
## Calculation failure and compact setup follow-up (2026-10-15)
|
||||
|
||||
The user reported calculation failure after at least 24 accepted captures; inspection found 35 observations still in memory. Logcat identified an OpenCV fisheye stereo fitting exception when successive views contained different corner counts (12–24 in this set). [OpenCV 4.13's implementation](https://github.com/opencv/opencv/blob/4.13.0/modules/calib3d/src/fisheye.cpp#L819) allocates its stereo residual blocks using the first view's count. The revised path fits fisheye intrinsics, undistorts every observed point, and fits stereo geometry with fixed intrinsics and zero residual distortion using the variable-count pinhole stereo solver. It retains every correspondence and still measures held-out error with the original lens model. Model failures are logged individually and the page shows the actionable reason instead of a generic failure.
|
||||
|
||||
All 35 observations were exported from the debug app as board landmarks and head poses before installation, backed up in app-private storage, and restored on the same connected glasses. No diagnostic camera recording was taken. Accepted captures now save this metadata automatically; automatic recovery requires a matching serial and geometry, and recovered batches cannot accept new-frame observations. This device did not expose a serial, so recovery here was explicit through the debug interface. The session-labelled backup is not automatically trusted on later unidentified connections.
|
||||
|
||||
The fixed fitter ran on the Fold7 with the restored set. Fisheye stereo fitting now completes, but both models fail stationary-board alignment: the best result varies by 25.5 cm (10.04 inches), above the unchanged 5 cm gate. The user confirmed the board was fixed. Offline replay showed a marked coordinate shift between the early and later observations; an exploratory fit of the first 13 observations agreed within 7.4 mm, while the full set did not. This suggests VIO frame motion/drift during collection; it does not establish its exact cause or validate a calibration from a selected subset. No profile was approved, and hand-controller acceptance remains blocked on a stable capture set plus measured-distance validation.
|
||||
|
||||
The calibration page now has a single Back/title row and no fixed subtitle or eyebrow. A landscape regression check requires at least 75% of page height for scrolling, and both its rendered preview and the installed Fold7 layout were inspected. Guidance explicitly asks for gentle turns and tilts with a pause before each capture. The final SDK-enabled APK and 29 targeted tests pass, including saved-point parsing, stale-frame behavior, restored-batch UI, and landscape viewport coverage. Runtime freshness verification passes. Installed APK SHA-256: `1ca44c6b2b480b197ec0069db026b765d7927c30b7cea7b80cb67d50d626d38a` (verified against `base.apk`). Artifact: `build/DroidDeck-calibration-calculation-fix.apk`. Evidence: `build/spatial-calculation-fix.log`, `build/calibration-observations-backup.json`, `build/calibration-alignment.log`, `build/calibration-calculation-device.json`, and `build/calibration-layout-after.png`.
|
||||
|
||||
## Distance-validation feedback follow-up (2026-10-15)
|
||||
|
||||
A new 25-view capture set produced a fitted pinhole calibration with 1.12094 px held-out reprojection error and passed the existing alignment checks. The user then reported that Validate distance appeared to do nothing. Live status showed that it had run and failed, but the message was attached to Calculate camera alignment, above the visible distance controls.
|
||||
|
||||
Validation now has independent pending/result state beside its own button. It reports partial-board corner counts, stereo-depth rejection, or measured versus selected distance in both units. It disables duplicate presses while pending and times out when no fresh camera image arrives. The complete-board requirement and 5 cm distance tolerance are unchanged. Processing errors are reported without stopping the camera worker.
|
||||
|
||||
The 25 original observations and fitted geometry were backed up, then explicitly restored onto the same connected glasses after installation, retaining `verified=false`. The device still provides no serial; no cross-device or automatic unidentified-profile reuse was introduced. A live validation attempt on the updated app reported 3 of 24 common corners rather than silently failing. Physical-distance validation remains pending; the existing fit and captures remain available in the open setup session.
|
||||
|
||||
Validation: 36 targeted tests pass, including visible validation feedback, pending-button state, complete-board/depth rejection, dual-unit output, camera-to-center distance, and unchanged tolerance. The SDK-enabled APK builds with runtime freshness verification. The result layout was rendered and visually inspected. APK: `build/DroidDeck-calibration-distance-fix.apk`; SHA-256 `1f0b288fdcb76231ecc9f5178f6621799b01f0eb83b574c79394fa1a04bb5f47`. Logs and metadata: `build/spatial-distance-fix.log`, `build/spatial-distance-apk.log`, `build/calibration-distance-fit.json`, `build/calibration-distance-captures.json`, and `build/calibration-distance-after.json`.
|
||||
|
||||
## Near-distance detection fix (2026-10-15)
|
||||
|
||||
The live stereo preview showed the complete printed board at the user's reported 20-inch position, but the pattern occupied only about 70 pixels across in the original 640 by 480 images. One explicitly requested diagnostic pair was saved. OpenCV 4.13 detected 16 left-camera checkerboard corners in the raw image and none after image undistortion; the right image detected 3 raw and 1 corrected. Neither version supplied a complete common board. This identified both limited marker resolution and additional loss from resampling, rather than paper edges being out of view.
|
||||
|
||||
Distance validation now detects the original image pair and undistorts the detected coordinates before stereo reconstruction. Pinhole inversion uses an explicit convergence criterion. The UI defaults to 12 inches (0.3048 m), also offers 30 cm, and retains the 20-inch and 50-cm choices. Feedback distinguishes detected checkerboard corners from paper edges. The 24-common-corner, stereo reprojection/depth, and 5-cm physical-distance gates remain unchanged.
|
||||
|
||||
Validation: all 36 targeted spatial/calibration/UI tests pass. Synthetic forward-projection and reconstruction checks exercise pinhole and fisheye point correction at 0.3048, 0.508, and 0.8 m; these verify geometry calculations, not hardware accuracy. The SDK-enabled build and runtime freshness checks pass, and the rendered result layout was inspected. APK: `build/DroidDeck-calibration-near-distance.apk`, SHA-256 `cd7e144e4fabb2e94872e4a762f1a641a61773902fa8fe40ec2c8c05d778f81d`; the installed Fold7 APK matches. Build log: `build/spatial-near-distance.log`. Synthetic results: `build/spatial-raw-landmark-geometry.json`.
|
||||
|
||||
The same connected glasses' 25 accepted views and 1.12094-px fitted alignment were restored after installation without marking them verified. A subsequent user-triggered distance check reported **12.01 inches measured against 12.00 inches selected**, passing the unchanged validation gates and marking the calibration verified for this connection. Status evidence is in `build/calibration-near-distance-after.json`. Native head tracking and stereo synchronization remained valid at approximately 25 Hz. No optical hands were valid in that status sample; hand accuracy, externally measured latency, and game performance remain untested. Device identity is still unavailable, so automatic reuse across unidentified reconnects remains disabled.
|
||||
|
||||
## Calibration persistence and hand-controller investigation (2026-10-15)
|
||||
|
||||
After launching SteamVR, the app reported `calibrated=false`, zero restored views, and no optical observations despite valid native head tracking. Android USB did not expose a serial number, so the earlier calibration had stayed in memory. SDK 2.4.0's documented `xr_device_provider_get_sn_hash` returns a stable 32-byte board-serial digest. The adapter now reads it under the SDK ownership lock, rejects missing/zero digests, and uses it to key fitted profiles and captures. A nonblank Android USB serial remains a fallback. An unavailable identity still does not permit automatic cross-connection reuse.
|
||||
|
||||
The existing 25 views and fitted geometry were explicitly imported onto the same physical glasses under the SDK identity with distance validation still required. The user repeated the check: **12.13 inches measured against 12.00 selected**. The app saved `verified=true`, and subsequent app installations/restarts automatically recovered the verified calibration and all 25 views without debugger restoration. This resolves the lost-calibration blocker; it does not establish reliable optical controllers.
|
||||
|
||||
Hand diagnostics now distinguish missing calibration, synchronization/freshness failure, per-camera detections, ambiguous matching, rejected stereo geometry, and accepted hands. They retain the last completed result while inference runs, and expose preprocessing/inference timing. Observations over 100 ms old are cleared before publication; the driver and session freshness checks remain in place. CPU stereo inference runs concurrently with separate image buffers and waits for both tasks before frame reuse, including after one task fails. GPU inference remains on its owning thread. Calibrated undistortion maps are cached; comparisons against the original image-undistortion operations were pixel-identical for both cameras and both lens-model paths.
|
||||
|
||||
SDK 2.4.0 documents the tracking cameras at 25 Hz and provides no stereo frame-rate setter. Faster application polling cannot raise that rate. The source-rate calculation now accounts for skipped frame sequence numbers; its earlier readout fell toward inference throughput when frames were dropped.
|
||||
|
||||
Short live samples on the Fold7 do **not** pass the combined hand gate. In the final 12-sample CPU/SteamVR check, samples with detected hands took 87–137 ms to process, and no fresh hand was observed by the status reader. In the comparable GPU sample, hand-containing observations took 59–81 ms to process, with one accepted reconstruction but no fresh hand caught by the status reader. Earlier setup-only GPU samples did briefly report a fresh left hand. Stereo-depth rejection and observations exceeding 100 ms remain frequent. These are sparse internal timing samples with uncontrolled pose/thermal conditions, not externally measured latency, hand-validity percentages, or a controlled game-performance comparison. The user reported no visible controllers in SteamVR. An earlier 8-second guest-channel check found the 176-byte channel active with both feature flags set and all sampled heads fresh, but no fresh optical hands; both virtual controller devices were activated in the SteamVR log.
|
||||
|
||||
One hand-landmark pair was inspected from the running reconstruction method without saving camera images. All 21 points triangulated within the broad depth range, but nine failed the unchanged 3-pixel reprojection gate. The wrist's error was 12.80 px; several fingers were approximately 3–4.3 px. Reconstructed palm width was 6.15 cm and depths were 28–41 cm. This is a correspondence/reconstruction failure, not a missing shared-memory channel. The single sample does not isolate calibration residuals from per-view landmark error, and no quality thresholds were relaxed. Data: `build/hand-stereo-point-diagnostics.json` and `build/hand-stereo-geometry-report.json`. Debugger pauses used for this inspection are excluded from performance evidence. The temporary GPU benchmark was ended with CPU mode restored; neither delegate has passed the combined gates.
|
||||
|
||||
Validation: 51 targeted tests pass, including device-identity binding, concurrent stereo work, frame ownership after detector failure, calibration, gestures, and setup UI. SDK-enabled and SDK-free Android builds pass runtime freshness checks. Installed candidate: `build/DroidDeck-parallel-hands.apk`, SHA-256 `5e8ffa6a89a91e3b1b22fe289c126dca96a77dfb7645da551e8e97e13673c761`. Evidence: `build/spatial-parallel-hands.log`, `build/spatial-parallel-sdkfree.log`, `build/spatial-cached-undistortion.json`, `build/spatial-parallel-cpu-steamvr.json`, `build/spatial-cached-gpu-steamvr.json`, and `build/spatial-steamvr-hands-channel.json`. Calibration backup: `build/calibration-verified-profile.json` and `build/calibration-verified-captures.json`. The combined feature remains an experimental prototype.
|
||||
|
||||
## Hand-mode color regression and CPU rollback (2026-10-15)
|
||||
|
||||
The parallel CPU trial was removed after failing to improve live hand results. The serial worker retains cached lens maps, device-bound calibration persistence, and detailed rejection diagnostics. All 49 remaining targeted tests pass; the SDK-enabled APK builds with runtime freshness checks. Installed candidate: `build/DroidDeck-serial-hands.apk`, SHA-256 `bcefc87dabfde3d29f2018c2b39368d4e68cb015dd390385e3c80c74d86531c6`. Build log: `build/spatial-serial-rollback.log`.
|
||||
|
||||
The user reported choppy head turns with hands and positional tracking disabled, then requested that this investigation be deferred because the phone was charging. A thermal snapshot reported status 4 and a skin sensor reading of 49.4 degrees C. The rendering log recorded approximately 55.5–60 fps at a selected 60 Hz. The retained 1.22-second prefix of a truncated head-channel diagnostic showed pose changes roughly every 3.2–6.6 ms, with no sampled gap above 7.5 ms. This small sample is not a controlled performance acceptance result. Evidence: `build/tracking-regression-sample.json`, `build/tracking-regression-analysis.json`, and `build/tracking-regression-wayland.log`.
|
||||
|
||||
The HDR-like darkening is reproduced in screenshots of the glasses output, with brightness 6, Sunblock off, and valid head tracking unchanged. Camera inference enabled independently of optical-controller mode leaves normal colors. Enabling optical-controller mode with inference paused darkens midtones and shadows. A further test switching only `SpatialTracking.setHands`, bypassing the settings/menu/display-refresh path, also reproduces it. The temporary debugger overrides were removed after each comparison. This isolates the trigger to the SteamVR optical/untracked-controller path; it does not yet identify the faulty color-conversion operation. HDR output is disabled in the session. Evidence: `build/hand-color-comparison.json`, `build/hand-color-isolation.json`, `build/hand-color-mode-isolation.json`, and their `build/hand-color-*.png` screenshots. The calibration and 100-ms observation expiry remain unchanged.
|
||||
|
||||
The serial candidate also passes the SDK-free build (`build/spatial-serial-sdkfree.log`). Temporary `TU_DEBUG=noubwc` did not remove the darkening, although the user changed display modes between that pair's screenshots. A subsequent `TU_DEBUG=gmem` comparison was interrupted by graphics recovery and is inconclusive. Neither diagnostic setting was retained: the following session's actual `vrserver` and `vrcompositor` environments both report an empty `TU_DEBUG`, and pending, active, and persistent agent overrides are empty (`build/hand-color-current-flags.json`). No gamma compensation or hand-validity workaround was added.
|
||||
|
||||
The user's observation that SBS felt smoother coincided with 2D running at 60 Hz and SBS at 90 Hz. A later 120-Hz 2D request still reported a 60-Hz physical mode; its failure cause remains undetermined. The next 90-Hz 2D selection succeeded: Android reports 1920 by 1200 at 90 Hz, and the SDK reports mode `0x43`. The app therefore has no fixed 60-Hz limit for 2D. 2D and SBS retain separate refresh preferences; actual display mode must be checked independently of the requested value. Evidence: `build/hand-color-display-modes.txt` and `build/refresh-mode-final.json`. This confirms the active refresh mode, not sustained 90-fps game rendering or a thermal/performance acceptance result.
|
||||
|
||||
## Color fix, original-image detection, and late synchronization (2026-10-16)
|
||||
|
||||
The HDR-like color change is fixed and confirmed by the user in the glasses. `NewSharedVulkanImage` requested compute access for compositor eye textures even though DroidDeck only needs rendering and sampling. SteamVR replaced the requested `R8G8B8A8_SRGB` format (43) with `R8G8B8A8_UNORM` (37). The untracked-controller compositor path then produced dark linear midtones, while its multisampled controller path encoded colors differently. Removing compute access preserves format 43 in both paths. Temporary Vulkan tracing established the before/after formats and was removed from the delivered build; the presenter retains its existing byte-preserving UNORM import. No gamma compensation was introduced. Evidence: `build/color-format-fixed-trace.json`, `build/color-format-fixed-states.json`, and the corresponding on/off output images. The user reported that colors now stay the same when toggling Hand controllers.
|
||||
|
||||
One explicitly authorized stereo image pair showed upright camera images and two visible open palms. Replaying it with the exact Android CPU detector found one left hand in each camera. All 21 landmarks passed the unchanged 3-pixel reprojection gate (maximum 2.227 px), with reconstructed palm width 7.31 cm and depth approximately 32–39 cm. The right hand was not detected consistently; lower confidence thresholds and exploratory crops did not establish an improvement and were not adopted. Hand detection now consumes the original grayscale images converted to RGBA and corrects landmark coordinates before triangulation, avoiding full-image resampling. This verifies one reconstructable sample, not sustained detection accuracy or two-hand operation. Evidence: `build/android-hand-pair-replay.json`, `build/android-hand-pair-geometry.json`, and `build/hand-pair-replay.json`.
|
||||
|
||||
Previously, camera acquisition waited up to 35 ms for a matching VIO callback before starting hand inference. Inference now starts with the newest pair and checks the exact source timestamp again after detection, allowing VIO processing to overlap detection. Publication still requires a matching callback within 5 ms, a nearby canonical head-history sample, unchanged connection and tracking frame, healthy current head tracking, and an observation no more than 100 ms old. The callback's undocumented pose basis remains unused. Calibration and preview-only acquisition retain their bounded wait. Four additional regression tests cover late confirmation with the original capture pose, missing confirmation, expiry, tracking loss/discontinuity, reconnects, and unavailable history.
|
||||
|
||||
Validation: 53 targeted Android tests pass with no failures or skips; SDK-enabled and SDK-free Android builds and runtime freshness checks pass. The ARM64 VR build passes its native layout, freshness, optical input recovery, render-target, synchronization, and scheduling checks. Temporary color tracing is absent from the final driver source and packaged runtime. Installed SDK-enabled artifact: `build/DroidDeck-spatial-color-sync.apk`, SHA-256 `885fb14b5a30ac65a255a6b0dd519d9f73b72489c08a8b74e1756673b86812ea`. Logs: `build/color-fix-final-native.log`, `build/color-fix-final-stage.log`, `build/spatial-late-sync-build.log`, `build/spatial-late-sync-sdkfree.log`, and `build/spatial-color-sync-install.json`.
|
||||
|
||||
During the subsequent grey-screen report, the compositor continued presenting approximately 60 frames per second, but the SDK returned unstable head status 1 and the shared head pose was invalid. USB timeout messages also appeared; their causal relationship to tracking loss is not established. The clean app/provider restart preserved all 25 calibration views and the verified profile. After tracking recovery and software recentering, the user confirmed both “6DoF active” and the visible SteamVR dashboard. Positional tracking was briefly disabled for diagnosis and restored. No synthetic valid pose or silent fallback was added. Evidence: `build/grey-launch-guest.json`, `build/grey-launch-app.log`, and `build/spatial-color-sync-recovered.json`.
|
||||
|
||||
The next 16-second left-palm test sampled the guest spatial channel 2,900 times: every head was fresh, but no optical controller pose was fresh. Six status samples showed one matched hand in both cameras and successful stereo reconstruction; inference took 139.7–174.3 ms and total processing 144.1–181.8 ms. Those observations correctly expired before publication. The other three status samples were waiting for synchronization. The app restarted between preparation of a temporary GPU benchmark and this measurement, so the active delegate was not verified in the new process and this is not a controlled GPU comparison. A later thermal snapshot reported status 4 and skin readings up to 51.0 degrees C; its contribution to the measured slowdown is unquantified. Evidence: `build/spatial-live-hands-left.json`, `build/spatial-live-hands-left.log`, and `build/spatial-hand-thermal.txt`. The glasses subsequently disconnected, preventing another live comparison. The next measurement needs a stable process/connection, explicit delegate verification, and controlled thermal conditions. Hand controls remain an experimental prototype and have not passed the combined release gate.
|
||||
|
||||
## Tracking-loss recovery screen (2026-10-16)
|
||||
|
||||
Active positional SteamVR and virtual-screen sessions now show a DroidDeck-themed recovery screen over the rendered output when head tracking becomes invalid. Separate states explain how to recover tracking, reconnect USB, recenter a changed origin, or hold still while recentering settles. The screen is rendered independently of the head pose, supports mono and SBS output, and closes automatically after valid tracking returns. The phone retains access to Recenter and glasses setup. No tracking validity, input-expiry, calibration, or driver behavior was relaxed.
|
||||
|
||||
Validation: 65 targeted Android tests pass without failures or skips, including callback-free stale-pose detection, recovery states, presentation surface preservation, mono/SBS rendering, and theme colors. Robolectric native-rendered previews were visually inspected at 1080 by 600 and 3840 by 1200; this is automated rendering evidence, not a physical glasses acceptance check. SDK-enabled and SDK-free Android builds and packaged-runtime freshness checks pass. Logs and previews: `build/tracking-recovery-build.log`, `build/tracking-recovery-sdkfree.log`, and `app/build/reports/tracking-recovery/`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-tracking-recovery.apk`, SHA-256 `2170c2e513d45af048b9aba01b76a694d333ba9848a91021a47d413fca3d282d`. Installation preserved app data and was performed with the session idle; SteamVR was not started. The installed APK hash matches the archived artifact (`build/tracking-recovery-install.json`). A live tracking-loss/recovery check through the glasses remains pending.
|
||||
|
||||
## Cooled-phone retest and visible left controller (2026-10-16)
|
||||
|
||||
The Fold7 initially reported thermal status 1, approximately 40.3 degrees C skin temperature, healthy USB access, and the saved 25-view calibration. Head tracking recovered to valid status 0. CPU inference was verified through the live setup control in process 17409, session 12. By the 16-second left-palm measurement, thermal status had returned to 4. All nine diagnostic samples detected and reconstructed a matching hand in both cameras, but inference took 124.4–173.3 ms. The guest channel had 3,038 fresh head reads and zero fresh hand reads. This does not establish an unthrottled CPU baseline. Evidence: `build/cooled-cpu-verified-ui.json`, `build/spatial-live-hands-cooled-cpu.json`.
|
||||
|
||||
Enabling GPU inference while that SteamVR session was running initialized both GPU graphs, including OpenCL loading, at 01:10:24–01:10:27 device-local time. The compositor first reported `vkerror=-4` at 01:10:29.516; automatic graphics recovery then killed and restarted the app. Process 27927, session 13, was back on CPU. The subsequently captured file named `build/spatial-live-hands-cooled-gpu.json` therefore contains CPU observations after recovery and must not be counted as a GPU comparison. Process IDs, session IDs, the setup control returning to Off, and Android exit records establish that reset. The timing associates GPU initialization with the device-loss event but does not establish its underlying driver cause. Evidence: `build/cooled-before-restart-logcat.txt`, `build/cooled-session12-vrcompositor.txt`, `build/cooled-session12-events.jsonl`, `build/cooled-exit-info.txt`, `build/cooled-gpu-after-ui.json`.
|
||||
|
||||
With SteamVR stopped, GPU inference was enabled and verified in process 27927. Twelve diagnostic samples then measured 10.3–14.1 ms inference and 12.0–19.7 ms total observation age, with healthy head tracking. These samples were not a prescribed palm-hold accuracy test; most saw a hand in only one camera and none produced accepted geometry. Evidence: `build/cooled-idle-gpu-verified-ui.json`, `build/cooled-idle-gpu-sample.json`.
|
||||
|
||||
Launching SteamVR after GPU initialization preserved process 27927 for session 14. During a fresh 16-second left-palm hold, the guest channel received 67 distinct valid left-hand observations. The left hand was fresh in 315 of 2,864 coherent channel reads (11.0%); all heads were fresh. This percentage measures sampled channel availability, not the acceptance target's percentage of camera observations. The user confirmed a controller appeared or flickered. Eight diagnostic samples all reconstructed the left hand, but inference varied from 41.8–101.6 ms and observation age from 72.5–131.9 ms. Thermal status remained 4, with reported skin temperature increasing from 49.1 to 51.3 degrees C. The session also logged another Vulkan device-loss event, so initializing GPU inference before SteamVR did not resolve stability. Evidence: `build/spatial-live-hands-prewarmed-gpu.json`, `build/cooled-prewarmed-start.json`, `build/cooled-session14-events.jsonl`.
|
||||
|
||||
This confirms a visible optical left-controller path, with no relaxed timestamp or geometry checks. Both-hand reliability, gesture operation, measured end-to-end latency, and sustained performance remain unvalidated. No camera images were saved during these tests, and calibration was not changed. The run was stopped after the repeated graphics fault. After confirming the session was idle and its artifacts complete, DroidDeck was closed to stop background inference and clear the temporary GPU benchmark; saved settings and calibration were retained. The next work must address GPU/SteamVR coexistence and inference latency under sustained thermal load; another calibration is not indicated by these successful left-hand reconstructions.
|
||||
|
||||
## Requested 200 ms experimental hand cutoff (2026-10-16)
|
||||
|
||||
The user requested accepting hand observations up to 200 ms old to reduce disappearing controllers while inference is slow. The cutoff now applies consistently to inference admission/completion, capture-pose lookup, Android publication, and SteamVR optical pose/input freshness. Observation timestamps are unchanged. Head freshness remains 100 ms, and detected loss, changed tracking frames, reconnects, and recentering still invalidate hands immediately. Missing observations can now retain trigger/grip until the 200 ms expiry; reacquisition still requires an observed release. This intentionally changes the earlier 100 ms hand-expiry requirement, without claiming lower processing latency or resolving the Vulkan faults. The diagnostics field `handMaxAgeMs` reports the active limit.
|
||||
|
||||
Validation: 73 targeted Android tests pass, including a 150 ms inference result using its original capture pose, the inclusive 200 ms boundary, rejection immediately after that boundary, unchanged 100 ms head expiry, and independent hand timestamps. Native tests pass the same hand/head boundaries, pose validity, stale-input release, and reacquisition behavior. The ARM64 driver was rebuilt, staged, and verified against source/output checksums; SDK-enabled and SDK-free Android builds pass. Logs: `build/hand-200ms-native.log`, `build/hand-200ms-stage.log`, `build/hand-200ms-build.log`, `build/hand-200ms-sdkfree.log`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-200ms.apk`, SHA-256 `dac9296cbc964fca87b65b7e1e8b87276deaaeb5d90ca731d47eae673e22c64d`. Its packaged driver matches the rebuilt ARM64 binary, SHA-256 `bf24667f9f288cd07efe6e100c3b237b1e68b734100ae99cf93974538ce3bc6a`. The installed APK hash and live `handMaxAgeMs: 200` were verified. Calibration and saved settings were preserved. SteamVR was not launched, and the app was closed after verification to stop background inference (`build/hand-200ms-install.json`). Live controller availability, gesture delay, and release behavior with the new cutoff remain unmeasured.
|
||||
|
||||
## GPU default requested by the user (2026-10-16)
|
||||
|
||||
The user explicitly requested GPU inference as the default. Every new stereo hand engine now selects GPU, including after app restart, USB reconnect, and provider restart. The setup control is renamed **GPU hand inference** and explains that disabling it selects CPU until the next tracking restart. Diagnostics distinguish `handGpuEnabled` (selection) from `handInferenceBackend` (GPU/CPU only after both detectors initialize, otherwise `not initialized`). This replaces the earlier CPU-default policy; it does not resolve the measured Vulkan failures or constitute performance acceptance. The 200 ms hand cutoff and gesture mappings are unchanged.
|
||||
|
||||
Validation: 73 targeted Android tests pass, including pinch/grip mapping, expiry, tracking recovery, and setup UI. SDK-enabled and SDK-free builds and runtime freshness checks pass. The packaged driver still matches the previously tested 200 ms ARM64 binary. The active SteamVR test was stopped cleanly before installation. GPU selection and the verified calibration were confirmed in two distinct app processes, 22127 and 23157, separated by a full app restart. Head tracking was valid, but hand diagnostics reported waiting for camera/head synchronization and the detectors reported `not initialized`; these checks verify default selection and profile persistence, not a new live GPU inference or gameplay result. SteamVR was not relaunched.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-gpu-default.apk`, SHA-256 `a32feb647b0ca9fb6f62bd31158450b58e5671de7c1b0324f6b0b56bd4c92a47`. The installed APK hash matches. Evidence: `build/hand-gpu-default-build.log`, `build/hand-gpu-default-sdkfree.log`, `build/hand-gpu-default-install.json`. Click instructions are documented: present an open hand, aim the virtual controller, pinch thumb and index finger to press, and separate them to release. Curling the middle, ring, and little fingers supplies grip; tracking loss requires another observed open/released gesture before input can resume.
|
||||
|
||||
## CPU default restored and model optimization investigated (2026-10-16)
|
||||
|
||||
After further SteamVR restarts, the user requested reverting to CPU. New engines, the no-engine diagnostics fallback, and the setup UI now default to CPU again. The optional GPU toggle explains the observed restart risk and resets on tracking restart. Calibration, the color fix, and the requested 200 ms hand cutoff are retained. GPU device loss has not been fixed or ruled out under other configurations.
|
||||
|
||||
Validation: 73 targeted Android tests pass with no failures, errors, or skips. SDK-enabled and SDK-free builds pass; the SDK-free APK excludes the optional VITURE bridge. The SDK-enabled APK's packaged ARM64 driver matches the previously tested 200 ms binary. CPU selection (`handGpuEnabled: false`) and the verified 25-view calibration were confirmed in two distinct processes, 3540 and 3717, across a full app restart. Head tracking was lost and hand detectors were `not initialized` during these checks, so this verifies the restored default and preserved calibration, not new live CPU performance or SteamVR stability.
|
||||
|
||||
Installed artifact: `build/DroidDeck-hand-cpu-restore.apk`, SHA-256 `5ae8f702fdce16501a69731fd5b0e718198e267276e918322b15228ae3b95282`. The installed APK hash matches. Driver SHA-256 remains `bf24667f9f288cd07efe6e100c3b237b1e68b734100ae99cf93974538ce3bc6a`. Evidence: `build/hand-cpu-restore-build.log`, `build/hand-cpu-restore-sdkfree.log`, and `build/hand-cpu-restore-install.json`. SteamVR was not launched for this verification.
|
||||
|
||||
The shipped bundle already uses float16 weights. This does not establish float16 CPU arithmetic or a speed advantage: Google's [float16 documentation](https://developers.google.com/edge/litert/conversion/tensorflow/quantization/post_training_float16_quant) describes CPU weight expansion to float32. [INT8 quantization](https://developers.google.com/edge/litert/quantization/model_optimization) is a possible separate experiment, subject to runtime/operator compatibility and stereo accuracy validation; no INT8 conversion or speed claim has been made.
|
||||
|
||||
An offline feasibility experiment compared Google's [legacy lite palm and hand landmark models](https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/models.md) against the installed full bundle using the previously authorized stereo image pair. No new images were captured. The lite models initially failed the Tasks loader because normalization metadata was absent. For this local experiment, input/output tensor names, shapes, and types were checked against the full models before copying their metadata and associated files. The 0–1 input normalization agrees with Google's [hand landmark graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/modules/hand_landmark/hand_landmark_cpu.pbtxt) and [palm graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/modules/palm_detection/palm_detection_cpu.pbtxt). Experimental bundles remain under `build/hand-lite-research/`; the app still ships the original checksum-pinned full bundle.
|
||||
|
||||
Windows CPU replay used MediaPipe 0.10.21, two sequential VIDEO instances, two-hand capacity, unchanged 0.6 confidence thresholds, 10 warmup pairs and 30 timed repetitions of one static pair. Times cover stereo inference only, excluding camera acquisition, preprocessing, geometry, rendering, and Android/game contention.
|
||||
|
||||
| Models | Median stereo inference | p95 stereo inference | Static-pair geometry |
|
||||
|---|---:|---:|---|
|
||||
| Full baseline, before alternatives | 84.54 ms | 97.28 ms | Left hand passes in 30/30 repetitions |
|
||||
| Lite palm + full landmarks | 37.11 ms | 43.94 ms | No hands detected; unusable for this sample |
|
||||
| Full palm + lite landmarks | 62.48 ms | 73.18 ms | Left hand passes in 30/30 repetitions |
|
||||
| Both lite | 36.75 ms | 43.69 ms | No hands detected; unusable for this sample |
|
||||
| Full baseline, after alternatives | 87.99 ms | 100.15 ms | Left hand passes in 30/30 repetitions |
|
||||
|
||||
Full palm plus lite landmarks is approximately 26–29% faster by median in this limited desktop replay. Its first-frame maximum reprojection error is 1.83 px versus 2.23 px for full; this single-pose result is not an accuracy comparison across real interactions. Repeated images do not validate motion, tracking availability, both hands, reacquisition, or thermal performance. Android Tasks 1.0.0 compatibility and temperature-matched Fold7/SteamVR measurements are still required before adopting it. The apparent speed of the lite-palm variants is not useful evidence of faster working tracking because they performed no landmark inference on a detected hand.
|
||||
|
||||
Research artifacts: `build/compare-hand-lite.py`, `build/hand-lite-research/results.json`, `build/hand-lite-research/without-metadata.json`, and `build/hand-lite-research-metadata.log`. Original downloaded SHA-256 values: palm lite `e9a4aaddf90dda56a87235303cf00e4c2d3fb28725f68fd88772997dac905c18`; landmarks lite `d7fde8ac11f8ce03f8663775bfc323f4fc9f2a38062b4f4efa142874ef5b2a48`.
|
||||
|
||||
Another candidate is reducing repeated palm searches without losing second-hand acquisition. The [upstream HandLandmarker graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc) runs detection whenever tracked hands are fewer than the configured maximum, including one visible hand with a two-hand limit. This is a profiling lead, not a measured attribution in the packaged Android runtime. Simply setting the limit to one would remove the required two-hand behavior; implementing a controlled search cadence would require a different graph or pipeline. The existing bounded latest-frame queue and VIDEO tracking already avoid unbounded backlog and unconditional IMAGE-mode detection.
|
||||
|
||||
## Brief visual hold and remaining detection gaps (2026-10-16)
|
||||
|
||||
After the user reported recovery following cooling, a 16-second CPU sample kept the same Android process (10308) and SteamVR session (`2026-10-16-29-steam`). Head tracking was fresh in 2,851 of 2,857 channel samples (99.8%), but the left hand was fresh in only 22 samples (0.77%), representing six distinct poses. Those poses first reached the monitor 162.2–198.3 ms after observation. Right-hand output was absent during the requested left-hand-only test. Diagnostic snapshots alternated between camera/head synchronization waits and no detected hands; measured inference in the snapshots that ran detection ranged from 76.1 to 136.8 ms. This is not a controlled cool-device benchmark: at the end Android reported thermal status 4 and the current HAL SKIN sensor read 51.5 C. Evidence: `build/spatial-live-hands-cooled-flicker.json` and `build/cooled-flicker-measure.log`.
|
||||
|
||||
The driver previously invalidated its rendered controller immediately whenever an inference result contained no hand, even if the previous observation was still younger than the requested 200 ms cutoff. It now caches only the most recent accepted optical transform for visual continuity through those short gaps. Expiry remains measured from the original camera observation; a missed result never extends it. Optical input processing remains separate and releases trigger/grip immediately on a miss. Reacquisition still requires a released gesture. Head loss, connection/origin changes, disconnect, and switching back to gamepad-positioned controllers clear the cache. Missing optical tracking no longer suppresses independently available gamepad menu and locomotion inputs. This is a partial flicker mitigation, not a remedy for the longer detection gaps or old poses measured above.
|
||||
|
||||
Native regressions pass for immediate gesture release during a held visual pose, its unchanged timestamp and inclusive 200 ms boundary, rejection after expiry, rearming, head loss/recovery, origin/connection changes, disconnect, mode changes, malformed/future/out-of-order samples, and gamepad availability during optical loss. The other native rendering, fence, and scheduling regressions pass. SDK-enabled and SDK-free Android packaging and runtime source/output freshness checks pass. Android application source was unchanged in this step; the existing 73 passing Android test results were checked during installation rather than rerun. Evidence: `build/hand-visual-hold-native.log`, `build/hand-visual-hold-stage.log`, `build/hand-visual-hold-build.log`, and `build/hand-visual-hold-sdkfree.log`.
|
||||
|
||||
Installed artifact: `build/DroidDeck-hand-visual-hold.apk`, SHA-256 `977b993433803f3d8abbd639ca636b874367467866b58f7af75a07a46b20daa2`. Its packaged ARM64 driver matches the tested build, SHA-256 `2d7ff8647f210248f6ea4b68e9b05e21ed7ebcc74da601da0f1d854067b7ab82`. Installation occurred with the session idle and preserved the calibration/profile files. CPU selection and the 200 ms setting were confirmed across two app processes. The glasses were disconnected during these checks, so calibration was not loaded and detectors were not initialized; this does not establish a live improvement. SteamVR was not relaunched. Evidence: `build/hand-visual-hold-install.json` and `build/hand-visual-hold-live.json`.
|
||||
|
||||
The user explicitly authorized one new diagnostic stereo pair and separately authorized copying that pair and timing metadata to the local workspace. Both 640 by 480 images visibly contain the palm, near the outer edge of the left image. The saved pair was not synchronized to a head pose. Desktop MediaPipe 0.10.21 replay of the current full model failed to detect either hand at palm-detection thresholds 0.6, 0.5, 0.4, and 0.3, with landmark-presence confidence held at 0.6. Undistorting the images before detection did not recover this sample. Lowering both confidence thresholds to 0.3 also failed. A fixed right-side image tile detected a left hand in only one camera, insufficient for stereo reconstruction. The earlier authorized pair still detected the left hand in both cameras and passed the unchanged stereo geometry gate in the same replay. These static diagnostic experiments do not justify changing thresholds, shipping a fixed crop, or claiming general detection improvement. No inference model, thresholds, or stereo quality gates were changed. Local artifacts: `build/flicker-pair-replay.json`, `build/flicker-pair-scale.json`, and their scripts/logs.
|
||||
|
||||
The earlier CPU crash investigation also observed Vulkan device loss with hand inference disabled. A temporary `TU_DEBUG=noubwc` session reproduced it and was stopped; the flag was not persisted and no production graphics change was made. Cooling may have contributed to the user's reported recovery, but neither overheating nor inference delegate choice is established as the sole crash cause.
|
||||
|
||||
## Half-rate stereo input requested by the user (2026-10-16)
|
||||
|
||||
The supplied SDK 2.4.0 Carina header documents 25 Hz camera/pose callbacks and exposes automatic/manual exposure controls, but no tracking-camera frame-rate setter. DroidDeck now selects every other callback for its own stereo pixel copies while hand controllers are enabled, yielding approximately 12.5 pairs per second with the observed 25 Hz input. The gate executes before native buffer resize/copy. Skipped pairs do not reach JNI transfer, grayscale/RGBA conversion, or hand inference. Vendor camera acquisition/VIO, pose callbacks, and the existing head polling interval are unchanged. This is software input decimation, not a firmware camera-rate change.
|
||||
|
||||
Calibration collection, distance validation, explicit diagnostic capture, and preview-only use bypass the half-rate gate. The first frame after a gate-mode change or camera-source reset is accepted. The latest eligible pending pair still replaces older eligible pending work. Camera-source/receipt timestamps and the existing 200 ms hand expiry are unchanged. JNI metadata carries the incoming frame index associated with the copied pair and the cumulative intentional skip count, so diagnostics continue to report the true incoming camera rate. `cameraFrameDivisor: 2` identifies hand input at half rate, and `rateLimitedFrames` is separate from processing-related `dropped` frames. The setup UI explains both rates and counters.
|
||||
|
||||
The native regression passed 250 source callbacks producing 125 copied pairs, full-rate setup, repeated configuration, source reset, invalid timestamps, and reconnect reset. All 75 targeted Android tests passed, including half-rate camera-clock mapping and UI separation of incoming rate versus intentionally skipped frames. All 13 runtime-preparation tests passed, including classifying the new native helper as locally compiled while rejecting unrelated donor-native reuse. SDK-enabled and SDK-free builds and runtime freshness checks passed. Tests: `tools/tests/test_stereo_frame_gate.cpp`; logs: `build/hand-half-rate-build.log`, `build/hand-half-rate-sdkfree.log`, `build/hand-half-rate-runtime-tests.log`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-half-rate.apk`, SHA-256 `c02d56e40802de5c5e9d9750234b24a85462652f472729fbc9696935654f95fb`. The installed hash was verified. The SteamVR driver is unchanged from the tested visual-hold build (`2d7ff8647f210248f6ea4b68e9b05e21ed7ebcc74da601da0f1d854067b7ab82`); the Android SDK bridge and app were rebuilt. Installation preserved app data and occurred with SteamVR stopped. CPU selection and the 200 ms expiry were confirmed across two processes. The glasses were disconnected, so the live half-rate cadence, inference cost, thermal benefit, and controller availability have not been measured. Evidence: `build/hand-half-rate-install.json`.
|
||||
|
||||
At 12.5 Hz, eligible pairs are approximately 80 ms apart. The earlier CPU inference snapshots already took 76–137 ms, so reducing input is not expected to halve total CPU load. It may save image-copy/preprocessing work and reduce inference load when the worker is fast enough, while also increasing frame waiting time and potentially reducing chances to recover a missed detection. It does not guarantee a successful observation or a visible response within 100 ms. A controlled comparison at matching temperatures is required before claiming smoother controllers or less overheating.
|
||||
|
||||
## SteamVR shared-eye image import correction (2026-10-16)
|
||||
|
||||
After the half-rate installation, sessions 31 and 32 hit `VK_ERROR_DEVICE_LOST` approximately 11.7 and 10.8 seconds after READY. DroidDeck's Android process and native 6DoF remained alive; CPU inference, a camera divisor of 2, approximately 25 Hz incoming cameras, and intentionally skipped frames were confirmed. This establishes live operation of the input limiter, but no thermal/performance benefit. The Android crash buffer contained an older VITURE abort in another process, not a new app crash. Evidence: `build/half-rate-crash-state.json`, `build/half-rate-crash-glasses.json`, and `build/half-rate-crash-guest.json`.
|
||||
|
||||
A temporary `DROIDDECK_VR_IMPORT_FORMAT_LIST=0` test ran 49 seconds without device loss with hands disabled. A second test with CPU hands was stopped after the user reported a grey, degraded image. The override is rejected and was not persisted. Restoring the normal import path reproduced device loss in session 35. Neither a successful launch nor removal of image-format metadata is an acceptable fix.
|
||||
|
||||
The normal-path Vulkan trace identifies an external-memory image-description mismatch: SteamVR exports each 1728 by 1080 eye image as format 43 (sRGB), flags `0x8`, usage `0x97`, with view formats 43/37, whereas the presenter imported it as format 37 (UNORM), flags `0x8`, usage `0x85`. Dedicated OPAQUE_FD imports require matching image descriptions. The presenter now retains the exported format, rendering/transfer usage, and view-format list. Presentation selects the corresponding sRGB or UNORM surface format, so an sRGB blit both decodes and re-encodes instead of darkening the display. It rejects incompatible surface encodings rather than silently selecting one, and checks imported-memory binding failures. The existing SteamVR compute-access color fix remains intact. Trace: `build/crash-import-trace/guest.json`. Requirement: [Vulkan dedicated allocation, VUID 01878](https://docs.vulkan.org/refpages/latest/refpages/source/VkMemoryDedicatedAllocateInfo.html).
|
||||
|
||||
The ARM64 build and native regressions pass, including a new regression against the captured SteamVR image description and display-format selection/rejection cases. SDK-enabled and SDK-free APK builds and runtime freshness checks pass. The prior 75 Android tests remain applicable; Android source was unchanged for this graphics correction. Installed SDK-enabled artifact: `build/DroidDeck-shared-eye-format.apk`, SHA-256 `c573169cd46b20b70d704cd92f9f07550d5aa128d0c4300c1c52e635f04958fb`; driver SHA-256 `51dbd719e56c43df6c37edef2d7f426e91183b2b8489285349df19c322090f3b`. Installation preserved the verified calibration, CPU inference, and 200 ms cutoff across two app processes. Logs: `build/shared-eye-format-{native,stage,build,sdkfree,install}.log` and `build/shared-eye-format-install.json`.
|
||||
|
||||
Two subsequent sessions completed 121.9 and 122.7 seconds after READY without a new device-loss event. All sampled head poses were valid, CPU inference stayed active, frame processing advanced, and Android PID 659 remained unchanged. Session 36 initially allocated 1080 by 1080 eyes; its trace confirms identical exported/imported format 43, usage `0x97`, and formats 43/37. Session 37 ran without tracing and allocated the original 1728 by 1080 eyes, removing the lower-resolution confound. Both used a 3840 by 1200 sRGB presentation surface (format 50) and showed approximately 600 frames per ten-second interval after startup. The cumulative compositor log contains older device-loss messages; filtering to the new sessions found no fresh device-loss error. Occasional `WaitForAcquire` warnings remain. Android thermal status reached 4, with current HAL skin temperatures 46.7 and 48.4 degrees Celsius after the respective runs; these are not controlled thermal comparisons.
|
||||
|
||||
These are startup stability checks, not a twenty-minute gameplay/performance pass or proof of every crash's cause. Both test sessions were stopped; all temporary environment overrides and the trace marker were cleared, with calibration and CPU hand settings verified afterward. Evidence: `build/shared-eye-format-live/`, `build/shared-eye-format-relaunch/`, `build/shared-eye-log-audit.json`, and `build/shared-eye-final.json`.
|
||||
|
||||
The user subsequently confirmed normal colors but reported another crash as soon as a hand appeared. Session 38 logged fresh Vulkan device loss at 08:44:25 UTC, about 42 seconds after READY, with a burst of geometry pipeline creation immediately beforehand. Recovery stopped the session, and Android later recorded SIGKILL for the app process; the initial failure was in SteamVR's compositor. The color correction is accepted, but the crash is not fixed. In session 39, the dashboard remained alive with valid head tracking and no detected optical hands, then failed less than four seconds after switching to the existing gamepad controller poses at 20.47 seconds. That reproduces the controller-visibility trigger independently of reconstructed hand coordinates. Evidence: `build/hand-visible-crash/`, `build/hand-visible-exits.txt`, and `build/controller-visibility/`. The preceding two-minute tests did not validate this transition.
|
||||
|
||||
## Visible-controller crash and compositor completion guard (2026-10-16)
|
||||
|
||||
An opt-in diagnostic waited for the compositor's graphics queue after each submission. Session 40 repeated the same optical-to-gamepad-controller transition at 18.01 seconds and stayed alive through 80.94 seconds, with approximately 600 presented frames per ten-second interval. Session 39 without the diagnostic failed less than four seconds after this transition. This supports a GPU synchronization problem in the visible-controller rendering path; it does not establish a specific bad hand sample or thermal cause. An [upstream SteamVR report](https://github.com/ValveSoftware/SteamVR-for-Linux/issues/952) describes premature command-buffer/descriptor-pool reuse, but equivalence to this device's failure is not directly proven. Diagnostic evidence: `build/controller-submit-idle/` and `build/submit-idle-install.json`.
|
||||
|
||||
The shipped experimental guard applies only inside `vrcompositor`. Each `vkQueueSubmit` or `vkQueueSubmit2` waits for its actual completion fence before returning. If SteamVR supplied its own fence, that fence remains attached to the original submission and a following empty submission signals DroidDeck's separate fence. Waiting is bounded to one second; timeout fails the submission path instead of reporting success. The tracker keeps the fence reserved during this wait, preventing the completion worker from resetting/reusing it; a timed-out but submitted fence stays pending until completion or device failure. Other processes retain asynchronous submission. This is a conservative workaround that reduces CPU/GPU overlap, so gameplay frame-time and long-duration acceptance remain required. The temporary `DROIDDECK_VR_SUBMIT_IDLE` probe was removed.
|
||||
|
||||
Native regressions cover both submission APIs, caller-owned fences, submission/marker failures, successful completion, device loss, timeout retention, tracker allocation failure, and bypass outside the compositor. Existing native rendering, spatial, optical-controller, synchronization, and scheduling checks also pass. SDK-enabled and SDK-free builds and runtime freshness checks pass; the prior 75 Android tests apply because Android source was unchanged. Installed APK: `build/DroidDeck-compositor-fence.apk`, SHA-256 `27d44a138d1f6bf6a0698abcd0a48107d31815c8e57a9f7ca07113fabb213195`. Installation preserved the verified calibration, CPU inference, 200 ms hand expiry, and half-rate camera input. Evidence: `build/compositor-fence-{native,stage,build,sdkfree,install}.log` and `build/compositor-fence-install.json`.
|
||||
|
||||
Session 42 repeated the controller-visibility regression on the installed bounded-fence build. Head tracking remained valid; switching to the existing gamepad controller poses at 17.55 seconds did not cause device loss through the final sample at 58.92 seconds. The presenter reported 593–601 frames per ten-second interval, with no late frames in those counters. Optical hands were absent before the switch, so this passes the reproduced synthetic-controller visibility trigger but does not yet validate an actual optical hand appearing. The successful session was left open with CPU optical tracking restored for that check. Evidence: `build/controller-fence-guard/` and `build/controller-fence-guard.log`.
|
||||
|
||||
The built, packaged, and installed Vulkan layers all have SHA-256 `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`; no pending or persistent diagnostic environment override remained. The live audit was repeated after the session ended and still confirmed the installed layer, saved calibration, CPU selection, and enabled hand-controller setting. Evidence: `build/compositor-fence-live-audit.json`.
|
||||
|
||||
## Optical appearance confirmed; flicker and pinch remain unresolved (2026-10-16)
|
||||
|
||||
The user confirmed that optical controllers appeared without freezing, but continued to flicker and could not click. This is a successful user-observed optical appearance on the compositor-fence build, not a sustained tracking or interaction pass. The color correction remains accepted.
|
||||
|
||||
A subsequent 16-second CPU baseline in session 43 sampled 2,846 coherent channel snapshots, all with fresh head tracking and none with a valid hand. The requested deliberate open/pinch/release sequence had not yet been confirmed, so absence of hands in this baseline cannot establish detector failure on a visible palm. Diagnostic samples measured 111.7–154.1 ms stereo inference; three observations exceeded the unchanged 200 ms cutoff, with completion ages 204.9–221.5 ms. Android stayed in process 32010 and the same READY SteamVR session. Thermal status was 4, with current HAL skin temperature 51.8 C at the end; this is not a controlled cool-device performance comparison. Evidence: `build/spatial-live-hands-pinch-baseline.json` and `build/pinch-baseline.log`.
|
||||
|
||||
The driver emits `/input/trigger/value` and `/input/trigger/click` from optical pinch. Any invalid/stale hand or head sample releases inputs and disarms gestures until an observed open/released hand, while the visual pose may remain only until its original observation reaches 200 ms. Repeated observation loss can therefore interrupt clicking as well as controller visibility. No deliberate pinch was measured in the new baseline, so this is an explanation of the implemented behavior, not proof of the user's missing-click cause. The next measurement records hand timestamps, validity transitions, trigger strengths, grip, and connection/origin changes without camera images (`build/measure-pinch.py`). No inference thresholds, model, expiry, delegate default, or graphics behavior were changed in this follow-up.
|
||||
|
||||
## Palm continuity and independent gestures implemented (2026-10-16)
|
||||
|
||||
Hand reconstruction now validates each triangulated landmark independently under the unchanged depth and 3-pixel reprojection limits. A controller requires wrist and four palm knuckles; pinch and grip each have separate required fingertips and validity. A rejected finger no longer removes an otherwise accepted palm. Board calibration and distance checks still reject a set containing any invalid point. Protocol version 2 retains the 176-byte layout and carries explicit per-gesture validity bits; old protocol versions are rejected. The native driver releases only the uncertain gesture and requires its observed release before rearming. Unknown finger data cannot arm a click.
|
||||
|
||||
The driver smooths palm translation and rotation with a 15 ms time constant. Three consecutive observations with consistent velocity can enable at most 30 ms / 2 cm of translation prediction. Prediction stops on missing/invalid observations and origin/connection/head changes; the original 200 ms observation expiry is unchanged. Pose filtering never generates or holds gesture input. Native regressions cover independent gesture loss/rearming, old-version rejection, quaternion sign equivalence, prediction bounds, loss holds, expiry, and origin resets. Android regressions cover rejected fingertip versus rejected palm behavior, independent pinch/grip validity, and unchanged metric/reprojection checks.
|
||||
|
||||
Both full and full-palm/lite-landmark models are packaged and SHA-256 pinned. The lighter bundle is reproducible with `tools/build_hand_lite.py` from the documented original models and MediaPipe 0.10.21 metadata tooling. Diagnostics offer temporary model and half/full-input controls; CPU, full model, and half-rate input remain defaults pending a live comparison. An opt-in debug command benchmarks existing saved stereo pairs on CPU without capturing images or publishing replayed hand poses. It clears live hands during replay, closes both temporary detectors, and resumes the selected live model afterward.
|
||||
|
||||
All 259 selected Android tests pass with no failures/errors/skips, including the glasses package and spatial-setup UI. The ARM64 native tests, SDK-enabled and SDK-free Android builds, and runtime freshness checks pass. Installed APK: `build/DroidDeck-hand-reliability.apk`, SHA-256 `38b9b67171b37b0bf80b540da165894ff4cbb24b609ae62edd829ecf618d01e3`. Driver SHA-256: `56b3a3eeaa96160e5cbf740f35b701f935b610417dbbba0aa2965ec109d5418a`. The graphics synchronization layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. Installation preserved the verified calibration and CPU selection across processes 2843 and 3996. The built, packaged, and deployed driver/layer match; the Android publisher produces a version-2, 176-byte spatial file. Evidence: `build/hand-reliability-{tests,native,stage,build,sdkfree,install,audit}.log` and corresponding install/audit JSON.
|
||||
|
||||
The Fold7 successfully loaded both model bundles using Android Tasks 1.0.0 on CPU. Saved-pair replay ran full/lite/full, each with four warmup and twenty measured pairs, outside SteamVR in the same process. Times below cover stereo inference, excluding model construction, camera acquisition, geometry, publication, and display latency.
|
||||
|
||||
| Saved pair / model | Median inference | p95 inference | Valid palms |
|
||||
|---|---:|---:|---:|
|
||||
| Previously detectable pair / full before | 104.95 ms | 107.95 ms | 20/20 |
|
||||
| Previously detectable pair / lite landmarks | 90.03 ms | 95.54 ms | 20/20 |
|
||||
| Previously detectable pair / full after | 103.89 ms | 106.20 ms | 20/20 |
|
||||
| Harder pair / full before | 60.77 ms | 63.88 ms | 0/20 |
|
||||
| Harder pair / lite landmarks | 62.64 ms | 66.53 ms | 0/20 |
|
||||
| Harder pair / full after | 62.48 ms | 65.74 ms | 0/20 |
|
||||
|
||||
The lite candidate is approximately 13–14% faster on the detectable static pair. Both models recognize a released pinch/grip and retain their validity on all twenty repetitions; their first measured palm positions differ by about 1.9 mm. This is agreement between models on one static observation, not ground-truth accuracy or stationary-jitter validation. Neither detects the harder saved pair, so reducing landmark cost does not solve missed palm detection. Thermal status ended at 2, with current HAL skin temperature 41.4 C. Evidence: `build/hand-reliability-benchmark-idle.json` and `.log`.
|
||||
|
||||
SteamVR attempts 44 and 45 were suspended by Android with the screen off/lock screen showing before their first displayed frame. Wake/dismiss requests did not dismiss the lock screen. A user unlock is required for the graphics-workload model comparison and a fresh optical open/pinch/release check; neither is counted as a pass. The full model remains the default until that comparison can be completed. The changes are installed as an experimental candidate; reduced live flicker and successful dashboard clicking have not yet been established.
|
||||
|
||||
## Concurrent CPU inference and controller cadence (2026-10-16)
|
||||
|
||||
Left and right CPU inference now run concurrently with separate image buffers and a stable owner for the right detector. Both calls finish before either frame buffer can be reused or either detector closed, including failure and interruption paths. GPU inference remains sequential and opt-in. Controller presentation now approaches the filtered pose on each driver update with a 15 ms time constant, rather than moving only when a new inference result arrives. Missing observations stop prediction and hold the last displayed transform only until its original 200 ms expiry. Gesture validity, input release, head freshness, and connection/origin resets are unchanged.
|
||||
|
||||
All 263 selected Android tests pass without failures, errors, or skips, including four concurrent-worker lifecycle tests. Native spatial regressions cover intermediate render updates, normalized rotations, loss holds, and unchanged expiry. The ARM64 build, SDK-enabled and SDK-free APK builds, and packaged-runtime freshness checks pass. Installed APK: `build/DroidDeck-hand-cadence.apk`, SHA-256 `7df588f0eacdbf4f5ddaae91c511e415b01fe39f37100c012e8f8da3fced52e0`. Driver SHA-256: `195261f27043b260052873a3b083c436457763caa5ebb7bcd1f52708e426bf11`. The graphics layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. Built, packaged, and deployed binaries match; the spatial channel remains version 2, 176 bytes. Installation preserved CPU selection and verified calibration across processes 13863 and 14542. Evidence: `build/hand-cadence-{native,stage,build,sdkfree,install,audit}.log` and corresponding install/audit JSON.
|
||||
|
||||
A saved-pair comparison ran under SteamVR in one Android process/session using the previously authorized pair `64390821246320`. Each configuration had four warmup and twenty measured pairs. Times cover stereo inference, not acquisition, reconstruction, publication, or display latency.
|
||||
|
||||
| CPU configuration, in test order | Median inference | p95 inference | Valid left palms |
|
||||
|---|---:|---:|---:|
|
||||
| Full, sequential before | 202.74 ms | 286.60 ms | 20/20 |
|
||||
| Full, concurrent | 133.19 ms | 162.33 ms | 20/20 |
|
||||
| Lite landmarks, concurrent | 104.18 ms | 165.52 ms | 20/20 |
|
||||
| Full, sequential after | 176.31 ms | 208.47 ms | 20/20 |
|
||||
|
||||
Concurrent full inference had a 24–34% lower median than the surrounding sequential runs. Its first palm position matched the full sequential result exactly; the lite result differed by about 1.9 mm. All repetitions retained released pinch/grip validity. These are repeated static-pair results, not physical accuracy, dynamic coverage, or clicking acceptance. Android thermal status was 4 throughout, with current HAL skin temperature increasing from 48.8 to 50.6 C. The ordered test was not temperature controlled. The lite model's lower median did not improve its p95 over concurrent full inference; the full model remains the default. Evidence: `build/hand-reliability-benchmark-steamvr-parallel.json` and `build/hand-cadence-benchmark.log`.
|
||||
|
||||
For the live tests, the user confirmed an open left palm approximately 12–18 inches away, moving slowly side to side. Each sample lasted 16 seconds. All coherent head samples were fresh. The sequential baseline used session 50; both concurrent-input samples used process 14542 and session 51 without a restart.
|
||||
|
||||
| Live configuration | Coherent reads | Reads with fresh left hand | Distinct left observations | Left arrival age, median / p95 | Sampled inference median |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Previous sequential CPU, 12.5 Hz input | 2,959 | 29.47% | 113 | 160.16 / 196.23 ms | 117.03 ms |
|
||||
| Concurrent CPU, 12.5 Hz input | 2,941 | 91.77% | 186 | 74.94 / 84.81 ms | 66.06 ms |
|
||||
| Concurrent CPU, 25 Hz input | 2,932 | 64.36% | 152 | 91.93 / 109.69 ms | 62.55 ms |
|
||||
|
||||
The freshness percentages measure sampled shared-channel availability; they are not the acceptance target's percentage of valid camera observations and cannot pass or fail that 95% gate. Arrival age measures the original observation's age when first seen by the channel monitor, not externally measured physical-motion-to-visible-controller latency. Natural hand motion, detected hand count, and temperature were not held constant. Both cameras often reported two hands during the requested left-palm test, so right-channel availability is not evidence of verified right-hand identity or aiming. Thermal status was 4, with HAL skin temperatures of 47.5–47.7 C for the baseline, 50.5 C for limited input, and 50.7–50.8 C for full input. These short, ordered samples support further testing of the candidate, not a causal attribution of the entire improvement or a general conclusion that full-rate input is worse. Full-rate input did not improve observed availability in this comparison; the original 12.5 Hz input cap was restored. The SDK cameras continue at approximately 25 Hz.
|
||||
|
||||
The user reported improved but still choppy motion, a reversed right-controller pointing direction while the left behaved as expected, and less flicker with both hands detected than with one. Right-hand orientation correction and investigation of the one-hand/two-hand difference are in progress; neither is validated by this candidate's measurements. No new camera images were saved. Session 51 later suspended and exited with status 0; a subsequent audit found session 52 READY, CPU/full/concurrent inference, the 12.5 Hz cap, verified calibration, and no temporary graphics overrides. Evidence: `build/spatial-live-hands-choppy-baseline-confirmed.json`, `build/spatial-live-hands-cadence-{half,full}.json`, `build/hand-input-comparison.json`, `build/hand-cadence-summary.json`, and `build/hand-cadence-audit.json`.
|
||||
|
||||
## Right-hand orientation and camera-clock continuity (2026-10-16)
|
||||
|
||||
The user's report of a reversed right controller exposed a handedness-independent palm basis. The right hand now reverses its lateral axis before constructing the rotation, giving mirrored palms the same normal convention while preserving the existing left mapping and wrist-to-finger direction. Both live reconstruction and saved-pair replay pass an explicit hand side. Tests cover mirrored palms, 180-degree and arbitrary rotations, camera-to-world transforms, and unchanged gesture values and timestamps. These tests verify the mathematical correction; the user subsequently confirmed that the right-controller orientation is correct on the installed build. Measured aiming accuracy across poses remains pending.
|
||||
|
||||
The one-hand/two-hand flicker difference has a plausible processing explanation: the [upstream MediaPipe video graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259) repeats palm detection until the previous tracked hand count reaches the configured maximum. DroidDeck requests two hands. Per-camera inference durations and current/previous raw hand counts are now reported separately from confidence-filtered counts so this hypothesis can be measured. No hand-count restriction, confidence reduction, or synthetic second hand was introduced.
|
||||
|
||||
A separate clock defect reset camera synchronization whenever the worker skipped more than 200 ms of source frames, even when host and source clocks advanced consistently. A processing stall could therefore add another 16 observations of warmup (roughly 1.3 seconds at 12.5 Hz). The mapping now survives an aligned long gap; offset discrepancies above 20 ms, invalid timestamps, or nonmonotonic source/receipt times still require warmup. Existing short-gap rolling-minimum/p90 jitter handling is retained. Six regressions cover 240 ms and one-second gaps, resets, offset changes in both directions, jitter boundaries, and recovery. Hand observation expiry remains 200 ms and head expiry remains 100 ms.
|
||||
|
||||
All 271 selected Android tests pass without failures, errors, or skips. SDK-enabled and SDK-free builds and packaged-runtime freshness checks pass. Native code is unchanged from the tested cadence build. Installed APK: `build/DroidDeck-hand-continuity.apk`, SHA-256 `46b99d939e5c7063840ae669128b40ba60a794e62a30c1a14f38da27362d2855`; driver remains `195261f27043b260052873a3b083c436457763caa5ebb7bcd1f52708e426bf11` and the graphics layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. The installed APK hash matches, and processes 29580 and 30045 both initialized CPU inference and loaded the verified calibration. CPU/full/concurrent inference and the 12.5 Hz input cap remain selected. No new images were captured. Evidence: `build/hand-continuity-{build,sdkfree,stop,install}.log` and `build/hand-continuity-install.json`. Subsequent live results follow; sustained smoothness acceptance remains pending.
|
||||
|
||||
SteamVR reopened as session 54 and reached READY with valid head tracking. A live audit verified the built, packaged, and deployed driver/layer hashes, the version-2 spatial channel, saved calibration, CPU/full/concurrent inference, the 12.5 Hz cap, and the new per-camera diagnostic fields. This is an installation/startup check, not the pending deliberate one-hand/two-hand comparison or a visual orientation pass. Evidence: `build/hand-continuity-launch.log`, `build/hand-continuity-audit.log`, and `build/hand-continuity-audit.json`.
|
||||
|
||||
## Live hand-count comparison and remaining startup delay (2026-10-16)
|
||||
|
||||
The subsequent 44-second live comparison retained Android process 30045 and SteamVR session `2026-10-16-54-steam`. All 8,146 coherent head reads were fresh; the left hand was fresh in 97.30% of channel reads. The right hand was introduced after the left, and both cameras reported two hands from approximately 13 seconds. Consequently, the nominal first-20-second/remaining-test stages are not clean one-hand/two-hand windows. The whole-run right-hand availability of 71.00% includes the period when it was absent and must not be interpreted as right-hand detection accuracy.
|
||||
|
||||
Per-camera timings were grouped by each detector's previous raw hand count, which is the count relevant to the upstream video graph's palm-search decision. The smaller count group contained eight diagnostic snapshots per camera, and the two-hand group contained nineteen.
|
||||
|
||||
| Camera | Previous raw hands | Diagnostic samples | Median inference | p95 inference |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Left camera | 1 | 8 | 73.36 ms | 80.27 ms |
|
||||
| Left camera | 2 | 19 | 55.96 ms | 69.10 ms |
|
||||
| Right camera | 1 | 8 | 71.94 ms | 87.04 ms |
|
||||
| Right camera | 2 | 19 | 53.15 ms | 65.69 ms |
|
||||
|
||||
The observed per-camera medians were approximately 17–19 ms lower with two previously tracked hands. This is consistent with the extra palm-search work described above, but the ordered, natural-motion test does not isolate that work from pose, visibility, or workload changes. Thermal status remained 4, with HAL skin temperature 50.7 C at both endpoints. These internal inference timings do not measure physical-motion-to-visible-controller latency. The 97.30% sampled left-channel availability is also distinct from the required 95% valid camera-observation rate; neither acceptance gate is established by this run. No new camera images were saved. Evidence: `build/hand-count-live.json`, `build/hand-count-live.log`, and `build/hand-count-summary.json`.
|
||||
|
||||
The user confirmed the right-controller orientation correction and described single-hand tracking as smoother and more reliable, but said it takes a while after SteamVR starts to become smooth. Startup choppiness remains unresolved. This measurement did not isolate startup behavior, and it does not establish a warmup, thermal, synchronization, or graphics cause for that remaining delay. Longer controlled runs, an observation-based coverage measure, external latency measurement, and deliberate optical interaction checks are still required.
|
||||
|
||||
The user clarified that startup choppiness lasts about a minute or longer after the dashboard appears. Both CPU detectors, camera synchronization, and verified calibration were already active in process 30045 before SteamVR launched, so initial model loading alone does not explain the delay. Session-54 compositor logs, aligned to their Denver timestamps, record 138 shader-pipeline creations during seconds 20–69 after the launch request and another 12 during seconds 120–129. This identifies ongoing graphics initialization as a plausible source of CPU contention, not a measured causal explanation of all startup stutter. The current session remained READY; its later presenter counters were about 600 frames per ten seconds. No matched `vkerror`, device-loss, fatal, or segmentation messages appeared in the filtered session window, although a `WaitForAcquire` timeout remains. No startup scheduling, cache, graphics, or loading-screen changes were made. A correlated startup trace of hand latency and CPU/GPU workload is still needed. Evidence: `build/hand-startup-review.json` and `build/hand-continuity-install.json`.
|
||||
|
||||
## Gesture, single-hand scheduling, and motion update (2026-10-16)
|
||||
|
||||
The next candidate keeps CPU/full-model/concurrent stereo inference. Calibrated stereo palm reconstruction is unchanged; pinch and finger closure now use agreement between each camera's image landmarks. Invalid or conflicting finger geometry clears the affected gesture, and the driver still requires an observed release before rearming it. This avoids making a valid palm depend on successful triangulation of every fingertip, without claiming that a lost finger is released.
|
||||
|
||||
CPU inference now retains separate one-hand and two-hand video trackers for each camera. Three consistent stereo observations enable the one-hand path; a two-hand search is due after 240 ms, and loss or ambiguous handedness returns to search. The selected bank's raw counts and hand limit are reported. Both per-camera tasks finish before images or detectors are reused or destroyed. GPU remains optional and serial. The adaptive copy gate starts at 500 permille of source callbacks (12.5 Hz with 25 Hz cameras), with 600/800 permille available only after eight consecutive healthy samples meet processing and age thresholds. Slow discovery samples can prevent an upshift; the presence of this controller is not evidence of achieving 15 or 20 Hz. Camera capture and vendor head tracking retain their normal rate.
|
||||
|
||||
The driver removes the additional observation-level low-pass and retains one 15 ms rendering correction. Stable translation advances from the time a result reaches the driver, bounded by 80 ms and 2 cm, instead of exhausting prediction during inference. Capture timestamps still control velocity and expiry. Explicit loss freezes the displayed pose, immediately releases invalid inputs, and preserves the 200 ms hand expiry and 100 ms head expiry. This is bounded motion estimation, not additional measured hand observations.
|
||||
|
||||
Validation: all 306 selected Android tracking/UI tests pass, with no failures, errors, or skips. The ARM64 VR build and native regressions pass, including delayed/jittered observations at 4 ms driver ticks, stops, reversals, outliers, quaternion normalization, exact loss freeze, expiry, and origin/connection changes. Fractional camera-gate regressions pass with strict compiler warnings. SDK-enabled and SDK-free Android builds, 13 runtime-preparation tests, and runtime checksums pass. The loading/first-frame flow and graphics color path were not changed in this update.
|
||||
|
||||
Installed on the existing Fold7 as `build/DroidDeck-hand-smoothness.apk`, final SHA-256 `9508370c42870e60a0070abf61d99792df581436f802049b068b96eba95fa027`; driver SHA-256 `52a96238afedf68c38866588db8225d2bd82450ed13635a5673ca7f9e0c6835b`. Android requested USB access after the first candidate installation; the user accepted it. Final verification matched the installed APK hash and confirmed CPU selection, automatic rate selection, scheduled single-hand tracking, and the existing verified calibration in two separate app processes (6139 and 6989). CPU inference initialized in 6139; process 6989 was initially waiting for valid head tracking, so its configured CPU delegate was not yet initialized. The saved 25-view calibration and distance check were retained. Evidence: `build/hand-smoothness-{native,stage,final-build,sdkfree,runtime-tests,install}.log` and `build/hand-smoothness-install.json`. Deliberate moving-hand, grab/release, external latency, and sustained game-performance checks remain required.
|
||||
|
||||
Saved-pair replay of the previously authorized capture `64390821246320` compared parallel full-model search, scheduled full-model tracking, scheduled lite-model tracking, then a repeated full-model search baseline. Each row measured 20 frames after four warmup frames. The scheduled rows used the one-hand path for 13 of those frames. All rows retained a valid left palm and valid released pinch/grip in all 20 measured frames; the source image is a stationary open palm, so this does not validate moving hands or pressing gestures.
|
||||
|
||||
| Workload | Full search median / p95 | Scheduled full median / p95 | Repeated full search median / p95 |
|
||||
|---|---:|---:|---:|
|
||||
| App idle | 70.93 / 76.19 ms | 36.74 / 74.04 ms | 71.28 / 75.77 ms |
|
||||
| SteamVR | 91.48 / 99.26 ms | 46.92 / 94.71 ms | 86.39 / 100.62 ms |
|
||||
|
||||
Scheduled lite-model medians/p95 were 25.41/67.04 ms idle and 40.26/86.77 ms under SteamVR. The full model remains selected because a repeated static image cannot establish the lighter model's live tracking or gesture accuracy. The SteamVR comparison retained process 333 and session `2026-10-16-57-steam`; thermal status was 4 and HAL skin temperature rose from 49.5 to 51.0 C. These short ordered runs show lower typical inference cost on this saved pair, while discovery-frame tail latency remains. They do not establish external motion-to-controller latency, sustained game frame time, or reduced thermal throttling. No new camera images were captured. Evidence: `build/hand-reliability-benchmark-smoothness-{idle,steamvr}.json`.
|
||||
|
||||
The final adaptive policy uses average processing cost for normal budget changes so occasional discovery frames do not veto otherwise sustained headroom. Any single processing cost at least 80 ms, observation age at least 140 ms, or invalid tracking falls back to 12.5 Hz. Regression tests verify that a repeating 36/36/74 ms pattern can select 15 Hz but not 20 Hz, while continuous 70 ms work stays at 12.5 Hz. This change does not assert that the observed SteamVR workload qualifies for a higher rate.
|
||||
|
||||
Final deployment audit: SteamVR session `2026-10-16-58-steam` reached READY with a first frame and valid head tracking. Built, packaged, and deployed driver hashes match; the unchanged graphics-layer hash is `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. The version-2 spatial channel and verified calibration were present, with no temporary environment overrides. CPU/full/concurrent inference, scheduled discovery, and adaptive input were active. The audit snapshot showed one accepted palm with valid pinch/grip geometry, the one-hand inference bank, 47.10 ms stereo inference, approximately 25 Hz source cameras, and a 500-permille input limit. A single snapshot does not demonstrate reliable movement or actual pinch/grip presses. Evidence: `build/hand-smoothness-{launch,audit}.log` and `build/hand-smoothness-audit.json`.
|
||||
|
||||
## Stationary jitter and persistent input release (2026-10-16)
|
||||
|
||||
The user reported stationary controller bounce and a dashboard drag that remained attached after a controller flickered and the hand returned open. The original 16-second reproduction recorded 2,905 coherent channel reads with fresh head tracking throughout. The right hand was fresh in 812 reads; 64 distinct poses were recorded. Right-hand transitions included five samples above the trigger-click threshold, 63 valid released-grip observations, and no pressed grip. Both hands were invalid at the end of the sample. This does not by itself locate the missing dashboard release: the original driver lacked per-input publication diagnostics. Evidence: `build/spatial-live-hands-sticky-grab-baseline.json`.
|
||||
|
||||
Inspection of the installed Vive compositor binding showed that dashboard left-click/drag uses the trigger, with grip mapped to Back. The installed dashboard `GrabHandle` attaches the panel to the initiating hand and clears that attachment on a window mouse-up event; it has no tracking-loss cleanup in that component. A missing mouse-up is consistent with the report, but the exposed code does not prove that native laser-mouse processing discards releases when a pose becomes invalid. Resource evidence: `resources/config/vrcompositor_bindings_vive_controller.json`, SHA-256 `16f19a170608c9b3450e7fc980644220a5bd3a34e1bb88050e65f51c7600a2d1`, and `resources/webinterface/dashboard/chunk~8012d0c89.js`, SHA-256 `8a529bd96141480a18efa44b912a81a1e27ac1bf8bae6b2ab8945ebaf6da2e73`. No SteamVR resource was modified.
|
||||
|
||||
The driver now stabilizes translation and rotation independently against fixed stationary anchors, holding within 3 mm and 1 degree. Breakaway uses the existing 15 ms rendering correction; it does not subtract a dead-zone distance from every observation. Returning to a hold requires 240 ms inside tighter 1.5 mm / 0.5-degree bounds and a check against sustained directional drift. Position prediction stops while anchored. Native tests cover still jitter, quaternion sign flips, accumulated slow translation/rotation, fast breakaway, settling, and the existing delayed-motion and loss behavior. These thresholds are candidates; stationary jitter and fine aiming still need hardware confirmation.
|
||||
|
||||
A confirmed latest-snapshot race could hide a transient invalid hand/head observation from the driver. Protocol version 3 retains the 176-byte layout and uses the offset-36 word for four independent 8-bit loss counters: left trigger/grip, then right trigger/grip. Loss remains visible in subsequent snapshots even if the invalid snapshot was overwritten. Head gaps clear cached hands, captures preceding head loss cannot rearm inputs, and expiry/invalid-gesture transitions update only the affected counters. The driver clears matching input state on counter changes, rejects old or duplicate observations for rearming, and detects observation gaps beyond 200 ms. It publishes released trigger scalar/click and grip before pose loss, clears inputs before deactivation, and logs state/error transitions for hardware verification. No synthetic presses or fresh poses were introduced. Versions 1 and 2 are rejected, so the Android and driver binaries must be updated together.
|
||||
|
||||
All 317 selected Android tests and the native regressions pass. New tests deliberately skip invalid snapshots, preserve independent trigger/grip state, reject stale release observations, verify publication ordering, and retry rejected OpenVR component writes. SDK-enabled and SDK-free Android builds and runtime checksum verification pass. Installed APK: `build/DroidDeck-hand-stability.apk`, SHA-256 `0e3f4a140e36eb48e91f5937748b0e10afe3491a0236484b21e9a3faa3ca6766`; driver SHA-256 `cca5723189a77cde606d11afd55d80567e7f72398e3f50e706bee164cb91eb5f`. Calibration survived two app processes (17259 and 17867); CPU initialized in the latter. SteamVR session `2026-10-16-60-steam` reached READY with a first frame. Evidence: `build/hand-stability-{native,stage,build,sdkfree,install,launch,audit}.log` and `build/hand-stability-{install,audit}.json`. End-to-end dashboard release and visible stationary improvement remain pending the deliberate post-installation check.
|
||||
|
||||
## Outstanding acceptance
|
||||
|
||||
### Investigation and shared base calibration (2026-10-16)
|
||||
|
||||
The phone's current verified 640×480 pinhole profile was exported directly over authorized ADB. Its 1.1209395390660726-px fit, lens coefficients, stereo transform, and camera-to-head transform are now bundled unchanged as `app/src/main/assets/calibration/luma-ultra-640x480.json`, with the source device identifier removed. Asset SHA-256: `d160f0c81d0328255f9afa69bc61505e5aa2988a9bb7a9faaeb611598eee7faa`. A matching personal profile takes precedence; new or unidentified Luma Ultras use this base for experimental tracking. Shared geometry is not falsely marked validated on other devices. UI and diagnostic status identify the shared base separately from individually verified profiles.
|
||||
|
||||
Hand tracking now displays **Experimental** throughout the relevant controls and setup status. Per-observation diagnostics distinguish detection/association failures, missing palm landmarks, palm width/basis/motion rejection, stale data, and tracking loss. Connection-lifetime counters retain independent palm, trigger, and grip acceptance/loss evidence without counting repeated status polls. Tracking and input thresholds are unchanged. The historical trace analysis and prioritized improvement experiments are recorded in `luma-hand-tracking-investigation.md`; no new moving-hand or game-usability pass is claimed.
|
||||
|
||||
All 388 focused glasses and setup/UI tests pass, including the new calibration selection and diagnostic-counter regressions. Exact geometry comparison against the phone export passes. The SDK-enabled APK builds with runtime freshness verification enabled. After the concurrent upstream Joy-Con merge, the already rebuilt VR outputs were validated against current source/header/configuration hashes and restaged with the updated helper scripts; unrelated prepared binaries retained their hashes. All 150 packaged runtime outputs match the resulting manifest. The packaged base calibration matches its source exactly, and the APK includes the new diagnostics. Artifact: `build/DroidDeckXR-luma-experimental.apk`, SHA-256 `11235040ad8c34abfd23482beb552fb09db4d9429703fbc8d7c5090b98da940e`. Runtime provenance is in `build/luma-hand-runtime-restage-20261016/provenance.json`. This change has not been installed on the phone. Logs: `build/luma-investigation-tests.log`, `build/luma-investigation-build.log`; metadata: `build/luma-investigation-validation.json`. The source phone profile backup stays in ignored `build/luma-ultra-phone-calibration-20261016.json`.
|
||||
|
||||
| Gate | Current evidence or blocker |
|
||||
|---|---|
|
||||
| Measured head axes and meter scale | Pending measured translation, leaning and crouching |
|
||||
| Floor and separate origins on hardware | Automated math passes; physical crouched recenter and picture switching remain pending |
|
||||
| Screen anchoring and plane crossing | Shader and parameter tests pass; visual movement and plane-crossing checks remain pending |
|
||||
| Camera image orientation | Upright 640 by 480 images and visible palms verified in one authorized diagnostic pair |
|
||||
| Calibration quality | 25-view set passes fit/alignment checks with 1.12094 px held-out reprojection error. Repeated distance check passed: 12.13 inches measured against 12.00 selected; SDK-identified verified profile survives app restarts |
|
||||
| Hand pose and aiming | User confirms corrected right-controller orientation and smoother, more reliable single-hand tracking. Latest 44-second comparison had 97.30% sampled left-channel availability; startup choppiness, measured aiming accuracy, and sustained continuity remain pending |
|
||||
| At least 95% valid hands at 30–80 cm | Camera-observation acceptance rate not measured; sampled channel availability above is a different metric |
|
||||
| At most 5 cm hand error and 2 cm jitter | Not measured |
|
||||
| External p95 controller latency below 100 ms | Not measured |
|
||||
| Hardware input release and reacquisition | State-machine tests pass; user-requested 200 ms hand expiry / 500 ms reacquisition not measured physically |
|
||||
| Twenty-minute game performance | No controlled baseline or CPU/GPU comparison |
|
||||
| Dashboard and Angry Birds VR interactions | Not validated with optical controllers |
|
||||
| Occlusion, crossed hands, lighting and thermal load | Pending |
|
||||
| Physical USB reconnect during inference | Generation/lifecycle guards implemented; no calibrated-inference unplug test |
|
||||
|
||||
The user confirmed corrected colors, optical controllers appearing without a freeze, corrected right-controller orientation, and smoother, more reliable single-hand tracking. The latest timing comparison is consistent with additional palm-search work when only one hand is tracked. Startup choppiness remains unresolved, with no established cause. A confirmed open/pinch/release sequence is still needed to distinguish missing/expired observations from gesture recognition or input-path failures. Full palm detection with the lite landmark model remains an experiment; the current full model stays selected. Controlled temperature-matched SteamVR measurements, right-hand correspondence across poses, and measured aiming validation remain necessary. The compositor guard passed the reproduced gamepad-controller visibility trigger; long-duration and GPU-inference stability remain unvalidated. Delegate and process identity must be checked across each measurement because graphics recovery recreates the hand engine; it now selects CPU by default and discards any temporary GPU override. The saved calibration no longer needs repeating after an identified-device reconnect. The combined feature must remain experimental until hand tracking and the remaining acceptance measurements pass. Native 6DoF availability alone is insufficient for release.
|
||||
|
||||
## Fork migration and latest feedback (2026-10-16)
|
||||
|
||||
The post-installation check did not establish a fix: the user could not start the dashboard drag and reported stationary controller jitter about the same. Dashboard interaction, release after tracking loss, and visible stationary stability remain unresolved.
|
||||
|
||||
At the user's request, the 63 uncommitted XR files were moved from `C:/Users/conno/Desktop/DroidDeck` to `C:/Users/conno/Desktop/DroidDeckXR` on branch `feat/3dfixes-6dof`. A three-way merge preserved the fork's resolution and menu changes; 13 non-overlapping fork files remained unchanged. Original files and patches from both checkouts are preserved in `build/xr-migration-20261016`. The source checkout is clean after hash-verified removal of the migrated edits.
|
||||
|
||||
The merged fork passes 373 selected Android tests and 13 packaging tests, plus SDK-enabled and SDK-free builds. The existing tested ARM64 VR outputs were reused after validating their source, SDK-header, configuration and binary checksums; they were not rebuilt during migration. All 133 prepared runtime outputs match the SDK-enabled APK. The new APK is `build/DroidDeckXR-migrated-sdk.apk`; it has not been installed. See `build/xr-migration-verification.json` for hashes and the retained historical runtime-provenance boundary. Subsequent work belongs in DroidDeckXR.
|
||||
|
||||
## Connected-device isolation review (2026-10-16)
|
||||
|
||||
Reviewed the outstanding DroidDeckXR changes together, including optical and Joy-Con tracking, calibration, display-mode routing, recovery overlays, graphics fixes, RGB diagnostics, and build/runtime provenance. Ultra controls now require a controllable, connected device whose Carina capability was confirmed by the VITURE SDK. Model names and saved Ultra preferences alone do not enable them. The experimental position/hand/Joy-Con controls, floor/eye-height setup, 6DoF label, calibration activity, recovery overlay, and RGB diagnostic entry follow this capability. Calibration closes when the connection disappears, and launching it directly without an Ultra is rejected.
|
||||
|
||||
Saved Ultra controller preferences no longer suppress the phone-camera permission or controller path for other glasses. Routing is refreshed on device changes as well as setting changes. Opening a non-Carina link clears the retained spatial-channel mode; temporary Ultra tracking loss still invalidates poses without silently switching to fixed-height legacy tracking. Tests cover disconnect, reconnect, unconfirmed model names, non-Ultra VITURE, XREAL and RayNeo states, rejected mode changes, and the native-channel flags.
|
||||
|
||||
The optional UVC directory now accepts an explicitly empty Gradle override, requires its license files before packaging, and is included in the local native-source preparation audit. All 456 selected Android tests pass with no failures, errors or skips. An earlier Compose calibration timeout passed in isolation and in the final complete selection without changing the test or product behavior. The ARM64 VR rebuild and native regressions pass. Linux packaging/helper tooling reports 47 passing tests; two shader compilation tests were skipped because glslangValidator is unavailable (this change set does not modify shader source or its generated header).
|
||||
|
||||
Both the SDK/UVC-enabled and SDK/UVC-free Android builds pass, and all 150 prepared runtime outputs match each APK. Package inspection confirms the enabled build includes the optional libraries and UVC notices, while the disabled build contains none of those optional libraries. Artifacts: `build/DroidDeckXR-ultra-reviewed.apk` (SHA-256 `a01706d56b876c2fec7b9386d33e9dc66854ead66b48a1392c492bde7c3591d3`) and `build/DroidDeckXR-ultra-reviewed-sdkfree.apk` (`6ce45e225a37c74355eb7d65e76b2ac63be7064ec24cbe389e006452e0e27390`). Results and logs are retained under `build/ultra-review-*`, including `build/ultra-review-verification.json`.
|
||||
|
||||
This review did not install a new package or repeat hardware acceptance. The existing dashboard interaction and stationary jitter reports remain unresolved; the combined tracking feature remains experimental.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Luma Ultra spatial tracking
|
||||
|
||||
For supported glasses across brands, feature levels, and app-wide setup, see the
|
||||
[smart-glasses support matrix](../smart-glasses-support.md).
|
||||
|
||||
This is an experimental implementation of native Luma Ultra head tracking, positional virtual screens, and optical SteamVR controllers. Positional tracking, bare-hand controllers, and Luma Joy-Con tracking all default to off and are grouped under Experimental. Combined release acceptance remains pending until the calibration, accuracy, latency, gameplay, and sustained performance targets below have been measured on hardware.
|
||||
|
||||
## Build and setup
|
||||
|
||||
Luma Ultra now includes a shared starting calibration exported from the user's verified Fold7 profile on 2026-10-16. The bundled `calibration/luma-ultra-640x480.json` contains the original lens, stereo, and camera-to-head geometry without the source glasses' serial identifier. It applies only to the SDK-confirmed Luma Ultra primary `left0/right0` pair at 640 by 480. Matching saved device calibration takes precedence, including an unfinished personal fit that still requires distance validation. Missing, corrupt, or foreign profiles fall back to the model base; unknown serials can also use the base.
|
||||
|
||||
The base enables experimental hand tracking without requiring every user to repeat board collection. It is not marked individually verified on other units. Tracking and setup show **Shared Luma Ultra calibration** while it is in use; diagnostics distinguish `calibrationSource` (`luma-ultra-base`, `device`, or `none`) from `calibrationVerified`. The existing `calibrated` availability field is true for either the base or a verified personal profile. The following board steps are optional refinement for a new device using the base; a newly calculated personal profile still needs the unchanged distance check before use. A successful distance check saves the profile for the connected identified glasses. Source validation does not establish manufacturing consistency or aiming accuracy across other units.
|
||||
|
||||
Use the existing optional `droiddeck.vitureSdk` local property with the external VITURE Android SDK 2.4.0. No vendor headers or libraries are checked into the repository. `-PvitureSdkDir=` explicitly builds without the SDK. The ordinary runtime freshness checks remain enabled. Rebuild the ARM64 VR components with `tools/vr/build_windows.py`; its manifest includes `spatialpose_shm.h`.
|
||||
|
||||
The hand pipeline pins MediaPipe Tasks Vision 1.0.0, OpenCV 4.13.0, and the model and SHA-256 recorded in `app/src/main/assets/models/hand_landmarker.provenance.json`. Both Gradle and the inference worker verify the model checksum. CPU inference, the full model, concurrent camera inference, scheduled hand discovery, and automatic input-rate adjustment are the defaults, including after app restarts, reconnects, and provider restarts. **Show diagnostics → GPU hand inference** temporarily enables GPU until the next tracking restart; GPU remains experimental because SteamVR restarts and Vulkan device loss were observed during testing. Diagnostics report the selection as `handGpuEnabled` and the initialized detectors as `handInferenceBackend`; `not initialized` means detector startup has not completed. Sustained performance acceptance remains unresolved.
|
||||
|
||||
Hand input starts by copying 500 permille of stereo callbacks, approximately 12.5 pairs per second at the SDK's documented 25 Hz camera rate. On CPU, automatic adjustment can raise this to 600 or 800 permille (about 15 or 20 Hz) only after eight consecutive valid observations have sufficient processing and freshness headroom. Promotion requires average processing at most 55 ms and observation age at most 110 ms for 15 Hz, then at most 40 ms and 90 ms for 20 Hz. A moving average allows occasional scheduled discovery frames while preserving a sustained processing budget. An average above the selected period reduces the rate; any single processing cost of 80 ms or observation age of 140 ms immediately returns it to 500 permille. Tracking failure and detector resets also return it to 500. This policy requests more input only when processing can accommodate it; it does not guarantee that many completed or valid hand updates.
|
||||
|
||||
Skipping happens before copying camera pixels into DroidDeck, so skipped pairs also avoid JNI transfer, preprocessing, and inference. The SDK exposes no tracking-camera rate setter: sensor capture, vendor VIO, pose callbacks, and head polling retain their normal rates. Calibration collection, distance checks, explicit diagnostic capture, and preview-only operation use full-rate input. Diagnostics keep incoming `cameraHz` distinct from `cameraCopyPermille`, `rateLimitedFrames` (intentional skips), and `dropped` (eligible pairs replaced while the worker is busy). The compatibility field `cameraFrameDivisor` is 2 at 500 permille, 1 at 1000, and 0 for intermediate fractions. The bounded latest-pair queue and original capture timestamps remain intact. **Automatic hand input rate** can be disabled to expose the manual half/full-rate switch; both choices reset with tracking. GPU uses that manual preference and does not adapt. Thermal, latency, and gameplay benefits require hardware comparison.
|
||||
|
||||
1. Connect a Luma Ultra and grant its existing USB permission. Spatial controls appear only after the SDK identifies the device as Carina.
|
||||
2. Enable **Positional tracking — experimental**. This restarts the provider in 6DoF mode. Switching picture modes does not restart it.
|
||||
3. Enter the current seated or standing eye height (shown in centimeters and feet/inches) and choose **Set floor from current height**. Normal VR recentering then preserves that floor, including while crouched. After reconnecting or explicitly restarting tracking, accept the new origin through setup/recenter. A recenter request during lost/stale tracking or recovery remains pending until 250 ms of fresh, steady poses arrive; repeated taps are unnecessary. The pending request is canceled on disconnect.
|
||||
4. Open **Stereo camera setup and diagnostics → Open setup**. The page uses the selected DroidDeck theme and app scale, with a single compact Back/title row so the controls have room to scroll in landscape. Check head tracking and camera synchronization first; capture controls become available when both are ready. Expand **Show diagnostics** for frame rate, frame age, dropped frames, processing time, and recovery tools.
|
||||
5. Print the 5 by 7 ChArUco board at actual size. Measure a square: it must be 30 mm (approximately 1.181 inches). Keep the board stationary and capture at least 24 sharp views by moving and rotating the glasses, covering the image center and edges. Include both yaw and pitch, roll, and varied distances. If the underlying tracking frame changes, start a new capture set; software recentering in an unchanged frame preserves the set.
|
||||
6. Calculate calibration. Accepted captures confirm board detection, not alignment quality. The fitter compares pinhole and fisheye models on held-out views. It rejects poor stereo reprojection, an implausible baseline, insufficient rotational diversity, and inconsistent stationary-board alignment. Turning and tilting the glasses is required; pause before each capture. If the board stayed fixed but alignment varies beyond 5 cm (about 2 inches), the head-tracking frame may have shifted during collection. Wait for steady tracking and start a new set; do not remove the quality gate to accept the old one.
|
||||
7. Start with **12 in (30.48 cm)** or **30 cm (11.81 in)** under **Measured distance**; **20 in** and **50 cm** remain available. The closer settings make the small printed markers easier to resolve in the 640 by 480 tracking cameras. Show the complete board with its center exactly the selected distance from the left camera, then choose **Validate distance**. You can hold or support the glasses without wearing them. Progress and the result appear beside this button. The count refers to detected checkerboard intersections, not the paper's edges; a fully visible but small or blurred board can still have too few detections. Validation detects corners in the original images, then corrects those coordinates for lens distortion before triangulating, avoiding marker blur from resampling the images. Rejected stereo geometry reports that depth could not be measured; a valid reconstruction shows measured and selected distances in both units. Each imperial or metric choice uses its exact distance. The complete 24-corner board and physical-distance check within 5 cm (about 2 inches) are still required before a newly fitted personal profile becomes usable. Profiles are keyed to the SDK board-serial SHA-256 identifier (falling back to a nonblank Android USB serial), dimensions, and format version. Verified profiles reload automatically after an app restart or reconnect to the same identified glasses. If neither identifier is available, profiles remain connection-local.
|
||||
8. Enable **Hand controllers — experimental**. Present an open hand first so inputs can arm. Aim the virtual controller at the target, pinch thumb and index finger together to press the trigger/click, then separate them to release. Moving the SteamVR dashboard by its handle uses this same pinch-and-release action. Keep the other fingers open when only clicking; curling the middle, ring, and little fingers activates the separate grip input, which the installed Vive dashboard binding maps to Back. Games can use grip for holding objects. After tracking loss, open the hand again before another press. Gamepad sticks supply locomotion and turning; buttons retain the Vive-compatible mapping. External VR controllers retain precedence.
|
||||
|
||||
The diagnostic capture button saves one stereo pair and metadata to the app-private `files/spatial/captures` directory. Merely opening diagnostics does not save images. Calibration profiles stay in `files/spatial`. A brief missed hand detection holds the last observed controller transform only until its original observation reaches 350 ms old. Input freshness remains limited to 200 ms. Pinch and grip release immediately on a detected miss and require an observed release before rearming. The hold does not extrapolate motion, refresh timestamps, or substitute a gamepad pose. Head tracking loss, origin/connection changes, disconnects, and controller-mode changes clear it. Gamepad menus and locomotion remain available during optical hand loss. Switch hand controllers off explicitly to return to gamepad-positioned controllers.
|
||||
|
||||
Accepted board landmarks and their capture-time head poses are saved separately in app-private `files/spatial/*-captures.json`, without camera images. A matching SDK or USB device identity and image geometry allow automatic recovery for recalculation. Restored observations cannot be appended to a new tracking frame; start a new set to capture again. Devices without either stable identifier keep a session-labelled diagnostic backup and require explicit developer recovery onto the same physical device. They do not automatically reuse another connection's unidentified capture set or personal calibration; they can still use the bundled model base.
|
||||
|
||||
The vendor SDK working directory is `noBackupFilesDir/viture`. Android can evict `cacheDir` during a live session, so active tracking files must not depend on that location. The SDK initializes its own files in the new directory on the next provider start; existing printed-board calibration profiles remain in `files/spatial`.
|
||||
|
||||
## Coordinates and lifecycle
|
||||
|
||||
`SpatialPose` uses meters and right/up/back OpenGL coordinates. Quaternions are normalized xyzw internally; SDK arrays are position followed by wxyz. Legacy orientation conversion and the 128-byte head-pose channel remain intact. The session coordinator survives `HeadTracker` recreation and owns separate full-pose screen and yaw/floor VR origins. The camera worker uses a generation guard, a bounded latest-frame queue, and no SDK lifecycle operations in callbacks.
|
||||
|
||||
Head polling is approximately every 4 ms with prediction disabled; this is not a claim of a 250 Hz sensor. Camera-rate callbacks provide a liveness check. Stale or unstable poses retain the last rendered transform and produce an invalid SteamVR pose. While a positional session is active, DroidDeck covers the resulting grey view with a themed recovery screen on the phone and glasses, including one readable copy per eye in SBS. It distinguishes tracking loss, USB disconnection, required recentering, and stabilization after recentering. Tracking-loss instructions ask the user to look around slowly in a well-lit room with visible detail, keep the cameras clear, and open the setup diagnostics to restart tracking if recovery stalls. Phone controls remain accessible, and the screen closes automatically when tracking is valid. Freshness is checked even if pose callbacks stop. Repeated USB timeouts require connection recovery rather than a synthetic valid pose. Large pose discontinuities latch a setup requirement. Provider shutdown invalidates queued work before closing the vendor provider and Java USB connection.
|
||||
|
||||
The 176-byte `droiddeck-spatialpose` snapshot uses protocol version 4 and sequence-counter publication. The driver validates file length, version, finite normalized poses, and monotonic age. Head and both hands use one snapshot and one VR origin. Hand pose flags independently identify valid palm tracking (bit 0), pinch (bit 1), and grip (bit 2); head flags use only bit 0. The formerly reserved 32-bit word at byte offset 36 is now `input_epochs`, containing independent 8-bit loss counters for left trigger, left grip, right trigger, and right grip, from least to most significant byte. The layout size and pose offsets are unchanged, but versions 1 through 3 are rejected; app and driver must be rebuilt together.
|
||||
|
||||
Snapshot flags select positional tracking (bit 0), bare-hand controllers (bit 1), or Luma Joy-Con
|
||||
controllers (bit 2). The latter two selections are mutually exclusive; either enables stereo inference.
|
||||
The Joy-Con mode consumes only wrist landmark 0 and the original capture timestamp; rotation and inputs
|
||||
come from the paired Joy-Con. `luma_joycon.h` selects fresh stereo positions or a 6DoF-head-anchored arm
|
||||
model independently per hand, rejects observations over 1.5 m from the head, and blends source switches
|
||||
with a 180 ms time constant. It never integrates accelerometer position or uses the phone-camera
|
||||
correction in this mode. Stale head poses invalidate output and release inputs. Selected mode flags
|
||||
survive disconnect so a lost Luma cannot silently switch to a fixed-origin controller. Origin/connection
|
||||
changes reset both positional blending and controller yaw alignment. This is hand-assisted tracking,
|
||||
not a detector for the controller shell; see [Joy-Con setup and limits](../cardboard-vr.md#joy-con-controllers).
|
||||
|
||||
Each counter advances when its gesture becomes ineligible, including invalid finger data, hand/head loss, and observation expiry. It survives later valid snapshots, allowing the driver to notice a short loss even if it never read the intervening invalid snapshot. Counters wrap independently within their bytes. Connection/origin changes reset driver input state, and a gap beyond 200 ms from its last accepted observation also releases inputs. Every hand retains its original camera observation timestamp: experimental hand poses and inputs expire after 200 ms, while head poses expire after 100 ms. A changed loss counter disarms only its affected gesture. Duplicate or older observations cannot rearm an input or replay a press; a new, valid released observation is required after loss before another press. Missing finger data never counts as a release gesture. Head updates cannot extend a hand observation's lifetime. Diagnostics expose the hand limit as `handMaxAgeMs`.
|
||||
|
||||
The driver submits trigger scalar, trigger click, and grip updates before submitting the controller pose, including when the pose is becoming invalid. Disconnect and deactivation also clear inputs first. Changes in input state, loss counters, or API error codes produce concise driver log records with raw gestures, output values, armed state, pose/input validity, observation age, connection, and origin. Successful input API calls confirm submission to SteamVR; they do not prove that the dashboard received its mouse-up event. The installed dashboard grab handler retains a hand attachment until mouse-up and has no tracking-loss cleanup in that JavaScript component. Recovery of the reported stuck dashboard drag therefore still requires hardware validation; the implementation does not manufacture another press or a valid pose to force release.
|
||||
|
||||
Controller display smoothing has no stationary anchors or dead zones. Small intentional motion remains visible; smoothing and prediction do not change the measured pose used for gesture validation.
|
||||
|
||||
The displayed controller converges toward each measured palm pose on every driver update with one 15 ms correction time constant and normalized, hemisphere-corrected quaternions. There is no additional per-observation smoothing stage. After three consecutive observations with consistent motion, translation advances from the result's receipt time for at most 80 ms and 2 cm; capture timestamps still determine velocity and freshness. Receipt-relative prediction can continue between delayed inference results instead of consuming its entire horizon before a result arrives. Prediction is disabled above 1.5 m/s or 6 m/s², on a missed/invalid observation, head loss, or changed origin/connection. A missed observation freezes the last displayed transform through duplicate old samples until a newer observation arrives or the original 350 ms visual-hold expiry is reached. Fast motion reaches the 2 cm bound sooner, and correction adds a small settling delay. Stabilization and prediction do not alter gesture values or observation timestamps and cannot replace measured latency validation.
|
||||
|
||||
The screen shader puts translation in `extra.yzw`, rotates the eye offset, and rejects intersections behind the eye. SBS and flat branches ignore translation. The push-constant allocation stays unchanged. A screen reference can include pitch and roll, preserving placement while reclining.
|
||||
|
||||
## Stereo reconstruction and limitations
|
||||
|
||||
Skipped camera pairs do not require clock warmup again when the source/host offset remains consistent. A source gap above 200 ms checks that offset against the existing 20 ms bound instead of unconditionally resetting synchronization. Nonmonotonic or invalid timestamps and inconsistent long gaps require 16 fresh observations; normal short-gap jitter retains the existing rolling-minimum and p90 policy. This preserves synchronization through a processing stall without publishing expired hand observations.
|
||||
|
||||
Palm orientation uses an explicit hand side. The lateral axis reverses for the right hand so mirrored palms have the same normal and finger-direction convention; the existing left-hand mapping is retained. This corrects the mathematical right-hand inversion, while physical controller aiming still needs confirmation.
|
||||
|
||||
The stock two-hand video graph can repeat whole-image palm detection when fewer than two hands were tracked previously. With both hands tracked it can skip that search; see the [MediaPipe graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259). CPU scheduling maintains two persistent detector banks, each containing one detector per camera: one bank permits one hand, and the other permits two. After three observations agree on exactly one confident hand in both cameras, the one-hand bank handles intervening frames, and a two-hand search becomes due after 240 ms of camera time, running on the next processed frame. Both hands, a miss, handedness disagreement, nonmonotonic time, or a gap above 400 ms return selection to the two-hand bank. Second-hand discovery therefore remains enabled but can wait for the next scheduled search and processing. Banks retain their video tracking state between selections; they are not rebuilt per frame. GPU retains the ordinary two-hand bank. Diagnostics expose the active bank as `inferenceHandLimit`, alongside raw and filtered hand counts and per-camera timing. These choices reduce unnecessary searches by design; their benefit and extra model-memory cost require device measurement.
|
||||
|
||||
The primary `left0/right0` images are copied only for diagnostics, calibration, or enabled hands. At most one pair is pending while the worker processes another. Automatic exposure is retained because the cameras also serve native VIO. SDK 2.4.0 documents the tracking-camera rate as 25 Hz and provides no stereo frame-rate setter. Polling more frequently does not create additional camera frames. The camera-rate readout accounts for skipped source-frame sequence numbers instead of treating processed-pair throughput as sensor rate. The selected video-mode Hand Landmarker pair receives matching timestamps. CPU inference runs the cameras concurrently with separate image buffers; each bank's right detector is created, used, and closed on one dedicated worker. Both calls must finish before pixels or detectors can be reused, including after failure or shutdown interruption. GPU inference remains serial on its owning thread. The pipeline detects landmarks in the original grayscale images replicated into RGB channels, without image normalization or fisheye remapping, corrects landmark coordinates for stereo reconstruction, matches unambiguous same-handed candidates, rejects excessive reprojection or implausible depth, and derives the palm pose from triangulated landmarks. Gesture recognition uses agreement between the two image-space finger estimates. MediaPipe world landmarks are not treated as headset-relative positions.
|
||||
|
||||
The public header does not document the coordinate basis of the camera-rate pose callback. Its pose values are therefore not used as canonical head transforms. A bounded history of polled OpenGL poses supplies the capture-time transform. Source-to-host clock offsets must first stabilize over at least 16 observations; callback timestamp agreement and a nearby canonical head sample are required. Hand inference starts without waiting for the matching VIO callback, then checks for that exact camera timestamp before reconstruction can publish a valid pose. Missing confirmation, changed connections/tracking frames, invalid head tracking, and hand observations older than 200 ms still discard the result. Calibration and preview-only acquisition retain their bounded pose wait. Receipt-based offset estimation cannot establish absolute camera latency. External motion-to-visible-controller measurement is still required. Increasing expiry can reduce disappearing controllers at the cost of displaying older poses; it does not resolve GPU device loss or establish the latency/performance gates.
|
||||
|
||||
Stereo validation retains the original 3-pixel reprojection, positive-depth, and 15–150 cm depth checks per landmark. Bare-hand controller position and orientation require a valid triangulated wrist and all four palm knuckles (0, 5, 9, 13, 17). Joy-Con mode reconstructs wrist landmark 0 only and takes orientation and inputs from the paired controller. Live pinch and grip no longer require trustworthy fingertip depth: each image estimates palm-normalized pinch and middle/ring/little finger curl, then both views must agree. Grip combines fingertip reach and finger-chain bending, with separate press/release thresholds and three curled fingers to press versus two to retain a press. Invalid, degenerate, or conflicting image evidence releases only its affected gesture and requires an observed release before rearming. A valid 3D palm is never sufficient by itself to invent a grab or click. Calibration and distance validation still require every board point to pass. Conservative stereo association rejects ambiguous hands rather than inventing depth. Crossed hands, edge-on palms, and occlusion may reduce availability. The 2D gesture consensus and aiming still require physical testing through grabs, pulls, and releases; they do not establish an accuracy or latency pass. No skeleton output, desktop gestures, fast-action guarantee, or hands-only locomotion is provided.
|
||||
|
||||
Diagnostics include automatic or manual half/full input rates and a Lite-model comparison. Choices reset with the tracking session; the default is full-model parallel CPU with scheduled discovery and an adaptive rate starting at 500 permille. The Lite bundle uses Google's Lite palm detector and Lite landmark model with matching Tasks metadata. Both bundles are checksum-pinned in Gradle and at runtime; `tools/build_hand_lite.py` reproduces the candidate from pinned inputs. Saved-pair CPU replay compares full-model two-hand inference, scheduled full, scheduled lite, then full-model two-hand inference again, all with concurrent cameras. Rows identify `parallel` and `scheduled`; samples identify `inferenceHandLimit`. Its timings exclude model construction and cannot establish live detection coverage, external latency, second-hand discovery delay, or game performance.
|
||||
|
||||
## Combined release gates
|
||||
|
||||
| Area | Required evidence |
|
||||
|---|---|
|
||||
| Translation | Measured right/up/back movement and approximately 1:1 meter scale; lean and crouch; no double application |
|
||||
| Origins | Crouched recenter preserves floor; picture switching preserves independent screen and VR references |
|
||||
| Screens | Anchoring while moving sideways, toward and away; stable output after crossing the screen plane |
|
||||
| Hands | At least 95% valid slow unobstructed observations at 30–80 cm; position error at most 5 cm; stationary jitter at most 2 cm |
|
||||
| Latency | External physical-motion-to-visible-controller p95 below 100 ms |
|
||||
| Recovery | Immediate release on detected loss; observation expiry extended to 200 ms by user request; clear-hand reacquisition within 500 ms |
|
||||
| Performance | At most 10% regression in p95 game frame time over 20 minutes, with baseline frame rate maintained |
|
||||
| Gameplay | Dashboard selection and Angry Birds VR aim, grip, pull, release |
|
||||
| Robustness | Crossed hands, partial occlusion, head motion, lighting, USB reconnect, thermal load |
|
||||
|
||||
Record results in `luma-spatial-evidence.md`. A build or head-tracking pass alone does not satisfy this combined gate.
|
||||
@@ -149,6 +149,11 @@ DroidDeck SteamVR session. Library launch diagnostics are in
|
||||
logs remain in `vrstartup.log` and Steam's logs directory. It does not intercept
|
||||
arbitrary external `steam://` URLs or change VR game launch arguments.
|
||||
|
||||
Only VR-only games (`vr_only`) wait for SteamVR and start it. Games where VR is
|
||||
optional, such as No Man's Sky, launch normally unless SteamVR is already
|
||||
running; before this they started (and if needed downloaded) SteamVR on every
|
||||
Play, so they opened in VR without the user asking for it.
|
||||
|
||||
Validation: seven bridge/dispatch tests passed under Node/Python, five Android
|
||||
SteamVR launch tests passed, shell syntax passed and the debug APK built with
|
||||
runtime input verification. The broader Linux compatibility suite cannot run
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# VITURE Android SDK integration
|
||||
|
||||
For the cross-brand feature matrix and user setup, see [smart-glasses support](../smart-glasses-support.md).
|
||||
|
||||
The optional adapter is built against the user-supplied VITURE XR Glasses SDK
|
||||
2.4.0. Set `droiddeck.vitureSdk=C:/path/to/VITURE_XR_Glasses_SDK_for_Android`
|
||||
in the ignored `local.properties`, or pass `-PvitureSdkDir=/path/to/sdk`.
|
||||
@@ -26,9 +28,10 @@ Device firmware can reject individual modes; Android's physical display size
|
||||
must confirm SBS before the app considers the switch successful.
|
||||
|
||||
All three feed the existing `Glasses.PoseSink`, head recentering, compositor and
|
||||
SteamVR pipeline. Ultra explicitly selects 3DoF before SDK initialization because
|
||||
the current pipeline consumes orientation. Positional 6DoF and camera passthrough
|
||||
are outside this integration. The SDK's optional statistics reporter is not enabled.
|
||||
SteamVR pipeline. Ultra selects 3DoF or the opt-in positional 6DoF mode before SDK
|
||||
initialization. Its positional poses and stereo cameras feed the
|
||||
[Luma spatial pipeline](luma-spatial-tracking.md); camera passthrough is not implemented.
|
||||
The SDK's optional statistics reporter is not enabled.
|
||||
|
||||
The SDK owns USB interface selection and I/O. `UsbDeviceConnection` remains open
|
||||
until stop/shutdown/destroy finish. A generation token rejects callbacks from an
|
||||
@@ -40,19 +43,64 @@ mode can fall back to the native table, with rollback on failure.
|
||||
|
||||
`Game on glasses` uses Android's existing Presentation output. USB control alone
|
||||
does not create a video display: the phone, cable and adapter must expose a usable
|
||||
external display. Discovery checks both display name and product metadata, with
|
||||
external display. Enabling 3D also enables and saves `Game on glasses` before
|
||||
requesting the stereo mode, so the picture uses the glasses' presentation instead
|
||||
of the mirrored phone output. Returning to 2D keeps the game on the glasses.
|
||||
Discovery checks both display name and product metadata, with
|
||||
a single unnamed-display fallback when recognized glasses USB is present.
|
||||
|
||||
Screen width and distance apply to **Virtual screen** rendering. In **Automatic**,
|
||||
ordinary 2D games stay flat, ordinary games with a confirmed SBS display use a
|
||||
virtual screen, and a running SteamVR session uses the direct stereo view.
|
||||
ordinary 2D games stay flat. On an SBS display, the complete flat frame is fitted
|
||||
into each eye without cropping or head tracking. A running SteamVR session uses
|
||||
the direct stereo view. Games or videos that already contain separate eye images
|
||||
use the explicit **Side-by-side game** picture setting. Switching SBS on does not
|
||||
activate a virtual screen or head tracking for ordinary games.
|
||||
The direct view uses SteamVR field of view instead of virtual-screen dimensions
|
||||
while SteamVR is running. A plain side-by-side game fills each eye and does not
|
||||
use the SteamVR field-of-view or recenter controls.
|
||||
The settings UI and session renderer share this mode selection through
|
||||
`GlassesRendering`; home-screen changes configure sessions rather than resizing
|
||||
Android's mirrored launcher. Regular refresh rate applies to 2D output; SBS uses
|
||||
the model's separate 3D refresh rate.
|
||||
|
||||
All supported glasses with an available head-tracking connection have a separate
|
||||
**3DoF virtual screen** toggle, including VITURE, RayNeo and XREAL 1S. Availability
|
||||
comes from the connected transport's IMU capability rather than a model whitelist.
|
||||
The app renders this screen without enabling a built-in 3DoF mode or changing
|
||||
brightness. Enabling it selects Virtual screen, head tracking and Game on glasses,
|
||||
and saves those preferences without changing the glasses' 2D/SBS mode. It works
|
||||
with flat games in either display mode. Disabling it returns the picture to Automatic; switching
|
||||
SBS on or off preserves the 3DoF selection. Virtual screens center on the current
|
||||
viewing direction when tracking starts or the user recenters, including the
|
||||
up/down angle. Direct SteamVR views retain their gravity-aligned recentering.
|
||||
|
||||
The settings UI, menu presentation, and session renderer share this mode selection
|
||||
through `GlassesRendering`. Setup → Smart glasses configures DroidDeck menus,
|
||||
non-Steam launchers, desktop sessions, and games. Tracked menus use the app's own
|
||||
glasses presentation while phone controls remain available; see
|
||||
[tracked menus](glasses-menu-tracking.md). Regular refresh rate applies to 2D
|
||||
output; SBS uses the model's separate 3D refresh rate.
|
||||
Both the home menu and session drawer use **Display**, **VR**, **Tracking**, and
|
||||
**Device** tabs. Display groups connection and output controls, 2D/3D resolution
|
||||
and active refresh rate, then picture and virtual-screen geometry. VR contains
|
||||
SteamVR controls, automatic 3D, VR resolution, and field of view. Tracking groups
|
||||
head orientation, position and floor, and hands with camera setup. Device contains
|
||||
brightness, volume, and sunblock. Unsupported hardware tabs are hidden. Recenter
|
||||
is the first action on every session tab while head tracking is enabled for a
|
||||
3DoF/6DoF virtual screen or SteamVR.
|
||||
|
||||
Display saves **2D resolution** and **3D resolution**; VR saves **VR resolution**. Values are per eye: 1920 x 1080 or, on tall VITURE
|
||||
models, 1920 x 1200. Automatic keeps the existing model default; VR defaults to
|
||||
following the active 2D/3D preference. Home settings expose all three profiles,
|
||||
while a running session shows the active profile next to its refresh control.
|
||||
Switching resolution retains the refresh preference and does not change brightness,
|
||||
color processing, or the session's game-rendering resolution. Android selects an
|
||||
advertised mode with matching dimensions and the closest available refresh rate,
|
||||
without crossing the 2D/SBS layout. VITURE's USB command selects the matching
|
||||
resolution family first. A saved resolution unsupported by another attached model
|
||||
falls back to its automatic behavior. Resolution controls wait for a pending SBS
|
||||
transition to finish before accepting another change.
|
||||
|
||||
Both lists offer 30 Hz only when Android advertises a matching native display mode at the current
|
||||
resolution and **Game on glasses** is enabled. Android's presentation selects that mode without a
|
||||
vendor command. A saved 30 Hz choice falls back to 60 Hz if the display or eye layout lacks it;
|
||||
mirrored output never substitutes a 30 fps rendering limit.
|
||||
Software picture controls remain available while a display is connected even if
|
||||
USB control is lost; hardware controls and head tracking still require USB.
|
||||
Presentation refresh preferences are revalidated on display changes even when
|
||||
|
||||
@@ -186,27 +186,43 @@ alone would then read as an installed client.
|
||||
restarted from the top forever (found with `perf_event_open` IP sampling, since proot holds
|
||||
ptrace). Such a range is now skipped whole and a second exhausted scan returns `-ENOMEM`, as the
|
||||
kernel does (`fex-2610-r3`).
|
||||
- **`build-fex.yml`.** The build job runs third-party images and toolchains with a read-only token,
|
||||
and inputs reach the shell as environment variables. A separate publish job (contents: write) runs
|
||||
only on a dispatch with a tag, host thunks on and no extra `cxxflags` (diagnostic builds such as
|
||||
`-DDEBUG_STRACE` stay artifacts). `droiddeck-steam-x64` pins each tag's sha256 and the archive is not
|
||||
reproducible, so a tag is never rebuilt over. The release is never the repo's Latest nor a
|
||||
pre-release: it is a component fetched by tag, not something users install.
|
||||
- **Building and publishing.** Droid-Deck/DroidDeck-Components' `build-fex.yml` builds `tools/fex` from
|
||||
this repository at a chosen ref (default `main`) with host thunks, in the same Arch Linux ARM container
|
||||
with a read-only token, then a separate job replaces `fex.tzst` on Components' single `fex` release,
|
||||
checks the digest GitHub stored, and records the build's version, sha256 and size in its `fex.json`
|
||||
(in that order: a device that reads the old `fex.json` meanwhile gets a mismatching archive and keeps
|
||||
its FEX). This repository's `build-fex.yml` only builds `tools/fex` when it changes (and on dispatch,
|
||||
for diagnostic builds such as `-DDEBUG_STRACE`); nothing is published from it.
|
||||
- Releases so far: `fex-2609-r2` (no thunks), `fex-2609-r3` (thunks), `fex-2610-r1`, `fex-2610-r2`,
|
||||
`fex-2610-r3`, `fex-2610-r4` (in-root resolution when openat2 is missing, exact MAP_32BIT
|
||||
marking), `fex-2610-r5` (current: bundles libxxhash, which the Linux runtime does not ship; the
|
||||
build fails when a program in the package needs a library that is neither bundled nor listed in
|
||||
`tools/fex/runtime-libs.txt`). They were built and published in The412Banner/DroidDeck; up to r3 their tags point at that
|
||||
repository's `main`, from r4 at the commit each was built from.
|
||||
repository's `main`, from r4 at the commit each was built from. Later builds are versions on the single
|
||||
`fex` release; `fex-2610-r5` stays published as the build `droiddeck-steam-x64` pins.
|
||||
- **Where the app gets it.** The app downloads FEX from Droid-Deck/DroidDeck-Components, where nothing is
|
||||
uploaded by hand: a new tag is published by `build-fex.yml`, recorded in that repository's `fex.json`
|
||||
(sha256 and size), and brought in by its `mirror-release.yml`, which refuses an archive that does not
|
||||
match. `FEX_TAG` and `FEX_SHA256` in `droiddeck-steam-x64` then move to the new tag. The rootfs is
|
||||
uploaded by hand. The app follows its `fex.json`: while the x86-64 client is selected it reads it from
|
||||
Components' `main` (directly, at most every ten minutes) and keeps the last copy, and a session passes
|
||||
that build to `droiddeck-steam-x64` (`BL_FEX_TAG`, `BL_FEX_SHA256`, `BL_FEX_URLS` with the download
|
||||
source's mirrors). The script installs a build only after its archive matches the sha256 and every
|
||||
program in it loads (`ld-linux --list`); the files it replaces are kept aside until the installed copy
|
||||
loads too and put back otherwise, and the tag is written last. Otherwise it keeps the installed one while that still loads,
|
||||
then falls back to `FEX_TAG`/`FEX_SHA256` built in, which stay the first-start default when no copy
|
||||
of `fex.json` has been read. Publishing a FEX build therefore takes a Components run only; a
|
||||
replaced archive cannot be pointed back to, so undoing a build means building an earlier ref. The rootfs is
|
||||
FEX-Emu's own image, unmodified: `fex-rootfs.json` there names its rootfs.fex-emu.gg link, sha256 and
|
||||
size, and `mirror-upstream.yml` fetches it into the catalog's release only when both match. A newer
|
||||
image is a new catalog release; `ROOTFS_URL` and `ROOTFS_SHA256` then move to it, and a device that
|
||||
already unpacked one keeps it.
|
||||
|
||||
- **What a session log shows.** `FEX ready (<tag>, <source>)` names the build and where it came from
|
||||
(`catalog`, `built-in`, or `kept` when a newer one did not install, with the reason a line above);
|
||||
`x86-64 rootfs:` names the image; `x86-64 client:` the client's ubuntu12 and steamrt64 builds at start
|
||||
(it may update itself after). The rootfs download reports every tenth of its size as a step. When the
|
||||
web helper's GPU process keeps exiting until Chromium gives up on the GPU (the menus then never appear),
|
||||
one line says so with the exit codes, followed by x86 `glxinfo -B` and `vulkaninfo --summary` through
|
||||
FEX's thunks.
|
||||
|
||||
## DroidDeck direct for Linux games and runtimes
|
||||
|
||||
- Steam runs native Linux games in the Steam Linux Runtime container (pressure-vessel), which needs
|
||||
|
||||
@@ -139,6 +139,27 @@ the timeline, then lessons and backlog. Companion to the README (what the app *d
|
||||
FEX uses). With the device's libxxhash hidden, r4 failed with the reported error and r5 loaded.
|
||||
Mirrored in DroidDeck-Components (#5); the app moves to fex-2610-r5 and an installed r4 is replaced
|
||||
at the next x86-64 start.
|
||||
- Merged as 7d21ba22.
|
||||
- **FEX from the catalog**: FEX builds may need to follow user reports faster than app builds, so the
|
||||
app now reads Components' `fex.json` (directly, kept between starts) and passes it to the session; a
|
||||
build is installed only once its sha256 matches and its programs load, the installed one stays while it
|
||||
loads, and the build pinned in the script remains the first-start default. **Session logs** for the
|
||||
x86-64 client now name the FEX build and its source, the rootfs and the client's builds, report the
|
||||
rootfs download in tenths instead of a progress bar (one report's log ended in hundreds of bar lines),
|
||||
and, when the web helper's GPU process keeps exiting (a report from an Adreno 840: the menus never
|
||||
appeared), say so in one line with x86 `glxinfo`/`vulkaninfo` through FEX. Settings shows the FEX
|
||||
build installed. FEX is now built and published by DroidDeck-Components' `build-fex.yml` from this
|
||||
repository's `tools/fex`, replacing the archive on its single `fex` release and recording the version
|
||||
in `fex.json`; this repository's `build-fex.yml` only checks the build. Tests: `test_fex_select.py`,
|
||||
`FexCatalogTest`.
|
||||
Components' first build there published `2610-r6` (run 39104443888). Device (Pocket FIT, r4
|
||||
installed): the app read `fex.json`, and the next x86-64 start logged `downloading FEX (fex-2610-r6)`
|
||||
then `FEX ready (fex-2610-r6, catalog)` a second later, with the rootfs and client lines; the client
|
||||
came up on it.
|
||||
- Review of #1053: an install keeps the files it replaces aside until the installed copy loads and puts
|
||||
them back otherwise (a failed copy could leave a partly replaced FEX marked as the new one;
|
||||
`test_fex_install.py`), and the catalog is read on its own thread, not before the library scan, and
|
||||
not in Offline Mode.
|
||||
|
||||
## 2026-10-15 - `feat/x86-64-steam-client`: SD card library, ARM parity, org main, Counter-Strike 2
|
||||
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
# DroidDeck 0.5.0
|
||||
|
||||
<p align="center"><a href="https://discord.gg/JRGAvawjsm"><img src="https://img.shields.io/badge/Discord-Join%20the%20community-5865F2?logo=discord&logoColor=white" alt="Join the DroidDeck Discord"></a></p>
|
||||
|
||||
### Features
|
||||
|
||||
- Added **SteamVR on Meta Quest 3 and 3S** from the standard APK. Quest headsets are detected automatically and get their own driver defaults. Immersive SteamVR runs through a native OpenXR activity with headset tracking, Touch controllers and haptics, and SteamVR's dashboard and Steam library work in VR. (@xXJSONDeruloXx, @Jagger15, [#642](https://github.com/Droid-Deck/DroidDeck/pull/642), [#910](https://github.com/Droid-Deck/DroidDeck/pull/910), [#733](https://github.com/Droid-Deck/DroidDeck/pull/733), [#853](https://github.com/Droid-Deck/DroidDeck/pull/853), [#905](https://github.com/Droid-Deck/DroidDeck/pull/905))
|
||||
|
||||
<img src="media/0.5.0/quest-steamvr.gif" width="50%" alt="On a Quest 3: DroidDeck in passthrough, Steam Big Picture in the headset, the SteamVR dashboard with a Touch controller, then playing Thumper in VR">
|
||||
|
||||
<details>
|
||||
<summary>Flat games on Quest with hand tracking</summary>
|
||||
|
||||
<img src="media/0.5.0/quest-flat-hands.gif" width="50%" alt="Plants vs. Zombies on a DroidDeck panel in Quest passthrough, played with hand tracking">
|
||||
|
||||
</details>
|
||||
- Added experimental **Meta Quest 2** support. (@Jagger15, [#676](https://github.com/Droid-Deck/DroidDeck/pull/676))
|
||||
- Added **XR glasses support for VITURE, RayNeo and XREAL**. Games play straight on the glasses at their own refresh rate while the phone becomes the controller, and the glasses' refresh rate, 3D mode, brightness, volume and sunblock can be changed from the app or the session menu. **Play SteamVR** appears beside Play Steam when supported glasses are connected, with head tracking from the glasses. Joy-Cons work as SteamVR hands, and the phone's camera can track bare hands with Quest-style pinch gestures. Supports VITURE Pro XR, RayNeo Air and GT glasses, and XREAL 1S (experimental). Glasses with head tracking can also show a **head-tracked virtual screen** everywhere in DroidDeck, in its menus, desktop sessions and games, without starting SteamVR, set up from the new **Setup → Smart glasses** page. (@maxjivi05, @informalTechCode, [#732](https://github.com/Droid-Deck/DroidDeck/pull/732), [#907](https://github.com/Droid-Deck/DroidDeck/pull/907), [#1035](https://github.com/Droid-Deck/DroidDeck/pull/1035))
|
||||
|
||||
<img src="media/0.5.0/play-steamvr-glasses.jpg" width="50%" alt="Play SteamVR beside Play Steam with VITURE Pro XR glasses connected">
|
||||
<img src="media/0.5.0/glasses-steamvr.gif" width="50%" alt="Angry Birds VR from Play SteamVR, rendered in stereo with head pose from the glasses and the slingshot pulled with the gamepad">
|
||||
|
||||
<details>
|
||||
<summary>On the glasses, and gamepad hands in SteamVR</summary>
|
||||
|
||||
<img src="media/0.5.0/glasses-2d.gif" width="50%" alt="Recorded from the glasses' own 1920×1080 display: Angry Birds VR with one eye's view, slingshot pulled with the gamepad">
|
||||
<img src="media/0.5.0/glasses-gamepad-hands.gif" width="50%" alt="Gamepad hands pointing at Angry Birds VR's level select">
|
||||
|
||||
</details>
|
||||
- Added **Cardboard VR mode** with Valve's ARM64 SteamVR, splitting the screen into two lens-corrected eyes with head tracking. Windows VR games such as Thumper and The Thrill of the Fight run through Proton. SteamVR and Cardboard share one **VR** tab in the session menu. (@xXJSONDeruloXx, [#704](https://github.com/Droid-Deck/DroidDeck/pull/704), [#959](https://github.com/Droid-Deck/DroidDeck/pull/959))
|
||||
|
||||
<img src="media/0.5.0/cardboard-thumper.jpg" width="50%" alt="Thumper in Cardboard VR mode, split into two lens-corrected eyes">
|
||||
<img src="media/0.5.0/cardboard-thrill-of-the-fight.jpg" width="50%" alt="The Thrill of the Fight's weigh-in scene in Cardboard VR mode">
|
||||
<img src="media/0.5.0/vr-drawer-tab.jpg" width="50%" alt="The session menu's VR tab with SteamVR and Cardboard sections">
|
||||
- Added **Epic Games Launcher for Linux**, Epic's own client, running under FEX and signed in automatically with your linked Epic account. Every Epic game now starts through Steam and the launcher, so games get the same sign-in and online services as on a PC, along with Steam Input and the performance overlay. The launcher can be driven with a gamepad or touch. Games installed from inside the launcher appear in the Games tab automatically, and your Steam games can be started from the launcher's library. (@maxjivi05, [#928](https://github.com/Droid-Deck/DroidDeck/pull/928))
|
||||
|
||||
<img src="media/0.5.0/epic-launcher.gif" width="50%" alt="Opening the Epic Games Launcher from the Epic tab, already signed in, and browsing its library">
|
||||
<img src="media/0.5.0/epic-launcher-games-tab.jpg" width="50%" alt="The Epic Games Launcher in the Games tab next to Epic games, with Launch and Uninstall">
|
||||
|
||||
<details>
|
||||
<summary>Gamepad navigation and launching through Steam</summary>
|
||||
|
||||
<img src="media/0.5.0/epic-launcher-gamepad.gif" width="50%" alt="Moving the selection ring across the launcher's library with the D-pad">
|
||||
<img src="media/0.5.0/epic-launch-through-steam.gif" width="50%" alt="Launching Alone With You from the Games tab: Steam starts the launcher in the background, which hands the game back to Steam">
|
||||
<img src="media/0.5.0/epic-launcher-library.jpg" width="50%" alt="The Epic Games Launcher's library running on DroidDeck">
|
||||
|
||||
</details>
|
||||
- Added an optional **x86-64 Steam client**: Valve's own Linux client, running under FEX beside the default ARM64 one. Choose it under **Steam client** in Steam settings; it applies at the next session. It has its own game library and shares your sign-in. Games with a Linux version, such as Team Fortress 2 and Half-Life, run their Linux builds on it, and Windows games use DroidDeck-Proton as before. New FEX builds reach it without an app update. (@The412Banner, [#152](https://github.com/Droid-Deck/DroidDeck/pull/152), [#1050](https://github.com/Droid-Deck/DroidDeck/pull/1050), [#1053](https://github.com/Droid-Deck/DroidDeck/pull/1053))
|
||||
- **Linux games run under FEX out of the box.** Proton Experimental ARM64, FEX and the Steam Linux Runtimes now download on the first Steam start instead of waiting as pending downloads. x86 Linux games start, keep running and take the controller under FEX, including Unity games, which used to crash or ignore the gamepad. Games that also have a Windows version stay on Proton by default, because the Windows build often runs better. To run a game's Linux build instead, choose **Linux x86 (FEX)** in its compatibility settings. (@maxjivi05, [#1045](https://github.com/Droid-Deck/DroidDeck/pull/1045))
|
||||
- Store game pages now show **HowLongToBeat times, Metacritic scores**, developer, publisher and release date for GOG and Epic games, along with an **achievements** card with filters for unlocked, locked and hidden achievements. (@phobos665, @xXJSONDeruloXx, [#641](https://github.com/Droid-Deck/DroidDeck/pull/641), [#712](https://github.com/Droid-Deck/DroidDeck/pull/712), [#775](https://github.com/Droid-Deck/DroidDeck/pull/775))
|
||||
|
||||
<img src="media/0.5.0/game-page-hltb-metacritic.jpg" width="50%" alt="Backpack Hero's Epic page with its Metacritic score of 76, HowLongToBeat times and achievements card">
|
||||
- Added opt-in **GOG achievements** syncing through Comet, with unlock notifications while you play. (@xXJSONDeruloXx, [#651](https://github.com/Droid-Deck/DroidDeck/pull/651))
|
||||
- Added **sorting and a list view** for the store Library and Installed tabs: sort by recently played, recently installed, title, size or release date, and switch between cards and a compact list. (@kingokksa, [#798](https://github.com/Droid-Deck/DroidDeck/pull/798))
|
||||
|
||||
<img src="media/0.5.0/sort-list-view.gif" width="50%" alt="Sorting the Epic library by size and by title, then switching from the card grid to the list view">
|
||||
- Added a **Storage** page to Setup that shows where DroidDeck's storage goes and frees what is safe to remove: unused Steam runtimes, leftovers from uninstalled games, old logs, shader caches, unused components and Flatpak runtimes. Starting Steam with less than 10 GB free offers to free up space. (@xXJSONDeruloXx, [#913](https://github.com/Droid-Deck/DroidDeck/pull/913))
|
||||
|
||||
<img src="media/0.5.0/storage-overview.jpg" width="50%" alt="Storage page overview with the internal storage bar and its categories">
|
||||
<img src="media/0.5.0/storage-cleanup.jpg" width="50%" alt="Storage Clean up checklist with sizes and the safe items selected">
|
||||
- Added **artwork changing for Steam games and GOG, Epic and Amazon games**, matching added games, with a SteamGridDB search in the art chooser. **Adjust logo** sets where a game's logo sits on its background and how big it is. Chosen art and logo positions are applied in Steam too. (@FX192, [#876](https://github.com/Droid-Deck/DroidDeck/pull/876), [#939](https://github.com/Droid-Deck/DroidDeck/pull/939), [#948](https://github.com/Droid-Deck/DroidDeck/pull/948), [#976](https://github.com/Droid-Deck/DroidDeck/pull/976), [#1004](https://github.com/Droid-Deck/DroidDeck/pull/1004))
|
||||
|
||||
<img src="media/0.5.0/steam-artwork.jpg" width="50%" alt="The Artwork editor for Moonlighter, a Steam game, with its cover, background, logo and icon">
|
||||
- Redesigned the **GPU drivers** page with three modes: **Auto**, **Matched pair**, or **Mix and match** display and runtime drivers. It also shows which drivers are in use now. Auto checks for new drivers when the app opens, and if a new pair never starts a session, it rolls back to the last pair that worked. (@xXJSONDeruloXx, [#858](https://github.com/Droid-Deck/DroidDeck/pull/858), [#972](https://github.com/Droid-Deck/DroidDeck/pull/972))
|
||||
|
||||
<img src="media/0.5.0/gpu-drivers.jpg" width="50%" alt="GPU drivers page with Auto, Matched pair and Mix and match, and the drivers in use">
|
||||
- Added **Boot movie** settings: the DroidDeck movie at 30 or 60 fps, with or without sound, or import your own video, which is converted on the device. (@xXJSONDeruloXx, [#969](https://github.com/Droid-Deck/DroidDeck/pull/969))
|
||||
|
||||
<img src="media/0.5.0/boot-movie.jpg" width="50%" alt="Boot movie settings in the Steam settings sheet: movie, sound and Import">
|
||||
- Added a per-game **Start without Steam** switch for games that close right away because of Proton's Steam setup. (@The412Banner, [#659](https://github.com/Droid-Deck/DroidDeck/pull/659))
|
||||
- Added **Low-latency controller** and **Low-latency touch** toggles to the session menu, which apply immediately while Steam is running. (@xXJSONDeruloXx, [#808](https://github.com/Droid-Deck/DroidDeck/pull/808))
|
||||
|
||||
<img src="media/0.5.0/low-latency.jpg" width="50%" alt="The session menu's Latency group with Low-latency controller and Low-latency touch">
|
||||
- A **three-finger tap** opens and closes Steam's on-screen keyboard in game. (@xXJSONDeruloXx, [#782](https://github.com/Droid-Deck/DroidDeck/pull/782))
|
||||
- Added **Picture position** (Top, Bottom or Center) for Steam sessions, so fixed-aspect games can sit away from on-screen controls on tall screens. (@asdf17128, [#871](https://github.com/Droid-Deck/DroidDeck/pull/871))
|
||||
|
||||
<img src="media/0.5.0/picture-position.jpg" width="50%" alt="Picture position in the Steam settings Display tab">
|
||||
- FEX, DXVK and VKD3D-Proton versions can be **downloaded from the session menu**. (@xXJSONDeruloXx, [#679](https://github.com/Droid-Deck/DroidDeck/pull/679))
|
||||
- Added opt-in **zRAM compression** for the Steam client while a game runs, freeing about 0.5–0.8 GB of memory in testing. (@maxjivi05, [#442](https://github.com/Droid-Deck/DroidDeck/pull/442))
|
||||
- Added **23 more languages**, for 32 in total besides English, selectable in the app or in Android's per-app language settings. Steam follows the chosen language. (@maxjivi05, [#669](https://github.com/Droid-Deck/DroidDeck/pull/669))
|
||||
- Downloads now **resume where they stopped** and fall back to mirrors, with a download source setting that picks the fastest source automatically. This helps especially in mainland China. (@kingokksa, [#623](https://github.com/Droid-Deck/DroidDeck/pull/623))
|
||||
- Added **Reset all settings** in Setup > Diagnostics. It keeps your games, sign-ins and downloads. (@0x1Nawaf, [#696](https://github.com/Droid-Deck/DroidDeck/pull/696))
|
||||
- Store installs can go to **internal storage, any SD card or your second library**. (@maxjivi05, [#928](https://github.com/Droid-Deck/DroidDeck/pull/928))
|
||||
- Controllers with two back paddles (such as the 8BitDo Mobile Controller) map them to the Deck's **L4 and R4**. (@Vynex, [#797](https://github.com/Droid-Deck/DroidDeck/pull/797))
|
||||
- The second screen's Deck controls now include the **Gamepad** tab. On foldables, the unfolded controls hide while a controller is connected. (@ovurrsl, @Gustash, [#895](https://github.com/Droid-Deck/DroidDeck/pull/895), [#829](https://github.com/Droid-Deck/DroidDeck/pull/829))
|
||||
- Added experimental **Android Auto** support. DroidDeck is only available while the car is parked, and needs Android Auto's developer mode with unknown sources enabled. (@zekena2, [#952](https://github.com/Droid-Deck/DroidDeck/pull/952))
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Rewrote the Epic and Amazon store modules**, with byte-exact download verification and proper controller focus on store game pages. Existing sign-ins and installs carry over. (@The412Banner, [#802](https://github.com/Droid-Deck/DroidDeck/pull/802), [#804](https://github.com/Droid-Deck/DroidDeck/pull/804))
|
||||
- **Epic installs need far less free space.** Files are assembled as they download instead of caching the whole game first, so a 71 GB game no longer needs about 142 GB free. Store installs also check for space first and recover better when they run short. (@jmarti326, [#710](https://github.com/Droid-Deck/DroidDeck/pull/710))
|
||||
- **Fewer background processes.** An idle Steam session went from about 41 processes to about 30, keeping it further under Android's child-process limit, and Steam's web helper uses about 250 MB less memory. (@xXJSONDeruloXx, [#566](https://github.com/Droid-Deck/DroidDeck/pull/566), [#575](https://github.com/Droid-Deck/DroidDeck/pull/575))
|
||||
- **Smoother frame pacing.** The session respects Android's peak refresh rate setting, frames are timed to the display's vsync, Steam's frame limiter paces each game by time, and frame generation follows the game's real frame rate. On a 60 Hz Thor, distinct frames on screen rose from about 49.5 to 59.4 a second. (@xXJSONDeruloXx, [#715](https://github.com/Droid-Deck/DroidDeck/pull/715), [#717](https://github.com/Droid-Deck/DroidDeck/pull/717), [#731](https://github.com/Droid-Deck/DroidDeck/pull/731), [#753](https://github.com/Droid-Deck/DroidDeck/pull/753), [#927](https://github.com/Droid-Deck/DroidDeck/pull/927))
|
||||
- The game's busiest threads are boosted onto the **prime cores on the Odin 3**, and Snapdragon 8 Gen 2 devices get default core assignments for games and Steam. (@kmnkv1, @xXJSONDeruloXx, [#682](https://github.com/Droid-Deck/DroidDeck/pull/682), [#654](https://github.com/Droid-Deck/DroidDeck/pull/654))
|
||||
- Games on **SD-card libraries launch faster**, about 46% faster in testing. (@xXJSONDeruloXx, [#757](https://github.com/Droid-Deck/DroidDeck/pull/757))
|
||||
- The Steam home wall now fills from **your own Steam library**, and the store sign-in screens show each store's featured art. (@xXJSONDeruloXx, [#835](https://github.com/Droid-Deck/DroidDeck/pull/835), [#878](https://github.com/Droid-Deck/DroidDeck/pull/878))
|
||||
- **Older and less common devices:** sessions start on kernels that drop fork events (such as the Xiaomi Pad 5), Flatpak installs work on older kernels, the Linux runtime installs on 16 KB page-size devices, and USB drives that only appear as raw storage can be granted All files access. (@xXPOLYGONXx, @xXJSONDeruloXx, @ovurrsl, @Imgau, [#457](https://github.com/Droid-Deck/DroidDeck/pull/457), [#752](https://github.com/Droid-Deck/DroidDeck/pull/752), [#900](https://github.com/Droid-Deck/DroidDeck/pull/900), [#760](https://github.com/Droid-Deck/DroidDeck/pull/760))
|
||||
- Controller-driven cursors now show in nested Wayland sessions. (@xXJSONDeruloXx, [#759](https://github.com/Droid-Deck/DroidDeck/pull/759))
|
||||
- An interrupted Linux runtime download resumes instead of starting again. (@ovurrsl, [#847](https://github.com/Droid-Deck/DroidDeck/pull/847))
|
||||
- Native Linux and FEX compatibility choices are kept instead of being reset to Proton. (@xXJSONDeruloXx, [#749](https://github.com/Droid-Deck/DroidDeck/pull/749))
|
||||
- DroidDeck-Proton releases now come from the [Droid-Deck/DroidDeck-Proton](https://github.com/Droid-Deck/DroidDeck-Proton) repo. (@The412Banner, [#875](https://github.com/Droid-Deck/DroidDeck/pull/875))
|
||||
- Folding or unfolding no longer restarts the front end, file manager or store sign-in. (@ovurrsl, [#832](https://github.com/Droid-Deck/DroidDeck/pull/832), [#886](https://github.com/Droid-Deck/DroidDeck/pull/886))
|
||||
- Session reports include the memory page size, why the app's last processes ended, the panel's refresh modes and the device's thermal state. Logs are synced to storage every 2 seconds, so a hard reset keeps them, and only the 3 newest session log folders are kept. (@ovurrsl, @kmnkv1, @xXJSONDeruloXx, [#897](https://github.com/Droid-Deck/DroidDeck/pull/897), [#513](https://github.com/Droid-Deck/DroidDeck/pull/513), [#852](https://github.com/Droid-Deck/DroidDeck/pull/852), [#990](https://github.com/Droid-Deck/DroidDeck/pull/990))
|
||||
- The store sign-in page can no longer read local files. (@ovurrsl, [#890](https://github.com/Droid-Deck/DroidDeck/pull/890))
|
||||
- Expanded and refreshed translations, including Chinese, Japanese, Turkish, German, French, Spanish, Italian, Portuguese, Russian, Korean and Hungarian. Every language now has every string, including the new Storage, GPU drivers and artwork pages, and Latin American Spanish is complete. (@Dantalian-dev, @ovurrsl, @kingokksa, [#539](https://github.com/Droid-Deck/DroidDeck/pull/539), [#826](https://github.com/Droid-Deck/DroidDeck/pull/826), [#849](https://github.com/Droid-Deck/DroidDeck/pull/849), [#850](https://github.com/Droid-Deck/DroidDeck/pull/850), [#884](https://github.com/Droid-Deck/DroidDeck/pull/884), [#1013](https://github.com/Droid-Deck/DroidDeck/pull/1013))
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed **Android UI flickering** during games on Adreno 740 devices such as the AYN Thor and Odin 2 Portal. (@xXJSONDeruloXx, [#963](https://github.com/Droid-Deck/DroidDeck/pull/963))
|
||||
- Fixed **game rumble stopping after one second** under Proton. (@TideGear, [#938](https://github.com/Droid-Deck/DroidDeck/pull/938))
|
||||
- Fixed **Shut Down and Restart** in Steam's power menu. Shut Down now ends the session, and Restart restarts the client. (@muazkadan, [#629](https://github.com/Droid-Deck/DroidDeck/pull/629))
|
||||
- Fixed the **Quick Access button** doing nothing when the pad is set to Xbox 360, DualShock 4 or DualSense. (@muazkadan, [#631](https://github.com/Droid-Deck/DroidDeck/pull/631))
|
||||
- Fixed **store games becoming Steam installs** when you also own them on Steam. (@xXJSONDeruloXx, [#816](https://github.com/Droid-Deck/DroidDeck/pull/816))
|
||||
- Fixed **added games sometimes never starting** on a cold launch with large Steam libraries. (@xXJSONDeruloXx, @ovurrsl, [#758](https://github.com/Droid-Deck/DroidDeck/pull/758), [#914](https://github.com/Droid-Deck/DroidDeck/pull/914), [#973](https://github.com/Droid-Deck/DroidDeck/pull/973))
|
||||
- Fixed games that never started after a long pause, and "Steam is paused" staying up after the first Resume tap. (@maxjivi05, [#928](https://github.com/Droid-Deck/DroidDeck/pull/928))
|
||||
- Fixed the session ending the third time Steam restarted, for example after changing language. (@KirkPatrickJunsay, [#754](https://github.com/Droid-Deck/DroidDeck/pull/754))
|
||||
- Fixed Steam touch controls ignoring the layout chosen in Steam, including community layouts. (@xXJSONDeruloXx, [#840](https://github.com/Droid-Deck/DroidDeck/pull/840))
|
||||
- Fixed **Batman: Arkham City** crashing at startup and getting stuck on "Running install script". (@TjebbeKerstens, [#792](https://github.com/Droid-Deck/DroidDeck/pull/792), [#793](https://github.com/Droid-Deck/DroidDeck/pull/793))
|
||||
- Fixed **.NET Desktop and .NET Core runtimes** failing to install from Windows components, which blocked MelonLoader mods and Baldur's Gate 3. (@The412Banner, [#740](https://github.com/Droid-Deck/DroidDeck/pull/740))
|
||||
- Fixed the microphone in DirectAudio games. (@xXJSONDeruloXx, [#747](https://github.com/Droid-Deck/DroidDeck/pull/747))
|
||||
- Fixed FEX games failing with "Couldn't connect to FEXServer socket". (@angelrenefm, [#841](https://github.com/Droid-Deck/DroidDeck/pull/841))
|
||||
- Fixed wired **PowerA and PDP** Xbox controllers' right stick and triggers. (@Raidennrl, [#692](https://github.com/Droid-Deck/DroidDeck/pull/692))
|
||||
- Fixed touchpad taps not registering in some games, and the touchpad not moving a game's own cursor when it locks the pointer. (@KirkPatrickJunsay, @Gustash, [#644](https://github.com/Droid-Deck/DroidDeck/pull/644), [#772](https://github.com/Droid-Deck/DroidDeck/pull/772))
|
||||
- Fixed native touch taps snapping the cursor to the top-left corner in some games. (@asdf17128, [#803](https://github.com/Droid-Deck/DroidDeck/pull/803))
|
||||
- Fixed the Android keyboard needing two taps to open after it was dismissed. (@KirkPatrickJunsay, [#665](https://github.com/Droid-Deck/DroidDeck/pull/665))
|
||||
- Fixed Shift staying held after typing capitals on a physical keyboard in the desktop. (@The412Banner, [#766](https://github.com/Droid-Deck/DroidDeck/pull/766))
|
||||
- Fixed an unfolded Follow screen session no longer turning to portrait. (@Gustash, [#626](https://github.com/Droid-Deck/DroidDeck/pull/626))
|
||||
- Fixed battery estimates showing thousands of hours on phones that report current in mA. (@muazkadan, [#820](https://github.com/Droid-Deck/DroidDeck/pull/820))
|
||||
- Fixed blurry home-screen shortcuts for Steam games. (@KirkPatrickJunsay, [#825](https://github.com/Droid-Deck/DroidDeck/pull/825))
|
||||
- Fixed SteamGridDB art staying "no match" after a failed lookup. (@FX192, [#876](https://github.com/Droid-Deck/DroidDeck/pull/876))
|
||||
- Fixed Steam crashes returning to the launcher silently. They now show DroidDeck's error screen, and Steam no longer crashes on Quest when a SteamVR runtime is installed. (@xXJSONDeruloXx, [#905](https://github.com/Droid-Deck/DroidDeck/pull/905))
|
||||
- Fixed the in-session achievements page staying on "Reading…". (@xXJSONDeruloXx, [#961](https://github.com/Droid-Deck/DroidDeck/pull/961))
|
||||
- Fixed crashes from failed store background jobs and cloud save recovery, and a memory error in the directory cache. (@KirkPatrickJunsay, @ovurrsl, @wallentx, [#846](https://github.com/Droid-Deck/DroidDeck/pull/846), [#898](https://github.com/Droid-Deck/DroidDeck/pull/898), [#685](https://github.com/Droid-Deck/DroidDeck/pull/685))
|
||||
- Fixed expired Epic and Amazon sign-ins failing with a generic error instead of asking you to sign in again. (@The412Banner, [#804](https://github.com/Droid-Deck/DroidDeck/pull/804))
|
||||
- Fixed the child-process limit check on Android 12, and its switch name now matches Android's wording. (@ovurrsl, [#955](https://github.com/Droid-Deck/DroidDeck/pull/955), [#848](https://github.com/Droid-Deck/DroidDeck/pull/848))
|
||||
- Fixed the launcher offering its own display as a second screen. (@ovurrsl, [#889](https://github.com/Droid-Deck/DroidDeck/pull/889))
|
||||
- Fixed Windows component detection being confused by folder names above the game folder. (@xXJSONDeruloXx, [#859](https://github.com/Droid-Deck/DroidDeck/pull/859))
|
||||
- Fixed FEX dropdown scaling and Flathub search focus. (@xXJSONDeruloXx, [#681](https://github.com/Droid-Deck/DroidDeck/pull/681))
|
||||
|
||||
### New contributors
|
||||
|
||||
- @xXPOLYGONXx made their first contribution in [#457](https://github.com/Droid-Deck/DroidDeck/pull/457).
|
||||
- @phobos665 made their first contribution in [#637](https://github.com/Droid-Deck/DroidDeck/pull/637).
|
||||
- @Jagger15 made their first contribution in [#642](https://github.com/Droid-Deck/DroidDeck/pull/642).
|
||||
- @KirkPatrickJunsay made their first contribution in [#644](https://github.com/Droid-Deck/DroidDeck/pull/644).
|
||||
- @wallentx made their first contribution in [#685](https://github.com/Droid-Deck/DroidDeck/pull/685).
|
||||
- @Raidennrl made their first contribution in [#692](https://github.com/Droid-Deck/DroidDeck/pull/692).
|
||||
- @0x1Nawaf made their first contribution in [#696](https://github.com/Droid-Deck/DroidDeck/pull/696).
|
||||
- @Producdevity made their first contribution in [#748](https://github.com/Droid-Deck/DroidDeck/pull/748).
|
||||
- @Imgau made their first contribution in [#760](https://github.com/Droid-Deck/DroidDeck/pull/760).
|
||||
- @TjebbeKerstens made their first contribution in [#792](https://github.com/Droid-Deck/DroidDeck/pull/792).
|
||||
- @Vynex made their first contribution in [#797](https://github.com/Droid-Deck/DroidDeck/pull/797).
|
||||
- @kingokksa made their first contribution in [#798](https://github.com/Droid-Deck/DroidDeck/pull/798).
|
||||
- @asdf17128 made their first contribution in [#803](https://github.com/Droid-Deck/DroidDeck/pull/803).
|
||||
- @ovurrsl made their first contribution in [#831](https://github.com/Droid-Deck/DroidDeck/pull/831).
|
||||
- @FX192 made their first contribution in [#876](https://github.com/Droid-Deck/DroidDeck/pull/876).
|
||||
- @angelrenefm made their first contribution in [#894](https://github.com/Droid-Deck/DroidDeck/pull/894).
|
||||
- @informalTechCode made their first contribution in [#907](https://github.com/Droid-Deck/DroidDeck/pull/907).
|
||||
- @TideGear made their first contribution in [#938](https://github.com/Droid-Deck/DroidDeck/pull/938).
|
||||
- @zekena2 made their first contribution in [#952](https://github.com/Droid-Deck/DroidDeck/pull/952).
|
||||
|
||||
[Full changelog](https://github.com/Droid-Deck/DroidDeck/compare/0.4.0...0.5.0)
|
||||
|
After Width: | Height: | Size: 181 KiB |
|
After Width: | Height: | Size: 68 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 2.9 MiB |
|
After Width: | Height: | Size: 4.6 MiB |
|
After Width: | Height: | Size: 144 KiB |
|
After Width: | Height: | Size: 313 KiB |
|
After Width: | Height: | Size: 6.0 MiB |
|
After Width: | Height: | Size: 173 KiB |
|
After Width: | Height: | Size: 2.5 MiB |
|
After Width: | Height: | Size: 1.9 MiB |
|
After Width: | Height: | Size: 2.6 MiB |
|
After Width: | Height: | Size: 163 KiB |
|
After Width: | Height: | Size: 165 KiB |
|
After Width: | Height: | Size: 161 KiB |
|
After Width: | Height: | Size: 171 KiB |
|
After Width: | Height: | Size: 4.5 MiB |
|
After Width: | Height: | Size: 9.3 MiB |
|
After Width: | Height: | Size: 5.3 MiB |
|
After Width: | Height: | Size: 124 KiB |
|
After Width: | Height: | Size: 142 KiB |
|
After Width: | Height: | Size: 182 KiB |
|
After Width: | Height: | Size: 143 KiB |
@@ -0,0 +1,81 @@
|
||||
# Smart-glasses support
|
||||
|
||||
This describes the integrations implemented in DroidDeck as of 2026-10-17. A recognized product name alone does not establish that every firmware, phone, cable, or adapter supports every mode. Controls become available from the connected device's capabilities and the display modes Android actually exposes.
|
||||
|
||||
**3DoF** tracks head rotation: turn, tilt, and roll around a virtual screen. **6DoF** also tracks head position, so leaning or moving changes the view. Both apply to DroidDeck menus, non-Steam launchers, desktop sessions, and games when the tracked virtual screen is enabled. Direct SteamVR rendering uses the corresponding headset pose instead of the virtual-screen layout.
|
||||
|
||||
## Support levels
|
||||
|
||||
- **Integrated:** a device transport and controls are implemented. Tracking requires a working connection and available orientation data.
|
||||
- **SDK-dependent:** integrated support requires an APK built with the optional VITURE SDK. Recognizing the display does not mean that SDK is included.
|
||||
- **Experimental:** implemented and opt-in, with the qualifications described below. This is not a general tracking-accuracy guarantee.
|
||||
- **Display-only:** Android can provide a picture through a compatible display connection, but DroidDeck has no supported control/tracking connection for that device.
|
||||
|
||||
## Device matrix
|
||||
|
||||
| Brand / model family | Level and connection | App-wide 3DoF | App-wide 6DoF / SteamVR position | Luma stereo hand / Joy-Con position |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| VITURE One / One Lite; Pro / Pro XR | Integrated legacy USB transport | Yes, when IMU connection is available | No | No |
|
||||
| VITURE Luma / Luma Pro; Luma Cyber when identified by the SDK | Integrated Luma-family USB transport; Cyber identification uses the SDK market name | Yes, when IMU connection is available | No | No |
|
||||
| VITURE Pro 2 / Pro 2 XR | SDK-dependent Gen2 transport and device identification | Yes | No | No |
|
||||
| VITURE Beast | SDK-dependent Gen2 transport | Yes | No | No |
|
||||
| **VITURE Luma Ultra** | SDK-dependent Carina transport | Yes | **Experimental, opt-in** | **Experimental, opt-in** |
|
||||
| Recognized RayNeo / TCL Air-series devices | Integrated RayNeo USB transport | Yes, with orientation calibration controls | No | No |
|
||||
| RayNeo GT-series devices matching the supported USB identity, including GT Max identification | Integrated GT USB/IMU path | Yes, with orientation calibration controls | No | No |
|
||||
| **XREAL 1S** | **Experimental**; exact USB identity plus the device's USB-Ethernet control/IMU connection | Yes, when that connection is ready | No | No |
|
||||
| Other XREAL models; unrecognized glasses; display adapters and docks | Display-only where Android exposes a usable display; automatic glasses routing also requires display recognition | No supported app tracking connection | No | No |
|
||||
|
||||
RayNeo's Air-series recognition uses its known USB identity or a RayNeo/TCL Air name under the accepted vendor IDs; it is not a per-model certification list. The GT vendor path requires the specific supported product ID. XREAL support currently targets **1S only**: One, One Pro, Air-family devices, and recovery-mode products are not enabled by the 1S transport. VITURE Pro 2 is identified through SDK validation rather than a guessed product ID. A dock does not supply the connected glasses' sensors by itself.
|
||||
|
||||
The final column refers specifically to Luma Ultra's stereo tracking cameras. DroidDeck's separate phone-camera and physical-controller features are independent of this matrix.
|
||||
|
||||
## Display and device controls
|
||||
|
||||
All display modes depend on the phone's video output, the cable/adapter, and Android's advertised modes. USB permission or a working sensor connection alone does not create a video display.
|
||||
|
||||
| Integration | Display selection | Device controls exposed by DroidDeck |
|
||||
| --- | --- | --- |
|
||||
| VITURE One/Lite and Pro families | 1080p per eye; 2D and SBS | Brightness, volume, dimming toggle according to the family profile |
|
||||
| VITURE Luma family | 1080p or 1200p per eye; 2D and SBS | Brightness, volume, dimming toggle |
|
||||
| VITURE Luma Ultra | 1080p or 1200p per eye; 2D and SBS, including the family's 90 Hz SBS selection | Brightness, volume, dimming toggle |
|
||||
| VITURE Beast | 1080p or 1200p per eye; SDK handles native/bypass display modes | Brightness, volume, dimming toggle |
|
||||
| VITURE Pro 2 | 1080p per eye; 2D and SBS | Brightness and volume; no electrochromic-film control |
|
||||
| RayNeo Air/GT | 2D/SBS switching; resolution and refresh follow available Android modes | Orientation calibration; no app brightness, volume, or dimming commands |
|
||||
| XREAL 1S | Supported 1S display-mode commands, including native 1200p/60 Hz and SBS | Brightness and dimming toggle; no app volume command |
|
||||
|
||||
DroidDeck saves separate **2D resolution**, **3D resolution**, and **VR resolution** preferences. Unsupported saved choices fall back to the connected model's automatic behavior. Native **30 Hz** is offered only when Android advertises a matching mode and glasses presentation output is enabled. Higher refresh selections are constrained by the active 2D/SBS layout and model; a firmware-rejected switch is not treated as successful until the physical display confirms it.
|
||||
|
||||
In **Automatic** picture mode, ordinary flat content is duplicated in full into both eyes on an SBS display. **Virtual screen** adds a configurable screen plane and tracking. **Side-by-side game** is for content that already contains separate eye images. SteamVR uses its direct stereo view and field-of-view control.
|
||||
|
||||
## Setup and app-wide behavior
|
||||
|
||||
1. Connect the glasses through a compatible video/data connection. Accept USB access when requested. For XREAL 1S, Android must also expose the glasses' USB-Ethernet network; the app leaves Steam's Internet connection on its normal network.
|
||||
2. Open **Setup → Smart glasses**. Enable glasses output and **3DoF virtual screen**. This selects the tracked screen and saves the setting without forcing SBS or changing brightness.
|
||||
3. Adjust screen width and distance, then recenter while looking where the screen should sit. The same settings apply when navigating DroidDeck, opening a non-Steam launcher, or entering a game/desktop session.
|
||||
4. On an SDK-enabled Luma Ultra, optionally enable positional tracking and set the floor from the current eye height. Use the tracking setup and recovery controls for calibration or recentering. SteamVR startup defers Luma positional initialization until two seconds after SteamVR reports running.
|
||||
|
||||
The phone remains the touch/control surface. DroidDeck mirrors its own visible UI into a tracked glasses presentation; it does not capture Android's entire desktop or other apps. Android 29+ can include app dialogs/popups, while Android 26–28 captures the activity's decor only. System keyboards and permission prompts, secure windows, and separately composed `SurfaceView` content stay on their existing surfaces. Menu recentering does not change the VR floor origin. See [tracked-menu behavior](development/glasses-menu-tracking.md).
|
||||
|
||||
## Luma Ultra tracking qualifications
|
||||
|
||||
Positional tracking, bare-hand controllers, and camera-assisted Joy-Cons are separate options and default off. Bare hands and camera-assisted Joy-Cons are mutually exclusive.
|
||||
|
||||
- **Head tracking:** the SDK supplies the positional pose. The app checks freshness and connection/origin generations and provides tracking-loss and recenter recovery. Screen and floor origins are separate.
|
||||
- **Bare hands:** Full palm + Full landmarks is the default. Raw grayscale is replicated into RGB, MediaPipe applies model tensor normalization, and calibration corrects landmark coordinates before stereo triangulation. Full landmarks support palm orientation, pinch, and grab. The small saved-image comparison favored Full over Lite but does not establish universal hand-detection quality; see the [comparison results](development/hand-model-comparison-2026-10-16.md).
|
||||
- **Controller visibility:** the last accepted visual pose can remain for up to 350 ms from capture time during a short detection gap. Invalid/stale gestures release independently. Normal hand mode has no stationary dead zone; shoulder-based **Steady hand aim** is a separate opt-in setting.
|
||||
- **Joy-Cons:** camera position uses wrist landmark 0. Physical buttons and IMU orientation remain the input sources, with an arm estimate when a usable camera position is unavailable. Camera wrist position does not make a physical controller fully optically tracked in every pose. Screen and Joy-Con recenter actions are separate.
|
||||
- **Calibration:** the included 640×480 Luma Ultra stereo profile is a model-specific starting point. A matching saved device profile takes precedence. The shared profile is labeled separately from an individually distance-verified calibration; optional board collection and distance validation refine a device profile.
|
||||
- **RGB camera:** the third camera remains diagnostic-only. It is not used for controller depth, hand tracking, or camera passthrough. The failed hybrid stereo matcher and temporary USB transport experiments are not part of the live pipeline.
|
||||
|
||||
## Build and troubleshooting references
|
||||
|
||||
Luma Ultra, Beast, and Pro 2 control/tracking require the optional, externally supplied **VITURE Android SDK 2.4.0** in the APK build. Without it, DroidDeck retains its legacy VITURE and RayNeo transports and the separate XREAL integration; it cannot advertise SDK-dependent tracking. Vendor SDK headers and binaries are not stored in this repository. See [VITURE SDK setup](development/viture-sdk.md).
|
||||
|
||||
If only a flat picture is available, check video output first, then the control connection. A disabled tracking toggle can mean missing SDK support, missing USB/network access, or a device without a supported IMU transport. SBS or a new resolution requires a confirmed physical display mode; selecting a preference cannot add a mode the phone/display does not expose.
|
||||
|
||||
- [Luma spatial setup, calibration, and diagnostics](development/luma-spatial-tracking.md)
|
||||
- [Tracked DroidDeck menu implementation](development/glasses-menu-tracking.md)
|
||||
- [Agent diagnostics and capture commands](agent-control.md)
|
||||
- [RayNeo attribution and protocol references](third-party/rayneo/README.md)
|
||||
|
||||
The capability definitions are maintained in [VITURE models](../app/src/main/java/com/droiddeck/launcher/glasses/VitureProtocol.kt), [RayNeo models](../app/src/main/java/com/droiddeck/launcher/glasses/RayNeoModel.kt), [XREAL 1S](../app/src/main/java/com/droiddeck/launcher/glasses/XrealModel.kt), and [connection capability routing](../app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt). Update this matrix when those integrations change.
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
## Included
|
||||
|
||||
[MediaPipe](https://github.com/google-ai-edge/mediapipe) Tasks Vision 0.10.21
|
||||
(`com.google.mediapipe:tasks-vision`, tag `v0.10.21` at cad7f3ab99ebf175947e40c5252c642612aae927;
|
||||
[MediaPipe](https://github.com/google-ai-edge/mediapipe) Tasks Vision 1.0.0
|
||||
(`com.google.mediapipe:tasks-vision`, shared with the Luma stereo tracking pipeline;
|
||||
copyright Google LLC and the MediaPipe authors, Apache License 2.0, `LICENSE_MEDIAPIPE` here) finds
|
||||
the player's body in each camera frame. It comes from Maven Central with its dependencies
|
||||
(Guava, Flogger, protobuf-javalite and Google's `datatransport`), all Apache-2.0.
|
||||
@@ -24,6 +24,16 @@ bundle, a palm detector and a hand landmark model, unchanged from
|
||||
licenses it under the Apache License 2.0; it gives 21 landmarks per hand in the picture and in metres,
|
||||
based on the GHUM hand model, and is meant for prominently shown hands.
|
||||
|
||||
The Luma stereo pipeline defaults to the Full bundle at
|
||||
`app/src/main/assets/models/hand_landmarker.task`, with the same checksum as the Full bundle
|
||||
above. The optional `app/src/main/assets/models/hand_landmarker_lite.task` is built from
|
||||
the official `palm_detection_lite.tflite` and `hand_landmark_lite.tflite` assets at
|
||||
`https://storage.googleapis.com/mediapipe-assets/`. It uses the matching tensor metadata and
|
||||
associated labels from the Full task bundle. These models share the MediaPipe hand tracking
|
||||
model card and Apache License 2.0. Source checksums and the generated bundle checksum are in
|
||||
`app/src/main/assets/models/hand_landmarker_lite.provenance.json`; `tools/build_hand_lite.py`
|
||||
reproduces the bundle with MediaPipe 0.10.21 metadata tooling.
|
||||
|
||||
[CameraX](https://developer.android.com/jetpack/androidx/releases/camera) 1.4.2
|
||||
(`androidx.camera:camera-camera2`, `camera-lifecycle`; copyright The Android Open Source Project,
|
||||
Apache License 2.0) delivers the frames.
|
||||
|
||||
@@ -0,0 +1,504 @@
|
||||
GNU LESSER GENERAL PUBLIC LICENSE
|
||||
Version 2.1, February 1999
|
||||
|
||||
Copyright (C) 1991, 1999 Free Software Foundation, Inc.
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
[This is the first released version of the Lesser GPL. It also counts
|
||||
as the successor of the GNU Library Public License, version 2, hence
|
||||
the version number 2.1.]
|
||||
|
||||
Preamble
|
||||
|
||||
The licenses for most software are designed to take away your
|
||||
freedom to share and change it. By contrast, the GNU General Public
|
||||
Licenses are intended to guarantee your freedom to share and change
|
||||
free software--to make sure the software is free for all its users.
|
||||
|
||||
This license, the Lesser General Public License, applies to some
|
||||
specially designated software packages--typically libraries--of the
|
||||
Free Software Foundation and other authors who decide to use it. You
|
||||
can use it too, but we suggest you first think carefully about whether
|
||||
this license or the ordinary General Public License is the better
|
||||
strategy to use in any particular case, based on the explanations below.
|
||||
|
||||
When we speak of free software, we are referring to freedom of use,
|
||||
not price. Our General Public Licenses are designed to make sure that
|
||||
you have the freedom to distribute copies of free software (and charge
|
||||
for this service if you wish); that you receive source code or can get
|
||||
it if you want it; that you can change the software and use pieces of
|
||||
it in new free programs; and that you are informed that you can do
|
||||
these things.
|
||||
|
||||
To protect your rights, we need to make restrictions that forbid
|
||||
distributors to deny you these rights or to ask you to surrender these
|
||||
rights. These restrictions translate to certain responsibilities for
|
||||
you if you distribute copies of the library or if you modify it.
|
||||
|
||||
For example, if you distribute copies of the library, whether gratis
|
||||
or for a fee, you must give the recipients all the rights that we gave
|
||||
you. You must make sure that they, too, receive or can get the source
|
||||
code. If you link other code with the library, you must provide
|
||||
complete object files to the recipients, so that they can relink them
|
||||
with the library after making changes to the library and recompiling
|
||||
it. And you must show them these terms so they know their rights.
|
||||
|
||||
We protect your rights with a two-step method: (1) we copyright the
|
||||
library, and (2) we offer you this license, which gives you legal
|
||||
permission to copy, distribute and/or modify the library.
|
||||
|
||||
To protect each distributor, we want to make it very clear that
|
||||
there is no warranty for the free library. Also, if the library is
|
||||
modified by someone else and passed on, the recipients should know
|
||||
that what they have is not the original version, so that the original
|
||||
author's reputation will not be affected by problems that might be
|
||||
introduced by others.
|
||||
|
||||
Finally, software patents pose a constant threat to the existence of
|
||||
any free program. We wish to make sure that a company cannot
|
||||
effectively restrict the users of a free program by obtaining a
|
||||
restrictive license from a patent holder. Therefore, we insist that
|
||||
any patent license obtained for a version of the library must be
|
||||
consistent with the full freedom of use specified in this license.
|
||||
|
||||
Most GNU software, including some libraries, is covered by the
|
||||
ordinary GNU General Public License. This license, the GNU Lesser
|
||||
General Public License, applies to certain designated libraries, and
|
||||
is quite different from the ordinary General Public License. We use
|
||||
this license for certain libraries in order to permit linking those
|
||||
libraries into non-free programs.
|
||||
|
||||
When a program is linked with a library, whether statically or using
|
||||
a shared library, the combination of the two is legally speaking a
|
||||
combined work, a derivative of the original library. The ordinary
|
||||
General Public License therefore permits such linking only if the
|
||||
entire combination fits its criteria of freedom. The Lesser General
|
||||
Public License permits more lax criteria for linking other code with
|
||||
the library.
|
||||
|
||||
We call this license the "Lesser" General Public License because it
|
||||
does Less to protect the user's freedom than the ordinary General
|
||||
Public License. It also provides other free software developers Less
|
||||
of an advantage over competing non-free programs. These disadvantages
|
||||
are the reason we use the ordinary General Public License for many
|
||||
libraries. However, the Lesser license provides advantages in certain
|
||||
special circumstances.
|
||||
|
||||
For example, on rare occasions, there may be a special need to
|
||||
encourage the widest possible use of a certain library, so that it becomes
|
||||
a de-facto standard. To achieve this, non-free programs must be
|
||||
allowed to use the library. A more frequent case is that a free
|
||||
library does the same job as widely used non-free libraries. In this
|
||||
case, there is little to gain by limiting the free library to free
|
||||
software only, so we use the Lesser General Public License.
|
||||
|
||||
In other cases, permission to use a particular library in non-free
|
||||
programs enables a greater number of people to use a large body of
|
||||
free software. For example, permission to use the GNU C Library in
|
||||
non-free programs enables many more people to use the whole GNU
|
||||
operating system, as well as its variant, the GNU/Linux operating
|
||||
system.
|
||||
|
||||
Although the Lesser General Public License is Less protective of the
|
||||
users' freedom, it does ensure that the user of a program that is
|
||||
linked with the Library has the freedom and the wherewithal to run
|
||||
that program using a modified version of the Library.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow. Pay close attention to the difference between a
|
||||
"work based on the library" and a "work that uses the library". The
|
||||
former contains code derived from the library, whereas the latter must
|
||||
be combined with the library in order to run.
|
||||
|
||||
GNU LESSER GENERAL PUBLIC LICENSE
|
||||
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
||||
|
||||
0. This License Agreement applies to any software library or other
|
||||
program which contains a notice placed by the copyright holder or
|
||||
other authorized party saying it may be distributed under the terms of
|
||||
this Lesser General Public License (also called "this License").
|
||||
Each licensee is addressed as "you".
|
||||
|
||||
A "library" means a collection of software functions and/or data
|
||||
prepared so as to be conveniently linked with application programs
|
||||
(which use some of those functions and data) to form executables.
|
||||
|
||||
The "Library", below, refers to any such software library or work
|
||||
which has been distributed under these terms. A "work based on the
|
||||
Library" means either the Library or any derivative work under
|
||||
copyright law: that is to say, a work containing the Library or a
|
||||
portion of it, either verbatim or with modifications and/or translated
|
||||
straightforwardly into another language. (Hereinafter, translation is
|
||||
included without limitation in the term "modification".)
|
||||
|
||||
"Source code" for a work means the preferred form of the work for
|
||||
making modifications to it. For a library, complete source code means
|
||||
all the source code for all modules it contains, plus any associated
|
||||
interface definition files, plus the scripts used to control compilation
|
||||
and installation of the library.
|
||||
|
||||
Activities other than copying, distribution and modification are not
|
||||
covered by this License; they are outside its scope. The act of
|
||||
running a program using the Library is not restricted, and output from
|
||||
such a program is covered only if its contents constitute a work based
|
||||
on the Library (independent of the use of the Library in a tool for
|
||||
writing it). Whether that is true depends on what the Library does
|
||||
and what the program that uses the Library does.
|
||||
|
||||
1. You may copy and distribute verbatim copies of the Library's
|
||||
complete source code as you receive it, in any medium, provided that
|
||||
you conspicuously and appropriately publish on each copy an
|
||||
appropriate copyright notice and disclaimer of warranty; keep intact
|
||||
all the notices that refer to this License and to the absence of any
|
||||
warranty; and distribute a copy of this License along with the
|
||||
Library.
|
||||
|
||||
You may charge a fee for the physical act of transferring a copy,
|
||||
and you may at your option offer warranty protection in exchange for a
|
||||
fee.
|
||||
|
||||
2. You may modify your copy or copies of the Library or any portion
|
||||
of it, thus forming a work based on the Library, and copy and
|
||||
distribute such modifications or work under the terms of Section 1
|
||||
above, provided that you also meet all of these conditions:
|
||||
|
||||
a) The modified work must itself be a software library.
|
||||
|
||||
b) You must cause the files modified to carry prominent notices
|
||||
stating that you changed the files and the date of any change.
|
||||
|
||||
c) You must cause the whole of the work to be licensed at no
|
||||
charge to all third parties under the terms of this License.
|
||||
|
||||
d) If a facility in the modified Library refers to a function or a
|
||||
table of data to be supplied by an application program that uses
|
||||
the facility, other than as an argument passed when the facility
|
||||
is invoked, then you must make a good faith effort to ensure that,
|
||||
in the event an application does not supply such function or
|
||||
table, the facility still operates, and performs whatever part of
|
||||
its purpose remains meaningful.
|
||||
|
||||
(For example, a function in a library to compute square roots has
|
||||
a purpose that is entirely well-defined independent of the
|
||||
application. Therefore, Subsection 2d requires that any
|
||||
application-supplied function or table used by this function must
|
||||
be optional: if the application does not supply it, the square
|
||||
root function must still compute square roots.)
|
||||
|
||||
These requirements apply to the modified work as a whole. If
|
||||
identifiable sections of that work are not derived from the Library,
|
||||
and can be reasonably considered independent and separate works in
|
||||
themselves, then this License, and its terms, do not apply to those
|
||||
sections when you distribute them as separate works. But when you
|
||||
distribute the same sections as part of a whole which is a work based
|
||||
on the Library, the distribution of the whole must be on the terms of
|
||||
this License, whose permissions for other licensees extend to the
|
||||
entire whole, and thus to each and every part regardless of who wrote
|
||||
it.
|
||||
|
||||
Thus, it is not the intent of this section to claim rights or contest
|
||||
your rights to work written entirely by you; rather, the intent is to
|
||||
exercise the right to control the distribution of derivative or
|
||||
collective works based on the Library.
|
||||
|
||||
In addition, mere aggregation of another work not based on the Library
|
||||
with the Library (or with a work based on the Library) on a volume of
|
||||
a storage or distribution medium does not bring the other work under
|
||||
the scope of this License.
|
||||
|
||||
3. You may opt to apply the terms of the ordinary GNU General Public
|
||||
License instead of this License to a given copy of the Library. To do
|
||||
this, you must alter all the notices that refer to this License, so
|
||||
that they refer to the ordinary GNU General Public License, version 2,
|
||||
instead of to this License. (If a newer version than version 2 of the
|
||||
ordinary GNU General Public License has appeared, then you can specify
|
||||
that version instead if you wish.) Do not make any other change in
|
||||
these notices.
|
||||
|
||||
Once this change is made in a given copy, it is irreversible for
|
||||
that copy, so the ordinary GNU General Public License applies to all
|
||||
subsequent copies and derivative works made from that copy.
|
||||
|
||||
This option is useful when you wish to copy part of the code of
|
||||
the Library into a program that is not a library.
|
||||
|
||||
4. You may copy and distribute the Library (or a portion or
|
||||
derivative of it, under Section 2) in object code or executable form
|
||||
under the terms of Sections 1 and 2 above provided that you accompany
|
||||
it with the complete corresponding machine-readable source code, which
|
||||
must be distributed under the terms of Sections 1 and 2 above on a
|
||||
medium customarily used for software interchange.
|
||||
|
||||
If distribution of object code is made by offering access to copy
|
||||
from a designated place, then offering equivalent access to copy the
|
||||
source code from the same place satisfies the requirement to
|
||||
distribute the source code, even though third parties are not
|
||||
compelled to copy the source along with the object code.
|
||||
|
||||
5. A program that contains no derivative of any portion of the
|
||||
Library, but is designed to work with the Library by being compiled or
|
||||
linked with it, is called a "work that uses the Library". Such a
|
||||
work, in isolation, is not a derivative work of the Library, and
|
||||
therefore falls outside the scope of this License.
|
||||
|
||||
However, linking a "work that uses the Library" with the Library
|
||||
creates an executable that is a derivative of the Library (because it
|
||||
contains portions of the Library), rather than a "work that uses the
|
||||
library". The executable is therefore covered by this License.
|
||||
Section 6 states terms for distribution of such executables.
|
||||
|
||||
When a "work that uses the Library" uses material from a header file
|
||||
that is part of the Library, the object code for the work may be a
|
||||
derivative work of the Library even though the source code is not.
|
||||
Whether this is true is especially significant if the work can be
|
||||
linked without the Library, or if the work is itself a library. The
|
||||
threshold for this to be true is not precisely defined by law.
|
||||
|
||||
If such an object file uses only numerical parameters, data
|
||||
structure layouts and accessors, and small macros and small inline
|
||||
functions (ten lines or less in length), then the use of the object
|
||||
file is unrestricted, regardless of whether it is legally a derivative
|
||||
work. (Executables containing this object code plus portions of the
|
||||
Library will still fall under Section 6.)
|
||||
|
||||
Otherwise, if the work is a derivative of the Library, you may
|
||||
distribute the object code for the work under the terms of Section 6.
|
||||
Any executables containing that work also fall under Section 6,
|
||||
whether or not they are linked directly with the Library itself.
|
||||
|
||||
6. As an exception to the Sections above, you may also combine or
|
||||
link a "work that uses the Library" with the Library to produce a
|
||||
work containing portions of the Library, and distribute that work
|
||||
under terms of your choice, provided that the terms permit
|
||||
modification of the work for the customer's own use and reverse
|
||||
engineering for debugging such modifications.
|
||||
|
||||
You must give prominent notice with each copy of the work that the
|
||||
Library is used in it and that the Library and its use are covered by
|
||||
this License. You must supply a copy of this License. If the work
|
||||
during execution displays copyright notices, you must include the
|
||||
copyright notice for the Library among them, as well as a reference
|
||||
directing the user to the copy of this License. Also, you must do one
|
||||
of these things:
|
||||
|
||||
a) Accompany the work with the complete corresponding
|
||||
machine-readable source code for the Library including whatever
|
||||
changes were used in the work (which must be distributed under
|
||||
Sections 1 and 2 above); and, if the work is an executable linked
|
||||
with the Library, with the complete machine-readable "work that
|
||||
uses the Library", as object code and/or source code, so that the
|
||||
user can modify the Library and then relink to produce a modified
|
||||
executable containing the modified Library. (It is understood
|
||||
that the user who changes the contents of definitions files in the
|
||||
Library will not necessarily be able to recompile the application
|
||||
to use the modified definitions.)
|
||||
|
||||
b) Use a suitable shared library mechanism for linking with the
|
||||
Library. A suitable mechanism is one that (1) uses at run time a
|
||||
copy of the library already present on the user's computer system,
|
||||
rather than copying library functions into the executable, and (2)
|
||||
will operate properly with a modified version of the library, if
|
||||
the user installs one, as long as the modified version is
|
||||
interface-compatible with the version that the work was made with.
|
||||
|
||||
c) Accompany the work with a written offer, valid for at
|
||||
least three years, to give the same user the materials
|
||||
specified in Subsection 6a, above, for a charge no more
|
||||
than the cost of performing this distribution.
|
||||
|
||||
d) If distribution of the work is made by offering access to copy
|
||||
from a designated place, offer equivalent access to copy the above
|
||||
specified materials from the same place.
|
||||
|
||||
e) Verify that the user has already received a copy of these
|
||||
materials or that you have already sent this user a copy.
|
||||
|
||||
For an executable, the required form of the "work that uses the
|
||||
Library" must include any data and utility programs needed for
|
||||
reproducing the executable from it. However, as a special exception,
|
||||
the materials to be distributed need not include anything that is
|
||||
normally distributed (in either source or binary form) with the major
|
||||
components (compiler, kernel, and so on) of the operating system on
|
||||
which the executable runs, unless that component itself accompanies
|
||||
the executable.
|
||||
|
||||
It may happen that this requirement contradicts the license
|
||||
restrictions of other proprietary libraries that do not normally
|
||||
accompany the operating system. Such a contradiction means you cannot
|
||||
use both them and the Library together in an executable that you
|
||||
distribute.
|
||||
|
||||
7. You may place library facilities that are a work based on the
|
||||
Library side-by-side in a single library together with other library
|
||||
facilities not covered by this License, and distribute such a combined
|
||||
library, provided that the separate distribution of the work based on
|
||||
the Library and of the other library facilities is otherwise
|
||||
permitted, and provided that you do these two things:
|
||||
|
||||
a) Accompany the combined library with a copy of the same work
|
||||
based on the Library, uncombined with any other library
|
||||
facilities. This must be distributed under the terms of the
|
||||
Sections above.
|
||||
|
||||
b) Give prominent notice with the combined library of the fact
|
||||
that part of it is a work based on the Library, and explaining
|
||||
where to find the accompanying uncombined form of the same work.
|
||||
|
||||
8. You may not copy, modify, sublicense, link with, or distribute
|
||||
the Library except as expressly provided under this License. Any
|
||||
attempt otherwise to copy, modify, sublicense, link with, or
|
||||
distribute the Library is void, and will automatically terminate your
|
||||
rights under this License. However, parties who have received copies,
|
||||
or rights, from you under this License will not have their licenses
|
||||
terminated so long as such parties remain in full compliance.
|
||||
|
||||
9. You are not required to accept this License, since you have not
|
||||
signed it. However, nothing else grants you permission to modify or
|
||||
distribute the Library or its derivative works. These actions are
|
||||
prohibited by law if you do not accept this License. Therefore, by
|
||||
modifying or distributing the Library (or any work based on the
|
||||
Library), you indicate your acceptance of this License to do so, and
|
||||
all its terms and conditions for copying, distributing or modifying
|
||||
the Library or works based on it.
|
||||
|
||||
10. Each time you redistribute the Library (or any work based on the
|
||||
Library), the recipient automatically receives a license from the
|
||||
original licensor to copy, distribute, link with or modify the Library
|
||||
subject to these terms and conditions. You may not impose any further
|
||||
restrictions on the recipients' exercise of the rights granted herein.
|
||||
You are not responsible for enforcing compliance by third parties with
|
||||
this License.
|
||||
|
||||
11. If, as a consequence of a court judgment or allegation of patent
|
||||
infringement or for any other reason (not limited to patent issues),
|
||||
conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot
|
||||
distribute so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you
|
||||
may not distribute the Library at all. For example, if a patent
|
||||
license would not permit royalty-free redistribution of the Library by
|
||||
all those who receive copies directly or indirectly through you, then
|
||||
the only way you could satisfy both it and this License would be to
|
||||
refrain entirely from distribution of the Library.
|
||||
|
||||
If any portion of this section is held invalid or unenforceable under any
|
||||
particular circumstance, the balance of the section is intended to apply,
|
||||
and the section as a whole is intended to apply in other circumstances.
|
||||
|
||||
It is not the purpose of this section to induce you to infringe any
|
||||
patents or other property right claims or to contest validity of any
|
||||
such claims; this section has the sole purpose of protecting the
|
||||
integrity of the free software distribution system which is
|
||||
implemented by public license practices. Many people have made
|
||||
generous contributions to the wide range of software distributed
|
||||
through that system in reliance on consistent application of that
|
||||
system; it is up to the author/donor to decide if he or she is willing
|
||||
to distribute software through any other system and a licensee cannot
|
||||
impose that choice.
|
||||
|
||||
This section is intended to make thoroughly clear what is believed to
|
||||
be a consequence of the rest of this License.
|
||||
|
||||
12. If the distribution and/or use of the Library is restricted in
|
||||
certain countries either by patents or by copyrighted interfaces, the
|
||||
original copyright holder who places the Library under this License may add
|
||||
an explicit geographical distribution limitation excluding those countries,
|
||||
so that distribution is permitted only in or among countries not thus
|
||||
excluded. In such case, this License incorporates the limitation as if
|
||||
written in the body of this License.
|
||||
|
||||
13. The Free Software Foundation may publish revised and/or new
|
||||
versions of the Lesser General Public License from time to time.
|
||||
Such new versions will be similar in spirit to the present version,
|
||||
but may differ in detail to address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the Library
|
||||
specifies a version number of this License which applies to it and
|
||||
"any later version", you have the option of following the terms and
|
||||
conditions either of that version or of any later version published by
|
||||
the Free Software Foundation. If the Library does not specify a
|
||||
license version number, you may choose any version ever published by
|
||||
the Free Software Foundation.
|
||||
|
||||
14. If you wish to incorporate parts of the Library into other free
|
||||
programs whose distribution conditions are incompatible with these,
|
||||
write to the author to ask for permission. For software which is
|
||||
copyrighted by the Free Software Foundation, write to the Free
|
||||
Software Foundation; we sometimes make exceptions for this. Our
|
||||
decision will be guided by the two goals of preserving the free status
|
||||
of all derivatives of our free software and of promoting the sharing
|
||||
and reuse of software generally.
|
||||
|
||||
NO WARRANTY
|
||||
|
||||
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
|
||||
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
|
||||
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
|
||||
OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
|
||||
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
|
||||
LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
|
||||
THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
|
||||
WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
|
||||
AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
|
||||
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
|
||||
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
|
||||
LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
|
||||
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
|
||||
FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
|
||||
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
|
||||
DAMAGES.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Libraries
|
||||
|
||||
If you develop a new library, and you want it to be of the greatest
|
||||
possible use to the public, we recommend making it free software that
|
||||
everyone can redistribute and change. You can do so by permitting
|
||||
redistribution under these terms (or, alternatively, under the terms of the
|
||||
ordinary General Public License).
|
||||
|
||||
To apply these terms, attach the following notices to the library. It is
|
||||
safest to attach them to the start of each source file to most effectively
|
||||
convey the exclusion of warranty; and each file should have at least the
|
||||
"copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the library's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This library is free software; you can redistribute it and/or
|
||||
modify it under the terms of the GNU Lesser General Public
|
||||
License as published by the Free Software Foundation; either
|
||||
version 2.1 of the License, or (at your option) any later version.
|
||||
|
||||
This library is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
Lesser General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Lesser General Public
|
||||
License along with this library; if not, write to the Free Software
|
||||
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or your
|
||||
school, if any, to sign a "copyright disclaimer" for the library, if
|
||||
necessary. Here is a sample; alter the names:
|
||||
|
||||
Yoyodyne, Inc., hereby disclaims all copyright interest in the
|
||||
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
|
||||
|
||||
<signature of Ty Coon>, 1 April 1990
|
||||
Ty Coon, President of Vice
|
||||
|
||||
That's all there is to it!
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
Software License Agreement (BSD License)
|
||||
|
||||
Copyright (C) 2010-2015 Ken Tossell
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions
|
||||
are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following
|
||||
disclaimer in the documentation and/or other materials provided
|
||||
with the distribution.
|
||||
* Neither the name of the author nor other contributors may be
|
||||
used to endorse or promote products derived from this software
|
||||
without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
|
||||
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
|
||||
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
|
||||
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
|
||||
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
|
||||
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
|
||||
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||
POSSIBILITY OF SUCH DAMAGE.
|
||||
@@ -0,0 +1,71 @@
|
||||
# RGB USB camera transport
|
||||
|
||||
The optional RGB diagnostic backend uses [libusb 1.0.30](https://github.com/libusb/libusb/releases/tag/v1.0.30)
|
||||
(LGPL 2.1 or later) and [libuvc 0.0.8](https://github.com/libuvc/libuvc/releases/tag/v0.0.8)
|
||||
(BSD 3-Clause). Full license texts are in `LICENSE_LIBUSB` and `LICENSE_LIBUVC` here.
|
||||
|
||||
`tools/build_rgb_uvc.py` obtains the official sources with these SHA-256 pins:
|
||||
|
||||
| Source | SHA-256 |
|
||||
| --- | --- |
|
||||
| [libusb-1.0.30.tar.bz2](https://github.com/libusb/libusb/releases/download/v1.0.30/libusb-1.0.30.tar.bz2) | `fea36f34f9156400209595e300840767ab1a385ede1dc7ee893015aea9c6dbaf` |
|
||||
| [libuvc v0.0.8 ZIP](https://github.com/libuvc/libuvc/archive/refs/tags/v0.0.8.zip) | `67922938ad81eb82b8ce14576f15a68ac75e30af738728a1f3070b6bbc938bce` |
|
||||
|
||||
## Build and replace
|
||||
|
||||
Use Python 3.12 or newer, Android NDK 27.3.13750724, CMake 3.22.1, and Ninja.
|
||||
From the repository root on Windows:
|
||||
|
||||
```powershell
|
||||
python tools/build_rgb_uvc.py --ndk "$env:LOCALAPPDATA/Android/Sdk/ndk/27.3.13750724" --cmake "$env:LOCALAPPDATA/Android/Sdk/cmake/3.22.1/bin/cmake.exe"
|
||||
```
|
||||
|
||||
The default output is `build/rgb-uvc/dist`: `arm64-v8a/libusb.so`, `arm64-v8a/libuvc.a`,
|
||||
public headers in `include`, and license texts in `licenses`. The script records archive,
|
||||
patch, recipe, and binary hashes plus complete build commands in `provenance.json`.
|
||||
Use `--cache` to reuse downloaded archives and `--output` for an independent build.
|
||||
Each run refreshes the extracted vendor sources from the pinned archives before patching.
|
||||
|
||||
Set `droiddeck.uvcDir=C:/path/to/build/rgb-uvc/dist` in the untracked `local.properties`, or
|
||||
pass `-PuvcDir=C:/path/to/build/rgb-uvc/dist` to Gradle. The optional backend and shared libusb
|
||||
are packaged through CMake; license texts are staged into APK assets under `licenses/uvc`.
|
||||
Without this input, the RGB SDK diagnostic remains available and direct UVC reports unavailable.
|
||||
|
||||
libusb is a separate shared library with SONAME `libusb.so`; the application must package
|
||||
it beside its JNI backend and keep it dynamically linked. libuvc is static PIC with hidden
|
||||
symbols. A backend links `libuvc.a`, `libusb.so`, and Android `log`, and includes `libusb.h`
|
||||
before `libuvc/libuvc.h`. Both native components target ARM64/API 26; shared libraries use
|
||||
16 KiB ELF LOAD alignment. No JPEG conversion dependency is included: camera payloads
|
||||
remain MJPEG and are decoded by the caller.
|
||||
|
||||
To rebuild libusb with modifications, run the script once, edit its extracted sources under
|
||||
the output's `src/libusb-1.0.30`, then use the recorded CMake configure/build commands directly
|
||||
without rerunning the Python extraction step. Package the resulting `libusb.so` in a rebuilt
|
||||
APK. Keep the corresponding sources, recipe, patches, and LGPL text with release source
|
||||
distribution; do not rely solely on the upstream download links. The generated manifest
|
||||
describes the pinned build; regenerate hashes when distributing modified binaries.
|
||||
|
||||
## Local changes and validation
|
||||
|
||||
libuvc's queue uses eight transfer buffers instead of the upstream Linux default of 100.
|
||||
The per-transfer limit remains 32 isochronous packets. The patch in
|
||||
`tools/uvc/libuvc-submit-failure.patch` preserves a USB submission error instead of returning
|
||||
success with an incomplete or empty queue. Unsubmitted transfers are removed under the
|
||||
callback mutex. The normal stop path then holds that mutex while checking and cancelling
|
||||
remaining transfers, waits for completion callbacks to clear them, and joins the frame
|
||||
callback thread before returning. This prevents cancellation of already-cleared slots.
|
||||
|
||||
`uvc_submit_failure_test` replaces submission, cancellation, alternate-setting, and companion
|
||||
descriptor USB calls with in-process fakes. It runs 60 starts covering failure before any
|
||||
transfer, failure after three queued transfers, and all eight submitted successfully. Each
|
||||
case checks the return code, cancellation count, stopped state, and cleared transfer slots.
|
||||
The Android executable needs `libusb.so` on its loader path; it does not access USB hardware.
|
||||
Building it verifies linking but does not execute the runtime tests. A separate device run
|
||||
is required to confirm them:
|
||||
|
||||
```sh
|
||||
LD_LIBRARY_PATH=. ./uvc_submit_failure_test
|
||||
```
|
||||
|
||||
The executable built under `build/rgb-uvc-reprotest` passed all 60 cases on the Fold7 on
|
||||
2026-10-16. That run used simulated USB calls and does not establish camera-stream reliability.
|
||||