Merge DroidDeck 0.5.0 into the non-Adreno test branch
Conflicts kept both sides: the 0.5.0 GPU drivers redesign (Auto / pair / mix modes, rollback of a pair that never starts a session) with the test branch's Mali, PowerVR and Xclipse rules (Mali display on the phone's own driver, bridge and system-Vulkan fallbacks, 1-hour list refresh for the open drivers); the test branch's Update button for rebuilt open pairs moved into 0.5.0's new GPU drivers panel; Venus GL probing, the Mali compatibility layer and oom_score_adj redirect kept with 0.5.0's x86-64 Steam client, VAC diagnostics and caller-aware preload hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No files matched your search
@@ -26,14 +26,14 @@ and reports every run.
|
||||
|
||||
Player controls and observations work over authorized ADB on any build: `state`, `start`, `stop`,
|
||||
`resume`, `wait`, `launch`, `quit`, `focus`, `input`, `ui`, `logs`, `screenshot`, `record`, `pacing`,
|
||||
`displays`, `wake`, and `access`.
|
||||
`displays`, `wake`, `glasses` (except `glasses raw`), and `access`.
|
||||
|
||||
**Setup > Diagnostics > Debugging tools** is off by default in every build, including debug builds.
|
||||
It enables `run`, `guest`, `cdp`, all `env` operations, `override`, and `prefs`. The toggle switches
|
||||
directly without a confirmation dialog. The bridge refuses to change it through `ui` or preference
|
||||
writes. The existing preference key, `setting-agent-commands` test tag, JSON command names, and
|
||||
`AGENT_COMMANDS_DISABLED` error code are unchanged. Refusals exit 7; `droiddeckctl access` reports
|
||||
whether commands are enabled.
|
||||
It enables `run`, `guest`, `cdp`, all `env` operations, `override`, `prefs`, and `glasses raw`. The
|
||||
toggle switches directly without a confirmation dialog. The bridge refuses to change it through `ui`
|
||||
or preference writes. The existing preference key, `setting-agent-commands` test tag, JSON command
|
||||
names, and `AGENT_COMMANDS_DISABLED` error code are unchanged. Refusals exit 7; `droiddeckctl access`
|
||||
reports whether commands are enabled.
|
||||
|
||||
Disabling the toggle clears queued and persistent agent environment settings and installed binary
|
||||
overrides. Disabled settings cannot apply at the next session start. Already running commands and
|
||||
@@ -80,6 +80,7 @@ artifact or file error, 7 debugging tools disabled.
|
||||
tools/droiddeckctl state
|
||||
tools/droiddeckctl start steam --wait
|
||||
tools/droiddeckctl start steam --wait --reuse
|
||||
tools/droiddeckctl start steam --steamvr --wait
|
||||
tools/droiddeckctl start desktop
|
||||
tools/droiddeckctl run /usr/bin/foo -- arg1 arg2
|
||||
tools/droiddeckctl wait ready|idle|failed
|
||||
@@ -117,12 +118,38 @@ tools/droiddeckctl prefs set session some_key true --type bool
|
||||
|
||||
tools/droiddeckctl logs latest ./session-artifacts
|
||||
tools/droiddeckctl screenshot ./screen.png --display 4
|
||||
tools/droiddeckctl xr-frame ./xr-capture
|
||||
tools/droiddeckctl record ./clip.mp4 --seconds 8 --display 0
|
||||
tools/droiddeckctl pacing --seconds 30
|
||||
tools/droiddeckctl displays
|
||||
tools/droiddeckctl wake
|
||||
|
||||
tools/droiddeckctl glasses status
|
||||
tools/droiddeckctl glasses set refreshHz 120
|
||||
tools/droiddeckctl glasses set 3d on
|
||||
tools/droiddeckctl glasses simulate --seconds 6 --yaw 15
|
||||
tools/droiddeckctl glasses stop-simulate
|
||||
tools/droiddeckctl glasses mode 0x34
|
||||
tools/droiddeckctl glasses raw 0x0007
|
||||
```
|
||||
|
||||
Glasses status reports `deviceMode` from the MCU separately from `sbs`, which
|
||||
requires a physical stereo display mode. `switching3d` marks the bounded
|
||||
confirmation window. A successful mode-command reply alone does not confirm 3D.
|
||||
`badFrames` counts reports and IMU packets dropped for a bad checksum.
|
||||
|
||||
`xr-frame` captures both app eye textures while SteamVR is open, saving `left.png`,
|
||||
`right.png` and `capture.json` with the matching source frame, projection poses,
|
||||
FOV and dimensions. It works over wired or wireless authorized ADB.
|
||||
Newer builds also include `source.png`, the incoming stereo texture before the
|
||||
Android eye draw, and sRGB attachment/write state in the metadata for color debugging.
|
||||
Source PNGs use the incoming frame's color space and eye PNGs use sRGB. Transfers use Android's
|
||||
gzip utility to avoid sending full uncompressed images over wireless ADB.
|
||||
A unique request ID prevents stale images from passing as a fresh capture; no valid XR
|
||||
layer produces an explicit error. Captures exclude Meta system overlays and
|
||||
the headset compositor's final reprojection. GPU readback runs only on request
|
||||
and can briefly affect frame timing. Remote files are overwritten each time.
|
||||
|
||||
A sleeping device suspends the session and hides the app's windows from `ui dump`; `wake` first.
|
||||
|
||||
`pacing` reads SurfaceFlinger's timestats for the game's layer (the session's SurfaceView when the
|
||||
@@ -141,7 +168,7 @@ mode except the desktop).
|
||||
`state` returns `schema`, `build`, `appVersion`, `runtime {installed, version}`,
|
||||
`agent {guest, guestFocus, commands}` and `session`:
|
||||
|
||||
- `id`, `phase`, `running`, `mode`, `program`, `steamUi`, `steamUrl`, `suspended`, `pip`,
|
||||
- `id`, `phase`, `running`, `mode`, `program`, `steamUi`, `steamUrl`, `steamVr`, `suspended`, `pip`,
|
||||
`firstFrame`, `output` [w, h], `refreshHz`, `lastTransitionAt`, `guestPid`, `installing`
|
||||
- `logDir`, `eventsFile`, `artifactsAvailable`, `artifactsComplete`, `failure {code, message, status}`
|
||||
- `focus`: gamescope's focus as the guest agent last published it, or null:
|
||||
@@ -158,6 +185,145 @@ there stays behind the client's loading screen however healthy its window looks
|
||||
server's windows with map state, override-redirect, app id, `WM_STATE`, pid, class, opacity and
|
||||
`_NET_WM_STATE`.
|
||||
|
||||
## Hand tracking diagnostics
|
||||
|
||||
With debugging tools enabled, the glasses provider accepts `spatial-hand-options` with optional
|
||||
boolean `lite`, `halfRate`, `parallel`, `adaptive`, and `scheduled` fields. These are temporary
|
||||
tracking-session controls and do not change the CPU/GPU delegate. `parallel` selects concurrent
|
||||
camera inference on CPU; GPU remains serial on its owning thread. `scheduled` enables persistent
|
||||
one-hand and two-hand detector banks on CPU, using the one-hand bank after stable agreement
|
||||
and periodically searching for another hand. GPU retains ordinary two-hand inference.
|
||||
`adaptive` lets CPU input rise from 500 permille to 600 or 800 only after sustained processing
|
||||
and observation-age headroom. Slow processing lowers the rate, and tracking loss resets it.
|
||||
Explicit `halfRate` selects manual input and disables adaptation; if both fields are supplied,
|
||||
`adaptive` is applied afterward. This allows restoring the manual preference and automatic mode
|
||||
in one request. CPU, full model, parallel inference, scheduled discovery, and automatic input
|
||||
starting at 500 permille are the defaults.
|
||||
|
||||
`handModel`, `handCpuParallel`, `handAdaptiveInput`, and `handScheduledTracking` identify the live
|
||||
selection. `handInputRatePermille` reports the current requested frame fraction, while
|
||||
`cameraCopyPermille` records the fraction used for the latest reported camera processing sample.
|
||||
At a 25 Hz source, 500/600/800 correspond to input limits of about 12.5/15/20 Hz, not completed
|
||||
tracking throughput. Calibration and explicit capture use 1000. `cameraFrameDivisor` remains
|
||||
2 for 500, 1 for 1000, and 0 for intermediate fractions; use `cameraCopyPermille` for rate calculations.
|
||||
`cameraHz` still describes the source cameras; `rateLimitedFrames` counts deliberate skips and
|
||||
`dropped` counts eligible pairs replaced while processing is busy. These controls do not change
|
||||
the glasses' sensor rate or native head tracking.
|
||||
|
||||
`spatial-hand-capture`, with debugging tools enabled, requests one local stereo pair in
|
||||
`files/spatial/captures`. Use only for an explicitly requested diagnostic capture; it includes
|
||||
the wearer's surroundings. It does not upload images.
|
||||
|
||||
`spatial-rgb-capture` requests one local JPEG from the Luma Ultra's separate UVC RGB camera.
|
||||
It requires Android camera permission, USB permission for that camera, and debugging tools.
|
||||
If USB permission is requested, approve it and explicitly repeat the capture request; permission
|
||||
approval alone does not save an image. Poll `spatial-rgb-status` for completion and the local
|
||||
`files/spatial/rgb-captures` path. The isolated worker stops the stream after the requested
|
||||
diagnostic window or a timeout, then exits. The default `transport: "sdk"` uses the vendor's
|
||||
fixed 1920×1080/30-fps stream. With the optional UVC build present, `transport: "uvc"` selects
|
||||
the independent eight-buffer transport; `width`, `height`, `fps`, and `durationMs` default to
|
||||
640, 480, 30, and 5000. Duration is bounded to 1000–30000 ms and camera modes must be supported
|
||||
by the device. This remains a manual debugging command, not a source of live controller positions.
|
||||
Both paths save `usb.log` and `usb-descriptors.bin`, including when acquisition fails.
|
||||
Direct UVC reports `snapshotAcquired` separately from `streamHealthy`. Health requires the
|
||||
full requested observation window, at least 80% of requested FPS, and no frame gap exceeding
|
||||
the larger of 250 ms or three frame periods. `observedFps`, `maxFrameGapMs`, and
|
||||
`tailFrameAgeMs` include silence at the end of the window. Status remains busy with
|
||||
`phase: "finishing"` until the capture worker exits, so a new request cannot race its teardown.
|
||||
`receiptNs` is the host callback time, not camera exposure time; the SDK timestamp can be zero.
|
||||
This diagnostic does not supply controller positions or replace the stereo tracking stream.
|
||||
|
||||
`spatial-hand-benchmark` takes a numeric `capture` directory name from the app's existing
|
||||
`files/spatial/captures` folder. It replays that saved stereo pair on CPU with full-model two-hand
|
||||
inference, scheduled full, scheduled lite, then full-model two-hand inference again, all with
|
||||
concurrent cameras. It clears live hand input during the comparison and never captures new images.
|
||||
Poll `glasses.spatial.handBenchmark` for completion, timing, and geometric results. Every replay
|
||||
row reports `parallel` and `scheduled`. `wristOnly` identifies the selected controller mode: bare-hand
|
||||
replay estimates `hands`, while Joy-Con replay estimates `joyCons` from wrist 0. Both report
|
||||
`wrists` after stereo geometry checks. Image preprocessing is raw grayscale replicated into RGB;
|
||||
lens correction applies once to landmark coordinates before triangulation. `wrists` reports landmark 0 after the existing stereo depth
|
||||
and reprojection checks, with the same unambiguous handedness pairing. Live Joy-Con tracking
|
||||
also uses wrist 0 directly, with motion and freshness checks; its `positionEstimator` is
|
||||
`joycon-wrist`. Samples identify the selected bank as `inferenceHandLimit`
|
||||
(1 or 2). Timings exclude model construction. Saved-pair results are not live accuracy,
|
||||
second-hand discovery delay, game performance, or motion-to-display latency measurements.
|
||||
|
||||
The live baseline is Full palm + Full landmarks. `handImagePreprocessing` is `raw-grayscale-rgb`;
|
||||
`handLensCorrection` reports `pinhole-points-only`, `fisheye-points-only`, or `none`.
|
||||
`steadyHandAim` is true only when bare hands and the optional shoulder-based aim setting are active.
|
||||
|
||||
`handTracking` includes `leftRawHands`/`rightRawHands` before handedness-confidence filtering,
|
||||
`previousLeftRawHands`/`previousRightRawHands` from the previous inference on those detectors
|
||||
(-1 after detector creation), and `leftInferenceMs`/`rightInferenceMs`. Existing
|
||||
`leftCameraHands`/`rightCameraHands` count confident handedness results. Compare camera timing
|
||||
against its previous raw count: the stock two-hand graph may repeat palm search when fewer
|
||||
than two hand regions were tracked. `inferenceHandLimit` identifies the bank that produced these
|
||||
counts; previous counts belong to that same persistent bank, not necessarily the preceding camera
|
||||
frame. These counts approximate graph conditions; they do not directly expose internal graph
|
||||
execution. Replay samples also include raw counts and camera timings. Accepted palms still
|
||||
require stereo depth, while pinch and grip validity now use agreement between image-space
|
||||
finger estimates from both cameras. Palm acceptance alone does not establish valid gesture input.
|
||||
|
||||
`handTracking.experimental` is always true. `observationTimeNs` identifies the processed camera
|
||||
observation; `observationAgeMs` is its age when diagnostics were published. Per-side `left` and
|
||||
`right` records expose `outcome`, `reconstructedPalmLandmarks`, raw trigger/grip estimates,
|
||||
geometry validity, and final input validity. `observations` contains connection-lifetime
|
||||
`processedPairs`, `inferencePairs`, and independent left/right acceptance, rejection-cause,
|
||||
tracking-loss, and gesture-loss counters. Each processed camera sequence is counted once;
|
||||
repeated status polling does not inflate these counters. Setup/off/clock/head failures count
|
||||
as processed pairs but not completed inference pairs. Skipped callbacks are not fabricated as
|
||||
observations. Sample deltas over a controlled hand-visible interval and inspect rejection
|
||||
causes before changing thresholds; these counters do not prove external latency or accuracy.
|
||||
|
||||
`spatial.calibrationSource` is `device`, `luma-ultra-base`, or `none`.
|
||||
`calibrationVerified` indicates an individual distance-validated profile; `calibrated` indicates
|
||||
geometry is available for experimental tracking, including the bundled model base. A shared base
|
||||
does not claim validation on every Luma Ultra. Matching local profiles take precedence.
|
||||
|
||||
`glasses set joyConTracking true` enables experimental Luma stereo-assisted Joy-Cons after
|
||||
positional tracking is enabled. It defaults off, excludes bare-hand controllers, and appears in
|
||||
both saved settings and spatial diagnostics. Driver logs identify each hand's transition between
|
||||
`stereo hand` and `6DoF arm estimate`; camera positions are hand proxies, while Joy-Con sensors and
|
||||
buttons supply orientation and input.
|
||||
|
||||
The shared-memory channel is protocol **4**, still 176 bytes. Snapshot flag bit 2 selects Luma
|
||||
Joy-Con tracking; bit 1 selects bare-hand controllers. Its word at byte offset 36 is
|
||||
`input_epochs`: four unsigned bytes in left-trigger, left-grip, right-trigger, right-grip order.
|
||||
Each byte increments modulo 256 when that gesture becomes ineligible. Inspect these counters
|
||||
when a loss may have happened between two reader samples: a later valid pose does not erase
|
||||
the intervening loss. Versions 1 through 3 are rejected, so a decoder, app, and driver must agree
|
||||
on version 4. Hand observation age remains limited to 200 ms, head age to 100 ms. For bare-hand
|
||||
controllers, a gap beyond
|
||||
200 ms from the driver's last accepted observation releases input state even when the next
|
||||
read contains a fresh replacement. Duplicate observations cannot rearm an input; recovery
|
||||
requires a newer valid released gesture before a subsequent press.
|
||||
|
||||
Driver lines containing `left optical input` or `right optical input` report state changes,
|
||||
changed connection/origin/loss counters, and changed input API errors, rather than every frame.
|
||||
`pose` is the displayed pose's validity and can remain true during its short held-pose window;
|
||||
`input` is the current observation's eligibility for input processing. Read `flags` for independent
|
||||
gesture validity. `raw` contains trigger strength/grip state; `out` contains submitted trigger
|
||||
strength/click/grip; `armed` and `epochs` are trigger/grip pairs. `error` is ordered **grip, trigger
|
||||
click, trigger scalar**, using OpenVR's `EVRInputError` codes (0 means `VRInputError_None`).
|
||||
The line also includes `ageMs`, `connection`, and `origin`. `controller inputs cleared: error=...`
|
||||
records neutralization on mode changes, disconnect, or deactivation; a nonzero value is the last
|
||||
failed component update in that clearing pass.
|
||||
|
||||
The driver publishes these input updates before the controller pose, including invalid poses,
|
||||
and clears components before deactivation. A zero API error establishes successful submission,
|
||||
not delivery of a dashboard mouse-up event. The installed Vive compositor binding maps trigger
|
||||
to dashboard `LeftClick` (0.65 activation, 0.60 release) and grip to `Back`. Its grab-handle
|
||||
JavaScript attaches to the initiating hand on mouse-down and clears that attachment on mouse-up;
|
||||
that component has no tracking-invalid cleanup. A stuck dashboard drag therefore needs the
|
||||
driver submission and actual dashboard behavior checked separately. Do not synthesize a press
|
||||
or a valid pose as a recovery shortcut.
|
||||
|
||||
Controller display smoothing is independent of these gesture logs: there is no stationary
|
||||
dead zone. Poses converge with one 15 ms correction and bounded receipt-relative translation
|
||||
prediction (80 ms / 2 cm). A missed observation holds the last pose for up to 350 ms from capture;
|
||||
input freshness remains limited to 200 ms, and a detected miss releases inputs immediately.
|
||||
Neither stabilization nor prediction refreshes camera timestamps or creates gesture input.
|
||||
|
||||
## Events
|
||||
|
||||
Each session writes `events.jsonl` beside its other artifacts: the lifecycle (`session.created`,
|
||||
@@ -176,7 +342,7 @@ Automator show them:
|
||||
| `play-steam` | Play on the Steam and Games pages |
|
||||
| `setting-<key>` | every toggle and choice row, by its key (`setting-logs`, `setting-agent-commands`, ...) |
|
||||
| `tab-<n>` | a page's tab strip (Setup's Overview, Controller, Session, Launcher...), by position |
|
||||
| `drawer-tab-controller`, `-second_screen`, `-display`, `-effects`, `-games`, `-session` | the session drawer's pages |
|
||||
| `drawer-tab-controller`, `-second_screen`, `-display`, `-effects`, `-vr`, `-games`, `-session` | the session drawer's pages |
|
||||
| `drawer-stop`, `drawer-stop-confirm`, `drawer-stop-cancel` | stopping a session from the drawer |
|
||||
|
||||
## Scenarios
|
||||
|
||||
@@ -61,7 +61,8 @@ switches the picture to side-by-side and runs SteamVR with three DroidDeck piece
|
||||
1. **`driver_droiddeck.so`**, an OpenVR driver with one HMD:
|
||||
- pose: the shared head pose file (3DoF, standing height);
|
||||
- projection: exactly the frustum the Cardboard pass samples, with identity distortion (the app does
|
||||
the lens);
|
||||
the lens). A new frustum mid-run (the glasses' SteamVR field of view) goes out with
|
||||
`SetDisplayProjectionRaw` and `VREvent_LensDistortionChanged`;
|
||||
- a proximity sensor that reads "worn" while Cardboard mode is on. Without it SteamVR takes a still
|
||||
headset (a phone on a table) for an unused one, enters standby after a few seconds, and the
|
||||
compositor stops submitting frames;
|
||||
@@ -96,15 +97,20 @@ switches the picture to side-by-side and runs SteamVR with three DroidDeck piece
|
||||
Turnip/KGSL cannot export. The layer creates it unshared and hands out a memfd holding a 64-bit
|
||||
counter instead. Every process with the semaphore keeps a real local one, and a poller thread
|
||||
carries values both ways (reached locally: up into the counter; seen in the counter: signalled
|
||||
locally from the host), so GPU waits work as written, a poll period late. Waits on an imported
|
||||
binary semaphore, which cannot be bridged that way, are dropped. With the semaphores dropped
|
||||
locally from the host), so GPU waits work as written, a poll period late. An imported binary
|
||||
semaphore cannot be bridged that way, and its import is refused. With the semaphores dropped
|
||||
outright (the first version) the dashboard showed torn, half-drawn tiles;
|
||||
- Turnip on KGSL does not defer a submit that waits on a timeline value nobody has signalled yet:
|
||||
`vkQueueSubmit` blocks until it is. SteamVR's client waits on a compositor value that the
|
||||
compositor signals only after the client has submitted and said so, which deadlocked every app
|
||||
before its first frame (`helloxr`'s swapchain acquire, DXVK's submit thread in a Proton game).
|
||||
A submit's wait on an imported shared value that is neither reached nor queued in that process
|
||||
gets 3 ms for the other process to reach it, then is dropped;
|
||||
gets 3 ms for the other process to reach it, then is dropped. On a counter that has not moved for
|
||||
100 ms it is dropped at once: Angry Birds VR waits on two such values a frame, and the grace cost
|
||||
its submit thread 6 ms a frame and held it at 60 fps. Keeping these waits stops a Proton game
|
||||
before its first frame (Angry Birds VR stayed on SteamVR's loading card). While a Quest
|
||||
session's frame socket (`/run/droiddeck/droiddeck-xr-frames`) exists, every wait is kept, as the
|
||||
Quest path was built and tested with them;
|
||||
- Steam's web helper has the semaphore bridge whenever SteamVR is installed: its GPU process
|
||||
outlives SteamVR runs, and one started without the layer crashed when SteamVR showed Steam's
|
||||
overlays, until Steam quit;
|
||||
@@ -118,7 +124,13 @@ switches the picture to side-by-side and runs SteamVR with three DroidDeck piece
|
||||
(values up to 3) submitted on them.
|
||||
3. **`droiddeck-steamvr`**, the guest helper (`status | install | start | stop`), which the app drives
|
||||
through the guest agent's fixed `vr` request (no agent commands needed). It also supplies an `xset`
|
||||
stand-in, because the dashboard's web helper checks the X server with it.
|
||||
stand-in, because the dashboard's web helper checks the X server with it. Before every start it
|
||||
removes SteamVR's crash marker (`config/vrserver_crash_timestamp.txt`, written when vrserver
|
||||
crashes) and clears `driver_droiddeck.blocked_by_safe_mode`: after a crash early in a run SteamVR
|
||||
starts in safe mode and blocks every add-on driver, so a single crash left the glasses blank on
|
||||
every later launch (`HmdNotFound`). The driver also stops its pose, vsync and presenter threads
|
||||
if vrserver exits without deactivating it, since a running `std::thread` destroyed at exit aborts
|
||||
the process.
|
||||
|
||||
Imported eye textures carry the same format list SteamVR's mutable sRGB images do. With it Turnip keeps
|
||||
UBWC compression on, and an import that disagreed read compressed data as garbage.
|
||||
@@ -136,20 +148,346 @@ Verified on the AYN Thor (SD 8 Gen 2, Turnip): SteamVR 2.17.10 starts with the D
|
||||
the compositor runs driver-direct at about 60 fps, the dashboard (Steam's library in VR, with the
|
||||
controller's laser) shows cleanly in both eyes at the panel's resolution while Steam keeps running, and
|
||||
a pose written to the shared file turns SteamVR's view. Started from the drawer button through the agent.
|
||||
Once a headset (VITURE, RayNeo or XREAL glasses) is connected and under DroidDeck's control, the
|
||||
Steam page shows **Play SteamVR** at the top left, with a **Settings** badge that drops that headset's
|
||||
settings from the top; before that neither appears. Play SteamVR starts Steam and then SteamVR
|
||||
(installing it first if needed) once the glasses show a picture, and SteamVR stops when the glasses
|
||||
are unplugged. **Play Steam** never starts it: while SteamVR runs, Steam keeps its own window in front
|
||||
of flat games. Through a VITURE Mobile Dock the glasses are only a display: the dock does not pass
|
||||
their controls or head tracking to the phone, so the Steam page shows neither, and only the session
|
||||
drawer's glasses page (output and **Game on glasses**) applies.
|
||||
|
||||
SteamVR defaults to 2D output: a single eye fills the glasses while head tracking remains active.
|
||||
Enable **3D (side-by-side)** manually or opt into **3D for VR games** in the glasses controls for stereo. Automatic
|
||||
switching waits until SteamVR reports that it is running; starting or installing it does not change
|
||||
the display mode. A saved **3D for VR games** preference is retained.
|
||||
|
||||
While SteamVR runs, its side-by-side source takes priority over the glasses' saved picture mode,
|
||||
including **3DoF virtual screen**. SteamVR already applies head tracking; projecting its combined
|
||||
eye pair onto a virtual screen would show both images in each eye and apply head rotation again.
|
||||
The glasses still supply poses to SteamVR. The saved picture mode resumes when SteamVR stops.
|
||||
|
||||
With **Game on glasses** off, SteamVR keeps the glasses' picture settings and IMU while Android
|
||||
mirrors the phone output to the glasses. Turning off the dedicated display must not turn off the
|
||||
renderer/HMD proximity flag: SteamVR otherwise reports the headset as taken off and enters standby,
|
||||
freezing the view on both displays. Without a glasses display or an enabled phone VR viewer, SteamVR
|
||||
uses a single-eye phone preview with the phone's tracking preference. These effective rendering
|
||||
settings last only while SteamVR runs; the saved Cardboard setting is unchanged.
|
||||
Verified on the VITURE Pro 2 with Game on glasses off: live glasses IMU, roughly 60 fps beyond
|
||||
the previous standby point, and user-confirmed updating images and correct head tracking on both
|
||||
screens. The rendering regression tests cover mirrored/dedicated parity and phone fallback.
|
||||
|
||||
A separate freeze occurred after swapping Pro 2 for GT Max and restarting SteamVR in the same
|
||||
DroidDeck process (2026-10-14, session 52): GT Max board `0x41` kept streaming poses and the HMD
|
||||
stayed active, but `vrcompositor` reported `vkerror=-4` (`VK_ERROR_DEVICE_LOST`) from 17:18:53.
|
||||
Restarting the Steam session alone reproduced it (sessions 53/54). Restarting DroidDeck itself
|
||||
cleared it; session 55 continued rendering with live GT Max tracking for over a minute without
|
||||
new device-loss errors. This is a recovery workaround, not a confirmed permanent fix; retained
|
||||
graphics state across restarts is a hypothesis. Frame counters alone do not prove recovery:
|
||||
the presenter can keep presenting an old SteamVR image while the compositor reports device loss.
|
||||
|
||||
### Joy-Con controllers
|
||||
|
||||
**Glasses → Tracking → Experimental → Luma Joy-Con tracking** is off by default. Enable
|
||||
positional tracking first, then this toggle. It uses the Luma Ultra primary stereo camera pair
|
||||
to locate each hand holding a Joy-Con, with the existing calibrated, capture-time head transform.
|
||||
Joy-Con IMU orientation, buttons, sticks, battery reporting and haptics remain in use. This mode
|
||||
also enables Joy-Con takeover for SteamVR even if the ordinary controller preference is off.
|
||||
Android 12+ and controller IMU support are still required.
|
||||
|
||||
Stereo positions and the 6DoF head share the same recentered floor origin. If a hand is hidden,
|
||||
invalid, or its observation is older than 200 ms, that side blends to the arm model anchored to
|
||||
the current head position. Reacquisition also blends (180 ms time constant). No usable calibration
|
||||
or visible palm means arm-model-only positioning. Loss of head tracking beyond 100 ms invalidates
|
||||
the controllers and releases their inputs. Reconnects, origin changes and Joy-Con recentering discard
|
||||
old positional corrections. Point the Joy-Cons forward when connecting or recentering; camera
|
||||
positions do not correct IMU yaw drift. Position extrapolation is disabled in this mode.
|
||||
|
||||
The camera observes the gripping **hand**, not the Joy-Con shell. Occluded palm landmarks, crossed
|
||||
hands, and handedness ambiguity can reduce tracking availability. This is an experimental hand
|
||||
position proxy with estimated fallback, not marker-based controller tracking. Controller grip offset,
|
||||
aiming, motion latency, and sustained performance still need physical validation with held Joy-Cons.
|
||||
|
||||
Luma Joy-Con tracking and stereo bare-hand controllers are mutually exclusive. Either selected
|
||||
Luma controller mode pauses phone-camera inference. External VR controllers retain priority.
|
||||
Disable both Luma controller toggles to return to the upstream Joy-Con/device-camera modes,
|
||||
which retain their existing fixed head-position anchor. Positional tracking, stereo controllers,
|
||||
and phone-camera controls are grouped at the bottom of the glasses settings under **Experimental**.
|
||||
|
||||
Nintendo Joy-Cons paired over Bluetooth become SteamVR hands while SteamVR runs, and stay an ordinary
|
||||
game pad otherwise. It needs Android 12 or later (motion sensors on an `InputDevice`) and a kernel whose
|
||||
`hid-nintendo` driver reports the IMU (Android common kernel `android14-6.1` or newer; older 5.x GKI
|
||||
builds expose the buttons only). A Joy-Con without motion sensors stays a game pad.
|
||||
**Controller settings > Joy-Cons as VR controllers** (on by default) turns it off.
|
||||
|
||||
`input/JoyConVr.kt` takes a Joy-Con once SteamVR reports that it is running. It reads the Joy-Con's
|
||||
gyroscope and accelerometer on a thread of its own. A Mahony filter turns them into an orientation,
|
||||
and the gyro bias is learned while the Joy-Con rests. Each IMU report carries three samples every
|
||||
10 ms. On the tested kernel they arrive newest first and about four in ten reports repeat the
|
||||
previous report's newest sample, while the kernel's timestamps run about twice as fast in an uneven
|
||||
3.3/3.3/13.3 ms pattern. `JoyConReports` collects a report's samples, learns their order from the
|
||||
repeats, drops the repeated one and times the rest from the report's arrival (report interval divided
|
||||
by new samples). That cut the gyro's error from 13 to 48 degrees per movement to about 2 to 3.
|
||||
|
||||
The accelerometer also measures movement. Linux removes the gyro's offset but not the
|
||||
accelerometer's (one Joy-Con read 10.08 m/s² lying still, the other 9.51), so `JoyConAccelCalibration`
|
||||
learns offset and scale by fitting a sphere to the readings of a Joy-Con resting in at least five
|
||||
different positions; it happens on its own while you play and is saved per Joy-Con. `JoyConMotion`
|
||||
takes gravity away in the world frame, integrates what is left into a velocity and a displacement,
|
||||
sets the velocity to zero whenever the Joy-Con is still, and leaks it away quickly once a movement
|
||||
lasts longer than a second, so the displacement holds for short lifts, pushes and punches and cannot
|
||||
run away. It also keeps a second velocity that is never zeroed and only leaks over 30 s, the plain
|
||||
integral of the acceleration, for camera tracking. The pose, displacement, both velocities, buttons,
|
||||
stick and battery go to a 224-byte seqlocked file, `joycon_shm.h`, at
|
||||
`/run/droiddeck/droiddeck-joycon`. The game pad no longer sees that Joy-Con;
|
||||
it gets it back when SteamVR stops, the Joy-Con disconnects or the setting is turned off.
|
||||
|
||||
The driver adds a left and a right Index controller (`knuckles`, Valve's Index input profile and
|
||||
models) when the first Joy-Con appears. The pad hand on that side switches off. A Joy-Con cannot
|
||||
measure where it is, only how it turns and how it speeds up. Without the camera, Google VR's arm model
|
||||
places the hand from its orientation and the head, and the measured displacement moves the hand on
|
||||
top of that before it settles back over 0.6 s (1.5 s while the trigger or grip is held, at most
|
||||
35 cm away), so a quick lift or punch shows but a slow sideways drift does not. With camera hand
|
||||
tracking (below) the camera places the hand instead. Its pointing direction is the Index controller's
|
||||
aim pose. Its heading is matched to the head when it connects. Holding Home (right) or Capture
|
||||
(left) for 0.8 s matches it again, with a short buzz; point the Joy-Con where you look first. A short
|
||||
press is the system button. A hand that sends nothing for 500 ms disconnects.
|
||||
|
||||
| Index | Left Joy-Con | Right Joy-Con |
|
||||
|--------------|------------------|----------------|
|
||||
| Trigger | ZL | ZR |
|
||||
| Grip | L, SL, SR | R, SL, SR |
|
||||
| A | down, right | A, B |
|
||||
| B | up, left, minus | X, Y, plus |
|
||||
| Thumbstick | stick, click | stick, click |
|
||||
| System | Capture (press) | Home (press) |
|
||||
|
||||
The fingers follow the buttons on a simulated hand: the index finger curls with the trigger, the
|
||||
other three with the grip, the thumb when it is on the stick or a face button. SteamVR's haptic
|
||||
events go to that hand's Joy-Con through `@droiddeck-rumble` (slots `0x100` and `0x101`) and respect
|
||||
the rumble setting.
|
||||
|
||||
Outside SteamVR two Joy-Cons are combined into one player: left stick, D-pad, L, ZL, minus and Capture
|
||||
from the left one, the face buttons, right stick, R, ZR, plus and Home from the right. Face buttons
|
||||
keep their positions (Nintendo X is the top button, Y on Xbox). SL and SR are shoulder buttons on a lone
|
||||
Joy-Con and are ignored while two are combined. A combined pair splits rumble the way a pad does,
|
||||
strong to the left and weak to the right. A lone Joy-Con plays the stronger of the two.
|
||||
|
||||
#### Camera hand tracking
|
||||
|
||||
The device's camera can watch the player so each Joy-Con hand sits where the real hand is, including
|
||||
lifting, lowering, reaching and moving sideways. With no controller connected it tracks bare hands
|
||||
instead (below). It needs Android 12 or later. The first time a headset connects, DroidDeck asks whether to use the camera. On a device
|
||||
with cameras on both sides it offers the back one (usually sharper and wider) or the front one; with
|
||||
cameras on one side only it uses those without asking. Android's camera permission is requested
|
||||
right after. **Headset sheet > Head tracking > Hand tracking** turns it on or off, switches the
|
||||
camera, asks for the permission again and opens the placement check.
|
||||
|
||||
When camera tracking is on, **Play SteamVR** first shows the placement check: the live picture with
|
||||
the detected body drawn on it, what to fix, and **Skip**. It reads the device's gravity sensor and
|
||||
asks to stand a device up when it lies flat face up or face down, or to stand it more upright when
|
||||
the camera points more than 45 degrees up or down. Then it asks to step into view, to step back or
|
||||
come closer (the shoulders should take 12 to 26 percent of the picture's width, about 0.75 to 1.6 m
|
||||
from a 90 degree camera), to step back when an elbow or wrist is near the edge of the picture, and
|
||||
to hold both hands in view. Once
|
||||
everything has been fine for 1.2 s SteamVR starts on its own. With the back camera the screen faces
|
||||
away from the player, so the advice is also spoken aloud.
|
||||
|
||||
`handtracking/HandCamera.kt` runs while SteamVR runs and either a Joy-Con is taken or no game
|
||||
controller is connected at all. It picks the widest
|
||||
camera on the chosen side (up to 100 degrees, zooming a logical camera out when it can), holds it at
|
||||
its fastest fixed frame rate up to 30 per second so exposures stay short and moving hands do not
|
||||
blur, takes 960x720 frames from CameraX, turns them upright by gravity, and runs MediaPipe's Pose
|
||||
Landmarker (Lite) on the CPU in video mode, which follows the body from frame to frame and smooths it,
|
||||
15 times a second (10 and then 6 when the device reports it is getting hot). The pose model's depth
|
||||
cannot tell a hand at the chest from one reaching out (it read about 25 cm in front of the shoulders
|
||||
for both), so each hand is measured on its own: `HandModel` runs MediaPipe's Hand Landmarker on a
|
||||
square around the hand. To find a hand it looks for up to two hands in a square around the pose's
|
||||
wrist and keeps the one whose wrist is nearest, taking turns between a square 4 shoulder widths across
|
||||
centred on the wrist and one 2.5 shoulder widths across moved toward the fingers. The palm detector
|
||||
misses many hands in a tight square: with the player about 0.65 m away the 2.5-wide square found 73
|
||||
percent of the hands in view (a 1.4-wide one 49, the whole picture 68), and with the player lying close
|
||||
to the camera, where hands look large, the wide square found 77 percent of the hands the tracker had
|
||||
seen and the two in turn 82 (the 2.5-wide one alone 54). After that it follows the hand in a square 2.8
|
||||
hands across, and finds it again in the same frame when it is lost. A hand whose wrist is nearer the other arm's wrist than its own is dropped.
|
||||
The hand's 21 landmarks come in metres and in pixels, so a least-squares fit gives the hand's
|
||||
distance from the size it appears (`BodyPose.handFix`); the middle of the palm is the point used.
|
||||
The head is fitted the same way from the pose model's face landmarks. Each hand is given relative to
|
||||
the eyes in a frame levelled by gravity, along with the face's heading from the nose and ears. When
|
||||
the hand model misses a hand the pose still sees, the hand keeps its last distance for half a second
|
||||
and is marked as estimated. The frame's exposure time, moved onto the monotonic clock, the hands, each hand-model hand's 21
|
||||
landmarks relative to the middle of its palm (in metres, in the same levelled frame), the face heading
|
||||
and whether bare hands are wanted go to a 568-byte seqlocked file, `handcam_shm.h`, at
|
||||
`/run/droiddeck/droiddeck-handcam`.
|
||||
No picture is kept or sent.
|
||||
|
||||
The driver turns the camera's frame to SteamVR's by comparing the face's heading with the headset's
|
||||
heading at the same moment, averaged over about 3 s and restarted when they disagree for half a
|
||||
second (after a recenter). Each hand then runs a Kalman filter (`HandKalman` in `hand_motion.h`) with
|
||||
position and velocity on each axis. The Joy-Con's plain velocity integral moves it forward between
|
||||
camera frames; its reports are timed 45 ms earlier than they arrive, which lines their movement up
|
||||
with the camera's. The Joy-Con is trusted less while it moves hard, since a small misalignment then
|
||||
turns into a large error. The filter's speed fades with a 0.3 s time constant (0.1 s once the camera
|
||||
has not seen the hand for 0.2 s, since a small tilt error then pushes the hand steadily), and while the Joy-Con
|
||||
has been still for 0.1 s (turning under 0.3 rad/s and accelerating under 0.6 m/s²) it is told the hand
|
||||
is not moving, so a hand the camera loses settles where it was instead of sliding away, and a resting
|
||||
hand does not shake (on a recording, 3.6 to 4.3 cm of shake fell to 1.0 to 1.5 cm). A camera sighting
|
||||
is applied at the moment the picture was taken, 60 to 140 ms in the past, and the Joy-Con movement
|
||||
since then is replayed on top, so the hand is where it is now, not where it was. A hand-model sighting
|
||||
counts as about 3 cm of camera noise, an estimated one as 6 cm.
|
||||
One sighting far from the filter (the model swapping left and right) is ignored unless three in a row
|
||||
agree. When the camera has not seen a hand for 0.75 s, the arm model takes over from where the hand
|
||||
was, and changes between the two blend over a quarter of a second.
|
||||
|
||||
#### Bare hands
|
||||
|
||||
With camera tracking on and no game controller or Joy-Con connected (checked again whenever an input
|
||||
device comes or goes), the app sets a flag in the camera file and the driver gives SteamVR two hands
|
||||
from the camera alone, the way a Quest's hand tracking does; the pad hands switch off. Each hand is
|
||||
still an Index controller to SteamVR, so games' Index bindings and the dashboard work, but its render
|
||||
model becomes a white glove with a light grey band at the wrist, like the plain hands a Quest draws
|
||||
(`{droiddeck}droiddeck_hand_left` and `_right`, built at build time by `tools/vr/make_hand_models.cpp`
|
||||
from the same skeleton). The glove is a smooth surface: tapered capsules along each finger bone, a
|
||||
domed palm and the band are blended into one distance field and meshed with surface nets (a 2.6 mm
|
||||
grid, up to about 10,000 vertices a part). SteamVR draws render models almost unlit, so the shading
|
||||
is baked in: ambient occlusion sampled from the distance field, and a soft light on the back of the
|
||||
hand, pick a column of a 64x64 texture whose rows run from the glove's white to the band's grey. It
|
||||
has two parts with two shapes each: the palm with the middle, ring and little fingers (open, or
|
||||
curled into a fist while `/input/grip/touch` is on, at half closed), and the thumb with the index
|
||||
finger (open, or pinching while `/input/trigger/click` is on). The thumb part carries a copy of the
|
||||
palm 0.3 mm inside the other part's, so the two blend without a seam. SteamVR only routes a part to
|
||||
its input when the part also names a legacy button (`controller_button`, as the Index model's buttons
|
||||
do; vrserver then logs `inputRouting=1` for it), and a hand whose laser points at the dashboard sends
|
||||
its trigger to the dashboard, so that hand clicks without drawing the pinch. Its pointer and grip poses are the Index
|
||||
controller's, so the laser leaves along the fingers. Games that draw hands from the
|
||||
skeleton get the camera's finger curls. Joy-Cons connected later take over; their hands go back to the
|
||||
Index model.
|
||||
|
||||
| Gesture | SteamVR input |
|
||||
| --- | --- |
|
||||
| Thumb and index finger together | Trigger: it starts pulling 6 cm apart, clicks under 3.5 cm and lets go over 4.5 cm |
|
||||
| Middle, ring and little fingers closed | Grip, clicking when fully closed |
|
||||
| Thumb to middle finger | B |
|
||||
| Thumb to ring finger | A |
|
||||
| Right palm toward the face, then pinch | System (the SteamVR dashboard) |
|
||||
| Left palm toward the face, then pinch | B (most games' menu) |
|
||||
|
||||
A gesture has to hold for 50 ms to press and be gone for 100 ms to release, and a palm has to face the
|
||||
headset for 0.3 s before its pinch becomes the menu. A and B only count with the hand open, take the
|
||||
fingertips within 2.5 cm (released over 3.5 cm), and while the palm faces the headset the trigger stays
|
||||
released. The distances were checked on 504 photos from the public HaGRID gesture set (50 each of
|
||||
eleven gestures) through the same hand model: MediaPipe puts touching thumb and index tips a median
|
||||
3.4 cm apart, so a 2.5 cm pinch caught only 27 percent of the "ok" signs; at 3.5 cm it catches 51
|
||||
percent while open hands, pointing, "stop", "four", "three" and thumbs-up misfire in 4 percent or
|
||||
fewer. A fist pinches in a quarter of them, which games read as grabbing with the trigger held. The
|
||||
grip read closed for 92 percent of fists and 96 percent of pointing hands and for none of the open
|
||||
hands.
|
||||
|
||||
`CameraHand` in `hand_pose.h` does the work. Each camera frame's landmarks are turned by the camera's
|
||||
yaw. The finger curls come from each finger's joint angles, looked up in a table of the skeleton's own
|
||||
angles. The skeleton's palm at those curls (wrist and four knuckles) is fitted to the camera's with
|
||||
Horn's closed-form quaternion method, and the root is placed so the skeleton's palm lands on the palm
|
||||
position. Only sightings with hand-model landmarks count; the pose-only estimates are left out. The
|
||||
position goes through a 1€ filter (Casiez, Roussel and Vogel, 2012): a low-pass at 1 Hz while the
|
||||
hand is still that opens up the further and faster the hand gets ahead of it. On a recording of bare hands it cut
|
||||
the wobble of a still hand from the 9 to 11 mm the Joy-Con Kalman filter left to 2.5 to 3 mm, and the
|
||||
largest jump between frames from 10 to 17 cm to about 1.3 cm. The turn goes through the same kind of
|
||||
filter (1 Hz when still, 0.2 Hz more per radian per second), and the curls follow within 80 ms. A fit
|
||||
more than 2 cm off is skipped, and a fit that turns the hand more than 45 degrees at once is held back
|
||||
until the next one agrees within 25 degrees (the hand model now and then flips a hand around; the
|
||||
largest turn between frames fell from 14 to 7.5 degrees). When the camera has not seen a hand's landmarks for half a second,
|
||||
the hand reports itself out of range and lets go of every input; it reappears wherever it is seen
|
||||
next. There is no stick or trackpad, so moving around needs a game's own hand tracking or pointing
|
||||
to teleport.
|
||||
|
||||
On MediaPipe's own example pictures (a raised index finger, a thumbs-up) the right-hand skeleton fits
|
||||
the palm within 7 mm, the curls read as the pose looks, and the palm direction points at the camera
|
||||
when the palm faces it.
|
||||
|
||||
Depth from one camera is approximate: the model expects one person, up to about 4 m away, with the
|
||||
head in view.
|
||||
|
||||
Joy-Con orientation and buttons were tested on a REDMAGIC tablet, and the movement code was tuned on
|
||||
recordings from its Joy-Cons. Camera tracking was recorded on the same tablet with the front camera:
|
||||
the camera holding 30 frames per second and video mode took the hands' frame-to-frame jitter from
|
||||
about 5 cm to 1 to 2 cm; the hand model and the filter were then tuned on that recording. For a
|
||||
diagnosis, an empty file named `handcam-record` in the app's files folder makes the next sessions save
|
||||
up to 6 minutes of small camera pictures with every landmark and timing to the app's cache
|
||||
(`handcam-rec/`), and the driver writes its camera, Joy-Con and filter values to
|
||||
`/run/droiddeck/droiddeck-handtrace.csv` while they run. `tools/vr/build.sh` runs `test_joycon` (arm
|
||||
model, aim pose, hand skeleton, movement, the filter's latency, noise and outlier handling, yaw,
|
||||
both file layouts, a lost hand settling, the palm fit, curls, gestures, bare camera hands, the turn
|
||||
filter and the hand render model). `JoyConTest`,
|
||||
`JoyConOrientationTest`, `JoyConMotionTest`, `JoyConShmTest`, `JoyConPadTest`, `RumbleComponentTest`,
|
||||
`BodyPoseTest`, `TrackingCamerasTest`, `PlacementAdviceTest`, `HandCameraShmTest`, `HandTrackingTest`
|
||||
and `HandCameraPromptTest` cover the app side.
|
||||
|
||||
### Device-loss recovery
|
||||
|
||||
During a SteamVR session, DroidDeck reads new `vrcompositor.txt` output every two seconds and detects
|
||||
`VK_ERROR_DEVICE_LOST` or `vkerror=-4`. Existing errors at session start, ordinary acquire timeouts,
|
||||
and unrelated failures do not trigger recovery. Reads are bounded to the newest 128 KiB and handle
|
||||
log replacement, truncation, and partial lines.
|
||||
|
||||
A detected fault shows an eight-second cancellable countdown while the session is foreground.
|
||||
Touches outside the prompt do not dismiss it; the user can explicitly choose Keep open.
|
||||
Recovery stops the guest, gives teardown and diagnostic collection up to 15 seconds, then uses a
|
||||
private `:vr_recovery` activity to kill only DroidDeck's main process and reopen SteamVR. That process
|
||||
does not initialize the glasses SDK or guest runtime. It recreates the native graphics context; an
|
||||
Activity recreation or SteamVR-only restart does not. Running games close and unsaved progress cannot
|
||||
be restored. A persisted ten-minute cooldown disables another automatic attempt, while still
|
||||
offering a manual restart. Session events record detection and recovery requests. This mitigates
|
||||
the freeze; it does not fix the underlying Vulkan device loss.
|
||||
|
||||
Validated on the phone with a clearly marked `SIMULATED_RECOVERY_TEST: VK_ERROR_DEVICE_LOST`
|
||||
log entry in session 58. Detection and the eight-second countdown led to a new main-process PID
|
||||
and session 59 automatically reached READY and requested SteamVR. The
|
||||
cooldown timestamp survived the process restart. This was a recovery-flow test, not a new GPU
|
||||
failure. Subsequent user feedback and `vrserver.txt` showed that SteamVR had blocked `driver_droiddeck`
|
||||
after a previous crash: the server ran, but reported `HmdNotFound` and never produced VR output.
|
||||
Recovery now clears only `driver_droiddeck.blocked_by_safe_mode`, atomically after stopping the old
|
||||
process and before launching the new session. It preserves global safe-mode preferences and other
|
||||
drivers' blocks. A running server or resumed sensor stream alone is not proof of successful recovery.
|
||||
The follow-up marked test in session 63 exercised the manual retry during the cooldown. Recovery
|
||||
logged the driver unblock, restarted into session 64, and SteamVR recognized the DroidDeck HMD.
|
||||
Fresh compositor output and a 3840x1080 `DroidDeck Cardboard VR` presenter appeared, with subsequent
|
||||
display intervals at 56–59 fps. The user confirmed the GT Max VR view and tracking work after this
|
||||
recovery. The 19 SteamVR unit tests and APK runtime-input verification passed.
|
||||
The glasses/SteamVR suite includes stale/rotated/split log handling, countdown,
|
||||
cancellation, foreground-only behavior, and cooldown checks.
|
||||
|
||||
Stereo also requires a physical side-by-side display mode. On the tested Astra 2
|
||||
firmware the Pro XR command succeeds but DisplayPort falls back to 640×480;
|
||||
direct-connection stereo remains unresolved. DroidDeck remembers such a fallback for that glasses
|
||||
model and stops asking for 3D on its own; **Retry** in the glasses settings tries again. See the
|
||||
[Astra/VITURE investigation](development/astra-viture-stereo.md) for measured
|
||||
results, app recovery behavior, and the unverified powered-adapter design.
|
||||
|
||||
### Not done yet
|
||||
|
||||
- **Controllers**: the driver adds two Vive-profile hands driven by the session's game pad. It reads
|
||||
the absolute-state snapshot the app already publishes in fake input ring 0
|
||||
(`FAKE_EVDEV_MEMFD_PATHS`), so the app needs no new plumbing. They are a gaze pointer: low in front
|
||||
of each hand, pointing where the head looks. Mapping, right hand: right stick = trackpad, A/R3 =
|
||||
trackpad click, RT = trigger, RB = grip, Start = menu, guide = system. Left hand: left stick, X/L3,
|
||||
LT, LB, Select. Both hands activate in SteamVR and their laser shows on the dashboard; guide, Start and
|
||||
A presses show up in the ring the driver reads.
|
||||
(`FAKE_EVDEV_MEMFD_PATHS`), so the app needs no new plumbing. They rest low in front of each side and
|
||||
point where the head looks. Each stick moves its hand sideways (40 cm), up (50 cm) and down (30 cm)
|
||||
and turns it as if from an elbow 45 cm behind, so pointers and aim follow it too; the hand eases
|
||||
back when the stick is let go. The d-pad's up and down move both hands in and out; while a hand's
|
||||
trigger or grip is held they move only that hand (a slingshot or bow pulled back), and it eases back
|
||||
on release. Clicking the left stick hands the sticks to the game as trackpads (walking, turning,
|
||||
teleport) and back. Mapping, right hand: A/R3 = trackpad click, RT = trigger, RB = grip, Start = menu, guide =
|
||||
system. Left hand: X, LT, LB, Select. Both hands activate in SteamVR and their laser shows on the
|
||||
dashboard; guide, Start and A presses show up in the ring the driver reads. They are drawn as Index
|
||||
controllers and switch off while another driver's controller is connected. A Joy-Con hand replaces
|
||||
the pad hand on its side, and bare camera hands replace both while no controller is connected
|
||||
(above).
|
||||
- **Layers**: only the first layer of each frame is drawn. SteamVR's compositor composites the
|
||||
dashboard into it, so this has not mattered yet.
|
||||
- **Timing**: no reprojection, and the presenter blits whatever was submitted last, a guard delay
|
||||
after it arrived. The chain is SteamVR → presenter → DroidDeck compositor → lens pass.
|
||||
- **Timing**: no reprojection. The presenter blits the frame submitted last once `vrcompositor`'s GPU
|
||||
work for it is done: the layer fences that process's submits and counts issued and finished ones in
|
||||
`present-sync` beside the `active` marker, and the presenter sleeps on the count (a futex). Without
|
||||
the file it falls back to a guard delay after the frame arrived (`DROIDDECK_VR_PRESENT_GUARD_MS`,
|
||||
4 ms). It hands its own picture to Wayland only once its blit is done, because KGSL does not order
|
||||
the DroidDeck compositor's copy after it. The chain is SteamVR → presenter → DroidDeck compositor →
|
||||
lens pass.
|
||||
- **Cross-process GPU sync** is a poll away from exact (see the semaphore note above); binary
|
||||
semaphores are not bridged.
|
||||
- **Apps and games**: Windows VR games run through Proton ARM64 (FEX for the x86-64 game, DXVK, and
|
||||
|
||||
@@ -47,7 +47,7 @@ shortcut appid is stored per folder at the first scan and never changes after.
|
||||
|
||||
The ✎ on a Custom game's page (`AddedGameEditor`) mirrors Steam's Properties for a
|
||||
non-Steam game: Name, Target (ranked exes, "?", Other exe… opening the picker in the
|
||||
game's folder), Start in (automatic = the exe's folder), Launch options, Artwork (Cover,
|
||||
game's folder), Start in (automatic = the exe's folder), Launch options, Artwork (Cover, Wide cover,
|
||||
Background, Logo, Icon, each with its thumbnail and source), and Remove (two presses; the
|
||||
files stay; adding it again brings it back). `AddedGameEdits` writes each change through
|
||||
the same shortcut: the listing (`name`, `exe`, `dir`, `launch`, `art`, the stored `appid`)
|
||||
@@ -68,7 +68,7 @@ store search whose result has the same name), then SteamGridDB, then the exe's i
|
||||
art chooser (an Artwork row) shows the game folder's images, Steam's official piece and
|
||||
SteamGridDB's best few (fetched when it opens); a tap applies one, "Search SteamGridDB…"
|
||||
opens a search (it starts with the game's name and opens the match at once when one is
|
||||
exact; "Show more" pages through the images), "Pick image…" takes a file through the same
|
||||
exact; it lists every image SteamGridDB returns for the slot, up to four pages of 50), "Pick image…" takes a file through the same
|
||||
picker (png, jpg, jpeg, webp, ico), "Reset to automatic" drops the choice. A choice is a
|
||||
copy in `files/added-art/<appid>/chosen-<slot>.<ext>`, its source in the preferences.
|
||||
|
||||
@@ -80,6 +80,34 @@ a failed request is never written as "none", a "none" older than the saved key i
|
||||
again, and the chooser always asks again when the user opens it. A background pass stops
|
||||
asking SteamGridDB after its first failure.
|
||||
|
||||
Steam games (installed through Steam, not Custom) get the same ✎ on their page: an Artwork-only
|
||||
editor (`SteamGameArtEditor`) with the same rows and the same chooser (Steam's official piece,
|
||||
SteamGridDB's best few, "Search SteamGridDB…", "Pick image…", "Reset to automatic"), shared
|
||||
through `ArtTarget`. A choice is a copy in `files/steam-art/<appid>/chosen-<slot>.<ext>`
|
||||
(`SteamGameArt`); `Library.steamGames` shows it in place of the librarycache image, and with a
|
||||
client running the Cover / Wide cover / Background / Logo is also set in Steam
|
||||
(`SetCustomArtworkForApp` / `ClearCustomArtworkForApp`). Icon is launcher-only.
|
||||
|
||||
`SteamGrid` keeps both sides in step through the client's grid folder
|
||||
(`userdata/<account>/config/grid`, `<appid>p`, `<appid>` for the wide cover, `_hero`, `_logo`), for Steam and Custom games alike.
|
||||
A choice is written there at once, so it reaches Steam while the client is closed. Every library
|
||||
scan (`Library.launchableGames`) compares the folder with what was last written or read
|
||||
(`grid-<slot>` in the game's art folder): a file Steam changed becomes the choice (source Steam),
|
||||
a file removed in Steam drops it, and files the shortcuts writer placed itself
|
||||
(`.droiddeck-art.json`) are not mistaken for choices. A chosen image is stored as
|
||||
`chosen-<slot>-<time>.<ext>`, so a new choice has a new path and shows at once. Installed GOG, Epic and Amazon games get the same rows through `StoreGameArtEditor`.
|
||||
|
||||
"Adjust logo" (last row of every art editor, `LogoPositionDialog`) shows the game's Background with
|
||||
its Logo on top and lets the logo be pinned to one of Steam's five places (top left / top center /
|
||||
center / bottom left / bottom center) and resized (10-100 %). The value is Steam's own
|
||||
`{"nVersion":1,"logoPosition":{"pinnedPosition","nWidthPct","nHeightPct"}}` (`LogoPosition`). It is
|
||||
kept in the game's art folder (`logo-position`), written to `<appid>.json` in the grid folder (other
|
||||
keys in that file are kept; clearing the position removes the file only when nothing else is in it),
|
||||
and, with the client running, set through `SetCustomLogoPositionForApp` /
|
||||
`ClearCustomLogoPositionForApp`. `SteamGrid.sync` takes over a position changed or removed in Steam
|
||||
the same way it does for images (`grid-json` holds the last seen signature). Reset removes the
|
||||
custom position so Steam's default comes back.
|
||||
|
||||
Setup › Launcher › Games holds the Artwork switch (automatic fetching) and the
|
||||
SteamGridDB API key: the user's own, sealed with `CredentialCipher`, wins over the build's
|
||||
(`BuildConfig.SGDB_API_KEY` from the `SGDB_API_KEY` secret); with neither, SteamGridDB is
|
||||
@@ -89,5 +117,5 @@ skipped.
|
||||
|
||||
`AddedExesTest`, `AddedGameEditsTest` (bulk add and skip, edits to the listing on a stable
|
||||
appid, Steam-side read-back, art choice and reset), `AddedGamesCandidatesTest`,
|
||||
`AddedGameArtLookupTest` (key precedence), `GameIdentifierTest`, `PickListingTest`, and
|
||||
`AddedGameArtLookupTest` (key precedence), `SteamGameArtTest` (Steam game art choice and reset), `SteamGridTest` (grid folder both ways), `GameIdentifierTest`, `PickListingTest`, and
|
||||
`tools/tests/test_steam_shortcuts.py` for the writer.
|
||||
@@ -0,0 +1,130 @@
|
||||
# Astra 2 and VITURE stereo
|
||||
|
||||
## Tested connection
|
||||
|
||||
2026-10-14: REDMAGIC Astra 2, model `NP06J`, Android 16,
|
||||
`RedMagicOS11.5.16_NP06J_GB`, Snapdragon `SM8850P`; VITURE Pro XR
|
||||
(`35ca:101d`), firmware `07.0.03.013_20250422`. The tablet reports a locked
|
||||
bootloader and green verified boot. Tests used the ordinary ADB shell, without root.
|
||||
|
||||
2D negotiates **1920×1080 at 120 Hz**. The glasses accept stereo command
|
||||
`0x0008 [32]` with reply `00 32`, but Android then exposes **only 640×480 at
|
||||
60 Hz**. An accepted USB command is not proof of a working stereo signal.
|
||||
In this state the glasses reject a 2D command (`01 31` or `01 34`) and their
|
||||
mode query returns `01 00`. A physical cable reconnect restored 1920×1080/120
|
||||
with this update; Android then requested USB permission again and control resumed.
|
||||
|
||||
On the fresh attempt, `0x030D [03]` arrived 3.1 seconds after the accepted stereo
|
||||
command. The vendor SDK maps that report to 2D/120 (`0x34`); the subsequent mode
|
||||
query was rejected. The report does not encode a fallback reason or establish a
|
||||
fixed timeout. The earlier command acknowledgement cannot prove stereo retention.
|
||||
|
||||
## What was checked
|
||||
|
||||
| Route | Result on this firmware |
|
||||
| --- | --- |
|
||||
| Pro full SBS command `0x32` | Accepted by MCU; physical output falls to 640×480 |
|
||||
| Android preferred mode 3840×1080/60 | Preference saved; physical supported modes remain 640×480 only; preference cleared after test |
|
||||
| USB source/host → source/device | Request issued; actual port stayed source/host |
|
||||
| USB data disable/enable | Data status changed and was restored; physical display remained 640×480 |
|
||||
| `IUsbManager.resetUsbPort` through ADB | HAL callback reported success (`0`); physical display mode did not change |
|
||||
| External-display power off/reset | Did not recover the signal; glasses still rejected 2D |
|
||||
| Physical cable reconnect | Restored 1920×1080/120 and USB controls after the permission prompt |
|
||||
| OEM mirror/3D controls, source review | Change virtual-display routing, not physical stereo timings |
|
||||
| OEM resolution selection, source review | Chooses cached resolution candidates and prefers stale 16:9 mode |
|
||||
| Pro startup mode / half SBS, SDK review | No supported Pro startup-mode setting or half-SBS input |
|
||||
|
||||
Public driver decompilations from the related REDMAGIC 11 Pro (`NX809J`, SM8850)
|
||||
[retain an active mode override](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube/blob/efee6f411184886b0fbdaf63fc9db36b3ed96036/decompiled/msm_drm/dp_display_get_modes.c#L37-L44)
|
||||
across a glasses hotplug. Their [USB-PD disconnect path](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube/blob/efee6f411184886b0fbdaf63fc9db36b3ed96036/decompiled/msm_drm/dp_display_usbpd_disconnect_cb.c#L127-L147)
|
||||
clears that override; their simulated hotplug path does not. This is consistent
|
||||
with the Astra's measured behavior, but does not prove its exact kernel
|
||||
implementation. That driver also [validates saved user modes against EDID](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube/blob/efee6f411184886b0fbdaf63fc9db36b3ed96036/decompiled/msm_drm/nubia_edid_modes.c#L226-L250),
|
||||
so active overrides and saved preferences must not be treated as the same
|
||||
mechanism. The independent [MagicDesk investigation](https://github.com/mekhontsev/magicdesk/blob/main/docs/xr-resolution-fix.md)
|
||||
documents a related failure on its separately identified REDMAGIC firmware.
|
||||
|
||||
The [VITURE SDK matrix](https://www.viture.com/en-SG/developer/glasses-sdk/glasses)
|
||||
specifies full SBS at 3840×1080/60 for Pro. Persistent startup mode is restricted
|
||||
to Luma/Luma Pro. Among the Luma families supported here, 90 Hz stereo is an
|
||||
Ultra capability. The connected Pro firmware matches both the N6PL entry in the
|
||||
[public updater catalog](https://static.viture.com/dfu-util/js/vers.js) and the newest
|
||||
N6P entry in the [public release-note service](https://cloud.viture.dev/api/v1/system/releasenote)
|
||||
checked on 2026-10-14. This does not rule out unpublished, device-specific updates.
|
||||
|
||||
## Powered dock results
|
||||
|
||||
A VITURE Pro Mobile Dock (`35ca:3011`) was connected between the Astra and Pro XR.
|
||||
Its official read-only P9 version query returned `2.01.03.0.03` with a valid
|
||||
checksum. The Astra negotiated 1920×1080/120, but enumerated only the dock's
|
||||
USB interfaces, not the glasses' `35ca:101d` control/IMU interfaces. A one-second
|
||||
DroidDeck IMU sample returned zero packets.
|
||||
|
||||
DroidDeck treats the glasses as a display behind the dock: no USB permission
|
||||
prompt, video at 1080p/120 on the glasses, no Play SteamVR, refresh, 3D or tracking
|
||||
settings. The dock alone, without glasses, does not count as glasses.
|
||||
|
||||
VITURE's updater calls this dock P9 (app `35ca:3011`, bootloader `35ca:3010`). Its
|
||||
HID messages cover dock versions, battery, inputs and LEDs; none sets a display
|
||||
mode or forwards glasses commands or IMU data. `2.01.03.0.03` is public release
|
||||
`1.03.0.03_20250709`; the latest, `1.03.0.06_20260409`, adds Beast support and
|
||||
updater fixes only. The updater also reads a Lontium LT8668 in the dock, a
|
||||
DisplayPort receiver with its own EDID handling, so the tablet most likely sees the
|
||||
dock as its DisplayPort sink rather than the glasses.
|
||||
|
||||
[VITURE's Pro Dock specifications](https://www.viture.com/product/viture-pro-mobile-dock)
|
||||
confirm 3840×1080/60 video support and exclude phone features requiring
|
||||
communication with the glasses. The vendor SDK also treats the dock as a separate
|
||||
accessory, not an IMU device. This dock may test the video path, but it does not
|
||||
provide DroidDeck's current head-tracking path. Stereo negotiation and mode
|
||||
retention during an upstream-only reconnect still require a physical test.
|
||||
|
||||
## No-root hardware design to test
|
||||
|
||||
A powered pass-through connection may let the tablet see the stereo EDID on a
|
||||
fresh USB-PD connection while the glasses retain their stereo mode:
|
||||
|
||||
1. Power the adapter and glasses independently of the tablet connection.
|
||||
2. Enable stereo on the glasses.
|
||||
3. Disconnect only the tablet's upstream USB-C connection promptly after the
|
||||
stereo acknowledgement, keeping the glasses powered. Compare with a delayed
|
||||
disconnect to test whether the observed mode report affects retention.
|
||||
4. Reconnect the tablet and check its physical supported modes before starting a game.
|
||||
|
||||
This requires an adapter that preserves glasses power and mode during the
|
||||
upstream disconnect, passes 3840×1080/60 video, and forwards USB HID for controls
|
||||
and head tracking. [VITURE's Adapter Ultra documentation](https://www.viture.com/academy/adapters/usb-c-xr-charging-adapter-ultra)
|
||||
explicitly excludes wired glasses data on Android, so it does not satisfy
|
||||
DroidDeck's current USB control/tracking path. Its power and video specifications
|
||||
also do not promise mode retention across upstream disconnect. This design
|
||||
remains **unverified**, not a purchasing recommendation. First verifying stereo
|
||||
retention through the adapter on a compatible host would provide a useful control.
|
||||
An intermediary presenting the stereo EDID before first host
|
||||
connection is another possible design; it must pass the actual stereo timing
|
||||
and preserve HID access, not merely advertise a wider desktop.
|
||||
|
||||
VITURE advertises full Android functionality for its separate
|
||||
[USB-C to Glasses and Charging Adapter](https://www.viture.com/en-MY/product/usb-c-to-glasses-and-charging-adapter).
|
||||
That makes it a candidate for the USB-data experiment, not a confirmed fix.
|
||||
First check USB enumeration, brightness controls and live IMU data through the
|
||||
adapter; then test power/mode retention and the stereo timing.
|
||||
|
||||
Success requires Android to expose and select 3840×1080/60, different left/right
|
||||
eye images to reach the lenses, live head tracking, and repeatable return to 2D.
|
||||
A wide screenshot or `wm size` override alone does not meet those conditions.
|
||||
|
||||
## App safeguards
|
||||
|
||||
DroidDeck waits up to 20 seconds for physical stereo confirmation. When the
|
||||
attempt ends at a low-resolution fallback, DroidDeck remembers it for that glasses
|
||||
model on this device, so Play SteamVR stops requesting 3D after a restart; **Retry** in the
|
||||
glasses settings clears it. A display that stays missing, such as an unanswered
|
||||
cast prompt, fails only that attempt. Automatic retries do not loop.
|
||||
USB mode reports are tracked separately from the physical display. When a session
|
||||
enabled 3D, stopping it requests a return to 2D, including during negotiation;
|
||||
this firmware can still require a cable reconnect. Output timing follows
|
||||
the active glasses display rather than the tablet's requested refresh rate.
|
||||
|
||||
Keep hardware experiments separate from game loads. End the session, force-stop
|
||||
DroidDeck, and reopen its home screen after each game test. Read current HAL
|
||||
temperatures rather than cached thermal-service samples.
|
||||
@@ -44,12 +44,16 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- `helloxr` blocked in a wait-idle in `vrclient.so`;
|
||||
- DXVK's `dxvk-queue` thread blocked in Proton games.
|
||||
|
||||
The layer now gives a submit's wait on an imported shared value 3 ms, then drops it. It is kept
|
||||
when the value is already reached, or when this process has queued the signal itself. Two details
|
||||
mattered:
|
||||
The layer now gives a submit's wait on an imported shared value 3 ms, then drops it, at once when
|
||||
the shared counter has not moved for 100 ms. It is kept when the value is already reached, when
|
||||
this process has queued the signal itself, or while a Quest session's frame socket exists (checked
|
||||
at most once a second). Three details mattered:
|
||||
- the waits are judged before the submit's own signals count, because an acquire waits on one
|
||||
value and signals the next on the same semaphore;
|
||||
- SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there.
|
||||
- SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there;
|
||||
- a Proton game waits on two values a frame that are never signalled (5,500 dropped, none reached
|
||||
within 200 ms afterwards). With the grace on each, Angry Birds VR's submit thread lost 6 ms a
|
||||
frame and the game ran at 60 fps; without it, 120.
|
||||
4. **UBWC layouts must agree.** An imported image has to be created with the same
|
||||
`VkImageFormatListCreateInfo` (sRGB + UNORM) as its creator, or Turnip lays out the compression
|
||||
differently and the reader sees garbage. The presenter's eye-texture import does this.
|
||||
@@ -83,10 +87,74 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- **Memory.** Steam, SteamVR and one VR game fit in the Thor's 12 GB. A large game such as The Lab, or
|
||||
two games at once, push the device into critical pressure, and the low-memory killer takes
|
||||
DroidDeck itself (`ended-without-teardown.txt` in the session logs, `lowmemorykiller` in logcat).
|
||||
- **Adreno 840 forced sysmem.** On the Fold with VITURE Luma Ultra, launching SteamVR from the library
|
||||
inherited `TU_DEBUG=sysmem` and produced `vkerror=-4`, triggering recovery. Adding `noubwc` still
|
||||
failed; replacing the flags with `noubwc` or an empty value worked. The shared launch helper removes
|
||||
only `sysmem` on `Adreno840*`, leaving other flags, the parent Steam session and other GPUs unchanged.
|
||||
The user confirmed improved landing-area brightness and colors with this configuration and asked
|
||||
to preserve them.
|
||||
- **Adreno 840 with DD-Turnip v0.2.0.** SteamVR reproduced `vkerror=-4` with `sysmem` removed,
|
||||
including on the previous app build and with the presenter disabled. After a phone reboot,
|
||||
the unchanged `fcf8c588` baseline rendered for 109 seconds without observed device loss.
|
||||
`TU_DEBUG=flushall` initially appeared promising: a global run rendered for over a minute,
|
||||
and a SteamVR-only run rendered for about 32 seconds before background suspension. The
|
||||
packaged SteamVR-only change then reproduced device loss with `flushall` confirmed in
|
||||
`vrcompositor`, so the automatic flag was removed. These bounded runs do not establish the
|
||||
underlying cause or a reliable workaround.
|
||||
- **Live files are not cache.** Android's `installd` evicted `cache/shm/u10528-*` while SteamVR was
|
||||
running, followed by Chromium shared-memory failures and lost IPC. Shared memory, emulated device
|
||||
files, sandbox trees and PRoot temporary files now use `no_backup/guest-runtime`, retaining their
|
||||
existing cleanup lifetimes. Session state must survive cache eviction.
|
||||
- **One host signal owner.** The submit-wait path must not mirror a shared timeline value itself:
|
||||
the poller can signal it between the submit thread's counter read and signal, producing an invalid
|
||||
duplicate signal. The poller now owns mirroring; a forced-interleaving test covers this race.
|
||||
- **Lost hand tracking is not a disconnect.** Publishing a disconnected pose and then a connected,
|
||||
untracked pose on every optical update floods SteamVR with device-change notifications. With
|
||||
uncalibrated hands this grew its web buffer to about 34 MB and triggered the Web thread watchdog.
|
||||
Release inputs and publish only the connected, untracked pose; reserve disconnection for an
|
||||
actual controller removal. The controller regression exercises repeated loss and reacquisition.
|
||||
- **Failed realtime scheduling can still install a CPU limit.** On the Fold, SteamVR's SDL RTKit
|
||||
fallback set `RLIMIT_RTTIME` to 100/200 ms despite realtime priority being refused. The render
|
||||
thread later received kernel `SIGXCPU` during `IOCTL_KGSL_GPU_COMMAND`, with policy `SCHED_OTHER`
|
||||
and the soft limit advanced to 1100 ms, confirming the realtime watchdog fired. The VR scheduling
|
||||
preload rejects lowering this limit only in `vrcompositor` when `RLIMIT_RTPRIO` is zero and
|
||||
`CAP_SYS_NICE` is absent. It preserves inherited limits, other processes and other resources;
|
||||
it does not intercept signals. SDL's scheduling hint cannot cover SteamVR's direct
|
||||
`SDL_LinuxSetThreadPriorityAndPolicy` call. A native preload regression covers rejection and
|
||||
forwarding. The exact Android kernel boost/accounting path was not established.
|
||||
- **Frames shown out of order.** Video from the glasses jumped back a few frames in bursts while the
|
||||
head turned. Two causes, each found by stamping a frame counter into the presenter's output:
|
||||
- the presenter copied SteamVR's eye texture a fixed 4 ms after `Present`, with no fence. Under
|
||||
GPU load `vrcompositor` had not drawn it yet, so the copy got that texture's previous frame. The
|
||||
layer now fences `vrcompositor`'s submits and the presenter waits for the frame's count;
|
||||
- DroidDeck's compositor copies a Wayland client's dma-buf without waiting for its implicit fence,
|
||||
and KGSL does not order the copy after the client's GPU work. The presenter's blit sat behind the
|
||||
game's frame, so the compositor showed the swapchain image's older picture. The presenter now
|
||||
waits for its blit before it presents.
|
||||
- **Flat games under SteamVR.** With SteamVR running, Steam lists a flat game after its own window in
|
||||
gamescope's base layer, so Big Picture keeps the focus and the game never shows, theatre setting or
|
||||
not. SteamVR is therefore started only by Play SteamVR, never with the glasses alone.
|
||||
- **A new projection needs an event.** `SetDisplayProjectionRaw` changed what `vrserver` reports, but
|
||||
`vrcompositor` kept the frustum it started with, so the glasses' field of view did nothing until
|
||||
SteamVR restarted. The driver now follows it with `VREvent_LensDistortionChanged`.
|
||||
- **Head-tracking axes.** With the display at `ROTATION_90`, screen x is minus device y and screen y is
|
||||
device x. Getting this wrong made a turn at the hips tilt the view. Recentring now keeps the world
|
||||
vertical and resets the heading only.
|
||||
|
||||
## Library launch validation (2026-10-15)
|
||||
|
||||
Two hands-off launches through the Steam library's Launch button completed 308 seconds and
|
||||
188 seconds, with an intentional stop between runs. Each retained its compositor process and
|
||||
session, sustained 60 fps, showed dashboard/world transitions, and kept USB control and valid
|
||||
6DoF head tracking. Neither run reported device loss, a watchdog abort, recovery, or a session
|
||||
restart. Hand controllers remained enabled but uncalibrated. Brightness stayed at 8; no shader
|
||||
color changes or temporary graphics overrides were used.
|
||||
|
||||
These timed runs used APK SHA-256
|
||||
`4ce7a623ceb6b1a100c0ed1184530ece2ea8e13a1cb9ee6c0ab24666a8249004`. A newer calibration/recenter
|
||||
APK was subsequently installed and verified; see `luma-spatial-evidence.md`. The timed results
|
||||
do not establish sustained gameplay or calibrated-hand acceptance for that newer build.
|
||||
|
||||
## Debugging a live session
|
||||
|
||||
- **Logs:** `~/.local/share/Steam/logs/{vrserver,vrcompositor,xrclient_*}.txt` and
|
||||
@@ -123,8 +191,9 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- **SteamVR Home** needs `IVROverlayView_002` in Proton's `vrclient`, or an ARM64 build of Home.
|
||||
- **The Lab and other large games** need memory headroom. Candidates: stop Steam's own web helpers
|
||||
while a VR game runs, or lower SteamVR's footprint.
|
||||
- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync. A submit thread in the
|
||||
layer, deferring submits until their waits are reachable, would be exact.
|
||||
- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync, and an idle counter
|
||||
skips it. A submit thread in the layer, deferring submits until their waits are reachable, would be
|
||||
exact.
|
||||
- **Layers:** only the first layer of each frame is presented. No reprojection.
|
||||
- **Controllers** are a gaze pointer driven by the pad. Pad presses did not close the SteamVR menu in
|
||||
testing. Untested: whether that came from gamescope focus being on Steam, or from the system button
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# Tracked DroidDeck menus
|
||||
|
||||
When glasses output and the tracked virtual screen are enabled, DroidDeck mirrors its own foreground menus into a glasses `Presentation`. Phone controls remain on the phone. The menu renderer does not start the guest compositor, request MediaProjection, or capture another application's windows.
|
||||
|
||||
`GlassesMenuCapture` draws the activity's visible view roots into a hardware Canvas at up to 30 fps. `GlassesMenuSurface` consumes that buffer as an external-OES texture and redraws the screen at the glasses refresh rate. The plane geometry follows the existing `cardboard.frag`: row-major head-to-anchor rotation, metres for translation, +Y up, -Z forward, and a top-left source image. Mono uses a central eye; SBS renders the full UI to both eyes with 64 mm eye separation.
|
||||
|
||||
The pose adapter keeps its 3DoF reference across menu pages and uses the shared spatial **screen** origin for 6DoF. Menu recenter does not change the VR floor origin. Stale or invalid tracking holds the last safe view. Explicitly changing between 3DoF and 6DoF can reanchor the screen because those modes have separate origins; recenter restores the chosen mode's view.
|
||||
|
||||
On Android 29 and later, capture includes visible app dialog/popup roots exposed by `WindowInspector`. Android 26–28 captures the activity decor only, so separate dialog and popup windows are not mirrored. System keyboards, system permission prompts, other apps, and separately composited `SurfaceView` content are outside this capture path. Secure windows are excluded. Phone interaction remains available for these cases.
|
||||
|
||||
## Shader regression check
|
||||
|
||||
The opt-in Python check compiles the actual embedded GLES shaders, imports an asymmetric texture through EGLImage/external-OES, renders it, and checks readback pixels. It covers image orientation, both SBS eyes, translation, yaw, and views behind the screen. It does not require Python packages beyond the standard library and is not part of default pytest discovery.
|
||||
|
||||
On Debian/Ubuntu with Python 3:
|
||||
|
||||
```sh
|
||||
sudo apt-get install libegl1 libegl-mesa0 libgles2 libgl1-mesa-dri
|
||||
python3 tools/check_glasses_menu_shader.py
|
||||
```
|
||||
|
||||
Run from any working directory by specifying the script's path. `--output-dir PATH` overrides the default `build/glasses-menu-shader`. The check defaults to headless Mesa software rendering and GLES 2.0. It writes `menu-shader-validation.json` with the shader hashes, GL version, and pixel comparisons, plus `menu-shader-mono.png` and `.ppm`.
|
||||
|
||||
The test supplies the Android consumer's vertical-flip texture transform. It verifies the real shader under that transform; it does not replace checking Android's actual Canvas → SurfaceTexture producer/consumer integration or display orientation on the phone.
|
||||
|
||||
## Phone and glasses smoke test
|
||||
|
||||
1. Connect supported glasses, open DroidDeck's setup page, and enable glasses output and the tracked virtual screen. Confirm the phone remains touchable and text in the glasses is upright with the same left/right ordering.
|
||||
2. Switch among Steam and non-Steam launcher pages without launching a game. Scroll, open menus and app dialogs, and change a setting from the phone. Check that the glasses follow the current page and retain the screen's head-tracked anchor.
|
||||
3. In 3DoF, turn and tilt the glasses, then use Recenter. Test mono and SBS. Each SBS eye must contain the full UI with matching vertical alignment.
|
||||
4. Enable 6DoF on supported hardware and complete its setup. Translate left/right, up/down, and toward/away from the plane. Recenter the menu and verify that VR floor height has not changed. Briefly obscure tracking; the held menu must remain finite and stable until tracking recovers.
|
||||
5. Launch a game, open and close the session menu, then return to the launcher. Confirm clean ownership handoff between the game output and menu presentation and that phone-only operation still works.
|
||||
6. Background/foreground DroidDeck, rotate the phone, change the glasses display mode, and unplug/replug the glasses. Check for stale frames, black output that persists after reopening, crashes, or a reused orientation from the previous device connection.
|
||||
|
||||
For evidence, `tools/droiddeckctl displays` lists logical display IDs and `tools/droiddeckctl screenshot ./glasses.png --display <id>` captures the selected output. Inspect `GlassesMenu` log messages for capture or renderer failures. A display screenshot verifies image content; head movement and stereo comfort still require the physical glasses.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Hybrid hand inference benchmark
|
||||
|
||||
Measured on the connected Samsung SM-F976U on 2026-10-17. This sparse-correspondence
|
||||
prototype did not improve useful hand throughput. It was slower than scheduled stereo
|
||||
on the central open-hand capture and accepted palms only on its dual-detector recovery
|
||||
frames. The earlier estimated 20–60% throughput improvement is not supported by this test.
|
||||
|
||||
Turning image lens correction off did not improve accepted-palm counts. The saved
|
||||
calibration uses the **pinhole** distortion model, not OpenCV's fisheye model. The benchmark did not change its diagnostic build's live algorithm or input-rate policy,
|
||||
and restored the prior settings after each sweep. That build ended with Lite selected.
|
||||
The shipping baseline was subsequently set to Full with raw grayscale-to-RGB input.
|
||||
The failed hybrid prototype and its runners have been removed; these results are historical.
|
||||
|
||||
## Central open-hand result
|
||||
|
||||
The table pools the initial and final central-scene runs in the lens on/off sweep.
|
||||
Each stereo entry combines the baselines before and after the hybrids. Times include
|
||||
per-frame image preparation, inference, matching when applicable, reconstruction, and
|
||||
palm/gesture estimation. Acceptance means at least one geometrically accepted palm in
|
||||
a repeated saved image, not live detection accuracy. This scene contains two open hands;
|
||||
Full consistently recovered only one.
|
||||
|
||||
| Full-model path | Image lens correction | Median / p95 processing | Frames with a palm | Sparse frames with a palm |
|
||||
|---|---|---:|---:|---:|
|
||||
| Scheduled stereo | On | 30.15 / 61.60 ms | 160 / 160 | — |
|
||||
| Left-led hybrid | On | 62.59 / 84.06 ms | 48 / 80 | 0 / 32 |
|
||||
| Right-led hybrid | On | 40.58 / 80.21 ms | 48 / 80 | 0 / 32 |
|
||||
| Scheduled stereo | Off | 30.51 / 61.18 ms | 160 / 160 | — |
|
||||
| Left-led hybrid | Off | 62.57 / 79.82 ms | 48 / 80 | 0 / 32 |
|
||||
| Right-led hybrid | Off | 61.75 / 83.05 ms | 48 / 80 | 0 / 32 |
|
||||
|
||||
All successful hybrid frames came from ordinary stereo inference. Correspondence failed
|
||||
to reconstruct the five required palm anchors; the original palm gates were retained.
|
||||
The first corrected-image trial measured about 19–21 ms for rectification and matching
|
||||
alone. Repeated failures also prevented the hybrid from sustaining the cheaper one-hand
|
||||
tracker schedule. The live input limit should not be raised on the strength of this candidate.
|
||||
|
||||
Lite accepted no palms with either lens setting or either architecture on the lens-sweep
|
||||
captures. The peripheral open hand and representative controller grip also yielded no
|
||||
accepted palms. The no-hand control remained negative. The earlier corrected-image sweep
|
||||
covered all nine saved pairs, including six controller grips, with no extra palms recovered
|
||||
by the hybrid. These are a small set of static scenes, not independent accuracy trials.
|
||||
|
||||
## What was measured
|
||||
|
||||
`HybridHandBenchmark` was reachable only through the saved-capture diagnostic. Both models
|
||||
use CPU VIDEO inference, the existing 0.6 thresholds, and the existing one-/two-hand
|
||||
schedule. Stereo uses concurrent camera inference. The hybrid runs one detector, rectifies
|
||||
both images, and searches for each landmark along its epipolar line using 11 × 11 normalized
|
||||
correlation patches. It requires correlation at least 0.80, a 0.04 uniqueness margin outside
|
||||
three pixels, patch standard deviation at least four, and forward/backward agreement within
|
||||
1.5 pixels. Search depth spans 0.15–1.5 m. Existing triangulation and palm/gesture checks
|
||||
remain in force. This is one concrete hybrid candidate, not a test of every stereo matcher.
|
||||
|
||||
Both detectors run every fifth frame and on the frame after failed sparse palm estimation.
|
||||
The hybrid's recovery costs and failures are included in the results. Each configuration
|
||||
has a separate cold frame, five warmups, and forty measured frames. Synthetic VIDEO
|
||||
timestamps advance by 50 ms, but processing runs unpaced; the experiment did not measure
|
||||
a live 20 Hz stream. Initialization, file decoding, camera transfer, head synchronization,
|
||||
pose publication, and rendering are excluded. Replayed poses are never published.
|
||||
|
||||
Contrast normalization and grayscale-to-RGB conversion stay enabled in both lens cases.
|
||||
With input lens correction off, detection runs on raw images; landmarks are still corrected
|
||||
using calibration for triangulation. The hybrid always uses corrected, rectified images
|
||||
for correspondence and transforms matched points back to the detector's image coordinates.
|
||||
This comparison does not disable the geometric calibration required for metric depth.
|
||||
|
||||
The lens sweep tested the central scene, peripheral scene, no-hand control, one grip scene,
|
||||
and then the central scene again. Full and Lite each ran stereo-before, left-led hybrid,
|
||||
right-led hybrid, and stereo-after for both lens settings: 3,200 measured replays. The
|
||||
separate initial corrected-image corpus sweep also recorded 3,200 replays across nine
|
||||
unique pairs, repeating the central scene last. Repetition is for timing, not extra accuracy
|
||||
samples. SteamVR was idle and live tracking was temporarily disabled during these runs.
|
||||
|
||||
## Validation and limits
|
||||
|
||||
The Android build and 33 focused JVM tests passed. On the phone, the matcher passed fifty
|
||||
synthetic correspondences in both directions and rejected textureless synthetic images
|
||||
before each capture benchmark. These checks establish basic coordinate/search behavior;
|
||||
they do not establish matching accuracy on real hands.
|
||||
|
||||
Android thermal status was 0–1 during the corrected-image sweep and 1 during the lens
|
||||
sweep. Timings varied substantially within the fixed configuration order, so the on/off
|
||||
timing difference cannot be attributed solely to lens correction. Full stereo medians in
|
||||
the first lens-sweep central trial ranged from 16.7 to 30.8 ms; the final trial was slower
|
||||
with correction enabled. Baselines bracket the hybrid cases, but the run is not randomized
|
||||
or temperature-controlled.
|
||||
|
||||
The initial corpus's final Full/stereo-before trial included a 2,611.74 ms inference stall.
|
||||
It remains in the raw samples and means; p95 alone hides that isolated stall. Geometric
|
||||
acceptance does not apply live capture-age gates, so that observation would be stale in
|
||||
live tracking. The later central lens-sweep samples had no processing times above 200 ms.
|
||||
No moving-hand, pinch/release, sustained SteamVR throughput, external latency, or independent
|
||||
position-accuracy pass is claimed. The present matcher needs better correspondence and
|
||||
lower cost before a live increased-input-rate trial is justified.
|
||||
|
||||
## Reproduction and evidence
|
||||
|
||||
The temporary runner and matcher are no longer shipped. Local evidence from those runs:
|
||||
|
||||
- Final lens comparison and raw samples: `build/hand-hybrid/lens/` and its `summary.json`.
|
||||
- Initial full corpus: `build/hand-hybrid/corpus/` and its `summary.json`.
|
||||
- First prototype trial: `build/hand-hybrid/first/`.
|
||||
- Build and focused tests: `build/hand-hybrid-validation.log`.
|
||||
- Lens comparison build: `build/hand-hybrid-lens-build.log`.
|
||||
- Installed lens-benchmark APK SHA-256: `d976329b96e6aaceec62ef38085e203942cf763c51102adfeb50c4e5614891f1`.
|
||||
- Calibration SHA-256: `3983438843877a321d3098f73496dfd9256f0088f095aa8d591602cf0b2cb213`.
|
||||
|
||||
Each result also includes both capture-image hashes and the model hashes. Build-directory
|
||||
evidence is local and may not be present in a fresh checkout.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Full versus Lite hand models and image preprocessing
|
||||
|
||||
The saved-image comparison favors the Full palm detector and Full landmark model.
|
||||
Neither contrast normalization nor lens correction improved usable bare-hand detections
|
||||
on the two available bare-hand scenes. This is a small static corpus, not a live tracking pass.
|
||||
|
||||
## Method
|
||||
|
||||
The same saved 640 × 480 grayscale stereo pairs were evaluated with eight configurations:
|
||||
Full or Lite palm/landmark bundles, each with raw input, calibrated lens correction,
|
||||
1st/99th-percentile contrast normalization, or both corrections. Grayscale-to-RGB replication
|
||||
and MediaPipe's model-required tensor scaling were enabled in every configuration.
|
||||
|
||||
The main comparison ran on the Samsung SM-F976U using the app's Android MediaPipe Tasks
|
||||
1.0.0 and OpenCV 4.13.0. Each configuration created fresh CPU VIDEO detectors, with two
|
||||
hands per camera, parallel camera inference, and all detection/presence/tracking thresholds
|
||||
at 0.6. Adaptive input and scheduled one-hand inference were excluded. Each pair received
|
||||
one cold inference, four warmups, and twelve measured repetitions at synthetic 80 ms
|
||||
timestamp intervals. Bare-hand mode was verified; wrist-only mode was false.
|
||||
|
||||
This was an offline replay with the game idle and tracking cameras disconnected. It loaded
|
||||
the phone's saved verified calibration and never initialized cameras or published replay
|
||||
poses. Raw-image landmarks were undistorted during triangulation; corrected-image landmarks
|
||||
were triangulated without applying distortion correction again. Both paths retained the
|
||||
existing handedness, depth, reprojection, palm geometry, and motion checks.
|
||||
|
||||
The corpus contains two bare-hand scenes, one no-hand control, and six controller-grip
|
||||
scenes. The central bare-hand scene was also repeated at the end of the phone run. Replaying
|
||||
one pair twelve times does not create twelve independent accuracy samples. Desktop IMAGE
|
||||
and VIDEO tests using MediaPipe 0.10.21/OpenCV 4.11.0 provide a separate cross-check.
|
||||
|
||||
## Bare-hand result
|
||||
|
||||
For capture `64390821246320`, two open hands are visible in both cameras:
|
||||
|
||||
| Image preprocessing | Full: usable stereo palms per replay | Lite: usable stereo palms per replay |
|
||||
|---|---:|---:|
|
||||
| Raw | 1 of 2 | 0 of 2 |
|
||||
| Lens correction only | 1 of 2 | 0 of 2 |
|
||||
| Contrast normalization only | 1 of 2 | 0 of 2 |
|
||||
| Lens correction + contrast normalization | 1 of 2 | 0 of 2 |
|
||||
|
||||
The cold frame and all twelve measured frames agreed. Full found one hand in each eye;
|
||||
Lite found no hands with raw, lens-only, or contrast-only input. With both corrections,
|
||||
Lite found one hand in the left camera and none in the right, so it could not reconstruct
|
||||
a stereo position.
|
||||
|
||||
Capture `72745789186883` contains one open hand, peripheral in the left image. Every
|
||||
configuration missed it in both cameras throughout cold and measured replay. The desktop
|
||||
IMAGE and VIDEO tests agreed with these bare-hand outcomes. The no-hand control
|
||||
`59405518228482` returned zero hands across all configurations.
|
||||
|
||||
The final repeat of the central scene reproduced the same cold and measured counts in
|
||||
every configuration. All six controller-grip pairs (`110854218431877`, `110858058677761`,
|
||||
`110862058272915`, `110935538464606`, `110939418512521`, `110943418461791`) produced
|
||||
zero accepted stereo palms and zero accepted wrists in all measured phone replays. Some
|
||||
Full configurations produced individual camera detections, but these did not become valid
|
||||
stereo positions. Grip scenes are reported separately from the bare-hand comparison.
|
||||
|
||||
## Phone timing
|
||||
|
||||
These are median milliseconds for a parallel stereo pair on the central scene. They include
|
||||
RGB preparation, inference, and result extraction, but exclude decoding, precomputed lens
|
||||
correction, model initialization, and stereo reconstruction. They are not end-to-end latency.
|
||||
|
||||
| Model | Preprocessing | First run | Final repeat |
|
||||
|---|---|---:|---:|
|
||||
| Full | Raw | 46.10 | 82.98 |
|
||||
| Full | Lens only | 75.17 | 84.46 |
|
||||
| Full | Contrast only | 85.64 | 82.99 |
|
||||
| Full | Lens + contrast | 84.10 | 80.49 |
|
||||
| Lite | Raw | 43.07 | 40.06 |
|
||||
| Lite | Lens only | 37.29 | 39.70 |
|
||||
| Lite | Contrast only | 38.73 | 42.54 |
|
||||
| Lite | Lens + contrast | 78.78 | 75.34 |
|
||||
|
||||
Android reported thermal status 2 throughout the saved-pair runs. Configuration order was
|
||||
fixed, and inference work differs when a hand is found. Full/raw's first-versus-last variation
|
||||
also prevents attributing the timing differences to preprocessing alone. Lite's lower timings
|
||||
mostly accompany zero returned hands; its two-camera usable-hand result remains zero.
|
||||
|
||||
## What normalization changes
|
||||
|
||||
The saved calibration uses pinhole radial distortion coefficients (`fisheye: false`). The
|
||||
comparison tests that calibrated lens correction, rather than forcing a fisheye model onto
|
||||
the coefficients. Keeping the original camera matrix and output dimensions narrows the
|
||||
field of view: the left output samples approximately raw x=96–544 and y=56–437, and the
|
||||
right approximately x=93–547 and y=57–440. Some peripheral fingers near raw x=600 disappear
|
||||
from the corrected left image. This is a concrete limitation of the current mapping, although
|
||||
the raw version also failed to detect that hand; the crop alone does not explain all failures.
|
||||
|
||||
Model tensor normalization was already provided by MediaPipe. The additional normalization
|
||||
tested here changes image contrast. Replicating grayscale into RGB satisfies the input channel
|
||||
contract but cannot recover missing color information.
|
||||
|
||||
## Recommendation and limits
|
||||
|
||||
Use Full with raw grayscale replicated into RGB as the next live baseline, retaining MediaPipe's
|
||||
tensor scaling and calibrated point correction for stereo geometry. Full has the stronger
|
||||
detection result here; extra contrast/lens preprocessing has no demonstrated acceptance gain
|
||||
on these scenes. Broader motion testing is still needed before calling any configuration best
|
||||
overall. Full itself still misses one of two central hands and the peripheral hand.
|
||||
|
||||
The comparison used a temporary diagnostic build with opt-in preprocessing and offline replay.
|
||||
That comparison code has been removed after selecting the Full/raw baseline; the live path
|
||||
keeps grayscale-to-RGB conversion and calibrated point correction. Build, twenty focused JVM
|
||||
tests, packaged runtime verification, and the installed Android replay were checked for the experiment.
|
||||
|
||||
Local evidence is under `build/hand-comparison/`: `phone/` contains full per-frame results,
|
||||
thermal snapshots and before/after status; `phone-summary.json` contains compact counts;
|
||||
`desktop/` and `desktop-video/` contain the independent comparisons. The scripts are
|
||||
`compare.py`, `compare_video.py`, `run_phone.py`, and `report_phone.py` in that directory.
|
||||
`input-manifest.json` records all capture, calibration, and model hashes plus runtime versions.
|
||||
@@ -0,0 +1,466 @@
|
||||
# Luma Ultra hand tracking investigation
|
||||
|
||||
Date: 2026-10-17. Status: **experimental; gameplay usability has not passed**.
|
||||
|
||||
## Current Full/raw baseline and hand rendering
|
||||
|
||||
The stereo pipeline defaults to Full palm detection plus Full hand landmarks following the
|
||||
[saved-image comparison](hand-model-comparison-2026-10-16.md). The optional Lite bundle contains
|
||||
Lite palm and Lite landmarks; earlier historical timings describe a different Full-palm/Lite-landmark
|
||||
bundle. `tools/build_hand_lite.py` pins both official Lite source models,
|
||||
checks their input/output tensor contracts, and copies Tasks metadata from the Full bundle.
|
||||
Both the outer task ZIP and the embedded label ZIPs have deterministic timestamps.
|
||||
|
||||
Live inference consumes original grayscale camera bytes, replicated into reusable RGB buffers.
|
||||
It skips full-frame lens remapping, contrast normalization, and the grayscale-to-RGBA-to-Bitmap
|
||||
round trip. Preview bitmaps are created only at preview cadence; grayscale Mats are populated
|
||||
only for calibration, validation, explicit capture, preview, or Joy-Con optical flow.
|
||||
|
||||
Detection, preview overlays, and Joy-Con flow share raw image coordinates. Calibration corrects
|
||||
the sparse landmark coordinates once before triangulation. The saved `fisheye` flag selects
|
||||
fisheye or pinhole correction; the bundled Luma profile uses pinhole coefficients. Calibration
|
||||
fitting, validation, and diagnostic captures retain their original raw images. The unsuccessful hybrid matcher and preprocessing-comparison code were removed
|
||||
after baseline selection. The existing saved-pair diagnostic uses the same raw RGB
|
||||
preprocessing and point correction as live tracking.
|
||||
|
||||
Each grayscale byte is copied into R, G, and B without changing exposure or image coordinates.
|
||||
[MediaPipe Tasks](https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker/android)
|
||||
then performs model-specific tensor normalization from metadata; the app does not divide these
|
||||
image bytes by 255 a second time.
|
||||
|
||||
Joy-Con mode retains only landmark 0 (the wrist), triangulates and follows that position, and
|
||||
does not estimate palm orientation, pinch, or grip. Replay follows the selected controller mode.
|
||||
The landmark network still predicts all 21 landmarks internally. Physical Joy-Con
|
||||
buttons and IMU orientation remain the controller input sources.
|
||||
|
||||
The user reported that the hand smoothing and dead zones made tracking feel worse. The
|
||||
common bare-hand `OpticalPose` path therefore removes the 3 mm position and 1-degree rotation
|
||||
anchors and restores the earlier fixed 15 ms rendering interpolation, replacing the added
|
||||
adaptive smoothing. Small accepted movements can change the rendered pose immediately.
|
||||
Position prediction still requires consecutive velocities to agree in direction, preventing
|
||||
alternating measurement noise from being extrapolated. The normal mode keeps measured palm
|
||||
orientation; only the opt-in steady-aim setting uses `ShoulderAim`. Joy-Con filtering, gesture
|
||||
release, observation freshness, and calibration gates are unchanged.
|
||||
|
||||
SteamVR startup defers Luma Ultra 6DoF until two seconds after the first confirmed RUNNING
|
||||
status. The launch gate applies before SDK initialization, with display controls available
|
||||
through a nonspatial connection during the wait. Repeated RUNNING polls do not extend the
|
||||
deadline. Stop, failed launch, and session cleanup cancel the pending start; reopening always
|
||||
uses the current device and saved tracking settings. This changes startup sequencing, not the
|
||||
tracking preference or hand-model configuration.
|
||||
Automatic restoration on the same device, including a cancelled launch or early stop,
|
||||
requests the existing stable-pose recenter. Status polls from superseded requests or ended
|
||||
sessions cannot restart the pending tracking transition.
|
||||
|
||||
Bare-hand controller rendering holds the last accepted pose for up to 350 ms from its original
|
||||
capture timestamp, extending the previous 200 ms visual limit. Missing, invalid, or stale
|
||||
observations freeze the rendered pose and stop prediction; repeated packets cannot extend the
|
||||
hold. New observations still require the existing 200 ms freshness check, and trigger/grip
|
||||
release immediately when their observations become invalid or stale. Head loss, origin or
|
||||
connection changes, and controller-mode changes clear the visual cache. This bridges short
|
||||
hand-detection gaps without adding a delay to fresh frames; it does not improve detection or
|
||||
preserve a gesture through tracking loss. Both Full and Lite predict the 21 landmarks used by
|
||||
the grab estimator; wrist-only Joy-Con mode uses physical buttons instead.
|
||||
|
||||
Before this rollback, synthetic 12.5 Hz alternating-noise replay reduced position RMS from 4.354 to 1.772 mm and
|
||||
rotation RMS from 1.722° to 1.398°. At 25 Hz, the respective changes were 1.242 to 0.988 mm
|
||||
and 1.058° to 0.800°. The tradeoff in the 12.5 Hz slow-motion case was slightly more lag:
|
||||
maximum position error changed from 0.423 to 0.593 mm and angular error from 0.489° to
|
||||
0.632°. These results describe the reverted adaptive filter, not the current behavior or
|
||||
measured live tracking improvements; the user's subsequent experience favored removing it.
|
||||
The 80 focused JVM tests and native spatial, controller, adaptive-filter, and Joy-Con tests
|
||||
passed. Build and installation evidence is recorded under `build/hand-baseline/`.
|
||||
|
||||
The visual-hold follow-up passed the full native suite, including capture-age boundaries,
|
||||
frozen rendering, stale-packet rejection, immediate gesture release, fresh reacquisition, and
|
||||
cache invalidation. Only the VR driver changed among the 150 staged runtime outputs.
|
||||
Build and installation evidence is recorded under `build/hand-visibility/`.
|
||||
|
||||
The subsequent dead-zone/adaptive-filter rollback passed the full native suite. Regression
|
||||
tests exercise 0.2 mm translation and 0.05-degree rotation immediately after settling, sustained
|
||||
small motion, reversal, and the retained visibility/input protections. Synthetic 95% step
|
||||
response was 44 ms after receipt at both 12.5 and 25 Hz observation rates. This measures the
|
||||
render filter, not end-to-end camera latency. Only the VR driver changed among the 150 runtime
|
||||
outputs; rollback evidence is recorded under `build/hand-filter-revert/`.
|
||||
|
||||
The reviewed startup-delay follow-up passed 129 focused JVM tests, and an isolated snapshot
|
||||
containing only the startup changes passed its 128 focused tests. Earlier phone logs recorded the spatial SDK
|
||||
open attempt 3.348 seconds after confirmed SteamVR RUNNING: the two-second delay plus SDK
|
||||
connection teardown. The resumed session reported valid 6DoF with no pending or required
|
||||
recenter. The native driver and hand configuration were unchanged. Evidence is recorded under
|
||||
`build/luma-vr-start-delay/`. The review fixes were rebuilt and installed, but the phone
|
||||
reported no glasses connected during the follow-up, so hardware cancellation recovery
|
||||
was not rechecked.
|
||||
|
||||
The installed APK SHA-256 is `fac0973a21e41db842d88617bb11f16672987bd0b955736ad510abceca46b00f`.
|
||||
Earlier connected-phone diagnostics confirmed `FULL`, `raw-grayscale-rgb`,
|
||||
`pinhole-points-only`, concurrent CPU inference, scheduled discovery, adaptive input,
|
||||
and `steadyHandAim=false`. Camera/head processing was active with the verified device
|
||||
calibration. The launch check reached READY and staged VR driver SHA-256
|
||||
`e79525c144e934761f08cdc1d3170d058df89bf55c8458ddc955b58c90c9b128`, matching the rebuilt
|
||||
payload. This verifies deployment and live processing, not moving-hand accuracy or comfort.
|
||||
|
||||
The optional Lite bundle can be rebuilt with `mediapipe==0.10.21` using:
|
||||
|
||||
```sh
|
||||
python tools/build_hand_lite.py --palm palm_detection_lite.tflite --lite hand_landmark_lite.tflite
|
||||
```
|
||||
|
||||
Source URLs and hashes are in `app/src/main/assets/models/hand_landmarker_lite.provenance.json`.
|
||||
These changes still need live moving-hand, closed-fist, gesture-continuity, and latency validation.
|
||||
Earlier Lite/normalized-input validation passed 52 focused JVM tests, and bundle rebuild and label-archive reproducibility
|
||||
checks passed. Desktop MediaPipe 0.10.21 inference succeeded on color hand fixtures. All six
|
||||
saved left-camera grayscale grip images still returned zero detections after correction and
|
||||
contrast normalization at the existing 0.6 confidence thresholds. This is not a tracking-accuracy
|
||||
pass. Synthetic projection/undistortion/triangulation checks stayed below 0.12 mm reconstruction
|
||||
error for the bundled pinhole profile and near numerical precision for a synthetic fisheye profile.
|
||||
Local reports for that implementation check are in `build/hand-lite-models/`; Android hardware
|
||||
inference was not run at that stage.
|
||||
|
||||
The subsequent [Full/Lite preprocessing comparison](hand-model-comparison-2026-10-16.md)
|
||||
ran all eight model/preprocessing combinations on the phone using nine saved stereo pairs.
|
||||
Full recovered one of two central open hands; Lite recovered no usable stereo hand. Neither
|
||||
model found the peripheral hand, and contrast/lens correction added no accepted bare hands.
|
||||
Full with raw grayscale-to-RGB input and calibrated point correction is now the live baseline;
|
||||
the earlier comparison diagnostic itself did not change the defaults.
|
||||
|
||||
There is room to improve the current implementation, but the available evidence does not support calling another smoothing adjustment a gameplay fix. The strongest current blockers are rejected palm reconstruction, interrupted gesture continuity, and processing latency under SteamVR load. The next useful milestone is a repeatable open → pinch → release interaction with a stable aiming pose, followed by sustained in-game testing.
|
||||
|
||||
## What the existing evidence establishes
|
||||
|
||||
The latest recorded post-installation interaction is `build/spatial-live-hands-sticky-grab-updated.json`, summarized by `build/sticky-grab-updated.log`. It belongs to Android process 17867 and SteamVR session `2026-10-16-60-steam`, using the stability candidate before the fork migration. It is historical evidence, not a measurement of the changes made during this investigation.
|
||||
|
||||
| Finding | Evidence | Interpretation and limits |
|
||||
|---|---|---|
|
||||
| Head tracking remained available while hands failed | All 2,823 coherent channel samples had fresh head tracking; right hand was fresh in 1,446 samples (51.2%) | The observed interruption is in the hand path. This is sampled channel availability, not accepted-camera-observation rate; the user's exact hand visibility was not recorded. |
|
||||
| Palm acceptance is a major failure point | Five of eight status snapshots reported matched/reconstructed hands but zero accepted palms | Detection alone is insufficient. The old message grouped missing palm anchors, implausible width, degenerate orientation, and temporal outliers together. It does not identify which gate caused each rejection. |
|
||||
| Gesture continuity can prevent clicks entirely | The right-hand trace contained valid open and pressed pinch values, separated by loss transitions | Requiring a fresh observed release after loss is correct, but unreliable tracking can prevent a usable press sequence. See the replay below. |
|
||||
| Processing already consumes most of the latency target | Eight reported processing times ranged from 63.3 to 114.0 ms; observation ages from 63.6 to 155.7 ms | These are internal status samples, not physical-motion-to-display latency. Some exceed the complete external p95 target of 100 ms before rendering is counted. |
|
||||
| The run was thermally constrained | Android thermal status was 4; HAL skin temperature increased from 53.9 to 54.9 °C | This run cannot establish cold performance or isolate thermal throttling from game workload. |
|
||||
| Latest stability changes did not establish usable interaction | The user reported similar stationary jitter and inability to start a dashboard drag | The previous candidate's automated tests and successful installation did not establish a hardware interaction pass. |
|
||||
|
||||
The trace used CPU, full model, concurrent stereo inference, automatic input rate, and scheduled tracking. All eight diagnostic snapshots selected the two-hand search bank and a 500-permille camera copy rate. Approximately 25 Hz camera delivery therefore supplied at most about 12.5 eligible stereo pairs per second before processing and rejection. The left-hand channel had only 14 fresh samples; this was not a controlled two-hand visibility test, so that number is not left-hand detection accuracy.
|
||||
|
||||
### Why a detected pinch may produce no click
|
||||
|
||||
The right-hand transition list spans 15.885 seconds and contains 142 rows. Applying the current trigger rules from `tools/vr/src/spatial_tracking.h` to those sampled rows gives:
|
||||
|
||||
| Trace quantity | Count |
|
||||
|---|---:|
|
||||
| Fresh, valid new observations below the 0.25 release threshold | 31 |
|
||||
| Fresh, valid new observations above the 0.65 press threshold | 12 |
|
||||
| Changes in the right trigger's loss-counter byte | 16 |
|
||||
| Valid-palm transition rows with invalid pinch geometry | 58 |
|
||||
| Above-threshold observations while the replayed trigger remained armed | 0 |
|
||||
|
||||
Replay starts disarmed because the first sampled hand is invalid. It compares the low byte of the right hand's `epochs`, releases on invalid/stale hand or pinch data, ignores duplicate observations for arming, checks the 200 ms gap against the previous accepted hand timestamp, and arms only on a fresh valid trigger value below 0.25. A valid palm with invalid pinch still advances the hand observation timestamp, as in the driver.
|
||||
|
||||
This is an inference from a sampled shared-memory trace. Sampling may omit intermediate observations, and this replay does not record what SteamVR or its dashboard consumed. It nevertheless gives a concrete explanation consistent with the reported inability to start a drag: the recognizer produces high pinch values, but continuity is lost before those values can form an armed press. Removing the release requirement would reintroduce presses on reacquisition and would not repair the observation stream.
|
||||
|
||||
## Prioritized work
|
||||
|
||||
1. **Measure each rejection before changing acceptance.** Count processed observations, valid palm/gesture observations, and rejection reasons separately for each hand. Distinguish no matching pair, missing triangulated palm anchors, palm width, degenerate orientation, implausible motion, synchronization, and age. Keep cumulative counters separate from sparse UI snapshots. These counters need no camera image recording. A broad reduction in stereo-quality thresholds could trade disappearing hands for incorrect depth and jumps.
|
||||
|
||||
2. **Establish one reliable interaction sequence.** With a single visible hand at 30, 50, then 80 cm, repeat open → pinch → hold → release while collecting the observation counters and existing driver input-transition logs. Repeat for the other hand, then with both visible. Record whether failure happens before palm acceptance, at pinch validation, at arming, at OpenVR publication, or inside the dashboard. The dashboard uses trigger for click/drag; the installed Vive binding maps grip to Back. Keep the other fingers open during the pinch test so the two actions remain distinguishable.
|
||||
|
||||
3. **Reduce measured processing cost and discovery spikes.** Run temperature-matched comparisons of full/scheduled CPU, lite/scheduled CPU, and input-rate settings, retaining the same game scene and deliberate motions. Measure completed-observation rate and accepted-observation rate, not camera rate or repeated channel reads. Existing saved-open-palm benchmarks show potential: under SteamVR, scheduled full inference had median/p95 46.92/94.71 ms versus full-search 91.48/99.26 ms; scheduled lite gave 40.26/86.77 ms. Those static-image results cannot establish moving-hand, pinch, or crossed-hand accuracy. The full model should remain the baseline until those comparisons pass. GPU requires a separate stability check because earlier evidence records graphics recovery/device-loss concerns.
|
||||
|
||||
4. **Tune pose and aiming with motion evidence.** Record stationary raw and rendered pose distributions and a repeatable slow aiming path. At the time of this trace, driver stabilization held position within 3 mm and orientation within 1 degree around a stationary anchor, applied a 15 ms rendering correction, and bounded prediction to 2 cm/80 ms. The anchors have since been removed at the user's request. Increasing smoothing without measuring raw noise and lag risks making aiming less responsive. Test motion-aware filtering or a more stable aim direction only against the same recorded motions and external latency measurement. Aiming direction and a game controller's grip pose may need separate treatment.
|
||||
|
||||
5. **Validate game compatibility after continuity.** Optical controllers currently expose Vive-compatible trigger/grip/trackpad inputs, with gamepad sticks and buttons supplying locomotion and menus; there is no skeletal hand output. `CardboardController::UpdateOptical` initializes `DriverPose_t` to zero and populates position and rotation, leaving linear and angular velocity zero. This is a source-confirmed omission to investigate for throwing and other velocity-dependent games, not an established cause of the dashboard problem. A velocity implementation needs capture-time derivatives, outlier/loss resets, a defined pose timestamp, and a game-level test before enabling it.
|
||||
|
||||
Startup should be measured separately from steady state. Earlier logs contained shader-pipeline creation well after the dashboard appeared, while both hand detectors were already initialized. Correlate observation latency and CPU/GPU workload through launch and the following two minutes; model warmup alone does not explain the reported minute-long startup delay.
|
||||
|
||||
The exported calibration also helps explain why a 3 mm stationary hold may have little visible effect. Its focal lengths are approximately 226–241 pixels and its stereo baseline is approximately 14 cm. The ideal rectified-stereo approximation `depth change ≈ depth² / (focal length × baseline) × disparity change` gives roughly 8 mm per disparity pixel at 50 cm and 20 mm at 80 cm. These are sensitivity estimates, not measured hand errors; the actual cameras have lens distortion and relative rotation. Independent landmark jitter can therefore exceed the driver's hold radius. If the new counters confirm palm reconstruction dominates, a robust fit across the available palm landmarks and temporal stereo correspondence are stronger candidates than simply enlarging that hold radius. Such a fit still needs explicit outlier rejection, side identity, capture-time freshness, and measured motion/gesture tests.
|
||||
|
||||
## Vendor runtime options
|
||||
|
||||
The public [VITURE Glasses C SDK documentation](https://www.viture.com/developer/glasses-sdk/glasses) describes USB management, head tracking, display control, and camera streaming, including Android support. It does not document a phone hand-tracking runtime that can replace this application's MediaPipe reconstruction directly. That is a statement about the published API reviewed on this date, not proof that VITURE has no private or forthcoming implementation.
|
||||
|
||||
The [VITURE Unity SDK documentation](https://www.viture.com/developer/unity-sdk/unity) does describe hand tracking, responsive/stable filter modes, and arm/wrist contributions to aiming. Its stated hardware requirement and deployment instructions target the Pro Neckband. Those controls provide useful design references, but the documentation does not establish that their runtime can be redistributed inside this phone app. Confirm phone runtime availability, platform support, and licensing with VITURE before planning that replacement.
|
||||
|
||||
## Calibration and completion criteria
|
||||
|
||||
The phone's current calibration can be shipped as the Luma Ultra starting profile. That reduces setup friction; it does not fix inference cost, hand occlusion, gesture continuity, or jitter by itself. Preserve each device's own calibration as the higher-priority choice, restrict the base profile to its matching model and camera geometry, and identify it as a shared starting calibration. A calibration validated on the source glasses has not been independently measured on every Luma Ultra unit.
|
||||
|
||||
Retain the existing acceptance gates in `luma-spatial-tracking.md`: at least 95% accepted slow unobstructed hand observations at 30–80 cm, no more than 5 cm position error and 2 cm stationary jitter, external p95 latency below 100 ms, reliable release and reacquisition, and no more than 10% regression in p95 game frame time over a 20-minute run. Dashboard selection and Angry Birds VR aim/grip/pull/release still require deliberate physical validation. Keep the feature visibly experimental until these tests pass.
|
||||
|
||||
Supporting records: `luma-spatial-evidence.md`, `build/spatial-live-hands-sticky-grab-updated.json`, `build/hand-reliability-benchmark-smoothness-steamvr.json`, and `build/hand-startup-review.json`. The build-directory records are local historical artifacts and may not accompany a fresh checkout.
|
||||
|
||||
## Joy-Con grip and arm extension investigation (2026-10-16)
|
||||
|
||||
The target is a closed fist holding a controller, with the wrist and forearm visible from the
|
||||
wearer's viewpoint. Open-palm performance is not sufficient evidence for this use case. In a
|
||||
requested 16-second extension exercise, 5 of 80 sampled left positions and 1 of 80 right positions
|
||||
were valid while head tracking stayed valid. These are channel availability samples, not 80
|
||||
independent detector results. The earlier driver also restarted its transition to the arm model
|
||||
on each invalid hand frame, preventing intermittent camera positions from converging.
|
||||
|
||||
The initial grip estimator required the wrist, middle base knuckle, and one supporting base
|
||||
knuckle, rather than all five palm landmarks and a hand-derived orientation. The later
|
||||
wrist-only update below removes the remaining knuckle dependency. It never generates hand
|
||||
gestures; Joy-Con IMU and buttons remain authoritative. Stereo optical flow can carry the
|
||||
identified wrist through brief model misses, with forward/backward consistency, photometric,
|
||||
stereo reprojection and motion checks. Tracks expire 750 ms after the last detector seed; frame
|
||||
gaps over 200 ms discard them. The driver retains a camera observation only until 200 ms after
|
||||
its original capture time. Neither mechanism turns stale coordinates into fresh measurements.
|
||||
|
||||
### Model assessment
|
||||
|
||||
- **MediaPipe Full (historical baseline):** an existing Android pipeline with separate palm
|
||||
detection and landmark regression. Curling fingers does not intrinsically exclude a hand,
|
||||
but occlusion and the egocentric view must be tested with actual grips. Its published
|
||||
monocular world landmarks are not used as metric camera distance; calibrated stereo supplies
|
||||
distance. [Official task documentation](https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker).
|
||||
- **Earlier bundled Lite:** that bundle changed the landmark regressor while retaining the Full
|
||||
palm detector (see `tools/build_hand_lite.py`). It can reduce processing cost, but cannot be
|
||||
assumed to fix the initial fist-detection problem. Earlier saved open-palm replay under
|
||||
SteamVR measured scheduled Full median/p95 46.92/94.71 ms and Lite 40.26/86.77 ms. Those figures
|
||||
do not establish closed-fist accuracy.
|
||||
- **RTMDet-nano hand detector plus RTMPose-m Hand5:** a concrete alternative detector and
|
||||
landmark pipeline worth evaluating against grip frames. The official demo uses separate
|
||||
hand boxes and landmarks. It is not a replacement `.task` asset: Android inference,
|
||||
preprocessing, handedness/identity association, and device timings need integration and
|
||||
verification. No claim of superior Joy-Con performance is established yet.
|
||||
[Official hand demo](https://github.com/open-mmlab/mmpose/blob/main/demo/docs/en/2d_hand_demo.md).
|
||||
- **HaMeR:** useful as an offline reference for occluded and egocentric hands, with published
|
||||
evaluation procedures. Its reference implementation uses PyTorch, ViTPose, and MANO assets;
|
||||
it is not an established real-time Android replacement for this app. Do not substitute its
|
||||
inferred mesh depth for measured stereo depth.
|
||||
[Official implementation](https://github.com/geopavlakos/hamer).
|
||||
- **Whole-body arm pose:** not the first candidate for cameras on the wearer's glasses,
|
||||
where the wearer's shoulders and torso are commonly outside view. An egocentric fist/wrist
|
||||
detector or forearm segmentation model is the more directly matched development target.
|
||||
This is an engineering assessment, not a measured comparison of those models on this device.
|
||||
|
||||
The third camera is a separately exposed UVC pass-through camera, not an unused member of the
|
||||
stereo callback. It may provide better RGB recognition, but requires its own capture/decode and
|
||||
camera-to-head calibration. It does not independently provide metric depth. The preferred use
|
||||
would be recognition or association assistance while stereo supplies distance; size-based
|
||||
monocular distance would need separate calibration and uncertainty handling.
|
||||
[VITURE SDK camera support](https://www.viture.com/developer/glasses-sdk/glasses).
|
||||
|
||||
Saved-grip replay reports `joyCons` acceptance separately from bare-hand `hands`, for both Full
|
||||
and Lite. Repeating one image measures model/geometry behavior on that image, not motion
|
||||
continuity. Live tests must still check reach response, left/right identity, reacquisition,
|
||||
camera loss, and head motion before considering the experiment reliable.
|
||||
|
||||
### Closed-fist model comparison
|
||||
|
||||
Three requested grip pairs were saved locally as captures `110935538464606`, `110939418512521`,
|
||||
and `110943418461791`. In on-phone saved-frame replay of the middle and last pair, Full detected
|
||||
one hand in the left image and none in the right image in all 20 measured repetitions per
|
||||
configuration; Lite detected none in either image. Neither produced a valid grip position.
|
||||
On the middle pair, scheduled Full median/p95 was 56.05/62.96 ms and Lite 46.00/47.99 ms. On
|
||||
the last pair, it was 21.17/23.33 ms and 18.08/20.06 ms respectively. Timing differences between
|
||||
frames are not accuracy improvements. Repeated static frames are not independent observations.
|
||||
|
||||
An isolated Windows evaluator tested the official RTMDet-nano hand / RTMPose-m Hand5 ONNX
|
||||
exports using `rtmlib` 0.0.16 and ONNX Runtime 1.31.0. Across the six raw images, hand-box counts
|
||||
were 0, 0, 1, 0, 2, 1. Lens-corrected images gave 0, 1, 1, 0, 2, 1. Neither variant found both
|
||||
hands in both cameras for any pair. These are box counts, not validated stereo positions or
|
||||
ground-truth accuracy. Desktop timings cannot predict Android cost. Model downloads and hashes
|
||||
are in `build/joycon-alternative-models/provenance.json`; results and overlays are local.
|
||||
|
||||
MediaPipe desktop preprocessing comparisons used the same Full model asset with Python Tasks
|
||||
1.1.1, versus Android Tasks 1.0.0. Overlapping 400-pixel crops recovered some missing detections;
|
||||
lens correction and lower thresholds also changed detections and sometimes handedness. Only
|
||||
isolated stereo matches passed the existing geometry checks. This does not justify enabling
|
||||
either preprocessing change in the live pipeline yet, or replacing Full with Lite/RTMPose.
|
||||
|
||||
The post-install shared-memory sample arrays in `build/joycon-reach-after.json` and
|
||||
`build/joycon-reach-grip.json` are not valid live-pose measurements: the SteamVR session had
|
||||
not restarted, so its pose publisher was inactive. Their before/after provider diagnostics and
|
||||
requested image captures remain useful. During the actual closed-fist interval, all 253
|
||||
processed pairs reported invalid head tracking, so no live reach claim can be made. The local
|
||||
measurement script now requires a running session, valid head tracking, and advancing head
|
||||
timestamps to reject this situation. The image comparisons above do not depend on a live head
|
||||
pose. A new live test is still required after head tracking and the session recover.
|
||||
|
||||
### Third-camera grip comparison
|
||||
|
||||
The separately exposed RGB camera delivered 1920 by 1080 JPEGs after Android Camera and USB
|
||||
permissions were granted. SDK 2.4.0 crashed in `xr_camera_provider_start` when reopening the
|
||||
provider in the main app process. RGB diagnostics now run in a dedicated process that skips
|
||||
normal app initialization. Four initial successful captures preserved the main app PID.
|
||||
Subsequent one-shot attempts intermittently returned zero callbacks despite a successful SDK
|
||||
start. Keeping a stream open between requests did not resolve this: four successive startup
|
||||
attempts returned zero callbacks. USB reconnection and a separate attempt with positional
|
||||
tracking temporarily disabled also returned no frames. The user's decision was to stop the RGB
|
||||
route and compare the models on stereo. The persistent-stream and RGB/stereo calibration-buffer
|
||||
prototypes were removed; the isolated one-shot command remains for manual diagnostics only.
|
||||
|
||||
Three requested normal-grip RGB captures (`112581486574499`, `112588468819444`, and
|
||||
`112595950634337`) were evaluated with the same local model assets and thresholds:
|
||||
|
||||
| Pipeline, full image | First | Second | Third |
|
||||
|---|---:|---:|---:|
|
||||
| MediaPipe Full | 0 hands | 0 hands | 2 hands |
|
||||
| MediaPipe Lite | 0 hands | 0 hands | 1 hand |
|
||||
| RTMDet-nano + RTMPose-m Hand5 | 2 hands | 2 hands | 2 hands |
|
||||
|
||||
Overlapping square crops recovered one grip in the first image with Full and Lite; neither
|
||||
recovered a hand in the second. The no-hands control image produced zero detections for all
|
||||
three pipelines. Visual inspection of the alternative model's overlays found plausible wrist
|
||||
locations on these grips, including the closer, partly cropped second view. Three selected
|
||||
images are not an accuracy dataset or evidence of continuous tracking, identity preservation,
|
||||
or measured reach. RGB results and overlays are in `build/joycon-rgb-evaluation`.
|
||||
|
||||
The unchanged official ONNX models were also replayed on the Fold7 using Android ONNX Runtime
|
||||
1.31.0 and OpenCV 4.13.0 preprocessing. Four warm-ups and ten timed iterations per image gave
|
||||
two-thread median times of 119.90, 119.88, and 129.24 ms; the largest sample was 151.04 ms.
|
||||
Four threads gave medians of 120.45, 120.79, and 121.05 ms. Timings include preprocessing,
|
||||
detection, and two hand landmark passes, but exclude JPEG decoding, camera acquisition,
|
||||
stereo depth, head fusion, rendering, and concurrent SteamVR load. Thus they already exceed
|
||||
the 100 ms end-to-end target. The app's OpenCV DNN importer alone rejected the detector's TopK
|
||||
operator; these measurements used a separate native benchmark, not a new production model
|
||||
dependency. [ONNX Runtime mobile support](https://onnxruntime.ai/docs/get-started/with-mobile.html).
|
||||
|
||||
The RGB SDK timestamps were zero on this device. Diagnostics record host callback receipt
|
||||
separately. A temporary prototype successfully collected one nearby stereo pair with surrounding
|
||||
frames, but the subsequent board attempt received no RGB frame. No RGB calibration was fitted.
|
||||
Existing stereo calibration does not supply RGB lens parameters or the RGB-to-stereo transform.
|
||||
RGB has never been used to publish controller positions in this experiment.
|
||||
|
||||
### Stereo-only follow-up and wrist landmark 0
|
||||
|
||||
The alternative RTMDet/RTMPose pipeline was replayed on the Fold7 against the same three raw
|
||||
stereo grip pairs, without RGB. With live Joy-Con inference temporarily paused, counts were
|
||||
0/0, 1/0, and 2/1 hands in the left/right images. Per-image medians were 45.63/45.42,
|
||||
129.35/45.03, and 199.50/123.13 ms. These are sequential single-image benchmark measurements,
|
||||
not concurrent stereo latency or end-to-end tracking. Both candidate pairings in the last
|
||||
capture failed the current geometry gates. The plausible wrist pair projected with 4.17 px
|
||||
and 3.27 px error, exceeding the 3 px limit; the other pairing reconstructed behind the camera.
|
||||
Neither a valid grip nor a valid isolated wrist was established. The log and geometry report
|
||||
are `build/joycon-stereo-phone-ort-controlled.log` and
|
||||
`build/joycon-stereo-phone-ort-geometry.json`.
|
||||
|
||||
“Full” in this project is the full MediaPipe Hands landmark model, not a full-body model.
|
||||
At the time of the model comparison, the live Joy-Con estimator used the midpoint of wrist 0
|
||||
and middle-finger base 9, validated with an additional base knuckle. The saved-pair diagnostic
|
||||
added `wrists`, testing landmark 0 without those palm anatomy requirements. This cannot
|
||||
recover a wrist when the upstream hand detector misses the grip in either camera.
|
||||
|
||||
The existing desktop Full outputs at the production 0.6 thresholds, using raw uncropped
|
||||
images, gave confident left/right counts of 0/0, 1/0, and 0/0. Thus none offered a wrist
|
||||
pair even before palm geometry checks. Desktop Tasks 1.1.1 static-image evaluation differs
|
||||
from Android Tasks 1.0.0 VIDEO replay and must not be presented as an Android measurement.
|
||||
The local report is `build/joycon-mediapipe-wrist-evaluation.json`. The updated Android
|
||||
wrist-only diagnostic was built and installed, but its replay is pending the glasses USB
|
||||
approval after installation. The last phone settings check showed positional tracking on,
|
||||
Joy-Con tracking off, and hand controllers off; there was no active stereo engine.
|
||||
|
||||
Keep MediaPipe Full as the live baseline. The alternative's better RGB detections did not
|
||||
transfer to dependable stereo grip positions on these captures. No new physical arm-extension
|
||||
validation has been completed.
|
||||
|
||||
### Live wrist-only Joy-Con position
|
||||
|
||||
Following the user's correction, Joy-Con mode triangulates only landmark 0 and publishes its
|
||||
position transformed through the camera-to-head and captured head poses. Knuckle availability,
|
||||
palm proportions, and the wrist/knuckle midpoint no longer affect Joy-Con acceptance or position.
|
||||
The estimator is `JoyConWristPosition`, and diagnostics report `joycon-wrist` with wrist-specific
|
||||
missing-landmark and motion outcomes. Bare-hand gesture tracking still uses its palm geometry.
|
||||
|
||||
Optical flow now seeds only the wrist and accepts a single tracked point per camera; the
|
||||
previous three-point minimum is removed. Forward/backward and photometric checks, stereo
|
||||
depth and reprojection limits, motion rejection, head synchronization, freshness, and track
|
||||
expiry remain in place. Controller rotation and buttons still come from the Joy-Con, and
|
||||
the driver still falls back to the arm model when fresh camera positions are unavailable.
|
||||
This removes the unnecessary knuckle dependency, but does not bypass MediaPipe's hand
|
||||
detector or establish continuous wrist visibility in the saved grip images.
|
||||
|
||||
### RGB transport retry: hidden USB submission failure
|
||||
|
||||
After the wrist-only test did not meet the user's needs, the RGB investigation resumed.
|
||||
Capture `116048611549114` added stderr collection with `LIBUSB_DEBUG=4`, separate from the
|
||||
vendor log hook. The SDK returned start status 0 and streaming status 1, but libusb reported
|
||||
`submit_iso_transfer: submiturb failed, errno=12` (`ENOMEM`) and subsequent isochronous packet
|
||||
errors. No RGB callbacks arrived. GET_INTERFACE showed alternate setting 5; the negotiated
|
||||
UVC payload was 2400 bytes per microframe. This establishes USB allocation/submission failure,
|
||||
not its exact resource limit: Android denied reading the USB memory-limit sysfs attribute.
|
||||
|
||||
Binary inspection also traced the earlier native crash to `libusb_cancel_transfer(NULL)`
|
||||
while cleaning up a failed start. Reopening alone was therefore too specific an explanation.
|
||||
The embedded SDK's roughly 100-transfer queue is a plausible source of pressure. Its public
|
||||
API cannot change the queue or the fixed 1080p30 mode.
|
||||
|
||||
An optional independent diagnostic transport now uses libusb 1.0.30 and libuvc 0.0.8 with eight
|
||||
queued transfers. The pinned libuvc source was patched to preserve submission errors and safely
|
||||
drain partial queues instead of reporting success. A hardware-free executable passed 60 starts
|
||||
on the phone covering failure before any submission, failure after three submissions, and all
|
||||
eight submitted. This validates failure cleanup, not camera capture. The reproducible recipe,
|
||||
source hashes, and licenses are documented in `docs/third-party/uvc/README.md`.
|
||||
|
||||
Raw USB descriptors advertise MJPEG 640×480, 1280×720, and 1920×1080 among other sizes, with
|
||||
30/25/20/15/10/5 fps intervals. The direct path can compare these modes while head/stereo tracking
|
||||
continues. It saves a local JPEG plus transfer diagnostics and stream statistics; it is not yet
|
||||
connected to controller tracking. RGB lens/extrinsic calibration and measured reach remain
|
||||
separate unfinished requirements.
|
||||
|
||||
### Direct UVC follow-up after the firmware update
|
||||
|
||||
The glasses now report firmware `12.0.02.002_20260901`, previously
|
||||
`12.0.01.100_20260326`. Direct UVC acquired short bursts, but no sustained stream:
|
||||
capture `140416689286120` delivered 22 callbacks over 708 ms, followed by 4.3 seconds
|
||||
without another callback. The main app stayed alive. These are transport observations,
|
||||
not arm-extension measurements.
|
||||
|
||||
The rejection logs exposed a separate parsing bug: valid JPEG end markers were followed
|
||||
by one to seven zero bytes, with buffers aligned to eight bytes. The diagnostic now trims
|
||||
up to seven trailing zeros before checking the end marker and saves the trimmed payload.
|
||||
A native test on the phone covers padding, truncation, garbage, null input, and size bounds.
|
||||
This fixes rejection of padded frames; it cannot recover missing USB callbacks. JPEG
|
||||
boundary validation also does not establish successful decoding.
|
||||
|
||||
A 32-transfer queue with eight packets per transfer was compared with the original
|
||||
eight transfers of 32 packets. Both fault-injection builds passed 60 simulated starts.
|
||||
The smaller transfers started 1080p30 without the earlier submission error, but received
|
||||
zero frames (`140593211685272`). At 640p the smaller transfers still stalled after seven
|
||||
callbacks (`140493747047289`). Neither configuration established reliable acquisition.
|
||||
|
||||
Explicit USB device resets returned success but produced no frames at 640×480/30 or
|
||||
320×240/5. GET_INTERFACE retained the selected alternate setting. Android's external
|
||||
camera provider reported zero devices. A physical reconnect with positional tracking
|
||||
disabled before attachment also yielded zero callbacks (`141363168667894`): capture
|
||||
waited five seconds for the first frame and never entered its requested ten-second
|
||||
observation window. The original positional and Joy-Con settings were restored afterward;
|
||||
bare-hand controllers remained off. This failure occurs without the stereo/VIO stream.
|
||||
|
||||
Capture evidence is under `build/joycon-rgb-captures/<id>`, including `usb.log`,
|
||||
`frame.json`, descriptors, and per-worker `libuvc-logcat.txt`. Teardown cancellation
|
||||
messages are not evidence that transfers were cancelled during acquisition. RGB remains
|
||||
diagnostic-only, with no RGB depth calibration or controller-position publication.
|
||||
|
||||
Detailed USB logging in capture `141627121555242` recorded 224 successful isochronous
|
||||
transfer completions over approximately 223 ms, each totaling 96 bytes, followed by no
|
||||
successful completions until capture teardown. These small transfers did not produce a
|
||||
decoded image or frame callback. A follow-up (`141844371790836`) read the UVC streaming
|
||||
error code before and after the stall: both returned code 0. Android denied reading the
|
||||
USB power-management sysfs attributes, so autosuspend was not verified as a cause.
|
||||
|
||||
The user confirmed a direct VITURE cable connection. With the same glasses and cable
|
||||
connected to Windows, FFmpeg/DirectShow decoded 300 frames in each of three ten-second
|
||||
runs: 640×480/30, 1920×1080/30, then 640×480/30 after reopening. Outputs used passthrough
|
||||
frame timing and frame hashes, with no frame-rate duplication or saved video. All exited
|
||||
successfully without timeout. Evidence is in `build/joycon-rgb-windows`, produced by
|
||||
`build/test-luma-rgb-windows.py`, which selects only USB VID:PID `0c45:636b`. This supports
|
||||
investigating the Android host/USB path further; it does not prove the camera will be
|
||||
reliable on every host or establish the exact Android fault.
|
||||
|
||||
A controlled APK replacement of only `libusb.so` with official libusb 1.0.27 also
|
||||
returned zero callbacks (`142296418051861`). Its source archive SHA-256 is
|
||||
`ffaa41d741a8a3bee244ac8e54a72ea05bf2879663c098c82fc5757853441575`; provenance and the
|
||||
one-entry APK comparison are under `build/rgb-libusb-1.0.27`. The production dependency
|
||||
pin remains 1.0.30. An isolated YUYV 320×240/5 diagnostic negotiated format 2 but likewise
|
||||
received zero raw frame callbacks (`142407601079475`), so JPEG validation was not its
|
||||
failure point. Neither comparison changed the production camera format.
|
||||
|
||||
Finally, 64 queued transfers of 32 packets increased the high-speed queue coverage from
|
||||
approximately 32 ms to 256 ms. Its fault-injection executable passed all 60 starts.
|
||||
Capture `142557405834522` recovered 18 valid JPEGs over 577 ms, then stalled for the
|
||||
remaining 9.42 seconds of its ten-second observation. All 18 passed the padding fix;
|
||||
the saved final JPEG also decoded at 640×480 in OpenCV. This is not sustained tracking
|
||||
and neither alternate queue was adopted. Temporary USB reset, queue configuration, and
|
||||
verbose logging experiments were removed. The retained UVC recipe uses eight transfers
|
||||
of at most 32 packets, and retains the JPEG-padding fix.
|
||||
@@ -0,0 +1,445 @@
|
||||
# Luma Ultra implementation evidence
|
||||
|
||||
Date: 2026-10-15. Status: experimental prototype, combined release gate not passed. Native 6DoF and synchronized stereo camera delivery were observed on hardware. Camera calibration and optical-hand acceptance were deferred at the user's request.
|
||||
|
||||
## Device and build
|
||||
|
||||
| Item | Observed value |
|
||||
|---|---|
|
||||
| Android host | Samsung Galaxy Z Fold7, SM-F976U |
|
||||
| Android version | 17 |
|
||||
| Glasses | SDK-confirmed VITURE Luma Ultra / Carina |
|
||||
| Glasses firmware | `12.0.01.100_20260326` |
|
||||
| Vendor SDK | External Android C SDK 2.4.0 |
|
||||
| Test application | `com.droiddeck.launcher.xr`, debug experimental build |
|
||||
| Camera pair | Primary left0/right0, both 640 by 480 |
|
||||
| Hand runtime | MediaPipe Tasks Vision 1.0.0; OpenCV 4.13.0; pinned model SHA-256 in the model provenance asset |
|
||||
| CPU or GPU inference benchmark | Not run; no accepted calibration |
|
||||
|
||||
Existing picture, refresh-rate, and graphics work was preserved. The test temporarily enabled positional tracking and in-memory camera diagnostics. Diagnostics were stopped and the positional preference restored to its previous off setting afterward. Hand controllers were never enabled. No camera image recordings were saved. Subsequent builds are local artifacts; the phone was left available for a separate active SteamVR test.
|
||||
|
||||
## Hardware observations
|
||||
|
||||
Initial VIO output was unstable. After the user wore/moved the glasses and the software origin was recentered, valid 6DoF output was observed. Changing position values confirms positional data is available, but does not establish correct physical axes or 1:1 scale without a measured reference.
|
||||
|
||||
Camera frames initially arrived before their matching VIO callback. A bounded wait of up to 35 ms in the latest-frame reader allowed matching timestamps without building a queue. The public callback's undocumented pose basis is not used; capture transforms come from canonical OpenGL pose history.
|
||||
|
||||
A subsequent diagnostic interval produced these results:
|
||||
|
||||
| Measurement | Result and scope |
|
||||
|---|---|
|
||||
| Sampled head validity | 30 of 30 status snapshots valid |
|
||||
| Sampled camera/head synchronization | 30 of 30 status snapshots synchronized |
|
||||
| Processed stereo pairs | Counter increased by 1,048 |
|
||||
| Dropped stereo pairs | Counter increased by 9 |
|
||||
| Median reported camera rate | 24.998 Hz |
|
||||
| Median estimated frame age | 24.73 ms |
|
||||
| Observed estimated frame-age range | 21.44–45.96 ms across the 30 snapshots |
|
||||
| Median diagnostic processing time | 1.62 ms; no hand inference or calibrated undistortion |
|
||||
| Host receipt age of latest head poll | 0.09–4.40 ms across sampled snapshots; not sensor latency |
|
||||
|
||||
These are sparse status samples in an uncontrolled workload, not a frame-by-frame latency benchmark. Receipt-based source-clock mapping does not measure absolute camera exposure delay. None of these values establishes physical-motion-to-visible-controller latency, hand availability, or game-frame-time performance.
|
||||
|
||||
Local raw metadata is in `build/spatial-hardware-observations.json`. The earlier startup/recenter observations are in `build/spatial-after-movement.json`, `build/spatial-recenter-state.json`, and `build/spatial-sync-state.json`. Those files contain pose/status metadata, not images.
|
||||
|
||||
## Automated and build validation
|
||||
|
||||
- SDK-enabled and explicitly SDK-free debug APK builds pass with runtime freshness checks enabled.
|
||||
- All 305 affected Kotlin glasses, head-pose, Cardboard, and glasses UI tests pass with zero failures or skips. New cases exercise canonical coordinates, normalization, floor anchoring, crouched recenter, independent origins, stale/discontinuous samples, generation rejection, pose history, camera-clock warmup/reset, gesture geometry, and inference-buffer ownership.
|
||||
- ARM64 VR driver and presentation-layer builds pass. Native tests cover ABI offsets and size, empty/truncated/version-mismatched or incoherent reads, independent hand timestamps, finite poses, stale tracking, trigger hysteresis, lost-input release and released-gesture rearming. Existing fence, timeline, and render-target tests also pass.
|
||||
- Native Cardboard parameter tests cover all three translation values, zero-translation equivalence, and unchanged SBS/flat parameters. The GLSL compiles; the committed SPIR-V matches regenerated glslang output, and push constants remain within the guaranteed range.
|
||||
- VR build manifests include the new shared header. Packaged runtime and APK payload checks reject mismatched source/output checksums. Both external vendor files and transient dependency-inspection files remain outside tracked source.
|
||||
- Packaging regressions pass: 8 Windows VR build tests, 13 Windows preparation tests, and 11 runtime-input tests. Runtime-input tests ran in the Linux container because the Windows host has no standalone zstd executable.
|
||||
|
||||
Build logs are `build/spatial-sdkfree.log`, `build/spatial-final-build.log`, `build/spatial-vr-build.log`, `build/spatial-shader.log`, and `build/spatial-runtime-evidence.json`. Test reports are under `app/build/reports/tests/testDebugUnitTest`.
|
||||
|
||||
The preserved SDK-enabled snapshot is `build/DroidDeck-luma-spatial-experimental.apk`; the SDK-free counterpart is `build/DroidDeck-spatial-sdkfree.apk`. SHA-256 hashes and final payload checks are recorded in `build/spatial-final-validation.json`. The final local snapshot includes additional buffer-ownership and conservative tracking-loss fixes after the hardware smoke test. It was not installed over the phone being used for the separate SteamVR tests. Integrate these sources into that chat's next build before later calibration; do not replace its newer runtime-storage fixes with this older snapshot.
|
||||
|
||||
A subsequent report from the parallel SteamVR test identified Android eviction of the live `cache/viture` directory. The SDK working-directory source was changed to `noBackupFilesDir/viture` afterward. The preserved APK snapshot and the hardware observations above predate this storage change. No additional phone restart or installation was performed here; the next shared-workspace build should include it.
|
||||
|
||||
## Calibration usability follow-up (2026-10-15)
|
||||
|
||||
The calibration screen now uses DroidDeck's saved palette, app scale, settings groups, and buttons, with separate tracking, capture, validation, and diagnostic sections. Eye height, virtual-screen dimensions, board instructions, and validation messages include imperial measurements. The distance check accepts either 20 inches (0.508 m) or 50 cm (0.5 m), using the selected value in reconstruction validation.
|
||||
|
||||
Recenter requests made while the newest pose is stale or invalid were previously ignored. Requests now remain pending and are acknowledged immediately in the UI; recovery waits for 250 ms of fresh, steady tracking. Normal recentering preserves floor height, new connections cancel old requests, and captures cannot mix different VIO coordinate frames. Looking straight upward also retains a stable heading instead of deriving yaw from an almost-zero forward projection. Camera diagnostics resume when the provider restarts while setup remains open.
|
||||
|
||||
Validation: 247 glasses and calibration/settings UI tests pass with zero failures or skips. The SDK-enabled debug APK builds, runtime input verification passes, and native Robolectric renders of the recovery and distance screens were visually checked. The APK's DEX files were checked for the new measurement controls. Logs are in `build/spatial-setup-build.log`; previews are in `app/build/reports/spatial-setup/`. The phone was not reinstalled or reconfigured during the other chat's SteamVR stability run. The user's reported repeated-recenter behavior still needs confirmation on the glasses after installing this build.
|
||||
|
||||
After those stability tests finished, the combined calibration/recenter APK was installed at 17:09 MDT on October 15. Its SHA-256 is `80bf99374a31676b675856c782298aecc1e1c649ad2fc5c1ce3b18b981979e3e`; hashing the installed `base.apk` confirmed an exact match. The app cold launch and agent status checks passed. The helper, VR driver, and scheduling preload match the preceding freeze-tested build. This install confirms delivery of the latest calibration changes, not hardware acceptance of recentering or calibrated hands.
|
||||
|
||||
## Capture counter fix (2026-10-15)
|
||||
|
||||
Board detection checked head freshness against the worker's start time. A newer head poll arriving during detection made that timestamp precede the latest pose and rejected an otherwise valid capture. Completion now checks freshness under the session lock using the current monotonic time, while still rejecting stale poses, changed connections, changed tracking frames, and pending recentering. The same timing check was corrected around hand inference. Capture progress now exposes pending work and shows success or the rejection reason beside the counter, including the number of common board corners when detection fails.
|
||||
|
||||
Validation: 24 targeted spatial and setup UI tests pass with no failures or skips, including the head-poll/detection timing regression and stale/disconnected-frame rejection. The SDK-enabled APK builds with runtime freshness verification. Build log: `build/spatial-capture-fix.log`. APK: `build/DroidDeck-spatial-capture-fix.apk`, SHA-256 `bf362bb606a3ec72665102b66e4a122c8fad9f5f7333f5a9a3ea3fef9a3bc9d5`. The installed Fold7 `base.apk` hash matches.
|
||||
|
||||
With USB permission granted, valid native tracking, and synchronized stereo images at approximately 25 Hz, a live Capture board view action advanced the counter from 0 to 1 of 24 and reported success. The accepted view remains in the open calibration session. Status-only evidence is in `build/spatial-capture-fix-device.json`; no diagnostic camera recording was saved. This confirms capture acceptance and counter progression, not a completed calibration or optical-hand acceptance.
|
||||
|
||||
## Calculation failure and compact setup follow-up (2026-10-15)
|
||||
|
||||
The user reported calculation failure after at least 24 accepted captures; inspection found 35 observations still in memory. Logcat identified an OpenCV fisheye stereo fitting exception when successive views contained different corner counts (12–24 in this set). [OpenCV 4.13's implementation](https://github.com/opencv/opencv/blob/4.13.0/modules/calib3d/src/fisheye.cpp#L819) allocates its stereo residual blocks using the first view's count. The revised path fits fisheye intrinsics, undistorts every observed point, and fits stereo geometry with fixed intrinsics and zero residual distortion using the variable-count pinhole stereo solver. It retains every correspondence and still measures held-out error with the original lens model. Model failures are logged individually and the page shows the actionable reason instead of a generic failure.
|
||||
|
||||
All 35 observations were exported from the debug app as board landmarks and head poses before installation, backed up in app-private storage, and restored on the same connected glasses. No diagnostic camera recording was taken. Accepted captures now save this metadata automatically; automatic recovery requires a matching serial and geometry, and recovered batches cannot accept new-frame observations. This device did not expose a serial, so recovery here was explicit through the debug interface. The session-labelled backup is not automatically trusted on later unidentified connections.
|
||||
|
||||
The fixed fitter ran on the Fold7 with the restored set. Fisheye stereo fitting now completes, but both models fail stationary-board alignment: the best result varies by 25.5 cm (10.04 inches), above the unchanged 5 cm gate. The user confirmed the board was fixed. Offline replay showed a marked coordinate shift between the early and later observations; an exploratory fit of the first 13 observations agreed within 7.4 mm, while the full set did not. This suggests VIO frame motion/drift during collection; it does not establish its exact cause or validate a calibration from a selected subset. No profile was approved, and hand-controller acceptance remains blocked on a stable capture set plus measured-distance validation.
|
||||
|
||||
The calibration page now has a single Back/title row and no fixed subtitle or eyebrow. A landscape regression check requires at least 75% of page height for scrolling, and both its rendered preview and the installed Fold7 layout were inspected. Guidance explicitly asks for gentle turns and tilts with a pause before each capture. The final SDK-enabled APK and 29 targeted tests pass, including saved-point parsing, stale-frame behavior, restored-batch UI, and landscape viewport coverage. Runtime freshness verification passes. Installed APK SHA-256: `1ca44c6b2b480b197ec0069db026b765d7927c30b7cea7b80cb67d50d626d38a` (verified against `base.apk`). Artifact: `build/DroidDeck-calibration-calculation-fix.apk`. Evidence: `build/spatial-calculation-fix.log`, `build/calibration-observations-backup.json`, `build/calibration-alignment.log`, `build/calibration-calculation-device.json`, and `build/calibration-layout-after.png`.
|
||||
|
||||
## Distance-validation feedback follow-up (2026-10-15)
|
||||
|
||||
A new 25-view capture set produced a fitted pinhole calibration with 1.12094 px held-out reprojection error and passed the existing alignment checks. The user then reported that Validate distance appeared to do nothing. Live status showed that it had run and failed, but the message was attached to Calculate camera alignment, above the visible distance controls.
|
||||
|
||||
Validation now has independent pending/result state beside its own button. It reports partial-board corner counts, stereo-depth rejection, or measured versus selected distance in both units. It disables duplicate presses while pending and times out when no fresh camera image arrives. The complete-board requirement and 5 cm distance tolerance are unchanged. Processing errors are reported without stopping the camera worker.
|
||||
|
||||
The 25 original observations and fitted geometry were backed up, then explicitly restored onto the same connected glasses after installation, retaining `verified=false`. The device still provides no serial; no cross-device or automatic unidentified-profile reuse was introduced. A live validation attempt on the updated app reported 3 of 24 common corners rather than silently failing. Physical-distance validation remains pending; the existing fit and captures remain available in the open setup session.
|
||||
|
||||
Validation: 36 targeted tests pass, including visible validation feedback, pending-button state, complete-board/depth rejection, dual-unit output, camera-to-center distance, and unchanged tolerance. The SDK-enabled APK builds with runtime freshness verification. The result layout was rendered and visually inspected. APK: `build/DroidDeck-calibration-distance-fix.apk`; SHA-256 `1f0b288fdcb76231ecc9f5178f6621799b01f0eb83b574c79394fa1a04bb5f47`. Logs and metadata: `build/spatial-distance-fix.log`, `build/spatial-distance-apk.log`, `build/calibration-distance-fit.json`, `build/calibration-distance-captures.json`, and `build/calibration-distance-after.json`.
|
||||
|
||||
## Near-distance detection fix (2026-10-15)
|
||||
|
||||
The live stereo preview showed the complete printed board at the user's reported 20-inch position, but the pattern occupied only about 70 pixels across in the original 640 by 480 images. One explicitly requested diagnostic pair was saved. OpenCV 4.13 detected 16 left-camera checkerboard corners in the raw image and none after image undistortion; the right image detected 3 raw and 1 corrected. Neither version supplied a complete common board. This identified both limited marker resolution and additional loss from resampling, rather than paper edges being out of view.
|
||||
|
||||
Distance validation now detects the original image pair and undistorts the detected coordinates before stereo reconstruction. Pinhole inversion uses an explicit convergence criterion. The UI defaults to 12 inches (0.3048 m), also offers 30 cm, and retains the 20-inch and 50-cm choices. Feedback distinguishes detected checkerboard corners from paper edges. The 24-common-corner, stereo reprojection/depth, and 5-cm physical-distance gates remain unchanged.
|
||||
|
||||
Validation: all 36 targeted spatial/calibration/UI tests pass. Synthetic forward-projection and reconstruction checks exercise pinhole and fisheye point correction at 0.3048, 0.508, and 0.8 m; these verify geometry calculations, not hardware accuracy. The SDK-enabled build and runtime freshness checks pass, and the rendered result layout was inspected. APK: `build/DroidDeck-calibration-near-distance.apk`, SHA-256 `cd7e144e4fabb2e94872e4a762f1a641a61773902fa8fe40ec2c8c05d778f81d`; the installed Fold7 APK matches. Build log: `build/spatial-near-distance.log`. Synthetic results: `build/spatial-raw-landmark-geometry.json`.
|
||||
|
||||
The same connected glasses' 25 accepted views and 1.12094-px fitted alignment were restored after installation without marking them verified. A subsequent user-triggered distance check reported **12.01 inches measured against 12.00 inches selected**, passing the unchanged validation gates and marking the calibration verified for this connection. Status evidence is in `build/calibration-near-distance-after.json`. Native head tracking and stereo synchronization remained valid at approximately 25 Hz. No optical hands were valid in that status sample; hand accuracy, externally measured latency, and game performance remain untested. Device identity is still unavailable, so automatic reuse across unidentified reconnects remains disabled.
|
||||
|
||||
## Calibration persistence and hand-controller investigation (2026-10-15)
|
||||
|
||||
After launching SteamVR, the app reported `calibrated=false`, zero restored views, and no optical observations despite valid native head tracking. Android USB did not expose a serial number, so the earlier calibration had stayed in memory. SDK 2.4.0's documented `xr_device_provider_get_sn_hash` returns a stable 32-byte board-serial digest. The adapter now reads it under the SDK ownership lock, rejects missing/zero digests, and uses it to key fitted profiles and captures. A nonblank Android USB serial remains a fallback. An unavailable identity still does not permit automatic cross-connection reuse.
|
||||
|
||||
The existing 25 views and fitted geometry were explicitly imported onto the same physical glasses under the SDK identity with distance validation still required. The user repeated the check: **12.13 inches measured against 12.00 selected**. The app saved `verified=true`, and subsequent app installations/restarts automatically recovered the verified calibration and all 25 views without debugger restoration. This resolves the lost-calibration blocker; it does not establish reliable optical controllers.
|
||||
|
||||
Hand diagnostics now distinguish missing calibration, synchronization/freshness failure, per-camera detections, ambiguous matching, rejected stereo geometry, and accepted hands. They retain the last completed result while inference runs, and expose preprocessing/inference timing. Observations over 100 ms old are cleared before publication; the driver and session freshness checks remain in place. CPU stereo inference runs concurrently with separate image buffers and waits for both tasks before frame reuse, including after one task fails. GPU inference remains on its owning thread. Calibrated undistortion maps are cached; comparisons against the original image-undistortion operations were pixel-identical for both cameras and both lens-model paths.
|
||||
|
||||
SDK 2.4.0 documents the tracking cameras at 25 Hz and provides no stereo frame-rate setter. Faster application polling cannot raise that rate. The source-rate calculation now accounts for skipped frame sequence numbers; its earlier readout fell toward inference throughput when frames were dropped.
|
||||
|
||||
Short live samples on the Fold7 do **not** pass the combined hand gate. In the final 12-sample CPU/SteamVR check, samples with detected hands took 87–137 ms to process, and no fresh hand was observed by the status reader. In the comparable GPU sample, hand-containing observations took 59–81 ms to process, with one accepted reconstruction but no fresh hand caught by the status reader. Earlier setup-only GPU samples did briefly report a fresh left hand. Stereo-depth rejection and observations exceeding 100 ms remain frequent. These are sparse internal timing samples with uncontrolled pose/thermal conditions, not externally measured latency, hand-validity percentages, or a controlled game-performance comparison. The user reported no visible controllers in SteamVR. An earlier 8-second guest-channel check found the 176-byte channel active with both feature flags set and all sampled heads fresh, but no fresh optical hands; both virtual controller devices were activated in the SteamVR log.
|
||||
|
||||
One hand-landmark pair was inspected from the running reconstruction method without saving camera images. All 21 points triangulated within the broad depth range, but nine failed the unchanged 3-pixel reprojection gate. The wrist's error was 12.80 px; several fingers were approximately 3–4.3 px. Reconstructed palm width was 6.15 cm and depths were 28–41 cm. This is a correspondence/reconstruction failure, not a missing shared-memory channel. The single sample does not isolate calibration residuals from per-view landmark error, and no quality thresholds were relaxed. Data: `build/hand-stereo-point-diagnostics.json` and `build/hand-stereo-geometry-report.json`. Debugger pauses used for this inspection are excluded from performance evidence. The temporary GPU benchmark was ended with CPU mode restored; neither delegate has passed the combined gates.
|
||||
|
||||
Validation: 51 targeted tests pass, including device-identity binding, concurrent stereo work, frame ownership after detector failure, calibration, gestures, and setup UI. SDK-enabled and SDK-free Android builds pass runtime freshness checks. Installed candidate: `build/DroidDeck-parallel-hands.apk`, SHA-256 `5e8ffa6a89a91e3b1b22fe289c126dca96a77dfb7645da551e8e97e13673c761`. Evidence: `build/spatial-parallel-hands.log`, `build/spatial-parallel-sdkfree.log`, `build/spatial-cached-undistortion.json`, `build/spatial-parallel-cpu-steamvr.json`, `build/spatial-cached-gpu-steamvr.json`, and `build/spatial-steamvr-hands-channel.json`. Calibration backup: `build/calibration-verified-profile.json` and `build/calibration-verified-captures.json`. The combined feature remains an experimental prototype.
|
||||
|
||||
## Hand-mode color regression and CPU rollback (2026-10-15)
|
||||
|
||||
The parallel CPU trial was removed after failing to improve live hand results. The serial worker retains cached lens maps, device-bound calibration persistence, and detailed rejection diagnostics. All 49 remaining targeted tests pass; the SDK-enabled APK builds with runtime freshness checks. Installed candidate: `build/DroidDeck-serial-hands.apk`, SHA-256 `bcefc87dabfde3d29f2018c2b39368d4e68cb015dd390385e3c80c74d86531c6`. Build log: `build/spatial-serial-rollback.log`.
|
||||
|
||||
The user reported choppy head turns with hands and positional tracking disabled, then requested that this investigation be deferred because the phone was charging. A thermal snapshot reported status 4 and a skin sensor reading of 49.4 degrees C. The rendering log recorded approximately 55.5–60 fps at a selected 60 Hz. The retained 1.22-second prefix of a truncated head-channel diagnostic showed pose changes roughly every 3.2–6.6 ms, with no sampled gap above 7.5 ms. This small sample is not a controlled performance acceptance result. Evidence: `build/tracking-regression-sample.json`, `build/tracking-regression-analysis.json`, and `build/tracking-regression-wayland.log`.
|
||||
|
||||
The HDR-like darkening is reproduced in screenshots of the glasses output, with brightness 6, Sunblock off, and valid head tracking unchanged. Camera inference enabled independently of optical-controller mode leaves normal colors. Enabling optical-controller mode with inference paused darkens midtones and shadows. A further test switching only `SpatialTracking.setHands`, bypassing the settings/menu/display-refresh path, also reproduces it. The temporary debugger overrides were removed after each comparison. This isolates the trigger to the SteamVR optical/untracked-controller path; it does not yet identify the faulty color-conversion operation. HDR output is disabled in the session. Evidence: `build/hand-color-comparison.json`, `build/hand-color-isolation.json`, `build/hand-color-mode-isolation.json`, and their `build/hand-color-*.png` screenshots. The calibration and 100-ms observation expiry remain unchanged.
|
||||
|
||||
The serial candidate also passes the SDK-free build (`build/spatial-serial-sdkfree.log`). Temporary `TU_DEBUG=noubwc` did not remove the darkening, although the user changed display modes between that pair's screenshots. A subsequent `TU_DEBUG=gmem` comparison was interrupted by graphics recovery and is inconclusive. Neither diagnostic setting was retained: the following session's actual `vrserver` and `vrcompositor` environments both report an empty `TU_DEBUG`, and pending, active, and persistent agent overrides are empty (`build/hand-color-current-flags.json`). No gamma compensation or hand-validity workaround was added.
|
||||
|
||||
The user's observation that SBS felt smoother coincided with 2D running at 60 Hz and SBS at 90 Hz. A later 120-Hz 2D request still reported a 60-Hz physical mode; its failure cause remains undetermined. The next 90-Hz 2D selection succeeded: Android reports 1920 by 1200 at 90 Hz, and the SDK reports mode `0x43`. The app therefore has no fixed 60-Hz limit for 2D. 2D and SBS retain separate refresh preferences; actual display mode must be checked independently of the requested value. Evidence: `build/hand-color-display-modes.txt` and `build/refresh-mode-final.json`. This confirms the active refresh mode, not sustained 90-fps game rendering or a thermal/performance acceptance result.
|
||||
|
||||
## Color fix, original-image detection, and late synchronization (2026-10-16)
|
||||
|
||||
The HDR-like color change is fixed and confirmed by the user in the glasses. `NewSharedVulkanImage` requested compute access for compositor eye textures even though DroidDeck only needs rendering and sampling. SteamVR replaced the requested `R8G8B8A8_SRGB` format (43) with `R8G8B8A8_UNORM` (37). The untracked-controller compositor path then produced dark linear midtones, while its multisampled controller path encoded colors differently. Removing compute access preserves format 43 in both paths. Temporary Vulkan tracing established the before/after formats and was removed from the delivered build; the presenter retains its existing byte-preserving UNORM import. No gamma compensation was introduced. Evidence: `build/color-format-fixed-trace.json`, `build/color-format-fixed-states.json`, and the corresponding on/off output images. The user reported that colors now stay the same when toggling Hand controllers.
|
||||
|
||||
One explicitly authorized stereo image pair showed upright camera images and two visible open palms. Replaying it with the exact Android CPU detector found one left hand in each camera. All 21 landmarks passed the unchanged 3-pixel reprojection gate (maximum 2.227 px), with reconstructed palm width 7.31 cm and depth approximately 32–39 cm. The right hand was not detected consistently; lower confidence thresholds and exploratory crops did not establish an improvement and were not adopted. Hand detection now consumes the original grayscale images converted to RGBA and corrects landmark coordinates before triangulation, avoiding full-image resampling. This verifies one reconstructable sample, not sustained detection accuracy or two-hand operation. Evidence: `build/android-hand-pair-replay.json`, `build/android-hand-pair-geometry.json`, and `build/hand-pair-replay.json`.
|
||||
|
||||
Previously, camera acquisition waited up to 35 ms for a matching VIO callback before starting hand inference. Inference now starts with the newest pair and checks the exact source timestamp again after detection, allowing VIO processing to overlap detection. Publication still requires a matching callback within 5 ms, a nearby canonical head-history sample, unchanged connection and tracking frame, healthy current head tracking, and an observation no more than 100 ms old. The callback's undocumented pose basis remains unused. Calibration and preview-only acquisition retain their bounded wait. Four additional regression tests cover late confirmation with the original capture pose, missing confirmation, expiry, tracking loss/discontinuity, reconnects, and unavailable history.
|
||||
|
||||
Validation: 53 targeted Android tests pass with no failures or skips; SDK-enabled and SDK-free Android builds and runtime freshness checks pass. The ARM64 VR build passes its native layout, freshness, optical input recovery, render-target, synchronization, and scheduling checks. Temporary color tracing is absent from the final driver source and packaged runtime. Installed SDK-enabled artifact: `build/DroidDeck-spatial-color-sync.apk`, SHA-256 `885fb14b5a30ac65a255a6b0dd519d9f73b72489c08a8b74e1756673b86812ea`. Logs: `build/color-fix-final-native.log`, `build/color-fix-final-stage.log`, `build/spatial-late-sync-build.log`, `build/spatial-late-sync-sdkfree.log`, and `build/spatial-color-sync-install.json`.
|
||||
|
||||
During the subsequent grey-screen report, the compositor continued presenting approximately 60 frames per second, but the SDK returned unstable head status 1 and the shared head pose was invalid. USB timeout messages also appeared; their causal relationship to tracking loss is not established. The clean app/provider restart preserved all 25 calibration views and the verified profile. After tracking recovery and software recentering, the user confirmed both “6DoF active” and the visible SteamVR dashboard. Positional tracking was briefly disabled for diagnosis and restored. No synthetic valid pose or silent fallback was added. Evidence: `build/grey-launch-guest.json`, `build/grey-launch-app.log`, and `build/spatial-color-sync-recovered.json`.
|
||||
|
||||
The next 16-second left-palm test sampled the guest spatial channel 2,900 times: every head was fresh, but no optical controller pose was fresh. Six status samples showed one matched hand in both cameras and successful stereo reconstruction; inference took 139.7–174.3 ms and total processing 144.1–181.8 ms. Those observations correctly expired before publication. The other three status samples were waiting for synchronization. The app restarted between preparation of a temporary GPU benchmark and this measurement, so the active delegate was not verified in the new process and this is not a controlled GPU comparison. A later thermal snapshot reported status 4 and skin readings up to 51.0 degrees C; its contribution to the measured slowdown is unquantified. Evidence: `build/spatial-live-hands-left.json`, `build/spatial-live-hands-left.log`, and `build/spatial-hand-thermal.txt`. The glasses subsequently disconnected, preventing another live comparison. The next measurement needs a stable process/connection, explicit delegate verification, and controlled thermal conditions. Hand controls remain an experimental prototype and have not passed the combined release gate.
|
||||
|
||||
## Tracking-loss recovery screen (2026-10-16)
|
||||
|
||||
Active positional SteamVR and virtual-screen sessions now show a DroidDeck-themed recovery screen over the rendered output when head tracking becomes invalid. Separate states explain how to recover tracking, reconnect USB, recenter a changed origin, or hold still while recentering settles. The screen is rendered independently of the head pose, supports mono and SBS output, and closes automatically after valid tracking returns. The phone retains access to Recenter and glasses setup. No tracking validity, input-expiry, calibration, or driver behavior was relaxed.
|
||||
|
||||
Validation: 65 targeted Android tests pass without failures or skips, including callback-free stale-pose detection, recovery states, presentation surface preservation, mono/SBS rendering, and theme colors. Robolectric native-rendered previews were visually inspected at 1080 by 600 and 3840 by 1200; this is automated rendering evidence, not a physical glasses acceptance check. SDK-enabled and SDK-free Android builds and packaged-runtime freshness checks pass. Logs and previews: `build/tracking-recovery-build.log`, `build/tracking-recovery-sdkfree.log`, and `app/build/reports/tracking-recovery/`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-tracking-recovery.apk`, SHA-256 `2170c2e513d45af048b9aba01b76a694d333ba9848a91021a47d413fca3d282d`. Installation preserved app data and was performed with the session idle; SteamVR was not started. The installed APK hash matches the archived artifact (`build/tracking-recovery-install.json`). A live tracking-loss/recovery check through the glasses remains pending.
|
||||
|
||||
## Cooled-phone retest and visible left controller (2026-10-16)
|
||||
|
||||
The Fold7 initially reported thermal status 1, approximately 40.3 degrees C skin temperature, healthy USB access, and the saved 25-view calibration. Head tracking recovered to valid status 0. CPU inference was verified through the live setup control in process 17409, session 12. By the 16-second left-palm measurement, thermal status had returned to 4. All nine diagnostic samples detected and reconstructed a matching hand in both cameras, but inference took 124.4–173.3 ms. The guest channel had 3,038 fresh head reads and zero fresh hand reads. This does not establish an unthrottled CPU baseline. Evidence: `build/cooled-cpu-verified-ui.json`, `build/spatial-live-hands-cooled-cpu.json`.
|
||||
|
||||
Enabling GPU inference while that SteamVR session was running initialized both GPU graphs, including OpenCL loading, at 01:10:24–01:10:27 device-local time. The compositor first reported `vkerror=-4` at 01:10:29.516; automatic graphics recovery then killed and restarted the app. Process 27927, session 13, was back on CPU. The subsequently captured file named `build/spatial-live-hands-cooled-gpu.json` therefore contains CPU observations after recovery and must not be counted as a GPU comparison. Process IDs, session IDs, the setup control returning to Off, and Android exit records establish that reset. The timing associates GPU initialization with the device-loss event but does not establish its underlying driver cause. Evidence: `build/cooled-before-restart-logcat.txt`, `build/cooled-session12-vrcompositor.txt`, `build/cooled-session12-events.jsonl`, `build/cooled-exit-info.txt`, `build/cooled-gpu-after-ui.json`.
|
||||
|
||||
With SteamVR stopped, GPU inference was enabled and verified in process 27927. Twelve diagnostic samples then measured 10.3–14.1 ms inference and 12.0–19.7 ms total observation age, with healthy head tracking. These samples were not a prescribed palm-hold accuracy test; most saw a hand in only one camera and none produced accepted geometry. Evidence: `build/cooled-idle-gpu-verified-ui.json`, `build/cooled-idle-gpu-sample.json`.
|
||||
|
||||
Launching SteamVR after GPU initialization preserved process 27927 for session 14. During a fresh 16-second left-palm hold, the guest channel received 67 distinct valid left-hand observations. The left hand was fresh in 315 of 2,864 coherent channel reads (11.0%); all heads were fresh. This percentage measures sampled channel availability, not the acceptance target's percentage of camera observations. The user confirmed a controller appeared or flickered. Eight diagnostic samples all reconstructed the left hand, but inference varied from 41.8–101.6 ms and observation age from 72.5–131.9 ms. Thermal status remained 4, with reported skin temperature increasing from 49.1 to 51.3 degrees C. The session also logged another Vulkan device-loss event, so initializing GPU inference before SteamVR did not resolve stability. Evidence: `build/spatial-live-hands-prewarmed-gpu.json`, `build/cooled-prewarmed-start.json`, `build/cooled-session14-events.jsonl`.
|
||||
|
||||
This confirms a visible optical left-controller path, with no relaxed timestamp or geometry checks. Both-hand reliability, gesture operation, measured end-to-end latency, and sustained performance remain unvalidated. No camera images were saved during these tests, and calibration was not changed. The run was stopped after the repeated graphics fault. After confirming the session was idle and its artifacts complete, DroidDeck was closed to stop background inference and clear the temporary GPU benchmark; saved settings and calibration were retained. The next work must address GPU/SteamVR coexistence and inference latency under sustained thermal load; another calibration is not indicated by these successful left-hand reconstructions.
|
||||
|
||||
## Requested 200 ms experimental hand cutoff (2026-10-16)
|
||||
|
||||
The user requested accepting hand observations up to 200 ms old to reduce disappearing controllers while inference is slow. The cutoff now applies consistently to inference admission/completion, capture-pose lookup, Android publication, and SteamVR optical pose/input freshness. Observation timestamps are unchanged. Head freshness remains 100 ms, and detected loss, changed tracking frames, reconnects, and recentering still invalidate hands immediately. Missing observations can now retain trigger/grip until the 200 ms expiry; reacquisition still requires an observed release. This intentionally changes the earlier 100 ms hand-expiry requirement, without claiming lower processing latency or resolving the Vulkan faults. The diagnostics field `handMaxAgeMs` reports the active limit.
|
||||
|
||||
Validation: 73 targeted Android tests pass, including a 150 ms inference result using its original capture pose, the inclusive 200 ms boundary, rejection immediately after that boundary, unchanged 100 ms head expiry, and independent hand timestamps. Native tests pass the same hand/head boundaries, pose validity, stale-input release, and reacquisition behavior. The ARM64 driver was rebuilt, staged, and verified against source/output checksums; SDK-enabled and SDK-free Android builds pass. Logs: `build/hand-200ms-native.log`, `build/hand-200ms-stage.log`, `build/hand-200ms-build.log`, `build/hand-200ms-sdkfree.log`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-200ms.apk`, SHA-256 `dac9296cbc964fca87b65b7e1e8b87276deaaeb5d90ca731d47eae673e22c64d`. Its packaged driver matches the rebuilt ARM64 binary, SHA-256 `bf24667f9f288cd07efe6e100c3b237b1e68b734100ae99cf93974538ce3bc6a`. The installed APK hash and live `handMaxAgeMs: 200` were verified. Calibration and saved settings were preserved. SteamVR was not launched, and the app was closed after verification to stop background inference (`build/hand-200ms-install.json`). Live controller availability, gesture delay, and release behavior with the new cutoff remain unmeasured.
|
||||
|
||||
## GPU default requested by the user (2026-10-16)
|
||||
|
||||
The user explicitly requested GPU inference as the default. Every new stereo hand engine now selects GPU, including after app restart, USB reconnect, and provider restart. The setup control is renamed **GPU hand inference** and explains that disabling it selects CPU until the next tracking restart. Diagnostics distinguish `handGpuEnabled` (selection) from `handInferenceBackend` (GPU/CPU only after both detectors initialize, otherwise `not initialized`). This replaces the earlier CPU-default policy; it does not resolve the measured Vulkan failures or constitute performance acceptance. The 200 ms hand cutoff and gesture mappings are unchanged.
|
||||
|
||||
Validation: 73 targeted Android tests pass, including pinch/grip mapping, expiry, tracking recovery, and setup UI. SDK-enabled and SDK-free builds and runtime freshness checks pass. The packaged driver still matches the previously tested 200 ms ARM64 binary. The active SteamVR test was stopped cleanly before installation. GPU selection and the verified calibration were confirmed in two distinct app processes, 22127 and 23157, separated by a full app restart. Head tracking was valid, but hand diagnostics reported waiting for camera/head synchronization and the detectors reported `not initialized`; these checks verify default selection and profile persistence, not a new live GPU inference or gameplay result. SteamVR was not relaunched.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-gpu-default.apk`, SHA-256 `a32feb647b0ca9fb6f62bd31158450b58e5671de7c1b0324f6b0b56bd4c92a47`. The installed APK hash matches. Evidence: `build/hand-gpu-default-build.log`, `build/hand-gpu-default-sdkfree.log`, `build/hand-gpu-default-install.json`. Click instructions are documented: present an open hand, aim the virtual controller, pinch thumb and index finger to press, and separate them to release. Curling the middle, ring, and little fingers supplies grip; tracking loss requires another observed open/released gesture before input can resume.
|
||||
|
||||
## CPU default restored and model optimization investigated (2026-10-16)
|
||||
|
||||
After further SteamVR restarts, the user requested reverting to CPU. New engines, the no-engine diagnostics fallback, and the setup UI now default to CPU again. The optional GPU toggle explains the observed restart risk and resets on tracking restart. Calibration, the color fix, and the requested 200 ms hand cutoff are retained. GPU device loss has not been fixed or ruled out under other configurations.
|
||||
|
||||
Validation: 73 targeted Android tests pass with no failures, errors, or skips. SDK-enabled and SDK-free builds pass; the SDK-free APK excludes the optional VITURE bridge. The SDK-enabled APK's packaged ARM64 driver matches the previously tested 200 ms binary. CPU selection (`handGpuEnabled: false`) and the verified 25-view calibration were confirmed in two distinct processes, 3540 and 3717, across a full app restart. Head tracking was lost and hand detectors were `not initialized` during these checks, so this verifies the restored default and preserved calibration, not new live CPU performance or SteamVR stability.
|
||||
|
||||
Installed artifact: `build/DroidDeck-hand-cpu-restore.apk`, SHA-256 `5ae8f702fdce16501a69731fd5b0e718198e267276e918322b15228ae3b95282`. The installed APK hash matches. Driver SHA-256 remains `bf24667f9f288cd07efe6e100c3b237b1e68b734100ae99cf93974538ce3bc6a`. Evidence: `build/hand-cpu-restore-build.log`, `build/hand-cpu-restore-sdkfree.log`, and `build/hand-cpu-restore-install.json`. SteamVR was not launched for this verification.
|
||||
|
||||
The shipped bundle already uses float16 weights. This does not establish float16 CPU arithmetic or a speed advantage: Google's [float16 documentation](https://developers.google.com/edge/litert/conversion/tensorflow/quantization/post_training_float16_quant) describes CPU weight expansion to float32. [INT8 quantization](https://developers.google.com/edge/litert/quantization/model_optimization) is a possible separate experiment, subject to runtime/operator compatibility and stereo accuracy validation; no INT8 conversion or speed claim has been made.
|
||||
|
||||
An offline feasibility experiment compared Google's [legacy lite palm and hand landmark models](https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/models.md) against the installed full bundle using the previously authorized stereo image pair. No new images were captured. The lite models initially failed the Tasks loader because normalization metadata was absent. For this local experiment, input/output tensor names, shapes, and types were checked against the full models before copying their metadata and associated files. The 0–1 input normalization agrees with Google's [hand landmark graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/modules/hand_landmark/hand_landmark_cpu.pbtxt) and [palm graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/modules/palm_detection/palm_detection_cpu.pbtxt). Experimental bundles remain under `build/hand-lite-research/`; the app still ships the original checksum-pinned full bundle.
|
||||
|
||||
Windows CPU replay used MediaPipe 0.10.21, two sequential VIDEO instances, two-hand capacity, unchanged 0.6 confidence thresholds, 10 warmup pairs and 30 timed repetitions of one static pair. Times cover stereo inference only, excluding camera acquisition, preprocessing, geometry, rendering, and Android/game contention.
|
||||
|
||||
| Models | Median stereo inference | p95 stereo inference | Static-pair geometry |
|
||||
|---|---:|---:|---|
|
||||
| Full baseline, before alternatives | 84.54 ms | 97.28 ms | Left hand passes in 30/30 repetitions |
|
||||
| Lite palm + full landmarks | 37.11 ms | 43.94 ms | No hands detected; unusable for this sample |
|
||||
| Full palm + lite landmarks | 62.48 ms | 73.18 ms | Left hand passes in 30/30 repetitions |
|
||||
| Both lite | 36.75 ms | 43.69 ms | No hands detected; unusable for this sample |
|
||||
| Full baseline, after alternatives | 87.99 ms | 100.15 ms | Left hand passes in 30/30 repetitions |
|
||||
|
||||
Full palm plus lite landmarks is approximately 26–29% faster by median in this limited desktop replay. Its first-frame maximum reprojection error is 1.83 px versus 2.23 px for full; this single-pose result is not an accuracy comparison across real interactions. Repeated images do not validate motion, tracking availability, both hands, reacquisition, or thermal performance. Android Tasks 1.0.0 compatibility and temperature-matched Fold7/SteamVR measurements are still required before adopting it. The apparent speed of the lite-palm variants is not useful evidence of faster working tracking because they performed no landmark inference on a detected hand.
|
||||
|
||||
Research artifacts: `build/compare-hand-lite.py`, `build/hand-lite-research/results.json`, `build/hand-lite-research/without-metadata.json`, and `build/hand-lite-research-metadata.log`. Original downloaded SHA-256 values: palm lite `e9a4aaddf90dda56a87235303cf00e4c2d3fb28725f68fd88772997dac905c18`; landmarks lite `d7fde8ac11f8ce03f8663775bfc323f4fc9f2a38062b4f4efa142874ef5b2a48`.
|
||||
|
||||
Another candidate is reducing repeated palm searches without losing second-hand acquisition. The [upstream HandLandmarker graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc) runs detection whenever tracked hands are fewer than the configured maximum, including one visible hand with a two-hand limit. This is a profiling lead, not a measured attribution in the packaged Android runtime. Simply setting the limit to one would remove the required two-hand behavior; implementing a controlled search cadence would require a different graph or pipeline. The existing bounded latest-frame queue and VIDEO tracking already avoid unbounded backlog and unconditional IMAGE-mode detection.
|
||||
|
||||
## Brief visual hold and remaining detection gaps (2026-10-16)
|
||||
|
||||
After the user reported recovery following cooling, a 16-second CPU sample kept the same Android process (10308) and SteamVR session (`2026-10-16-29-steam`). Head tracking was fresh in 2,851 of 2,857 channel samples (99.8%), but the left hand was fresh in only 22 samples (0.77%), representing six distinct poses. Those poses first reached the monitor 162.2–198.3 ms after observation. Right-hand output was absent during the requested left-hand-only test. Diagnostic snapshots alternated between camera/head synchronization waits and no detected hands; measured inference in the snapshots that ran detection ranged from 76.1 to 136.8 ms. This is not a controlled cool-device benchmark: at the end Android reported thermal status 4 and the current HAL SKIN sensor read 51.5 C. Evidence: `build/spatial-live-hands-cooled-flicker.json` and `build/cooled-flicker-measure.log`.
|
||||
|
||||
The driver previously invalidated its rendered controller immediately whenever an inference result contained no hand, even if the previous observation was still younger than the requested 200 ms cutoff. It now caches only the most recent accepted optical transform for visual continuity through those short gaps. Expiry remains measured from the original camera observation; a missed result never extends it. Optical input processing remains separate and releases trigger/grip immediately on a miss. Reacquisition still requires a released gesture. Head loss, connection/origin changes, disconnect, and switching back to gamepad-positioned controllers clear the cache. Missing optical tracking no longer suppresses independently available gamepad menu and locomotion inputs. This is a partial flicker mitigation, not a remedy for the longer detection gaps or old poses measured above.
|
||||
|
||||
Native regressions pass for immediate gesture release during a held visual pose, its unchanged timestamp and inclusive 200 ms boundary, rejection after expiry, rearming, head loss/recovery, origin/connection changes, disconnect, mode changes, malformed/future/out-of-order samples, and gamepad availability during optical loss. The other native rendering, fence, and scheduling regressions pass. SDK-enabled and SDK-free Android packaging and runtime source/output freshness checks pass. Android application source was unchanged in this step; the existing 73 passing Android test results were checked during installation rather than rerun. Evidence: `build/hand-visual-hold-native.log`, `build/hand-visual-hold-stage.log`, `build/hand-visual-hold-build.log`, and `build/hand-visual-hold-sdkfree.log`.
|
||||
|
||||
Installed artifact: `build/DroidDeck-hand-visual-hold.apk`, SHA-256 `977b993433803f3d8abbd639ca636b874367467866b58f7af75a07a46b20daa2`. Its packaged ARM64 driver matches the tested build, SHA-256 `2d7ff8647f210248f6ea4b68e9b05e21ed7ebcc74da601da0f1d854067b7ab82`. Installation occurred with the session idle and preserved the calibration/profile files. CPU selection and the 200 ms setting were confirmed across two app processes. The glasses were disconnected during these checks, so calibration was not loaded and detectors were not initialized; this does not establish a live improvement. SteamVR was not relaunched. Evidence: `build/hand-visual-hold-install.json` and `build/hand-visual-hold-live.json`.
|
||||
|
||||
The user explicitly authorized one new diagnostic stereo pair and separately authorized copying that pair and timing metadata to the local workspace. Both 640 by 480 images visibly contain the palm, near the outer edge of the left image. The saved pair was not synchronized to a head pose. Desktop MediaPipe 0.10.21 replay of the current full model failed to detect either hand at palm-detection thresholds 0.6, 0.5, 0.4, and 0.3, with landmark-presence confidence held at 0.6. Undistorting the images before detection did not recover this sample. Lowering both confidence thresholds to 0.3 also failed. A fixed right-side image tile detected a left hand in only one camera, insufficient for stereo reconstruction. The earlier authorized pair still detected the left hand in both cameras and passed the unchanged stereo geometry gate in the same replay. These static diagnostic experiments do not justify changing thresholds, shipping a fixed crop, or claiming general detection improvement. No inference model, thresholds, or stereo quality gates were changed. Local artifacts: `build/flicker-pair-replay.json`, `build/flicker-pair-scale.json`, and their scripts/logs.
|
||||
|
||||
The earlier CPU crash investigation also observed Vulkan device loss with hand inference disabled. A temporary `TU_DEBUG=noubwc` session reproduced it and was stopped; the flag was not persisted and no production graphics change was made. Cooling may have contributed to the user's reported recovery, but neither overheating nor inference delegate choice is established as the sole crash cause.
|
||||
|
||||
## Half-rate stereo input requested by the user (2026-10-16)
|
||||
|
||||
The supplied SDK 2.4.0 Carina header documents 25 Hz camera/pose callbacks and exposes automatic/manual exposure controls, but no tracking-camera frame-rate setter. DroidDeck now selects every other callback for its own stereo pixel copies while hand controllers are enabled, yielding approximately 12.5 pairs per second with the observed 25 Hz input. The gate executes before native buffer resize/copy. Skipped pairs do not reach JNI transfer, grayscale/RGBA conversion, or hand inference. Vendor camera acquisition/VIO, pose callbacks, and the existing head polling interval are unchanged. This is software input decimation, not a firmware camera-rate change.
|
||||
|
||||
Calibration collection, distance validation, explicit diagnostic capture, and preview-only use bypass the half-rate gate. The first frame after a gate-mode change or camera-source reset is accepted. The latest eligible pending pair still replaces older eligible pending work. Camera-source/receipt timestamps and the existing 200 ms hand expiry are unchanged. JNI metadata carries the incoming frame index associated with the copied pair and the cumulative intentional skip count, so diagnostics continue to report the true incoming camera rate. `cameraFrameDivisor: 2` identifies hand input at half rate, and `rateLimitedFrames` is separate from processing-related `dropped` frames. The setup UI explains both rates and counters.
|
||||
|
||||
The native regression passed 250 source callbacks producing 125 copied pairs, full-rate setup, repeated configuration, source reset, invalid timestamps, and reconnect reset. All 75 targeted Android tests passed, including half-rate camera-clock mapping and UI separation of incoming rate versus intentionally skipped frames. All 13 runtime-preparation tests passed, including classifying the new native helper as locally compiled while rejecting unrelated donor-native reuse. SDK-enabled and SDK-free builds and runtime freshness checks passed. Tests: `tools/tests/test_stereo_frame_gate.cpp`; logs: `build/hand-half-rate-build.log`, `build/hand-half-rate-sdkfree.log`, `build/hand-half-rate-runtime-tests.log`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-half-rate.apk`, SHA-256 `c02d56e40802de5c5e9d9750234b24a85462652f472729fbc9696935654f95fb`. The installed hash was verified. The SteamVR driver is unchanged from the tested visual-hold build (`2d7ff8647f210248f6ea4b68e9b05e21ed7ebcc74da601da0f1d854067b7ab82`); the Android SDK bridge and app were rebuilt. Installation preserved app data and occurred with SteamVR stopped. CPU selection and the 200 ms expiry were confirmed across two processes. The glasses were disconnected, so the live half-rate cadence, inference cost, thermal benefit, and controller availability have not been measured. Evidence: `build/hand-half-rate-install.json`.
|
||||
|
||||
At 12.5 Hz, eligible pairs are approximately 80 ms apart. The earlier CPU inference snapshots already took 76–137 ms, so reducing input is not expected to halve total CPU load. It may save image-copy/preprocessing work and reduce inference load when the worker is fast enough, while also increasing frame waiting time and potentially reducing chances to recover a missed detection. It does not guarantee a successful observation or a visible response within 100 ms. A controlled comparison at matching temperatures is required before claiming smoother controllers or less overheating.
|
||||
|
||||
## SteamVR shared-eye image import correction (2026-10-16)
|
||||
|
||||
After the half-rate installation, sessions 31 and 32 hit `VK_ERROR_DEVICE_LOST` approximately 11.7 and 10.8 seconds after READY. DroidDeck's Android process and native 6DoF remained alive; CPU inference, a camera divisor of 2, approximately 25 Hz incoming cameras, and intentionally skipped frames were confirmed. This establishes live operation of the input limiter, but no thermal/performance benefit. The Android crash buffer contained an older VITURE abort in another process, not a new app crash. Evidence: `build/half-rate-crash-state.json`, `build/half-rate-crash-glasses.json`, and `build/half-rate-crash-guest.json`.
|
||||
|
||||
A temporary `DROIDDECK_VR_IMPORT_FORMAT_LIST=0` test ran 49 seconds without device loss with hands disabled. A second test with CPU hands was stopped after the user reported a grey, degraded image. The override is rejected and was not persisted. Restoring the normal import path reproduced device loss in session 35. Neither a successful launch nor removal of image-format metadata is an acceptable fix.
|
||||
|
||||
The normal-path Vulkan trace identifies an external-memory image-description mismatch: SteamVR exports each 1728 by 1080 eye image as format 43 (sRGB), flags `0x8`, usage `0x97`, with view formats 43/37, whereas the presenter imported it as format 37 (UNORM), flags `0x8`, usage `0x85`. Dedicated OPAQUE_FD imports require matching image descriptions. The presenter now retains the exported format, rendering/transfer usage, and view-format list. Presentation selects the corresponding sRGB or UNORM surface format, so an sRGB blit both decodes and re-encodes instead of darkening the display. It rejects incompatible surface encodings rather than silently selecting one, and checks imported-memory binding failures. The existing SteamVR compute-access color fix remains intact. Trace: `build/crash-import-trace/guest.json`. Requirement: [Vulkan dedicated allocation, VUID 01878](https://docs.vulkan.org/refpages/latest/refpages/source/VkMemoryDedicatedAllocateInfo.html).
|
||||
|
||||
The ARM64 build and native regressions pass, including a new regression against the captured SteamVR image description and display-format selection/rejection cases. SDK-enabled and SDK-free APK builds and runtime freshness checks pass. The prior 75 Android tests remain applicable; Android source was unchanged for this graphics correction. Installed SDK-enabled artifact: `build/DroidDeck-shared-eye-format.apk`, SHA-256 `c573169cd46b20b70d704cd92f9f07550d5aa128d0c4300c1c52e635f04958fb`; driver SHA-256 `51dbd719e56c43df6c37edef2d7f426e91183b2b8489285349df19c322090f3b`. Installation preserved the verified calibration, CPU inference, and 200 ms cutoff across two app processes. Logs: `build/shared-eye-format-{native,stage,build,sdkfree,install}.log` and `build/shared-eye-format-install.json`.
|
||||
|
||||
Two subsequent sessions completed 121.9 and 122.7 seconds after READY without a new device-loss event. All sampled head poses were valid, CPU inference stayed active, frame processing advanced, and Android PID 659 remained unchanged. Session 36 initially allocated 1080 by 1080 eyes; its trace confirms identical exported/imported format 43, usage `0x97`, and formats 43/37. Session 37 ran without tracing and allocated the original 1728 by 1080 eyes, removing the lower-resolution confound. Both used a 3840 by 1200 sRGB presentation surface (format 50) and showed approximately 600 frames per ten-second interval after startup. The cumulative compositor log contains older device-loss messages; filtering to the new sessions found no fresh device-loss error. Occasional `WaitForAcquire` warnings remain. Android thermal status reached 4, with current HAL skin temperatures 46.7 and 48.4 degrees Celsius after the respective runs; these are not controlled thermal comparisons.
|
||||
|
||||
These are startup stability checks, not a twenty-minute gameplay/performance pass or proof of every crash's cause. Both test sessions were stopped; all temporary environment overrides and the trace marker were cleared, with calibration and CPU hand settings verified afterward. Evidence: `build/shared-eye-format-live/`, `build/shared-eye-format-relaunch/`, `build/shared-eye-log-audit.json`, and `build/shared-eye-final.json`.
|
||||
|
||||
The user subsequently confirmed normal colors but reported another crash as soon as a hand appeared. Session 38 logged fresh Vulkan device loss at 08:44:25 UTC, about 42 seconds after READY, with a burst of geometry pipeline creation immediately beforehand. Recovery stopped the session, and Android later recorded SIGKILL for the app process; the initial failure was in SteamVR's compositor. The color correction is accepted, but the crash is not fixed. In session 39, the dashboard remained alive with valid head tracking and no detected optical hands, then failed less than four seconds after switching to the existing gamepad controller poses at 20.47 seconds. That reproduces the controller-visibility trigger independently of reconstructed hand coordinates. Evidence: `build/hand-visible-crash/`, `build/hand-visible-exits.txt`, and `build/controller-visibility/`. The preceding two-minute tests did not validate this transition.
|
||||
|
||||
## Visible-controller crash and compositor completion guard (2026-10-16)
|
||||
|
||||
An opt-in diagnostic waited for the compositor's graphics queue after each submission. Session 40 repeated the same optical-to-gamepad-controller transition at 18.01 seconds and stayed alive through 80.94 seconds, with approximately 600 presented frames per ten-second interval. Session 39 without the diagnostic failed less than four seconds after this transition. This supports a GPU synchronization problem in the visible-controller rendering path; it does not establish a specific bad hand sample or thermal cause. An [upstream SteamVR report](https://github.com/ValveSoftware/SteamVR-for-Linux/issues/952) describes premature command-buffer/descriptor-pool reuse, but equivalence to this device's failure is not directly proven. Diagnostic evidence: `build/controller-submit-idle/` and `build/submit-idle-install.json`.
|
||||
|
||||
The shipped experimental guard applies only inside `vrcompositor`. Each `vkQueueSubmit` or `vkQueueSubmit2` waits for its actual completion fence before returning. If SteamVR supplied its own fence, that fence remains attached to the original submission and a following empty submission signals DroidDeck's separate fence. Waiting is bounded to one second; timeout fails the submission path instead of reporting success. The tracker keeps the fence reserved during this wait, preventing the completion worker from resetting/reusing it; a timed-out but submitted fence stays pending until completion or device failure. Other processes retain asynchronous submission. This is a conservative workaround that reduces CPU/GPU overlap, so gameplay frame-time and long-duration acceptance remain required. The temporary `DROIDDECK_VR_SUBMIT_IDLE` probe was removed.
|
||||
|
||||
Native regressions cover both submission APIs, caller-owned fences, submission/marker failures, successful completion, device loss, timeout retention, tracker allocation failure, and bypass outside the compositor. Existing native rendering, spatial, optical-controller, synchronization, and scheduling checks also pass. SDK-enabled and SDK-free builds and runtime freshness checks pass; the prior 75 Android tests apply because Android source was unchanged. Installed APK: `build/DroidDeck-compositor-fence.apk`, SHA-256 `27d44a138d1f6bf6a0698abcd0a48107d31815c8e57a9f7ca07113fabb213195`. Installation preserved the verified calibration, CPU inference, 200 ms hand expiry, and half-rate camera input. Evidence: `build/compositor-fence-{native,stage,build,sdkfree,install}.log` and `build/compositor-fence-install.json`.
|
||||
|
||||
Session 42 repeated the controller-visibility regression on the installed bounded-fence build. Head tracking remained valid; switching to the existing gamepad controller poses at 17.55 seconds did not cause device loss through the final sample at 58.92 seconds. The presenter reported 593–601 frames per ten-second interval, with no late frames in those counters. Optical hands were absent before the switch, so this passes the reproduced synthetic-controller visibility trigger but does not yet validate an actual optical hand appearing. The successful session was left open with CPU optical tracking restored for that check. Evidence: `build/controller-fence-guard/` and `build/controller-fence-guard.log`.
|
||||
|
||||
The built, packaged, and installed Vulkan layers all have SHA-256 `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`; no pending or persistent diagnostic environment override remained. The live audit was repeated after the session ended and still confirmed the installed layer, saved calibration, CPU selection, and enabled hand-controller setting. Evidence: `build/compositor-fence-live-audit.json`.
|
||||
|
||||
## Optical appearance confirmed; flicker and pinch remain unresolved (2026-10-16)
|
||||
|
||||
The user confirmed that optical controllers appeared without freezing, but continued to flicker and could not click. This is a successful user-observed optical appearance on the compositor-fence build, not a sustained tracking or interaction pass. The color correction remains accepted.
|
||||
|
||||
A subsequent 16-second CPU baseline in session 43 sampled 2,846 coherent channel snapshots, all with fresh head tracking and none with a valid hand. The requested deliberate open/pinch/release sequence had not yet been confirmed, so absence of hands in this baseline cannot establish detector failure on a visible palm. Diagnostic samples measured 111.7–154.1 ms stereo inference; three observations exceeded the unchanged 200 ms cutoff, with completion ages 204.9–221.5 ms. Android stayed in process 32010 and the same READY SteamVR session. Thermal status was 4, with current HAL skin temperature 51.8 C at the end; this is not a controlled cool-device performance comparison. Evidence: `build/spatial-live-hands-pinch-baseline.json` and `build/pinch-baseline.log`.
|
||||
|
||||
The driver emits `/input/trigger/value` and `/input/trigger/click` from optical pinch. Any invalid/stale hand or head sample releases inputs and disarms gestures until an observed open/released hand, while the visual pose may remain only until its original observation reaches 200 ms. Repeated observation loss can therefore interrupt clicking as well as controller visibility. No deliberate pinch was measured in the new baseline, so this is an explanation of the implemented behavior, not proof of the user's missing-click cause. The next measurement records hand timestamps, validity transitions, trigger strengths, grip, and connection/origin changes without camera images (`build/measure-pinch.py`). No inference thresholds, model, expiry, delegate default, or graphics behavior were changed in this follow-up.
|
||||
|
||||
## Palm continuity and independent gestures implemented (2026-10-16)
|
||||
|
||||
Hand reconstruction now validates each triangulated landmark independently under the unchanged depth and 3-pixel reprojection limits. A controller requires wrist and four palm knuckles; pinch and grip each have separate required fingertips and validity. A rejected finger no longer removes an otherwise accepted palm. Board calibration and distance checks still reject a set containing any invalid point. Protocol version 2 retains the 176-byte layout and carries explicit per-gesture validity bits; old protocol versions are rejected. The native driver releases only the uncertain gesture and requires its observed release before rearming. Unknown finger data cannot arm a click.
|
||||
|
||||
The driver smooths palm translation and rotation with a 15 ms time constant. Three consecutive observations with consistent velocity can enable at most 30 ms / 2 cm of translation prediction. Prediction stops on missing/invalid observations and origin/connection/head changes; the original 200 ms observation expiry is unchanged. Pose filtering never generates or holds gesture input. Native regressions cover independent gesture loss/rearming, old-version rejection, quaternion sign equivalence, prediction bounds, loss holds, expiry, and origin resets. Android regressions cover rejected fingertip versus rejected palm behavior, independent pinch/grip validity, and unchanged metric/reprojection checks.
|
||||
|
||||
Both full and full-palm/lite-landmark models are packaged and SHA-256 pinned. The lighter bundle is reproducible with `tools/build_hand_lite.py` from the documented original models and MediaPipe 0.10.21 metadata tooling. Diagnostics offer temporary model and half/full-input controls; CPU, full model, and half-rate input remain defaults pending a live comparison. An opt-in debug command benchmarks existing saved stereo pairs on CPU without capturing images or publishing replayed hand poses. It clears live hands during replay, closes both temporary detectors, and resumes the selected live model afterward.
|
||||
|
||||
All 259 selected Android tests pass with no failures/errors/skips, including the glasses package and spatial-setup UI. The ARM64 native tests, SDK-enabled and SDK-free Android builds, and runtime freshness checks pass. Installed APK: `build/DroidDeck-hand-reliability.apk`, SHA-256 `38b9b67171b37b0bf80b540da165894ff4cbb24b609ae62edd829ecf618d01e3`. Driver SHA-256: `56b3a3eeaa96160e5cbf740f35b701f935b610417dbbba0aa2965ec109d5418a`. The graphics synchronization layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. Installation preserved the verified calibration and CPU selection across processes 2843 and 3996. The built, packaged, and deployed driver/layer match; the Android publisher produces a version-2, 176-byte spatial file. Evidence: `build/hand-reliability-{tests,native,stage,build,sdkfree,install,audit}.log` and corresponding install/audit JSON.
|
||||
|
||||
The Fold7 successfully loaded both model bundles using Android Tasks 1.0.0 on CPU. Saved-pair replay ran full/lite/full, each with four warmup and twenty measured pairs, outside SteamVR in the same process. Times below cover stereo inference, excluding model construction, camera acquisition, geometry, publication, and display latency.
|
||||
|
||||
| Saved pair / model | Median inference | p95 inference | Valid palms |
|
||||
|---|---:|---:|---:|
|
||||
| Previously detectable pair / full before | 104.95 ms | 107.95 ms | 20/20 |
|
||||
| Previously detectable pair / lite landmarks | 90.03 ms | 95.54 ms | 20/20 |
|
||||
| Previously detectable pair / full after | 103.89 ms | 106.20 ms | 20/20 |
|
||||
| Harder pair / full before | 60.77 ms | 63.88 ms | 0/20 |
|
||||
| Harder pair / lite landmarks | 62.64 ms | 66.53 ms | 0/20 |
|
||||
| Harder pair / full after | 62.48 ms | 65.74 ms | 0/20 |
|
||||
|
||||
The lite candidate is approximately 13–14% faster on the detectable static pair. Both models recognize a released pinch/grip and retain their validity on all twenty repetitions; their first measured palm positions differ by about 1.9 mm. This is agreement between models on one static observation, not ground-truth accuracy or stationary-jitter validation. Neither detects the harder saved pair, so reducing landmark cost does not solve missed palm detection. Thermal status ended at 2, with current HAL skin temperature 41.4 C. Evidence: `build/hand-reliability-benchmark-idle.json` and `.log`.
|
||||
|
||||
SteamVR attempts 44 and 45 were suspended by Android with the screen off/lock screen showing before their first displayed frame. Wake/dismiss requests did not dismiss the lock screen. A user unlock is required for the graphics-workload model comparison and a fresh optical open/pinch/release check; neither is counted as a pass. The full model remains the default until that comparison can be completed. The changes are installed as an experimental candidate; reduced live flicker and successful dashboard clicking have not yet been established.
|
||||
|
||||
## Concurrent CPU inference and controller cadence (2026-10-16)
|
||||
|
||||
Left and right CPU inference now run concurrently with separate image buffers and a stable owner for the right detector. Both calls finish before either frame buffer can be reused or either detector closed, including failure and interruption paths. GPU inference remains sequential and opt-in. Controller presentation now approaches the filtered pose on each driver update with a 15 ms time constant, rather than moving only when a new inference result arrives. Missing observations stop prediction and hold the last displayed transform only until its original 200 ms expiry. Gesture validity, input release, head freshness, and connection/origin resets are unchanged.
|
||||
|
||||
All 263 selected Android tests pass without failures, errors, or skips, including four concurrent-worker lifecycle tests. Native spatial regressions cover intermediate render updates, normalized rotations, loss holds, and unchanged expiry. The ARM64 build, SDK-enabled and SDK-free APK builds, and packaged-runtime freshness checks pass. Installed APK: `build/DroidDeck-hand-cadence.apk`, SHA-256 `7df588f0eacdbf4f5ddaae91c511e415b01fe39f37100c012e8f8da3fced52e0`. Driver SHA-256: `195261f27043b260052873a3b083c436457763caa5ebb7bcd1f52708e426bf11`. The graphics layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. Built, packaged, and deployed binaries match; the spatial channel remains version 2, 176 bytes. Installation preserved CPU selection and verified calibration across processes 13863 and 14542. Evidence: `build/hand-cadence-{native,stage,build,sdkfree,install,audit}.log` and corresponding install/audit JSON.
|
||||
|
||||
A saved-pair comparison ran under SteamVR in one Android process/session using the previously authorized pair `64390821246320`. Each configuration had four warmup and twenty measured pairs. Times cover stereo inference, not acquisition, reconstruction, publication, or display latency.
|
||||
|
||||
| CPU configuration, in test order | Median inference | p95 inference | Valid left palms |
|
||||
|---|---:|---:|---:|
|
||||
| Full, sequential before | 202.74 ms | 286.60 ms | 20/20 |
|
||||
| Full, concurrent | 133.19 ms | 162.33 ms | 20/20 |
|
||||
| Lite landmarks, concurrent | 104.18 ms | 165.52 ms | 20/20 |
|
||||
| Full, sequential after | 176.31 ms | 208.47 ms | 20/20 |
|
||||
|
||||
Concurrent full inference had a 24–34% lower median than the surrounding sequential runs. Its first palm position matched the full sequential result exactly; the lite result differed by about 1.9 mm. All repetitions retained released pinch/grip validity. These are repeated static-pair results, not physical accuracy, dynamic coverage, or clicking acceptance. Android thermal status was 4 throughout, with current HAL skin temperature increasing from 48.8 to 50.6 C. The ordered test was not temperature controlled. The lite model's lower median did not improve its p95 over concurrent full inference; the full model remains the default. Evidence: `build/hand-reliability-benchmark-steamvr-parallel.json` and `build/hand-cadence-benchmark.log`.
|
||||
|
||||
For the live tests, the user confirmed an open left palm approximately 12–18 inches away, moving slowly side to side. Each sample lasted 16 seconds. All coherent head samples were fresh. The sequential baseline used session 50; both concurrent-input samples used process 14542 and session 51 without a restart.
|
||||
|
||||
| Live configuration | Coherent reads | Reads with fresh left hand | Distinct left observations | Left arrival age, median / p95 | Sampled inference median |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Previous sequential CPU, 12.5 Hz input | 2,959 | 29.47% | 113 | 160.16 / 196.23 ms | 117.03 ms |
|
||||
| Concurrent CPU, 12.5 Hz input | 2,941 | 91.77% | 186 | 74.94 / 84.81 ms | 66.06 ms |
|
||||
| Concurrent CPU, 25 Hz input | 2,932 | 64.36% | 152 | 91.93 / 109.69 ms | 62.55 ms |
|
||||
|
||||
The freshness percentages measure sampled shared-channel availability; they are not the acceptance target's percentage of valid camera observations and cannot pass or fail that 95% gate. Arrival age measures the original observation's age when first seen by the channel monitor, not externally measured physical-motion-to-visible-controller latency. Natural hand motion, detected hand count, and temperature were not held constant. Both cameras often reported two hands during the requested left-palm test, so right-channel availability is not evidence of verified right-hand identity or aiming. Thermal status was 4, with HAL skin temperatures of 47.5–47.7 C for the baseline, 50.5 C for limited input, and 50.7–50.8 C for full input. These short, ordered samples support further testing of the candidate, not a causal attribution of the entire improvement or a general conclusion that full-rate input is worse. Full-rate input did not improve observed availability in this comparison; the original 12.5 Hz input cap was restored. The SDK cameras continue at approximately 25 Hz.
|
||||
|
||||
The user reported improved but still choppy motion, a reversed right-controller pointing direction while the left behaved as expected, and less flicker with both hands detected than with one. Right-hand orientation correction and investigation of the one-hand/two-hand difference are in progress; neither is validated by this candidate's measurements. No new camera images were saved. Session 51 later suspended and exited with status 0; a subsequent audit found session 52 READY, CPU/full/concurrent inference, the 12.5 Hz cap, verified calibration, and no temporary graphics overrides. Evidence: `build/spatial-live-hands-choppy-baseline-confirmed.json`, `build/spatial-live-hands-cadence-{half,full}.json`, `build/hand-input-comparison.json`, `build/hand-cadence-summary.json`, and `build/hand-cadence-audit.json`.
|
||||
|
||||
## Right-hand orientation and camera-clock continuity (2026-10-16)
|
||||
|
||||
The user's report of a reversed right controller exposed a handedness-independent palm basis. The right hand now reverses its lateral axis before constructing the rotation, giving mirrored palms the same normal convention while preserving the existing left mapping and wrist-to-finger direction. Both live reconstruction and saved-pair replay pass an explicit hand side. Tests cover mirrored palms, 180-degree and arbitrary rotations, camera-to-world transforms, and unchanged gesture values and timestamps. These tests verify the mathematical correction; the user subsequently confirmed that the right-controller orientation is correct on the installed build. Measured aiming accuracy across poses remains pending.
|
||||
|
||||
The one-hand/two-hand flicker difference has a plausible processing explanation: the [upstream MediaPipe video graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259) repeats palm detection until the previous tracked hand count reaches the configured maximum. DroidDeck requests two hands. Per-camera inference durations and current/previous raw hand counts are now reported separately from confidence-filtered counts so this hypothesis can be measured. No hand-count restriction, confidence reduction, or synthetic second hand was introduced.
|
||||
|
||||
A separate clock defect reset camera synchronization whenever the worker skipped more than 200 ms of source frames, even when host and source clocks advanced consistently. A processing stall could therefore add another 16 observations of warmup (roughly 1.3 seconds at 12.5 Hz). The mapping now survives an aligned long gap; offset discrepancies above 20 ms, invalid timestamps, or nonmonotonic source/receipt times still require warmup. Existing short-gap rolling-minimum/p90 jitter handling is retained. Six regressions cover 240 ms and one-second gaps, resets, offset changes in both directions, jitter boundaries, and recovery. Hand observation expiry remains 200 ms and head expiry remains 100 ms.
|
||||
|
||||
All 271 selected Android tests pass without failures, errors, or skips. SDK-enabled and SDK-free builds and packaged-runtime freshness checks pass. Native code is unchanged from the tested cadence build. Installed APK: `build/DroidDeck-hand-continuity.apk`, SHA-256 `46b99d939e5c7063840ae669128b40ba60a794e62a30c1a14f38da27362d2855`; driver remains `195261f27043b260052873a3b083c436457763caa5ebb7bcd1f52708e426bf11` and the graphics layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. The installed APK hash matches, and processes 29580 and 30045 both initialized CPU inference and loaded the verified calibration. CPU/full/concurrent inference and the 12.5 Hz input cap remain selected. No new images were captured. Evidence: `build/hand-continuity-{build,sdkfree,stop,install}.log` and `build/hand-continuity-install.json`. Subsequent live results follow; sustained smoothness acceptance remains pending.
|
||||
|
||||
SteamVR reopened as session 54 and reached READY with valid head tracking. A live audit verified the built, packaged, and deployed driver/layer hashes, the version-2 spatial channel, saved calibration, CPU/full/concurrent inference, the 12.5 Hz cap, and the new per-camera diagnostic fields. This is an installation/startup check, not the pending deliberate one-hand/two-hand comparison or a visual orientation pass. Evidence: `build/hand-continuity-launch.log`, `build/hand-continuity-audit.log`, and `build/hand-continuity-audit.json`.
|
||||
|
||||
## Live hand-count comparison and remaining startup delay (2026-10-16)
|
||||
|
||||
The subsequent 44-second live comparison retained Android process 30045 and SteamVR session `2026-10-16-54-steam`. All 8,146 coherent head reads were fresh; the left hand was fresh in 97.30% of channel reads. The right hand was introduced after the left, and both cameras reported two hands from approximately 13 seconds. Consequently, the nominal first-20-second/remaining-test stages are not clean one-hand/two-hand windows. The whole-run right-hand availability of 71.00% includes the period when it was absent and must not be interpreted as right-hand detection accuracy.
|
||||
|
||||
Per-camera timings were grouped by each detector's previous raw hand count, which is the count relevant to the upstream video graph's palm-search decision. The smaller count group contained eight diagnostic snapshots per camera, and the two-hand group contained nineteen.
|
||||
|
||||
| Camera | Previous raw hands | Diagnostic samples | Median inference | p95 inference |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Left camera | 1 | 8 | 73.36 ms | 80.27 ms |
|
||||
| Left camera | 2 | 19 | 55.96 ms | 69.10 ms |
|
||||
| Right camera | 1 | 8 | 71.94 ms | 87.04 ms |
|
||||
| Right camera | 2 | 19 | 53.15 ms | 65.69 ms |
|
||||
|
||||
The observed per-camera medians were approximately 17–19 ms lower with two previously tracked hands. This is consistent with the extra palm-search work described above, but the ordered, natural-motion test does not isolate that work from pose, visibility, or workload changes. Thermal status remained 4, with HAL skin temperature 50.7 C at both endpoints. These internal inference timings do not measure physical-motion-to-visible-controller latency. The 97.30% sampled left-channel availability is also distinct from the required 95% valid camera-observation rate; neither acceptance gate is established by this run. No new camera images were saved. Evidence: `build/hand-count-live.json`, `build/hand-count-live.log`, and `build/hand-count-summary.json`.
|
||||
|
||||
The user confirmed the right-controller orientation correction and described single-hand tracking as smoother and more reliable, but said it takes a while after SteamVR starts to become smooth. Startup choppiness remains unresolved. This measurement did not isolate startup behavior, and it does not establish a warmup, thermal, synchronization, or graphics cause for that remaining delay. Longer controlled runs, an observation-based coverage measure, external latency measurement, and deliberate optical interaction checks are still required.
|
||||
|
||||
The user clarified that startup choppiness lasts about a minute or longer after the dashboard appears. Both CPU detectors, camera synchronization, and verified calibration were already active in process 30045 before SteamVR launched, so initial model loading alone does not explain the delay. Session-54 compositor logs, aligned to their Denver timestamps, record 138 shader-pipeline creations during seconds 20–69 after the launch request and another 12 during seconds 120–129. This identifies ongoing graphics initialization as a plausible source of CPU contention, not a measured causal explanation of all startup stutter. The current session remained READY; its later presenter counters were about 600 frames per ten seconds. No matched `vkerror`, device-loss, fatal, or segmentation messages appeared in the filtered session window, although a `WaitForAcquire` timeout remains. No startup scheduling, cache, graphics, or loading-screen changes were made. A correlated startup trace of hand latency and CPU/GPU workload is still needed. Evidence: `build/hand-startup-review.json` and `build/hand-continuity-install.json`.
|
||||
|
||||
## Gesture, single-hand scheduling, and motion update (2026-10-16)
|
||||
|
||||
The next candidate keeps CPU/full-model/concurrent stereo inference. Calibrated stereo palm reconstruction is unchanged; pinch and finger closure now use agreement between each camera's image landmarks. Invalid or conflicting finger geometry clears the affected gesture, and the driver still requires an observed release before rearming it. This avoids making a valid palm depend on successful triangulation of every fingertip, without claiming that a lost finger is released.
|
||||
|
||||
CPU inference now retains separate one-hand and two-hand video trackers for each camera. Three consistent stereo observations enable the one-hand path; a two-hand search is due after 240 ms, and loss or ambiguous handedness returns to search. The selected bank's raw counts and hand limit are reported. Both per-camera tasks finish before images or detectors are reused or destroyed. GPU remains optional and serial. The adaptive copy gate starts at 500 permille of source callbacks (12.5 Hz with 25 Hz cameras), with 600/800 permille available only after eight consecutive healthy samples meet processing and age thresholds. Slow discovery samples can prevent an upshift; the presence of this controller is not evidence of achieving 15 or 20 Hz. Camera capture and vendor head tracking retain their normal rate.
|
||||
|
||||
The driver removes the additional observation-level low-pass and retains one 15 ms rendering correction. Stable translation advances from the time a result reaches the driver, bounded by 80 ms and 2 cm, instead of exhausting prediction during inference. Capture timestamps still control velocity and expiry. Explicit loss freezes the displayed pose, immediately releases invalid inputs, and preserves the 200 ms hand expiry and 100 ms head expiry. This is bounded motion estimation, not additional measured hand observations.
|
||||
|
||||
Validation: all 306 selected Android tracking/UI tests pass, with no failures, errors, or skips. The ARM64 VR build and native regressions pass, including delayed/jittered observations at 4 ms driver ticks, stops, reversals, outliers, quaternion normalization, exact loss freeze, expiry, and origin/connection changes. Fractional camera-gate regressions pass with strict compiler warnings. SDK-enabled and SDK-free Android builds, 13 runtime-preparation tests, and runtime checksums pass. The loading/first-frame flow and graphics color path were not changed in this update.
|
||||
|
||||
Installed on the existing Fold7 as `build/DroidDeck-hand-smoothness.apk`, final SHA-256 `9508370c42870e60a0070abf61d99792df581436f802049b068b96eba95fa027`; driver SHA-256 `52a96238afedf68c38866588db8225d2bd82450ed13635a5673ca7f9e0c6835b`. Android requested USB access after the first candidate installation; the user accepted it. Final verification matched the installed APK hash and confirmed CPU selection, automatic rate selection, scheduled single-hand tracking, and the existing verified calibration in two separate app processes (6139 and 6989). CPU inference initialized in 6139; process 6989 was initially waiting for valid head tracking, so its configured CPU delegate was not yet initialized. The saved 25-view calibration and distance check were retained. Evidence: `build/hand-smoothness-{native,stage,final-build,sdkfree,runtime-tests,install}.log` and `build/hand-smoothness-install.json`. Deliberate moving-hand, grab/release, external latency, and sustained game-performance checks remain required.
|
||||
|
||||
Saved-pair replay of the previously authorized capture `64390821246320` compared parallel full-model search, scheduled full-model tracking, scheduled lite-model tracking, then a repeated full-model search baseline. Each row measured 20 frames after four warmup frames. The scheduled rows used the one-hand path for 13 of those frames. All rows retained a valid left palm and valid released pinch/grip in all 20 measured frames; the source image is a stationary open palm, so this does not validate moving hands or pressing gestures.
|
||||
|
||||
| Workload | Full search median / p95 | Scheduled full median / p95 | Repeated full search median / p95 |
|
||||
|---|---:|---:|---:|
|
||||
| App idle | 70.93 / 76.19 ms | 36.74 / 74.04 ms | 71.28 / 75.77 ms |
|
||||
| SteamVR | 91.48 / 99.26 ms | 46.92 / 94.71 ms | 86.39 / 100.62 ms |
|
||||
|
||||
Scheduled lite-model medians/p95 were 25.41/67.04 ms idle and 40.26/86.77 ms under SteamVR. The full model remains selected because a repeated static image cannot establish the lighter model's live tracking or gesture accuracy. The SteamVR comparison retained process 333 and session `2026-10-16-57-steam`; thermal status was 4 and HAL skin temperature rose from 49.5 to 51.0 C. These short ordered runs show lower typical inference cost on this saved pair, while discovery-frame tail latency remains. They do not establish external motion-to-controller latency, sustained game frame time, or reduced thermal throttling. No new camera images were captured. Evidence: `build/hand-reliability-benchmark-smoothness-{idle,steamvr}.json`.
|
||||
|
||||
The final adaptive policy uses average processing cost for normal budget changes so occasional discovery frames do not veto otherwise sustained headroom. Any single processing cost at least 80 ms, observation age at least 140 ms, or invalid tracking falls back to 12.5 Hz. Regression tests verify that a repeating 36/36/74 ms pattern can select 15 Hz but not 20 Hz, while continuous 70 ms work stays at 12.5 Hz. This change does not assert that the observed SteamVR workload qualifies for a higher rate.
|
||||
|
||||
Final deployment audit: SteamVR session `2026-10-16-58-steam` reached READY with a first frame and valid head tracking. Built, packaged, and deployed driver hashes match; the unchanged graphics-layer hash is `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. The version-2 spatial channel and verified calibration were present, with no temporary environment overrides. CPU/full/concurrent inference, scheduled discovery, and adaptive input were active. The audit snapshot showed one accepted palm with valid pinch/grip geometry, the one-hand inference bank, 47.10 ms stereo inference, approximately 25 Hz source cameras, and a 500-permille input limit. A single snapshot does not demonstrate reliable movement or actual pinch/grip presses. Evidence: `build/hand-smoothness-{launch,audit}.log` and `build/hand-smoothness-audit.json`.
|
||||
|
||||
## Stationary jitter and persistent input release (2026-10-16)
|
||||
|
||||
The user reported stationary controller bounce and a dashboard drag that remained attached after a controller flickered and the hand returned open. The original 16-second reproduction recorded 2,905 coherent channel reads with fresh head tracking throughout. The right hand was fresh in 812 reads; 64 distinct poses were recorded. Right-hand transitions included five samples above the trigger-click threshold, 63 valid released-grip observations, and no pressed grip. Both hands were invalid at the end of the sample. This does not by itself locate the missing dashboard release: the original driver lacked per-input publication diagnostics. Evidence: `build/spatial-live-hands-sticky-grab-baseline.json`.
|
||||
|
||||
Inspection of the installed Vive compositor binding showed that dashboard left-click/drag uses the trigger, with grip mapped to Back. The installed dashboard `GrabHandle` attaches the panel to the initiating hand and clears that attachment on a window mouse-up event; it has no tracking-loss cleanup in that component. A missing mouse-up is consistent with the report, but the exposed code does not prove that native laser-mouse processing discards releases when a pose becomes invalid. Resource evidence: `resources/config/vrcompositor_bindings_vive_controller.json`, SHA-256 `16f19a170608c9b3450e7fc980644220a5bd3a34e1bb88050e65f51c7600a2d1`, and `resources/webinterface/dashboard/chunk~8012d0c89.js`, SHA-256 `8a529bd96141480a18efa44b912a81a1e27ac1bf8bae6b2ab8945ebaf6da2e73`. No SteamVR resource was modified.
|
||||
|
||||
The driver now stabilizes translation and rotation independently against fixed stationary anchors, holding within 3 mm and 1 degree. Breakaway uses the existing 15 ms rendering correction; it does not subtract a dead-zone distance from every observation. Returning to a hold requires 240 ms inside tighter 1.5 mm / 0.5-degree bounds and a check against sustained directional drift. Position prediction stops while anchored. Native tests cover still jitter, quaternion sign flips, accumulated slow translation/rotation, fast breakaway, settling, and the existing delayed-motion and loss behavior. These thresholds are candidates; stationary jitter and fine aiming still need hardware confirmation.
|
||||
|
||||
A confirmed latest-snapshot race could hide a transient invalid hand/head observation from the driver. Protocol version 3 retains the 176-byte layout and uses the offset-36 word for four independent 8-bit loss counters: left trigger/grip, then right trigger/grip. Loss remains visible in subsequent snapshots even if the invalid snapshot was overwritten. Head gaps clear cached hands, captures preceding head loss cannot rearm inputs, and expiry/invalid-gesture transitions update only the affected counters. The driver clears matching input state on counter changes, rejects old or duplicate observations for rearming, and detects observation gaps beyond 200 ms. It publishes released trigger scalar/click and grip before pose loss, clears inputs before deactivation, and logs state/error transitions for hardware verification. No synthetic presses or fresh poses were introduced. Versions 1 and 2 are rejected, so the Android and driver binaries must be updated together.
|
||||
|
||||
All 317 selected Android tests and the native regressions pass. New tests deliberately skip invalid snapshots, preserve independent trigger/grip state, reject stale release observations, verify publication ordering, and retry rejected OpenVR component writes. SDK-enabled and SDK-free Android builds and runtime checksum verification pass. Installed APK: `build/DroidDeck-hand-stability.apk`, SHA-256 `0e3f4a140e36eb48e91f5937748b0e10afe3491a0236484b21e9a3faa3ca6766`; driver SHA-256 `cca5723189a77cde606d11afd55d80567e7f72398e3f50e706bee164cb91eb5f`. Calibration survived two app processes (17259 and 17867); CPU initialized in the latter. SteamVR session `2026-10-16-60-steam` reached READY with a first frame. Evidence: `build/hand-stability-{native,stage,build,sdkfree,install,launch,audit}.log` and `build/hand-stability-{install,audit}.json`. End-to-end dashboard release and visible stationary improvement remain pending the deliberate post-installation check.
|
||||
|
||||
## Outstanding acceptance
|
||||
|
||||
### Investigation and shared base calibration (2026-10-16)
|
||||
|
||||
The phone's current verified 640×480 pinhole profile was exported directly over authorized ADB. Its 1.1209395390660726-px fit, lens coefficients, stereo transform, and camera-to-head transform are now bundled unchanged as `app/src/main/assets/calibration/luma-ultra-640x480.json`, with the source device identifier removed. Asset SHA-256: `d160f0c81d0328255f9afa69bc61505e5aa2988a9bb7a9faaeb611598eee7faa`. A matching personal profile takes precedence; new or unidentified Luma Ultras use this base for experimental tracking. Shared geometry is not falsely marked validated on other devices. UI and diagnostic status identify the shared base separately from individually verified profiles.
|
||||
|
||||
Hand tracking now displays **Experimental** throughout the relevant controls and setup status. Per-observation diagnostics distinguish detection/association failures, missing palm landmarks, palm width/basis/motion rejection, stale data, and tracking loss. Connection-lifetime counters retain independent palm, trigger, and grip acceptance/loss evidence without counting repeated status polls. Tracking and input thresholds are unchanged. The historical trace analysis and prioritized improvement experiments are recorded in `luma-hand-tracking-investigation.md`; no new moving-hand or game-usability pass is claimed.
|
||||
|
||||
All 388 focused glasses and setup/UI tests pass, including the new calibration selection and diagnostic-counter regressions. Exact geometry comparison against the phone export passes. The SDK-enabled APK builds with runtime freshness verification enabled. After the concurrent upstream Joy-Con merge, the already rebuilt VR outputs were validated against current source/header/configuration hashes and restaged with the updated helper scripts; unrelated prepared binaries retained their hashes. All 150 packaged runtime outputs match the resulting manifest. The packaged base calibration matches its source exactly, and the APK includes the new diagnostics. Artifact: `build/DroidDeckXR-luma-experimental.apk`, SHA-256 `11235040ad8c34abfd23482beb552fb09db4d9429703fbc8d7c5090b98da940e`. Runtime provenance is in `build/luma-hand-runtime-restage-20261016/provenance.json`. This change has not been installed on the phone. Logs: `build/luma-investigation-tests.log`, `build/luma-investigation-build.log`; metadata: `build/luma-investigation-validation.json`. The source phone profile backup stays in ignored `build/luma-ultra-phone-calibration-20261016.json`.
|
||||
|
||||
| Gate | Current evidence or blocker |
|
||||
|---|---|
|
||||
| Measured head axes and meter scale | Pending measured translation, leaning and crouching |
|
||||
| Floor and separate origins on hardware | Automated math passes; physical crouched recenter and picture switching remain pending |
|
||||
| Screen anchoring and plane crossing | Shader and parameter tests pass; visual movement and plane-crossing checks remain pending |
|
||||
| Camera image orientation | Upright 640 by 480 images and visible palms verified in one authorized diagnostic pair |
|
||||
| Calibration quality | 25-view set passes fit/alignment checks with 1.12094 px held-out reprojection error. Repeated distance check passed: 12.13 inches measured against 12.00 selected; SDK-identified verified profile survives app restarts |
|
||||
| Hand pose and aiming | User confirms corrected right-controller orientation and smoother, more reliable single-hand tracking. Latest 44-second comparison had 97.30% sampled left-channel availability; startup choppiness, measured aiming accuracy, and sustained continuity remain pending |
|
||||
| At least 95% valid hands at 30–80 cm | Camera-observation acceptance rate not measured; sampled channel availability above is a different metric |
|
||||
| At most 5 cm hand error and 2 cm jitter | Not measured |
|
||||
| External p95 controller latency below 100 ms | Not measured |
|
||||
| Hardware input release and reacquisition | State-machine tests pass; user-requested 200 ms hand expiry / 500 ms reacquisition not measured physically |
|
||||
| Twenty-minute game performance | No controlled baseline or CPU/GPU comparison |
|
||||
| Dashboard and Angry Birds VR interactions | Not validated with optical controllers |
|
||||
| Occlusion, crossed hands, lighting and thermal load | Pending |
|
||||
| Physical USB reconnect during inference | Generation/lifecycle guards implemented; no calibrated-inference unplug test |
|
||||
|
||||
The user confirmed corrected colors, optical controllers appearing without a freeze, corrected right-controller orientation, and smoother, more reliable single-hand tracking. The latest timing comparison is consistent with additional palm-search work when only one hand is tracked. Startup choppiness remains unresolved, with no established cause. A confirmed open/pinch/release sequence is still needed to distinguish missing/expired observations from gesture recognition or input-path failures. Full palm detection with the lite landmark model remains an experiment; the current full model stays selected. Controlled temperature-matched SteamVR measurements, right-hand correspondence across poses, and measured aiming validation remain necessary. The compositor guard passed the reproduced gamepad-controller visibility trigger; long-duration and GPU-inference stability remain unvalidated. Delegate and process identity must be checked across each measurement because graphics recovery recreates the hand engine; it now selects CPU by default and discards any temporary GPU override. The saved calibration no longer needs repeating after an identified-device reconnect. The combined feature must remain experimental until hand tracking and the remaining acceptance measurements pass. Native 6DoF availability alone is insufficient for release.
|
||||
|
||||
## Fork migration and latest feedback (2026-10-16)
|
||||
|
||||
The post-installation check did not establish a fix: the user could not start the dashboard drag and reported stationary controller jitter about the same. Dashboard interaction, release after tracking loss, and visible stationary stability remain unresolved.
|
||||
|
||||
At the user's request, the 63 uncommitted XR files were moved from `C:/Users/conno/Desktop/DroidDeck` to `C:/Users/conno/Desktop/DroidDeckXR` on branch `feat/3dfixes-6dof`. A three-way merge preserved the fork's resolution and menu changes; 13 non-overlapping fork files remained unchanged. Original files and patches from both checkouts are preserved in `build/xr-migration-20261016`. The source checkout is clean after hash-verified removal of the migrated edits.
|
||||
|
||||
The merged fork passes 373 selected Android tests and 13 packaging tests, plus SDK-enabled and SDK-free builds. The existing tested ARM64 VR outputs were reused after validating their source, SDK-header, configuration and binary checksums; they were not rebuilt during migration. All 133 prepared runtime outputs match the SDK-enabled APK. The new APK is `build/DroidDeckXR-migrated-sdk.apk`; it has not been installed. See `build/xr-migration-verification.json` for hashes and the retained historical runtime-provenance boundary. Subsequent work belongs in DroidDeckXR.
|
||||
|
||||
## Connected-device isolation review (2026-10-16)
|
||||
|
||||
Reviewed the outstanding DroidDeckXR changes together, including optical and Joy-Con tracking, calibration, display-mode routing, recovery overlays, graphics fixes, RGB diagnostics, and build/runtime provenance. Ultra controls now require a controllable, connected device whose Carina capability was confirmed by the VITURE SDK. Model names and saved Ultra preferences alone do not enable them. The experimental position/hand/Joy-Con controls, floor/eye-height setup, 6DoF label, calibration activity, recovery overlay, and RGB diagnostic entry follow this capability. Calibration closes when the connection disappears, and launching it directly without an Ultra is rejected.
|
||||
|
||||
Saved Ultra controller preferences no longer suppress the phone-camera permission or controller path for other glasses. Routing is refreshed on device changes as well as setting changes. Opening a non-Carina link clears the retained spatial-channel mode; temporary Ultra tracking loss still invalidates poses without silently switching to fixed-height legacy tracking. Tests cover disconnect, reconnect, unconfirmed model names, non-Ultra VITURE, XREAL and RayNeo states, rejected mode changes, and the native-channel flags.
|
||||
|
||||
The optional UVC directory now accepts an explicitly empty Gradle override, requires its license files before packaging, and is included in the local native-source preparation audit. All 456 selected Android tests pass with no failures, errors or skips. An earlier Compose calibration timeout passed in isolation and in the final complete selection without changing the test or product behavior. The ARM64 VR rebuild and native regressions pass. Linux packaging/helper tooling reports 47 passing tests; two shader compilation tests were skipped because glslangValidator is unavailable (this change set does not modify shader source or its generated header).
|
||||
|
||||
Both the SDK/UVC-enabled and SDK/UVC-free Android builds pass, and all 150 prepared runtime outputs match each APK. Package inspection confirms the enabled build includes the optional libraries and UVC notices, while the disabled build contains none of those optional libraries. Artifacts: `build/DroidDeckXR-ultra-reviewed.apk` (SHA-256 `a01706d56b876c2fec7b9386d33e9dc66854ead66b48a1392c492bde7c3591d3`) and `build/DroidDeckXR-ultra-reviewed-sdkfree.apk` (`6ce45e225a37c74355eb7d65e76b2ac63be7064ec24cbe389e006452e0e27390`). Results and logs are retained under `build/ultra-review-*`, including `build/ultra-review-verification.json`.
|
||||
|
||||
This review did not install a new package or repeat hardware acceptance. The existing dashboard interaction and stationary jitter reports remain unresolved; the combined tracking feature remains experimental.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Luma Ultra spatial tracking
|
||||
|
||||
For supported glasses across brands, feature levels, and app-wide setup, see the
|
||||
[smart-glasses support matrix](../smart-glasses-support.md).
|
||||
|
||||
This is an experimental implementation of native Luma Ultra head tracking, positional virtual screens, and optical SteamVR controllers. Positional tracking, bare-hand controllers, and Luma Joy-Con tracking all default to off and are grouped under Experimental. Combined release acceptance remains pending until the calibration, accuracy, latency, gameplay, and sustained performance targets below have been measured on hardware.
|
||||
|
||||
## Build and setup
|
||||
|
||||
Luma Ultra now includes a shared starting calibration exported from the user's verified Fold7 profile on 2026-10-16. The bundled `calibration/luma-ultra-640x480.json` contains the original lens, stereo, and camera-to-head geometry without the source glasses' serial identifier. It applies only to the SDK-confirmed Luma Ultra primary `left0/right0` pair at 640 by 480. Matching saved device calibration takes precedence, including an unfinished personal fit that still requires distance validation. Missing, corrupt, or foreign profiles fall back to the model base; unknown serials can also use the base.
|
||||
|
||||
The base enables experimental hand tracking without requiring every user to repeat board collection. It is not marked individually verified on other units. Tracking and setup show **Shared Luma Ultra calibration** while it is in use; diagnostics distinguish `calibrationSource` (`luma-ultra-base`, `device`, or `none`) from `calibrationVerified`. The existing `calibrated` availability field is true for either the base or a verified personal profile. The following board steps are optional refinement for a new device using the base; a newly calculated personal profile still needs the unchanged distance check before use. A successful distance check saves the profile for the connected identified glasses. Source validation does not establish manufacturing consistency or aiming accuracy across other units.
|
||||
|
||||
Use the existing optional `droiddeck.vitureSdk` local property with the external VITURE Android SDK 2.4.0. No vendor headers or libraries are checked into the repository. `-PvitureSdkDir=` explicitly builds without the SDK. The ordinary runtime freshness checks remain enabled. Rebuild the ARM64 VR components with `tools/vr/build_windows.py`; its manifest includes `spatialpose_shm.h`.
|
||||
|
||||
The hand pipeline pins MediaPipe Tasks Vision 1.0.0, OpenCV 4.13.0, and the model and SHA-256 recorded in `app/src/main/assets/models/hand_landmarker.provenance.json`. Both Gradle and the inference worker verify the model checksum. CPU inference, the full model, concurrent camera inference, scheduled hand discovery, and automatic input-rate adjustment are the defaults, including after app restarts, reconnects, and provider restarts. **Show diagnostics → GPU hand inference** temporarily enables GPU until the next tracking restart; GPU remains experimental because SteamVR restarts and Vulkan device loss were observed during testing. Diagnostics report the selection as `handGpuEnabled` and the initialized detectors as `handInferenceBackend`; `not initialized` means detector startup has not completed. Sustained performance acceptance remains unresolved.
|
||||
|
||||
Hand input starts by copying 500 permille of stereo callbacks, approximately 12.5 pairs per second at the SDK's documented 25 Hz camera rate. On CPU, automatic adjustment can raise this to 600 or 800 permille (about 15 or 20 Hz) only after eight consecutive valid observations have sufficient processing and freshness headroom. Promotion requires average processing at most 55 ms and observation age at most 110 ms for 15 Hz, then at most 40 ms and 90 ms for 20 Hz. A moving average allows occasional scheduled discovery frames while preserving a sustained processing budget. An average above the selected period reduces the rate; any single processing cost of 80 ms or observation age of 140 ms immediately returns it to 500 permille. Tracking failure and detector resets also return it to 500. This policy requests more input only when processing can accommodate it; it does not guarantee that many completed or valid hand updates.
|
||||
|
||||
Skipping happens before copying camera pixels into DroidDeck, so skipped pairs also avoid JNI transfer, preprocessing, and inference. The SDK exposes no tracking-camera rate setter: sensor capture, vendor VIO, pose callbacks, and head polling retain their normal rates. Calibration collection, distance checks, explicit diagnostic capture, and preview-only operation use full-rate input. Diagnostics keep incoming `cameraHz` distinct from `cameraCopyPermille`, `rateLimitedFrames` (intentional skips), and `dropped` (eligible pairs replaced while the worker is busy). The compatibility field `cameraFrameDivisor` is 2 at 500 permille, 1 at 1000, and 0 for intermediate fractions. The bounded latest-pair queue and original capture timestamps remain intact. **Automatic hand input rate** can be disabled to expose the manual half/full-rate switch; both choices reset with tracking. GPU uses that manual preference and does not adapt. Thermal, latency, and gameplay benefits require hardware comparison.
|
||||
|
||||
1. Connect a Luma Ultra and grant its existing USB permission. Spatial controls appear only after the SDK identifies the device as Carina.
|
||||
2. Enable **Positional tracking — experimental**. This restarts the provider in 6DoF mode. Switching picture modes does not restart it.
|
||||
3. Enter the current seated or standing eye height (shown in centimeters and feet/inches) and choose **Set floor from current height**. Normal VR recentering then preserves that floor, including while crouched. After reconnecting or explicitly restarting tracking, accept the new origin through setup/recenter. A recenter request during lost/stale tracking or recovery remains pending until 250 ms of fresh, steady poses arrive; repeated taps are unnecessary. The pending request is canceled on disconnect.
|
||||
4. Open **Stereo camera setup and diagnostics → Open setup**. The page uses the selected DroidDeck theme and app scale, with a single compact Back/title row so the controls have room to scroll in landscape. Check head tracking and camera synchronization first; capture controls become available when both are ready. Expand **Show diagnostics** for frame rate, frame age, dropped frames, processing time, and recovery tools.
|
||||
5. Print the 5 by 7 ChArUco board at actual size. Measure a square: it must be 30 mm (approximately 1.181 inches). Keep the board stationary and capture at least 24 sharp views by moving and rotating the glasses, covering the image center and edges. Include both yaw and pitch, roll, and varied distances. If the underlying tracking frame changes, start a new capture set; software recentering in an unchanged frame preserves the set.
|
||||
6. Calculate calibration. Accepted captures confirm board detection, not alignment quality. The fitter compares pinhole and fisheye models on held-out views. It rejects poor stereo reprojection, an implausible baseline, insufficient rotational diversity, and inconsistent stationary-board alignment. Turning and tilting the glasses is required; pause before each capture. If the board stayed fixed but alignment varies beyond 5 cm (about 2 inches), the head-tracking frame may have shifted during collection. Wait for steady tracking and start a new set; do not remove the quality gate to accept the old one.
|
||||
7. Start with **12 in (30.48 cm)** or **30 cm (11.81 in)** under **Measured distance**; **20 in** and **50 cm** remain available. The closer settings make the small printed markers easier to resolve in the 640 by 480 tracking cameras. Show the complete board with its center exactly the selected distance from the left camera, then choose **Validate distance**. You can hold or support the glasses without wearing them. Progress and the result appear beside this button. The count refers to detected checkerboard intersections, not the paper's edges; a fully visible but small or blurred board can still have too few detections. Validation detects corners in the original images, then corrects those coordinates for lens distortion before triangulating, avoiding marker blur from resampling the images. Rejected stereo geometry reports that depth could not be measured; a valid reconstruction shows measured and selected distances in both units. Each imperial or metric choice uses its exact distance. The complete 24-corner board and physical-distance check within 5 cm (about 2 inches) are still required before a newly fitted personal profile becomes usable. Profiles are keyed to the SDK board-serial SHA-256 identifier (falling back to a nonblank Android USB serial), dimensions, and format version. Verified profiles reload automatically after an app restart or reconnect to the same identified glasses. If neither identifier is available, profiles remain connection-local.
|
||||
8. Enable **Hand controllers — experimental**. Present an open hand first so inputs can arm. Aim the virtual controller at the target, pinch thumb and index finger together to press the trigger/click, then separate them to release. Moving the SteamVR dashboard by its handle uses this same pinch-and-release action. Keep the other fingers open when only clicking; curling the middle, ring, and little fingers activates the separate grip input, which the installed Vive dashboard binding maps to Back. Games can use grip for holding objects. After tracking loss, open the hand again before another press. Gamepad sticks supply locomotion and turning; buttons retain the Vive-compatible mapping. External VR controllers retain precedence.
|
||||
|
||||
The diagnostic capture button saves one stereo pair and metadata to the app-private `files/spatial/captures` directory. Merely opening diagnostics does not save images. Calibration profiles stay in `files/spatial`. A brief missed hand detection holds the last observed controller transform only until its original observation reaches 350 ms old. Input freshness remains limited to 200 ms. Pinch and grip release immediately on a detected miss and require an observed release before rearming. The hold does not extrapolate motion, refresh timestamps, or substitute a gamepad pose. Head tracking loss, origin/connection changes, disconnects, and controller-mode changes clear it. Gamepad menus and locomotion remain available during optical hand loss. Switch hand controllers off explicitly to return to gamepad-positioned controllers.
|
||||
|
||||
Accepted board landmarks and their capture-time head poses are saved separately in app-private `files/spatial/*-captures.json`, without camera images. A matching SDK or USB device identity and image geometry allow automatic recovery for recalculation. Restored observations cannot be appended to a new tracking frame; start a new set to capture again. Devices without either stable identifier keep a session-labelled diagnostic backup and require explicit developer recovery onto the same physical device. They do not automatically reuse another connection's unidentified capture set or personal calibration; they can still use the bundled model base.
|
||||
|
||||
The vendor SDK working directory is `noBackupFilesDir/viture`. Android can evict `cacheDir` during a live session, so active tracking files must not depend on that location. The SDK initializes its own files in the new directory on the next provider start; existing printed-board calibration profiles remain in `files/spatial`.
|
||||
|
||||
## Coordinates and lifecycle
|
||||
|
||||
`SpatialPose` uses meters and right/up/back OpenGL coordinates. Quaternions are normalized xyzw internally; SDK arrays are position followed by wxyz. Legacy orientation conversion and the 128-byte head-pose channel remain intact. The session coordinator survives `HeadTracker` recreation and owns separate full-pose screen and yaw/floor VR origins. The camera worker uses a generation guard, a bounded latest-frame queue, and no SDK lifecycle operations in callbacks.
|
||||
|
||||
Head polling is approximately every 4 ms with prediction disabled; this is not a claim of a 250 Hz sensor. Camera-rate callbacks provide a liveness check. Stale or unstable poses retain the last rendered transform and produce an invalid SteamVR pose. While a positional session is active, DroidDeck covers the resulting grey view with a themed recovery screen on the phone and glasses, including one readable copy per eye in SBS. It distinguishes tracking loss, USB disconnection, required recentering, and stabilization after recentering. Tracking-loss instructions ask the user to look around slowly in a well-lit room with visible detail, keep the cameras clear, and open the setup diagnostics to restart tracking if recovery stalls. Phone controls remain accessible, and the screen closes automatically when tracking is valid. Freshness is checked even if pose callbacks stop. Repeated USB timeouts require connection recovery rather than a synthetic valid pose. Large pose discontinuities latch a setup requirement. Provider shutdown invalidates queued work before closing the vendor provider and Java USB connection.
|
||||
|
||||
The 176-byte `droiddeck-spatialpose` snapshot uses protocol version 4 and sequence-counter publication. The driver validates file length, version, finite normalized poses, and monotonic age. Head and both hands use one snapshot and one VR origin. Hand pose flags independently identify valid palm tracking (bit 0), pinch (bit 1), and grip (bit 2); head flags use only bit 0. The formerly reserved 32-bit word at byte offset 36 is now `input_epochs`, containing independent 8-bit loss counters for left trigger, left grip, right trigger, and right grip, from least to most significant byte. The layout size and pose offsets are unchanged, but versions 1 through 3 are rejected; app and driver must be rebuilt together.
|
||||
|
||||
Snapshot flags select positional tracking (bit 0), bare-hand controllers (bit 1), or Luma Joy-Con
|
||||
controllers (bit 2). The latter two selections are mutually exclusive; either enables stereo inference.
|
||||
The Joy-Con mode consumes only wrist landmark 0 and the original capture timestamp; rotation and inputs
|
||||
come from the paired Joy-Con. `luma_joycon.h` selects fresh stereo positions or a 6DoF-head-anchored arm
|
||||
model independently per hand, rejects observations over 1.5 m from the head, and blends source switches
|
||||
with a 180 ms time constant. It never integrates accelerometer position or uses the phone-camera
|
||||
correction in this mode. Stale head poses invalidate output and release inputs. Selected mode flags
|
||||
survive disconnect so a lost Luma cannot silently switch to a fixed-origin controller. Origin/connection
|
||||
changes reset both positional blending and controller yaw alignment. This is hand-assisted tracking,
|
||||
not a detector for the controller shell; see [Joy-Con setup and limits](../cardboard-vr.md#joy-con-controllers).
|
||||
|
||||
Each counter advances when its gesture becomes ineligible, including invalid finger data, hand/head loss, and observation expiry. It survives later valid snapshots, allowing the driver to notice a short loss even if it never read the intervening invalid snapshot. Counters wrap independently within their bytes. Connection/origin changes reset driver input state, and a gap beyond 200 ms from its last accepted observation also releases inputs. Every hand retains its original camera observation timestamp: experimental hand poses and inputs expire after 200 ms, while head poses expire after 100 ms. A changed loss counter disarms only its affected gesture. Duplicate or older observations cannot rearm an input or replay a press; a new, valid released observation is required after loss before another press. Missing finger data never counts as a release gesture. Head updates cannot extend a hand observation's lifetime. Diagnostics expose the hand limit as `handMaxAgeMs`.
|
||||
|
||||
The driver submits trigger scalar, trigger click, and grip updates before submitting the controller pose, including when the pose is becoming invalid. Disconnect and deactivation also clear inputs first. Changes in input state, loss counters, or API error codes produce concise driver log records with raw gestures, output values, armed state, pose/input validity, observation age, connection, and origin. Successful input API calls confirm submission to SteamVR; they do not prove that the dashboard received its mouse-up event. The installed dashboard grab handler retains a hand attachment until mouse-up and has no tracking-loss cleanup in that JavaScript component. Recovery of the reported stuck dashboard drag therefore still requires hardware validation; the implementation does not manufacture another press or a valid pose to force release.
|
||||
|
||||
Controller display smoothing has no stationary anchors or dead zones. Small intentional motion remains visible; smoothing and prediction do not change the measured pose used for gesture validation.
|
||||
|
||||
The displayed controller converges toward each measured palm pose on every driver update with one 15 ms correction time constant and normalized, hemisphere-corrected quaternions. There is no additional per-observation smoothing stage. After three consecutive observations with consistent motion, translation advances from the result's receipt time for at most 80 ms and 2 cm; capture timestamps still determine velocity and freshness. Receipt-relative prediction can continue between delayed inference results instead of consuming its entire horizon before a result arrives. Prediction is disabled above 1.5 m/s or 6 m/s², on a missed/invalid observation, head loss, or changed origin/connection. A missed observation freezes the last displayed transform through duplicate old samples until a newer observation arrives or the original 350 ms visual-hold expiry is reached. Fast motion reaches the 2 cm bound sooner, and correction adds a small settling delay. Stabilization and prediction do not alter gesture values or observation timestamps and cannot replace measured latency validation.
|
||||
|
||||
The screen shader puts translation in `extra.yzw`, rotates the eye offset, and rejects intersections behind the eye. SBS and flat branches ignore translation. The push-constant allocation stays unchanged. A screen reference can include pitch and roll, preserving placement while reclining.
|
||||
|
||||
## Stereo reconstruction and limitations
|
||||
|
||||
Skipped camera pairs do not require clock warmup again when the source/host offset remains consistent. A source gap above 200 ms checks that offset against the existing 20 ms bound instead of unconditionally resetting synchronization. Nonmonotonic or invalid timestamps and inconsistent long gaps require 16 fresh observations; normal short-gap jitter retains the existing rolling-minimum and p90 policy. This preserves synchronization through a processing stall without publishing expired hand observations.
|
||||
|
||||
Palm orientation uses an explicit hand side. The lateral axis reverses for the right hand so mirrored palms have the same normal and finger-direction convention; the existing left-hand mapping is retained. This corrects the mathematical right-hand inversion, while physical controller aiming still needs confirmation.
|
||||
|
||||
The stock two-hand video graph can repeat whole-image palm detection when fewer than two hands were tracked previously. With both hands tracked it can skip that search; see the [MediaPipe graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259). CPU scheduling maintains two persistent detector banks, each containing one detector per camera: one bank permits one hand, and the other permits two. After three observations agree on exactly one confident hand in both cameras, the one-hand bank handles intervening frames, and a two-hand search becomes due after 240 ms of camera time, running on the next processed frame. Both hands, a miss, handedness disagreement, nonmonotonic time, or a gap above 400 ms return selection to the two-hand bank. Second-hand discovery therefore remains enabled but can wait for the next scheduled search and processing. Banks retain their video tracking state between selections; they are not rebuilt per frame. GPU retains the ordinary two-hand bank. Diagnostics expose the active bank as `inferenceHandLimit`, alongside raw and filtered hand counts and per-camera timing. These choices reduce unnecessary searches by design; their benefit and extra model-memory cost require device measurement.
|
||||
|
||||
The primary `left0/right0` images are copied only for diagnostics, calibration, or enabled hands. At most one pair is pending while the worker processes another. Automatic exposure is retained because the cameras also serve native VIO. SDK 2.4.0 documents the tracking-camera rate as 25 Hz and provides no stereo frame-rate setter. Polling more frequently does not create additional camera frames. The camera-rate readout accounts for skipped source-frame sequence numbers instead of treating processed-pair throughput as sensor rate. The selected video-mode Hand Landmarker pair receives matching timestamps. CPU inference runs the cameras concurrently with separate image buffers; each bank's right detector is created, used, and closed on one dedicated worker. Both calls must finish before pixels or detectors can be reused, including after failure or shutdown interruption. GPU inference remains serial on its owning thread. The pipeline detects landmarks in the original grayscale images replicated into RGB channels, without image normalization or fisheye remapping, corrects landmark coordinates for stereo reconstruction, matches unambiguous same-handed candidates, rejects excessive reprojection or implausible depth, and derives the palm pose from triangulated landmarks. Gesture recognition uses agreement between the two image-space finger estimates. MediaPipe world landmarks are not treated as headset-relative positions.
|
||||
|
||||
The public header does not document the coordinate basis of the camera-rate pose callback. Its pose values are therefore not used as canonical head transforms. A bounded history of polled OpenGL poses supplies the capture-time transform. Source-to-host clock offsets must first stabilize over at least 16 observations; callback timestamp agreement and a nearby canonical head sample are required. Hand inference starts without waiting for the matching VIO callback, then checks for that exact camera timestamp before reconstruction can publish a valid pose. Missing confirmation, changed connections/tracking frames, invalid head tracking, and hand observations older than 200 ms still discard the result. Calibration and preview-only acquisition retain their bounded pose wait. Receipt-based offset estimation cannot establish absolute camera latency. External motion-to-visible-controller measurement is still required. Increasing expiry can reduce disappearing controllers at the cost of displaying older poses; it does not resolve GPU device loss or establish the latency/performance gates.
|
||||
|
||||
Stereo validation retains the original 3-pixel reprojection, positive-depth, and 15–150 cm depth checks per landmark. Bare-hand controller position and orientation require a valid triangulated wrist and all four palm knuckles (0, 5, 9, 13, 17). Joy-Con mode reconstructs wrist landmark 0 only and takes orientation and inputs from the paired controller. Live pinch and grip no longer require trustworthy fingertip depth: each image estimates palm-normalized pinch and middle/ring/little finger curl, then both views must agree. Grip combines fingertip reach and finger-chain bending, with separate press/release thresholds and three curled fingers to press versus two to retain a press. Invalid, degenerate, or conflicting image evidence releases only its affected gesture and requires an observed release before rearming. A valid 3D palm is never sufficient by itself to invent a grab or click. Calibration and distance validation still require every board point to pass. Conservative stereo association rejects ambiguous hands rather than inventing depth. Crossed hands, edge-on palms, and occlusion may reduce availability. The 2D gesture consensus and aiming still require physical testing through grabs, pulls, and releases; they do not establish an accuracy or latency pass. No skeleton output, desktop gestures, fast-action guarantee, or hands-only locomotion is provided.
|
||||
|
||||
Diagnostics include automatic or manual half/full input rates and a Lite-model comparison. Choices reset with the tracking session; the default is full-model parallel CPU with scheduled discovery and an adaptive rate starting at 500 permille. The Lite bundle uses Google's Lite palm detector and Lite landmark model with matching Tasks metadata. Both bundles are checksum-pinned in Gradle and at runtime; `tools/build_hand_lite.py` reproduces the candidate from pinned inputs. Saved-pair CPU replay compares full-model two-hand inference, scheduled full, scheduled lite, then full-model two-hand inference again, all with concurrent cameras. Rows identify `parallel` and `scheduled`; samples identify `inferenceHandLimit`. Its timings exclude model construction and cannot establish live detection coverage, external latency, second-hand discovery delay, or game performance.
|
||||
|
||||
## Combined release gates
|
||||
|
||||
| Area | Required evidence |
|
||||
|---|---|
|
||||
| Translation | Measured right/up/back movement and approximately 1:1 meter scale; lean and crouch; no double application |
|
||||
| Origins | Crouched recenter preserves floor; picture switching preserves independent screen and VR references |
|
||||
| Screens | Anchoring while moving sideways, toward and away; stable output after crossing the screen plane |
|
||||
| Hands | At least 95% valid slow unobstructed observations at 30–80 cm; position error at most 5 cm; stationary jitter at most 2 cm |
|
||||
| Latency | External physical-motion-to-visible-controller p95 below 100 ms |
|
||||
| Recovery | Immediate release on detected loss; observation expiry extended to 200 ms by user request; clear-hand reacquisition within 500 ms |
|
||||
| Performance | At most 10% regression in p95 game frame time over 20 minutes, with baseline frame rate maintained |
|
||||
| Gameplay | Dashboard selection and Angry Birds VR aim, grip, pull, release |
|
||||
| Robustness | Crossed hands, partial occlusion, head motion, lighting, USB reconnect, thermal load |
|
||||
|
||||
Record results in `luma-spatial-evidence.md`. A build or head-tracking pass alone does not satisfy this combined gate.
|
||||
@@ -187,7 +187,7 @@ Checked against bionic `android16-release`:
|
||||
|---|---|---|
|
||||
| `rseq`, `set_robust_list` | **blocked** | glibc, at the start of every thread, while all signals are blocked |
|
||||
| `faccessat2` | **blocked** | glibc's `faccessat()`, which tries it on **every** call before falling back, so 2 round trips (`access()` calls the plain `faccessat` syscall directly) |
|
||||
| `fchmodat2`, `openat2` | **blocked** | newer glibc, Flatpak/libglnx, FEX's rootfs lookups (`libblsession.so` turns `openat2` into `openat`) |
|
||||
| `fchmodat2`, `openat2` | **blocked** | newer glibc, Flatpak/libglnx, FEX's rootfs lookups (`libblsession.so` turns `openat2` into `openat`, resolving `RESOLVE_IN_ROOT` paths component by component inside the root) |
|
||||
| `setuid`, `setgid` and the rest of the family | **blocked** | Xwayland (hence `-i`) |
|
||||
| `futex_waitv`, `io_uring_*`, `landlock_*` | **blocked** | Proton's fsync (droiddeck-fsync answers `futex_waitv` in `libblsession.so`), some engines |
|
||||
| `clone3` | blocked before Android 15 | glibc `pthread_create` |
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
# RayNeo Air / GT XR integration
|
||||
|
||||
Built on the current `feat/viture-xr` PR #732 at bdf876d47cc9bb6fa810d014370da5bbb3c28039.
|
||||
The supplied historical hashes a127e66, 7f9dd75, e8e3e6b and f0b3297 were not
|
||||
resolvable in the fetched branch; no native projection or synchronization files are changed.
|
||||
|
||||
## Discovery and transport
|
||||
|
||||
`Glasses.kt` owns USB permission, attachment/detachment and presentation-display
|
||||
selection. There is no manifest USB VID filter: discovery uses `UsbManager.deviceList`.
|
||||
VITURE discovery is retained. `RayNeoModel` accepts the scaffold's confirmed
|
||||
`1bbb:af50` and verncat's confirmed GT identity `3941:af50`; other products under `1bbb` or bridge vendor `22d9` must identify
|
||||
as RayNeo/TCL Air or GT in their USB product name. This covers named Air 2, 2s,
|
||||
3, 4 Pro and GT devices without claiming arbitrary TCL phones or bridge chips.
|
||||
Additional unnamed VID/PID pairs require captures before being allowlisted.
|
||||
Other products under `3941` are rejected. Without a meaningful USB product name,
|
||||
`3941:af50` is labeled "RayNeo GT series"; a supplied GT MAX name is preserved.
|
||||
GT Max was observed on the Samsung SM-F976U with this USB identity on 2026-10-14;
|
||||
USB permission and the command handshake succeeded.
|
||||
EDID/product display names containing RayNeo, Ray Neo, or TCL Air/GT use the existing
|
||||
external-screen path. Android's supported modes, not a synthetic refresh value,
|
||||
determine whether 120 Hz can be selected. Stereo modes may have a lower maximum rate.
|
||||
GT Max's observed display identity is `SmartGlasses`, EDID manufacturer `TCL`,
|
||||
product `980` (1920x1080, 60/120 Hz in 2D). This exact metadata combination is
|
||||
recognized even though neither display name contains RayNeo. Other generic
|
||||
SmartGlasses/TCL products are not accepted by this rule.
|
||||
|
||||
Game output uses the same `GlassesScreen` presentation as VITURE, with the phone
|
||||
retaining the session controls; no screen-capture permission is needed. Discovery
|
||||
checks both Android's display name and its product name. On Android 12 and newer,
|
||||
a missing product-info object is treated like an unnamed panel: a recognized
|
||||
glasses USB device can identify a single presentation display. Unrelated named
|
||||
monitors and multiple unnamed displays are not automatically selected by that
|
||||
fallback. `Game on glasses` remains enabled by default.
|
||||
The presentation holder is selected before `show()`, and the phone SurfaceView
|
||||
is hidden only after the compositor attaches to the glasses holder. Late phone
|
||||
surface callbacks cannot reset the external output.
|
||||
|
||||
`RayNeoLink` claims a paired IN/OUT interface and translates the controller's
|
||||
internal commands to RayNeo packets. For `3941:af50`, selection prefers interface
|
||||
number 5 (not array index 5), as tested in verncat's GT contribution. Only endpoints
|
||||
that can carry 64-byte packets qualify. Firmware with a different interface number
|
||||
can use a unique qualifying IN/OUT pair; ambiguous alternatives are rejected.
|
||||
It requests device info (0x66/0), enables IMU
|
||||
(0x66/1) and waits for acknowledgments, warms up for two seconds, and stops IMU
|
||||
(0x66/2) when no pose consumer remains or the link closes. SBS uses commands 6/7.
|
||||
Unsupported VITURE brightness, volume and electrochromic commands are never sent;
|
||||
the VITURE hardware-control UI is hidden for RayNeo.
|
||||
GT spatial-button notification 0x08 and volume/display ACKs remain non-device-info
|
||||
responses and cannot satisfy unrelated startup commands. The spatial button is
|
||||
left to the glasses' firmware; no undocumented spatial-mode command is sent.
|
||||
|
||||
The parser/filter are adapted from the user's scaffold at
|
||||
581dbf712ed601ece4cf5e4a7c1f52e780299c88. Sensor packets are framed with 0x99/0x65,
|
||||
with little-endian acceleration at byte 4, gyro (degrees/sec) at 16 and a uint32
|
||||
100us device tick at 40. Fragmented and coalesced reports are assembled before decoding.
|
||||
Board 0x3a (Air 4 Pro) receives the scaffold's -20-degree X IMU mounting correction.
|
||||
GT Max (`3941:af50`, board `0x41`) uses the mapping measured on 2026-10-14:
|
||||
gyro and acceleration `(-y, z, -x)`, then +35 degrees around X.
|
||||
At the user's level pose the raw acceleration was approximately
|
||||
`(5.70, 0.00, 7.95)` m/s². A directed left-turn-and-hold capture produced positive
|
||||
X/Z gyro (e.g. `(20.5, 2.0, 28.2)` deg/s), and upward looks produced negative Y
|
||||
gyro. The 35-degree mounting estimate comes from that level capture,
|
||||
not manufacturer calibration; frame fit and other GT variants remain unverified.
|
||||
Gravity initializes orientation before the first published pose. Subsequent
|
||||
body-to-world fusion uses `measuredUp × predictedUp`, correcting tilt toward gravity
|
||||
instead of amplifying the error. These fusion corrections also apply to Air;
|
||||
the GT Max sensor remapping does not apply to Air or VITURE.
|
||||
|
||||
The non-Max GT shares USB `3941:af50` but reports board `0x40`, confirmed by
|
||||
switching between both physical pairs on the same phone. Its sensor axes use
|
||||
the ordinary XYZ basis with a measured -25-degree X mounting correction, rather
|
||||
than the Max permutation. Its confirmed upside-down resting acceleration was
|
||||
approximately `(0.020, -8.819, -4.106)` m/s²; the worn level capture was near
|
||||
`(0.09, 9.26, 3.11)`. Left turns produced positive Y/Z gyro. The mounting estimate
|
||||
uses the resting surface; per-device calibration refines its tilt and gyro bias.
|
||||
Both the pose filter and calibration collector select the same board profile.
|
||||
Non-Max calibration keys have a `:gt-standard-v1` suffix so they cannot replace
|
||||
existing Max profiles when USB identity fields coincide. Calibration loading
|
||||
waits for the device-info response; detached-device callbacks cannot change the
|
||||
replacement device's calibration key. Existing Max keys remain compatible.
|
||||
After installing this correction, the non-Max completed the upside-down
|
||||
calibration on the phone with `saved=true`, phase `SAVED`, progress 100.
|
||||
All 186 glasses/shared-pose/SteamVR-launch/UI regression checks passed and the
|
||||
installed APK's packaged runtime inputs verified. After calibration and the
|
||||
recenter check, the wearer confirmed on 2026-10-14 that the non-Max GT view
|
||||
looks and feels good.
|
||||
|
||||
## Existing pose and SteamVR pipeline
|
||||
|
||||
The scaffold fuses gyro/accel into wxyz. The Android adapter converts its Y-up
|
||||
coordinate basis and component order into VITURE's Z-up xyzw convention before
|
||||
calling the same `Glasses.PoseSink` used by VITURE. `HeadTracker` and `HeadPose`
|
||||
retain their existing recenter/seated-reference behavior. The path continues through
|
||||
`WaylandCompositor.nativeSetCardboardPose`, the compositor's `headpose_shm.h`
|
||||
shared-memory record and `tools/vr/src/driver_droiddeck.cpp`'s pose-file reader.
|
||||
The OpenVR projection, presenter, Vulkan wait logic and vblank counters are untouched.
|
||||
An explicit Play SteamVR request polls the guest runtime independently of display
|
||||
attachment and USB-control readiness. Both RayNeo and VITURE call the same
|
||||
`droiddeck-steamvr start` helper. Stopping cancels the pending launch request;
|
||||
automatic hardware 3D switching still respects the shared Auto 3D setting.
|
||||
|
||||
## Optional stationary calibration
|
||||
|
||||
Glasses → Tracking → Tracking calibration guides the user to rest the connected
|
||||
RayNeo upside down on a level surface. It allows three seconds to settle, then
|
||||
requires five continuous seconds of stationary samples (at least 200 reports).
|
||||
Invalid gravity, the wrong orientation, motion/variance, or gaps longer than 250 ms
|
||||
restart the window. A 30-second deadline fails without replacing a saved profile;
|
||||
USB stream loss also terminates the operation. Cancellation and link replacement
|
||||
invalidate queued callbacks by operation revision and connection generation.
|
||||
|
||||
The profile contains a raw gyro bias and a small mounting tilt correction, applied
|
||||
to both sensor vectors before fusion. It does not estimate heading, full accelerometer
|
||||
scale/bias, or translation. Profiles persist by vendor/product/serial; when a serial
|
||||
is unavailable the product and model name share a fallback profile. Reset deletes
|
||||
only the current profile and restores the model mapping. VITURE's fused SDK poses
|
||||
do not expose the raw inputs this measurement needs, so this control is RayNeo-only.
|
||||
|
||||
While measuring, live pose publication pauses. On completion the user wears the
|
||||
glasses, looks ahead and taps Done to recenter the active session. Dismissing the
|
||||
dialog or leaving the tracking panel cancels an in-progress measurement. Agent
|
||||
diagnostics expose the calibration state through `glasses`; operations are
|
||||
`calibrate`, `cancel-calibration`, and `reset-calibration`.
|
||||
The calibration build passed 157 targeted sensor, lifecycle, pose, launch and
|
||||
Compose UI tests. Installing this feature and validating a real upside-down capture
|
||||
on the phone remain pending a debugging connection.
|
||||
|
||||
## Regression and hardware validation
|
||||
|
||||
### SteamVR library launch routing (2026-10-14)
|
||||
|
||||
DroidDeck's SteamVR library entry (appid 250820), including its game shortcut,
|
||||
now sets the same `EXTRA_STEAMVR` request as Play SteamVR instead of sending a
|
||||
normal Steam game URL. Existing sessions resume with that request as well.
|
||||
Other games retain their normal launch path and arguments.
|
||||
|
||||
Inside Steam's library, the existing SharedJSContext bridge wraps
|
||||
`SteamClient.Apps.RunGame` for exactly 250820 and dispatches it to
|
||||
`droiddeck-steamvr launch`. This calls the same install/start helper used by the
|
||||
button, including OpenVR registration, the ARM64 runtime and Vulkan layer setup.
|
||||
It does not change Steam's saved launch options. The library bridge is installed
|
||||
for both desktop and Big Picture. It relies on the live Steam UI API and needs
|
||||
verification against the actual phone's Steam version after installation.
|
||||
|
||||
The helper serializes installation/startup and avoids restarting an active
|
||||
DroidDeck SteamVR session. Library launch diagnostics are in
|
||||
`${XDG_RUNTIME_DIR:-/tmp}/droiddeck-steamvr/library-launch.log`; normal runtime
|
||||
logs remain in `vrstartup.log` and Steam's logs directory. It does not intercept
|
||||
arbitrary external `steam://` URLs or change VR game launch arguments.
|
||||
|
||||
Only VR-only games (`vr_only`) wait for SteamVR and start it. Games where VR is
|
||||
optional, such as No Man's Sky, launch normally unless SteamVR is already
|
||||
running; before this they started (and if needed downloaded) SteamVR on every
|
||||
Play, so they opened in VR without the user asking for it.
|
||||
|
||||
Validation: seven bridge/dispatch tests passed under Node/Python, five Android
|
||||
SteamVR launch tests passed, shell syntax passed and the debug APK built with
|
||||
runtime input verification. The broader Linux compatibility suite cannot run
|
||||
unmodified on Windows because it requires `fcntl`; no Linux runtime was available
|
||||
for the helper's lock/process integration tests. Installation and live library
|
||||
Play verification are pending while the phone charges.
|
||||
|
||||
Run `./gradlew -PskipRust=true :app:testDebugUnitTest --tests 'com.droiddeck.launcher.glasses.*'`.
|
||||
Protocol tests cover malformed/fragmented packets, unsigned timestamps, model gating,
|
||||
quaternion normalization, yaw basis and tick rollover. USB integration tests cover
|
||||
permissions, GT interface preference/fallback, interleaved GT notifications and
|
||||
ACKs during startup, 2D/3D writes, and stale pose callbacks. Hardware validation is still required per model:
|
||||
USB enumeration, acknowledgment/stream startup, axes and recenter, hot unplug/replug,
|
||||
SBS negotiation, Android's actual 120Hz mode, and SteamVR pose/frame pacing.
|
||||
GT Max was verified on SM-F976U with USB `3941:af50` and the TCL/980 display:
|
||||
the compositor attached to its separate 1920x1080 display, the Play SteamVR
|
||||
request started native `vrserver`/`vrcompositor` through the shared helper, and
|
||||
the presenter logged roughly 60 frames/s while the IMU packet count increased.
|
||||
The test retained the user's disabled Auto 3D setting. After installing the corrected
|
||||
GT sensor mapping, the user confirmed an upright starting view and correct movement
|
||||
in all directions. SteamVR and the glasses IMU remained running. Stereo negotiation,
|
||||
long-term drift, and alternate bridge firmware variants still need their own
|
||||
hardware validation. The targeted glasses, Cardboard pose, and SteamVR launch
|
||||
suite passed 134 tests after the tracking correction. Tracking regression coverage
|
||||
checks initial gravity alignment, tilt convergence, the recorded GT Max level pose,
|
||||
and yaw/pitch/roll through the shared VITURE basis conversion and recenter path.
|
||||
|
||||
Attribution and the scaffold's Apache-2.0 license are in `docs/third-party/rayneo/`.
|
||||
@@ -0,0 +1,140 @@
|
||||
# VITURE Android SDK integration
|
||||
|
||||
For the cross-brand feature matrix and user setup, see [smart-glasses support](../smart-glasses-support.md).
|
||||
|
||||
The optional adapter is built against the user-supplied VITURE XR Glasses SDK
|
||||
2.4.0. Set `droiddeck.vitureSdk=C:/path/to/VITURE_XR_Glasses_SDK_for_Android`
|
||||
in the ignored `local.properties`, or pass `-PvitureSdkDir=/path/to/sdk`.
|
||||
The directory must contain `include/` and `android/arm64-v8a/`.
|
||||
|
||||
Gradle compiles `droiddeck_viture`, packages `libglasses`, `libcarina_vio`, and
|
||||
`libcloud_protocol`, and includes the SDK's LICENSE and NOTICE under
|
||||
`assets/licenses/viture`. Vendor headers and binaries stay outside the repository.
|
||||
A build without the optional SDK still supports the existing legacy VITURE and
|
||||
RayNeo transports, but does not advertise control for SDK-dependent glasses.
|
||||
|
||||
## Supported app functions
|
||||
|
||||
| Device | Transport and tracking | App controls |
|
||||
| --- | --- | --- |
|
||||
| Pro 2 XR | SDK Gen2 pose callbacks, observed OpenGL axes converted to NWU; negotiated pose frequency | 1080p 2D through 120 Hz, SBS 60 Hz, brightness 0–8, volume 0–8 |
|
||||
| Luma Ultra | SDK Carina USB transport; OpenGL poses converted to NWU | 1200p 2D through 120 Hz, SBS 60/90 Hz, brightness, volume, binary dimming |
|
||||
| Beast | SDK Gen2; native/bypass display commands selected by capability and current mode | 1200p output, SBS, brightness, volume 0–15, dimming toggle |
|
||||
|
||||
Pro 2 identification uses the SDK's PID validation and market name, not a guessed
|
||||
USB ID. Pro 2 has no electrochromic film control. Its display profile uses the
|
||||
[official 50° FOV and 1080p/SBS limits](https://www.viture.com/academy/product/viture-pro-2-xr-glasses).
|
||||
Device firmware can reject individual modes; Android's physical display size
|
||||
must confirm SBS before the app considers the switch successful.
|
||||
|
||||
All three feed the existing `Glasses.PoseSink`, head recentering, compositor and
|
||||
SteamVR pipeline. Ultra selects 3DoF or the opt-in positional 6DoF mode before SDK
|
||||
initialization. Its positional poses and stereo cameras feed the
|
||||
[Luma spatial pipeline](luma-spatial-tracking.md); camera passthrough is not implemented.
|
||||
The SDK's optional statistics reporter is not enabled.
|
||||
|
||||
The SDK owns USB interface selection and I/O. `UsbDeviceConnection` remains open
|
||||
until stop/shutdown/destroy finish. A generation token rejects callbacks from an
|
||||
old connection. Device brightness, volume and dimming events update the app.
|
||||
Beast native and standard display codes have separate meanings; they are mapped
|
||||
explicitly. Host tracking suspends native DOF to prevent applying rotation twice
|
||||
and restores the previous native DOF on stop. Older firmware rejecting a bypass
|
||||
mode can fall back to the native table, with rollback on failure.
|
||||
|
||||
`Game on glasses` uses Android's existing Presentation output. USB control alone
|
||||
does not create a video display: the phone, cable and adapter must expose a usable
|
||||
external display. Enabling 3D also enables and saves `Game on glasses` before
|
||||
requesting the stereo mode, so the picture uses the glasses' presentation instead
|
||||
of the mirrored phone output. Returning to 2D keeps the game on the glasses.
|
||||
Discovery checks both display name and product metadata, with
|
||||
a single unnamed-display fallback when recognized glasses USB is present.
|
||||
|
||||
Screen width and distance apply to **Virtual screen** rendering. In **Automatic**,
|
||||
ordinary 2D games stay flat. On an SBS display, the complete flat frame is fitted
|
||||
into each eye without cropping or head tracking. A running SteamVR session uses
|
||||
the direct stereo view. Games or videos that already contain separate eye images
|
||||
use the explicit **Side-by-side game** picture setting. Switching SBS on does not
|
||||
activate a virtual screen or head tracking for ordinary games.
|
||||
The direct view uses SteamVR field of view instead of virtual-screen dimensions
|
||||
while SteamVR is running. A plain side-by-side game fills each eye and does not
|
||||
use the SteamVR field-of-view or recenter controls.
|
||||
|
||||
All supported glasses with an available head-tracking connection have a separate
|
||||
**3DoF virtual screen** toggle, including VITURE, RayNeo and XREAL 1S. Availability
|
||||
comes from the connected transport's IMU capability rather than a model whitelist.
|
||||
The app renders this screen without enabling a built-in 3DoF mode or changing
|
||||
brightness. Enabling it selects Virtual screen, head tracking and Game on glasses,
|
||||
and saves those preferences without changing the glasses' 2D/SBS mode. It works
|
||||
with flat games in either display mode. Disabling it returns the picture to Automatic; switching
|
||||
SBS on or off preserves the 3DoF selection. Virtual screens center on the current
|
||||
viewing direction when tracking starts or the user recenters, including the
|
||||
up/down angle. Direct SteamVR views retain their gravity-aligned recentering.
|
||||
|
||||
The settings UI, menu presentation, and session renderer share this mode selection
|
||||
through `GlassesRendering`. Setup → Smart glasses configures DroidDeck menus,
|
||||
non-Steam launchers, desktop sessions, and games. Tracked menus use the app's own
|
||||
glasses presentation while phone controls remain available; see
|
||||
[tracked menus](glasses-menu-tracking.md). Regular refresh rate applies to 2D
|
||||
output; SBS uses the model's separate 3D refresh rate.
|
||||
Both the home menu and session drawer use **Display**, **VR**, **Tracking**, and
|
||||
**Device** tabs. Display groups connection and output controls, 2D/3D resolution
|
||||
and active refresh rate, then picture and virtual-screen geometry. VR contains
|
||||
SteamVR controls, automatic 3D, VR resolution, and field of view. Tracking groups
|
||||
head orientation, position and floor, and hands with camera setup. Device contains
|
||||
brightness, volume, and sunblock. Unsupported hardware tabs are hidden. Recenter
|
||||
is the first action on every session tab while head tracking is enabled for a
|
||||
3DoF/6DoF virtual screen or SteamVR.
|
||||
|
||||
Display saves **2D resolution** and **3D resolution**; VR saves **VR resolution**. Values are per eye: 1920 x 1080 or, on tall VITURE
|
||||
models, 1920 x 1200. Automatic keeps the existing model default; VR defaults to
|
||||
following the active 2D/3D preference. Home settings expose all three profiles,
|
||||
while a running session shows the active profile next to its refresh control.
|
||||
Switching resolution retains the refresh preference and does not change brightness,
|
||||
color processing, or the session's game-rendering resolution. Android selects an
|
||||
advertised mode with matching dimensions and the closest available refresh rate,
|
||||
without crossing the 2D/SBS layout. VITURE's USB command selects the matching
|
||||
resolution family first. A saved resolution unsupported by another attached model
|
||||
falls back to its automatic behavior. Resolution controls wait for a pending SBS
|
||||
transition to finish before accepting another change.
|
||||
|
||||
Both lists offer 30 Hz only when Android advertises a matching native display mode at the current
|
||||
resolution and **Game on glasses** is enabled. Android's presentation selects that mode without a
|
||||
vendor command. A saved 30 Hz choice falls back to 60 Hz if the display or eye layout lacks it;
|
||||
mirrored output never substitutes a 30 fps rendering limit.
|
||||
Software picture controls remain available while a display is connected even if
|
||||
USB control is lost; hardware controls and head tracking still require USB.
|
||||
Presentation refresh preferences are revalidated on display changes even when
|
||||
the requested Hz is unchanged, and keep both dimensions of the active mode.
|
||||
|
||||
## Validation
|
||||
|
||||
Run `:app:externalNativeBuildDebug` and
|
||||
`:app:testDebugUnitTest --tests 'com.droiddeck.launcher.glasses.*'` with the SDK
|
||||
configured. Compile `app/src/test/cpp/viture_display_modes_test.cpp` with the NDK
|
||||
C++17 compiler and the SDK include directory to check native/standard mappings
|
||||
against the vendor constants. The Kotlin tests cover model capabilities, axis
|
||||
conversion, invalid poses, rejected tracking starts and stale callbacks.
|
||||
|
||||
Physical acceptance checks remain: connect each model, grant USB permission,
|
||||
verify the Play SteamVR control, verify game output moves to the glasses,
|
||||
test head rotation/recenter and SBS, then unplug/reconnect. Building and unit
|
||||
testing do not establish hardware compatibility for every firmware revision.
|
||||
|
||||
### Pro 2 pose-axis correction, 2026-10-14
|
||||
|
||||
The connected Pro 2 delivered approximately 200 normalized poses/sec. The wearer
|
||||
reported yaw moving the view vertically, pitch rolling it, and roll turning it
|
||||
horizontally. Directed captures showed a left turn primarily increasing quaternion
|
||||
Y (approximately 0.004 to 0.253), while repeated upward looks varied quaternion X
|
||||
(approximately -0.187 to +0.082) with Y/Z remaining near zero. This is the OpenGL
|
||||
right/up/back basis, despite the SDK 2.4.0 generic Gen2 callback documentation
|
||||
describing NWU.
|
||||
|
||||
The adapter therefore selects the existing OpenGL-to-shared-glasses conversion
|
||||
for the SDK-identified Pro 2 family as well as Carina. Other Gen2 models retain
|
||||
their previous mapping. Tests exercise model selection and yaw/pitch/roll through
|
||||
the complete shared `HeadPose` conversion and recenter path. This fixes a model
|
||||
coordinate mismatch; it does not claim to run a vendor sensor-calibration command
|
||||
or save a RayNeo-style calibration for VITURE. All 188 glasses/shared-pose/
|
||||
SteamVR-launch/UI regression checks passed, packaged runtime inputs verified,
|
||||
and the corrected build was installed on the phone. Visual verification is pending.
|
||||
@@ -0,0 +1,451 @@
|
||||
# x86-64 Steam client
|
||||
|
||||
Valve's x86-64 Linux Steam client, run unchanged under FEX beside the default arm64 client. It is
|
||||
opt-in: Steam settings, **Steam client: x86-64** (`steamClientArch`, passed to the
|
||||
session as `BL_STEAM_CLIENT_ARCH=x86_64`, applies from the next session), or the desktop menu entry
|
||||
**Steam x86-64 (FEX, experimental)**. The Games tab lists the active client's games and tags its
|
||||
header "x86-64" while that client's list is shown. This page records why each workaround exists and what was seen
|
||||
on devices. The timeline is in `docs/progress-log.md`.
|
||||
|
||||
**Why.** The arm64 Linux client never runs VAC: Left 4 Dead 2 and Team Fortress 2 are refused by
|
||||
VAC-secured servers ("Connection Blocked ... VAC secured server") and no client log mentions VAC. VAC
|
||||
modules are x86 code. Under FEX the x86-64 client has a module it can run, as SteamLite does in Wine.
|
||||
|
||||
**Layout.** Everything lives in its own home, `$X64_HOME` (`BL_X64_HOME`, default `~/.droiddeck-x64`):
|
||||
its own `~/.steam` links, config, libraries, x86 rootfs and log (`steam-x64.log`). Shared with the arm64
|
||||
client: the sign-in, and its `config/uioverrides` (Steam's startup movie overrides), which the x86 client
|
||||
links. One client runs at a time, since both use the same account; the standalone launcher shuts the
|
||||
arm64 client down first.
|
||||
|
||||
| Command | Does |
|
||||
| --- | --- |
|
||||
| `droiddeck-steam-x64 [steam:// URL or client arguments]` | standalone run (the desktop menu entry) |
|
||||
| `droiddeck-steam-x64 --prepare` | install and set up, then exit |
|
||||
| `droiddeck-steam-x64 --exec ARGS...` | run the client binary in its environment (how the session starts it) |
|
||||
| `droiddeck-steam-x64 --sync-login` | copy the newer sign-in to the other client |
|
||||
| `droiddeck-steam-x64 --map-direct [--live]` | map Linux titles to DroidDeck direct |
|
||||
| `droiddeck-steam-x64 --default-tool` | print the default ARM64 Proton wrapper's tool name |
|
||||
| `droiddeck-steam-x64 --gpu` / `--webhelper-gpu` | exit status: client / x86 web helper on the GPU |
|
||||
|
||||
## Client install and update
|
||||
|
||||
- The first run downloads FEX (release `FEX_TAG` in Droid-Deck/DroidDeck-Components, sha256-pinned), FEX's Ubuntu 24.04 x86
|
||||
rootfs (about 525 MB, 1.5 GB unpacked, sha256-pinned; the same repository's copy of FEX-Emu's 2026-08-11 image) and the client (about 600 MB) from Valve's CDN,
|
||||
part by part with sha256 checks. A part-done rootfs download resumes; a finished one answers 416,
|
||||
which is not a failure. `unsquashfs -no-xattrs -ignore-errors`: ownership and xattrs mean nothing in
|
||||
the guest, and device nodes cannot be made and are not needed (FEX passes `/dev` through).
|
||||
- Channel `steamdeck_publicbeta`, the `-clientbeta` the session starts it with. `package/beta` must
|
||||
match `-clientbeta` before every start, or the client loops on rc 42 (its restart request, which the
|
||||
standalone launcher honours up to five times, as `steam.sh` does).
|
||||
- Before each start `x64_sync_client` brings the client to Valve's current build, downloading only the
|
||||
changed parts (9 of 28 on a device). If that fails, the client updates itself.
|
||||
- `.steam-enable-steamrt64-client` in the client folder opts into the SteamRT3 build
|
||||
(`steamrt64/steam`). Without it the client re-executes itself in the scout environment through
|
||||
`steam.sh`, whose start-up checks need what proot cannot give; the session stalled before the
|
||||
interface. The client is started directly, as the session starts `steamrtarm64/steam`:
|
||||
`STEAM_RUNTIME=1`, its own `steamrt64` libraries first, and the `~/.steam` links `steam.sh` makes.
|
||||
- **User namespaces.** Unlike the arm64 client, this one carries pressure-vessel and refuses to start
|
||||
("Steam now requires user namespaces") when `steam-runtime-check-requirements`' bwrap probe cannot
|
||||
create a user namespace, which no Android app can. `STEAM_RUNTIME_STEAMRT` points at
|
||||
`steamrt-shim`: Valve's runtime linked file by file with only the probe replaced (`exit 0`). proot
|
||||
is the container, as for the arm64 client.
|
||||
- **Re-exec after an update.** The client re-executes itself inside the SteamRT container through
|
||||
`pressure-vessel-wrap` (bwrap). `STEAM_STEAMRT_RECURSION_GUARD=1` says it is already inside one; the
|
||||
FEX rootfs with the client's `steamrt64` libraries is that container.
|
||||
- **Starting games.** Believing itself inside the container, the client starts games and system
|
||||
commands "alongside Steam" through `steam-runtime-launch-client`. Valve's asks a launcher service on
|
||||
the session bus (there is none: the game ended at once), and with none on `PATH` the launch fails
|
||||
(No such file). The shim's version applies `--directory`, `--env` and `--unset-env` and runs the
|
||||
command.
|
||||
- **Libraries.** The rootfs lacks what the SteamRT container gives the client (GTK 2 for
|
||||
`steamui.so`, ...). Valve's steamrt3c platform libraries are linked into `steamrt-libs`; they ship
|
||||
without soname links (pressure-vessel makes those when it builds the container), so the rootfs's
|
||||
`ldconfig` adds them there. `LD_LIBRARY_PATH` beats the system folders whatever its order, so only
|
||||
libraries the rootfs lacks are kept: the platform's older glibc must not shadow the rootfs's.
|
||||
- **System V IPC.** Android kernels have none, and the client's tier0 gave up on start ("Function not
|
||||
implemented" from `semget`; traced with a `-DDEBUG_STRACE` FEX). The session shim (`preload/*.c`) is
|
||||
also built for x86-64 and i386 and preloaded through the rootfs's own `/etc/ld.so.preload`, one copy
|
||||
per architecture under the multiarch folder named through glibc's `$LIB`. Without a list in the
|
||||
rootfs, FEX falls through to the guest's, which names the session's arm64 preloads.
|
||||
- **Tracer.** proot is a tracer, so tier0 believes a debugger is attached and breaks into it with int3
|
||||
on the first failed assertion: the web helper died of SIGTRAP a second after every start.
|
||||
`preload/tracer.c` exempts web helpers on purpose (Chromium traps on a status fd that is not
|
||||
procfs); with `BL_WEBHELPER_HIDE_TRACER=1` only opens called from `libtier0_s.so` get the copy
|
||||
without the tracer.
|
||||
- **FEXServer owns the rootfs.** A FEX process takes its rootfs from FEXServer, and a server started
|
||||
earlier without `FEX_ROOTFS` (any FEX run in the guest) serves none: every x86 library lookup fell
|
||||
through to the arm64 guest. It is stopped while no x86 client runs.
|
||||
- **`/dev/dri`.** x86 programs enumerated `/dev/dri`, found the session's KGSL node and spun on it (x86
|
||||
Turnip "failed to query kernel driver version", KMS dumb-buffer ioctls in a loop); the web helper
|
||||
never got a browser up. FEX looks in the rootfs first, so an empty `rootfs/dev/dri` hides the node.
|
||||
- **Scout runtime.** Valve ships it as `ubuntu12_32/steam-runtime.tar.xz` and `steam.sh` unpacks it.
|
||||
Since `steam.sh` is not used, `--prepare` unpacks it when its checksum changes.
|
||||
- **Repair.** Repair Steam (`BL_X64_REPAIR=1`), or two starts in a row that never showed the client
|
||||
(`package/droiddeck-unconfirmed`; the app clears the count once the client is on screen), removes
|
||||
the install stamp so the client is reinstalled. Counts from before this was tracked are dropped
|
||||
rather than taken as failed starts. `droiddeck-steam-install` removes the repair request when it is
|
||||
done, so the session reads it first and both clients are repaired.
|
||||
- **Stale state.** A killed client leaves `.crash`, which opens the update UI on its spinner, and
|
||||
`htmlcache/Singleton*` locks, which can leave the UI blank. Both are removed while no client runs.
|
||||
- **Network.** The client's connection header and Settings > Internet come from NetworkManager on the
|
||||
system bus; without it the client shows no connection while downloads run. The standalone launcher
|
||||
starts the session's system bus configuration and the `droiddeck-netmanager` stand-in when no
|
||||
system bus is up (a desktop has none).
|
||||
- **FEX settings.** The app's PERFORMANCE_TSO preset (`core/FexPreset.kt`, WinNative's default). FEX's
|
||||
own defaults keep half-barrier TSO and full-precision x87, which cost every x86 program time. Values
|
||||
already in the environment win.
|
||||
- **Fast path.** libblfastpath, preloaded into every guest process and so into FEX, broke FEX's 32-bit
|
||||
mode (SIGILL at start). It stands down when FEX is about to run a 32-bit program; 64-bit programs
|
||||
keep it. `BL_X64_FP_OFF=1` turns it off for everything.
|
||||
- **Diagnostics.** `BL_FEX` runs another FEX build (a `DEBUG_STRACE` one, say). `BL_FEXSERVER_LOG`
|
||||
starts a FEXServer in the foreground logging there; every FEX process that logs to "server"
|
||||
(`FEX_OUTPUTLOG` or an AppConfig) then has its lines tagged with its pid (file logs overwrite each
|
||||
other).
|
||||
|
||||
## Web helper on the GPU, QAM/WSI and flicker
|
||||
|
||||
The client's web helper draws its whole UI. There are two:
|
||||
|
||||
- **The arm64 client's helper** (`droiddeck-x64-arm-webhelper`), used when both clients are the same
|
||||
Steam build (unless `BL_X64_X86_WEBHELPER=1`). FEX hands the arm64 program to the CPU, so it runs
|
||||
natively with the session's preloads (KGSL poll fix, fast path) and the arm64 session's GPU
|
||||
settings. It talks to the client over IPC, and a helper of another build never comes up: the client
|
||||
restarted it every 10 s (device: 1790721607 against 1790904859). The session therefore updates the
|
||||
arm64 client's parts on the same channel first (`droiddeck-steam-install --sync
|
||||
steamdeck_publicbeta`). With this helper the client itself gets the GPU: it shows every Big Picture
|
||||
frame the helper draws, and in software (llvmpipe under FEX) that held the UI at about 20 fps
|
||||
(`BL_X64_CLIENT_SOFTWARE=1` keeps it in software). Device: 144 fps menus.
|
||||
- **The client's own x86 helper on the GPU** (`droiddeck-x64-webhelper`), otherwise: ANGLE's GL
|
||||
backend through FEX's GL thunk to zink and Turnip. On ANGLE's Vulkan backend the x86 compositor
|
||||
thread was the bottleneck (16 fps against 60). Never `--in-process-gpu`, the preload's default: a
|
||||
GPU crash there took the whole helper down (102 restarts). `BL_X64_CEF_GPU=0` keeps it in software.
|
||||
|
||||
The session shim (`preload/net.c`) turns `steamrt64/steamwebhelper.sh`'s exec of `./steamwebhelper`
|
||||
into `BL_X64_ARM_WEBHELPER`. Nothing in the client's folder is edited: the client checksums its files
|
||||
as it starts and reinstalled itself over an edited launcher (2-3 minutes) on every start.
|
||||
|
||||
Quick Access Menu and flicker, x86 helper:
|
||||
|
||||
- Steam's compositor in the helper draws each browser window (main UI, QAM) from that window's X
|
||||
pixmap. gamescope's WSI layer took the helper's zink swapchains and sent the frames to gamescope
|
||||
instead: the QAM opened blank until a d-pad press and the screen behind it went black. The helper
|
||||
runs with `DISABLE_GAMESCOPE_WSI=1`; games keep the layer. Not FEX: a texture-from-pixmap test
|
||||
behaved the same under FEX and natively.
|
||||
- gamescope sets `vk_xwayland_wait_ready=false` for its children because its layer does the waiting.
|
||||
Without the layer, unfinished frames reached the compositor as black flashes (3 in 10 QAM opens,
|
||||
screen-recorded and counted frame by frame). The helper sets `vk_xwayland_wait_ready=true`.
|
||||
- zink's kopper swapchain still flashed on about 1 in 20 opens with the wait. With
|
||||
`LIBGL_KOPPER_DISABLE=true` none did in 60 opens, at 60 fps instead of about 140 (Xwayland's
|
||||
per-window timing for child windows). No flashes was chosen over frame rate.
|
||||
|
||||
Software path (no host thunks, or `BL_X64_SOFTWARE=1`): x86 Mesa has no driver for this GPU, so
|
||||
llvmpipe for GL (the client's own window needs GLX) and the rootfs's own Vulkan ICDs (lavapipe). GLX
|
||||
first failed because the desktop exports zink over the arm64 Turnip ICD; those variables are cleared.
|
||||
`LIBGL_DRI3_DISABLE=1`: over DRI3 the X server hands out the GPU node and Mesa's software path then
|
||||
tries KMS dumb buffers on it in a loop.
|
||||
|
||||
Decky's loader talks to the client's CEF debugging port. Its marker
|
||||
(`.cef-enable-remote-debugging`) is kept in both clients' roots, because the session reads it from the
|
||||
client it runs. The app creates it in the x86-64 root only once that client is installed: the folder
|
||||
alone would then read as an installed client.
|
||||
|
||||
## FEX build and patches
|
||||
|
||||
- `tools/fex/build-in-arch.sh` runs in an Arch Linux ARM container (`menci/archlinuxarm:base-devel`)
|
||||
on an arm64 runner: Arch Linux ARM has no FEX package and FEX publishes no Linux binaries. It packs
|
||||
FEX and the runtime's own `unsquashfs` as `fex.tzst` (`usr/local/...`), with `fex-needed.txt` (every
|
||||
needed soname must resolve in a stock guest) and `fex-files.txt`. `TUNE_CPU=none`: the default
|
||||
(native) would tune for the runner's Neoverse cores, not the device.
|
||||
- **Host thunks** (`FEX_THUNKS=1`): an x86 program's libvulkan, libGL, libX11, ... calls go to the
|
||||
runtime's arm64 libraries (Turnip, Mesa). The host half needs the graphics development packages. The
|
||||
guest half is cross-compiled with clang against an x86 sysroot of Arch x86_64 and multilib i686
|
||||
packages, unpacked by pacman into their own root and never run (no scriptlets, no signature checks).
|
||||
FEX's x86 toolchain files assume an x86 build machine, so on arm64 the sysroot also supplies headers,
|
||||
C runtime objects and libgcc.
|
||||
- **Stack protector.** Arch's clang builds the guest thunks with stack protection. The canary load in
|
||||
`CallHostFunction`'s prologue (`mov %fs:0x28, %r11`) overwrote r11, which carries the host function's
|
||||
address, and FEX called the canary. 9 of 521 Vulkan and 66 of 736 GL host calls (those with 7 or
|
||||
more arguments) were affected, `vkCmdBindDescriptorSets` from TF2's DXVK among them; 32-bit games pass
|
||||
the host address in mm0. The toolchain files add `-fno-stack-protector` (`fex-2610-r2`).
|
||||
- **`vkGetDeviceProcAddr`.** FEX-2609's guest version passed a lookup of `vkGetDeviceProcAddr` itself
|
||||
to the host and handed back the raw host function, so every pointer fetched through it was unlinked.
|
||||
TF2's native DXVK (2.3) makes that lookup. FEX-2610 answers self-lookups in the guest (FEX-Emu/FEX
|
||||
3397eb9b).
|
||||
- Patches in `tools/fex/patches`:
|
||||
- `0001-openat2-enosys-fallback`: our proot answers `openat2` with ENOSYS (`tools/proot/PATCHES.md`,
|
||||
0008). FEX opened every rootfs path with `openat2(RESOLVE_IN_ROOT)` and fell through to the arm64
|
||||
guest on anything but EXDEV; x86 bash could not find libtinfo. Since r4 the fallback resolves the
|
||||
path inside the rootfs itself (absolute symlinks restart at the rootfs, as `RESOLVE_IN_ROOT` does);
|
||||
before, they resolved against the arm64 guest. `libblsession.so` (`preload/robust.c`) answers
|
||||
`openat2` first in practice and does the same walk for `RESOLVE_IN_ROOT` callers, so Valve's FEX-Emu
|
||||
on the arm64 client gets it too. Each path component is then its own system call.
|
||||
- `0002-thunkgen-newer-clang`: the container's newer clang has no `FileManager::clearStatCache()`,
|
||||
which the thunk generator called after each run.
|
||||
- `0003-allocator32-map32bit-exhaustion`: Counter-Strike 2 reserves the low 2 GB with `MAP_32BIT`
|
||||
in shrinking sizes. FEX's `MemAllocator32Bit` met EEXIST, stepped down one page per retry and
|
||||
restarted from the top forever (found with `perf_event_open` IP sampling, since proot holds
|
||||
ptrace). Such a range is now skipped whole and a second exhausted scan returns `-ENOMEM`, as the
|
||||
kernel does (`fex-2610-r3`).
|
||||
- **Building and publishing.** Droid-Deck/DroidDeck-Components' `build-fex.yml` builds `tools/fex` from
|
||||
this repository at a chosen ref (default `main`) with host thunks, in the same Arch Linux ARM container
|
||||
with a read-only token, then a separate job replaces `fex.tzst` on Components' single `fex` release,
|
||||
checks the digest GitHub stored, and records the build's version, sha256 and size in its `fex.json`
|
||||
(in that order: a device that reads the old `fex.json` meanwhile gets a mismatching archive and keeps
|
||||
its FEX). This repository's `build-fex.yml` only builds `tools/fex` when it changes (and on dispatch,
|
||||
for diagnostic builds such as `-DDEBUG_STRACE`); nothing is published from it.
|
||||
- Releases so far: `fex-2609-r2` (no thunks), `fex-2609-r3` (thunks), `fex-2610-r1`, `fex-2610-r2`,
|
||||
`fex-2610-r3`, `fex-2610-r4` (in-root resolution when openat2 is missing, exact MAP_32BIT
|
||||
marking), `fex-2610-r5` (current: bundles libxxhash, which the Linux runtime does not ship; the
|
||||
build fails when a program in the package needs a library that is neither bundled nor listed in
|
||||
`tools/fex/runtime-libs.txt`). They were built and published in The412Banner/DroidDeck; up to r3 their tags point at that
|
||||
repository's `main`, from r4 at the commit each was built from. Later builds are versions on the single
|
||||
`fex` release; `fex-2610-r5` stays published as the build `droiddeck-steam-x64` pins.
|
||||
- **Where the app gets it.** The app downloads FEX from Droid-Deck/DroidDeck-Components, where nothing is
|
||||
uploaded by hand. The app follows its `fex.json`: while the x86-64 client is selected it reads it from
|
||||
Components' `main` (directly, at most every ten minutes) and keeps the last copy, and a session passes
|
||||
that build to `droiddeck-steam-x64` (`BL_FEX_TAG`, `BL_FEX_SHA256`, `BL_FEX_URLS` with the download
|
||||
source's mirrors). The script installs a build only after its archive matches the sha256 and every
|
||||
program in it loads (`ld-linux --list`); the files it replaces are kept aside until the installed copy
|
||||
loads too and put back otherwise, and the tag is written last. Otherwise it keeps the installed one while that still loads,
|
||||
then falls back to `FEX_TAG`/`FEX_SHA256` built in, which stay the first-start default when no copy
|
||||
of `fex.json` has been read. Publishing a FEX build therefore takes a Components run only; a
|
||||
replaced archive cannot be pointed back to, so undoing a build means building an earlier ref. The rootfs is
|
||||
FEX-Emu's own image, unmodified: `fex-rootfs.json` there names its rootfs.fex-emu.gg link, sha256 and
|
||||
size, and `mirror-upstream.yml` fetches it into the catalog's release only when both match. A newer
|
||||
image is a new catalog release; `ROOTFS_URL` and `ROOTFS_SHA256` then move to it, and a device that
|
||||
already unpacked one keeps it.
|
||||
|
||||
- **What a session log shows.** `FEX ready (<tag>, <source>)` names the build and where it came from
|
||||
(`catalog`, `built-in`, or `kept` when a newer one did not install, with the reason a line above);
|
||||
`x86-64 rootfs:` names the image; `x86-64 client:` the client's ubuntu12 and steamrt64 builds at start
|
||||
(it may update itself after). The rootfs download reports every tenth of its size as a step. When the
|
||||
web helper's GPU process keeps exiting until Chromium gives up on the GPU (the menus then never appear),
|
||||
one line says so with the exit codes, followed by x86 `glxinfo -B` and `vulkaninfo --summary` through
|
||||
FEX's thunks.
|
||||
|
||||
## DroidDeck direct for Linux games and runtimes
|
||||
|
||||
- Steam runs native Linux games in the Steam Linux Runtime container (pressure-vessel), which needs
|
||||
bwrap and user namespaces. **DroidDeck direct** (`compatibilitytools.d/droiddeck-direct`, linux to
|
||||
linux) runs them under FEX with no container. Scout titles run through Valve's scout library runtime
|
||||
(`ubuntu12_32/steam-runtime/run.sh`). Titles Valve gives a newer runtime (sniper, soldier; listed in
|
||||
`direct-sniper`) are built for newer libraries than scout's and run on the rootfs (Ubuntu 24.04) as
|
||||
it is.
|
||||
- **Mapping** (`x64_map_direct`): every game the client's appinfo cache shows with a Linux build (type
|
||||
game, oslist with linux; runtime from `config/app_mappings`: SteamLinuxRuntime_sniper or _soldier,
|
||||
none means scout), installed or not, so Steam downloads the Linux depots. A title on a tool chosen in
|
||||
Properties keeps it, except `BL_X64_DIRECT_APPS` (Left 4 Dead 2, 550, by default), which is always
|
||||
taken back. `direct-auto.json` lists what was mapped. One pass over the mapping: a 2000-title fixture
|
||||
went from 16.5 s to 0.2 s with the same output.
|
||||
- A game whose Windows build is installed (it ran on a Proton before) stays a Windows game on the
|
||||
default Proton. Steam does not swap an installed build when the tool changes: DroidDeck direct was
|
||||
handed `Launcher.exe` and the game ended at once (Borderlands 2), and getting the Linux build back
|
||||
means downloading the game again.
|
||||
- The client rewrites `config.vdf` from memory as it exits, so the mapping is written only while no
|
||||
x86 client runs: by `--prepare`, and by the session after each exit, when the appinfo cache also
|
||||
knows titles bought during the run. `--live` (the session's pass while the client runs) writes
|
||||
nothing; installed Linux titles it would map go to the session's pending list.
|
||||
- A new user's client has no `config.vdf` before its first start, so Steam Play's default and the
|
||||
Linux titles would wait a session. `--prepare` writes the `CompatToolMapping` block alone, which the
|
||||
client keeps.
|
||||
- **Sniper check.** `tf.sh` reads `VERSION_CODENAME` from `/etc/os-release` and stops unless it is
|
||||
`sniper`. For `direct-sniper` titles the run script runs such a script through `bash -c` with that
|
||||
line replaced by `VERSION_CODENAME=sniper`.
|
||||
- **HarfBuzz.** The rootfs's HarfBuzz needs a newer FreeType than a game may bring. Counter-Strike 2
|
||||
ships an older `libfreetype.so.6` and no HarfBuzz: its menu text library failed to load, the game
|
||||
flagged its files corrupt (a verify each start) and waited behind an error box that could not open.
|
||||
Newer-runtime titles get Valve's runtime HarfBuzz (2.7) preloaded from `sniper-compat`, as they would
|
||||
in its container.
|
||||
- **GPU.** Games get FEX's host thunks and the session's GPU settings (Turnip, Zink), saved by the
|
||||
launcher as `game-gpu.env` and rewritten whenever it runs; `game-gpu.local` is the user's, never
|
||||
rewritten, and is read after it. `BL_X64_GAME_SOFTWARE=1` keeps a game in software, with
|
||||
`mesa_glthread=false` (GL threading is no help with software GL under FEX).
|
||||
- Source games (`hl2.sh`, `hl2_linux`, `srcds`) get `-condebug` for the VAC diagnostics.
|
||||
- **Controllers.** `libfakeinput` is built for x86-64 and preloaded in the rootfs; Big Picture works
|
||||
with it. The 32-bit copy crashed Left 4 Dead 2: its fake devices reach the game's own SDL2 through
|
||||
udev with a missing attribute and SDL `strdup()`s the NULL (fault backtrace: strlen in libc <- `__strdup`
|
||||
<- libSDL2 <- `udev_device_get_sysattr_value`). 32-bit programs get only the session shim. The
|
||||
preload line names the x86_64 copy by full path, since with `$LIB` FEX falls through to the
|
||||
session's own `/usr/local/lib/i386-linux-gnu` copy.
|
||||
- Device: TF2 main menu at 358 fps on the GPU and about 84 fps in a match; Half-Life (32-bit) at
|
||||
101 fps; Left 4 Dead 2 at 51 fps (about 11 in software).
|
||||
- The arm64 client's route to x86 Linux titles, `droiddeck-fex`, is not wrapped for this client. Its
|
||||
wrapper ran when the client probed its tools after logon; the arm64 client took that for an x86
|
||||
program wanting FEX and installed Valve's FEX.
|
||||
- `droiddeck-fex` itself (arm64 client): Valve's FEX-Emu keeps its server socket in
|
||||
`$STEAM_COMPAT_DATA_PATH/fex-emu/Server` whatever `FEX_APP_DATA_LOCATION` or `FEX_SERVERSOCKETPATH`
|
||||
say. Under the app's data folder that path passed the 108 bytes a socket path may have (Half-Life:
|
||||
136), FEXServer never listened and the game ended at once. FEX's data folder in the runtime directory
|
||||
is short enough; the one under HOME is not, so `/tmp/fex` stands in for it.
|
||||
|
||||
## Libraries, SD card and sharing between clients
|
||||
|
||||
- **Separate libraries.** With folders shared, each client could update, verify or switch the
|
||||
other's games. A title with a Linux build runs as that in the x86 client and as the Windows build in
|
||||
the arm64 one, so opening it in the other client switched the files and the first client downloaded
|
||||
its version again (Left 4 Dead 2, 8.6 GB). Hard links do not help: switching platform makes Steam
|
||||
download depot 551 anyway. A game wanted in both clients is installed in both. Folders an earlier
|
||||
version shared are taken out of the x86 client's lists (written only while no x86 client runs); the
|
||||
games stay in the arm64 library.
|
||||
- **SD card.** The x86 client's card library is a folder inside the arm64 one (`/mnt/droiddeck-sd/x86-64`,
|
||||
`GameStorage.X64_CARD_DIR`, passed as `BL_X64_STEAM_LIBRARIES`), so the install dialog offers Internal
|
||||
Storage and SD card. compatdata, shadercache and runtime tools stay private, as in the arm64 card
|
||||
library. FEX runs x86 programs from the noexec exFAT card. The folder appears with the client's first
|
||||
session on the card and its `steamapps` with the first library write. Desktop sessions prepare it as
|
||||
well: desktop mode offers the x86 client whichever is selected, and `droiddeck-steam-x64` drops
|
||||
library folders it is not given. The arm64 card library may hold the arm64 copy of the same title,
|
||||
so the session's installed check looks only in the x86 card folder.
|
||||
- **Card downloads.** `storage.c` (no zero-filled reservation where `fallocate` is refused) and
|
||||
`dircache.c` recognised only `steamrtarm64/steam`, and the x86 client wrote each download in full as
|
||||
zeros first (Audiosurf: 415 MB written, 0 downloaded). After the fix: FlatOut, 1.19 GB, to the card
|
||||
in 51 s.
|
||||
- **Sign-in.** Before either client starts, the newer sign-in (`config/loginusers.vdf`, the login token
|
||||
in `local.vdf`, `AutoLoginUser` and `RememberPassword` in `registry.vdf`) is copied to the other, so
|
||||
signing in or switching accounts in one counts for both. Steam renews its token every few days and
|
||||
refuses an old copy (a 3-day-old one asked for the password), so the newest is copied at every start,
|
||||
never while either client runs, since each rewrites these files. A client that refused its token
|
||||
rewrites `local.vdf` with an empty `ConnectCache`: newest, but copied over the other it would sign
|
||||
that one out too (seen after the x86 client's "Invalid Password"), so it is never the source. The copy keeps
|
||||
the time stamp, so both match until one client signs in again. The rest of `registry.vdf`
|
||||
(language, skin) stays each client's.
|
||||
- **App side.** The library, art picker, grid sync and boot movie follow the running client, else the
|
||||
selected one; a changed client applies from the next session. Offline Mode is written to both
|
||||
clients' `loginusers.vdf`; until the x86 client's first start its sign-in is the arm64 one. The log
|
||||
redactor learns account names from both clients. The startup movie is staged into the x86 client's
|
||||
config only once its first start has created it; `--prepare` links the arm64 `config/uioverrides`
|
||||
and copies the `StartupMovie` choice.
|
||||
- **Storage page.** It lists both clients' games, the x86 ones labelled; a game can be installed in
|
||||
both and an id left over in several libraries. `steam://uninstall` reaches only the session's client,
|
||||
so the other client's copy is uninstalled after switching clients. Tools in the x86 client's
|
||||
libraries are shown as required (it runs its Linux games with Valve's runtimes). Proton tool files
|
||||
live only in the arm64 root; the x86 client has wrappers. An x86 client without `config.vdf` has
|
||||
never mapped a game.
|
||||
|
||||
## Proton for Windows titles in the x86 client
|
||||
|
||||
- Windows games run on the arm64 client's Protons; x86 Proton would need pressure-vessel. For each
|
||||
arm64 client tool (droiddeck-proton-arm64, DroidDeck-Proton, GE-Proton, CachyOS) `--prepare` makes a
|
||||
wrapper in the x86 client's `compatibilitytools.d` with the same folder and display name: Components
|
||||
swaps and per-title mappings are keyed by them. In the tool name `arm64` and `aarch64` become `a64`,
|
||||
because the x86 client ignores a tool named for arm64 ("for a different target platform linux
|
||||
arm64"). The session runs `--prepare` again after it installs a new Proton.
|
||||
- The wrapper is an x86 script under FEX. It starts `droiddeck-x64-arm-proton` through an arm64 binary
|
||||
(a copy of `/usr/bin/env`), which FEX hands straight to the CPU, so Wine and the game run as from the
|
||||
arm64 client: same Proton, launcher, Components swaps, per-game environment and Steam client library
|
||||
(`~/.steam/sdk64`).
|
||||
- `droiddeck-x64-arm-proton` drops FEX's settings and the x86 library paths and preloads, then restores
|
||||
the arm64 session's environment that `droiddeck-steam-x64` saved before it built the x86 one
|
||||
(`arm-session.env`: HOME, PATH, GPU, audio, input, fast path). Steam's per-game variables are not in
|
||||
it. It keeps the game display Steam chose: the saved `DISPLAY=:0` hid the game window and made
|
||||
DiRT Showdown exit after about 7 s. Proton takes its Steam client parts from the client's install
|
||||
folder, and the x86 client's are x86, so `STEAM_COMPAT_CLIENT_INSTALL_PATH` points at the arm64 root
|
||||
and `BL_X64_STEAM_ROOT` keeps the x86 one for install-script lookups.
|
||||
- **Game overlay.** The x86 client preloads its own x86 overlay; the arm64 one takes its place. The
|
||||
overlay marks the game's windows with the Steam app ID, and without it gamescope never shows them
|
||||
(FlatOut ran and played sound behind Steam's launch screen, "steam app id: 0"). The agent also tags
|
||||
untagged `steam_app_<id>` windows with `STEAM_GAME` on the game display when the x86 client runs
|
||||
(DiRT 3, an added game, then reached its title screen at 144 fps).
|
||||
- The x86 client's tools are not adopted as arm64 Protons: `adopt_extras` on its
|
||||
`compatibilitytools.d` rewrote DroidDeck direct as a Windows tool, and Steam installed Half-Life's
|
||||
Windows files. The arm64 Proton is adopted on demand at an x86 launch.
|
||||
- **Default.** The newest DroidDeck-Proton, then the newest GE-Proton (aarch64), else
|
||||
`droiddeck-proton-a64`. FlatOut from the card ran on DroidDeck-Proton 11-9.1 and exited on
|
||||
GE-Proton11-7 (no log). With droiddeck-proton-arm64 (Proton Experimental) Xwayland never got its
|
||||
window buffers back and FlatOut took the device's RAM with it (10.5 GB of dma-buf in 35 s);
|
||||
GE-Proton, same game and client, stayed flat. The session asks for GE-Proton when none is installed.
|
||||
`default-tool` records the default the script set; when it changes, the titles still on the old one
|
||||
move along (device: 113 titles off GE-Proton11-7). A default or title the user set to another tool
|
||||
keeps it. Before `default-tool` existed, the default the script set was an `-a64` wrapper at
|
||||
priority 75.
|
||||
- `steam-compatibility --x64` repoints entries the client aimed at a Proton it cannot start (Valve's
|
||||
`proton_*` names, and `droiddeck-proton-arm64` left by an arm64 compat pass) to the default wrapper.
|
||||
Tools in its own `compatibilitytools.d` are kept: the wrappers include `proton-cachyos-*-a64`, which
|
||||
starts with "proton" like Valve's.
|
||||
- **Added games** are written by the client the session runs and pointed at its default wrapper. The
|
||||
shortcuts pass keeps the x86 client's own default and state (`droiddeck-compat` beside its Steam
|
||||
folder), because the arm64 client's `state.json` lists which of that client's titles follow its
|
||||
default.
|
||||
- **Install scripts.** A game's own installers without a `HasRunKey` (Borderlands 2: VC++ 2005, 2008
|
||||
and 2010, DirectX, .NET 4.0) are marked under `Software\Valve\Steam\Apps\<appid>` (and the
|
||||
Wow6432Node view), so Steam skips them; this applies to both clients. Windows components install into
|
||||
the x86 client's prefix (DiRT Showdown and DiRT 3 needed OpenAL).
|
||||
|
||||
## VAC findings
|
||||
|
||||
- x86-64 client with Linux builds: secure. Left 4 Dead 2 joined a live Quick Match game, the server
|
||||
type that refuses the arm64 client; on the GPU an online server answered the A2S info query as
|
||||
"Valve Left4Dead 2 US East Server", VAC=1. TF2 Casual matchmaking put it on a VAC-secured Valve
|
||||
server. Half-Life's native build joined a VAC-secured server.
|
||||
- x86-64 client with a Windows build (TF2 on DroidDeck-Proton): insecure. Proton reaches Steam through
|
||||
the arm64 client's `steamclient.so`.
|
||||
- arm64 client: Left 4 Dead 2 was blocked on both Protons; Half-Life (Windows build) joined a vac=1
|
||||
GoldSrc server. Why the arm64 client fails is not established; it has no VAC module of its own to run.
|
||||
- Headless VAC testing is not possible: current Steam starts the saved login from its UI.
|
||||
- **VAC diagnostics** (Setup page). A Source game decides secure or insecure once, as it starts, and
|
||||
says so only in its own console ("You are in insecure mode. You must restart before you can connect
|
||||
to secure servers."). That console reaches a file only with `-condebug`: `<game>/<mod>/console.log`,
|
||||
or `<game>/game/<mod>/console.log` for Source 2. At session start the app writes
|
||||
`~/.droiddeck-vac-diagnostics` (start time, client, each console.log's length, since `-condebug`
|
||||
appends). The launch wrapper (`bl_vacdiag`) then adds `-condebug` to Source titles (found by
|
||||
`gameinfo.txt` or `gameinfo.gi`), flags `-insecure` in launch options and notes each launch. At the
|
||||
end the session copies what each console.log gained into `vac/`, scrubbed, and writes `vac.txt`: the
|
||||
game's secure/insecure lines, VAC lines in the client's logs, and which client files are present. A
|
||||
stopped session keeps its own copy of the marker and launch list; a new session waits for earlier
|
||||
collections, because it replaces the marker and Steam's logs.
|
||||
- "Connecting to" is only an attempt. L4D2 logs it, retries, and then refuses a VAC server with a
|
||||
dialog ("Connection Blocked: Please remove plug-ins before attempting to connect to a VAC secured
|
||||
server") that never reaches the console. Only "Connected to" means the game got in.
|
||||
- SteamLite's "INSECURE - VAC may kick" warning was our own parser reading the wrong log while the
|
||||
client was secure all along. `vac.txt` therefore quotes the lines it found and says "inconclusive"
|
||||
when the game never reached a server.
|
||||
|
||||
## Session integration, restarts and mapping
|
||||
|
||||
- With `BL_STEAM_CLIENT_ARCH=x86_64`, `droiddeck-session` updates the arm64 client's parts
|
||||
(`droiddeck-steam-install`, then `--sync steamdeck_publicbeta`), runs `droiddeck-steam-x64 --prepare`
|
||||
and starts the client with `droiddeck-steam-x64 --exec`. Steps only for the arm64 client (library
|
||||
registration, the ARM64 Proton fetch, the live compat helper) are skipped; added games,
|
||||
`steam-compatibility --x64`, UI scale and redistributables run for the x86 client.
|
||||
- **Deck mode.** With `-steamos3` the client's SteamOS audio manager passed libpulse the NULL
|
||||
operation of a failed request, and libpulse aborted the client right after start ("Assertion 'o'
|
||||
failed at ../src/pulse/operation.c:136 pa_operation_get_state()", the runtime's PulseAudio 13).
|
||||
`preload/pulseguard.c` (x86 builds only; the arm64 client does not hit it) answers a missing operation
|
||||
as cancelled and ignores its unref and cancel. The session passes `-steamdeck -steamos3` unless
|
||||
`BL_X64_DECK=0`. Switch to Desktop is expected to need no change (an x86 shell under FEX runs
|
||||
`steamos-session-select`); the full switch has not been tested on a device.
|
||||
- **Process names.** Under FEX a process's argv[0] is FEX and the program is its first argument. The
|
||||
session's game detection, CPU pinning and the zRAM watcher read that argument. Steam pins itself and
|
||||
its web helper to a few cores, so the session's CPU set is applied again every 5 s.
|
||||
- **Live mapping and restarts.** While the client runs, the session's background loop runs
|
||||
`steam-compatibility --x64 --live` and `droiddeck-steam-x64 --map-direct --live`, every 15 s and
|
||||
only in the session's first hour (720 beats of 5 s). A title installed after that is mapped when the
|
||||
client exits and takes effect from the next session. Steam reads a
|
||||
mapping changed while it runs only at its next start, so changed titles go to `.bl-compat-pending`.
|
||||
Once they have installed (`StateFlags` 4 in the client's root or its card folder) and no game runs,
|
||||
the client is restarted once per title, at most 3 times a session.
|
||||
- **After exit.** Added games, `steam-compatibility --x64` and `--map-direct` run again: the client
|
||||
wrote its mapping from memory as it exited, over what the live passes set.
|
||||
- The Lossless Scaling ownership probe and the Steam import watcher reach the x86 client through the
|
||||
arm64 client's `steamclient.so` (`BL_STEAMCLIENT`); the watcher restarts the client with
|
||||
`droiddeck-steam-x64 --exec -shutdown`.
|
||||
- Steam logs and the Steam language come from the x86 client's home. On the desktop, `steam` and
|
||||
`steamdeck-steam` start the selected client, and the desktop game entries list its libraries.
|
||||
- The standalone launcher (desktop menu) shuts the arm64 client down first, starts the system bus and
|
||||
the NetworkManager stand-in when missing, clears stale locks, pins CPUs and restarts the client on
|
||||
rc 42.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Experimental and opt-in; one client runs at a time.
|
||||
- Windows titles in the x86 client are not VAC-secure: Proton reaches Steam through the arm64
|
||||
`steamclient.so`.
|
||||
- 32-bit games get no controller reader (the i386 `libfakeinput` is left out).
|
||||
- 32-bit Linux titles through the arm64 client's `droiddeck-fex` still fail (not pursued).
|
||||
- The x86 web helper's menus run at 60 fps without kopper.
|
||||
- Counter-Strike 2 reaches its menu (and loaded Dust II at 78 fps), then Android's `lmkd` kills
|
||||
DroidDeck. The game's GPU memory is pinned: KGSL `page_alloc` followed Shmem through the load, to
|
||||
about 3.2 GB at the kill, with Unevictable just above it. This device's `lmkd` runs in minfree mode,
|
||||
which does not count swap, while MemAvailable counted that memory as available. Paging the game out
|
||||
with a root tool (`process_madvise(MADV_PAGEOUT)`) let it draw its menu, but `lmkd` still killed the
|
||||
session. Not solved.
|
||||
- Sharing libraries between the clients is parked.
|
||||
@@ -0,0 +1,242 @@
|
||||
# XREAL 1S / One Pro feasibility — 2026-10-14
|
||||
|
||||
The user owns XREAL 1S and also wants One Pro support. Supporting these in DroidDeck
|
||||
is feasible based on existing open drivers, but the supplied Monado Air driver is
|
||||
not their transport. The initial investigation below preceded implementation.
|
||||
An experimental native 1S driver is now enabled; see the hardware validation
|
||||
section at the end for what has actually been verified.
|
||||
|
||||
## Supplied Monado source
|
||||
|
||||
Inspected `C:/Users/conno/Downloads/monado-main/monado-main/src/xrt/drivers/xreal_air/`
|
||||
and `src/xrt/targets/common/target_builder_xreal_air.c`.
|
||||
|
||||
| Model | USB vendor:product | Sensor/control interface | Sensor report |
|
||||
| --- | --- | --- | --- |
|
||||
| Air | 3318:0424 | 3 / 4 | 64 bytes |
|
||||
| Air 2 | 3318:0428 | 3 / 4 | 64 bytes |
|
||||
| Air 2 Pro | 3318:0432 | 3 / 4 | 64 bytes |
|
||||
| Air 2 Ultra | 3318:0426 | 2 / 0 | 512 bytes |
|
||||
|
||||
This driver implements HID IMU input, segmented factory-calibration JSON reads,
|
||||
gravity/gyro fusion, brightness 0–7 and 2D/3D mode commands. It does not register
|
||||
1S or One Pro. Copying those Air HID commands to the new products is unjustified.
|
||||
Its source carries Tobias Frisch's BSL-1.0 notice.
|
||||
|
||||
## Applicable One-family references
|
||||
|
||||
* [XRLinuxDriver](https://github.com/wheaney/XRLinuxDriver), inspected commit
|
||||
`36e2139dd482f5984112d0c3ebafcc07a78f99c0`. `src/devices/xreal.c` lists vendor
|
||||
`3318`, One Pro products `0435/0436`, and 1S products `043d/043e`. These use the
|
||||
non-HID transport. Its README explicitly lists both models and requires current
|
||||
firmware with glasses stabilization/anchor disabled when using external tracking.
|
||||
* [xrealOneDeviceKit](https://github.com/wheaney/xrealOneDeviceKit), inspected commit
|
||||
`f5833f611265f418b1ea549da20dc4764d86a4ec`. Its IMU adapter consumes the One TCP
|
||||
driver, interprets gyro as radians/sec and acceleration as m/s², then feeds fusion.
|
||||
Its factory-calibration JSON hook returns unsupported. The maintained Linux
|
||||
integration disables its SBS-control capability when no MCU connection is open;
|
||||
a static model table alone is not proof of working display switching.
|
||||
* [xreal_one_driver](https://github.com/rohitsangwan01/xreal_one_driver), MIT,
|
||||
inspected commit `3e912315ec16e94e9a47688770d649cfef293624`. Opens TCP
|
||||
`169.254.2.1:52998` with a two-second timeout, assembles 84-byte IMU records,
|
||||
reads a timestamp at byte 14, gyro floats at 34 and acceleration floats at 46.
|
||||
This is a transport/parser reference, not a drop-in Android glasses controller.
|
||||
|
||||
The source check confirms a practical path for both requested models. It does not
|
||||
establish that this Samsung phone exposes the glasses' USB Ethernet interface to
|
||||
ordinary Android apps, or establish the physical tracking axes on this hardware.
|
||||
|
||||
## DroidDeck integration path
|
||||
|
||||
1. Attach the 1S and inspect its USB descriptors, presentation display identity,
|
||||
Ethernet network/link properties and whether the IMU endpoint is reachable.
|
||||
Obtain the same evidence for One Pro before claiming hardware validation.
|
||||
2. Add exact model discovery and external-display selection. Preserve each panel's
|
||||
Android-advertised dimensions and modes; do not assume the Air's resolution.
|
||||
3. Implement a bounded One-family stream parser and cancellable socket transport.
|
||||
On Android, bind the socket to the relevant Ethernet `Network` when available,
|
||||
leaving Steam's Internet route alone. If Android does not expose that link,
|
||||
a USB network transport is additional work; no root requirement is assumed.
|
||||
4. Establish gyro units, timestamp deltas and axis mapping using stationary,
|
||||
directed yaw, pitch and roll captures. Route the validated fused pose through
|
||||
the same glasses/HeadTracker/SteamVR pipeline as VITURE and RayNeo. Avoid layering
|
||||
DroidDeck tracking on top of the glasses' own anchor/stabilization.
|
||||
5. Gate brightness and automatic stereo switching separately on a verified control
|
||||
protocol. IMU streaming alone does not justify showing those controls as working.
|
||||
6. Test malformed/fragmented/coalesced records, missing network, timeout/reconnect,
|
||||
USB detach, screen handoff, SteamVR launch and each physical movement direction.
|
||||
|
||||
At this initial investigation stage, no external SDK was installed, firmware was
|
||||
modified, or XREAL commands were sent.
|
||||
|
||||
## Further source investigation while the phone charges
|
||||
|
||||
Inspected [xr-tools](https://github.com/0xcaff/xr-tools), commit
|
||||
`65c11105434a998f09ac4a5f8e4de8736bb4d36c`, particularly
|
||||
`packages/xreal_one_driver/src/proto/net/`, `src/proto/usb/mod.rs`, and the IMU
|
||||
example. This describes considerably more of the One protocol than the simple
|
||||
TCP IMU reader above:
|
||||
|
||||
* Reports use TCP `169.254.2.1:52998`; controls use `169.254.2.1:52999`.
|
||||
Both have a two-byte message identifier followed by a four-byte **big-endian**
|
||||
body length. Control messages additionally contain a big-endian transaction ID
|
||||
and protobuf payload. A port must bound lengths, use unique transaction IDs,
|
||||
enforce deadlines, and close both sockets on detach.
|
||||
* Sensor messages `28 36` have a **128-byte body**, hence 134 bytes including the
|
||||
header. The older reader's 84-byte extraction is not the complete frame format.
|
||||
Body offsets: device ID 0, device timestamp 8, report type 24, gyro 28,
|
||||
acceleration 40, magnetometer 52, temperature 64 and IMU ID 68. Numeric sensor
|
||||
fields are little-endian; timestamp is documented as nanoseconds, gyro rad/s,
|
||||
acceleration m/s². Type 11 is IMU; type 4 is magnetometer. Do not interpret every
|
||||
report as gyro data or use arrival time as the device's integration interval.
|
||||
* Control `27 1f` reads factory JSON: IMU bias, temperature-dependent gyro bias,
|
||||
and display-to-IMU transforms. Its example subtracts the factory biases, fuses
|
||||
gyro/acceleration and applies the display rotation. This is preferable to
|
||||
guessing a fixed mounting angle, but the example's frame convention still
|
||||
needs conversion and directed-motion verification in DroidDeck.
|
||||
* `27 1c` sets brightness 0–9; `27 27` sets dimming; `28 22` selects regular/SBS
|
||||
input; `27 5e/5f` reads/sets display configuration; `28 a6/a7` reads/sets EIS.
|
||||
Documented display configurations are 1920×1080 at 60/90/120 Hz and 3840×1080
|
||||
at 60 Hz. These are evidence for One/One Pro, not confirmed 1S modes. Preserve
|
||||
the Android-advertised modes and verify controls separately on 1S firmware.
|
||||
* `28 29` is labelled buttons-enabled/buttons-disabled scene mode. Do not assume
|
||||
that command disables stabilization; the separate EIS control exists.
|
||||
* This project's USB detection labels `3318:0436` One Pro and `3318:0438` One as
|
||||
normal devices, and `0435/0437` as **recovery** devices. XRLinuxDriver accepting
|
||||
both IDs does not justify opening recovery products as ordinary glasses or
|
||||
sending firmware commands. xr-tools does not register 1S; its applicability
|
||||
there is a protocol-family inference, not hardware validation.
|
||||
|
||||
No license file or Cargo license declaration was found in this inspected xr-tools
|
||||
revision. No implementation was copied into DroidDeck. Use protocol observations
|
||||
for an independently written adapter, or resolve source licensing before a port.
|
||||
The MIT TCP driver and GPL XRLinuxDriver remain explicitly licensed references.
|
||||
|
||||
Also inspected XRLinuxDriver commit
|
||||
[`05d0cfb`](https://github.com/wheaney/XRLinuxDriver/commit/05d0cfb86abaacbf073e1a285ad4eabf681ccb05).
|
||||
It moves per-device pitch adjustment into the core pose path, after coordinate
|
||||
conversion; it does not add an Android transport or new One control protocol.
|
||||
Its pitch constants should not replace the GT mapping already verified by the user.
|
||||
|
||||
Next hardware check remains read-only: enumerate the 1S USB interfaces, display
|
||||
and Ethernet link; attempt bounded reads from both TCP services, save the factory
|
||||
configuration, then capture stationary/yaw/pitch/roll data. No XREAL support is
|
||||
advertised in this build. Android Ethernet availability and 1S control compatibility
|
||||
are still unverified.
|
||||
|
||||
## Official XREAL documentation and SDK inspection
|
||||
|
||||
Checked the user's [official documentation](https://docs.xreal.com/) on
|
||||
2026-10-14. This establishes an additional integration candidate alongside the
|
||||
direct TCP driver; it does not make that candidate a drop-in Android controller.
|
||||
|
||||
* The [device page](https://docs.xreal.com/XREALDevices/XREAL%20Glasses) explicitly
|
||||
includes One, One Pro and 1S in the X1-based One family with rotational tracking.
|
||||
* [SDK 3.1.0 release notes](https://docs.xreal.com/Release%20Note/XREAL%20SDK%203.1.0)
|
||||
add positional tracking for the One series with XREAL Eye and updated firmware.
|
||||
Beam Pro and Samsung S25 are its listed tested hosts. This does not establish
|
||||
6DoF on a camera-less 1S or validation on the user's SM-F976U.
|
||||
* [ControlGlasses](https://docs.xreal.com/Tools/ControlGlasses) is a required server
|
||||
for official SDK apps on compatible Android phones. It supplies device services,
|
||||
3D switching and dynamically loaded algorithm libraries. Beam Pro uses MyGlasses
|
||||
instead. An SDK integration would have a companion-app/version dependency.
|
||||
* The [quickstart](https://docs.xreal.com/Getting%20Started%20with%20XREAL%20SDK)
|
||||
describes Unity XR Plugin integration, ARM64 and OpenGL ES3. The
|
||||
[dual-display guide](https://docs.xreal.com/Tools/Dual%20Screen%20Display)
|
||||
confirms separate phone/glasses views and continued glasses output while the
|
||||
phone runs another app. This demonstrates the desired behavior in the SDK;
|
||||
it does not itself supply a native Kotlin implementation.
|
||||
* The [compatibility page](https://docs.xreal.com/XREALDevices/Compatibility)
|
||||
currently flags unresolved Android 16 compatibility. The
|
||||
[FAQ](https://docs.xreal.com/Frequently%20Asked%20Questions) also says DeX can
|
||||
interfere with SDK display operation, and Android 16 USB permission may require
|
||||
camera permission first when the device exposes an enabled UVC interface. Apply
|
||||
that permission requirement only when relevant; ordinary tracking must not
|
||||
start camera capture or enable UVC automatically.
|
||||
|
||||
Downloaded, but did not install or bundle, the official release archive linked
|
||||
from the 3.1.0 release notes:
|
||||
`https://public-resource.xreal.com/download/XREALSDK_Release_3.1.0.20251125/com.xreal.xr.tar.gz`.
|
||||
SHA-256: `fd7d0fce103712e7acc85bfcbb8b127007e1d84f64596efb6c11580bea058d90`.
|
||||
Size: 248,169,074 bytes.
|
||||
|
||||
Package inspection found Android AARs including `nr_api` and `nr_loader`, ARM64
|
||||
`libXREALXRPlugin.so` / `libXREALNativeSessionManager.so`, and Java
|
||||
`com.xreal.sdk.display.GlassesDisplay` plus display/surface listeners. The C# layer
|
||||
calls native exports for tracking type, resolution, refresh rate, component poses
|
||||
and session lifecycle; the loader also integrates Unity subsystems. Presence of
|
||||
these libraries does not establish a supported standalone JNI API or permission
|
||||
to redistribute a stripped subset. No C headers were present in the archive file
|
||||
listing inspected. No SDK source or binaries were added to the app.
|
||||
|
||||
Decision: retain both candidates. Before choosing the official native-library
|
||||
route, establish non-Unity initialization, head-pose access, ControlGlasses
|
||||
compatibility, display-surface ownership and distribution terms. Before choosing
|
||||
direct TCP, establish the Android network link and 1S protocol behavior. Compare
|
||||
both against DroidDeck's existing compositor instead of replacing its rendering
|
||||
architecture merely to fit the SDK's Unity sample.
|
||||
|
||||
## Native 1S implementation and hardware validation
|
||||
|
||||
The pre-implementation checkpoint is `4b045519`. The new driver uses independently
|
||||
written Kotlin TCP framing, control transactions, sensor parsing and gyro/gravity
|
||||
fusion. It does not bundle XREAL SDK binaries or copy the unlicensed xr-tools
|
||||
source. Protocol observations came from the references above and
|
||||
[xr-tools](https://github.com/0xcaff/xr-tools), inspected commit
|
||||
`65c11105434a998f09ac4a5f8e4de8736bb4d36c`.
|
||||
|
||||
Verified on the connected SM-F976U (Android reports release 17):
|
||||
|
||||
* Normal 1S USB identity is `3318:043e`, product name `XREAL 1S`. Only that exact
|
||||
product enables this driver; Air, One Pro and alternate/recovery IDs are excluded.
|
||||
* Android exposes NCM/ECM Ethernet networks at `169.254.2.10/24` and
|
||||
`169.254.1.10/24`. Individual sockets bound to the first network reach the
|
||||
glasses at `169.254.2.1`, control port 52999 and report port 52998. The process
|
||||
default network remains unchanged, preserving Steam's Internet connectivity.
|
||||
* Requests require the high bit of the transaction ID; responses clear that bit.
|
||||
Replies must match both command and transaction before values are published.
|
||||
* GET_DISPLAY returned payload `22 02 10 1d`: mode **29**, with Android reporting
|
||||
**1920 x 1200 at 60 Hz**. This differs from the older One's mode 2 (1080 lines).
|
||||
The 1S default 60 Hz mapping preserves mode 29.
|
||||
* A two-second capture delivered **2,000 valid IMU poses (1,000 Hz)**. Reports
|
||||
use the 128-byte body, type 11, little-endian device nanoseconds and float
|
||||
radians/sec / m/s² format. The framing handles partial and coalesced TCP reads.
|
||||
|
||||
The driver plugs into the shared glasses display and head-pose paths, including
|
||||
SteamVR. The FOV calculation now uses the actual per-eye aspect ratio, including
|
||||
the 1S's 16:10 display. Controls use acknowledged commands, bounded queues and
|
||||
timeouts; connection failures clear tracking state and retry. Volume is hidden
|
||||
because no volume protocol has been established. Missing Ethernet is reported
|
||||
without requesting HID permission or claiming USB interfaces from Android.
|
||||
|
||||
Directed wearer captures on the same device established the 1S sensor basis:
|
||||
level acceleration is approximately `(0, -9.65, -1.07)` m/s²; turning left gives
|
||||
negative gyro Y; looking up gives positive gyro X; tilting toward the right
|
||||
shoulder gives positive gyro Z with negative acceleration X. Both vectors are
|
||||
mapped as `(x, -y, -z)` into the shared right/up/back basis before initial gravity
|
||||
alignment and fusion. No RayNeo GT mounting angle is reused. Regression tests
|
||||
exercise those measured signs through `HeadPose`, check an upright first pose
|
||||
before recenter, and preserve the device's radians/sec and nanosecond units.
|
||||
|
||||
Stereo switching was verified at 3840 x 1080 / 60 Hz, returning to native
|
||||
1920 x 1200. On 1S, SET_INPUT performs the EDID switch itself; immediately sending
|
||||
SET_DISPLAY returns busy status 5001 (`22 03 08 89 27`). The driver now polls
|
||||
GET_DISPLAY across the temporary empty reply (`22 00`) and only sends a second
|
||||
mode command if the settled mode differs. Hardware retesting kept the connection
|
||||
and sensor stream running through the switch. The busy status is never accepted
|
||||
as a successful acknowledgement.
|
||||
|
||||
After installing the corrected build on 2026-10-14, the wearer confirmed that
|
||||
the view and motion feel good in response to the upright-view, yaw, pitch and
|
||||
head-tilt check. The SteamVR-requested session was running with a rendered first
|
||||
frame, stereo output and active tracking. All 181 glasses, shared-pose, SteamVR
|
||||
launch and glasses-UI regression checks passed; packaged runtime inputs verified.
|
||||
|
||||
The wearer also confirmed on 2026-10-14 that XREAL 1S brightness control works.
|
||||
Higher refresh rates were not tested.
|
||||
|
||||
Still experimental: dimmer, 90/120 Hz compatibility modes and
|
||||
extended gameplay stability need hardware validation. One successful view/motion
|
||||
check does not establish long-session stability. Users must select Follow mode and disable the glasses' own
|
||||
stabilization. Factory calibration and XREAL Eye/6DoF are not implemented.
|
||||
One Pro is not enabled or claimed as tested.
|
||||
@@ -7,6 +7,357 @@ the timeline, then lessons and backlog. Companion to the README (what the app *d
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-16/17 - `feat/x86-64-steam-client`: web helper on the GPU, org main, review rounds
|
||||
|
||||
- **x86-64 web helper on the GPU** (`f603a085`): the client's own steamwebhelper runs ANGLE's GL backend
|
||||
through FEX's GL thunk to zink/Turnip (ANGLE-Vulkan was CPU-bound at 16 fps). No arm64 helper needed.
|
||||
- **Quick Access Menu** (`26d80895`): the QAM opened blank until a d-pad press and the screen behind it went
|
||||
black. Steam's compositor draws each browser window from its X pixmap; gamescope's WSI layer took the
|
||||
helper's zink swapchains, so the pixmaps never got frames. The helper now runs with
|
||||
`DISABLE_GAMESCOPE_WSI=1` (games keep the layer). Not FEX: a texture-from-pixmap test behaved the same
|
||||
under FEX and native, so no fex-2610-r4.
|
||||
- **Black flashes** (`1b52df77`): gamescope sets `vk_xwayland_wait_ready=false` for its children; without
|
||||
its layer, unfinished frames reached the compositor (3 flashes in 10 QAM opens, screen-recorded and
|
||||
counted frame by frame). The helper waits for frames and presents without kopper: 0 flashes in 60 opens.
|
||||
The x86-64 menus run at 60 fps on that path (Xwayland's per-window timing for child windows); the user
|
||||
chose no flashes over ~140 fps.
|
||||
- **Org main**: rebased twice (522e755b, then e95a4226), linear, each final tree checked equal to a plain
|
||||
merge. Main's art picker, grid sync and boot movie follow the selected client (`32bbf79a`).
|
||||
- **CodeRabbit** on personal PRs #68 and #69 (full review of 76 files on #69). Fixed: one Steam channel for
|
||||
the x86-64 client; VAC report reads only console logs this session wrote, and a stopped session keeps its
|
||||
own marker; segvlog reads stack words without faulting and opens files with raw syscalls; the x86-64
|
||||
shortcuts pass no longer writes the arm64 client's compat `state.json`; DroidDeck direct mapping in one
|
||||
pass (2000-title fixture: same output, 16.5 s -> 0.2 s); session stop/start handoff under one lock;
|
||||
desktop sessions prepare the x86-64 card library; Decky's x86-64 marker only for an installed client;
|
||||
library follows the running client; download monitor restarts on the other client's log; compat
|
||||
restart reads the x86-64 card folder; CPU pinning sees FEX processes; build-fex has a read-only build
|
||||
job, a separate publish job, never republishes a tag, requires thunks and tags the built commit
|
||||
(`8e2751f9`..`f9de849c`). Skipped with evidence: IPC home for the probes (device: ownership probe
|
||||
succeeds in x86-64 sessions); shortened `STEAM_COMPAT_DATA_PATH` (read before the override).
|
||||
- **Device** (5004cf4e): TF2, Half-Life, Borderlands 2 and DiRT Showdown launch and play on the x86-64
|
||||
client; the arm64 client starts. BL2/Half-Life "downloads" were Steam shader pre-caching (6.1 GB Deck
|
||||
`steam_pipeline_cache.foz` for BL2), not game updates. Half-Life is the Windows build in the arm64 client
|
||||
and the Linux build in the x86-64 one, in separate folders.
|
||||
- **Org readiness** (2026-10-17): org main's rules are a PR with the `Build APK` check (ruleset "Protect main")
|
||||
and `AGENTS.md`'s one line on comments. Comments added by the branch went from about 440 lines to under
|
||||
100, each a non-obvious external constraint; the device history and reasons moved to
|
||||
`docs/development/x86-64-steam-client.md` (code unchanged, checked by stripping comments from both sides).
|
||||
The Steam client setting now says the library is the client's own and the sign-in is shared. Rebased
|
||||
onto 83eda433 (VITURE XR); squashed as personal PR #70, `feat:` title. FEX tags r1-r3 point at personal
|
||||
main, not their build commits; build-fex now tags the built commit and needs thunks to publish.
|
||||
- **PR #70 review**: added Linux-only titles in the x86-64 client went to `droiddeck-fex`, a tool that
|
||||
client lacks (now DroidDeck direct, and `droiddeck-fex` is retaken); pulseguard finds a libpulse loaded
|
||||
RTLD_LOCAL (Wine's winepulse.so); segvlog passes faults on to the program's own handlers after logging;
|
||||
Lossless's DLL search includes the x86-64 card folder; the log scrubber keeps account names when a
|
||||
`loginusers.vdf` cannot be read; build-diag runs with a read-only token.
|
||||
- **Org main 97f976ef** (Epic Games Launcher for Linux, #928): rebased onto it. Its new session code
|
||||
launched, restarted and recovered games through `steamrtarm64/steam`, and its added-game shortcut check
|
||||
read the arm64 client's Steam folder; both now use the session's client. Added games marked native are
|
||||
not given a Proton by either client's shortcuts pass.
|
||||
- **FEX hosted by the org**: `fex-2610-r3` is mirrored into Droid-Deck/DroidDeck-Components by its
|
||||
`mirror-release.yml`, checked against a new `fex.json` catalog there (Components PR #2), and
|
||||
`droiddeck-steam-x64` downloads it from there instead of The412Banner/DroidDeck.
|
||||
- **FEX fex-2610-r4** (PR #71 review): without openat2 (proot) FEX opened rootfs paths with absolute
|
||||
symlinks resolved against the arm64 guest, and `libblsession.so` converted every openat2 the same way
|
||||
before FEX saw ENOSYS. `preload/robust.c` now resolves `RESOLVE_IN_ROOT` paths component by component
|
||||
inside the root (59 cases; 100,000-path differential fuzz; new `test_openat2_preload.py`), and patch
|
||||
0001 does the same in FEX. Patch 0003 marks only the host mappings that overlap a failed MAP_32BIT
|
||||
range (the low-2 GB case: about a million mmap tries before, one now). Built with thunks, tagged at the
|
||||
built commit, mirrored into DroidDeck-Components (PR #3). The Steam client setting names the FEX build.
|
||||
The bring-up crash logger (`tools/diag`) left the branch. Device: FEX r4 downloaded on first start,
|
||||
client UI in 23 s (19-33 s before), TF2 played. The first session on the new build drew no SVG fills
|
||||
in Steam's UI (icons blank, outlines and text fine); a restart drew them, cause not found.
|
||||
- **Added games**: the scan treated every folder of the card library as a game folder except
|
||||
`steamapps`, so the x86-64 client's `x86-64/` library showed up as a custom game named "x86-64"
|
||||
with the exe of a Steam title installed there (FlatOut). `AddedGames.isSteamLibraryFolder` skips it
|
||||
like `steamapps` in the scan, the exe-to-folder lookup and the folder list (`AddedGamesScanTest`).
|
||||
- Rebased onto org main 45513c90; squashed again for a full automated review (personal PR #72,
|
||||
replacing #71).
|
||||
- **PR #72 review**: grid art, the content-log monitor, the repair marker, storage leftovers and the
|
||||
VAC diagnostics files are kept per client (the x86-64 client's first art sync was deleting chosen
|
||||
art as "removed in Steam"); the old games-folder migration skips the x86-64 library too.
|
||||
- **Which client**: the Games header shows a plain "x86-64" chip while the x86-64 client's list is
|
||||
shown (on its own line in a narrow pane); nothing changes for arm64. The setting reads "Applies next
|
||||
session. FEX 2610 r4" with the choices ARM64 and x86-64 (`GamesClientTagTest`).
|
||||
- **Stale list**: every refresh drops the Steam-client games when they belong to another client than
|
||||
the active one (a setting change, the end of a session, a resume), keeping added and store games;
|
||||
the cached list records its client in the same file (`library-cache.json`, saved under one lock). The next review of PR #72 had no
|
||||
findings. Device: the header chip and the removed "x86-64" entry seen on 04568806.
|
||||
- **x86-64 rootfs in DroidDeck-Components**: FEX-Emu's Ubuntu 24.04 image (2026-08-11, 525,615,104
|
||||
bytes, unmodified) is now fetched from Droid-Deck/DroidDeck-Components release
|
||||
`fex-rootfs-ubuntu-24.04-2026-08-11` instead of rootfs.fex-emu.gg. Components PR #4 added
|
||||
`fex-rootfs.json` and `mirror-upstream.yml` (fetches a catalog's upstream link, refuses it unless
|
||||
sha256 and size match, checks the digest GitHub stored); its run 39084452789 brought the file in.
|
||||
The sha256 the app pins is unchanged, so an unpacked rootfs is kept.
|
||||
- **One cache file**: the review of that change found the list and its client marker written as two
|
||||
files, so concurrent saves could pair one client's list with the other's marker. `library-cache.json`
|
||||
is now one object (`client`, `games`) written by one rename under a lock; a bare list from earlier
|
||||
builds reads as arm64's. Device (73e2a80b): the first start after install read the old list,
|
||||
rescanned and wrote `{"client":"x86_64","games":[...]}` with the same 23 games.
|
||||
- A scan reads the client it tagged the cache with (`Library.launchableGames(arch = ...)`), so a switch
|
||||
during a scan cannot pair one client's games with the other's tag; a failed rename drops the cache
|
||||
instead of leaving the previous list behind (`LibraryArchTest`).
|
||||
- The next review of PR #72 (a2552c58) had no findings; every review thread is resolved. Device: 43f9a8a8
|
||||
installed.
|
||||
- Device checks on 43f9a8a8: the setting reads "Applies next session. FEX 2610 r4" with the choices
|
||||
ARM64 and x86-64; picking ARM64 drops the chip and the Games tab showed the arm64 list on first
|
||||
look, and back on x86-64 the chip and the x86-64 list return (the cache records each client). Two
|
||||
custom games' chosen background art is kept, with the same file in both clients' grid folders.
|
||||
Team Fortress 2 launched from the Games tab with the x86-64 client selected: the session started
|
||||
the client, handed it app 440 and the Linux build (`tf_linux64` under FEX) took focus about 60 s
|
||||
after the tap.
|
||||
- **Translations**: the branch's 11 strings (the Steam client setting and its values, VAC diagnostics,
|
||||
the x86-64 loading title and the Storage labels) are in all 31 other locales, after
|
||||
`stores_registering` as in the English file, reusing each locale's wording for "Steam client",
|
||||
"Session logs" and "Uninstall" and its formal or informal address. Every file parses, every
|
||||
translation keeps its English placeholders, and no locale has a key the English file lacks.
|
||||
- The translations went to PR #72 (a43217c4) for review; org main is still 45513c90, so the branch
|
||||
needs no rebase before the org PR. Device: 9afb9bd5 installed.
|
||||
- **Org PR**: the draft #152 from 2026-10-02 (same branch name, 12 commits, conflicting with main) is
|
||||
refreshed as one commit on org main 45513c90 and marked ready, once the review of the translations
|
||||
is clean. Its old head is kept at `refs/backup/20261018/x86-64-steam-client-afa479ee`.
|
||||
- Korean: the review of the translations found `을(를)` left in the two uninstall strings; both client
|
||||
names end in a vowel when read, so they take `를`. The next review was clean.
|
||||
- **Submitted**: #152 is one commit (a53fd117) on org main 45513c90, ready for review; Build APK passed.
|
||||
Its `kwin` job (Build desktop package, KDE Plasma) failed for an unrelated reason: `build-kwin.sh`
|
||||
requires Arch Linux ARM's `kwin-6.7.5`, which `extra` replaced with 6.8.0 on 2026-10-15; the job ran
|
||||
only because refreshing the branch from its 2026-10-14 head spanned main's own `tools/desktop-kde`
|
||||
changes. This branch changes nothing there.
|
||||
- **Org review** (CodeRabbit on #152, 7 points): the library cache is saved only by the newest scan; a
|
||||
session records the client it ran (`steam-client.txt`) so an abandoned session's Steam logs come
|
||||
from that client; two console logs with the same game and mod get distinct names in the VAC report;
|
||||
CPU pinning also unwraps `FEXInterpreter`; the VAC hint names the game in pl, ru, uk, cs and bg.
|
||||
Pinning `actions/download-artifact` to a SHA was left out: no workflow here pins one. Follow-up
|
||||
commit eb8634ea.
|
||||
- Merged as ae94d9db.
|
||||
- **FEX r5** (report: a Galaxy S25 without the KDE desktop): the x86-64 client exited 127 at its first
|
||||
FEX call, `error while loading shared libraries: libxxhash.so.0`. FEX, FEXServer, FEXRootFSFetcher
|
||||
and FEXOfflineCompiler link libxxhash; linuxfs-r9 does not ship it, and the test device had it only
|
||||
from the KDE desktop package. r5 bundles it in `usr/local/lib` with RUNPATH `$ORIGIN/../lib` on
|
||||
those programs (the runtime has no ld.so.cache), and `build-in-arch.sh` now fails when a program
|
||||
needs a library that is neither bundled nor listed in `tools/fex/runtime-libs.txt` (the r9 libraries
|
||||
FEX uses). With the device's libxxhash hidden, r4 failed with the reported error and r5 loaded.
|
||||
Mirrored in DroidDeck-Components (#5); the app moves to fex-2610-r5 and an installed r4 is replaced
|
||||
at the next x86-64 start.
|
||||
- Merged as 7d21ba22.
|
||||
- **FEX from the catalog**: FEX builds may need to follow user reports faster than app builds, so the
|
||||
app now reads Components' `fex.json` (directly, kept between starts) and passes it to the session; a
|
||||
build is installed only once its sha256 matches and its programs load, the installed one stays while it
|
||||
loads, and the build pinned in the script remains the first-start default. **Session logs** for the
|
||||
x86-64 client now name the FEX build and its source, the rootfs and the client's builds, report the
|
||||
rootfs download in tenths instead of a progress bar (one report's log ended in hundreds of bar lines),
|
||||
and, when the web helper's GPU process keeps exiting (a report from an Adreno 840: the menus never
|
||||
appeared), say so in one line with x86 `glxinfo`/`vulkaninfo` through FEX. Settings shows the FEX
|
||||
build installed. FEX is now built and published by DroidDeck-Components' `build-fex.yml` from this
|
||||
repository's `tools/fex`, replacing the archive on its single `fex` release and recording the version
|
||||
in `fex.json`; this repository's `build-fex.yml` only checks the build. Tests: `test_fex_select.py`,
|
||||
`FexCatalogTest`.
|
||||
Components' first build there published `2610-r6` (run 39104443888). Device (Pocket FIT, r4
|
||||
installed): the app read `fex.json`, and the next x86-64 start logged `downloading FEX (fex-2610-r6)`
|
||||
then `FEX ready (fex-2610-r6, catalog)` a second later, with the rootfs and client lines; the client
|
||||
came up on it.
|
||||
- Review of #1053: an install keeps the files it replaces aside until the installed copy loads and puts
|
||||
them back otherwise (a failed copy could leave a partly replaced FEX marked as the new one;
|
||||
`test_fex_install.py`), and the catalog is read on its own thread, not before the library scan, and
|
||||
not in Offline Mode.
|
||||
|
||||
## 2026-10-15 - `feat/x86-64-steam-client`: SD card library, ARM parity, org main, Counter-Strike 2
|
||||
|
||||
- **SD card library** (`3fc74f3e`): the x86-64 client had internal storage as its only library, so Steam's
|
||||
install dialog never offered a place. It now has `x86-64/` inside the card library
|
||||
(`/mnt/droiddeck-sd/x86-64`; compatdata, shadercache and runtime tools private like the arm64 card
|
||||
library). Before building, FEX was shown to run x86 programs from the noexec exFAT card. Device: the
|
||||
dialog offers Internal Storage and SD card.
|
||||
- **Card downloads** (`f77113fb`): `storage.c` (no zero-filled reservation where `fallocate` is refused)
|
||||
and `dircache.c` recognised only `steamrtarm64/steam`; the x86-64 client wrote each download in full as
|
||||
zeros first (Audiosurf: 415 MB written, 0 downloaded). Device: FlatOut, 1.19 GB, to the card in 51 s.
|
||||
- **zRAM** (`877c27eb`): the watcher read argv[0], which under FEX is FEX; it now reads FEX's first argument.
|
||||
- **ARM parity** (`b0923f05`): touch layouts read the active client's files; Offline Mode is written to
|
||||
both clients' `loginusers.vdf`; the Lossless Scaling ownership check runs for x86-64; Repair Steam repairs
|
||||
the x86-64 client too (and two unshown starts do, like arm64; the app clears its start count). Device: no
|
||||
repair on the stale 31-start count, count cleared once the client was on screen. Switch to Desktop needs
|
||||
no change (Deck mode: an x86 shell under FEX runs `steamos-session-select`).
|
||||
- **DroidDeck-Proton default** (`2bd752aa`): newest DroidDeck-Proton before GE-Proton; `default-tool`
|
||||
records the default this script set, and on a change the titles still on it move along (device: 113 off
|
||||
GE-Proton11-7). A default the user chose is kept. FlatOut from the card: in game on DroidDeck-Proton
|
||||
11-9.1, exits on GE-Proton11-7 (no log).
|
||||
- **Org main** merged (`714198bf`, 36 commits; strings and `steam_started` conflicts kept both sides).
|
||||
A stale line from the 10-07 sync ran the live compat helper in the foreground on the arm64 restart path:
|
||||
black screen after a Steam import restart (`02e7c503` matches main again).
|
||||
- **VAC findings:** x86-64 client + Linux builds secure (TF2, HL, L4D2). x86-64 client + Windows build
|
||||
(TF2 on DroidDeck-Proton) insecure: Proton reaches Steam through the arm64 client's `steamclient.so`.
|
||||
arm64 client + L4D2 blocked on both Protons; arm64 + Half-Life (Windows) joined a vac=1 GoldSrc server.
|
||||
Why the arm64 client fails is not established. Sharing libraries between the clients stays parked.
|
||||
- **arm64 `droiddeck-fex`** (`a53b49ef`): Valve's FEX-Emu keeps its server socket in
|
||||
`$STEAM_COMPAT_DATA_PATH/fex-emu/Server` whatever `FEX_APP_DATA_LOCATION`/`FEX_SERVERSOCKETPATH` say;
|
||||
under the app's folder that path passed 108 bytes (Half-Life: 136) and FEXServer never listened. A short
|
||||
path is used now. 32-bit Linux titles through it still fail (not pursued).
|
||||
- **Counter-Strike 2** (Linux, newer runtime) hung at 100% of one core before loading its engine. With
|
||||
ptrace taken by proot, `perf_event_open` IP sampling put the loop in FEX's `MemAllocator32Bit`: CS2
|
||||
reserves the low 2 GB with `MAP_32BIT` in shrinking sizes, FEX met `EEXIST`, stepped down one page per
|
||||
retry and restarted from the top for ever. **`fex-2610-r3`** (patch `0003`, `b0d316dc`): such a range is
|
||||
skipped whole and a second exhausted scan returns `-ENOMEM`, as the kernel does. Next, its menu text
|
||||
library failed: CS2 ships an older `libfreetype.so.6` but no HarfBuzz, and the rootfs's needs a newer
|
||||
FreeType; the game flagged its files corrupt (a verify each start) and waited behind an error box that
|
||||
could not open. DroidDeck direct now preloads Valve's runtime HarfBuzz (2.7) for newer-runtime titles
|
||||
(`c9eb7d30`). Device with both done by hand: engine and modules loaded (5 GB) with no error, then Android
|
||||
killed DroidDeck for memory while it was in the background. Not yet seen at the menu.
|
||||
- **CS2 on the r3 APK** (`c9eb7d30`, no launch options): no hang, the menu text library loads, the game
|
||||
client and modules load and the window resizes. Every run then ended in Android's low-memory killer:
|
||||
CS2 goes from 3.5 to 5.3 GB in about 15 s and `lmkd` killed DroidDeck (adj 200, in the background
|
||||
while the device was in Termux) with zRAM on and still filling (1.5 to 3.2 GB of 6 GB). This device's
|
||||
`lmkd` runs in minfree mode (`persist.device_config.lmkd_native.use_minfree_levels=true`; background
|
||||
apps go below about 120 MB free), which does not count swap. Proposed, not applied: switch it to the
|
||||
PSI mode (`use_minfree_levels=false`, `lmkd.reinit=1`; reverted with `true`). CS2 has not been seen
|
||||
at its menu yet.
|
||||
- **CS2 reached its menu (14:47, DroidDeck in the foreground):** still loading, 11 fps, 11.6 GiB used.
|
||||
Five seconds later `lmkd` killed nearly every app, then DroidDeck itself at adj 0 (foreground), while
|
||||
swap was about 22 of 23 GB free (zRAM 6 GB with 0.9 GB used, `/data/swapfile` 16 GB almost unused).
|
||||
Steam and CS2 died with the app; neither crashed on its own.
|
||||
- **Root test without changing Android (lmkd left as is):** DroidDeck's zRAM watcher only pages out
|
||||
`steam` and `steamwebhelper`, never the game. A root tool (`/data/local/tmp/pageout cs2 3000 <log>`)
|
||||
calls `process_madvise(MADV_PAGEOUT)` on CS2's private anonymous memory whenever MemAvailable drops
|
||||
below 3 GB, so free RAM stays above the killer's levels. One call per mapping in 1 GB pieces: a single
|
||||
call with many ranges is capped at about 2 GB in total, and FEX reserves huge ranges, so batched calls
|
||||
silently skipped most of a FEX process. With that fixed, the x86 Steam client moved 223 of 256 MB to
|
||||
swap. If CS2 survives to its menu, the app version is the zRAM watcher also paging out the running
|
||||
game during its load (no root).
|
||||
- **Page-out result (14:59):** the tool fired at 2.9 GB available and moved 4.8 GB of CS2 to swap
|
||||
(resident 5.8 → 1.5 GB). CS2 drew its full menu (2–34 fps). 27 s later `lmkd` still killed down to
|
||||
DroidDeck at adj 0. Its kill record: MemFree 111 MB, Cached 6.3 GB, but **Shmem 3.4 GB and
|
||||
Unevictable 3.5 GB** (Mlocked only 120 MB), swap 17 GB free. In minfree mode the "file" figure is
|
||||
Cached minus Shmem, Unevictable and SwapCached, so about 0, and it kills to the foreground. After the
|
||||
session died Shmem fell to 0.3 GB, so the 3.4 GB of pinned shared memory belonged to the CS2 session.
|
||||
MemAvailable counted it as available, which is why the tool saw 3.7 GB of headroom 2 s before the
|
||||
kill. Page-out cannot move unevictable pages.
|
||||
- **Next run (15:04):** a root logger (`/data/local/tmp/shmlog.sh` → `shm-cs2.log`, every 3 s) records
|
||||
MemFree / Cached / Shmem / Unevictable / Mlocked / SwapFree and every process holding more than 50 MB
|
||||
of shared (`RssShmem`) or locked (`VmLck`) memory, to find who pins the 3.4 GB (FEX, the X server,
|
||||
Vulkan buffers, a ramfs). No SysV shared memory on the device. Then the trigger moves from
|
||||
MemAvailable to MemFree plus droppable file pages.
|
||||
- **Dust II (15:27):** CS2 loaded Dust II "Training Day" (bots) at 78 fps, then was killed at 15:28:05 the
|
||||
same way. From 15:25:30 Shmem and Unevictable rose from 0.25 to 3.9 / 4.0 GB, yet the largest process
|
||||
mapping shared memory (an `exe`) held only about 535 MB, so about 3.3 GB is not mapped by any process.
|
||||
- **Open files ruled out (15:32-15:36):** a second logger (`fdlog.sh` → `fd-cs2.log`) listed every memfd,
|
||||
ashmem, deleted or tmpfs file of 20 MB or more held by the app. Only 64 MB PulseAudio memfds showed up
|
||||
while Shmem reached 2.7 GB. The run was killed at 15:36:28 (Shmem 3.4 GB, Unevictable 3.5 GB).
|
||||
- **Lead: GPU memory.** `/sys/class/kgsl/kgsl/page_alloc_max` reads 6.07 GB (the peak since boot), and the
|
||||
driver has `max_reclaim_limit` / `page_reclaim_per_call`, i.e. a KGSL build with process reclaim, where
|
||||
GPU pages are shmem-backed and stay unevictable while the process is in the foreground. That fits
|
||||
memory that no process maps, that has no file, and that is freed when the session dies. Not proven.
|
||||
- **Low-settings test (15:42):** `cs2_video.txt` in the x86 userdata: `gpu_mem_level` 3 → 0,
|
||||
`cpu_level` 3 → 0, resolution 1024x576 → 854x480 (everything else was already at its lowest; backup
|
||||
`cs2_video.txt.before-low`). A third logger (`gpulog.sh` → `gpu-cs2.log`, every 3 s) records KGSL
|
||||
`page_alloc`, MemFree, Shmem, Unevictable and each process with more than 200 MB of GPU memory. If
|
||||
Shmem rises with `page_alloc`, the GPU memory is what is pinned. Background apps were force-stopped
|
||||
and caches dropped first (about 11 GB free).
|
||||
- **GPU memory confirmed (15:47):** with low settings CS2 reached its menu at 94 fps and was killed at
|
||||
15:47:59 the same way. KGSL `page_alloc` followed Shmem within 20-120 MB through the whole load
|
||||
(0.23 GB at 15:44:50, 0.93 GB at 15:46:47, then +2.2 GB in 3 s, about 3.2 GB at the kill), and
|
||||
Unevictable stayed about 100 MB above it. One process (`exe`, pid 23906) held nearly all of it. So the
|
||||
pinned 3.4 GB is the game's GPU memory, and the low settings did not make it smaller. (The negative
|
||||
values in that log were the device shell's 32-bit arithmetic; the logger now divides in awk.)
|
||||
- **New page-out trigger (16:23):** the tool now acts on what lmkd's minfree mode looks at, MemFree plus
|
||||
`Cached + Buffers - Shmem - Unevictable`, instead of MemAvailable (which counted the GPU memory as
|
||||
free). Threshold 2500 MB, checked every 0.5 s, page-out at most once a second. The idea: push CS2's
|
||||
swappable memory out early so the 3.2 GB of GPU memory fits beside it. The open-file logger was stopped.
|
||||
- **Still to test:** CS2 without the kill; touch layouts, Offline Mode, Switch to Desktop, a Linux game from
|
||||
the card. TF2 on the device is the Windows build after the VAC test.
|
||||
|
||||
## 2026-10-14 (evening) - `feat/x86-64-steam-client`: Team Fortress 2 runs (sniper check, FEX thunks)
|
||||
|
||||
- **Sniper check:** `tf.sh` reads `VERSION_CODENAME` from `/etc/os-release` and stops unless it is
|
||||
`sniper`. DroidDeck direct cannot use Valve's sniper container (no user namespaces under Android), so
|
||||
for titles in `direct-sniper` it runs such a script through `bash -c` with that line replaced by
|
||||
`VERSION_CODENAME=sniper` (`92d65e3d`). Device: TF2 gets past the check and loads on the rootfs
|
||||
libraries; sniper itself is not needed.
|
||||
- **Then a crash at the first frames**, with both launch options (Vulkan and Legacy OpenGL). Strict FEX
|
||||
memory ordering and turning off the Gamescope WSI layer changed nothing. With FEX thunks off (lavapipe,
|
||||
software) TF2 reached its main menu and loaded a match at 17 fps, so the thunks were the cause; with
|
||||
only the Vulkan thunk on it still crashed.
|
||||
- **Cause (from a SIGSEGV report taken inside the game):** the main thread was executing Turnip's
|
||||
`vk_common_CmdBindDescriptorSets` as x86 code, an address FEX never linked. FEX-2609's guest
|
||||
`vkGetDeviceProcAddr` passes a lookup of `vkGetDeviceProcAddr` itself to the host and hands back the
|
||||
raw host function; every pointer fetched through it is unlinked. TF2's native DXVK (2.3) makes that
|
||||
lookup; the Windows games through Proton do not. FEX-2610 answers self-lookups in the guest
|
||||
(FEX-Emu/FEX 3397eb9b).
|
||||
- **FEX 2610** (`af49e545`): `tools/fex/build-in-arch.sh` builds FEX-2610 with the same openat2 patch and
|
||||
host thunks as `fex-2609-r3`. The container's newer clang has no `FileManager::clearStatCache()`,
|
||||
which the thunk generator called after each run; patch `0002` drops that call (`31d66e67`). Published
|
||||
as `fex-2610-r1` on the personal repo.
|
||||
- **2610 alone did not fix it.** An ARM-side signal hook in FEX (wrapping its `syscall(rt_sigaction)`)
|
||||
logged no fault in Turnip, and the guest thunk's disassembly showed why: Arch's clang builds the x86
|
||||
guest thunks with stack protection, and the canary load in `CallHostFunction`'s prologue
|
||||
(`mov %fs:0x28, %r11`) overwrote r11, which carries the host function's address, before the thunk
|
||||
stored it. FEX called the canary. 9 of 521 Vulkan and 66 of 736 GL host calls (those with 7+
|
||||
arguments) were affected, `vkCmdBindDescriptorSets` among them. Windows games never reach these
|
||||
thunks (they run on the arm64 Proton) and 32-bit games pass the host address in mm0.
|
||||
- **`fex-2610-r2`** (`56e5a712`): the toolchain files add `-fno-stack-protector` for the guest thunks;
|
||||
no guest thunk library has a canary load. The x86 client fetches it (`9cc0ebcb`). Device, stock
|
||||
settings: TF2 main menu at 358 fps on the GPU, Casual matchmaking onto a VAC-secured Valve server,
|
||||
in game at about 84 fps; Half-Life (32-bit, direct) runs on it. Windows games are not on this FEX:
|
||||
in x86 client mode they still run on the arm64 Proton (Borderlands 2: GE-Proton11-7-aarch64), whose
|
||||
own FEX translates the game.
|
||||
- **Which FEX runs what in x86 client mode:** the client (`steamrt64/steam`) and every Linux game on
|
||||
DroidDeck direct run on the runtime's FEX (`fex-2610-r2`); Windows games go to the arm64 Proton and
|
||||
its own FEX. At each start every library game with a Linux build, installed or not, is mapped to
|
||||
DroidDeck direct, so Steam downloads the Linux depots; Windows-only games stay on the default
|
||||
Proton, and a game already installed as its Windows build stays there (Borderlands 2 here:
|
||||
`Binaries/Win32`, depot 49521). A game new to the library mid-session is mapped at the next start.
|
||||
- APK run 38670180921 (`9cc0ebcb`), signed and staged as `DroidDeck-x64-fex2610r2-9cc0ebcb.apk`.
|
||||
|
||||
## 2026-10-14 - `feat/x86-64-steam-client`: brought up to main; added games run through the x86-64 client
|
||||
|
||||
Work lives on The412Banner/DroidDeck only until it runs correctly; the org branch and draft PR #152
|
||||
stay as they were (`afa479ee`).
|
||||
|
||||
- **Merged main (94 commits).** Main dropped the Steam channel and Deck-mode settings (#546): only the
|
||||
client choice (`steamClientArch`), `activeSteamRoot` and VAC diagnostics remain. Main's web-helper
|
||||
arguments (#566, `bl_webhelper_argv`) are composed with the x86 client's arm64 web-helper redirect in
|
||||
`preload/net.c`. Main's single background loop (#575) now carries the x86 hooks: the client's own
|
||||
compat pass (`--x64 --live`), `steamrt64` in the pinning and stale-tmp patterns, Lossless probe off for
|
||||
x86, and the once-per-title restart after a new mapping. The text-strings merge had dropped three of
|
||||
main's strings; main's files were taken and the six x86 strings added back.
|
||||
- **Builds on the personal repo** need the pinned Debian image in a release of their own:
|
||||
`ci-debian-bullseye-r1` was copied (same sha256) as a pre-release. The bullseye compilers come from
|
||||
`deb.debian.org` first; `archive.debian.org` no longer serves `libmpfr6_4.1.0-3`.
|
||||
- **Client update before start** (`x64_sync_client`): Valve serves one build to both clients; the x86
|
||||
client is brought to it (changed parts only, 9 of 28 on device) so it can use the arm64 web helper on
|
||||
the GPU from the first frame (device: 144 fps menus).
|
||||
- **Web helper loop:** with main's in-process GPU arguments the x86 web helper's GPU crash took the
|
||||
whole helper down (102 restarts). The x86 helper now starts with stock arguments.
|
||||
- **Loading screen:** "Starting Steam x86-64", and the x86 setup and update steps reach it.
|
||||
- **Added games:** written first, then pointed at the x86 client's Proton (they were mapped to the
|
||||
arm64 client's tool name, which this client ignores). Windows components install into the x86
|
||||
prefix (DiRT Showdown and DiRT 3 needed OpenAL).
|
||||
- **Hidden game window, two causes.** The arm64 launcher loaded the saved arm session's `DISPLAY=:0`
|
||||
over the game display Steam chose; it now keeps Steam's. And the game window had no `STEAM_GAME`
|
||||
tag, so gamescope never listed it as focusable; the agent's focus watcher now tags untagged
|
||||
`steam_app_<id>` windows on the game display when the x86 client runs. Device: DiRT 3 (added game)
|
||||
at its title screen at 144 fps.
|
||||
- **Parity with the arm64 client:** Steam UI scale and the Steam import watcher run for the x86 client
|
||||
too (the import is untested). An SD-card library stays arm64-only: FEX cannot run Linux games from it.
|
||||
- **Later the same day:**
|
||||
- DiRT Showdown's ~7 s exit was the same `DISPLAY=:0` problem; it runs now.
|
||||
- **Tools adopted as arm64 Protons:** main's `droiddeck-steam-shortcuts` calls `adopt_extras` on the
|
||||
client's `compatibilitytools.d`; in the x86 client that rewrote DroidDeck direct as a Windows tool,
|
||||
so Steam installed Half-Life's Windows files. The x86 client no longer adopts its tools, and its
|
||||
preparation removes what an adoption left. The arm64 Proton is adopted on demand at an x86 launch.
|
||||
- **Scout runtime:** shipped as `steam-runtime.tar.xz`; `steam.sh` unpacks it, and the x86 client
|
||||
is started without it. Its preparation unpacks it now. Device: Half-Life's native Linux build at
|
||||
101 fps on the GPU under FEX (32-bit), and a VAC-secured multiplayer server.
|
||||
- **Install scripts:** the evaluator script is looked up in the x86 client's folder. A game's own
|
||||
installers without a `HasRunKey` (Borderlands 2: VC++ 2005/2008/2010, DirectX, .NET 4.0) are now
|
||||
marked under `Software\Valve\Steam\Apps\<appid>` (and the Wow6432Node view); Steam skips them.
|
||||
This applies to the arm64 client too. Device: Borderlands 2 started past its install script, and
|
||||
on the build that marks them itself it went straight into the game.
|
||||
- **Windows components** detection follows the chosen client (in-session install watch and
|
||||
`appinfo.vdf`).
|
||||
- **Next:** Counter-Strike: Source, the first game on the newer (sniper) runtime path, where DroidDeck
|
||||
direct runs it on the rootfs instead of the scout runtime.
|
||||
- **Open:** newer-runtime Linux games (sniper) untested; the arm64 client untested on this build; the
|
||||
Steam import watcher untested; an SD-card library for the x86 client deferred.
|
||||
|
||||
## 2026-10-10 - `feat/wincomp-auto-sources`: Windows components picked by the game's source
|
||||
|
||||
- **Per source** (`WinCompSources`): Steam - Steamworks Shared depots from the appmanifest or the
|
||||
@@ -87,6 +438,104 @@ the timeline, then lessons and backlog. Companion to the README (what the app *d
|
||||
idempotent, store and steamapps skipped, libraries empty, same-name folders), `GameStorageTest`
|
||||
updated. Not yet on a device.
|
||||
|
||||
## 2026-10-02 - `feat/x86-64-steam-client`: Valve's x86-64 Linux client under FEX (experiment)
|
||||
|
||||
Why: the arm64 Linux client never runs VAC. L4D2 (and TF2 in Bannerlator) are refused by VAC
|
||||
servers ("Connection Blocked ... VAC secured server") and no client log mentions VAC; VAC modules
|
||||
are x86 code. The question is whether Valve's x86-64 client under FEX gets a VAC module it can run.
|
||||
Branched from `feat/vac-diagnostics`. Opt-in: `bannerlator-steam-x64` (desktop menu "Steam x86-64
|
||||
(FEX, experimental)") runs the client in its own home (`~/.droiddeck-x64`) with its own rootfs,
|
||||
login and library. The arm64 client stays the default and is not touched.
|
||||
|
||||
Device-proven on the FIT (Genshin repack): **the x86-64 client runs, self-updates, renders its full
|
||||
UI (software GL), logs in, shows friends, and downloads games.** VAC itself is not tested yet.
|
||||
|
||||
What it took, in order (each step was the next failure on device):
|
||||
- **FEX for the guest** - `tools/fex/build-in-arch.sh` + `build-fex.yml` (FEX-2609, Arch Linux ARM
|
||||
container, no thunks), released as `fex-2609-r2` (not Latest). Rootfs: FEX's Ubuntu 24.04 image.
|
||||
- **openat2** - our proot answers it with ENOSYS; FEX opened every rootfs path with
|
||||
`openat2(RESOLVE_IN_ROOT)`, fell through to the arm64 guest on anything but EXDEV, and x86 bash
|
||||
could not find libtinfo. `tools/fex/patches/0001-openat2-enosys-fallback.patch`.
|
||||
- **FEXServer owns the rootfs** - a server started before `FEX_ROOTFS` (any earlier FEX run) answers
|
||||
every client with no rootfs. The launcher stops it first.
|
||||
- **32-bit x86 died with SIGILL** - libblfastpath (the proot fast path, preloaded into every guest
|
||||
process, FEX itself included) breaks FEX's 32-bit mode. `PROOT_FP_OFF=1` for FEX.
|
||||
- **"Steam now requires user namespaces"** - steam-runtime-check-requirements' bwrap probe; the
|
||||
arm64 client has no such probe. The SteamRT3 client (`steamrt64/steam`) is started directly, the
|
||||
way the session starts `steamrtarm64/steam`, with `STEAM_RUNTIME_STEAMRT` pointing at Valve's
|
||||
runtime linked file by file and only the probe replaced.
|
||||
- **tier0 assertion "Function not implemented"** - semget: Android has no System V IPC. Traced with
|
||||
a `-DDEBUG_STRACE` FEX (build-fex.yml `cxxflags` input, artifact only). The session shim
|
||||
(`preload/*.c`) is now also built natively as `libblsession-x86_64.so`, staged, and preloaded
|
||||
through the rootfs's own `/etc/ld.so.preload`.
|
||||
- **re-exec into pressure-vessel after self-update** - `STEAM_STEAMRT_RECURSION_GUARD=1`.
|
||||
- **libgtk-x11-2.0 missing / soname links / old glibc shadowing** - Valve's steamrt3c platform
|
||||
libraries linked into `~/.droiddeck-x64/steamrt-libs`, soname links from the rootfs `ldconfig`,
|
||||
and only libraries the rootfs lacks (LD_LIBRARY_PATH beats system folders whatever its order).
|
||||
- **GLX failed** - the desktop exports zink over the arm64 Turnip ICD; cleared, llvmpipe used.
|
||||
- **web helper died of SIGTRAP a second after start** - found with pid-tagged traces (FEX AppConfig
|
||||
`OutputLog=server` + a foreground FEXServer, `BL_FEXSERVER_LOG`; file logs overwrite each other):
|
||||
every web helper read `/proc/self/status` from libtier0_s.so, saw proot as a tracer, and tier0 broke
|
||||
into the "debugger" with int3. `preload/tracer.c` exempts the web helper on purpose (Chromium traps
|
||||
on a status fd that is not procfs); with `BL_WEBHELPER_HIDE_TRACER=1` only opens called from
|
||||
libtier0_s.so get the copy.
|
||||
- `SessionFiles` stages `bannerlator-steam-x64` and the x86-64 shim (they were missing from the list).
|
||||
|
||||
Dead ends worth remembering: headless VAC testing is impossible (current Steam starts the saved
|
||||
login from its UI); `-cef-*` switches are not forwarded to the x86 web helper; the KMS
|
||||
`CREATE_DUMB` errors are a red herring (40 fast ENOTTY per process); not an address-space problem
|
||||
(0 ENOMEM in 618k traced syscalls, although the device is 39-bit VA).
|
||||
|
||||
Later the same day (device-proven unless noted):
|
||||
- **Controllers**: libfakeinput is now also built for x86-64 and preloaded in the FEX rootfs; the
|
||||
client registers the pad, and **Big Picture works with it** (Xbox prompts, D-pad moves focus).
|
||||
- **Online**: the launcher starts the system bus and the NetworkManager stand-in like the session.
|
||||
- From an audit of `bannerlator-session`: stale lock cleanup, branch sync before every start, core
|
||||
re-pinning, path cache and `perms.c` for `steamrt64`, the app's FEX preset (PERFORMANCE_TSO).
|
||||
- **Play button (built, not yet device-tested)**: Steam settings > Client > "Steam client: ARM64 /
|
||||
x86-64 (FEX, experimental)" (`BL_STEAM_CLIENT_ARCH`). The session then starts the x86 client via
|
||||
`bannerlator-steam-x64 --prepare/--exec` and gives it everything it gives the arm64 one; steps
|
||||
only for the arm64 client (ARM64 Proton, mapping, seeding, Decky, steamclient.so probes) are
|
||||
skipped.
|
||||
- L4D2 hard-link reuse does not work: switching platform makes Steam redownload depot 551 anyway.
|
||||
|
||||
Evening (device-proven unless noted):
|
||||
- **Play button with the x86-64 client works** (Steam settings > Client > Steam client: x86-64).
|
||||
Deck mode is skipped for it: with -steamos3 its SteamOS audio manager aborts on the runtime's
|
||||
PulseAudio 13 (pa_operation_get_state assertion).
|
||||
- **Left 4 Dead 2 runs in the x86-64 client and joined a live online game** (Quick Match, Swamp
|
||||
Fever, other players, 138 ms) - the server type that refused the arm64 client with "Connection
|
||||
Blocked ... VAC secured server". Not yet hard proof of VAC (no client log line; -condebug added
|
||||
for the server's secure flag). Software rendering: ~11 fps.
|
||||
- What the launch needed: "DroidDeck direct", a compatibility tool running native Linux games with
|
||||
Valve's scout library runtime instead of the Steam Linux Runtime container; a stand-in
|
||||
steam-runtime-launch-client (the client, believing itself inside the SteamRT container, starts
|
||||
every game "alongside Steam" through it, and Valve's needs a launcher service on a session bus);
|
||||
i386 builds of the session shim (the i386 controller reader crashes the game and is left out for
|
||||
now). A steam:// URL written to the launch dir now reaches the running client.
|
||||
- **FEX with host thunks** (fex-2609-r3): FEX's Vulkan/GL/EGL/DRM/Wayland/ALSA thunks, 64- and
|
||||
32-bit guests, guest halves cross-compiled against an Arch x86_64/multilib sysroot. The x86 client
|
||||
uses them when installed (GPU mode, the arm64 client's ANGLE-on-Vulkan flags) - built, not yet
|
||||
device-tested.
|
||||
- The proot fast path now stands down in FEX only for 32-bit programs (it broke FEX's 32-bit mode).
|
||||
- **On the GPU, VAC proven (19:40):** games get FEX's host thunks and the session's Turnip/Zink
|
||||
settings (the client and its browser stay in software: with the thunks the browser's GPU process
|
||||
dies and the client never opens a window). Left 4 Dead 2 in The Passing on an online server:
|
||||
51 fps (was ~11 in software). The server answered the A2S info query as "Valve Left4Dead 2 US
|
||||
East Server", VAC=1 - an official VAC-secured server, which the arm64 client is refused.
|
||||
- The crash that held this up was not the GPU: the 32-bit controller reader was still preloaded
|
||||
(FEX falls through to the session's own i386 copy), and the game's SDL2 strdup()s a NULL udev
|
||||
attribute of its fake devices. Found with a SIGSEGV logger (tools/diag/segvlog.c).
|
||||
- The Linux FEX as a component file: /sdcard/Download/FEX-2609-r2-x86-64-guest.wcp (type
|
||||
FEX-Guest); the Components section does not know that type yet.
|
||||
|
||||
Open:
|
||||
- VAC test: L4D2's native Linux build in the x86 client's own library (hard-linked from the arm64
|
||||
copy, same filesystem, so only the Linux depots download). Native Linux games start inside the
|
||||
Steam Linux Runtime container (pressure-vessel) - expected to need the same treatment.
|
||||
- No x86 GPU driver: games render with llvmpipe. Playable needs FEX host thunks.
|
||||
- Windows games in the x86 client need x86 Proton = pressure-vessel.
|
||||
- One client per account: integration would be a per-title hand-off, not side by side.
|
||||
## 2026-10-09 - `feat/games-add-edit`: add games from the Games tab, edit them like Steam's Properties
|
||||
|
||||
- **Steam ⚙ settings:** the Games section is gone (it was a feature, not a setting). Games
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Experimental Quest 2/3/3S support
|
||||
|
||||
Quest 3/3S runs DroidDeck as a 2D panel. This does not add immersive VR or
|
||||
OpenXR support. Quest 2 has a separate experimental Adreno 6xx driver path.
|
||||
Quest runs DroidDeck as a 2D panel, with an experimental immersive SteamVR path
|
||||
in the standard app. Quest 2 has a separate experimental Adreno 6xx driver path.
|
||||
|
||||
Quest 2 selects `turnip26.3.0-quest2`, a pinned Banners-Turnip A6xx/A7xx build
|
||||
with the same GPUOBJ adapter. It does not select the Quest 3 A7xx binary.
|
||||
@@ -64,3 +64,147 @@ On a headset, check the Steam login panel, sign-in, game installation and
|
||||
then an owned game's rendering/input. Capture the device/OS, session logs,
|
||||
`files/quest-startup.txt` and any `files/quest-last-crash.txt` failure.
|
||||
Host allocation and preload-chain tests do not establish headset operation.
|
||||
|
||||
## Experimental immersive SteamVR
|
||||
|
||||
In a running Steam session, open the session menu's SteamVR tab, install SteamVR
|
||||
once, then choose Enter SteamVR. Cardboard has its own tab and can remain off. On supported Quest devices this opens the standard app's
|
||||
OpenXR activity. The guest driver receives headset poses, Touch controller input
|
||||
and display timing, and returns owned stereo images tagged with the latest eye poses.
|
||||
SteamVR haptics return through the tracking bridge. No root or separate XR APK
|
||||
is required. Other devices retain their existing SteamVR start behavior.
|
||||
|
||||
The left Touch menu button is SteamVR's system button and toggles its dashboard.
|
||||
The right Meta button remains reserved by the headset OS.
|
||||
|
||||
The driver supplies SteamVR's version-5 chaperone configuration through
|
||||
`Prop_DriverProvidedChaperoneJson_String`, following Valve's driver API. Standing
|
||||
tracking keeps the OpenXR stage coordinates; seated tracking starts at the head's
|
||||
position and yaw. No synthetic play area or collision walls are advertised.
|
||||
Without a seated transform, SteamVR reported valid raw and standing poses but
|
||||
invalid seated poses, causing Thumper to remain behind its tracking-loss fade.
|
||||
On Quest, supplying both transforms made all three origins valid; seated height
|
||||
started within a millimetre of zero while standing and raw height stayed unchanged.
|
||||
|
||||
Quest uses SteamVR's normal Linux compositor through its display component.
|
||||
The Vulkan layer captures only the HMD display swapchain, retains its present
|
||||
waits, and chains a GPU copy before presentation. Three bounded capture buffers
|
||||
allow rendering and transfer to overlap. Where native GLES supports
|
||||
`GL_EXT_external_buffer` and texture views, Android allocates three
|
||||
`AHardwareBuffer` BLOBs and passes their DMA-BUF allocations to Linux through
|
||||
`SCM_RIGHTS`. Vulkan imports them as buffers and transfers foreign queue-family
|
||||
ownership before and after copying the display image. A native compute shader
|
||||
reads each shared buffer into a private RGBA8 texture; an sRGB texture view
|
||||
preserves hardware transfer decoding. Native GPU completion precedes the
|
||||
acknowledgement, so Linux cannot overwrite an active reader. This removes the CPU
|
||||
pixel copy and upload, but still includes GPU copies and explicit completion waits.
|
||||
|
||||
If capability negotiation or buffer import fails before frames are submitted,
|
||||
the existing CPU transport remains available. It brackets the DMA-BUF read with
|
||||
`DMA_BUF_IOCTL_SYNC`, copies to owned pixels, then acknowledges and uploads them.
|
||||
Both channels authenticate the peer UID and bound buffer sizes. Missing
|
||||
acknowledgements after submission quarantine the transfer and fail presentation
|
||||
instead of falling back while a reader might still own a buffer.
|
||||
|
||||
The helper temporarily enables SteamVR's Linux Vulkan asynchronous queue and disables motion
|
||||
smoothing, backing up those preferences and restoring them on stop or the next start
|
||||
after an interrupted session. It excludes the pinned Steam Frame package's optional CVP backend,
|
||||
whose library constructor exits on Quest's unsupported SoC. The original library
|
||||
is retained with a `.droiddeck-disabled` suffix. SteamVR binaries are not patched.
|
||||
Cardboard retains its existing direct-mode path; `questframes` remains an explicit
|
||||
diagnostic override for the earlier driver presenter.
|
||||
|
||||
The panel compositor detaches before OpenXR starts displaying guest frames.
|
||||
Session suspension accounts for the immersive activity while the panel is hidden.
|
||||
Closing the immersive activity stops SteamVR. The panel remains available for
|
||||
returning to the normal Steam session.
|
||||
The immersive activity belongs to the Steam session that opened it. Ending that
|
||||
session closes XR; a failed startup or stopped SteamVR also returns to the panel
|
||||
with an error instead of leaving a black scene. Enter SteamVR can then start a
|
||||
fresh connection.
|
||||
|
||||
SteamVR scene-application and dashboard events also guide Gamescope's game focus.
|
||||
While the dashboard is hidden, the guest agent resolves the live scene process's
|
||||
Steam app ID and promotes its focusable window. Opening the dashboard or ending
|
||||
the scene restores the previous order if no other client has replaced it. This
|
||||
keeps keyboard/gamepad input with the VR scene without game-specific IDs.
|
||||
|
||||
With Steam already running, `python3 tools/test_quest_xr.py --serial SERIAL`
|
||||
launches the immersive activity and verifies advancing owned images, tracked
|
||||
6DoF, focus and SteamVR status. Add `--require-controllers` while holding both
|
||||
controllers. Results include telemetry and logcat in `build/questxr-validation`.
|
||||
Host tests and APK installation alone do not establish headset operation.
|
||||
|
||||
On Quest 3, a 45-second hardware run maintained focused tracked 6DoF, both
|
||||
controllers active, and advancing stereo images. A headset tester confirmed
|
||||
controller models and trigger animation. That run delivered about 8.5 new stereo
|
||||
images per second despite a 90 Hz XR session.
|
||||
That earlier measurement used the driver-direct presenter. With SteamVR's normal
|
||||
compositor, a native Vulkan scene passed its first submission and continued
|
||||
submitting stereo frames. Thumper and The Thrill of the Fight produced clean
|
||||
immersive scenes. SUPERHOT's Windows OpenXR build initially showed shared-image
|
||||
corruption: DXVK supplied a restricted mutable-image format list while SteamVR
|
||||
created its corresponding opaque shared images without one. Turnip can select
|
||||
different memory layouts for those descriptions. On the Quest bridge, the layer
|
||||
normalizes the known opaque-FD stereo-array creation chains to SteamVR's unrestricted list,
|
||||
without changing application-owned single-layer OpenVR images, DMA-BUF/modifier
|
||||
images or other targets. A headset capture then
|
||||
showed clean SUPERHOT scene geometry; interactive gameplay remains unverified.
|
||||
A subsequent 60-second shared-buffer dashboard run delivered about
|
||||
43 new stereo images per second. Reusing Android pixel buffers raised the next
|
||||
60-second run to about 68 received and 65 uploaded images per second, with valid
|
||||
focused tracking and no rejected frames. These dashboard results do not establish refresh-rate gameplay: game input,
|
||||
sustained game performance, and exact association with SteamVR's final compositor
|
||||
pose still require validation.
|
||||
|
||||
The native GPU-buffer channel passed 256 full-size buffer reuse cycles, including
|
||||
head/tail pixel checks and sRGB texture-view decoding. A subsequent SUPERHOT
|
||||
scene run delivered about 43 new stereo images/s and 58 XR draws/s over one
|
||||
minute, with focused tracking and no rejected frames. The earlier CPU-path
|
||||
SUPERHOT run uploaded about 28 images/s; these are separate runs, not a controlled
|
||||
benchmark. Capturing the source and eye images can delay frame consumption;
|
||||
the GPU receiver acknowledges stale frames without reading them and keeps the
|
||||
last private image until a fresh frame arrives. Missing completion still fails
|
||||
closed. `frame_verify` explicitly selects the CPU path for byte comparison.
|
||||
|
||||
The frame channel supports linear and sRGB RGBA8. SteamVR's final display image
|
||||
uses sRGB display values; Android decodes those during texture sampling and writes
|
||||
to an sRGB eye swapchain, encoding the display values once. A Quest color-chart
|
||||
capture verified that 64, 128, 192 and 224 grey patches retain those values;
|
||||
the previous sRGB source upload darkened them to approximately 13, 55, 134 and
|
||||
190. The transport remains 8-bit SDR; this correction does not add HDR or higher
|
||||
precision gradients.
|
||||
|
||||
When available, `XR_EXT_performance_settings` requests sustained-high CPU and GPU
|
||||
performance for the running OpenXR session. These are runtime-managed hints;
|
||||
device power and thermal policies remain authoritative. Unsupported or rejected
|
||||
requests do not prevent startup. This does not change render resolution.
|
||||
|
||||
The manifest declares the supported Quest models using Meta's standard
|
||||
`com.oculus.supportedDevices` metadata. Without it, Quest 3 used compatibility
|
||||
mode and its refresh-rate API returned an empty capability list. With the
|
||||
declaration, the test runtime reported 72/80/90/120 Hz; a fresh normal launch
|
||||
used 72 Hz in both OpenXR telemetry and SteamVR's HMD property. DroidDeck leaves
|
||||
the refresh choice to the runtime. No forced refresh override is included.
|
||||
|
||||
The OpenVR driver maps OpenXR's asymmetric eye FOV to OpenVR's lower/upper Y
|
||||
projection bounds. A live Quest 3 check confirmed that SteamVR and native
|
||||
OpenXR now report the same vertical projection center; the previous mapping
|
||||
mirrored that offset. This applies to every VR scene, without game-specific
|
||||
rendering changes.
|
||||
|
||||
### Experimental flat games in SteamVR
|
||||
|
||||
Start SteamVR first, then add `droiddeck-steamvr-game %command%` to a flat
|
||||
game's Steam launch options. Remove that prefix to restore normal panel launch.
|
||||
This runs a nested, unmodified Gamescope OpenVR backend for that game. Gamescope
|
||||
owns the GPU overlay, placement and input routing; no CPU screen capture or
|
||||
PipeWire service is used. The existing DroidDeck Vulkan synchronization layer
|
||||
remains enabled. On Quest, a connected XR receiver selects Gamescope's Vulkan
|
||||
texture sharing instead of DMA-BUF modifiers. Explicit environment overrides
|
||||
are preserved; other devices retain the upstream modifier default.
|
||||
|
||||
An upstream GPU cube overlay has been observed on Quest 3. Flat-game stability,
|
||||
controller interaction and performance still require hardware validation. This
|
||||
uses Valve's public Gamescope backend; the exact Steam Frame OS launch setup
|
||||
has not been verified.
|
||||
@@ -0,0 +1,162 @@
|
||||
# DroidDeck 0.5.0
|
||||
|
||||
<p align="center"><a href="https://discord.gg/JRGAvawjsm"><img src="https://img.shields.io/badge/Discord-Join%20the%20community-5865F2?logo=discord&logoColor=white" alt="Join the DroidDeck Discord"></a></p>
|
||||
|
||||
### Features
|
||||
|
||||
- Added **SteamVR on Meta Quest 3 and 3S** from the standard APK. Quest headsets are detected automatically and get their own driver defaults. Immersive SteamVR runs through a native OpenXR activity with headset tracking, Touch controllers and haptics, and SteamVR's dashboard and Steam library work in VR. (@xXJSONDeruloXx, @Jagger15, [#642](https://github.com/Droid-Deck/DroidDeck/pull/642), [#910](https://github.com/Droid-Deck/DroidDeck/pull/910), [#733](https://github.com/Droid-Deck/DroidDeck/pull/733), [#853](https://github.com/Droid-Deck/DroidDeck/pull/853), [#905](https://github.com/Droid-Deck/DroidDeck/pull/905))
|
||||
|
||||
<img src="media/0.5.0/quest-steamvr.gif" width="50%" alt="On a Quest 3: DroidDeck in passthrough, Steam Big Picture in the headset, the SteamVR dashboard with a Touch controller, then playing Thumper in VR">
|
||||
|
||||
<details>
|
||||
<summary>Flat games on Quest with hand tracking</summary>
|
||||
|
||||
<img src="media/0.5.0/quest-flat-hands.gif" width="50%" alt="Plants vs. Zombies on a DroidDeck panel in Quest passthrough, played with hand tracking">
|
||||
|
||||
</details>
|
||||
- Added experimental **Meta Quest 2** support. (@Jagger15, [#676](https://github.com/Droid-Deck/DroidDeck/pull/676))
|
||||
- Added **XR glasses support for VITURE, RayNeo and XREAL**. Games play straight on the glasses at their own refresh rate while the phone becomes the controller, and the glasses' refresh rate, 3D mode, brightness, volume and sunblock can be changed from the app or the session menu. **Play SteamVR** appears beside Play Steam when supported glasses are connected, with head tracking from the glasses. Joy-Cons work as SteamVR hands, and the phone's camera can track bare hands with Quest-style pinch gestures. Supports VITURE Pro XR, RayNeo Air and GT glasses, and XREAL 1S (experimental). Glasses with head tracking can also show a **head-tracked virtual screen** everywhere in DroidDeck, in its menus, desktop sessions and games, without starting SteamVR, set up from the new **Setup → Smart glasses** page. (@maxjivi05, @informalTechCode, [#732](https://github.com/Droid-Deck/DroidDeck/pull/732), [#907](https://github.com/Droid-Deck/DroidDeck/pull/907), [#1035](https://github.com/Droid-Deck/DroidDeck/pull/1035))
|
||||
|
||||
<img src="media/0.5.0/play-steamvr-glasses.jpg" width="50%" alt="Play SteamVR beside Play Steam with VITURE Pro XR glasses connected">
|
||||
<img src="media/0.5.0/glasses-steamvr.gif" width="50%" alt="Angry Birds VR from Play SteamVR, rendered in stereo with head pose from the glasses and the slingshot pulled with the gamepad">
|
||||
|
||||
<details>
|
||||
<summary>On the glasses, and gamepad hands in SteamVR</summary>
|
||||
|
||||
<img src="media/0.5.0/glasses-2d.gif" width="50%" alt="Recorded from the glasses' own 1920×1080 display: Angry Birds VR with one eye's view, slingshot pulled with the gamepad">
|
||||
<img src="media/0.5.0/glasses-gamepad-hands.gif" width="50%" alt="Gamepad hands pointing at Angry Birds VR's level select">
|
||||
|
||||
</details>
|
||||
- Added **Cardboard VR mode** with Valve's ARM64 SteamVR, splitting the screen into two lens-corrected eyes with head tracking. Windows VR games such as Thumper and The Thrill of the Fight run through Proton. SteamVR and Cardboard share one **VR** tab in the session menu. (@xXJSONDeruloXx, [#704](https://github.com/Droid-Deck/DroidDeck/pull/704), [#959](https://github.com/Droid-Deck/DroidDeck/pull/959))
|
||||
|
||||
<img src="media/0.5.0/cardboard-thumper.jpg" width="50%" alt="Thumper in Cardboard VR mode, split into two lens-corrected eyes">
|
||||
<img src="media/0.5.0/cardboard-thrill-of-the-fight.jpg" width="50%" alt="The Thrill of the Fight's weigh-in scene in Cardboard VR mode">
|
||||
<img src="media/0.5.0/vr-drawer-tab.jpg" width="50%" alt="The session menu's VR tab with SteamVR and Cardboard sections">
|
||||
- Added **Epic Games Launcher for Linux**, Epic's own client, running under FEX and signed in automatically with your linked Epic account. Every Epic game now starts through Steam and the launcher, so games get the same sign-in and online services as on a PC, along with Steam Input and the performance overlay. The launcher can be driven with a gamepad or touch. Games installed from inside the launcher appear in the Games tab automatically, and your Steam games can be started from the launcher's library. (@maxjivi05, [#928](https://github.com/Droid-Deck/DroidDeck/pull/928))
|
||||
|
||||
<img src="media/0.5.0/epic-launcher.gif" width="50%" alt="Opening the Epic Games Launcher from the Epic tab, already signed in, and browsing its library">
|
||||
<img src="media/0.5.0/epic-launcher-games-tab.jpg" width="50%" alt="The Epic Games Launcher in the Games tab next to Epic games, with Launch and Uninstall">
|
||||
|
||||
<details>
|
||||
<summary>Gamepad navigation and launching through Steam</summary>
|
||||
|
||||
<img src="media/0.5.0/epic-launcher-gamepad.gif" width="50%" alt="Moving the selection ring across the launcher's library with the D-pad">
|
||||
<img src="media/0.5.0/epic-launch-through-steam.gif" width="50%" alt="Launching Alone With You from the Games tab: Steam starts the launcher in the background, which hands the game back to Steam">
|
||||
<img src="media/0.5.0/epic-launcher-library.jpg" width="50%" alt="The Epic Games Launcher's library running on DroidDeck">
|
||||
|
||||
</details>
|
||||
- Added an optional **x86-64 Steam client**: Valve's own Linux client, running under FEX beside the default ARM64 one. Choose it under **Steam client** in Steam settings; it applies at the next session. It has its own game library and shares your sign-in. Games with a Linux version, such as Team Fortress 2 and Half-Life, run their Linux builds on it, and Windows games use DroidDeck-Proton as before. New FEX builds reach it without an app update. (@The412Banner, [#152](https://github.com/Droid-Deck/DroidDeck/pull/152), [#1050](https://github.com/Droid-Deck/DroidDeck/pull/1050), [#1053](https://github.com/Droid-Deck/DroidDeck/pull/1053))
|
||||
- **Linux games run under FEX out of the box.** Proton Experimental ARM64, FEX and the Steam Linux Runtimes now download on the first Steam start instead of waiting as pending downloads. x86 Linux games start, keep running and take the controller under FEX, including Unity games, which used to crash or ignore the gamepad. Games that also have a Windows version stay on Proton by default, because the Windows build often runs better. To run a game's Linux build instead, choose **Linux x86 (FEX)** in its compatibility settings. (@maxjivi05, [#1045](https://github.com/Droid-Deck/DroidDeck/pull/1045))
|
||||
- Store game pages now show **HowLongToBeat times, Metacritic scores**, developer, publisher and release date for GOG and Epic games, along with an **achievements** card with filters for unlocked, locked and hidden achievements. (@phobos665, @xXJSONDeruloXx, [#641](https://github.com/Droid-Deck/DroidDeck/pull/641), [#712](https://github.com/Droid-Deck/DroidDeck/pull/712), [#775](https://github.com/Droid-Deck/DroidDeck/pull/775))
|
||||
|
||||
<img src="media/0.5.0/game-page-hltb-metacritic.jpg" width="50%" alt="Backpack Hero's Epic page with its Metacritic score of 76, HowLongToBeat times and achievements card">
|
||||
- Added opt-in **GOG achievements** syncing through Comet, with unlock notifications while you play. (@xXJSONDeruloXx, [#651](https://github.com/Droid-Deck/DroidDeck/pull/651))
|
||||
- Added **sorting and a list view** for the store Library and Installed tabs: sort by recently played, recently installed, title, size or release date, and switch between cards and a compact list. (@kingokksa, [#798](https://github.com/Droid-Deck/DroidDeck/pull/798))
|
||||
|
||||
<img src="media/0.5.0/sort-list-view.gif" width="50%" alt="Sorting the Epic library by size and by title, then switching from the card grid to the list view">
|
||||
- Added a **Storage** page to Setup that shows where DroidDeck's storage goes and frees what is safe to remove: unused Steam runtimes, leftovers from uninstalled games, old logs, shader caches, unused components and Flatpak runtimes. Starting Steam with less than 10 GB free offers to free up space. (@xXJSONDeruloXx, [#913](https://github.com/Droid-Deck/DroidDeck/pull/913))
|
||||
|
||||
<img src="media/0.5.0/storage-overview.jpg" width="50%" alt="Storage page overview with the internal storage bar and its categories">
|
||||
<img src="media/0.5.0/storage-cleanup.jpg" width="50%" alt="Storage Clean up checklist with sizes and the safe items selected">
|
||||
- Added **artwork changing for Steam games and GOG, Epic and Amazon games**, matching added games, with a SteamGridDB search in the art chooser. **Adjust logo** sets where a game's logo sits on its background and how big it is. Chosen art and logo positions are applied in Steam too. (@FX192, [#876](https://github.com/Droid-Deck/DroidDeck/pull/876), [#939](https://github.com/Droid-Deck/DroidDeck/pull/939), [#948](https://github.com/Droid-Deck/DroidDeck/pull/948), [#976](https://github.com/Droid-Deck/DroidDeck/pull/976), [#1004](https://github.com/Droid-Deck/DroidDeck/pull/1004))
|
||||
|
||||
<img src="media/0.5.0/steam-artwork.jpg" width="50%" alt="The Artwork editor for Moonlighter, a Steam game, with its cover, background, logo and icon">
|
||||
- Redesigned the **GPU drivers** page with three modes: **Auto**, **Matched pair**, or **Mix and match** display and runtime drivers. It also shows which drivers are in use now. Auto checks for new drivers when the app opens, and if a new pair never starts a session, it rolls back to the last pair that worked. (@xXJSONDeruloXx, [#858](https://github.com/Droid-Deck/DroidDeck/pull/858), [#972](https://github.com/Droid-Deck/DroidDeck/pull/972))
|
||||
|
||||
<img src="media/0.5.0/gpu-drivers.jpg" width="50%" alt="GPU drivers page with Auto, Matched pair and Mix and match, and the drivers in use">
|
||||
- Added **Boot movie** settings: the DroidDeck movie at 30 or 60 fps, with or without sound, or import your own video, which is converted on the device. (@xXJSONDeruloXx, [#969](https://github.com/Droid-Deck/DroidDeck/pull/969))
|
||||
|
||||
<img src="media/0.5.0/boot-movie.jpg" width="50%" alt="Boot movie settings in the Steam settings sheet: movie, sound and Import">
|
||||
- Added a per-game **Start without Steam** switch for games that close right away because of Proton's Steam setup. (@The412Banner, [#659](https://github.com/Droid-Deck/DroidDeck/pull/659))
|
||||
- Added **Low-latency controller** and **Low-latency touch** toggles to the session menu, which apply immediately while Steam is running. (@xXJSONDeruloXx, [#808](https://github.com/Droid-Deck/DroidDeck/pull/808))
|
||||
|
||||
<img src="media/0.5.0/low-latency.jpg" width="50%" alt="The session menu's Latency group with Low-latency controller and Low-latency touch">
|
||||
- A **three-finger tap** opens and closes Steam's on-screen keyboard in game. (@xXJSONDeruloXx, [#782](https://github.com/Droid-Deck/DroidDeck/pull/782))
|
||||
- Added **Picture position** (Top, Bottom or Center) for Steam sessions, so fixed-aspect games can sit away from on-screen controls on tall screens. (@asdf17128, [#871](https://github.com/Droid-Deck/DroidDeck/pull/871))
|
||||
|
||||
<img src="media/0.5.0/picture-position.jpg" width="50%" alt="Picture position in the Steam settings Display tab">
|
||||
- FEX, DXVK and VKD3D-Proton versions can be **downloaded from the session menu**. (@xXJSONDeruloXx, [#679](https://github.com/Droid-Deck/DroidDeck/pull/679))
|
||||
- Added opt-in **zRAM compression** for the Steam client while a game runs, freeing about 0.5–0.8 GB of memory in testing. (@maxjivi05, [#442](https://github.com/Droid-Deck/DroidDeck/pull/442))
|
||||
- Added **23 more languages**, for 32 in total besides English, selectable in the app or in Android's per-app language settings. Steam follows the chosen language. (@maxjivi05, [#669](https://github.com/Droid-Deck/DroidDeck/pull/669))
|
||||
- Downloads now **resume where they stopped** and fall back to mirrors, with a download source setting that picks the fastest source automatically. This helps especially in mainland China. (@kingokksa, [#623](https://github.com/Droid-Deck/DroidDeck/pull/623))
|
||||
- Added **Reset all settings** in Setup > Diagnostics. It keeps your games, sign-ins and downloads. (@0x1Nawaf, [#696](https://github.com/Droid-Deck/DroidDeck/pull/696))
|
||||
- Store installs can go to **internal storage, any SD card or your second library**. (@maxjivi05, [#928](https://github.com/Droid-Deck/DroidDeck/pull/928))
|
||||
- Controllers with two back paddles (such as the 8BitDo Mobile Controller) map them to the Deck's **L4 and R4**. (@Vynex, [#797](https://github.com/Droid-Deck/DroidDeck/pull/797))
|
||||
- The second screen's Deck controls now include the **Gamepad** tab. On foldables, the unfolded controls hide while a controller is connected. (@ovurrsl, @Gustash, [#895](https://github.com/Droid-Deck/DroidDeck/pull/895), [#829](https://github.com/Droid-Deck/DroidDeck/pull/829))
|
||||
- Added experimental **Android Auto** support. DroidDeck is only available while the car is parked, and needs Android Auto's developer mode with unknown sources enabled. (@zekena2, [#952](https://github.com/Droid-Deck/DroidDeck/pull/952))
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Rewrote the Epic and Amazon store modules**, with byte-exact download verification and proper controller focus on store game pages. Existing sign-ins and installs carry over. (@The412Banner, [#802](https://github.com/Droid-Deck/DroidDeck/pull/802), [#804](https://github.com/Droid-Deck/DroidDeck/pull/804))
|
||||
- **Epic installs need far less free space.** Files are assembled as they download instead of caching the whole game first, so a 71 GB game no longer needs about 142 GB free. Store installs also check for space first and recover better when they run short. (@jmarti326, [#710](https://github.com/Droid-Deck/DroidDeck/pull/710))
|
||||
- **Fewer background processes.** An idle Steam session went from about 41 processes to about 30, keeping it further under Android's child-process limit, and Steam's web helper uses about 250 MB less memory. (@xXJSONDeruloXx, [#566](https://github.com/Droid-Deck/DroidDeck/pull/566), [#575](https://github.com/Droid-Deck/DroidDeck/pull/575))
|
||||
- **Smoother frame pacing.** The session respects Android's peak refresh rate setting, frames are timed to the display's vsync, Steam's frame limiter paces each game by time, and frame generation follows the game's real frame rate. On a 60 Hz Thor, distinct frames on screen rose from about 49.5 to 59.4 a second. (@xXJSONDeruloXx, [#715](https://github.com/Droid-Deck/DroidDeck/pull/715), [#717](https://github.com/Droid-Deck/DroidDeck/pull/717), [#731](https://github.com/Droid-Deck/DroidDeck/pull/731), [#753](https://github.com/Droid-Deck/DroidDeck/pull/753), [#927](https://github.com/Droid-Deck/DroidDeck/pull/927))
|
||||
- The game's busiest threads are boosted onto the **prime cores on the Odin 3**, and Snapdragon 8 Gen 2 devices get default core assignments for games and Steam. (@kmnkv1, @xXJSONDeruloXx, [#682](https://github.com/Droid-Deck/DroidDeck/pull/682), [#654](https://github.com/Droid-Deck/DroidDeck/pull/654))
|
||||
- Games on **SD-card libraries launch faster**, about 46% faster in testing. (@xXJSONDeruloXx, [#757](https://github.com/Droid-Deck/DroidDeck/pull/757))
|
||||
- The Steam home wall now fills from **your own Steam library**, and the store sign-in screens show each store's featured art. (@xXJSONDeruloXx, [#835](https://github.com/Droid-Deck/DroidDeck/pull/835), [#878](https://github.com/Droid-Deck/DroidDeck/pull/878))
|
||||
- **Older and less common devices:** sessions start on kernels that drop fork events (such as the Xiaomi Pad 5), Flatpak installs work on older kernels, the Linux runtime installs on 16 KB page-size devices, and USB drives that only appear as raw storage can be granted All files access. (@xXPOLYGONXx, @xXJSONDeruloXx, @ovurrsl, @Imgau, [#457](https://github.com/Droid-Deck/DroidDeck/pull/457), [#752](https://github.com/Droid-Deck/DroidDeck/pull/752), [#900](https://github.com/Droid-Deck/DroidDeck/pull/900), [#760](https://github.com/Droid-Deck/DroidDeck/pull/760))
|
||||
- Controller-driven cursors now show in nested Wayland sessions. (@xXJSONDeruloXx, [#759](https://github.com/Droid-Deck/DroidDeck/pull/759))
|
||||
- An interrupted Linux runtime download resumes instead of starting again. (@ovurrsl, [#847](https://github.com/Droid-Deck/DroidDeck/pull/847))
|
||||
- Native Linux and FEX compatibility choices are kept instead of being reset to Proton. (@xXJSONDeruloXx, [#749](https://github.com/Droid-Deck/DroidDeck/pull/749))
|
||||
- DroidDeck-Proton releases now come from the [Droid-Deck/DroidDeck-Proton](https://github.com/Droid-Deck/DroidDeck-Proton) repo. (@The412Banner, [#875](https://github.com/Droid-Deck/DroidDeck/pull/875))
|
||||
- Folding or unfolding no longer restarts the front end, file manager or store sign-in. (@ovurrsl, [#832](https://github.com/Droid-Deck/DroidDeck/pull/832), [#886](https://github.com/Droid-Deck/DroidDeck/pull/886))
|
||||
- Session reports include the memory page size, why the app's last processes ended, the panel's refresh modes and the device's thermal state. Logs are synced to storage every 2 seconds, so a hard reset keeps them, and only the 3 newest session log folders are kept. (@ovurrsl, @kmnkv1, @xXJSONDeruloXx, [#897](https://github.com/Droid-Deck/DroidDeck/pull/897), [#513](https://github.com/Droid-Deck/DroidDeck/pull/513), [#852](https://github.com/Droid-Deck/DroidDeck/pull/852), [#990](https://github.com/Droid-Deck/DroidDeck/pull/990))
|
||||
- The store sign-in page can no longer read local files. (@ovurrsl, [#890](https://github.com/Droid-Deck/DroidDeck/pull/890))
|
||||
- Expanded and refreshed translations, including Chinese, Japanese, Turkish, German, French, Spanish, Italian, Portuguese, Russian, Korean and Hungarian. Every language now has every string, including the new Storage, GPU drivers and artwork pages, and Latin American Spanish is complete. (@Dantalian-dev, @ovurrsl, @kingokksa, [#539](https://github.com/Droid-Deck/DroidDeck/pull/539), [#826](https://github.com/Droid-Deck/DroidDeck/pull/826), [#849](https://github.com/Droid-Deck/DroidDeck/pull/849), [#850](https://github.com/Droid-Deck/DroidDeck/pull/850), [#884](https://github.com/Droid-Deck/DroidDeck/pull/884), [#1013](https://github.com/Droid-Deck/DroidDeck/pull/1013))
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed **Android UI flickering** during games on Adreno 740 devices such as the AYN Thor and Odin 2 Portal. (@xXJSONDeruloXx, [#963](https://github.com/Droid-Deck/DroidDeck/pull/963))
|
||||
- Fixed **game rumble stopping after one second** under Proton. (@TideGear, [#938](https://github.com/Droid-Deck/DroidDeck/pull/938))
|
||||
- Fixed **Shut Down and Restart** in Steam's power menu. Shut Down now ends the session, and Restart restarts the client. (@muazkadan, [#629](https://github.com/Droid-Deck/DroidDeck/pull/629))
|
||||
- Fixed the **Quick Access button** doing nothing when the pad is set to Xbox 360, DualShock 4 or DualSense. (@muazkadan, [#631](https://github.com/Droid-Deck/DroidDeck/pull/631))
|
||||
- Fixed **store games becoming Steam installs** when you also own them on Steam. (@xXJSONDeruloXx, [#816](https://github.com/Droid-Deck/DroidDeck/pull/816))
|
||||
- Fixed **added games sometimes never starting** on a cold launch with large Steam libraries. (@xXJSONDeruloXx, @ovurrsl, [#758](https://github.com/Droid-Deck/DroidDeck/pull/758), [#914](https://github.com/Droid-Deck/DroidDeck/pull/914), [#973](https://github.com/Droid-Deck/DroidDeck/pull/973))
|
||||
- Fixed games that never started after a long pause, and "Steam is paused" staying up after the first Resume tap. (@maxjivi05, [#928](https://github.com/Droid-Deck/DroidDeck/pull/928))
|
||||
- Fixed the session ending the third time Steam restarted, for example after changing language. (@KirkPatrickJunsay, [#754](https://github.com/Droid-Deck/DroidDeck/pull/754))
|
||||
- Fixed Steam touch controls ignoring the layout chosen in Steam, including community layouts. (@xXJSONDeruloXx, [#840](https://github.com/Droid-Deck/DroidDeck/pull/840))
|
||||
- Fixed **Batman: Arkham City** crashing at startup and getting stuck on "Running install script". (@TjebbeKerstens, [#792](https://github.com/Droid-Deck/DroidDeck/pull/792), [#793](https://github.com/Droid-Deck/DroidDeck/pull/793))
|
||||
- Fixed **.NET Desktop and .NET Core runtimes** failing to install from Windows components, which blocked MelonLoader mods and Baldur's Gate 3. (@The412Banner, [#740](https://github.com/Droid-Deck/DroidDeck/pull/740))
|
||||
- Fixed the microphone in DirectAudio games. (@xXJSONDeruloXx, [#747](https://github.com/Droid-Deck/DroidDeck/pull/747))
|
||||
- Fixed FEX games failing with "Couldn't connect to FEXServer socket". (@angelrenefm, [#841](https://github.com/Droid-Deck/DroidDeck/pull/841))
|
||||
- Fixed wired **PowerA and PDP** Xbox controllers' right stick and triggers. (@Raidennrl, [#692](https://github.com/Droid-Deck/DroidDeck/pull/692))
|
||||
- Fixed touchpad taps not registering in some games, and the touchpad not moving a game's own cursor when it locks the pointer. (@KirkPatrickJunsay, @Gustash, [#644](https://github.com/Droid-Deck/DroidDeck/pull/644), [#772](https://github.com/Droid-Deck/DroidDeck/pull/772))
|
||||
- Fixed native touch taps snapping the cursor to the top-left corner in some games. (@asdf17128, [#803](https://github.com/Droid-Deck/DroidDeck/pull/803))
|
||||
- Fixed the Android keyboard needing two taps to open after it was dismissed. (@KirkPatrickJunsay, [#665](https://github.com/Droid-Deck/DroidDeck/pull/665))
|
||||
- Fixed Shift staying held after typing capitals on a physical keyboard in the desktop. (@The412Banner, [#766](https://github.com/Droid-Deck/DroidDeck/pull/766))
|
||||
- Fixed an unfolded Follow screen session no longer turning to portrait. (@Gustash, [#626](https://github.com/Droid-Deck/DroidDeck/pull/626))
|
||||
- Fixed battery estimates showing thousands of hours on phones that report current in mA. (@muazkadan, [#820](https://github.com/Droid-Deck/DroidDeck/pull/820))
|
||||
- Fixed blurry home-screen shortcuts for Steam games. (@KirkPatrickJunsay, [#825](https://github.com/Droid-Deck/DroidDeck/pull/825))
|
||||
- Fixed SteamGridDB art staying "no match" after a failed lookup. (@FX192, [#876](https://github.com/Droid-Deck/DroidDeck/pull/876))
|
||||
- Fixed Steam crashes returning to the launcher silently. They now show DroidDeck's error screen, and Steam no longer crashes on Quest when a SteamVR runtime is installed. (@xXJSONDeruloXx, [#905](https://github.com/Droid-Deck/DroidDeck/pull/905))
|
||||
- Fixed the in-session achievements page staying on "Reading…". (@xXJSONDeruloXx, [#961](https://github.com/Droid-Deck/DroidDeck/pull/961))
|
||||
- Fixed crashes from failed store background jobs and cloud save recovery, and a memory error in the directory cache. (@KirkPatrickJunsay, @ovurrsl, @wallentx, [#846](https://github.com/Droid-Deck/DroidDeck/pull/846), [#898](https://github.com/Droid-Deck/DroidDeck/pull/898), [#685](https://github.com/Droid-Deck/DroidDeck/pull/685))
|
||||
- Fixed expired Epic and Amazon sign-ins failing with a generic error instead of asking you to sign in again. (@The412Banner, [#804](https://github.com/Droid-Deck/DroidDeck/pull/804))
|
||||
- Fixed the child-process limit check on Android 12, and its switch name now matches Android's wording. (@ovurrsl, [#955](https://github.com/Droid-Deck/DroidDeck/pull/955), [#848](https://github.com/Droid-Deck/DroidDeck/pull/848))
|
||||
- Fixed the launcher offering its own display as a second screen. (@ovurrsl, [#889](https://github.com/Droid-Deck/DroidDeck/pull/889))
|
||||
- Fixed Windows component detection being confused by folder names above the game folder. (@xXJSONDeruloXx, [#859](https://github.com/Droid-Deck/DroidDeck/pull/859))
|
||||
- Fixed FEX dropdown scaling and Flathub search focus. (@xXJSONDeruloXx, [#681](https://github.com/Droid-Deck/DroidDeck/pull/681))
|
||||
|
||||
### New contributors
|
||||
|
||||
- @xXPOLYGONXx made their first contribution in [#457](https://github.com/Droid-Deck/DroidDeck/pull/457).
|
||||
- @phobos665 made their first contribution in [#637](https://github.com/Droid-Deck/DroidDeck/pull/637).
|
||||
- @Jagger15 made their first contribution in [#642](https://github.com/Droid-Deck/DroidDeck/pull/642).
|
||||
- @KirkPatrickJunsay made their first contribution in [#644](https://github.com/Droid-Deck/DroidDeck/pull/644).
|
||||
- @wallentx made their first contribution in [#685](https://github.com/Droid-Deck/DroidDeck/pull/685).
|
||||
- @Raidennrl made their first contribution in [#692](https://github.com/Droid-Deck/DroidDeck/pull/692).
|
||||
- @0x1Nawaf made their first contribution in [#696](https://github.com/Droid-Deck/DroidDeck/pull/696).
|
||||
- @Producdevity made their first contribution in [#748](https://github.com/Droid-Deck/DroidDeck/pull/748).
|
||||
- @Imgau made their first contribution in [#760](https://github.com/Droid-Deck/DroidDeck/pull/760).
|
||||
- @TjebbeKerstens made their first contribution in [#792](https://github.com/Droid-Deck/DroidDeck/pull/792).
|
||||
- @Vynex made their first contribution in [#797](https://github.com/Droid-Deck/DroidDeck/pull/797).
|
||||
- @kingokksa made their first contribution in [#798](https://github.com/Droid-Deck/DroidDeck/pull/798).
|
||||
- @asdf17128 made their first contribution in [#803](https://github.com/Droid-Deck/DroidDeck/pull/803).
|
||||
- @ovurrsl made their first contribution in [#831](https://github.com/Droid-Deck/DroidDeck/pull/831).
|
||||
- @FX192 made their first contribution in [#876](https://github.com/Droid-Deck/DroidDeck/pull/876).
|
||||
- @angelrenefm made their first contribution in [#894](https://github.com/Droid-Deck/DroidDeck/pull/894).
|
||||
- @informalTechCode made their first contribution in [#907](https://github.com/Droid-Deck/DroidDeck/pull/907).
|
||||
- @TideGear made their first contribution in [#938](https://github.com/Droid-Deck/DroidDeck/pull/938).
|
||||
- @zekena2 made their first contribution in [#952](https://github.com/Droid-Deck/DroidDeck/pull/952).
|
||||
|
||||
[Full changelog](https://github.com/Droid-Deck/DroidDeck/compare/0.4.0...0.5.0)
|
||||
|
After Width: | Height: | Size: 181 KiB |
|
After Width: | Height: | Size: 68 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 2.9 MiB |
|
After Width: | Height: | Size: 4.6 MiB |
|
After Width: | Height: | Size: 144 KiB |
|
After Width: | Height: | Size: 313 KiB |
|
After Width: | Height: | Size: 6.0 MiB |
|
After Width: | Height: | Size: 173 KiB |
|
After Width: | Height: | Size: 2.5 MiB |
|
After Width: | Height: | Size: 1.9 MiB |
|
After Width: | Height: | Size: 2.6 MiB |
|
After Width: | Height: | Size: 163 KiB |
|
After Width: | Height: | Size: 165 KiB |
|
After Width: | Height: | Size: 161 KiB |
|
After Width: | Height: | Size: 171 KiB |
|
After Width: | Height: | Size: 4.5 MiB |
|
After Width: | Height: | Size: 9.3 MiB |
|
After Width: | Height: | Size: 5.3 MiB |
|
After Width: | Height: | Size: 124 KiB |
|
After Width: | Height: | Size: 142 KiB |
|
After Width: | Height: | Size: 182 KiB |
|
After Width: | Height: | Size: 143 KiB |
@@ -0,0 +1,81 @@
|
||||
# Smart-glasses support
|
||||
|
||||
This describes the integrations implemented in DroidDeck as of 2026-10-17. A recognized product name alone does not establish that every firmware, phone, cable, or adapter supports every mode. Controls become available from the connected device's capabilities and the display modes Android actually exposes.
|
||||
|
||||
**3DoF** tracks head rotation: turn, tilt, and roll around a virtual screen. **6DoF** also tracks head position, so leaning or moving changes the view. Both apply to DroidDeck menus, non-Steam launchers, desktop sessions, and games when the tracked virtual screen is enabled. Direct SteamVR rendering uses the corresponding headset pose instead of the virtual-screen layout.
|
||||
|
||||
## Support levels
|
||||
|
||||
- **Integrated:** a device transport and controls are implemented. Tracking requires a working connection and available orientation data.
|
||||
- **SDK-dependent:** integrated support requires an APK built with the optional VITURE SDK. Recognizing the display does not mean that SDK is included.
|
||||
- **Experimental:** implemented and opt-in, with the qualifications described below. This is not a general tracking-accuracy guarantee.
|
||||
- **Display-only:** Android can provide a picture through a compatible display connection, but DroidDeck has no supported control/tracking connection for that device.
|
||||
|
||||
## Device matrix
|
||||
|
||||
| Brand / model family | Level and connection | App-wide 3DoF | App-wide 6DoF / SteamVR position | Luma stereo hand / Joy-Con position |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| VITURE One / One Lite; Pro / Pro XR | Integrated legacy USB transport | Yes, when IMU connection is available | No | No |
|
||||
| VITURE Luma / Luma Pro; Luma Cyber when identified by the SDK | Integrated Luma-family USB transport; Cyber identification uses the SDK market name | Yes, when IMU connection is available | No | No |
|
||||
| VITURE Pro 2 / Pro 2 XR | SDK-dependent Gen2 transport and device identification | Yes | No | No |
|
||||
| VITURE Beast | SDK-dependent Gen2 transport | Yes | No | No |
|
||||
| **VITURE Luma Ultra** | SDK-dependent Carina transport | Yes | **Experimental, opt-in** | **Experimental, opt-in** |
|
||||
| Recognized RayNeo / TCL Air-series devices | Integrated RayNeo USB transport | Yes, with orientation calibration controls | No | No |
|
||||
| RayNeo GT-series devices matching the supported USB identity, including GT Max identification | Integrated GT USB/IMU path | Yes, with orientation calibration controls | No | No |
|
||||
| **XREAL 1S** | **Experimental**; exact USB identity plus the device's USB-Ethernet control/IMU connection | Yes, when that connection is ready | No | No |
|
||||
| Other XREAL models; unrecognized glasses; display adapters and docks | Display-only where Android exposes a usable display; automatic glasses routing also requires display recognition | No supported app tracking connection | No | No |
|
||||
|
||||
RayNeo's Air-series recognition uses its known USB identity or a RayNeo/TCL Air name under the accepted vendor IDs; it is not a per-model certification list. The GT vendor path requires the specific supported product ID. XREAL support currently targets **1S only**: One, One Pro, Air-family devices, and recovery-mode products are not enabled by the 1S transport. VITURE Pro 2 is identified through SDK validation rather than a guessed product ID. A dock does not supply the connected glasses' sensors by itself.
|
||||
|
||||
The final column refers specifically to Luma Ultra's stereo tracking cameras. DroidDeck's separate phone-camera and physical-controller features are independent of this matrix.
|
||||
|
||||
## Display and device controls
|
||||
|
||||
All display modes depend on the phone's video output, the cable/adapter, and Android's advertised modes. USB permission or a working sensor connection alone does not create a video display.
|
||||
|
||||
| Integration | Display selection | Device controls exposed by DroidDeck |
|
||||
| --- | --- | --- |
|
||||
| VITURE One/Lite and Pro families | 1080p per eye; 2D and SBS | Brightness, volume, dimming toggle according to the family profile |
|
||||
| VITURE Luma family | 1080p or 1200p per eye; 2D and SBS | Brightness, volume, dimming toggle |
|
||||
| VITURE Luma Ultra | 1080p or 1200p per eye; 2D and SBS, including the family's 90 Hz SBS selection | Brightness, volume, dimming toggle |
|
||||
| VITURE Beast | 1080p or 1200p per eye; SDK handles native/bypass display modes | Brightness, volume, dimming toggle |
|
||||
| VITURE Pro 2 | 1080p per eye; 2D and SBS | Brightness and volume; no electrochromic-film control |
|
||||
| RayNeo Air/GT | 2D/SBS switching; resolution and refresh follow available Android modes | Orientation calibration; no app brightness, volume, or dimming commands |
|
||||
| XREAL 1S | Supported 1S display-mode commands, including native 1200p/60 Hz and SBS | Brightness and dimming toggle; no app volume command |
|
||||
|
||||
DroidDeck saves separate **2D resolution**, **3D resolution**, and **VR resolution** preferences. Unsupported saved choices fall back to the connected model's automatic behavior. Native **30 Hz** is offered only when Android advertises a matching mode and glasses presentation output is enabled. Higher refresh selections are constrained by the active 2D/SBS layout and model; a firmware-rejected switch is not treated as successful until the physical display confirms it.
|
||||
|
||||
In **Automatic** picture mode, ordinary flat content is duplicated in full into both eyes on an SBS display. **Virtual screen** adds a configurable screen plane and tracking. **Side-by-side game** is for content that already contains separate eye images. SteamVR uses its direct stereo view and field-of-view control.
|
||||
|
||||
## Setup and app-wide behavior
|
||||
|
||||
1. Connect the glasses through a compatible video/data connection. Accept USB access when requested. For XREAL 1S, Android must also expose the glasses' USB-Ethernet network; the app leaves Steam's Internet connection on its normal network.
|
||||
2. Open **Setup → Smart glasses**. Enable glasses output and **3DoF virtual screen**. This selects the tracked screen and saves the setting without forcing SBS or changing brightness.
|
||||
3. Adjust screen width and distance, then recenter while looking where the screen should sit. The same settings apply when navigating DroidDeck, opening a non-Steam launcher, or entering a game/desktop session.
|
||||
4. On an SDK-enabled Luma Ultra, optionally enable positional tracking and set the floor from the current eye height. Use the tracking setup and recovery controls for calibration or recentering. SteamVR startup defers Luma positional initialization until two seconds after SteamVR reports running.
|
||||
|
||||
The phone remains the touch/control surface. DroidDeck mirrors its own visible UI into a tracked glasses presentation; it does not capture Android's entire desktop or other apps. Android 29+ can include app dialogs/popups, while Android 26–28 captures the activity's decor only. System keyboards and permission prompts, secure windows, and separately composed `SurfaceView` content stay on their existing surfaces. Menu recentering does not change the VR floor origin. See [tracked-menu behavior](development/glasses-menu-tracking.md).
|
||||
|
||||
## Luma Ultra tracking qualifications
|
||||
|
||||
Positional tracking, bare-hand controllers, and camera-assisted Joy-Cons are separate options and default off. Bare hands and camera-assisted Joy-Cons are mutually exclusive.
|
||||
|
||||
- **Head tracking:** the SDK supplies the positional pose. The app checks freshness and connection/origin generations and provides tracking-loss and recenter recovery. Screen and floor origins are separate.
|
||||
- **Bare hands:** Full palm + Full landmarks is the default. Raw grayscale is replicated into RGB, MediaPipe applies model tensor normalization, and calibration corrects landmark coordinates before stereo triangulation. Full landmarks support palm orientation, pinch, and grab. The small saved-image comparison favored Full over Lite but does not establish universal hand-detection quality; see the [comparison results](development/hand-model-comparison-2026-10-16.md).
|
||||
- **Controller visibility:** the last accepted visual pose can remain for up to 350 ms from capture time during a short detection gap. Invalid/stale gestures release independently. Normal hand mode has no stationary dead zone; shoulder-based **Steady hand aim** is a separate opt-in setting.
|
||||
- **Joy-Cons:** camera position uses wrist landmark 0. Physical buttons and IMU orientation remain the input sources, with an arm estimate when a usable camera position is unavailable. Camera wrist position does not make a physical controller fully optically tracked in every pose. Screen and Joy-Con recenter actions are separate.
|
||||
- **Calibration:** the included 640×480 Luma Ultra stereo profile is a model-specific starting point. A matching saved device profile takes precedence. The shared profile is labeled separately from an individually distance-verified calibration; optional board collection and distance validation refine a device profile.
|
||||
- **RGB camera:** the third camera remains diagnostic-only. It is not used for controller depth, hand tracking, or camera passthrough. The failed hybrid stereo matcher and temporary USB transport experiments are not part of the live pipeline.
|
||||
|
||||
## Build and troubleshooting references
|
||||
|
||||
Luma Ultra, Beast, and Pro 2 control/tracking require the optional, externally supplied **VITURE Android SDK 2.4.0** in the APK build. Without it, DroidDeck retains its legacy VITURE and RayNeo transports and the separate XREAL integration; it cannot advertise SDK-dependent tracking. Vendor SDK headers and binaries are not stored in this repository. See [VITURE SDK setup](development/viture-sdk.md).
|
||||
|
||||
If only a flat picture is available, check video output first, then the control connection. A disabled tracking toggle can mean missing SDK support, missing USB/network access, or a device without a supported IMU transport. SBS or a new resolution requires a confirmed physical display mode; selecting a preference cannot add a mode the phone/display does not expose.
|
||||
|
||||
- [Luma spatial setup, calibration, and diagnostics](development/luma-spatial-tracking.md)
|
||||
- [Tracked DroidDeck menu implementation](development/glasses-menu-tracking.md)
|
||||
- [Agent diagnostics and capture commands](agent-control.md)
|
||||
- [RayNeo attribution and protocol references](third-party/rayneo/README.md)
|
||||
|
||||
The capability definitions are maintained in [VITURE models](../app/src/main/java/com/droiddeck/launcher/glasses/VitureProtocol.kt), [RayNeo models](../app/src/main/java/com/droiddeck/launcher/glasses/RayNeoModel.kt), [XREAL 1S](../app/src/main/java/com/droiddeck/launcher/glasses/XrealModel.kt), and [connection capability routing](../app/src/main/java/com/droiddeck/launcher/glasses/Glasses.kt). Update this matrix when those integrations change.
|
||||
@@ -0,0 +1,218 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
===========================================================================
|
||||
For files under tasks/cc/text/language_detector/custom_ops/utils/utf/
|
||||
===========================================================================
|
||||
/*
|
||||
* The authors of this software are Rob Pike and Ken Thompson.
|
||||
* Copyright (c) 2002 by Lucent Technologies.
|
||||
* Permission to use, copy, modify, and distribute this software for any
|
||||
* purpose without fee is hereby granted, provided that this entire notice
|
||||
* is included in all copies of any software which is or includes a copy
|
||||
* or modification of this software and in all copies of the supporting
|
||||
* documentation for such software.
|
||||
* THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED
|
||||
* WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY
|
||||
* REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY
|
||||
* OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE.
|
||||
*/
|
||||
@@ -0,0 +1,3 @@
|
||||
The MIT License
|
||||
Copyright © 2019 Inigo Quilez
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Camera hand tracking attribution
|
||||
|
||||
## Included
|
||||
|
||||
[MediaPipe](https://github.com/google-ai-edge/mediapipe) Tasks Vision 1.0.0
|
||||
(`com.google.mediapipe:tasks-vision`, shared with the Luma stereo tracking pipeline;
|
||||
copyright Google LLC and the MediaPipe authors, Apache License 2.0, `LICENSE_MEDIAPIPE` here) finds
|
||||
the player's body in each camera frame. It comes from Maven Central with its dependencies
|
||||
(Guava, Flogger, protobuf-javalite and Google's `datatransport`), all Apache-2.0.
|
||||
|
||||
`app/src/main/assets/handtracking/pose_landmarker_lite.task` is MediaPipe's Pose Landmarker (Lite)
|
||||
model, BlazePose GHUM 3D, unchanged from
|
||||
`https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task`
|
||||
(SHA-256 `59929e1d1ee95287735ddd833b19cf4ac46d29bc7afddbbf6753c459690d574a`). Its
|
||||
[model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20BlazePose%20GHUM%203D.pdf)
|
||||
licenses it under the Apache License 2.0 and lists its limits: one person, up to about 4 metres
|
||||
from the camera, the head in view, and depth that is approximate rather than metric.
|
||||
|
||||
`app/src/main/assets/handtracking/hand_landmarker.task` is MediaPipe's Hand Landmarker (full) model
|
||||
bundle, a palm detector and a hand landmark model, unchanged from
|
||||
`https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task`
|
||||
(SHA-256 `fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1`). Its
|
||||
[model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Hand%20Tracking%20(Lite_Full)%20with%20Fairness%20Oct%202021.pdf)
|
||||
licenses it under the Apache License 2.0; it gives 21 landmarks per hand in the picture and in metres,
|
||||
based on the GHUM hand model, and is meant for prominently shown hands.
|
||||
|
||||
The Luma stereo pipeline defaults to the Full bundle at
|
||||
`app/src/main/assets/models/hand_landmarker.task`, with the same checksum as the Full bundle
|
||||
above. The optional `app/src/main/assets/models/hand_landmarker_lite.task` is built from
|
||||
the official `palm_detection_lite.tflite` and `hand_landmark_lite.tflite` assets at
|
||||
`https://storage.googleapis.com/mediapipe-assets/`. It uses the matching tensor metadata and
|
||||
associated labels from the Full task bundle. These models share the MediaPipe hand tracking
|
||||
model card and Apache License 2.0. Source checksums and the generated bundle checksum are in
|
||||
`app/src/main/assets/models/hand_landmarker_lite.provenance.json`; `tools/build_hand_lite.py`
|
||||
reproduces the bundle with MediaPipe 0.10.21 metadata tooling.
|
||||
|
||||
[CameraX](https://developer.android.com/jetpack/androidx/releases/camera) 1.4.2
|
||||
(`androidx.camera:camera-camera2`, `camera-lifecycle`; copyright The Android Open Source Project,
|
||||
Apache License 2.0) delivers the frames.
|
||||
|
||||
`tools/vr/src/hand_mesh.h` ports Inigo Quilez's signed distance to a polygon (`sdPolygon`, from
|
||||
[2D distance functions](https://iquilezles.org/articles/distfunctions2d/) and his Shadertoy
|
||||
"Polygon - exact", `wdBXRW`; copyright 2019 Inigo Quilez, MIT License, `LICENSE_SDPOLYGON` here) for
|
||||
the glove's palm outline.
|
||||
|
||||
MediaPipe's task runner reports usage statistics through Google's `datatransport` library. The app
|
||||
manifest removes that library's `CctBackendFactory` entry, so those reports have nowhere to go and
|
||||
are dropped on the device. Camera frames are never stored or sent anywhere.
|
||||
|
||||
## References
|
||||
|
||||
No code from these projects is included. They were read to design and check the camera tracking
|
||||
and the Joy-Con movement that it corrects:
|
||||
|
||||
- [JoyConVr](https://github.com/F8LEFT/JoyConVr) by F8LEFT and contributors (MIT), `android-camera`
|
||||
at 76466c28829396651be3c68143f6297ed68923e4: an Android app that runs MediaPipe's pose landmarker
|
||||
for Joy-Con VR hands, levels the camera with gravity and slows inference down as needed.
|
||||
- Google's `GvrArmModel.cs` in the [Google VR SDK for Unity](https://github.com/googlevr/gvr-unity-sdk)
|
||||
v1.30.0 (Apache-2.0): its optional accelerometer mode moves the elbow by a filtered, integrated
|
||||
velocity and keeps the offset within a range, the idea behind the short Joy-Con movements here.
|
||||
- MediaPipe's [pose landmarker guide](https://developers.google.com/edge/mediapipe/solutions/vision/pose_landmarker),
|
||||
[hand landmarker guide](https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker),
|
||||
`pose_landmarker_graph.cc` and `pose_landmarks_detector_graph.cc` (Apache-2.0): image and world
|
||||
landmarks, and the tracking and landmark smoothing that video mode applies to a single pose, which
|
||||
the app uses. MediaPipe Holistic's idea of finding the hands from the pose's wrists informed the hand
|
||||
crops.
|
||||
- R. E. Kalman, "A New Approach to Linear Filtering and Prediction Problems", Journal of Basic
|
||||
Engineering 82(1), 1960, and the usual treatment of late measurements by replaying the stored
|
||||
inputs after them: the per-hand filter that joins Joy-Con movement and camera sightings.
|
||||
- Android's [Camera2 reference](https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics)
|
||||
(`SENSOR_ORIENTATION`, `LENS_FACING`, `LENS_INTRINSIC_CALIBRATION`, `CONTROL_ZOOM_RATIO_RANGE`,
|
||||
`SENSOR_INFO_TIMESTAMP_SOURCE`): choosing the camera, its focal length and turning frames upright.
|
||||
- [hid-nintendo](https://github.com/torvalds/linux/blob/master/drivers/hid/hid-nintendo.c)
|
||||
(GPL-2.0+): the Joy-Con IMU report, three samples every 10 ms, and its timestamps.
|
||||
- E. Foxlin, "Pedestrian Tracking with Shoe-Mounted Inertial Sensors", IEEE Computer Graphics and
|
||||
Applications 25(6), 2005: zero-velocity updates while the sensor is still.
|
||||
- I. Frosio, F. Pedersini, N. A. Borghese, "Autocalibration of MEMS Accelerometers", IEEE
|
||||
Transactions on Instrumentation and Measurement 58(6), 2009: learning an accelerometer's offset
|
||||
and scale from resting positions.
|
||||
|
||||
- [ALVR](https://github.com/alvr-org/ALVR) (MIT), `alvr/server_core/src/hand_gestures.rs` at
|
||||
ebec107b7e5f81677fb9c631d239daf2110bf573: turning tracked hands into controller input (thumb pinches
|
||||
to each finger, a pinch range from touching to pulling, grip from the other fingers, a held palm
|
||||
for the menu). The distances, mapping and timing here are DroidDeck's own.
|
||||
- Meta Quest's hand tracking: the palm-toward-the-face pinch that opens the system menu, and the
|
||||
plain white hands it draws, which the gesture and the glove model follow.
|
||||
- B. K. P. Horn, "Closed-form solution of absolute orientation using unit quaternions", Journal of
|
||||
the Optical Society of America A 4(4), 1987: fitting the skeleton's palm to the camera's.
|
||||
- G. Casiez, N. Roussel, D. Vogel, "1€ Filter: A Simple Speed-based Low-pass Filter for Noisy Input in
|
||||
Interactive Systems", CHI 2012: the bare hands' position and turn filters.
|
||||
- Valve's [OpenVR wiki](https://github.com/ValveSoftware/openvr/wiki) pages "Render Model Reference"
|
||||
and "Driver Render Models": the render model JSON, its components, `component_path` and visibility.
|
||||
The glove's tip, base, handgrip and grip poses are the numbers in SteamVR's Valve Index controller
|
||||
model (`valve_controller_knu_1_0_left` and `_right`), so the pointer leaves the hand as it leaves an
|
||||
Index controller, and its button parts showed which fields SteamVR needs to route a part to an input.
|
||||
- S. F. F. Gibson, "Constrained Elastic Surface Nets", MICCAI 1998, and Mikola Lysenko's
|
||||
[Smooth Voxel Terrain (Part 2)](https://0fps.net/2012/07/12/smooth-voxel-terrain-part-2/): meshing
|
||||
the glove's distance field with surface nets.
|
||||
- [HaGRID](https://github.com/hukenovs/hagrid) (A. Kapitanov et al., "HaGRID - HAnd Gesture Recognition
|
||||
Image Dataset", WACV 2024; CC BY-SA 4.0), through the 384p sample
|
||||
[cj-mills/hagrid-sample-30k-384p](https://huggingface.co/datasets/cj-mills/hagrid-sample-30k-384p):
|
||||
550 of its photos were used, on the development machine only, to check the pinch and grip distances.
|
||||
No picture is included.
|
||||
- Inigo Quilez's articles on [smooth minimum](https://iquilezles.org/articles/smin/) and
|
||||
[3D distance functions](https://iquilezles.org/articles/distfunctions/) (the polynomial smooth
|
||||
minimum and the ellipsoid bound) and his talk "Rendering Worlds with Two Triangles" (NVScene 2008;
|
||||
ambient occlusion from a distance field): blending the glove's capsules, palm and wrist band, and
|
||||
baking its shading.
|
||||
|
||||
Nintendo Switch and Joy-Con are trademarks of Nintendo. This project is not affiliated with Nintendo,
|
||||
Google, Meta or Valve.
|
||||
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,23 @@
|
||||
Boost Software License - Version 1.0 - August 17th, 2003
|
||||
|
||||
Permission is hereby granted, free of charge, to any person or organization
|
||||
obtaining a copy of the software and accompanying documentation covered by
|
||||
this license (the "Software") to use, reproduce, display, distribute, execute,
|
||||
and transmit the Software, and to prepare derivative works of the Software,
|
||||
and to permit third-parties to whom the Software is furnished to do so, all
|
||||
subject to the following:
|
||||
|
||||
The copyright notices in the Software and this entire statement, including
|
||||
the above license grant, this restriction and the following disclaimer, must
|
||||
be included in all copies of the Software, in whole or in part, and all derivative
|
||||
works of the Software, unless such copies or derivative works are solely in
|
||||
the form of machine-executable object code generated by a source language
|
||||
processor.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES
|
||||
OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Joy-Con VR controller attribution
|
||||
|
||||
## Adapted code
|
||||
|
||||
`tools/vr/src/arm_model.h` ports the arm model of Google's `GvrArmModel.cs` from the
|
||||
[Google VR SDK for Unity](https://github.com/googlevr/gvr-unity-sdk)
|
||||
(`Assets/GoogleVR/Scripts/Controller/ArmModel/GvrArmModel.cs` at
|
||||
f999ed727a1d238c246ba4a504b77d55b59c9e9a; copyright 2016 Google Inc., Apache License 2.0, LICENSE_GVR
|
||||
here). It is rewritten in C++ in OpenVR's coordinate frame. The rest positions, extension
|
||||
angles, elbow bend, extension weight and the torso following the gaze are Google's. The neck
|
||||
offset is left out, as `GvrArmModel` leaves it out by default.
|
||||
|
||||
`tools/vr/src/hand_skeleton.h` adapts `hand_simulation.cpp` from Valve's OpenVR
|
||||
`handskeletonsimulation` driver sample
|
||||
([ValveSoftware/openvr](https://github.com/ValveSoftware/openvr), `samples/drivers/drivers/handskeletonsimulation`
|
||||
at 0924064316de3effbcd1acf1e309182a2deb1c05; copyright Valve Corporation, BSD-3-Clause,
|
||||
`tools/vr/src/OPENVR_LICENSE`). Valve's sample is inspired by Moshi Turner's hand simulation in
|
||||
[Monado](https://gitlab.freedesktop.org/monado/monado) (`src/xrt/auxiliary/util/u_hand_simulation.c`).
|
||||
|
||||
`tools/vr/src/index_pose.h` uses the Valve Index tip rotation and grip offset listed in Monado's
|
||||
`src/xrt/auxiliary/vive/vive_poses.c` (at 8b259763fdd23c46592fd4cdd30b4eea997d4b70; copyright
|
||||
Collabora, Ltd., author Daniel Willmott, Boost Software License 1.0, LICENSE_MONADO here).
|
||||
|
||||
## References
|
||||
|
||||
No code from these projects is included. They were read to learn how Joy-Cons, Android and
|
||||
SteamVR behave, and to check this implementation:
|
||||
|
||||
- [hid-nintendo](https://github.com/torvalds/linux/blob/master/drivers/hid/hid-nintendo.c), the Linux
|
||||
Joy-Con driver by Daniel J. Ogorchock and contributors (GPL-2.0+): button codes, stick axes, IMU
|
||||
axes and units, and the rumble channels Android exposes.
|
||||
- [Nintendo_Switch_Reverse_Engineering](https://github.com/dekuNukem/Nintendo_Switch_Reverse_Engineering)
|
||||
by dekuNukem: the Joy-Con's buttons, sensors and rumble.
|
||||
- [SDL](https://github.com/libsdl-org/SDL) (zlib): how Switch controller sensors map to a game
|
||||
controller's axes.
|
||||
- [Android Open Source Project](https://github.com/aosp-mirror/platform_frameworks_base) (Apache-2.0):
|
||||
`data/keyboards/Generic.kl`, which Android applies to Joy-Cons, and the `InputDevice` sensor and
|
||||
vibrator APIs.
|
||||
- [ALVR](https://github.com/alvr-org/ALVR) (MIT): presenting controllers to SteamVR as Index
|
||||
controllers and handling their haptic events.
|
||||
- [OpenComposite](https://gitlab.com/znixian/OpenOVR) (GPL-3.0): cross-check of the Index input
|
||||
profile and skeleton.
|
||||
- [PSMoveSteamVRBridge](https://github.com/HipsterSloth/PSMoveSteamVRBridge) (Apache-2.0): driving
|
||||
SteamVR hands from orientation-only controllers.
|
||||
- [JoyConVr](https://github.com/F8LEFT/JoyConVr) by F8LEFT (MIT): a Joy-Con button layout for SteamVR
|
||||
and recentering by holding a button.
|
||||
- [Daydream-Controller-SteamVR](https://github.com/krnelpan1c/Daydream-Controller-SteamVR) by
|
||||
krnelpan1c: snapping a 3DoF controller's heading to the view by holding Home.
|
||||
- [joyconvr](https://github.com/yoyobuae/joyconvr) by yoyobuae: Joy-Con sensor axes in SteamVR.
|
||||
- [Moonlight for Android](https://github.com/moonlight-stream/moonlight-android) (GPL-3.0) and
|
||||
[Dolphin](https://github.com/dolphin-emu/dolphin) (GPL-2.0+): reading a game controller's motion
|
||||
sensors through Android's `InputDevice` API.
|
||||
|
||||
Nintendo Switch and Joy-Con are trademarks of Nintendo. Valve Index and SteamVR are trademarks of
|
||||
Valve Corporation. This project is not affiliated with either.
|
||||
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 verncat
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,13 @@
|
||||
Monado Reference Attribution
|
||||
============================
|
||||
|
||||
This project's RayNeo implementation was developed by referencing behavior and protocol details from the Monado project:
|
||||
|
||||
- Project: https://gitlab.freedesktop.org/monado/monado
|
||||
- Relevant area: src/xrt/drivers/rayneo/
|
||||
|
||||
Notes:
|
||||
|
||||
- The app code in this repository was written for this project.
|
||||
- Protocol understanding (packet IDs/layout, command flow, and initialization sequencing) was informed by the Monado RayNeo driver implementation.
|
||||
- Monado is distributed under the Boost Software License 1.0 (BSL-1.0). See the upstream repository for full licensing and notices.
|
||||
@@ -0,0 +1,21 @@
|
||||
# RayNeo protocol and orientation filter attribution
|
||||
|
||||
RayNeoProtocol.kt and RayNeoOrientationFilter.kt are adapted from
|
||||
https://github.com/informalTechCode/RayNeo-Air-Series-3DoF-Scaffold
|
||||
at 581dbf712ed601ece4cf5e4a7c1f52e780299c88 (Apache License 2.0; LICENSE here).
|
||||
The scaffold references Monado for protocol details; its notice is preserved here.
|
||||
|
||||
DroidDeck adaptations add frame validation, unsigned device ticks, endpoint filtering,
|
||||
and conversion into the existing VITURE pose coordinate convention. The transport,
|
||||
USB discovery and DroidDeck integration are implemented for this project.
|
||||
|
||||
GT USB identification and interface selection reference
|
||||
https://github.com/verncat/RayNeo-Air-3S-Pro-OpenVR
|
||||
at dfd2e8abab7052d88a1047266d89260347d0dd67, especially merged PR #3:
|
||||
https://github.com/verncat/RayNeo-Air-3S-Pro-OpenVR/pull/3
|
||||
(MIT; copyright 2025 verncat; full notice in LICENSE_VERNcat).
|
||||
That contribution reports GT hardware tests for 3941:af50 on interface 5,
|
||||
IMU streaming, spatial-button notification 0x08, and display/volume ACK handling.
|
||||
DroidDeck keeps its Android transport and existing command-specific ACK parser;
|
||||
it prefers interface 5 for this identity and falls back only to a unique pair
|
||||
of protocol-sized endpoints. GT Max is not separately verified by this source.
|
||||
@@ -0,0 +1,504 @@
|
||||
GNU LESSER GENERAL PUBLIC LICENSE
|
||||
Version 2.1, February 1999
|
||||
|
||||
Copyright (C) 1991, 1999 Free Software Foundation, Inc.
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
[This is the first released version of the Lesser GPL. It also counts
|
||||
as the successor of the GNU Library Public License, version 2, hence
|
||||
the version number 2.1.]
|
||||
|
||||
Preamble
|
||||
|
||||
The licenses for most software are designed to take away your
|
||||
freedom to share and change it. By contrast, the GNU General Public
|
||||
Licenses are intended to guarantee your freedom to share and change
|
||||
free software--to make sure the software is free for all its users.
|
||||
|
||||
This license, the Lesser General Public License, applies to some
|
||||
specially designated software packages--typically libraries--of the
|
||||
Free Software Foundation and other authors who decide to use it. You
|
||||
can use it too, but we suggest you first think carefully about whether
|
||||
this license or the ordinary General Public License is the better
|
||||
strategy to use in any particular case, based on the explanations below.
|
||||
|
||||
When we speak of free software, we are referring to freedom of use,
|
||||
not price. Our General Public Licenses are designed to make sure that
|
||||
you have the freedom to distribute copies of free software (and charge
|
||||
for this service if you wish); that you receive source code or can get
|
||||
it if you want it; that you can change the software and use pieces of
|
||||
it in new free programs; and that you are informed that you can do
|
||||
these things.
|
||||
|
||||
To protect your rights, we need to make restrictions that forbid
|
||||
distributors to deny you these rights or to ask you to surrender these
|
||||
rights. These restrictions translate to certain responsibilities for
|
||||
you if you distribute copies of the library or if you modify it.
|
||||
|
||||
For example, if you distribute copies of the library, whether gratis
|
||||
or for a fee, you must give the recipients all the rights that we gave
|
||||
you. You must make sure that they, too, receive or can get the source
|
||||
code. If you link other code with the library, you must provide
|
||||
complete object files to the recipients, so that they can relink them
|
||||
with the library after making changes to the library and recompiling
|
||||
it. And you must show them these terms so they know their rights.
|
||||
|
||||
We protect your rights with a two-step method: (1) we copyright the
|
||||
library, and (2) we offer you this license, which gives you legal
|
||||
permission to copy, distribute and/or modify the library.
|
||||
|
||||
To protect each distributor, we want to make it very clear that
|
||||
there is no warranty for the free library. Also, if the library is
|
||||
modified by someone else and passed on, the recipients should know
|
||||
that what they have is not the original version, so that the original
|
||||
author's reputation will not be affected by problems that might be
|
||||
introduced by others.
|
||||
|
||||
Finally, software patents pose a constant threat to the existence of
|
||||
any free program. We wish to make sure that a company cannot
|
||||
effectively restrict the users of a free program by obtaining a
|
||||
restrictive license from a patent holder. Therefore, we insist that
|
||||
any patent license obtained for a version of the library must be
|
||||
consistent with the full freedom of use specified in this license.
|
||||
|
||||
Most GNU software, including some libraries, is covered by the
|
||||
ordinary GNU General Public License. This license, the GNU Lesser
|
||||
General Public License, applies to certain designated libraries, and
|
||||
is quite different from the ordinary General Public License. We use
|
||||
this license for certain libraries in order to permit linking those
|
||||
libraries into non-free programs.
|
||||
|
||||
When a program is linked with a library, whether statically or using
|
||||
a shared library, the combination of the two is legally speaking a
|
||||
combined work, a derivative of the original library. The ordinary
|
||||
General Public License therefore permits such linking only if the
|
||||
entire combination fits its criteria of freedom. The Lesser General
|
||||
Public License permits more lax criteria for linking other code with
|
||||
the library.
|
||||
|
||||
We call this license the "Lesser" General Public License because it
|
||||
does Less to protect the user's freedom than the ordinary General
|
||||
Public License. It also provides other free software developers Less
|
||||
of an advantage over competing non-free programs. These disadvantages
|
||||
are the reason we use the ordinary General Public License for many
|
||||
libraries. However, the Lesser license provides advantages in certain
|
||||
special circumstances.
|
||||
|
||||
For example, on rare occasions, there may be a special need to
|
||||
encourage the widest possible use of a certain library, so that it becomes
|
||||
a de-facto standard. To achieve this, non-free programs must be
|
||||
allowed to use the library. A more frequent case is that a free
|
||||
library does the same job as widely used non-free libraries. In this
|
||||
case, there is little to gain by limiting the free library to free
|
||||
software only, so we use the Lesser General Public License.
|
||||
|
||||
In other cases, permission to use a particular library in non-free
|
||||
programs enables a greater number of people to use a large body of
|
||||
free software. For example, permission to use the GNU C Library in
|
||||
non-free programs enables many more people to use the whole GNU
|
||||
operating system, as well as its variant, the GNU/Linux operating
|
||||
system.
|
||||
|
||||
Although the Lesser General Public License is Less protective of the
|
||||
users' freedom, it does ensure that the user of a program that is
|
||||
linked with the Library has the freedom and the wherewithal to run
|
||||
that program using a modified version of the Library.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow. Pay close attention to the difference between a
|
||||
"work based on the library" and a "work that uses the library". The
|
||||
former contains code derived from the library, whereas the latter must
|
||||
be combined with the library in order to run.
|
||||
|
||||
GNU LESSER GENERAL PUBLIC LICENSE
|
||||
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
||||
|
||||
0. This License Agreement applies to any software library or other
|
||||
program which contains a notice placed by the copyright holder or
|
||||
other authorized party saying it may be distributed under the terms of
|
||||
this Lesser General Public License (also called "this License").
|
||||
Each licensee is addressed as "you".
|
||||
|
||||
A "library" means a collection of software functions and/or data
|
||||
prepared so as to be conveniently linked with application programs
|
||||
(which use some of those functions and data) to form executables.
|
||||
|
||||
The "Library", below, refers to any such software library or work
|
||||
which has been distributed under these terms. A "work based on the
|
||||
Library" means either the Library or any derivative work under
|
||||
copyright law: that is to say, a work containing the Library or a
|
||||
portion of it, either verbatim or with modifications and/or translated
|
||||
straightforwardly into another language. (Hereinafter, translation is
|
||||
included without limitation in the term "modification".)
|
||||
|
||||
"Source code" for a work means the preferred form of the work for
|
||||
making modifications to it. For a library, complete source code means
|
||||
all the source code for all modules it contains, plus any associated
|
||||
interface definition files, plus the scripts used to control compilation
|
||||
and installation of the library.
|
||||
|
||||
Activities other than copying, distribution and modification are not
|
||||
covered by this License; they are outside its scope. The act of
|
||||
running a program using the Library is not restricted, and output from
|
||||
such a program is covered only if its contents constitute a work based
|
||||
on the Library (independent of the use of the Library in a tool for
|
||||
writing it). Whether that is true depends on what the Library does
|
||||
and what the program that uses the Library does.
|
||||
|
||||
1. You may copy and distribute verbatim copies of the Library's
|
||||
complete source code as you receive it, in any medium, provided that
|
||||
you conspicuously and appropriately publish on each copy an
|
||||
appropriate copyright notice and disclaimer of warranty; keep intact
|
||||
all the notices that refer to this License and to the absence of any
|
||||
warranty; and distribute a copy of this License along with the
|
||||
Library.
|
||||
|
||||
You may charge a fee for the physical act of transferring a copy,
|
||||
and you may at your option offer warranty protection in exchange for a
|
||||
fee.
|
||||
|
||||
2. You may modify your copy or copies of the Library or any portion
|
||||
of it, thus forming a work based on the Library, and copy and
|
||||
distribute such modifications or work under the terms of Section 1
|
||||
above, provided that you also meet all of these conditions:
|
||||
|
||||
a) The modified work must itself be a software library.
|
||||
|
||||
b) You must cause the files modified to carry prominent notices
|
||||
stating that you changed the files and the date of any change.
|
||||
|
||||
c) You must cause the whole of the work to be licensed at no
|
||||
charge to all third parties under the terms of this License.
|
||||
|
||||
d) If a facility in the modified Library refers to a function or a
|
||||
table of data to be supplied by an application program that uses
|
||||
the facility, other than as an argument passed when the facility
|
||||
is invoked, then you must make a good faith effort to ensure that,
|
||||
in the event an application does not supply such function or
|
||||
table, the facility still operates, and performs whatever part of
|
||||
its purpose remains meaningful.
|
||||
|
||||
(For example, a function in a library to compute square roots has
|
||||
a purpose that is entirely well-defined independent of the
|
||||
application. Therefore, Subsection 2d requires that any
|
||||
application-supplied function or table used by this function must
|
||||
be optional: if the application does not supply it, the square
|
||||
root function must still compute square roots.)
|
||||
|
||||
These requirements apply to the modified work as a whole. If
|
||||
identifiable sections of that work are not derived from the Library,
|
||||
and can be reasonably considered independent and separate works in
|
||||
themselves, then this License, and its terms, do not apply to those
|
||||
sections when you distribute them as separate works. But when you
|
||||
distribute the same sections as part of a whole which is a work based
|
||||
on the Library, the distribution of the whole must be on the terms of
|
||||
this License, whose permissions for other licensees extend to the
|
||||
entire whole, and thus to each and every part regardless of who wrote
|
||||
it.
|
||||
|
||||
Thus, it is not the intent of this section to claim rights or contest
|
||||
your rights to work written entirely by you; rather, the intent is to
|
||||
exercise the right to control the distribution of derivative or
|
||||
collective works based on the Library.
|
||||
|
||||
In addition, mere aggregation of another work not based on the Library
|
||||
with the Library (or with a work based on the Library) on a volume of
|
||||
a storage or distribution medium does not bring the other work under
|
||||
the scope of this License.
|
||||
|
||||
3. You may opt to apply the terms of the ordinary GNU General Public
|
||||
License instead of this License to a given copy of the Library. To do
|
||||
this, you must alter all the notices that refer to this License, so
|
||||
that they refer to the ordinary GNU General Public License, version 2,
|
||||
instead of to this License. (If a newer version than version 2 of the
|
||||
ordinary GNU General Public License has appeared, then you can specify
|
||||
that version instead if you wish.) Do not make any other change in
|
||||
these notices.
|
||||
|
||||
Once this change is made in a given copy, it is irreversible for
|
||||
that copy, so the ordinary GNU General Public License applies to all
|
||||
subsequent copies and derivative works made from that copy.
|
||||
|
||||
This option is useful when you wish to copy part of the code of
|
||||
the Library into a program that is not a library.
|
||||
|
||||
4. You may copy and distribute the Library (or a portion or
|
||||
derivative of it, under Section 2) in object code or executable form
|
||||
under the terms of Sections 1 and 2 above provided that you accompany
|
||||
it with the complete corresponding machine-readable source code, which
|
||||
must be distributed under the terms of Sections 1 and 2 above on a
|
||||
medium customarily used for software interchange.
|
||||
|
||||
If distribution of object code is made by offering access to copy
|
||||
from a designated place, then offering equivalent access to copy the
|
||||
source code from the same place satisfies the requirement to
|
||||
distribute the source code, even though third parties are not
|
||||
compelled to copy the source along with the object code.
|
||||
|
||||
5. A program that contains no derivative of any portion of the
|
||||
Library, but is designed to work with the Library by being compiled or
|
||||
linked with it, is called a "work that uses the Library". Such a
|
||||
work, in isolation, is not a derivative work of the Library, and
|
||||
therefore falls outside the scope of this License.
|
||||
|
||||
However, linking a "work that uses the Library" with the Library
|
||||
creates an executable that is a derivative of the Library (because it
|
||||
contains portions of the Library), rather than a "work that uses the
|
||||
library". The executable is therefore covered by this License.
|
||||
Section 6 states terms for distribution of such executables.
|
||||
|
||||
When a "work that uses the Library" uses material from a header file
|
||||
that is part of the Library, the object code for the work may be a
|
||||
derivative work of the Library even though the source code is not.
|
||||
Whether this is true is especially significant if the work can be
|
||||
linked without the Library, or if the work is itself a library. The
|
||||
threshold for this to be true is not precisely defined by law.
|
||||
|
||||
If such an object file uses only numerical parameters, data
|
||||
structure layouts and accessors, and small macros and small inline
|
||||
functions (ten lines or less in length), then the use of the object
|
||||
file is unrestricted, regardless of whether it is legally a derivative
|
||||
work. (Executables containing this object code plus portions of the
|
||||
Library will still fall under Section 6.)
|
||||
|
||||
Otherwise, if the work is a derivative of the Library, you may
|
||||
distribute the object code for the work under the terms of Section 6.
|
||||
Any executables containing that work also fall under Section 6,
|
||||
whether or not they are linked directly with the Library itself.
|
||||
|
||||
6. As an exception to the Sections above, you may also combine or
|
||||
link a "work that uses the Library" with the Library to produce a
|
||||
work containing portions of the Library, and distribute that work
|
||||
under terms of your choice, provided that the terms permit
|
||||
modification of the work for the customer's own use and reverse
|
||||
engineering for debugging such modifications.
|
||||
|
||||
You must give prominent notice with each copy of the work that the
|
||||
Library is used in it and that the Library and its use are covered by
|
||||
this License. You must supply a copy of this License. If the work
|
||||
during execution displays copyright notices, you must include the
|
||||
copyright notice for the Library among them, as well as a reference
|
||||
directing the user to the copy of this License. Also, you must do one
|
||||
of these things:
|
||||
|
||||
a) Accompany the work with the complete corresponding
|
||||
machine-readable source code for the Library including whatever
|
||||
changes were used in the work (which must be distributed under
|
||||
Sections 1 and 2 above); and, if the work is an executable linked
|
||||
with the Library, with the complete machine-readable "work that
|
||||
uses the Library", as object code and/or source code, so that the
|
||||
user can modify the Library and then relink to produce a modified
|
||||
executable containing the modified Library. (It is understood
|
||||
that the user who changes the contents of definitions files in the
|
||||
Library will not necessarily be able to recompile the application
|
||||
to use the modified definitions.)
|
||||
|
||||
b) Use a suitable shared library mechanism for linking with the
|
||||
Library. A suitable mechanism is one that (1) uses at run time a
|
||||
copy of the library already present on the user's computer system,
|
||||
rather than copying library functions into the executable, and (2)
|
||||
will operate properly with a modified version of the library, if
|
||||
the user installs one, as long as the modified version is
|
||||
interface-compatible with the version that the work was made with.
|
||||
|
||||
c) Accompany the work with a written offer, valid for at
|
||||
least three years, to give the same user the materials
|
||||
specified in Subsection 6a, above, for a charge no more
|
||||
than the cost of performing this distribution.
|
||||
|
||||
d) If distribution of the work is made by offering access to copy
|
||||
from a designated place, offer equivalent access to copy the above
|
||||
specified materials from the same place.
|
||||
|
||||
e) Verify that the user has already received a copy of these
|
||||
materials or that you have already sent this user a copy.
|
||||
|
||||
For an executable, the required form of the "work that uses the
|
||||
Library" must include any data and utility programs needed for
|
||||
reproducing the executable from it. However, as a special exception,
|
||||
the materials to be distributed need not include anything that is
|
||||
normally distributed (in either source or binary form) with the major
|
||||
components (compiler, kernel, and so on) of the operating system on
|
||||
which the executable runs, unless that component itself accompanies
|
||||
the executable.
|
||||
|
||||
It may happen that this requirement contradicts the license
|
||||
restrictions of other proprietary libraries that do not normally
|
||||
accompany the operating system. Such a contradiction means you cannot
|
||||
use both them and the Library together in an executable that you
|
||||
distribute.
|
||||
|
||||
7. You may place library facilities that are a work based on the
|
||||
Library side-by-side in a single library together with other library
|
||||
facilities not covered by this License, and distribute such a combined
|
||||
library, provided that the separate distribution of the work based on
|
||||
the Library and of the other library facilities is otherwise
|
||||
permitted, and provided that you do these two things:
|
||||
|
||||
a) Accompany the combined library with a copy of the same work
|
||||
based on the Library, uncombined with any other library
|
||||
facilities. This must be distributed under the terms of the
|
||||
Sections above.
|
||||
|
||||
b) Give prominent notice with the combined library of the fact
|
||||
that part of it is a work based on the Library, and explaining
|
||||
where to find the accompanying uncombined form of the same work.
|
||||
|
||||
8. You may not copy, modify, sublicense, link with, or distribute
|
||||
the Library except as expressly provided under this License. Any
|
||||
attempt otherwise to copy, modify, sublicense, link with, or
|
||||
distribute the Library is void, and will automatically terminate your
|
||||
rights under this License. However, parties who have received copies,
|
||||
or rights, from you under this License will not have their licenses
|
||||
terminated so long as such parties remain in full compliance.
|
||||
|
||||
9. You are not required to accept this License, since you have not
|
||||
signed it. However, nothing else grants you permission to modify or
|
||||
distribute the Library or its derivative works. These actions are
|
||||
prohibited by law if you do not accept this License. Therefore, by
|
||||
modifying or distributing the Library (or any work based on the
|
||||
Library), you indicate your acceptance of this License to do so, and
|
||||
all its terms and conditions for copying, distributing or modifying
|
||||
the Library or works based on it.
|
||||
|
||||
10. Each time you redistribute the Library (or any work based on the
|
||||
Library), the recipient automatically receives a license from the
|
||||
original licensor to copy, distribute, link with or modify the Library
|
||||
subject to these terms and conditions. You may not impose any further
|
||||
restrictions on the recipients' exercise of the rights granted herein.
|
||||
You are not responsible for enforcing compliance by third parties with
|
||||
this License.
|
||||
|
||||
11. If, as a consequence of a court judgment or allegation of patent
|
||||
infringement or for any other reason (not limited to patent issues),
|
||||
conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot
|
||||
distribute so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you
|
||||
may not distribute the Library at all. For example, if a patent
|
||||
license would not permit royalty-free redistribution of the Library by
|
||||
all those who receive copies directly or indirectly through you, then
|
||||
the only way you could satisfy both it and this License would be to
|
||||
refrain entirely from distribution of the Library.
|
||||
|
||||
If any portion of this section is held invalid or unenforceable under any
|
||||
particular circumstance, the balance of the section is intended to apply,
|
||||
and the section as a whole is intended to apply in other circumstances.
|
||||
|
||||
It is not the purpose of this section to induce you to infringe any
|
||||
patents or other property right claims or to contest validity of any
|
||||
such claims; this section has the sole purpose of protecting the
|
||||
integrity of the free software distribution system which is
|
||||
implemented by public license practices. Many people have made
|
||||
generous contributions to the wide range of software distributed
|
||||
through that system in reliance on consistent application of that
|
||||
system; it is up to the author/donor to decide if he or she is willing
|
||||
to distribute software through any other system and a licensee cannot
|
||||
impose that choice.
|
||||
|
||||
This section is intended to make thoroughly clear what is believed to
|
||||
be a consequence of the rest of this License.
|
||||
|
||||
12. If the distribution and/or use of the Library is restricted in
|
||||
certain countries either by patents or by copyrighted interfaces, the
|
||||
original copyright holder who places the Library under this License may add
|
||||
an explicit geographical distribution limitation excluding those countries,
|
||||
so that distribution is permitted only in or among countries not thus
|
||||
excluded. In such case, this License incorporates the limitation as if
|
||||
written in the body of this License.
|
||||
|
||||
13. The Free Software Foundation may publish revised and/or new
|
||||
versions of the Lesser General Public License from time to time.
|
||||
Such new versions will be similar in spirit to the present version,
|
||||
but may differ in detail to address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the Library
|
||||
specifies a version number of this License which applies to it and
|
||||
"any later version", you have the option of following the terms and
|
||||
conditions either of that version or of any later version published by
|
||||
the Free Software Foundation. If the Library does not specify a
|
||||
license version number, you may choose any version ever published by
|
||||
the Free Software Foundation.
|
||||
|
||||
14. If you wish to incorporate parts of the Library into other free
|
||||
programs whose distribution conditions are incompatible with these,
|
||||
write to the author to ask for permission. For software which is
|
||||
copyrighted by the Free Software Foundation, write to the Free
|
||||
Software Foundation; we sometimes make exceptions for this. Our
|
||||
decision will be guided by the two goals of preserving the free status
|
||||
of all derivatives of our free software and of promoting the sharing
|
||||
and reuse of software generally.
|
||||
|
||||
NO WARRANTY
|
||||
|
||||
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
|
||||
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
|
||||
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
|
||||
OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
|
||||
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
|
||||
LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
|
||||
THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
|
||||
WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
|
||||
AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
|
||||
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
|
||||
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
|
||||
LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
|
||||
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
|
||||
FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
|
||||
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
|
||||
DAMAGES.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Libraries
|
||||
|
||||
If you develop a new library, and you want it to be of the greatest
|
||||
possible use to the public, we recommend making it free software that
|
||||
everyone can redistribute and change. You can do so by permitting
|
||||
redistribution under these terms (or, alternatively, under the terms of the
|
||||
ordinary General Public License).
|
||||
|
||||
To apply these terms, attach the following notices to the library. It is
|
||||
safest to attach them to the start of each source file to most effectively
|
||||
convey the exclusion of warranty; and each file should have at least the
|
||||
"copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the library's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This library is free software; you can redistribute it and/or
|
||||
modify it under the terms of the GNU Lesser General Public
|
||||
License as published by the Free Software Foundation; either
|
||||
version 2.1 of the License, or (at your option) any later version.
|
||||
|
||||
This library is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
Lesser General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Lesser General Public
|
||||
License along with this library; if not, write to the Free Software
|
||||
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or your
|
||||
school, if any, to sign a "copyright disclaimer" for the library, if
|
||||
necessary. Here is a sample; alter the names:
|
||||
|
||||
Yoyodyne, Inc., hereby disclaims all copyright interest in the
|
||||
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
|
||||
|
||||
<signature of Ty Coon>, 1 April 1990
|
||||
Ty Coon, President of Vice
|
||||
|
||||
That's all there is to it!
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
Software License Agreement (BSD License)
|
||||
|
||||
Copyright (C) 2010-2015 Ken Tossell
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions
|
||||
are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following
|
||||
disclaimer in the documentation and/or other materials provided
|
||||
with the distribution.
|
||||
* Neither the name of the author nor other contributors may be
|
||||
used to endorse or promote products derived from this software
|
||||
without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
|
||||
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
|
||||
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
|
||||
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
|
||||
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
|
||||
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
|
||||
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||
POSSIBILITY OF SUCH DAMAGE.
|
||||
@@ -0,0 +1,71 @@
|
||||
# RGB USB camera transport
|
||||
|
||||
The optional RGB diagnostic backend uses [libusb 1.0.30](https://github.com/libusb/libusb/releases/tag/v1.0.30)
|
||||
(LGPL 2.1 or later) and [libuvc 0.0.8](https://github.com/libuvc/libuvc/releases/tag/v0.0.8)
|
||||
(BSD 3-Clause). Full license texts are in `LICENSE_LIBUSB` and `LICENSE_LIBUVC` here.
|
||||
|
||||
`tools/build_rgb_uvc.py` obtains the official sources with these SHA-256 pins:
|
||||
|
||||
| Source | SHA-256 |
|
||||
| --- | --- |
|
||||
| [libusb-1.0.30.tar.bz2](https://github.com/libusb/libusb/releases/download/v1.0.30/libusb-1.0.30.tar.bz2) | `fea36f34f9156400209595e300840767ab1a385ede1dc7ee893015aea9c6dbaf` |
|
||||
| [libuvc v0.0.8 ZIP](https://github.com/libuvc/libuvc/archive/refs/tags/v0.0.8.zip) | `67922938ad81eb82b8ce14576f15a68ac75e30af738728a1f3070b6bbc938bce` |
|
||||
|
||||
## Build and replace
|
||||
|
||||
Use Python 3.12 or newer, Android NDK 27.3.13750724, CMake 3.22.1, and Ninja.
|
||||
From the repository root on Windows:
|
||||
|
||||
```powershell
|
||||
python tools/build_rgb_uvc.py --ndk "$env:LOCALAPPDATA/Android/Sdk/ndk/27.3.13750724" --cmake "$env:LOCALAPPDATA/Android/Sdk/cmake/3.22.1/bin/cmake.exe"
|
||||
```
|
||||
|
||||
The default output is `build/rgb-uvc/dist`: `arm64-v8a/libusb.so`, `arm64-v8a/libuvc.a`,
|
||||
public headers in `include`, and license texts in `licenses`. The script records archive,
|
||||
patch, recipe, and binary hashes plus complete build commands in `provenance.json`.
|
||||
Use `--cache` to reuse downloaded archives and `--output` for an independent build.
|
||||
Each run refreshes the extracted vendor sources from the pinned archives before patching.
|
||||
|
||||
Set `droiddeck.uvcDir=C:/path/to/build/rgb-uvc/dist` in the untracked `local.properties`, or
|
||||
pass `-PuvcDir=C:/path/to/build/rgb-uvc/dist` to Gradle. The optional backend and shared libusb
|
||||
are packaged through CMake; license texts are staged into APK assets under `licenses/uvc`.
|
||||
Without this input, the RGB SDK diagnostic remains available and direct UVC reports unavailable.
|
||||
|
||||
libusb is a separate shared library with SONAME `libusb.so`; the application must package
|
||||
it beside its JNI backend and keep it dynamically linked. libuvc is static PIC with hidden
|
||||
symbols. A backend links `libuvc.a`, `libusb.so`, and Android `log`, and includes `libusb.h`
|
||||
before `libuvc/libuvc.h`. Both native components target ARM64/API 26; shared libraries use
|
||||
16 KiB ELF LOAD alignment. No JPEG conversion dependency is included: camera payloads
|
||||
remain MJPEG and are decoded by the caller.
|
||||
|
||||
To rebuild libusb with modifications, run the script once, edit its extracted sources under
|
||||
the output's `src/libusb-1.0.30`, then use the recorded CMake configure/build commands directly
|
||||
without rerunning the Python extraction step. Package the resulting `libusb.so` in a rebuilt
|
||||
APK. Keep the corresponding sources, recipe, patches, and LGPL text with release source
|
||||
distribution; do not rely solely on the upstream download links. The generated manifest
|
||||
describes the pinned build; regenerate hashes when distributing modified binaries.
|
||||
|
||||
## Local changes and validation
|
||||
|
||||
libuvc's queue uses eight transfer buffers instead of the upstream Linux default of 100.
|
||||
The per-transfer limit remains 32 isochronous packets. The patch in
|
||||
`tools/uvc/libuvc-submit-failure.patch` preserves a USB submission error instead of returning
|
||||
success with an incomplete or empty queue. Unsubmitted transfers are removed under the
|
||||
callback mutex. The normal stop path then holds that mutex while checking and cancelling
|
||||
remaining transfers, waits for completion callbacks to clear them, and joins the frame
|
||||
callback thread before returning. This prevents cancellation of already-cleared slots.
|
||||
|
||||
`uvc_submit_failure_test` replaces submission, cancellation, alternate-setting, and companion
|
||||
descriptor USB calls with in-process fakes. It runs 60 starts covering failure before any
|
||||
transfer, failure after three queued transfers, and all eight submitted successfully. Each
|
||||
case checks the return code, cancellation count, stopped state, and cleared transfer slots.
|
||||
The Android executable needs `libusb.so` on its loader path; it does not access USB hardware.
|
||||
Building it verifies linking but does not execute the runtime tests. A separate device run
|
||||
is required to confirm them:
|
||||
|
||||
```sh
|
||||
LD_LIBRARY_PATH=. ./uvc_submit_failure_test
|
||||
```
|
||||
|
||||
The executable built under `build/rgb-uvc-reprotest` passed all 60 cases on the Fold7 on
|
||||
2026-10-16. That run used simulated USB calls and does not establish camera-stream reliability.
|
||||