mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-20 21:35:59 +00:00
Merge DroidDeck 0.5.0 into the non-Adreno test branch
Conflicts kept both sides: the 0.5.0 GPU drivers redesign (Auto / pair / mix modes, rollback of a pair that never starts a session) with the test branch's Mali, PowerVR and Xclipse rules (Mali display on the phone's own driver, bridge and system-Vulkan fallbacks, 1-hour list refresh for the open drivers); the test branch's Update button for rebuilt open pairs moved into 0.5.0's new GPU drivers panel; Venus GL probing, the Mali compatibility layer and oom_score_adj redirect kept with 0.5.0's x86-64 Steam client, VAC diagnostics and caller-aware preload hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
66b4fd777c
616 files changed
+83918
-2233
No files matched your search
@@ -47,7 +47,7 @@ shortcut appid is stored per folder at the first scan and never changes after.
|
||||
|
||||
The ✎ on a Custom game's page (`AddedGameEditor`) mirrors Steam's Properties for a
|
||||
non-Steam game: Name, Target (ranked exes, "?", Other exe… opening the picker in the
|
||||
game's folder), Start in (automatic = the exe's folder), Launch options, Artwork (Cover,
|
||||
game's folder), Start in (automatic = the exe's folder), Launch options, Artwork (Cover, Wide cover,
|
||||
Background, Logo, Icon, each with its thumbnail and source), and Remove (two presses; the
|
||||
files stay; adding it again brings it back). `AddedGameEdits` writes each change through
|
||||
the same shortcut: the listing (`name`, `exe`, `dir`, `launch`, `art`, the stored `appid`)
|
||||
@@ -68,7 +68,7 @@ store search whose result has the same name), then SteamGridDB, then the exe's i
|
||||
art chooser (an Artwork row) shows the game folder's images, Steam's official piece and
|
||||
SteamGridDB's best few (fetched when it opens); a tap applies one, "Search SteamGridDB…"
|
||||
opens a search (it starts with the game's name and opens the match at once when one is
|
||||
exact; "Show more" pages through the images), "Pick image…" takes a file through the same
|
||||
exact; it lists every image SteamGridDB returns for the slot, up to four pages of 50), "Pick image…" takes a file through the same
|
||||
picker (png, jpg, jpeg, webp, ico), "Reset to automatic" drops the choice. A choice is a
|
||||
copy in `files/added-art/<appid>/chosen-<slot>.<ext>`, its source in the preferences.
|
||||
|
||||
@@ -80,6 +80,34 @@ a failed request is never written as "none", a "none" older than the saved key i
|
||||
again, and the chooser always asks again when the user opens it. A background pass stops
|
||||
asking SteamGridDB after its first failure.
|
||||
|
||||
Steam games (installed through Steam, not Custom) get the same ✎ on their page: an Artwork-only
|
||||
editor (`SteamGameArtEditor`) with the same rows and the same chooser (Steam's official piece,
|
||||
SteamGridDB's best few, "Search SteamGridDB…", "Pick image…", "Reset to automatic"), shared
|
||||
through `ArtTarget`. A choice is a copy in `files/steam-art/<appid>/chosen-<slot>.<ext>`
|
||||
(`SteamGameArt`); `Library.steamGames` shows it in place of the librarycache image, and with a
|
||||
client running the Cover / Wide cover / Background / Logo is also set in Steam
|
||||
(`SetCustomArtworkForApp` / `ClearCustomArtworkForApp`). Icon is launcher-only.
|
||||
|
||||
`SteamGrid` keeps both sides in step through the client's grid folder
|
||||
(`userdata/<account>/config/grid`, `<appid>p`, `<appid>` for the wide cover, `_hero`, `_logo`), for Steam and Custom games alike.
|
||||
A choice is written there at once, so it reaches Steam while the client is closed. Every library
|
||||
scan (`Library.launchableGames`) compares the folder with what was last written or read
|
||||
(`grid-<slot>` in the game's art folder): a file Steam changed becomes the choice (source Steam),
|
||||
a file removed in Steam drops it, and files the shortcuts writer placed itself
|
||||
(`.droiddeck-art.json`) are not mistaken for choices. A chosen image is stored as
|
||||
`chosen-<slot>-<time>.<ext>`, so a new choice has a new path and shows at once. Installed GOG, Epic and Amazon games get the same rows through `StoreGameArtEditor`.
|
||||
|
||||
"Adjust logo" (last row of every art editor, `LogoPositionDialog`) shows the game's Background with
|
||||
its Logo on top and lets the logo be pinned to one of Steam's five places (top left / top center /
|
||||
center / bottom left / bottom center) and resized (10-100 %). The value is Steam's own
|
||||
`{"nVersion":1,"logoPosition":{"pinnedPosition","nWidthPct","nHeightPct"}}` (`LogoPosition`). It is
|
||||
kept in the game's art folder (`logo-position`), written to `<appid>.json` in the grid folder (other
|
||||
keys in that file are kept; clearing the position removes the file only when nothing else is in it),
|
||||
and, with the client running, set through `SetCustomLogoPositionForApp` /
|
||||
`ClearCustomLogoPositionForApp`. `SteamGrid.sync` takes over a position changed or removed in Steam
|
||||
the same way it does for images (`grid-json` holds the last seen signature). Reset removes the
|
||||
custom position so Steam's default comes back.
|
||||
|
||||
Setup › Launcher › Games holds the Artwork switch (automatic fetching) and the
|
||||
SteamGridDB API key: the user's own, sealed with `CredentialCipher`, wins over the build's
|
||||
(`BuildConfig.SGDB_API_KEY` from the `SGDB_API_KEY` secret); with neither, SteamGridDB is
|
||||
@@ -89,5 +117,5 @@ skipped.
|
||||
|
||||
`AddedExesTest`, `AddedGameEditsTest` (bulk add and skip, edits to the listing on a stable
|
||||
appid, Steam-side read-back, art choice and reset), `AddedGamesCandidatesTest`,
|
||||
`AddedGameArtLookupTest` (key precedence), `GameIdentifierTest`, `PickListingTest`, and
|
||||
`AddedGameArtLookupTest` (key precedence), `SteamGameArtTest` (Steam game art choice and reset), `SteamGridTest` (grid folder both ways), `GameIdentifierTest`, `PickListingTest`, and
|
||||
`tools/tests/test_steam_shortcuts.py` for the writer.
|
||||
@@ -0,0 +1,130 @@
|
||||
# Astra 2 and VITURE stereo
|
||||
|
||||
## Tested connection
|
||||
|
||||
2026-10-14: REDMAGIC Astra 2, model `NP06J`, Android 16,
|
||||
`RedMagicOS11.5.16_NP06J_GB`, Snapdragon `SM8850P`; VITURE Pro XR
|
||||
(`35ca:101d`), firmware `07.0.03.013_20250422`. The tablet reports a locked
|
||||
bootloader and green verified boot. Tests used the ordinary ADB shell, without root.
|
||||
|
||||
2D negotiates **1920×1080 at 120 Hz**. The glasses accept stereo command
|
||||
`0x0008 [32]` with reply `00 32`, but Android then exposes **only 640×480 at
|
||||
60 Hz**. An accepted USB command is not proof of a working stereo signal.
|
||||
In this state the glasses reject a 2D command (`01 31` or `01 34`) and their
|
||||
mode query returns `01 00`. A physical cable reconnect restored 1920×1080/120
|
||||
with this update; Android then requested USB permission again and control resumed.
|
||||
|
||||
On the fresh attempt, `0x030D [03]` arrived 3.1 seconds after the accepted stereo
|
||||
command. The vendor SDK maps that report to 2D/120 (`0x34`); the subsequent mode
|
||||
query was rejected. The report does not encode a fallback reason or establish a
|
||||
fixed timeout. The earlier command acknowledgement cannot prove stereo retention.
|
||||
|
||||
## What was checked
|
||||
|
||||
| Route | Result on this firmware |
|
||||
| --- | --- |
|
||||
| Pro full SBS command `0x32` | Accepted by MCU; physical output falls to 640×480 |
|
||||
| Android preferred mode 3840×1080/60 | Preference saved; physical supported modes remain 640×480 only; preference cleared after test |
|
||||
| USB source/host → source/device | Request issued; actual port stayed source/host |
|
||||
| USB data disable/enable | Data status changed and was restored; physical display remained 640×480 |
|
||||
| `IUsbManager.resetUsbPort` through ADB | HAL callback reported success (`0`); physical display mode did not change |
|
||||
| External-display power off/reset | Did not recover the signal; glasses still rejected 2D |
|
||||
| Physical cable reconnect | Restored 1920×1080/120 and USB controls after the permission prompt |
|
||||
| OEM mirror/3D controls, source review | Change virtual-display routing, not physical stereo timings |
|
||||
| OEM resolution selection, source review | Chooses cached resolution candidates and prefers stale 16:9 mode |
|
||||
| Pro startup mode / half SBS, SDK review | No supported Pro startup-mode setting or half-SBS input |
|
||||
|
||||
Public driver decompilations from the related REDMAGIC 11 Pro (`NX809J`, SM8850)
|
||||
[retain an active mode override](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube/blob/efee6f411184886b0fbdaf63fc9db36b3ed96036/decompiled/msm_drm/dp_display_get_modes.c#L37-L44)
|
||||
across a glasses hotplug. Their [USB-PD disconnect path](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube/blob/efee6f411184886b0fbdaf63fc9db36b3ed96036/decompiled/msm_drm/dp_display_usbpd_disconnect_cb.c#L127-L147)
|
||||
clears that override; their simulated hotplug path does not. This is consistent
|
||||
with the Astra's measured behavior, but does not prove its exact kernel
|
||||
implementation. That driver also [validates saved user modes against EDID](https://github.com/Coding-BR/android_kernel_nubia_sm8850_qwjujube/blob/efee6f411184886b0fbdaf63fc9db36b3ed96036/decompiled/msm_drm/nubia_edid_modes.c#L226-L250),
|
||||
so active overrides and saved preferences must not be treated as the same
|
||||
mechanism. The independent [MagicDesk investigation](https://github.com/mekhontsev/magicdesk/blob/main/docs/xr-resolution-fix.md)
|
||||
documents a related failure on its separately identified REDMAGIC firmware.
|
||||
|
||||
The [VITURE SDK matrix](https://www.viture.com/en-SG/developer/glasses-sdk/glasses)
|
||||
specifies full SBS at 3840×1080/60 for Pro. Persistent startup mode is restricted
|
||||
to Luma/Luma Pro. Among the Luma families supported here, 90 Hz stereo is an
|
||||
Ultra capability. The connected Pro firmware matches both the N6PL entry in the
|
||||
[public updater catalog](https://static.viture.com/dfu-util/js/vers.js) and the newest
|
||||
N6P entry in the [public release-note service](https://cloud.viture.dev/api/v1/system/releasenote)
|
||||
checked on 2026-10-14. This does not rule out unpublished, device-specific updates.
|
||||
|
||||
## Powered dock results
|
||||
|
||||
A VITURE Pro Mobile Dock (`35ca:3011`) was connected between the Astra and Pro XR.
|
||||
Its official read-only P9 version query returned `2.01.03.0.03` with a valid
|
||||
checksum. The Astra negotiated 1920×1080/120, but enumerated only the dock's
|
||||
USB interfaces, not the glasses' `35ca:101d` control/IMU interfaces. A one-second
|
||||
DroidDeck IMU sample returned zero packets.
|
||||
|
||||
DroidDeck treats the glasses as a display behind the dock: no USB permission
|
||||
prompt, video at 1080p/120 on the glasses, no Play SteamVR, refresh, 3D or tracking
|
||||
settings. The dock alone, without glasses, does not count as glasses.
|
||||
|
||||
VITURE's updater calls this dock P9 (app `35ca:3011`, bootloader `35ca:3010`). Its
|
||||
HID messages cover dock versions, battery, inputs and LEDs; none sets a display
|
||||
mode or forwards glasses commands or IMU data. `2.01.03.0.03` is public release
|
||||
`1.03.0.03_20250709`; the latest, `1.03.0.06_20260409`, adds Beast support and
|
||||
updater fixes only. The updater also reads a Lontium LT8668 in the dock, a
|
||||
DisplayPort receiver with its own EDID handling, so the tablet most likely sees the
|
||||
dock as its DisplayPort sink rather than the glasses.
|
||||
|
||||
[VITURE's Pro Dock specifications](https://www.viture.com/product/viture-pro-mobile-dock)
|
||||
confirm 3840×1080/60 video support and exclude phone features requiring
|
||||
communication with the glasses. The vendor SDK also treats the dock as a separate
|
||||
accessory, not an IMU device. This dock may test the video path, but it does not
|
||||
provide DroidDeck's current head-tracking path. Stereo negotiation and mode
|
||||
retention during an upstream-only reconnect still require a physical test.
|
||||
|
||||
## No-root hardware design to test
|
||||
|
||||
A powered pass-through connection may let the tablet see the stereo EDID on a
|
||||
fresh USB-PD connection while the glasses retain their stereo mode:
|
||||
|
||||
1. Power the adapter and glasses independently of the tablet connection.
|
||||
2. Enable stereo on the glasses.
|
||||
3. Disconnect only the tablet's upstream USB-C connection promptly after the
|
||||
stereo acknowledgement, keeping the glasses powered. Compare with a delayed
|
||||
disconnect to test whether the observed mode report affects retention.
|
||||
4. Reconnect the tablet and check its physical supported modes before starting a game.
|
||||
|
||||
This requires an adapter that preserves glasses power and mode during the
|
||||
upstream disconnect, passes 3840×1080/60 video, and forwards USB HID for controls
|
||||
and head tracking. [VITURE's Adapter Ultra documentation](https://www.viture.com/academy/adapters/usb-c-xr-charging-adapter-ultra)
|
||||
explicitly excludes wired glasses data on Android, so it does not satisfy
|
||||
DroidDeck's current USB control/tracking path. Its power and video specifications
|
||||
also do not promise mode retention across upstream disconnect. This design
|
||||
remains **unverified**, not a purchasing recommendation. First verifying stereo
|
||||
retention through the adapter on a compatible host would provide a useful control.
|
||||
An intermediary presenting the stereo EDID before first host
|
||||
connection is another possible design; it must pass the actual stereo timing
|
||||
and preserve HID access, not merely advertise a wider desktop.
|
||||
|
||||
VITURE advertises full Android functionality for its separate
|
||||
[USB-C to Glasses and Charging Adapter](https://www.viture.com/en-MY/product/usb-c-to-glasses-and-charging-adapter).
|
||||
That makes it a candidate for the USB-data experiment, not a confirmed fix.
|
||||
First check USB enumeration, brightness controls and live IMU data through the
|
||||
adapter; then test power/mode retention and the stereo timing.
|
||||
|
||||
Success requires Android to expose and select 3840×1080/60, different left/right
|
||||
eye images to reach the lenses, live head tracking, and repeatable return to 2D.
|
||||
A wide screenshot or `wm size` override alone does not meet those conditions.
|
||||
|
||||
## App safeguards
|
||||
|
||||
DroidDeck waits up to 20 seconds for physical stereo confirmation. When the
|
||||
attempt ends at a low-resolution fallback, DroidDeck remembers it for that glasses
|
||||
model on this device, so Play SteamVR stops requesting 3D after a restart; **Retry** in the
|
||||
glasses settings clears it. A display that stays missing, such as an unanswered
|
||||
cast prompt, fails only that attempt. Automatic retries do not loop.
|
||||
USB mode reports are tracked separately from the physical display. When a session
|
||||
enabled 3D, stopping it requests a return to 2D, including during negotiation;
|
||||
this firmware can still require a cable reconnect. Output timing follows
|
||||
the active glasses display rather than the tablet's requested refresh rate.
|
||||
|
||||
Keep hardware experiments separate from game loads. End the session, force-stop
|
||||
DroidDeck, and reopen its home screen after each game test. Read current HAL
|
||||
temperatures rather than cached thermal-service samples.
|
||||
@@ -44,12 +44,16 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- `helloxr` blocked in a wait-idle in `vrclient.so`;
|
||||
- DXVK's `dxvk-queue` thread blocked in Proton games.
|
||||
|
||||
The layer now gives a submit's wait on an imported shared value 3 ms, then drops it. It is kept
|
||||
when the value is already reached, or when this process has queued the signal itself. Two details
|
||||
mattered:
|
||||
The layer now gives a submit's wait on an imported shared value 3 ms, then drops it, at once when
|
||||
the shared counter has not moved for 100 ms. It is kept when the value is already reached, when
|
||||
this process has queued the signal itself, or while a Quest session's frame socket exists (checked
|
||||
at most once a second). Three details mattered:
|
||||
- the waits are judged before the submit's own signals count, because an acquire waits on one
|
||||
value and signals the next on the same semaphore;
|
||||
- SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there.
|
||||
- SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there;
|
||||
- a Proton game waits on two values a frame that are never signalled (5,500 dropped, none reached
|
||||
within 200 ms afterwards). With the grace on each, Angry Birds VR's submit thread lost 6 ms a
|
||||
frame and the game ran at 60 fps; without it, 120.
|
||||
4. **UBWC layouts must agree.** An imported image has to be created with the same
|
||||
`VkImageFormatListCreateInfo` (sRGB + UNORM) as its creator, or Turnip lays out the compression
|
||||
differently and the reader sees garbage. The presenter's eye-texture import does this.
|
||||
@@ -83,10 +87,74 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- **Memory.** Steam, SteamVR and one VR game fit in the Thor's 12 GB. A large game such as The Lab, or
|
||||
two games at once, push the device into critical pressure, and the low-memory killer takes
|
||||
DroidDeck itself (`ended-without-teardown.txt` in the session logs, `lowmemorykiller` in logcat).
|
||||
- **Adreno 840 forced sysmem.** On the Fold with VITURE Luma Ultra, launching SteamVR from the library
|
||||
inherited `TU_DEBUG=sysmem` and produced `vkerror=-4`, triggering recovery. Adding `noubwc` still
|
||||
failed; replacing the flags with `noubwc` or an empty value worked. The shared launch helper removes
|
||||
only `sysmem` on `Adreno840*`, leaving other flags, the parent Steam session and other GPUs unchanged.
|
||||
The user confirmed improved landing-area brightness and colors with this configuration and asked
|
||||
to preserve them.
|
||||
- **Adreno 840 with DD-Turnip v0.2.0.** SteamVR reproduced `vkerror=-4` with `sysmem` removed,
|
||||
including on the previous app build and with the presenter disabled. After a phone reboot,
|
||||
the unchanged `fcf8c588` baseline rendered for 109 seconds without observed device loss.
|
||||
`TU_DEBUG=flushall` initially appeared promising: a global run rendered for over a minute,
|
||||
and a SteamVR-only run rendered for about 32 seconds before background suspension. The
|
||||
packaged SteamVR-only change then reproduced device loss with `flushall` confirmed in
|
||||
`vrcompositor`, so the automatic flag was removed. These bounded runs do not establish the
|
||||
underlying cause or a reliable workaround.
|
||||
- **Live files are not cache.** Android's `installd` evicted `cache/shm/u10528-*` while SteamVR was
|
||||
running, followed by Chromium shared-memory failures and lost IPC. Shared memory, emulated device
|
||||
files, sandbox trees and PRoot temporary files now use `no_backup/guest-runtime`, retaining their
|
||||
existing cleanup lifetimes. Session state must survive cache eviction.
|
||||
- **One host signal owner.** The submit-wait path must not mirror a shared timeline value itself:
|
||||
the poller can signal it between the submit thread's counter read and signal, producing an invalid
|
||||
duplicate signal. The poller now owns mirroring; a forced-interleaving test covers this race.
|
||||
- **Lost hand tracking is not a disconnect.** Publishing a disconnected pose and then a connected,
|
||||
untracked pose on every optical update floods SteamVR with device-change notifications. With
|
||||
uncalibrated hands this grew its web buffer to about 34 MB and triggered the Web thread watchdog.
|
||||
Release inputs and publish only the connected, untracked pose; reserve disconnection for an
|
||||
actual controller removal. The controller regression exercises repeated loss and reacquisition.
|
||||
- **Failed realtime scheduling can still install a CPU limit.** On the Fold, SteamVR's SDL RTKit
|
||||
fallback set `RLIMIT_RTTIME` to 100/200 ms despite realtime priority being refused. The render
|
||||
thread later received kernel `SIGXCPU` during `IOCTL_KGSL_GPU_COMMAND`, with policy `SCHED_OTHER`
|
||||
and the soft limit advanced to 1100 ms, confirming the realtime watchdog fired. The VR scheduling
|
||||
preload rejects lowering this limit only in `vrcompositor` when `RLIMIT_RTPRIO` is zero and
|
||||
`CAP_SYS_NICE` is absent. It preserves inherited limits, other processes and other resources;
|
||||
it does not intercept signals. SDL's scheduling hint cannot cover SteamVR's direct
|
||||
`SDL_LinuxSetThreadPriorityAndPolicy` call. A native preload regression covers rejection and
|
||||
forwarding. The exact Android kernel boost/accounting path was not established.
|
||||
- **Frames shown out of order.** Video from the glasses jumped back a few frames in bursts while the
|
||||
head turned. Two causes, each found by stamping a frame counter into the presenter's output:
|
||||
- the presenter copied SteamVR's eye texture a fixed 4 ms after `Present`, with no fence. Under
|
||||
GPU load `vrcompositor` had not drawn it yet, so the copy got that texture's previous frame. The
|
||||
layer now fences `vrcompositor`'s submits and the presenter waits for the frame's count;
|
||||
- DroidDeck's compositor copies a Wayland client's dma-buf without waiting for its implicit fence,
|
||||
and KGSL does not order the copy after the client's GPU work. The presenter's blit sat behind the
|
||||
game's frame, so the compositor showed the swapchain image's older picture. The presenter now
|
||||
waits for its blit before it presents.
|
||||
- **Flat games under SteamVR.** With SteamVR running, Steam lists a flat game after its own window in
|
||||
gamescope's base layer, so Big Picture keeps the focus and the game never shows, theatre setting or
|
||||
not. SteamVR is therefore started only by Play SteamVR, never with the glasses alone.
|
||||
- **A new projection needs an event.** `SetDisplayProjectionRaw` changed what `vrserver` reports, but
|
||||
`vrcompositor` kept the frustum it started with, so the glasses' field of view did nothing until
|
||||
SteamVR restarted. The driver now follows it with `VREvent_LensDistortionChanged`.
|
||||
- **Head-tracking axes.** With the display at `ROTATION_90`, screen x is minus device y and screen y is
|
||||
device x. Getting this wrong made a turn at the hips tilt the view. Recentring now keeps the world
|
||||
vertical and resets the heading only.
|
||||
|
||||
## Library launch validation (2026-10-15)
|
||||
|
||||
Two hands-off launches through the Steam library's Launch button completed 308 seconds and
|
||||
188 seconds, with an intentional stop between runs. Each retained its compositor process and
|
||||
session, sustained 60 fps, showed dashboard/world transitions, and kept USB control and valid
|
||||
6DoF head tracking. Neither run reported device loss, a watchdog abort, recovery, or a session
|
||||
restart. Hand controllers remained enabled but uncalibrated. Brightness stayed at 8; no shader
|
||||
color changes or temporary graphics overrides were used.
|
||||
|
||||
These timed runs used APK SHA-256
|
||||
`4ce7a623ceb6b1a100c0ed1184530ece2ea8e13a1cb9ee6c0ab24666a8249004`. A newer calibration/recenter
|
||||
APK was subsequently installed and verified; see `luma-spatial-evidence.md`. The timed results
|
||||
do not establish sustained gameplay or calibrated-hand acceptance for that newer build.
|
||||
|
||||
## Debugging a live session
|
||||
|
||||
- **Logs:** `~/.local/share/Steam/logs/{vrserver,vrcompositor,xrclient_*}.txt` and
|
||||
@@ -123,8 +191,9 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
|
||||
- **SteamVR Home** needs `IVROverlayView_002` in Proton's `vrclient`, or an ARM64 build of Home.
|
||||
- **The Lab and other large games** need memory headroom. Candidates: stop Steam's own web helpers
|
||||
while a VR game runs, or lower SteamVR's footprint.
|
||||
- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync. A submit thread in the
|
||||
layer, deferring submits until their waits are reachable, would be exact.
|
||||
- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync, and an idle counter
|
||||
skips it. A submit thread in the layer, deferring submits until their waits are reachable, would be
|
||||
exact.
|
||||
- **Layers:** only the first layer of each frame is presented. No reprojection.
|
||||
- **Controllers** are a gaze pointer driven by the pad. Pad presses did not close the SteamVR menu in
|
||||
testing. Untested: whether that came from gamescope focus being on Steam, or from the system button
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# Tracked DroidDeck menus
|
||||
|
||||
When glasses output and the tracked virtual screen are enabled, DroidDeck mirrors its own foreground menus into a glasses `Presentation`. Phone controls remain on the phone. The menu renderer does not start the guest compositor, request MediaProjection, or capture another application's windows.
|
||||
|
||||
`GlassesMenuCapture` draws the activity's visible view roots into a hardware Canvas at up to 30 fps. `GlassesMenuSurface` consumes that buffer as an external-OES texture and redraws the screen at the glasses refresh rate. The plane geometry follows the existing `cardboard.frag`: row-major head-to-anchor rotation, metres for translation, +Y up, -Z forward, and a top-left source image. Mono uses a central eye; SBS renders the full UI to both eyes with 64 mm eye separation.
|
||||
|
||||
The pose adapter keeps its 3DoF reference across menu pages and uses the shared spatial **screen** origin for 6DoF. Menu recenter does not change the VR floor origin. Stale or invalid tracking holds the last safe view. Explicitly changing between 3DoF and 6DoF can reanchor the screen because those modes have separate origins; recenter restores the chosen mode's view.
|
||||
|
||||
On Android 29 and later, capture includes visible app dialog/popup roots exposed by `WindowInspector`. Android 26–28 captures the activity decor only, so separate dialog and popup windows are not mirrored. System keyboards, system permission prompts, other apps, and separately composited `SurfaceView` content are outside this capture path. Secure windows are excluded. Phone interaction remains available for these cases.
|
||||
|
||||
## Shader regression check
|
||||
|
||||
The opt-in Python check compiles the actual embedded GLES shaders, imports an asymmetric texture through EGLImage/external-OES, renders it, and checks readback pixels. It covers image orientation, both SBS eyes, translation, yaw, and views behind the screen. It does not require Python packages beyond the standard library and is not part of default pytest discovery.
|
||||
|
||||
On Debian/Ubuntu with Python 3:
|
||||
|
||||
```sh
|
||||
sudo apt-get install libegl1 libegl-mesa0 libgles2 libgl1-mesa-dri
|
||||
python3 tools/check_glasses_menu_shader.py
|
||||
```
|
||||
|
||||
Run from any working directory by specifying the script's path. `--output-dir PATH` overrides the default `build/glasses-menu-shader`. The check defaults to headless Mesa software rendering and GLES 2.0. It writes `menu-shader-validation.json` with the shader hashes, GL version, and pixel comparisons, plus `menu-shader-mono.png` and `.ppm`.
|
||||
|
||||
The test supplies the Android consumer's vertical-flip texture transform. It verifies the real shader under that transform; it does not replace checking Android's actual Canvas → SurfaceTexture producer/consumer integration or display orientation on the phone.
|
||||
|
||||
## Phone and glasses smoke test
|
||||
|
||||
1. Connect supported glasses, open DroidDeck's setup page, and enable glasses output and the tracked virtual screen. Confirm the phone remains touchable and text in the glasses is upright with the same left/right ordering.
|
||||
2. Switch among Steam and non-Steam launcher pages without launching a game. Scroll, open menus and app dialogs, and change a setting from the phone. Check that the glasses follow the current page and retain the screen's head-tracked anchor.
|
||||
3. In 3DoF, turn and tilt the glasses, then use Recenter. Test mono and SBS. Each SBS eye must contain the full UI with matching vertical alignment.
|
||||
4. Enable 6DoF on supported hardware and complete its setup. Translate left/right, up/down, and toward/away from the plane. Recenter the menu and verify that VR floor height has not changed. Briefly obscure tracking; the held menu must remain finite and stable until tracking recovers.
|
||||
5. Launch a game, open and close the session menu, then return to the launcher. Confirm clean ownership handoff between the game output and menu presentation and that phone-only operation still works.
|
||||
6. Background/foreground DroidDeck, rotate the phone, change the glasses display mode, and unplug/replug the glasses. Check for stale frames, black output that persists after reopening, crashes, or a reused orientation from the previous device connection.
|
||||
|
||||
For evidence, `tools/droiddeckctl displays` lists logical display IDs and `tools/droiddeckctl screenshot ./glasses.png --display <id>` captures the selected output. Inspect `GlassesMenu` log messages for capture or renderer failures. A display screenshot verifies image content; head movement and stereo comfort still require the physical glasses.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Hybrid hand inference benchmark
|
||||
|
||||
Measured on the connected Samsung SM-F976U on 2026-10-17. This sparse-correspondence
|
||||
prototype did not improve useful hand throughput. It was slower than scheduled stereo
|
||||
on the central open-hand capture and accepted palms only on its dual-detector recovery
|
||||
frames. The earlier estimated 20–60% throughput improvement is not supported by this test.
|
||||
|
||||
Turning image lens correction off did not improve accepted-palm counts. The saved
|
||||
calibration uses the **pinhole** distortion model, not OpenCV's fisheye model. The benchmark did not change its diagnostic build's live algorithm or input-rate policy,
|
||||
and restored the prior settings after each sweep. That build ended with Lite selected.
|
||||
The shipping baseline was subsequently set to Full with raw grayscale-to-RGB input.
|
||||
The failed hybrid prototype and its runners have been removed; these results are historical.
|
||||
|
||||
## Central open-hand result
|
||||
|
||||
The table pools the initial and final central-scene runs in the lens on/off sweep.
|
||||
Each stereo entry combines the baselines before and after the hybrids. Times include
|
||||
per-frame image preparation, inference, matching when applicable, reconstruction, and
|
||||
palm/gesture estimation. Acceptance means at least one geometrically accepted palm in
|
||||
a repeated saved image, not live detection accuracy. This scene contains two open hands;
|
||||
Full consistently recovered only one.
|
||||
|
||||
| Full-model path | Image lens correction | Median / p95 processing | Frames with a palm | Sparse frames with a palm |
|
||||
|---|---|---:|---:|---:|
|
||||
| Scheduled stereo | On | 30.15 / 61.60 ms | 160 / 160 | — |
|
||||
| Left-led hybrid | On | 62.59 / 84.06 ms | 48 / 80 | 0 / 32 |
|
||||
| Right-led hybrid | On | 40.58 / 80.21 ms | 48 / 80 | 0 / 32 |
|
||||
| Scheduled stereo | Off | 30.51 / 61.18 ms | 160 / 160 | — |
|
||||
| Left-led hybrid | Off | 62.57 / 79.82 ms | 48 / 80 | 0 / 32 |
|
||||
| Right-led hybrid | Off | 61.75 / 83.05 ms | 48 / 80 | 0 / 32 |
|
||||
|
||||
All successful hybrid frames came from ordinary stereo inference. Correspondence failed
|
||||
to reconstruct the five required palm anchors; the original palm gates were retained.
|
||||
The first corrected-image trial measured about 19–21 ms for rectification and matching
|
||||
alone. Repeated failures also prevented the hybrid from sustaining the cheaper one-hand
|
||||
tracker schedule. The live input limit should not be raised on the strength of this candidate.
|
||||
|
||||
Lite accepted no palms with either lens setting or either architecture on the lens-sweep
|
||||
captures. The peripheral open hand and representative controller grip also yielded no
|
||||
accepted palms. The no-hand control remained negative. The earlier corrected-image sweep
|
||||
covered all nine saved pairs, including six controller grips, with no extra palms recovered
|
||||
by the hybrid. These are a small set of static scenes, not independent accuracy trials.
|
||||
|
||||
## What was measured
|
||||
|
||||
`HybridHandBenchmark` was reachable only through the saved-capture diagnostic. Both models
|
||||
use CPU VIDEO inference, the existing 0.6 thresholds, and the existing one-/two-hand
|
||||
schedule. Stereo uses concurrent camera inference. The hybrid runs one detector, rectifies
|
||||
both images, and searches for each landmark along its epipolar line using 11 × 11 normalized
|
||||
correlation patches. It requires correlation at least 0.80, a 0.04 uniqueness margin outside
|
||||
three pixels, patch standard deviation at least four, and forward/backward agreement within
|
||||
1.5 pixels. Search depth spans 0.15–1.5 m. Existing triangulation and palm/gesture checks
|
||||
remain in force. This is one concrete hybrid candidate, not a test of every stereo matcher.
|
||||
|
||||
Both detectors run every fifth frame and on the frame after failed sparse palm estimation.
|
||||
The hybrid's recovery costs and failures are included in the results. Each configuration
|
||||
has a separate cold frame, five warmups, and forty measured frames. Synthetic VIDEO
|
||||
timestamps advance by 50 ms, but processing runs unpaced; the experiment did not measure
|
||||
a live 20 Hz stream. Initialization, file decoding, camera transfer, head synchronization,
|
||||
pose publication, and rendering are excluded. Replayed poses are never published.
|
||||
|
||||
Contrast normalization and grayscale-to-RGB conversion stay enabled in both lens cases.
|
||||
With input lens correction off, detection runs on raw images; landmarks are still corrected
|
||||
using calibration for triangulation. The hybrid always uses corrected, rectified images
|
||||
for correspondence and transforms matched points back to the detector's image coordinates.
|
||||
This comparison does not disable the geometric calibration required for metric depth.
|
||||
|
||||
The lens sweep tested the central scene, peripheral scene, no-hand control, one grip scene,
|
||||
and then the central scene again. Full and Lite each ran stereo-before, left-led hybrid,
|
||||
right-led hybrid, and stereo-after for both lens settings: 3,200 measured replays. The
|
||||
separate initial corrected-image corpus sweep also recorded 3,200 replays across nine
|
||||
unique pairs, repeating the central scene last. Repetition is for timing, not extra accuracy
|
||||
samples. SteamVR was idle and live tracking was temporarily disabled during these runs.
|
||||
|
||||
## Validation and limits
|
||||
|
||||
The Android build and 33 focused JVM tests passed. On the phone, the matcher passed fifty
|
||||
synthetic correspondences in both directions and rejected textureless synthetic images
|
||||
before each capture benchmark. These checks establish basic coordinate/search behavior;
|
||||
they do not establish matching accuracy on real hands.
|
||||
|
||||
Android thermal status was 0–1 during the corrected-image sweep and 1 during the lens
|
||||
sweep. Timings varied substantially within the fixed configuration order, so the on/off
|
||||
timing difference cannot be attributed solely to lens correction. Full stereo medians in
|
||||
the first lens-sweep central trial ranged from 16.7 to 30.8 ms; the final trial was slower
|
||||
with correction enabled. Baselines bracket the hybrid cases, but the run is not randomized
|
||||
or temperature-controlled.
|
||||
|
||||
The initial corpus's final Full/stereo-before trial included a 2,611.74 ms inference stall.
|
||||
It remains in the raw samples and means; p95 alone hides that isolated stall. Geometric
|
||||
acceptance does not apply live capture-age gates, so that observation would be stale in
|
||||
live tracking. The later central lens-sweep samples had no processing times above 200 ms.
|
||||
No moving-hand, pinch/release, sustained SteamVR throughput, external latency, or independent
|
||||
position-accuracy pass is claimed. The present matcher needs better correspondence and
|
||||
lower cost before a live increased-input-rate trial is justified.
|
||||
|
||||
## Reproduction and evidence
|
||||
|
||||
The temporary runner and matcher are no longer shipped. Local evidence from those runs:
|
||||
|
||||
- Final lens comparison and raw samples: `build/hand-hybrid/lens/` and its `summary.json`.
|
||||
- Initial full corpus: `build/hand-hybrid/corpus/` and its `summary.json`.
|
||||
- First prototype trial: `build/hand-hybrid/first/`.
|
||||
- Build and focused tests: `build/hand-hybrid-validation.log`.
|
||||
- Lens comparison build: `build/hand-hybrid-lens-build.log`.
|
||||
- Installed lens-benchmark APK SHA-256: `d976329b96e6aaceec62ef38085e203942cf763c51102adfeb50c4e5614891f1`.
|
||||
- Calibration SHA-256: `3983438843877a321d3098f73496dfd9256f0088f095aa8d591602cf0b2cb213`.
|
||||
|
||||
Each result also includes both capture-image hashes and the model hashes. Build-directory
|
||||
evidence is local and may not be present in a fresh checkout.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Full versus Lite hand models and image preprocessing
|
||||
|
||||
The saved-image comparison favors the Full palm detector and Full landmark model.
|
||||
Neither contrast normalization nor lens correction improved usable bare-hand detections
|
||||
on the two available bare-hand scenes. This is a small static corpus, not a live tracking pass.
|
||||
|
||||
## Method
|
||||
|
||||
The same saved 640 × 480 grayscale stereo pairs were evaluated with eight configurations:
|
||||
Full or Lite palm/landmark bundles, each with raw input, calibrated lens correction,
|
||||
1st/99th-percentile contrast normalization, or both corrections. Grayscale-to-RGB replication
|
||||
and MediaPipe's model-required tensor scaling were enabled in every configuration.
|
||||
|
||||
The main comparison ran on the Samsung SM-F976U using the app's Android MediaPipe Tasks
|
||||
1.0.0 and OpenCV 4.13.0. Each configuration created fresh CPU VIDEO detectors, with two
|
||||
hands per camera, parallel camera inference, and all detection/presence/tracking thresholds
|
||||
at 0.6. Adaptive input and scheduled one-hand inference were excluded. Each pair received
|
||||
one cold inference, four warmups, and twelve measured repetitions at synthetic 80 ms
|
||||
timestamp intervals. Bare-hand mode was verified; wrist-only mode was false.
|
||||
|
||||
This was an offline replay with the game idle and tracking cameras disconnected. It loaded
|
||||
the phone's saved verified calibration and never initialized cameras or published replay
|
||||
poses. Raw-image landmarks were undistorted during triangulation; corrected-image landmarks
|
||||
were triangulated without applying distortion correction again. Both paths retained the
|
||||
existing handedness, depth, reprojection, palm geometry, and motion checks.
|
||||
|
||||
The corpus contains two bare-hand scenes, one no-hand control, and six controller-grip
|
||||
scenes. The central bare-hand scene was also repeated at the end of the phone run. Replaying
|
||||
one pair twelve times does not create twelve independent accuracy samples. Desktop IMAGE
|
||||
and VIDEO tests using MediaPipe 0.10.21/OpenCV 4.11.0 provide a separate cross-check.
|
||||
|
||||
## Bare-hand result
|
||||
|
||||
For capture `64390821246320`, two open hands are visible in both cameras:
|
||||
|
||||
| Image preprocessing | Full: usable stereo palms per replay | Lite: usable stereo palms per replay |
|
||||
|---|---:|---:|
|
||||
| Raw | 1 of 2 | 0 of 2 |
|
||||
| Lens correction only | 1 of 2 | 0 of 2 |
|
||||
| Contrast normalization only | 1 of 2 | 0 of 2 |
|
||||
| Lens correction + contrast normalization | 1 of 2 | 0 of 2 |
|
||||
|
||||
The cold frame and all twelve measured frames agreed. Full found one hand in each eye;
|
||||
Lite found no hands with raw, lens-only, or contrast-only input. With both corrections,
|
||||
Lite found one hand in the left camera and none in the right, so it could not reconstruct
|
||||
a stereo position.
|
||||
|
||||
Capture `72745789186883` contains one open hand, peripheral in the left image. Every
|
||||
configuration missed it in both cameras throughout cold and measured replay. The desktop
|
||||
IMAGE and VIDEO tests agreed with these bare-hand outcomes. The no-hand control
|
||||
`59405518228482` returned zero hands across all configurations.
|
||||
|
||||
The final repeat of the central scene reproduced the same cold and measured counts in
|
||||
every configuration. All six controller-grip pairs (`110854218431877`, `110858058677761`,
|
||||
`110862058272915`, `110935538464606`, `110939418512521`, `110943418461791`) produced
|
||||
zero accepted stereo palms and zero accepted wrists in all measured phone replays. Some
|
||||
Full configurations produced individual camera detections, but these did not become valid
|
||||
stereo positions. Grip scenes are reported separately from the bare-hand comparison.
|
||||
|
||||
## Phone timing
|
||||
|
||||
These are median milliseconds for a parallel stereo pair on the central scene. They include
|
||||
RGB preparation, inference, and result extraction, but exclude decoding, precomputed lens
|
||||
correction, model initialization, and stereo reconstruction. They are not end-to-end latency.
|
||||
|
||||
| Model | Preprocessing | First run | Final repeat |
|
||||
|---|---|---:|---:|
|
||||
| Full | Raw | 46.10 | 82.98 |
|
||||
| Full | Lens only | 75.17 | 84.46 |
|
||||
| Full | Contrast only | 85.64 | 82.99 |
|
||||
| Full | Lens + contrast | 84.10 | 80.49 |
|
||||
| Lite | Raw | 43.07 | 40.06 |
|
||||
| Lite | Lens only | 37.29 | 39.70 |
|
||||
| Lite | Contrast only | 38.73 | 42.54 |
|
||||
| Lite | Lens + contrast | 78.78 | 75.34 |
|
||||
|
||||
Android reported thermal status 2 throughout the saved-pair runs. Configuration order was
|
||||
fixed, and inference work differs when a hand is found. Full/raw's first-versus-last variation
|
||||
also prevents attributing the timing differences to preprocessing alone. Lite's lower timings
|
||||
mostly accompany zero returned hands; its two-camera usable-hand result remains zero.
|
||||
|
||||
## What normalization changes
|
||||
|
||||
The saved calibration uses pinhole radial distortion coefficients (`fisheye: false`). The
|
||||
comparison tests that calibrated lens correction, rather than forcing a fisheye model onto
|
||||
the coefficients. Keeping the original camera matrix and output dimensions narrows the
|
||||
field of view: the left output samples approximately raw x=96–544 and y=56–437, and the
|
||||
right approximately x=93–547 and y=57–440. Some peripheral fingers near raw x=600 disappear
|
||||
from the corrected left image. This is a concrete limitation of the current mapping, although
|
||||
the raw version also failed to detect that hand; the crop alone does not explain all failures.
|
||||
|
||||
Model tensor normalization was already provided by MediaPipe. The additional normalization
|
||||
tested here changes image contrast. Replicating grayscale into RGB satisfies the input channel
|
||||
contract but cannot recover missing color information.
|
||||
|
||||
## Recommendation and limits
|
||||
|
||||
Use Full with raw grayscale replicated into RGB as the next live baseline, retaining MediaPipe's
|
||||
tensor scaling and calibrated point correction for stereo geometry. Full has the stronger
|
||||
detection result here; extra contrast/lens preprocessing has no demonstrated acceptance gain
|
||||
on these scenes. Broader motion testing is still needed before calling any configuration best
|
||||
overall. Full itself still misses one of two central hands and the peripheral hand.
|
||||
|
||||
The comparison used a temporary diagnostic build with opt-in preprocessing and offline replay.
|
||||
That comparison code has been removed after selecting the Full/raw baseline; the live path
|
||||
keeps grayscale-to-RGB conversion and calibrated point correction. Build, twenty focused JVM
|
||||
tests, packaged runtime verification, and the installed Android replay were checked for the experiment.
|
||||
|
||||
Local evidence is under `build/hand-comparison/`: `phone/` contains full per-frame results,
|
||||
thermal snapshots and before/after status; `phone-summary.json` contains compact counts;
|
||||
`desktop/` and `desktop-video/` contain the independent comparisons. The scripts are
|
||||
`compare.py`, `compare_video.py`, `run_phone.py`, and `report_phone.py` in that directory.
|
||||
`input-manifest.json` records all capture, calibration, and model hashes plus runtime versions.
|
||||
@@ -0,0 +1,466 @@
|
||||
# Luma Ultra hand tracking investigation
|
||||
|
||||
Date: 2026-10-17. Status: **experimental; gameplay usability has not passed**.
|
||||
|
||||
## Current Full/raw baseline and hand rendering
|
||||
|
||||
The stereo pipeline defaults to Full palm detection plus Full hand landmarks following the
|
||||
[saved-image comparison](hand-model-comparison-2026-10-16.md). The optional Lite bundle contains
|
||||
Lite palm and Lite landmarks; earlier historical timings describe a different Full-palm/Lite-landmark
|
||||
bundle. `tools/build_hand_lite.py` pins both official Lite source models,
|
||||
checks their input/output tensor contracts, and copies Tasks metadata from the Full bundle.
|
||||
Both the outer task ZIP and the embedded label ZIPs have deterministic timestamps.
|
||||
|
||||
Live inference consumes original grayscale camera bytes, replicated into reusable RGB buffers.
|
||||
It skips full-frame lens remapping, contrast normalization, and the grayscale-to-RGBA-to-Bitmap
|
||||
round trip. Preview bitmaps are created only at preview cadence; grayscale Mats are populated
|
||||
only for calibration, validation, explicit capture, preview, or Joy-Con optical flow.
|
||||
|
||||
Detection, preview overlays, and Joy-Con flow share raw image coordinates. Calibration corrects
|
||||
the sparse landmark coordinates once before triangulation. The saved `fisheye` flag selects
|
||||
fisheye or pinhole correction; the bundled Luma profile uses pinhole coefficients. Calibration
|
||||
fitting, validation, and diagnostic captures retain their original raw images. The unsuccessful hybrid matcher and preprocessing-comparison code were removed
|
||||
after baseline selection. The existing saved-pair diagnostic uses the same raw RGB
|
||||
preprocessing and point correction as live tracking.
|
||||
|
||||
Each grayscale byte is copied into R, G, and B without changing exposure or image coordinates.
|
||||
[MediaPipe Tasks](https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker/android)
|
||||
then performs model-specific tensor normalization from metadata; the app does not divide these
|
||||
image bytes by 255 a second time.
|
||||
|
||||
Joy-Con mode retains only landmark 0 (the wrist), triangulates and follows that position, and
|
||||
does not estimate palm orientation, pinch, or grip. Replay follows the selected controller mode.
|
||||
The landmark network still predicts all 21 landmarks internally. Physical Joy-Con
|
||||
buttons and IMU orientation remain the controller input sources.
|
||||
|
||||
The user reported that the hand smoothing and dead zones made tracking feel worse. The
|
||||
common bare-hand `OpticalPose` path therefore removes the 3 mm position and 1-degree rotation
|
||||
anchors and restores the earlier fixed 15 ms rendering interpolation, replacing the added
|
||||
adaptive smoothing. Small accepted movements can change the rendered pose immediately.
|
||||
Position prediction still requires consecutive velocities to agree in direction, preventing
|
||||
alternating measurement noise from being extrapolated. The normal mode keeps measured palm
|
||||
orientation; only the opt-in steady-aim setting uses `ShoulderAim`. Joy-Con filtering, gesture
|
||||
release, observation freshness, and calibration gates are unchanged.
|
||||
|
||||
SteamVR startup defers Luma Ultra 6DoF until two seconds after the first confirmed RUNNING
|
||||
status. The launch gate applies before SDK initialization, with display controls available
|
||||
through a nonspatial connection during the wait. Repeated RUNNING polls do not extend the
|
||||
deadline. Stop, failed launch, and session cleanup cancel the pending start; reopening always
|
||||
uses the current device and saved tracking settings. This changes startup sequencing, not the
|
||||
tracking preference or hand-model configuration.
|
||||
Automatic restoration on the same device, including a cancelled launch or early stop,
|
||||
requests the existing stable-pose recenter. Status polls from superseded requests or ended
|
||||
sessions cannot restart the pending tracking transition.
|
||||
|
||||
Bare-hand controller rendering holds the last accepted pose for up to 350 ms from its original
|
||||
capture timestamp, extending the previous 200 ms visual limit. Missing, invalid, or stale
|
||||
observations freeze the rendered pose and stop prediction; repeated packets cannot extend the
|
||||
hold. New observations still require the existing 200 ms freshness check, and trigger/grip
|
||||
release immediately when their observations become invalid or stale. Head loss, origin or
|
||||
connection changes, and controller-mode changes clear the visual cache. This bridges short
|
||||
hand-detection gaps without adding a delay to fresh frames; it does not improve detection or
|
||||
preserve a gesture through tracking loss. Both Full and Lite predict the 21 landmarks used by
|
||||
the grab estimator; wrist-only Joy-Con mode uses physical buttons instead.
|
||||
|
||||
Before this rollback, synthetic 12.5 Hz alternating-noise replay reduced position RMS from 4.354 to 1.772 mm and
|
||||
rotation RMS from 1.722° to 1.398°. At 25 Hz, the respective changes were 1.242 to 0.988 mm
|
||||
and 1.058° to 0.800°. The tradeoff in the 12.5 Hz slow-motion case was slightly more lag:
|
||||
maximum position error changed from 0.423 to 0.593 mm and angular error from 0.489° to
|
||||
0.632°. These results describe the reverted adaptive filter, not the current behavior or
|
||||
measured live tracking improvements; the user's subsequent experience favored removing it.
|
||||
The 80 focused JVM tests and native spatial, controller, adaptive-filter, and Joy-Con tests
|
||||
passed. Build and installation evidence is recorded under `build/hand-baseline/`.
|
||||
|
||||
The visual-hold follow-up passed the full native suite, including capture-age boundaries,
|
||||
frozen rendering, stale-packet rejection, immediate gesture release, fresh reacquisition, and
|
||||
cache invalidation. Only the VR driver changed among the 150 staged runtime outputs.
|
||||
Build and installation evidence is recorded under `build/hand-visibility/`.
|
||||
|
||||
The subsequent dead-zone/adaptive-filter rollback passed the full native suite. Regression
|
||||
tests exercise 0.2 mm translation and 0.05-degree rotation immediately after settling, sustained
|
||||
small motion, reversal, and the retained visibility/input protections. Synthetic 95% step
|
||||
response was 44 ms after receipt at both 12.5 and 25 Hz observation rates. This measures the
|
||||
render filter, not end-to-end camera latency. Only the VR driver changed among the 150 runtime
|
||||
outputs; rollback evidence is recorded under `build/hand-filter-revert/`.
|
||||
|
||||
The reviewed startup-delay follow-up passed 129 focused JVM tests, and an isolated snapshot
|
||||
containing only the startup changes passed its 128 focused tests. Earlier phone logs recorded the spatial SDK
|
||||
open attempt 3.348 seconds after confirmed SteamVR RUNNING: the two-second delay plus SDK
|
||||
connection teardown. The resumed session reported valid 6DoF with no pending or required
|
||||
recenter. The native driver and hand configuration were unchanged. Evidence is recorded under
|
||||
`build/luma-vr-start-delay/`. The review fixes were rebuilt and installed, but the phone
|
||||
reported no glasses connected during the follow-up, so hardware cancellation recovery
|
||||
was not rechecked.
|
||||
|
||||
The installed APK SHA-256 is `fac0973a21e41db842d88617bb11f16672987bd0b955736ad510abceca46b00f`.
|
||||
Earlier connected-phone diagnostics confirmed `FULL`, `raw-grayscale-rgb`,
|
||||
`pinhole-points-only`, concurrent CPU inference, scheduled discovery, adaptive input,
|
||||
and `steadyHandAim=false`. Camera/head processing was active with the verified device
|
||||
calibration. The launch check reached READY and staged VR driver SHA-256
|
||||
`e79525c144e934761f08cdc1d3170d058df89bf55c8458ddc955b58c90c9b128`, matching the rebuilt
|
||||
payload. This verifies deployment and live processing, not moving-hand accuracy or comfort.
|
||||
|
||||
The optional Lite bundle can be rebuilt with `mediapipe==0.10.21` using:
|
||||
|
||||
```sh
|
||||
python tools/build_hand_lite.py --palm palm_detection_lite.tflite --lite hand_landmark_lite.tflite
|
||||
```
|
||||
|
||||
Source URLs and hashes are in `app/src/main/assets/models/hand_landmarker_lite.provenance.json`.
|
||||
These changes still need live moving-hand, closed-fist, gesture-continuity, and latency validation.
|
||||
Earlier Lite/normalized-input validation passed 52 focused JVM tests, and bundle rebuild and label-archive reproducibility
|
||||
checks passed. Desktop MediaPipe 0.10.21 inference succeeded on color hand fixtures. All six
|
||||
saved left-camera grayscale grip images still returned zero detections after correction and
|
||||
contrast normalization at the existing 0.6 confidence thresholds. This is not a tracking-accuracy
|
||||
pass. Synthetic projection/undistortion/triangulation checks stayed below 0.12 mm reconstruction
|
||||
error for the bundled pinhole profile and near numerical precision for a synthetic fisheye profile.
|
||||
Local reports for that implementation check are in `build/hand-lite-models/`; Android hardware
|
||||
inference was not run at that stage.
|
||||
|
||||
The subsequent [Full/Lite preprocessing comparison](hand-model-comparison-2026-10-16.md)
|
||||
ran all eight model/preprocessing combinations on the phone using nine saved stereo pairs.
|
||||
Full recovered one of two central open hands; Lite recovered no usable stereo hand. Neither
|
||||
model found the peripheral hand, and contrast/lens correction added no accepted bare hands.
|
||||
Full with raw grayscale-to-RGB input and calibrated point correction is now the live baseline;
|
||||
the earlier comparison diagnostic itself did not change the defaults.
|
||||
|
||||
There is room to improve the current implementation, but the available evidence does not support calling another smoothing adjustment a gameplay fix. The strongest current blockers are rejected palm reconstruction, interrupted gesture continuity, and processing latency under SteamVR load. The next useful milestone is a repeatable open → pinch → release interaction with a stable aiming pose, followed by sustained in-game testing.
|
||||
|
||||
## What the existing evidence establishes
|
||||
|
||||
The latest recorded post-installation interaction is `build/spatial-live-hands-sticky-grab-updated.json`, summarized by `build/sticky-grab-updated.log`. It belongs to Android process 17867 and SteamVR session `2026-10-16-60-steam`, using the stability candidate before the fork migration. It is historical evidence, not a measurement of the changes made during this investigation.
|
||||
|
||||
| Finding | Evidence | Interpretation and limits |
|
||||
|---|---|---|
|
||||
| Head tracking remained available while hands failed | All 2,823 coherent channel samples had fresh head tracking; right hand was fresh in 1,446 samples (51.2%) | The observed interruption is in the hand path. This is sampled channel availability, not accepted-camera-observation rate; the user's exact hand visibility was not recorded. |
|
||||
| Palm acceptance is a major failure point | Five of eight status snapshots reported matched/reconstructed hands but zero accepted palms | Detection alone is insufficient. The old message grouped missing palm anchors, implausible width, degenerate orientation, and temporal outliers together. It does not identify which gate caused each rejection. |
|
||||
| Gesture continuity can prevent clicks entirely | The right-hand trace contained valid open and pressed pinch values, separated by loss transitions | Requiring a fresh observed release after loss is correct, but unreliable tracking can prevent a usable press sequence. See the replay below. |
|
||||
| Processing already consumes most of the latency target | Eight reported processing times ranged from 63.3 to 114.0 ms; observation ages from 63.6 to 155.7 ms | These are internal status samples, not physical-motion-to-display latency. Some exceed the complete external p95 target of 100 ms before rendering is counted. |
|
||||
| The run was thermally constrained | Android thermal status was 4; HAL skin temperature increased from 53.9 to 54.9 °C | This run cannot establish cold performance or isolate thermal throttling from game workload. |
|
||||
| Latest stability changes did not establish usable interaction | The user reported similar stationary jitter and inability to start a dashboard drag | The previous candidate's automated tests and successful installation did not establish a hardware interaction pass. |
|
||||
|
||||
The trace used CPU, full model, concurrent stereo inference, automatic input rate, and scheduled tracking. All eight diagnostic snapshots selected the two-hand search bank and a 500-permille camera copy rate. Approximately 25 Hz camera delivery therefore supplied at most about 12.5 eligible stereo pairs per second before processing and rejection. The left-hand channel had only 14 fresh samples; this was not a controlled two-hand visibility test, so that number is not left-hand detection accuracy.
|
||||
|
||||
### Why a detected pinch may produce no click
|
||||
|
||||
The right-hand transition list spans 15.885 seconds and contains 142 rows. Applying the current trigger rules from `tools/vr/src/spatial_tracking.h` to those sampled rows gives:
|
||||
|
||||
| Trace quantity | Count |
|
||||
|---|---:|
|
||||
| Fresh, valid new observations below the 0.25 release threshold | 31 |
|
||||
| Fresh, valid new observations above the 0.65 press threshold | 12 |
|
||||
| Changes in the right trigger's loss-counter byte | 16 |
|
||||
| Valid-palm transition rows with invalid pinch geometry | 58 |
|
||||
| Above-threshold observations while the replayed trigger remained armed | 0 |
|
||||
|
||||
Replay starts disarmed because the first sampled hand is invalid. It compares the low byte of the right hand's `epochs`, releases on invalid/stale hand or pinch data, ignores duplicate observations for arming, checks the 200 ms gap against the previous accepted hand timestamp, and arms only on a fresh valid trigger value below 0.25. A valid palm with invalid pinch still advances the hand observation timestamp, as in the driver.
|
||||
|
||||
This is an inference from a sampled shared-memory trace. Sampling may omit intermediate observations, and this replay does not record what SteamVR or its dashboard consumed. It nevertheless gives a concrete explanation consistent with the reported inability to start a drag: the recognizer produces high pinch values, but continuity is lost before those values can form an armed press. Removing the release requirement would reintroduce presses on reacquisition and would not repair the observation stream.
|
||||
|
||||
## Prioritized work
|
||||
|
||||
1. **Measure each rejection before changing acceptance.** Count processed observations, valid palm/gesture observations, and rejection reasons separately for each hand. Distinguish no matching pair, missing triangulated palm anchors, palm width, degenerate orientation, implausible motion, synchronization, and age. Keep cumulative counters separate from sparse UI snapshots. These counters need no camera image recording. A broad reduction in stereo-quality thresholds could trade disappearing hands for incorrect depth and jumps.
|
||||
|
||||
2. **Establish one reliable interaction sequence.** With a single visible hand at 30, 50, then 80 cm, repeat open → pinch → hold → release while collecting the observation counters and existing driver input-transition logs. Repeat for the other hand, then with both visible. Record whether failure happens before palm acceptance, at pinch validation, at arming, at OpenVR publication, or inside the dashboard. The dashboard uses trigger for click/drag; the installed Vive binding maps grip to Back. Keep the other fingers open during the pinch test so the two actions remain distinguishable.
|
||||
|
||||
3. **Reduce measured processing cost and discovery spikes.** Run temperature-matched comparisons of full/scheduled CPU, lite/scheduled CPU, and input-rate settings, retaining the same game scene and deliberate motions. Measure completed-observation rate and accepted-observation rate, not camera rate or repeated channel reads. Existing saved-open-palm benchmarks show potential: under SteamVR, scheduled full inference had median/p95 46.92/94.71 ms versus full-search 91.48/99.26 ms; scheduled lite gave 40.26/86.77 ms. Those static-image results cannot establish moving-hand, pinch, or crossed-hand accuracy. The full model should remain the baseline until those comparisons pass. GPU requires a separate stability check because earlier evidence records graphics recovery/device-loss concerns.
|
||||
|
||||
4. **Tune pose and aiming with motion evidence.** Record stationary raw and rendered pose distributions and a repeatable slow aiming path. At the time of this trace, driver stabilization held position within 3 mm and orientation within 1 degree around a stationary anchor, applied a 15 ms rendering correction, and bounded prediction to 2 cm/80 ms. The anchors have since been removed at the user's request. Increasing smoothing without measuring raw noise and lag risks making aiming less responsive. Test motion-aware filtering or a more stable aim direction only against the same recorded motions and external latency measurement. Aiming direction and a game controller's grip pose may need separate treatment.
|
||||
|
||||
5. **Validate game compatibility after continuity.** Optical controllers currently expose Vive-compatible trigger/grip/trackpad inputs, with gamepad sticks and buttons supplying locomotion and menus; there is no skeletal hand output. `CardboardController::UpdateOptical` initializes `DriverPose_t` to zero and populates position and rotation, leaving linear and angular velocity zero. This is a source-confirmed omission to investigate for throwing and other velocity-dependent games, not an established cause of the dashboard problem. A velocity implementation needs capture-time derivatives, outlier/loss resets, a defined pose timestamp, and a game-level test before enabling it.
|
||||
|
||||
Startup should be measured separately from steady state. Earlier logs contained shader-pipeline creation well after the dashboard appeared, while both hand detectors were already initialized. Correlate observation latency and CPU/GPU workload through launch and the following two minutes; model warmup alone does not explain the reported minute-long startup delay.
|
||||
|
||||
The exported calibration also helps explain why a 3 mm stationary hold may have little visible effect. Its focal lengths are approximately 226–241 pixels and its stereo baseline is approximately 14 cm. The ideal rectified-stereo approximation `depth change ≈ depth² / (focal length × baseline) × disparity change` gives roughly 8 mm per disparity pixel at 50 cm and 20 mm at 80 cm. These are sensitivity estimates, not measured hand errors; the actual cameras have lens distortion and relative rotation. Independent landmark jitter can therefore exceed the driver's hold radius. If the new counters confirm palm reconstruction dominates, a robust fit across the available palm landmarks and temporal stereo correspondence are stronger candidates than simply enlarging that hold radius. Such a fit still needs explicit outlier rejection, side identity, capture-time freshness, and measured motion/gesture tests.
|
||||
|
||||
## Vendor runtime options
|
||||
|
||||
The public [VITURE Glasses C SDK documentation](https://www.viture.com/developer/glasses-sdk/glasses) describes USB management, head tracking, display control, and camera streaming, including Android support. It does not document a phone hand-tracking runtime that can replace this application's MediaPipe reconstruction directly. That is a statement about the published API reviewed on this date, not proof that VITURE has no private or forthcoming implementation.
|
||||
|
||||
The [VITURE Unity SDK documentation](https://www.viture.com/developer/unity-sdk/unity) does describe hand tracking, responsive/stable filter modes, and arm/wrist contributions to aiming. Its stated hardware requirement and deployment instructions target the Pro Neckband. Those controls provide useful design references, but the documentation does not establish that their runtime can be redistributed inside this phone app. Confirm phone runtime availability, platform support, and licensing with VITURE before planning that replacement.
|
||||
|
||||
## Calibration and completion criteria
|
||||
|
||||
The phone's current calibration can be shipped as the Luma Ultra starting profile. That reduces setup friction; it does not fix inference cost, hand occlusion, gesture continuity, or jitter by itself. Preserve each device's own calibration as the higher-priority choice, restrict the base profile to its matching model and camera geometry, and identify it as a shared starting calibration. A calibration validated on the source glasses has not been independently measured on every Luma Ultra unit.
|
||||
|
||||
Retain the existing acceptance gates in `luma-spatial-tracking.md`: at least 95% accepted slow unobstructed hand observations at 30–80 cm, no more than 5 cm position error and 2 cm stationary jitter, external p95 latency below 100 ms, reliable release and reacquisition, and no more than 10% regression in p95 game frame time over a 20-minute run. Dashboard selection and Angry Birds VR aim/grip/pull/release still require deliberate physical validation. Keep the feature visibly experimental until these tests pass.
|
||||
|
||||
Supporting records: `luma-spatial-evidence.md`, `build/spatial-live-hands-sticky-grab-updated.json`, `build/hand-reliability-benchmark-smoothness-steamvr.json`, and `build/hand-startup-review.json`. The build-directory records are local historical artifacts and may not accompany a fresh checkout.
|
||||
|
||||
## Joy-Con grip and arm extension investigation (2026-10-16)
|
||||
|
||||
The target is a closed fist holding a controller, with the wrist and forearm visible from the
|
||||
wearer's viewpoint. Open-palm performance is not sufficient evidence for this use case. In a
|
||||
requested 16-second extension exercise, 5 of 80 sampled left positions and 1 of 80 right positions
|
||||
were valid while head tracking stayed valid. These are channel availability samples, not 80
|
||||
independent detector results. The earlier driver also restarted its transition to the arm model
|
||||
on each invalid hand frame, preventing intermittent camera positions from converging.
|
||||
|
||||
The initial grip estimator required the wrist, middle base knuckle, and one supporting base
|
||||
knuckle, rather than all five palm landmarks and a hand-derived orientation. The later
|
||||
wrist-only update below removes the remaining knuckle dependency. It never generates hand
|
||||
gestures; Joy-Con IMU and buttons remain authoritative. Stereo optical flow can carry the
|
||||
identified wrist through brief model misses, with forward/backward consistency, photometric,
|
||||
stereo reprojection and motion checks. Tracks expire 750 ms after the last detector seed; frame
|
||||
gaps over 200 ms discard them. The driver retains a camera observation only until 200 ms after
|
||||
its original capture time. Neither mechanism turns stale coordinates into fresh measurements.
|
||||
|
||||
### Model assessment
|
||||
|
||||
- **MediaPipe Full (historical baseline):** an existing Android pipeline with separate palm
|
||||
detection and landmark regression. Curling fingers does not intrinsically exclude a hand,
|
||||
but occlusion and the egocentric view must be tested with actual grips. Its published
|
||||
monocular world landmarks are not used as metric camera distance; calibrated stereo supplies
|
||||
distance. [Official task documentation](https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker).
|
||||
- **Earlier bundled Lite:** that bundle changed the landmark regressor while retaining the Full
|
||||
palm detector (see `tools/build_hand_lite.py`). It can reduce processing cost, but cannot be
|
||||
assumed to fix the initial fist-detection problem. Earlier saved open-palm replay under
|
||||
SteamVR measured scheduled Full median/p95 46.92/94.71 ms and Lite 40.26/86.77 ms. Those figures
|
||||
do not establish closed-fist accuracy.
|
||||
- **RTMDet-nano hand detector plus RTMPose-m Hand5:** a concrete alternative detector and
|
||||
landmark pipeline worth evaluating against grip frames. The official demo uses separate
|
||||
hand boxes and landmarks. It is not a replacement `.task` asset: Android inference,
|
||||
preprocessing, handedness/identity association, and device timings need integration and
|
||||
verification. No claim of superior Joy-Con performance is established yet.
|
||||
[Official hand demo](https://github.com/open-mmlab/mmpose/blob/main/demo/docs/en/2d_hand_demo.md).
|
||||
- **HaMeR:** useful as an offline reference for occluded and egocentric hands, with published
|
||||
evaluation procedures. Its reference implementation uses PyTorch, ViTPose, and MANO assets;
|
||||
it is not an established real-time Android replacement for this app. Do not substitute its
|
||||
inferred mesh depth for measured stereo depth.
|
||||
[Official implementation](https://github.com/geopavlakos/hamer).
|
||||
- **Whole-body arm pose:** not the first candidate for cameras on the wearer's glasses,
|
||||
where the wearer's shoulders and torso are commonly outside view. An egocentric fist/wrist
|
||||
detector or forearm segmentation model is the more directly matched development target.
|
||||
This is an engineering assessment, not a measured comparison of those models on this device.
|
||||
|
||||
The third camera is a separately exposed UVC pass-through camera, not an unused member of the
|
||||
stereo callback. It may provide better RGB recognition, but requires its own capture/decode and
|
||||
camera-to-head calibration. It does not independently provide metric depth. The preferred use
|
||||
would be recognition or association assistance while stereo supplies distance; size-based
|
||||
monocular distance would need separate calibration and uncertainty handling.
|
||||
[VITURE SDK camera support](https://www.viture.com/developer/glasses-sdk/glasses).
|
||||
|
||||
Saved-grip replay reports `joyCons` acceptance separately from bare-hand `hands`, for both Full
|
||||
and Lite. Repeating one image measures model/geometry behavior on that image, not motion
|
||||
continuity. Live tests must still check reach response, left/right identity, reacquisition,
|
||||
camera loss, and head motion before considering the experiment reliable.
|
||||
|
||||
### Closed-fist model comparison
|
||||
|
||||
Three requested grip pairs were saved locally as captures `110935538464606`, `110939418512521`,
|
||||
and `110943418461791`. In on-phone saved-frame replay of the middle and last pair, Full detected
|
||||
one hand in the left image and none in the right image in all 20 measured repetitions per
|
||||
configuration; Lite detected none in either image. Neither produced a valid grip position.
|
||||
On the middle pair, scheduled Full median/p95 was 56.05/62.96 ms and Lite 46.00/47.99 ms. On
|
||||
the last pair, it was 21.17/23.33 ms and 18.08/20.06 ms respectively. Timing differences between
|
||||
frames are not accuracy improvements. Repeated static frames are not independent observations.
|
||||
|
||||
An isolated Windows evaluator tested the official RTMDet-nano hand / RTMPose-m Hand5 ONNX
|
||||
exports using `rtmlib` 0.0.16 and ONNX Runtime 1.31.0. Across the six raw images, hand-box counts
|
||||
were 0, 0, 1, 0, 2, 1. Lens-corrected images gave 0, 1, 1, 0, 2, 1. Neither variant found both
|
||||
hands in both cameras for any pair. These are box counts, not validated stereo positions or
|
||||
ground-truth accuracy. Desktop timings cannot predict Android cost. Model downloads and hashes
|
||||
are in `build/joycon-alternative-models/provenance.json`; results and overlays are local.
|
||||
|
||||
MediaPipe desktop preprocessing comparisons used the same Full model asset with Python Tasks
|
||||
1.1.1, versus Android Tasks 1.0.0. Overlapping 400-pixel crops recovered some missing detections;
|
||||
lens correction and lower thresholds also changed detections and sometimes handedness. Only
|
||||
isolated stereo matches passed the existing geometry checks. This does not justify enabling
|
||||
either preprocessing change in the live pipeline yet, or replacing Full with Lite/RTMPose.
|
||||
|
||||
The post-install shared-memory sample arrays in `build/joycon-reach-after.json` and
|
||||
`build/joycon-reach-grip.json` are not valid live-pose measurements: the SteamVR session had
|
||||
not restarted, so its pose publisher was inactive. Their before/after provider diagnostics and
|
||||
requested image captures remain useful. During the actual closed-fist interval, all 253
|
||||
processed pairs reported invalid head tracking, so no live reach claim can be made. The local
|
||||
measurement script now requires a running session, valid head tracking, and advancing head
|
||||
timestamps to reject this situation. The image comparisons above do not depend on a live head
|
||||
pose. A new live test is still required after head tracking and the session recover.
|
||||
|
||||
### Third-camera grip comparison
|
||||
|
||||
The separately exposed RGB camera delivered 1920 by 1080 JPEGs after Android Camera and USB
|
||||
permissions were granted. SDK 2.4.0 crashed in `xr_camera_provider_start` when reopening the
|
||||
provider in the main app process. RGB diagnostics now run in a dedicated process that skips
|
||||
normal app initialization. Four initial successful captures preserved the main app PID.
|
||||
Subsequent one-shot attempts intermittently returned zero callbacks despite a successful SDK
|
||||
start. Keeping a stream open between requests did not resolve this: four successive startup
|
||||
attempts returned zero callbacks. USB reconnection and a separate attempt with positional
|
||||
tracking temporarily disabled also returned no frames. The user's decision was to stop the RGB
|
||||
route and compare the models on stereo. The persistent-stream and RGB/stereo calibration-buffer
|
||||
prototypes were removed; the isolated one-shot command remains for manual diagnostics only.
|
||||
|
||||
Three requested normal-grip RGB captures (`112581486574499`, `112588468819444`, and
|
||||
`112595950634337`) were evaluated with the same local model assets and thresholds:
|
||||
|
||||
| Pipeline, full image | First | Second | Third |
|
||||
|---|---:|---:|---:|
|
||||
| MediaPipe Full | 0 hands | 0 hands | 2 hands |
|
||||
| MediaPipe Lite | 0 hands | 0 hands | 1 hand |
|
||||
| RTMDet-nano + RTMPose-m Hand5 | 2 hands | 2 hands | 2 hands |
|
||||
|
||||
Overlapping square crops recovered one grip in the first image with Full and Lite; neither
|
||||
recovered a hand in the second. The no-hands control image produced zero detections for all
|
||||
three pipelines. Visual inspection of the alternative model's overlays found plausible wrist
|
||||
locations on these grips, including the closer, partly cropped second view. Three selected
|
||||
images are not an accuracy dataset or evidence of continuous tracking, identity preservation,
|
||||
or measured reach. RGB results and overlays are in `build/joycon-rgb-evaluation`.
|
||||
|
||||
The unchanged official ONNX models were also replayed on the Fold7 using Android ONNX Runtime
|
||||
1.31.0 and OpenCV 4.13.0 preprocessing. Four warm-ups and ten timed iterations per image gave
|
||||
two-thread median times of 119.90, 119.88, and 129.24 ms; the largest sample was 151.04 ms.
|
||||
Four threads gave medians of 120.45, 120.79, and 121.05 ms. Timings include preprocessing,
|
||||
detection, and two hand landmark passes, but exclude JPEG decoding, camera acquisition,
|
||||
stereo depth, head fusion, rendering, and concurrent SteamVR load. Thus they already exceed
|
||||
the 100 ms end-to-end target. The app's OpenCV DNN importer alone rejected the detector's TopK
|
||||
operator; these measurements used a separate native benchmark, not a new production model
|
||||
dependency. [ONNX Runtime mobile support](https://onnxruntime.ai/docs/get-started/with-mobile.html).
|
||||
|
||||
The RGB SDK timestamps were zero on this device. Diagnostics record host callback receipt
|
||||
separately. A temporary prototype successfully collected one nearby stereo pair with surrounding
|
||||
frames, but the subsequent board attempt received no RGB frame. No RGB calibration was fitted.
|
||||
Existing stereo calibration does not supply RGB lens parameters or the RGB-to-stereo transform.
|
||||
RGB has never been used to publish controller positions in this experiment.
|
||||
|
||||
### Stereo-only follow-up and wrist landmark 0
|
||||
|
||||
The alternative RTMDet/RTMPose pipeline was replayed on the Fold7 against the same three raw
|
||||
stereo grip pairs, without RGB. With live Joy-Con inference temporarily paused, counts were
|
||||
0/0, 1/0, and 2/1 hands in the left/right images. Per-image medians were 45.63/45.42,
|
||||
129.35/45.03, and 199.50/123.13 ms. These are sequential single-image benchmark measurements,
|
||||
not concurrent stereo latency or end-to-end tracking. Both candidate pairings in the last
|
||||
capture failed the current geometry gates. The plausible wrist pair projected with 4.17 px
|
||||
and 3.27 px error, exceeding the 3 px limit; the other pairing reconstructed behind the camera.
|
||||
Neither a valid grip nor a valid isolated wrist was established. The log and geometry report
|
||||
are `build/joycon-stereo-phone-ort-controlled.log` and
|
||||
`build/joycon-stereo-phone-ort-geometry.json`.
|
||||
|
||||
“Full” in this project is the full MediaPipe Hands landmark model, not a full-body model.
|
||||
At the time of the model comparison, the live Joy-Con estimator used the midpoint of wrist 0
|
||||
and middle-finger base 9, validated with an additional base knuckle. The saved-pair diagnostic
|
||||
added `wrists`, testing landmark 0 without those palm anatomy requirements. This cannot
|
||||
recover a wrist when the upstream hand detector misses the grip in either camera.
|
||||
|
||||
The existing desktop Full outputs at the production 0.6 thresholds, using raw uncropped
|
||||
images, gave confident left/right counts of 0/0, 1/0, and 0/0. Thus none offered a wrist
|
||||
pair even before palm geometry checks. Desktop Tasks 1.1.1 static-image evaluation differs
|
||||
from Android Tasks 1.0.0 VIDEO replay and must not be presented as an Android measurement.
|
||||
The local report is `build/joycon-mediapipe-wrist-evaluation.json`. The updated Android
|
||||
wrist-only diagnostic was built and installed, but its replay is pending the glasses USB
|
||||
approval after installation. The last phone settings check showed positional tracking on,
|
||||
Joy-Con tracking off, and hand controllers off; there was no active stereo engine.
|
||||
|
||||
Keep MediaPipe Full as the live baseline. The alternative's better RGB detections did not
|
||||
transfer to dependable stereo grip positions on these captures. No new physical arm-extension
|
||||
validation has been completed.
|
||||
|
||||
### Live wrist-only Joy-Con position
|
||||
|
||||
Following the user's correction, Joy-Con mode triangulates only landmark 0 and publishes its
|
||||
position transformed through the camera-to-head and captured head poses. Knuckle availability,
|
||||
palm proportions, and the wrist/knuckle midpoint no longer affect Joy-Con acceptance or position.
|
||||
The estimator is `JoyConWristPosition`, and diagnostics report `joycon-wrist` with wrist-specific
|
||||
missing-landmark and motion outcomes. Bare-hand gesture tracking still uses its palm geometry.
|
||||
|
||||
Optical flow now seeds only the wrist and accepts a single tracked point per camera; the
|
||||
previous three-point minimum is removed. Forward/backward and photometric checks, stereo
|
||||
depth and reprojection limits, motion rejection, head synchronization, freshness, and track
|
||||
expiry remain in place. Controller rotation and buttons still come from the Joy-Con, and
|
||||
the driver still falls back to the arm model when fresh camera positions are unavailable.
|
||||
This removes the unnecessary knuckle dependency, but does not bypass MediaPipe's hand
|
||||
detector or establish continuous wrist visibility in the saved grip images.
|
||||
|
||||
### RGB transport retry: hidden USB submission failure
|
||||
|
||||
After the wrist-only test did not meet the user's needs, the RGB investigation resumed.
|
||||
Capture `116048611549114` added stderr collection with `LIBUSB_DEBUG=4`, separate from the
|
||||
vendor log hook. The SDK returned start status 0 and streaming status 1, but libusb reported
|
||||
`submit_iso_transfer: submiturb failed, errno=12` (`ENOMEM`) and subsequent isochronous packet
|
||||
errors. No RGB callbacks arrived. GET_INTERFACE showed alternate setting 5; the negotiated
|
||||
UVC payload was 2400 bytes per microframe. This establishes USB allocation/submission failure,
|
||||
not its exact resource limit: Android denied reading the USB memory-limit sysfs attribute.
|
||||
|
||||
Binary inspection also traced the earlier native crash to `libusb_cancel_transfer(NULL)`
|
||||
while cleaning up a failed start. Reopening alone was therefore too specific an explanation.
|
||||
The embedded SDK's roughly 100-transfer queue is a plausible source of pressure. Its public
|
||||
API cannot change the queue or the fixed 1080p30 mode.
|
||||
|
||||
An optional independent diagnostic transport now uses libusb 1.0.30 and libuvc 0.0.8 with eight
|
||||
queued transfers. The pinned libuvc source was patched to preserve submission errors and safely
|
||||
drain partial queues instead of reporting success. A hardware-free executable passed 60 starts
|
||||
on the phone covering failure before any submission, failure after three submissions, and all
|
||||
eight submitted. This validates failure cleanup, not camera capture. The reproducible recipe,
|
||||
source hashes, and licenses are documented in `docs/third-party/uvc/README.md`.
|
||||
|
||||
Raw USB descriptors advertise MJPEG 640×480, 1280×720, and 1920×1080 among other sizes, with
|
||||
30/25/20/15/10/5 fps intervals. The direct path can compare these modes while head/stereo tracking
|
||||
continues. It saves a local JPEG plus transfer diagnostics and stream statistics; it is not yet
|
||||
connected to controller tracking. RGB lens/extrinsic calibration and measured reach remain
|
||||
separate unfinished requirements.
|
||||
|
||||
### Direct UVC follow-up after the firmware update
|
||||
|
||||
The glasses now report firmware `12.0.02.002_20260901`, previously
|
||||
`12.0.01.100_20260326`. Direct UVC acquired short bursts, but no sustained stream:
|
||||
capture `140416689286120` delivered 22 callbacks over 708 ms, followed by 4.3 seconds
|
||||
without another callback. The main app stayed alive. These are transport observations,
|
||||
not arm-extension measurements.
|
||||
|
||||
The rejection logs exposed a separate parsing bug: valid JPEG end markers were followed
|
||||
by one to seven zero bytes, with buffers aligned to eight bytes. The diagnostic now trims
|
||||
up to seven trailing zeros before checking the end marker and saves the trimmed payload.
|
||||
A native test on the phone covers padding, truncation, garbage, null input, and size bounds.
|
||||
This fixes rejection of padded frames; it cannot recover missing USB callbacks. JPEG
|
||||
boundary validation also does not establish successful decoding.
|
||||
|
||||
A 32-transfer queue with eight packets per transfer was compared with the original
|
||||
eight transfers of 32 packets. Both fault-injection builds passed 60 simulated starts.
|
||||
The smaller transfers started 1080p30 without the earlier submission error, but received
|
||||
zero frames (`140593211685272`). At 640p the smaller transfers still stalled after seven
|
||||
callbacks (`140493747047289`). Neither configuration established reliable acquisition.
|
||||
|
||||
Explicit USB device resets returned success but produced no frames at 640×480/30 or
|
||||
320×240/5. GET_INTERFACE retained the selected alternate setting. Android's external
|
||||
camera provider reported zero devices. A physical reconnect with positional tracking
|
||||
disabled before attachment also yielded zero callbacks (`141363168667894`): capture
|
||||
waited five seconds for the first frame and never entered its requested ten-second
|
||||
observation window. The original positional and Joy-Con settings were restored afterward;
|
||||
bare-hand controllers remained off. This failure occurs without the stereo/VIO stream.
|
||||
|
||||
Capture evidence is under `build/joycon-rgb-captures/<id>`, including `usb.log`,
|
||||
`frame.json`, descriptors, and per-worker `libuvc-logcat.txt`. Teardown cancellation
|
||||
messages are not evidence that transfers were cancelled during acquisition. RGB remains
|
||||
diagnostic-only, with no RGB depth calibration or controller-position publication.
|
||||
|
||||
Detailed USB logging in capture `141627121555242` recorded 224 successful isochronous
|
||||
transfer completions over approximately 223 ms, each totaling 96 bytes, followed by no
|
||||
successful completions until capture teardown. These small transfers did not produce a
|
||||
decoded image or frame callback. A follow-up (`141844371790836`) read the UVC streaming
|
||||
error code before and after the stall: both returned code 0. Android denied reading the
|
||||
USB power-management sysfs attributes, so autosuspend was not verified as a cause.
|
||||
|
||||
The user confirmed a direct VITURE cable connection. With the same glasses and cable
|
||||
connected to Windows, FFmpeg/DirectShow decoded 300 frames in each of three ten-second
|
||||
runs: 640×480/30, 1920×1080/30, then 640×480/30 after reopening. Outputs used passthrough
|
||||
frame timing and frame hashes, with no frame-rate duplication or saved video. All exited
|
||||
successfully without timeout. Evidence is in `build/joycon-rgb-windows`, produced by
|
||||
`build/test-luma-rgb-windows.py`, which selects only USB VID:PID `0c45:636b`. This supports
|
||||
investigating the Android host/USB path further; it does not prove the camera will be
|
||||
reliable on every host or establish the exact Android fault.
|
||||
|
||||
A controlled APK replacement of only `libusb.so` with official libusb 1.0.27 also
|
||||
returned zero callbacks (`142296418051861`). Its source archive SHA-256 is
|
||||
`ffaa41d741a8a3bee244ac8e54a72ea05bf2879663c098c82fc5757853441575`; provenance and the
|
||||
one-entry APK comparison are under `build/rgb-libusb-1.0.27`. The production dependency
|
||||
pin remains 1.0.30. An isolated YUYV 320×240/5 diagnostic negotiated format 2 but likewise
|
||||
received zero raw frame callbacks (`142407601079475`), so JPEG validation was not its
|
||||
failure point. Neither comparison changed the production camera format.
|
||||
|
||||
Finally, 64 queued transfers of 32 packets increased the high-speed queue coverage from
|
||||
approximately 32 ms to 256 ms. Its fault-injection executable passed all 60 starts.
|
||||
Capture `142557405834522` recovered 18 valid JPEGs over 577 ms, then stalled for the
|
||||
remaining 9.42 seconds of its ten-second observation. All 18 passed the padding fix;
|
||||
the saved final JPEG also decoded at 640×480 in OpenCV. This is not sustained tracking
|
||||
and neither alternate queue was adopted. Temporary USB reset, queue configuration, and
|
||||
verbose logging experiments were removed. The retained UVC recipe uses eight transfers
|
||||
of at most 32 packets, and retains the JPEG-padding fix.
|
||||
@@ -0,0 +1,445 @@
|
||||
# Luma Ultra implementation evidence
|
||||
|
||||
Date: 2026-10-15. Status: experimental prototype, combined release gate not passed. Native 6DoF and synchronized stereo camera delivery were observed on hardware. Camera calibration and optical-hand acceptance were deferred at the user's request.
|
||||
|
||||
## Device and build
|
||||
|
||||
| Item | Observed value |
|
||||
|---|---|
|
||||
| Android host | Samsung Galaxy Z Fold7, SM-F976U |
|
||||
| Android version | 17 |
|
||||
| Glasses | SDK-confirmed VITURE Luma Ultra / Carina |
|
||||
| Glasses firmware | `12.0.01.100_20260326` |
|
||||
| Vendor SDK | External Android C SDK 2.4.0 |
|
||||
| Test application | `com.droiddeck.launcher.xr`, debug experimental build |
|
||||
| Camera pair | Primary left0/right0, both 640 by 480 |
|
||||
| Hand runtime | MediaPipe Tasks Vision 1.0.0; OpenCV 4.13.0; pinned model SHA-256 in the model provenance asset |
|
||||
| CPU or GPU inference benchmark | Not run; no accepted calibration |
|
||||
|
||||
Existing picture, refresh-rate, and graphics work was preserved. The test temporarily enabled positional tracking and in-memory camera diagnostics. Diagnostics were stopped and the positional preference restored to its previous off setting afterward. Hand controllers were never enabled. No camera image recordings were saved. Subsequent builds are local artifacts; the phone was left available for a separate active SteamVR test.
|
||||
|
||||
## Hardware observations
|
||||
|
||||
Initial VIO output was unstable. After the user wore/moved the glasses and the software origin was recentered, valid 6DoF output was observed. Changing position values confirms positional data is available, but does not establish correct physical axes or 1:1 scale without a measured reference.
|
||||
|
||||
Camera frames initially arrived before their matching VIO callback. A bounded wait of up to 35 ms in the latest-frame reader allowed matching timestamps without building a queue. The public callback's undocumented pose basis is not used; capture transforms come from canonical OpenGL pose history.
|
||||
|
||||
A subsequent diagnostic interval produced these results:
|
||||
|
||||
| Measurement | Result and scope |
|
||||
|---|---|
|
||||
| Sampled head validity | 30 of 30 status snapshots valid |
|
||||
| Sampled camera/head synchronization | 30 of 30 status snapshots synchronized |
|
||||
| Processed stereo pairs | Counter increased by 1,048 |
|
||||
| Dropped stereo pairs | Counter increased by 9 |
|
||||
| Median reported camera rate | 24.998 Hz |
|
||||
| Median estimated frame age | 24.73 ms |
|
||||
| Observed estimated frame-age range | 21.44–45.96 ms across the 30 snapshots |
|
||||
| Median diagnostic processing time | 1.62 ms; no hand inference or calibrated undistortion |
|
||||
| Host receipt age of latest head poll | 0.09–4.40 ms across sampled snapshots; not sensor latency |
|
||||
|
||||
These are sparse status samples in an uncontrolled workload, not a frame-by-frame latency benchmark. Receipt-based source-clock mapping does not measure absolute camera exposure delay. None of these values establishes physical-motion-to-visible-controller latency, hand availability, or game-frame-time performance.
|
||||
|
||||
Local raw metadata is in `build/spatial-hardware-observations.json`. The earlier startup/recenter observations are in `build/spatial-after-movement.json`, `build/spatial-recenter-state.json`, and `build/spatial-sync-state.json`. Those files contain pose/status metadata, not images.
|
||||
|
||||
## Automated and build validation
|
||||
|
||||
- SDK-enabled and explicitly SDK-free debug APK builds pass with runtime freshness checks enabled.
|
||||
- All 305 affected Kotlin glasses, head-pose, Cardboard, and glasses UI tests pass with zero failures or skips. New cases exercise canonical coordinates, normalization, floor anchoring, crouched recenter, independent origins, stale/discontinuous samples, generation rejection, pose history, camera-clock warmup/reset, gesture geometry, and inference-buffer ownership.
|
||||
- ARM64 VR driver and presentation-layer builds pass. Native tests cover ABI offsets and size, empty/truncated/version-mismatched or incoherent reads, independent hand timestamps, finite poses, stale tracking, trigger hysteresis, lost-input release and released-gesture rearming. Existing fence, timeline, and render-target tests also pass.
|
||||
- Native Cardboard parameter tests cover all three translation values, zero-translation equivalence, and unchanged SBS/flat parameters. The GLSL compiles; the committed SPIR-V matches regenerated glslang output, and push constants remain within the guaranteed range.
|
||||
- VR build manifests include the new shared header. Packaged runtime and APK payload checks reject mismatched source/output checksums. Both external vendor files and transient dependency-inspection files remain outside tracked source.
|
||||
- Packaging regressions pass: 8 Windows VR build tests, 13 Windows preparation tests, and 11 runtime-input tests. Runtime-input tests ran in the Linux container because the Windows host has no standalone zstd executable.
|
||||
|
||||
Build logs are `build/spatial-sdkfree.log`, `build/spatial-final-build.log`, `build/spatial-vr-build.log`, `build/spatial-shader.log`, and `build/spatial-runtime-evidence.json`. Test reports are under `app/build/reports/tests/testDebugUnitTest`.
|
||||
|
||||
The preserved SDK-enabled snapshot is `build/DroidDeck-luma-spatial-experimental.apk`; the SDK-free counterpart is `build/DroidDeck-spatial-sdkfree.apk`. SHA-256 hashes and final payload checks are recorded in `build/spatial-final-validation.json`. The final local snapshot includes additional buffer-ownership and conservative tracking-loss fixes after the hardware smoke test. It was not installed over the phone being used for the separate SteamVR tests. Integrate these sources into that chat's next build before later calibration; do not replace its newer runtime-storage fixes with this older snapshot.
|
||||
|
||||
A subsequent report from the parallel SteamVR test identified Android eviction of the live `cache/viture` directory. The SDK working-directory source was changed to `noBackupFilesDir/viture` afterward. The preserved APK snapshot and the hardware observations above predate this storage change. No additional phone restart or installation was performed here; the next shared-workspace build should include it.
|
||||
|
||||
## Calibration usability follow-up (2026-10-15)
|
||||
|
||||
The calibration screen now uses DroidDeck's saved palette, app scale, settings groups, and buttons, with separate tracking, capture, validation, and diagnostic sections. Eye height, virtual-screen dimensions, board instructions, and validation messages include imperial measurements. The distance check accepts either 20 inches (0.508 m) or 50 cm (0.5 m), using the selected value in reconstruction validation.
|
||||
|
||||
Recenter requests made while the newest pose is stale or invalid were previously ignored. Requests now remain pending and are acknowledged immediately in the UI; recovery waits for 250 ms of fresh, steady tracking. Normal recentering preserves floor height, new connections cancel old requests, and captures cannot mix different VIO coordinate frames. Looking straight upward also retains a stable heading instead of deriving yaw from an almost-zero forward projection. Camera diagnostics resume when the provider restarts while setup remains open.
|
||||
|
||||
Validation: 247 glasses and calibration/settings UI tests pass with zero failures or skips. The SDK-enabled debug APK builds, runtime input verification passes, and native Robolectric renders of the recovery and distance screens were visually checked. The APK's DEX files were checked for the new measurement controls. Logs are in `build/spatial-setup-build.log`; previews are in `app/build/reports/spatial-setup/`. The phone was not reinstalled or reconfigured during the other chat's SteamVR stability run. The user's reported repeated-recenter behavior still needs confirmation on the glasses after installing this build.
|
||||
|
||||
After those stability tests finished, the combined calibration/recenter APK was installed at 17:09 MDT on October 15. Its SHA-256 is `80bf99374a31676b675856c782298aecc1e1c649ad2fc5c1ce3b18b981979e3e`; hashing the installed `base.apk` confirmed an exact match. The app cold launch and agent status checks passed. The helper, VR driver, and scheduling preload match the preceding freeze-tested build. This install confirms delivery of the latest calibration changes, not hardware acceptance of recentering or calibrated hands.
|
||||
|
||||
## Capture counter fix (2026-10-15)
|
||||
|
||||
Board detection checked head freshness against the worker's start time. A newer head poll arriving during detection made that timestamp precede the latest pose and rejected an otherwise valid capture. Completion now checks freshness under the session lock using the current monotonic time, while still rejecting stale poses, changed connections, changed tracking frames, and pending recentering. The same timing check was corrected around hand inference. Capture progress now exposes pending work and shows success or the rejection reason beside the counter, including the number of common board corners when detection fails.
|
||||
|
||||
Validation: 24 targeted spatial and setup UI tests pass with no failures or skips, including the head-poll/detection timing regression and stale/disconnected-frame rejection. The SDK-enabled APK builds with runtime freshness verification. Build log: `build/spatial-capture-fix.log`. APK: `build/DroidDeck-spatial-capture-fix.apk`, SHA-256 `bf362bb606a3ec72665102b66e4a122c8fad9f5f7333f5a9a3ea3fef9a3bc9d5`. The installed Fold7 `base.apk` hash matches.
|
||||
|
||||
With USB permission granted, valid native tracking, and synchronized stereo images at approximately 25 Hz, a live Capture board view action advanced the counter from 0 to 1 of 24 and reported success. The accepted view remains in the open calibration session. Status-only evidence is in `build/spatial-capture-fix-device.json`; no diagnostic camera recording was saved. This confirms capture acceptance and counter progression, not a completed calibration or optical-hand acceptance.
|
||||
|
||||
## Calculation failure and compact setup follow-up (2026-10-15)
|
||||
|
||||
The user reported calculation failure after at least 24 accepted captures; inspection found 35 observations still in memory. Logcat identified an OpenCV fisheye stereo fitting exception when successive views contained different corner counts (12–24 in this set). [OpenCV 4.13's implementation](https://github.com/opencv/opencv/blob/4.13.0/modules/calib3d/src/fisheye.cpp#L819) allocates its stereo residual blocks using the first view's count. The revised path fits fisheye intrinsics, undistorts every observed point, and fits stereo geometry with fixed intrinsics and zero residual distortion using the variable-count pinhole stereo solver. It retains every correspondence and still measures held-out error with the original lens model. Model failures are logged individually and the page shows the actionable reason instead of a generic failure.
|
||||
|
||||
All 35 observations were exported from the debug app as board landmarks and head poses before installation, backed up in app-private storage, and restored on the same connected glasses. No diagnostic camera recording was taken. Accepted captures now save this metadata automatically; automatic recovery requires a matching serial and geometry, and recovered batches cannot accept new-frame observations. This device did not expose a serial, so recovery here was explicit through the debug interface. The session-labelled backup is not automatically trusted on later unidentified connections.
|
||||
|
||||
The fixed fitter ran on the Fold7 with the restored set. Fisheye stereo fitting now completes, but both models fail stationary-board alignment: the best result varies by 25.5 cm (10.04 inches), above the unchanged 5 cm gate. The user confirmed the board was fixed. Offline replay showed a marked coordinate shift between the early and later observations; an exploratory fit of the first 13 observations agreed within 7.4 mm, while the full set did not. This suggests VIO frame motion/drift during collection; it does not establish its exact cause or validate a calibration from a selected subset. No profile was approved, and hand-controller acceptance remains blocked on a stable capture set plus measured-distance validation.
|
||||
|
||||
The calibration page now has a single Back/title row and no fixed subtitle or eyebrow. A landscape regression check requires at least 75% of page height for scrolling, and both its rendered preview and the installed Fold7 layout were inspected. Guidance explicitly asks for gentle turns and tilts with a pause before each capture. The final SDK-enabled APK and 29 targeted tests pass, including saved-point parsing, stale-frame behavior, restored-batch UI, and landscape viewport coverage. Runtime freshness verification passes. Installed APK SHA-256: `1ca44c6b2b480b197ec0069db026b765d7927c30b7cea7b80cb67d50d626d38a` (verified against `base.apk`). Artifact: `build/DroidDeck-calibration-calculation-fix.apk`. Evidence: `build/spatial-calculation-fix.log`, `build/calibration-observations-backup.json`, `build/calibration-alignment.log`, `build/calibration-calculation-device.json`, and `build/calibration-layout-after.png`.
|
||||
|
||||
## Distance-validation feedback follow-up (2026-10-15)
|
||||
|
||||
A new 25-view capture set produced a fitted pinhole calibration with 1.12094 px held-out reprojection error and passed the existing alignment checks. The user then reported that Validate distance appeared to do nothing. Live status showed that it had run and failed, but the message was attached to Calculate camera alignment, above the visible distance controls.
|
||||
|
||||
Validation now has independent pending/result state beside its own button. It reports partial-board corner counts, stereo-depth rejection, or measured versus selected distance in both units. It disables duplicate presses while pending and times out when no fresh camera image arrives. The complete-board requirement and 5 cm distance tolerance are unchanged. Processing errors are reported without stopping the camera worker.
|
||||
|
||||
The 25 original observations and fitted geometry were backed up, then explicitly restored onto the same connected glasses after installation, retaining `verified=false`. The device still provides no serial; no cross-device or automatic unidentified-profile reuse was introduced. A live validation attempt on the updated app reported 3 of 24 common corners rather than silently failing. Physical-distance validation remains pending; the existing fit and captures remain available in the open setup session.
|
||||
|
||||
Validation: 36 targeted tests pass, including visible validation feedback, pending-button state, complete-board/depth rejection, dual-unit output, camera-to-center distance, and unchanged tolerance. The SDK-enabled APK builds with runtime freshness verification. The result layout was rendered and visually inspected. APK: `build/DroidDeck-calibration-distance-fix.apk`; SHA-256 `1f0b288fdcb76231ecc9f5178f6621799b01f0eb83b574c79394fa1a04bb5f47`. Logs and metadata: `build/spatial-distance-fix.log`, `build/spatial-distance-apk.log`, `build/calibration-distance-fit.json`, `build/calibration-distance-captures.json`, and `build/calibration-distance-after.json`.
|
||||
|
||||
## Near-distance detection fix (2026-10-15)
|
||||
|
||||
The live stereo preview showed the complete printed board at the user's reported 20-inch position, but the pattern occupied only about 70 pixels across in the original 640 by 480 images. One explicitly requested diagnostic pair was saved. OpenCV 4.13 detected 16 left-camera checkerboard corners in the raw image and none after image undistortion; the right image detected 3 raw and 1 corrected. Neither version supplied a complete common board. This identified both limited marker resolution and additional loss from resampling, rather than paper edges being out of view.
|
||||
|
||||
Distance validation now detects the original image pair and undistorts the detected coordinates before stereo reconstruction. Pinhole inversion uses an explicit convergence criterion. The UI defaults to 12 inches (0.3048 m), also offers 30 cm, and retains the 20-inch and 50-cm choices. Feedback distinguishes detected checkerboard corners from paper edges. The 24-common-corner, stereo reprojection/depth, and 5-cm physical-distance gates remain unchanged.
|
||||
|
||||
Validation: all 36 targeted spatial/calibration/UI tests pass. Synthetic forward-projection and reconstruction checks exercise pinhole and fisheye point correction at 0.3048, 0.508, and 0.8 m; these verify geometry calculations, not hardware accuracy. The SDK-enabled build and runtime freshness checks pass, and the rendered result layout was inspected. APK: `build/DroidDeck-calibration-near-distance.apk`, SHA-256 `cd7e144e4fabb2e94872e4a762f1a641a61773902fa8fe40ec2c8c05d778f81d`; the installed Fold7 APK matches. Build log: `build/spatial-near-distance.log`. Synthetic results: `build/spatial-raw-landmark-geometry.json`.
|
||||
|
||||
The same connected glasses' 25 accepted views and 1.12094-px fitted alignment were restored after installation without marking them verified. A subsequent user-triggered distance check reported **12.01 inches measured against 12.00 inches selected**, passing the unchanged validation gates and marking the calibration verified for this connection. Status evidence is in `build/calibration-near-distance-after.json`. Native head tracking and stereo synchronization remained valid at approximately 25 Hz. No optical hands were valid in that status sample; hand accuracy, externally measured latency, and game performance remain untested. Device identity is still unavailable, so automatic reuse across unidentified reconnects remains disabled.
|
||||
|
||||
## Calibration persistence and hand-controller investigation (2026-10-15)
|
||||
|
||||
After launching SteamVR, the app reported `calibrated=false`, zero restored views, and no optical observations despite valid native head tracking. Android USB did not expose a serial number, so the earlier calibration had stayed in memory. SDK 2.4.0's documented `xr_device_provider_get_sn_hash` returns a stable 32-byte board-serial digest. The adapter now reads it under the SDK ownership lock, rejects missing/zero digests, and uses it to key fitted profiles and captures. A nonblank Android USB serial remains a fallback. An unavailable identity still does not permit automatic cross-connection reuse.
|
||||
|
||||
The existing 25 views and fitted geometry were explicitly imported onto the same physical glasses under the SDK identity with distance validation still required. The user repeated the check: **12.13 inches measured against 12.00 selected**. The app saved `verified=true`, and subsequent app installations/restarts automatically recovered the verified calibration and all 25 views without debugger restoration. This resolves the lost-calibration blocker; it does not establish reliable optical controllers.
|
||||
|
||||
Hand diagnostics now distinguish missing calibration, synchronization/freshness failure, per-camera detections, ambiguous matching, rejected stereo geometry, and accepted hands. They retain the last completed result while inference runs, and expose preprocessing/inference timing. Observations over 100 ms old are cleared before publication; the driver and session freshness checks remain in place. CPU stereo inference runs concurrently with separate image buffers and waits for both tasks before frame reuse, including after one task fails. GPU inference remains on its owning thread. Calibrated undistortion maps are cached; comparisons against the original image-undistortion operations were pixel-identical for both cameras and both lens-model paths.
|
||||
|
||||
SDK 2.4.0 documents the tracking cameras at 25 Hz and provides no stereo frame-rate setter. Faster application polling cannot raise that rate. The source-rate calculation now accounts for skipped frame sequence numbers; its earlier readout fell toward inference throughput when frames were dropped.
|
||||
|
||||
Short live samples on the Fold7 do **not** pass the combined hand gate. In the final 12-sample CPU/SteamVR check, samples with detected hands took 87–137 ms to process, and no fresh hand was observed by the status reader. In the comparable GPU sample, hand-containing observations took 59–81 ms to process, with one accepted reconstruction but no fresh hand caught by the status reader. Earlier setup-only GPU samples did briefly report a fresh left hand. Stereo-depth rejection and observations exceeding 100 ms remain frequent. These are sparse internal timing samples with uncontrolled pose/thermal conditions, not externally measured latency, hand-validity percentages, or a controlled game-performance comparison. The user reported no visible controllers in SteamVR. An earlier 8-second guest-channel check found the 176-byte channel active with both feature flags set and all sampled heads fresh, but no fresh optical hands; both virtual controller devices were activated in the SteamVR log.
|
||||
|
||||
One hand-landmark pair was inspected from the running reconstruction method without saving camera images. All 21 points triangulated within the broad depth range, but nine failed the unchanged 3-pixel reprojection gate. The wrist's error was 12.80 px; several fingers were approximately 3–4.3 px. Reconstructed palm width was 6.15 cm and depths were 28–41 cm. This is a correspondence/reconstruction failure, not a missing shared-memory channel. The single sample does not isolate calibration residuals from per-view landmark error, and no quality thresholds were relaxed. Data: `build/hand-stereo-point-diagnostics.json` and `build/hand-stereo-geometry-report.json`. Debugger pauses used for this inspection are excluded from performance evidence. The temporary GPU benchmark was ended with CPU mode restored; neither delegate has passed the combined gates.
|
||||
|
||||
Validation: 51 targeted tests pass, including device-identity binding, concurrent stereo work, frame ownership after detector failure, calibration, gestures, and setup UI. SDK-enabled and SDK-free Android builds pass runtime freshness checks. Installed candidate: `build/DroidDeck-parallel-hands.apk`, SHA-256 `5e8ffa6a89a91e3b1b22fe289c126dca96a77dfb7645da551e8e97e13673c761`. Evidence: `build/spatial-parallel-hands.log`, `build/spatial-parallel-sdkfree.log`, `build/spatial-cached-undistortion.json`, `build/spatial-parallel-cpu-steamvr.json`, `build/spatial-cached-gpu-steamvr.json`, and `build/spatial-steamvr-hands-channel.json`. Calibration backup: `build/calibration-verified-profile.json` and `build/calibration-verified-captures.json`. The combined feature remains an experimental prototype.
|
||||
|
||||
## Hand-mode color regression and CPU rollback (2026-10-15)
|
||||
|
||||
The parallel CPU trial was removed after failing to improve live hand results. The serial worker retains cached lens maps, device-bound calibration persistence, and detailed rejection diagnostics. All 49 remaining targeted tests pass; the SDK-enabled APK builds with runtime freshness checks. Installed candidate: `build/DroidDeck-serial-hands.apk`, SHA-256 `bcefc87dabfde3d29f2018c2b39368d4e68cb015dd390385e3c80c74d86531c6`. Build log: `build/spatial-serial-rollback.log`.
|
||||
|
||||
The user reported choppy head turns with hands and positional tracking disabled, then requested that this investigation be deferred because the phone was charging. A thermal snapshot reported status 4 and a skin sensor reading of 49.4 degrees C. The rendering log recorded approximately 55.5–60 fps at a selected 60 Hz. The retained 1.22-second prefix of a truncated head-channel diagnostic showed pose changes roughly every 3.2–6.6 ms, with no sampled gap above 7.5 ms. This small sample is not a controlled performance acceptance result. Evidence: `build/tracking-regression-sample.json`, `build/tracking-regression-analysis.json`, and `build/tracking-regression-wayland.log`.
|
||||
|
||||
The HDR-like darkening is reproduced in screenshots of the glasses output, with brightness 6, Sunblock off, and valid head tracking unchanged. Camera inference enabled independently of optical-controller mode leaves normal colors. Enabling optical-controller mode with inference paused darkens midtones and shadows. A further test switching only `SpatialTracking.setHands`, bypassing the settings/menu/display-refresh path, also reproduces it. The temporary debugger overrides were removed after each comparison. This isolates the trigger to the SteamVR optical/untracked-controller path; it does not yet identify the faulty color-conversion operation. HDR output is disabled in the session. Evidence: `build/hand-color-comparison.json`, `build/hand-color-isolation.json`, `build/hand-color-mode-isolation.json`, and their `build/hand-color-*.png` screenshots. The calibration and 100-ms observation expiry remain unchanged.
|
||||
|
||||
The serial candidate also passes the SDK-free build (`build/spatial-serial-sdkfree.log`). Temporary `TU_DEBUG=noubwc` did not remove the darkening, although the user changed display modes between that pair's screenshots. A subsequent `TU_DEBUG=gmem` comparison was interrupted by graphics recovery and is inconclusive. Neither diagnostic setting was retained: the following session's actual `vrserver` and `vrcompositor` environments both report an empty `TU_DEBUG`, and pending, active, and persistent agent overrides are empty (`build/hand-color-current-flags.json`). No gamma compensation or hand-validity workaround was added.
|
||||
|
||||
The user's observation that SBS felt smoother coincided with 2D running at 60 Hz and SBS at 90 Hz. A later 120-Hz 2D request still reported a 60-Hz physical mode; its failure cause remains undetermined. The next 90-Hz 2D selection succeeded: Android reports 1920 by 1200 at 90 Hz, and the SDK reports mode `0x43`. The app therefore has no fixed 60-Hz limit for 2D. 2D and SBS retain separate refresh preferences; actual display mode must be checked independently of the requested value. Evidence: `build/hand-color-display-modes.txt` and `build/refresh-mode-final.json`. This confirms the active refresh mode, not sustained 90-fps game rendering or a thermal/performance acceptance result.
|
||||
|
||||
## Color fix, original-image detection, and late synchronization (2026-10-16)
|
||||
|
||||
The HDR-like color change is fixed and confirmed by the user in the glasses. `NewSharedVulkanImage` requested compute access for compositor eye textures even though DroidDeck only needs rendering and sampling. SteamVR replaced the requested `R8G8B8A8_SRGB` format (43) with `R8G8B8A8_UNORM` (37). The untracked-controller compositor path then produced dark linear midtones, while its multisampled controller path encoded colors differently. Removing compute access preserves format 43 in both paths. Temporary Vulkan tracing established the before/after formats and was removed from the delivered build; the presenter retains its existing byte-preserving UNORM import. No gamma compensation was introduced. Evidence: `build/color-format-fixed-trace.json`, `build/color-format-fixed-states.json`, and the corresponding on/off output images. The user reported that colors now stay the same when toggling Hand controllers.
|
||||
|
||||
One explicitly authorized stereo image pair showed upright camera images and two visible open palms. Replaying it with the exact Android CPU detector found one left hand in each camera. All 21 landmarks passed the unchanged 3-pixel reprojection gate (maximum 2.227 px), with reconstructed palm width 7.31 cm and depth approximately 32–39 cm. The right hand was not detected consistently; lower confidence thresholds and exploratory crops did not establish an improvement and were not adopted. Hand detection now consumes the original grayscale images converted to RGBA and corrects landmark coordinates before triangulation, avoiding full-image resampling. This verifies one reconstructable sample, not sustained detection accuracy or two-hand operation. Evidence: `build/android-hand-pair-replay.json`, `build/android-hand-pair-geometry.json`, and `build/hand-pair-replay.json`.
|
||||
|
||||
Previously, camera acquisition waited up to 35 ms for a matching VIO callback before starting hand inference. Inference now starts with the newest pair and checks the exact source timestamp again after detection, allowing VIO processing to overlap detection. Publication still requires a matching callback within 5 ms, a nearby canonical head-history sample, unchanged connection and tracking frame, healthy current head tracking, and an observation no more than 100 ms old. The callback's undocumented pose basis remains unused. Calibration and preview-only acquisition retain their bounded wait. Four additional regression tests cover late confirmation with the original capture pose, missing confirmation, expiry, tracking loss/discontinuity, reconnects, and unavailable history.
|
||||
|
||||
Validation: 53 targeted Android tests pass with no failures or skips; SDK-enabled and SDK-free Android builds and runtime freshness checks pass. The ARM64 VR build passes its native layout, freshness, optical input recovery, render-target, synchronization, and scheduling checks. Temporary color tracing is absent from the final driver source and packaged runtime. Installed SDK-enabled artifact: `build/DroidDeck-spatial-color-sync.apk`, SHA-256 `885fb14b5a30ac65a255a6b0dd519d9f73b72489c08a8b74e1756673b86812ea`. Logs: `build/color-fix-final-native.log`, `build/color-fix-final-stage.log`, `build/spatial-late-sync-build.log`, `build/spatial-late-sync-sdkfree.log`, and `build/spatial-color-sync-install.json`.
|
||||
|
||||
During the subsequent grey-screen report, the compositor continued presenting approximately 60 frames per second, but the SDK returned unstable head status 1 and the shared head pose was invalid. USB timeout messages also appeared; their causal relationship to tracking loss is not established. The clean app/provider restart preserved all 25 calibration views and the verified profile. After tracking recovery and software recentering, the user confirmed both “6DoF active” and the visible SteamVR dashboard. Positional tracking was briefly disabled for diagnosis and restored. No synthetic valid pose or silent fallback was added. Evidence: `build/grey-launch-guest.json`, `build/grey-launch-app.log`, and `build/spatial-color-sync-recovered.json`.
|
||||
|
||||
The next 16-second left-palm test sampled the guest spatial channel 2,900 times: every head was fresh, but no optical controller pose was fresh. Six status samples showed one matched hand in both cameras and successful stereo reconstruction; inference took 139.7–174.3 ms and total processing 144.1–181.8 ms. Those observations correctly expired before publication. The other three status samples were waiting for synchronization. The app restarted between preparation of a temporary GPU benchmark and this measurement, so the active delegate was not verified in the new process and this is not a controlled GPU comparison. A later thermal snapshot reported status 4 and skin readings up to 51.0 degrees C; its contribution to the measured slowdown is unquantified. Evidence: `build/spatial-live-hands-left.json`, `build/spatial-live-hands-left.log`, and `build/spatial-hand-thermal.txt`. The glasses subsequently disconnected, preventing another live comparison. The next measurement needs a stable process/connection, explicit delegate verification, and controlled thermal conditions. Hand controls remain an experimental prototype and have not passed the combined release gate.
|
||||
|
||||
## Tracking-loss recovery screen (2026-10-16)
|
||||
|
||||
Active positional SteamVR and virtual-screen sessions now show a DroidDeck-themed recovery screen over the rendered output when head tracking becomes invalid. Separate states explain how to recover tracking, reconnect USB, recenter a changed origin, or hold still while recentering settles. The screen is rendered independently of the head pose, supports mono and SBS output, and closes automatically after valid tracking returns. The phone retains access to Recenter and glasses setup. No tracking validity, input-expiry, calibration, or driver behavior was relaxed.
|
||||
|
||||
Validation: 65 targeted Android tests pass without failures or skips, including callback-free stale-pose detection, recovery states, presentation surface preservation, mono/SBS rendering, and theme colors. Robolectric native-rendered previews were visually inspected at 1080 by 600 and 3840 by 1200; this is automated rendering evidence, not a physical glasses acceptance check. SDK-enabled and SDK-free Android builds and packaged-runtime freshness checks pass. Logs and previews: `build/tracking-recovery-build.log`, `build/tracking-recovery-sdkfree.log`, and `app/build/reports/tracking-recovery/`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-tracking-recovery.apk`, SHA-256 `2170c2e513d45af048b9aba01b76a694d333ba9848a91021a47d413fca3d282d`. Installation preserved app data and was performed with the session idle; SteamVR was not started. The installed APK hash matches the archived artifact (`build/tracking-recovery-install.json`). A live tracking-loss/recovery check through the glasses remains pending.
|
||||
|
||||
## Cooled-phone retest and visible left controller (2026-10-16)
|
||||
|
||||
The Fold7 initially reported thermal status 1, approximately 40.3 degrees C skin temperature, healthy USB access, and the saved 25-view calibration. Head tracking recovered to valid status 0. CPU inference was verified through the live setup control in process 17409, session 12. By the 16-second left-palm measurement, thermal status had returned to 4. All nine diagnostic samples detected and reconstructed a matching hand in both cameras, but inference took 124.4–173.3 ms. The guest channel had 3,038 fresh head reads and zero fresh hand reads. This does not establish an unthrottled CPU baseline. Evidence: `build/cooled-cpu-verified-ui.json`, `build/spatial-live-hands-cooled-cpu.json`.
|
||||
|
||||
Enabling GPU inference while that SteamVR session was running initialized both GPU graphs, including OpenCL loading, at 01:10:24–01:10:27 device-local time. The compositor first reported `vkerror=-4` at 01:10:29.516; automatic graphics recovery then killed and restarted the app. Process 27927, session 13, was back on CPU. The subsequently captured file named `build/spatial-live-hands-cooled-gpu.json` therefore contains CPU observations after recovery and must not be counted as a GPU comparison. Process IDs, session IDs, the setup control returning to Off, and Android exit records establish that reset. The timing associates GPU initialization with the device-loss event but does not establish its underlying driver cause. Evidence: `build/cooled-before-restart-logcat.txt`, `build/cooled-session12-vrcompositor.txt`, `build/cooled-session12-events.jsonl`, `build/cooled-exit-info.txt`, `build/cooled-gpu-after-ui.json`.
|
||||
|
||||
With SteamVR stopped, GPU inference was enabled and verified in process 27927. Twelve diagnostic samples then measured 10.3–14.1 ms inference and 12.0–19.7 ms total observation age, with healthy head tracking. These samples were not a prescribed palm-hold accuracy test; most saw a hand in only one camera and none produced accepted geometry. Evidence: `build/cooled-idle-gpu-verified-ui.json`, `build/cooled-idle-gpu-sample.json`.
|
||||
|
||||
Launching SteamVR after GPU initialization preserved process 27927 for session 14. During a fresh 16-second left-palm hold, the guest channel received 67 distinct valid left-hand observations. The left hand was fresh in 315 of 2,864 coherent channel reads (11.0%); all heads were fresh. This percentage measures sampled channel availability, not the acceptance target's percentage of camera observations. The user confirmed a controller appeared or flickered. Eight diagnostic samples all reconstructed the left hand, but inference varied from 41.8–101.6 ms and observation age from 72.5–131.9 ms. Thermal status remained 4, with reported skin temperature increasing from 49.1 to 51.3 degrees C. The session also logged another Vulkan device-loss event, so initializing GPU inference before SteamVR did not resolve stability. Evidence: `build/spatial-live-hands-prewarmed-gpu.json`, `build/cooled-prewarmed-start.json`, `build/cooled-session14-events.jsonl`.
|
||||
|
||||
This confirms a visible optical left-controller path, with no relaxed timestamp or geometry checks. Both-hand reliability, gesture operation, measured end-to-end latency, and sustained performance remain unvalidated. No camera images were saved during these tests, and calibration was not changed. The run was stopped after the repeated graphics fault. After confirming the session was idle and its artifacts complete, DroidDeck was closed to stop background inference and clear the temporary GPU benchmark; saved settings and calibration were retained. The next work must address GPU/SteamVR coexistence and inference latency under sustained thermal load; another calibration is not indicated by these successful left-hand reconstructions.
|
||||
|
||||
## Requested 200 ms experimental hand cutoff (2026-10-16)
|
||||
|
||||
The user requested accepting hand observations up to 200 ms old to reduce disappearing controllers while inference is slow. The cutoff now applies consistently to inference admission/completion, capture-pose lookup, Android publication, and SteamVR optical pose/input freshness. Observation timestamps are unchanged. Head freshness remains 100 ms, and detected loss, changed tracking frames, reconnects, and recentering still invalidate hands immediately. Missing observations can now retain trigger/grip until the 200 ms expiry; reacquisition still requires an observed release. This intentionally changes the earlier 100 ms hand-expiry requirement, without claiming lower processing latency or resolving the Vulkan faults. The diagnostics field `handMaxAgeMs` reports the active limit.
|
||||
|
||||
Validation: 73 targeted Android tests pass, including a 150 ms inference result using its original capture pose, the inclusive 200 ms boundary, rejection immediately after that boundary, unchanged 100 ms head expiry, and independent hand timestamps. Native tests pass the same hand/head boundaries, pose validity, stale-input release, and reacquisition behavior. The ARM64 driver was rebuilt, staged, and verified against source/output checksums; SDK-enabled and SDK-free Android builds pass. Logs: `build/hand-200ms-native.log`, `build/hand-200ms-stage.log`, `build/hand-200ms-build.log`, `build/hand-200ms-sdkfree.log`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-200ms.apk`, SHA-256 `dac9296cbc964fca87b65b7e1e8b87276deaaeb5d90ca731d47eae673e22c64d`. Its packaged driver matches the rebuilt ARM64 binary, SHA-256 `bf24667f9f288cd07efe6e100c3b237b1e68b734100ae99cf93974538ce3bc6a`. The installed APK hash and live `handMaxAgeMs: 200` were verified. Calibration and saved settings were preserved. SteamVR was not launched, and the app was closed after verification to stop background inference (`build/hand-200ms-install.json`). Live controller availability, gesture delay, and release behavior with the new cutoff remain unmeasured.
|
||||
|
||||
## GPU default requested by the user (2026-10-16)
|
||||
|
||||
The user explicitly requested GPU inference as the default. Every new stereo hand engine now selects GPU, including after app restart, USB reconnect, and provider restart. The setup control is renamed **GPU hand inference** and explains that disabling it selects CPU until the next tracking restart. Diagnostics distinguish `handGpuEnabled` (selection) from `handInferenceBackend` (GPU/CPU only after both detectors initialize, otherwise `not initialized`). This replaces the earlier CPU-default policy; it does not resolve the measured Vulkan failures or constitute performance acceptance. The 200 ms hand cutoff and gesture mappings are unchanged.
|
||||
|
||||
Validation: 73 targeted Android tests pass, including pinch/grip mapping, expiry, tracking recovery, and setup UI. SDK-enabled and SDK-free builds and runtime freshness checks pass. The packaged driver still matches the previously tested 200 ms ARM64 binary. The active SteamVR test was stopped cleanly before installation. GPU selection and the verified calibration were confirmed in two distinct app processes, 22127 and 23157, separated by a full app restart. Head tracking was valid, but hand diagnostics reported waiting for camera/head synchronization and the detectors reported `not initialized`; these checks verify default selection and profile persistence, not a new live GPU inference or gameplay result. SteamVR was not relaunched.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-gpu-default.apk`, SHA-256 `a32feb647b0ca9fb6f62bd31158450b58e5671de7c1b0324f6b0b56bd4c92a47`. The installed APK hash matches. Evidence: `build/hand-gpu-default-build.log`, `build/hand-gpu-default-sdkfree.log`, `build/hand-gpu-default-install.json`. Click instructions are documented: present an open hand, aim the virtual controller, pinch thumb and index finger to press, and separate them to release. Curling the middle, ring, and little fingers supplies grip; tracking loss requires another observed open/released gesture before input can resume.
|
||||
|
||||
## CPU default restored and model optimization investigated (2026-10-16)
|
||||
|
||||
After further SteamVR restarts, the user requested reverting to CPU. New engines, the no-engine diagnostics fallback, and the setup UI now default to CPU again. The optional GPU toggle explains the observed restart risk and resets on tracking restart. Calibration, the color fix, and the requested 200 ms hand cutoff are retained. GPU device loss has not been fixed or ruled out under other configurations.
|
||||
|
||||
Validation: 73 targeted Android tests pass with no failures, errors, or skips. SDK-enabled and SDK-free builds pass; the SDK-free APK excludes the optional VITURE bridge. The SDK-enabled APK's packaged ARM64 driver matches the previously tested 200 ms binary. CPU selection (`handGpuEnabled: false`) and the verified 25-view calibration were confirmed in two distinct processes, 3540 and 3717, across a full app restart. Head tracking was lost and hand detectors were `not initialized` during these checks, so this verifies the restored default and preserved calibration, not new live CPU performance or SteamVR stability.
|
||||
|
||||
Installed artifact: `build/DroidDeck-hand-cpu-restore.apk`, SHA-256 `5ae8f702fdce16501a69731fd5b0e718198e267276e918322b15228ae3b95282`. The installed APK hash matches. Driver SHA-256 remains `bf24667f9f288cd07efe6e100c3b237b1e68b734100ae99cf93974538ce3bc6a`. Evidence: `build/hand-cpu-restore-build.log`, `build/hand-cpu-restore-sdkfree.log`, and `build/hand-cpu-restore-install.json`. SteamVR was not launched for this verification.
|
||||
|
||||
The shipped bundle already uses float16 weights. This does not establish float16 CPU arithmetic or a speed advantage: Google's [float16 documentation](https://developers.google.com/edge/litert/conversion/tensorflow/quantization/post_training_float16_quant) describes CPU weight expansion to float32. [INT8 quantization](https://developers.google.com/edge/litert/quantization/model_optimization) is a possible separate experiment, subject to runtime/operator compatibility and stereo accuracy validation; no INT8 conversion or speed claim has been made.
|
||||
|
||||
An offline feasibility experiment compared Google's [legacy lite palm and hand landmark models](https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/models.md) against the installed full bundle using the previously authorized stereo image pair. No new images were captured. The lite models initially failed the Tasks loader because normalization metadata was absent. For this local experiment, input/output tensor names, shapes, and types were checked against the full models before copying their metadata and associated files. The 0–1 input normalization agrees with Google's [hand landmark graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/modules/hand_landmark/hand_landmark_cpu.pbtxt) and [palm graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/modules/palm_detection/palm_detection_cpu.pbtxt). Experimental bundles remain under `build/hand-lite-research/`; the app still ships the original checksum-pinned full bundle.
|
||||
|
||||
Windows CPU replay used MediaPipe 0.10.21, two sequential VIDEO instances, two-hand capacity, unchanged 0.6 confidence thresholds, 10 warmup pairs and 30 timed repetitions of one static pair. Times cover stereo inference only, excluding camera acquisition, preprocessing, geometry, rendering, and Android/game contention.
|
||||
|
||||
| Models | Median stereo inference | p95 stereo inference | Static-pair geometry |
|
||||
|---|---:|---:|---|
|
||||
| Full baseline, before alternatives | 84.54 ms | 97.28 ms | Left hand passes in 30/30 repetitions |
|
||||
| Lite palm + full landmarks | 37.11 ms | 43.94 ms | No hands detected; unusable for this sample |
|
||||
| Full palm + lite landmarks | 62.48 ms | 73.18 ms | Left hand passes in 30/30 repetitions |
|
||||
| Both lite | 36.75 ms | 43.69 ms | No hands detected; unusable for this sample |
|
||||
| Full baseline, after alternatives | 87.99 ms | 100.15 ms | Left hand passes in 30/30 repetitions |
|
||||
|
||||
Full palm plus lite landmarks is approximately 26–29% faster by median in this limited desktop replay. Its first-frame maximum reprojection error is 1.83 px versus 2.23 px for full; this single-pose result is not an accuracy comparison across real interactions. Repeated images do not validate motion, tracking availability, both hands, reacquisition, or thermal performance. Android Tasks 1.0.0 compatibility and temperature-matched Fold7/SteamVR measurements are still required before adopting it. The apparent speed of the lite-palm variants is not useful evidence of faster working tracking because they performed no landmark inference on a detected hand.
|
||||
|
||||
Research artifacts: `build/compare-hand-lite.py`, `build/hand-lite-research/results.json`, `build/hand-lite-research/without-metadata.json`, and `build/hand-lite-research-metadata.log`. Original downloaded SHA-256 values: palm lite `e9a4aaddf90dda56a87235303cf00e4c2d3fb28725f68fd88772997dac905c18`; landmarks lite `d7fde8ac11f8ce03f8663775bfc323f4fc9f2a38062b4f4efa142874ef5b2a48`.
|
||||
|
||||
Another candidate is reducing repeated palm searches without losing second-hand acquisition. The [upstream HandLandmarker graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc) runs detection whenever tracked hands are fewer than the configured maximum, including one visible hand with a two-hand limit. This is a profiling lead, not a measured attribution in the packaged Android runtime. Simply setting the limit to one would remove the required two-hand behavior; implementing a controlled search cadence would require a different graph or pipeline. The existing bounded latest-frame queue and VIDEO tracking already avoid unbounded backlog and unconditional IMAGE-mode detection.
|
||||
|
||||
## Brief visual hold and remaining detection gaps (2026-10-16)
|
||||
|
||||
After the user reported recovery following cooling, a 16-second CPU sample kept the same Android process (10308) and SteamVR session (`2026-10-16-29-steam`). Head tracking was fresh in 2,851 of 2,857 channel samples (99.8%), but the left hand was fresh in only 22 samples (0.77%), representing six distinct poses. Those poses first reached the monitor 162.2–198.3 ms after observation. Right-hand output was absent during the requested left-hand-only test. Diagnostic snapshots alternated between camera/head synchronization waits and no detected hands; measured inference in the snapshots that ran detection ranged from 76.1 to 136.8 ms. This is not a controlled cool-device benchmark: at the end Android reported thermal status 4 and the current HAL SKIN sensor read 51.5 C. Evidence: `build/spatial-live-hands-cooled-flicker.json` and `build/cooled-flicker-measure.log`.
|
||||
|
||||
The driver previously invalidated its rendered controller immediately whenever an inference result contained no hand, even if the previous observation was still younger than the requested 200 ms cutoff. It now caches only the most recent accepted optical transform for visual continuity through those short gaps. Expiry remains measured from the original camera observation; a missed result never extends it. Optical input processing remains separate and releases trigger/grip immediately on a miss. Reacquisition still requires a released gesture. Head loss, connection/origin changes, disconnect, and switching back to gamepad-positioned controllers clear the cache. Missing optical tracking no longer suppresses independently available gamepad menu and locomotion inputs. This is a partial flicker mitigation, not a remedy for the longer detection gaps or old poses measured above.
|
||||
|
||||
Native regressions pass for immediate gesture release during a held visual pose, its unchanged timestamp and inclusive 200 ms boundary, rejection after expiry, rearming, head loss/recovery, origin/connection changes, disconnect, mode changes, malformed/future/out-of-order samples, and gamepad availability during optical loss. The other native rendering, fence, and scheduling regressions pass. SDK-enabled and SDK-free Android packaging and runtime source/output freshness checks pass. Android application source was unchanged in this step; the existing 73 passing Android test results were checked during installation rather than rerun. Evidence: `build/hand-visual-hold-native.log`, `build/hand-visual-hold-stage.log`, `build/hand-visual-hold-build.log`, and `build/hand-visual-hold-sdkfree.log`.
|
||||
|
||||
Installed artifact: `build/DroidDeck-hand-visual-hold.apk`, SHA-256 `977b993433803f3d8abbd639ca636b874367467866b58f7af75a07a46b20daa2`. Its packaged ARM64 driver matches the tested build, SHA-256 `2d7ff8647f210248f6ea4b68e9b05e21ed7ebcc74da601da0f1d854067b7ab82`. Installation occurred with the session idle and preserved the calibration/profile files. CPU selection and the 200 ms setting were confirmed across two app processes. The glasses were disconnected during these checks, so calibration was not loaded and detectors were not initialized; this does not establish a live improvement. SteamVR was not relaunched. Evidence: `build/hand-visual-hold-install.json` and `build/hand-visual-hold-live.json`.
|
||||
|
||||
The user explicitly authorized one new diagnostic stereo pair and separately authorized copying that pair and timing metadata to the local workspace. Both 640 by 480 images visibly contain the palm, near the outer edge of the left image. The saved pair was not synchronized to a head pose. Desktop MediaPipe 0.10.21 replay of the current full model failed to detect either hand at palm-detection thresholds 0.6, 0.5, 0.4, and 0.3, with landmark-presence confidence held at 0.6. Undistorting the images before detection did not recover this sample. Lowering both confidence thresholds to 0.3 also failed. A fixed right-side image tile detected a left hand in only one camera, insufficient for stereo reconstruction. The earlier authorized pair still detected the left hand in both cameras and passed the unchanged stereo geometry gate in the same replay. These static diagnostic experiments do not justify changing thresholds, shipping a fixed crop, or claiming general detection improvement. No inference model, thresholds, or stereo quality gates were changed. Local artifacts: `build/flicker-pair-replay.json`, `build/flicker-pair-scale.json`, and their scripts/logs.
|
||||
|
||||
The earlier CPU crash investigation also observed Vulkan device loss with hand inference disabled. A temporary `TU_DEBUG=noubwc` session reproduced it and was stopped; the flag was not persisted and no production graphics change was made. Cooling may have contributed to the user's reported recovery, but neither overheating nor inference delegate choice is established as the sole crash cause.
|
||||
|
||||
## Half-rate stereo input requested by the user (2026-10-16)
|
||||
|
||||
The supplied SDK 2.4.0 Carina header documents 25 Hz camera/pose callbacks and exposes automatic/manual exposure controls, but no tracking-camera frame-rate setter. DroidDeck now selects every other callback for its own stereo pixel copies while hand controllers are enabled, yielding approximately 12.5 pairs per second with the observed 25 Hz input. The gate executes before native buffer resize/copy. Skipped pairs do not reach JNI transfer, grayscale/RGBA conversion, or hand inference. Vendor camera acquisition/VIO, pose callbacks, and the existing head polling interval are unchanged. This is software input decimation, not a firmware camera-rate change.
|
||||
|
||||
Calibration collection, distance validation, explicit diagnostic capture, and preview-only use bypass the half-rate gate. The first frame after a gate-mode change or camera-source reset is accepted. The latest eligible pending pair still replaces older eligible pending work. Camera-source/receipt timestamps and the existing 200 ms hand expiry are unchanged. JNI metadata carries the incoming frame index associated with the copied pair and the cumulative intentional skip count, so diagnostics continue to report the true incoming camera rate. `cameraFrameDivisor: 2` identifies hand input at half rate, and `rateLimitedFrames` is separate from processing-related `dropped` frames. The setup UI explains both rates and counters.
|
||||
|
||||
The native regression passed 250 source callbacks producing 125 copied pairs, full-rate setup, repeated configuration, source reset, invalid timestamps, and reconnect reset. All 75 targeted Android tests passed, including half-rate camera-clock mapping and UI separation of incoming rate versus intentionally skipped frames. All 13 runtime-preparation tests passed, including classifying the new native helper as locally compiled while rejecting unrelated donor-native reuse. SDK-enabled and SDK-free builds and runtime freshness checks passed. Tests: `tools/tests/test_stereo_frame_gate.cpp`; logs: `build/hand-half-rate-build.log`, `build/hand-half-rate-sdkfree.log`, `build/hand-half-rate-runtime-tests.log`.
|
||||
|
||||
Installed SDK-enabled artifact: `build/DroidDeck-hand-half-rate.apk`, SHA-256 `c02d56e40802de5c5e9d9750234b24a85462652f472729fbc9696935654f95fb`. The installed hash was verified. The SteamVR driver is unchanged from the tested visual-hold build (`2d7ff8647f210248f6ea4b68e9b05e21ed7ebcc74da601da0f1d854067b7ab82`); the Android SDK bridge and app were rebuilt. Installation preserved app data and occurred with SteamVR stopped. CPU selection and the 200 ms expiry were confirmed across two processes. The glasses were disconnected, so the live half-rate cadence, inference cost, thermal benefit, and controller availability have not been measured. Evidence: `build/hand-half-rate-install.json`.
|
||||
|
||||
At 12.5 Hz, eligible pairs are approximately 80 ms apart. The earlier CPU inference snapshots already took 76–137 ms, so reducing input is not expected to halve total CPU load. It may save image-copy/preprocessing work and reduce inference load when the worker is fast enough, while also increasing frame waiting time and potentially reducing chances to recover a missed detection. It does not guarantee a successful observation or a visible response within 100 ms. A controlled comparison at matching temperatures is required before claiming smoother controllers or less overheating.
|
||||
|
||||
## SteamVR shared-eye image import correction (2026-10-16)
|
||||
|
||||
After the half-rate installation, sessions 31 and 32 hit `VK_ERROR_DEVICE_LOST` approximately 11.7 and 10.8 seconds after READY. DroidDeck's Android process and native 6DoF remained alive; CPU inference, a camera divisor of 2, approximately 25 Hz incoming cameras, and intentionally skipped frames were confirmed. This establishes live operation of the input limiter, but no thermal/performance benefit. The Android crash buffer contained an older VITURE abort in another process, not a new app crash. Evidence: `build/half-rate-crash-state.json`, `build/half-rate-crash-glasses.json`, and `build/half-rate-crash-guest.json`.
|
||||
|
||||
A temporary `DROIDDECK_VR_IMPORT_FORMAT_LIST=0` test ran 49 seconds without device loss with hands disabled. A second test with CPU hands was stopped after the user reported a grey, degraded image. The override is rejected and was not persisted. Restoring the normal import path reproduced device loss in session 35. Neither a successful launch nor removal of image-format metadata is an acceptable fix.
|
||||
|
||||
The normal-path Vulkan trace identifies an external-memory image-description mismatch: SteamVR exports each 1728 by 1080 eye image as format 43 (sRGB), flags `0x8`, usage `0x97`, with view formats 43/37, whereas the presenter imported it as format 37 (UNORM), flags `0x8`, usage `0x85`. Dedicated OPAQUE_FD imports require matching image descriptions. The presenter now retains the exported format, rendering/transfer usage, and view-format list. Presentation selects the corresponding sRGB or UNORM surface format, so an sRGB blit both decodes and re-encodes instead of darkening the display. It rejects incompatible surface encodings rather than silently selecting one, and checks imported-memory binding failures. The existing SteamVR compute-access color fix remains intact. Trace: `build/crash-import-trace/guest.json`. Requirement: [Vulkan dedicated allocation, VUID 01878](https://docs.vulkan.org/refpages/latest/refpages/source/VkMemoryDedicatedAllocateInfo.html).
|
||||
|
||||
The ARM64 build and native regressions pass, including a new regression against the captured SteamVR image description and display-format selection/rejection cases. SDK-enabled and SDK-free APK builds and runtime freshness checks pass. The prior 75 Android tests remain applicable; Android source was unchanged for this graphics correction. Installed SDK-enabled artifact: `build/DroidDeck-shared-eye-format.apk`, SHA-256 `c573169cd46b20b70d704cd92f9f07550d5aa128d0c4300c1c52e635f04958fb`; driver SHA-256 `51dbd719e56c43df6c37edef2d7f426e91183b2b8489285349df19c322090f3b`. Installation preserved the verified calibration, CPU inference, and 200 ms cutoff across two app processes. Logs: `build/shared-eye-format-{native,stage,build,sdkfree,install}.log` and `build/shared-eye-format-install.json`.
|
||||
|
||||
Two subsequent sessions completed 121.9 and 122.7 seconds after READY without a new device-loss event. All sampled head poses were valid, CPU inference stayed active, frame processing advanced, and Android PID 659 remained unchanged. Session 36 initially allocated 1080 by 1080 eyes; its trace confirms identical exported/imported format 43, usage `0x97`, and formats 43/37. Session 37 ran without tracing and allocated the original 1728 by 1080 eyes, removing the lower-resolution confound. Both used a 3840 by 1200 sRGB presentation surface (format 50) and showed approximately 600 frames per ten-second interval after startup. The cumulative compositor log contains older device-loss messages; filtering to the new sessions found no fresh device-loss error. Occasional `WaitForAcquire` warnings remain. Android thermal status reached 4, with current HAL skin temperatures 46.7 and 48.4 degrees Celsius after the respective runs; these are not controlled thermal comparisons.
|
||||
|
||||
These are startup stability checks, not a twenty-minute gameplay/performance pass or proof of every crash's cause. Both test sessions were stopped; all temporary environment overrides and the trace marker were cleared, with calibration and CPU hand settings verified afterward. Evidence: `build/shared-eye-format-live/`, `build/shared-eye-format-relaunch/`, `build/shared-eye-log-audit.json`, and `build/shared-eye-final.json`.
|
||||
|
||||
The user subsequently confirmed normal colors but reported another crash as soon as a hand appeared. Session 38 logged fresh Vulkan device loss at 08:44:25 UTC, about 42 seconds after READY, with a burst of geometry pipeline creation immediately beforehand. Recovery stopped the session, and Android later recorded SIGKILL for the app process; the initial failure was in SteamVR's compositor. The color correction is accepted, but the crash is not fixed. In session 39, the dashboard remained alive with valid head tracking and no detected optical hands, then failed less than four seconds after switching to the existing gamepad controller poses at 20.47 seconds. That reproduces the controller-visibility trigger independently of reconstructed hand coordinates. Evidence: `build/hand-visible-crash/`, `build/hand-visible-exits.txt`, and `build/controller-visibility/`. The preceding two-minute tests did not validate this transition.
|
||||
|
||||
## Visible-controller crash and compositor completion guard (2026-10-16)
|
||||
|
||||
An opt-in diagnostic waited for the compositor's graphics queue after each submission. Session 40 repeated the same optical-to-gamepad-controller transition at 18.01 seconds and stayed alive through 80.94 seconds, with approximately 600 presented frames per ten-second interval. Session 39 without the diagnostic failed less than four seconds after this transition. This supports a GPU synchronization problem in the visible-controller rendering path; it does not establish a specific bad hand sample or thermal cause. An [upstream SteamVR report](https://github.com/ValveSoftware/SteamVR-for-Linux/issues/952) describes premature command-buffer/descriptor-pool reuse, but equivalence to this device's failure is not directly proven. Diagnostic evidence: `build/controller-submit-idle/` and `build/submit-idle-install.json`.
|
||||
|
||||
The shipped experimental guard applies only inside `vrcompositor`. Each `vkQueueSubmit` or `vkQueueSubmit2` waits for its actual completion fence before returning. If SteamVR supplied its own fence, that fence remains attached to the original submission and a following empty submission signals DroidDeck's separate fence. Waiting is bounded to one second; timeout fails the submission path instead of reporting success. The tracker keeps the fence reserved during this wait, preventing the completion worker from resetting/reusing it; a timed-out but submitted fence stays pending until completion or device failure. Other processes retain asynchronous submission. This is a conservative workaround that reduces CPU/GPU overlap, so gameplay frame-time and long-duration acceptance remain required. The temporary `DROIDDECK_VR_SUBMIT_IDLE` probe was removed.
|
||||
|
||||
Native regressions cover both submission APIs, caller-owned fences, submission/marker failures, successful completion, device loss, timeout retention, tracker allocation failure, and bypass outside the compositor. Existing native rendering, spatial, optical-controller, synchronization, and scheduling checks also pass. SDK-enabled and SDK-free builds and runtime freshness checks pass; the prior 75 Android tests apply because Android source was unchanged. Installed APK: `build/DroidDeck-compositor-fence.apk`, SHA-256 `27d44a138d1f6bf6a0698abcd0a48107d31815c8e57a9f7ca07113fabb213195`. Installation preserved the verified calibration, CPU inference, 200 ms hand expiry, and half-rate camera input. Evidence: `build/compositor-fence-{native,stage,build,sdkfree,install}.log` and `build/compositor-fence-install.json`.
|
||||
|
||||
Session 42 repeated the controller-visibility regression on the installed bounded-fence build. Head tracking remained valid; switching to the existing gamepad controller poses at 17.55 seconds did not cause device loss through the final sample at 58.92 seconds. The presenter reported 593–601 frames per ten-second interval, with no late frames in those counters. Optical hands were absent before the switch, so this passes the reproduced synthetic-controller visibility trigger but does not yet validate an actual optical hand appearing. The successful session was left open with CPU optical tracking restored for that check. Evidence: `build/controller-fence-guard/` and `build/controller-fence-guard.log`.
|
||||
|
||||
The built, packaged, and installed Vulkan layers all have SHA-256 `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`; no pending or persistent diagnostic environment override remained. The live audit was repeated after the session ended and still confirmed the installed layer, saved calibration, CPU selection, and enabled hand-controller setting. Evidence: `build/compositor-fence-live-audit.json`.
|
||||
|
||||
## Optical appearance confirmed; flicker and pinch remain unresolved (2026-10-16)
|
||||
|
||||
The user confirmed that optical controllers appeared without freezing, but continued to flicker and could not click. This is a successful user-observed optical appearance on the compositor-fence build, not a sustained tracking or interaction pass. The color correction remains accepted.
|
||||
|
||||
A subsequent 16-second CPU baseline in session 43 sampled 2,846 coherent channel snapshots, all with fresh head tracking and none with a valid hand. The requested deliberate open/pinch/release sequence had not yet been confirmed, so absence of hands in this baseline cannot establish detector failure on a visible palm. Diagnostic samples measured 111.7–154.1 ms stereo inference; three observations exceeded the unchanged 200 ms cutoff, with completion ages 204.9–221.5 ms. Android stayed in process 32010 and the same READY SteamVR session. Thermal status was 4, with current HAL skin temperature 51.8 C at the end; this is not a controlled cool-device performance comparison. Evidence: `build/spatial-live-hands-pinch-baseline.json` and `build/pinch-baseline.log`.
|
||||
|
||||
The driver emits `/input/trigger/value` and `/input/trigger/click` from optical pinch. Any invalid/stale hand or head sample releases inputs and disarms gestures until an observed open/released hand, while the visual pose may remain only until its original observation reaches 200 ms. Repeated observation loss can therefore interrupt clicking as well as controller visibility. No deliberate pinch was measured in the new baseline, so this is an explanation of the implemented behavior, not proof of the user's missing-click cause. The next measurement records hand timestamps, validity transitions, trigger strengths, grip, and connection/origin changes without camera images (`build/measure-pinch.py`). No inference thresholds, model, expiry, delegate default, or graphics behavior were changed in this follow-up.
|
||||
|
||||
## Palm continuity and independent gestures implemented (2026-10-16)
|
||||
|
||||
Hand reconstruction now validates each triangulated landmark independently under the unchanged depth and 3-pixel reprojection limits. A controller requires wrist and four palm knuckles; pinch and grip each have separate required fingertips and validity. A rejected finger no longer removes an otherwise accepted palm. Board calibration and distance checks still reject a set containing any invalid point. Protocol version 2 retains the 176-byte layout and carries explicit per-gesture validity bits; old protocol versions are rejected. The native driver releases only the uncertain gesture and requires its observed release before rearming. Unknown finger data cannot arm a click.
|
||||
|
||||
The driver smooths palm translation and rotation with a 15 ms time constant. Three consecutive observations with consistent velocity can enable at most 30 ms / 2 cm of translation prediction. Prediction stops on missing/invalid observations and origin/connection/head changes; the original 200 ms observation expiry is unchanged. Pose filtering never generates or holds gesture input. Native regressions cover independent gesture loss/rearming, old-version rejection, quaternion sign equivalence, prediction bounds, loss holds, expiry, and origin resets. Android regressions cover rejected fingertip versus rejected palm behavior, independent pinch/grip validity, and unchanged metric/reprojection checks.
|
||||
|
||||
Both full and full-palm/lite-landmark models are packaged and SHA-256 pinned. The lighter bundle is reproducible with `tools/build_hand_lite.py` from the documented original models and MediaPipe 0.10.21 metadata tooling. Diagnostics offer temporary model and half/full-input controls; CPU, full model, and half-rate input remain defaults pending a live comparison. An opt-in debug command benchmarks existing saved stereo pairs on CPU without capturing images or publishing replayed hand poses. It clears live hands during replay, closes both temporary detectors, and resumes the selected live model afterward.
|
||||
|
||||
All 259 selected Android tests pass with no failures/errors/skips, including the glasses package and spatial-setup UI. The ARM64 native tests, SDK-enabled and SDK-free Android builds, and runtime freshness checks pass. Installed APK: `build/DroidDeck-hand-reliability.apk`, SHA-256 `38b9b67171b37b0bf80b540da165894ff4cbb24b609ae62edd829ecf618d01e3`. Driver SHA-256: `56b3a3eeaa96160e5cbf740f35b701f935b610417dbbba0aa2965ec109d5418a`. The graphics synchronization layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. Installation preserved the verified calibration and CPU selection across processes 2843 and 3996. The built, packaged, and deployed driver/layer match; the Android publisher produces a version-2, 176-byte spatial file. Evidence: `build/hand-reliability-{tests,native,stage,build,sdkfree,install,audit}.log` and corresponding install/audit JSON.
|
||||
|
||||
The Fold7 successfully loaded both model bundles using Android Tasks 1.0.0 on CPU. Saved-pair replay ran full/lite/full, each with four warmup and twenty measured pairs, outside SteamVR in the same process. Times below cover stereo inference, excluding model construction, camera acquisition, geometry, publication, and display latency.
|
||||
|
||||
| Saved pair / model | Median inference | p95 inference | Valid palms |
|
||||
|---|---:|---:|---:|
|
||||
| Previously detectable pair / full before | 104.95 ms | 107.95 ms | 20/20 |
|
||||
| Previously detectable pair / lite landmarks | 90.03 ms | 95.54 ms | 20/20 |
|
||||
| Previously detectable pair / full after | 103.89 ms | 106.20 ms | 20/20 |
|
||||
| Harder pair / full before | 60.77 ms | 63.88 ms | 0/20 |
|
||||
| Harder pair / lite landmarks | 62.64 ms | 66.53 ms | 0/20 |
|
||||
| Harder pair / full after | 62.48 ms | 65.74 ms | 0/20 |
|
||||
|
||||
The lite candidate is approximately 13–14% faster on the detectable static pair. Both models recognize a released pinch/grip and retain their validity on all twenty repetitions; their first measured palm positions differ by about 1.9 mm. This is agreement between models on one static observation, not ground-truth accuracy or stationary-jitter validation. Neither detects the harder saved pair, so reducing landmark cost does not solve missed palm detection. Thermal status ended at 2, with current HAL skin temperature 41.4 C. Evidence: `build/hand-reliability-benchmark-idle.json` and `.log`.
|
||||
|
||||
SteamVR attempts 44 and 45 were suspended by Android with the screen off/lock screen showing before their first displayed frame. Wake/dismiss requests did not dismiss the lock screen. A user unlock is required for the graphics-workload model comparison and a fresh optical open/pinch/release check; neither is counted as a pass. The full model remains the default until that comparison can be completed. The changes are installed as an experimental candidate; reduced live flicker and successful dashboard clicking have not yet been established.
|
||||
|
||||
## Concurrent CPU inference and controller cadence (2026-10-16)
|
||||
|
||||
Left and right CPU inference now run concurrently with separate image buffers and a stable owner for the right detector. Both calls finish before either frame buffer can be reused or either detector closed, including failure and interruption paths. GPU inference remains sequential and opt-in. Controller presentation now approaches the filtered pose on each driver update with a 15 ms time constant, rather than moving only when a new inference result arrives. Missing observations stop prediction and hold the last displayed transform only until its original 200 ms expiry. Gesture validity, input release, head freshness, and connection/origin resets are unchanged.
|
||||
|
||||
All 263 selected Android tests pass without failures, errors, or skips, including four concurrent-worker lifecycle tests. Native spatial regressions cover intermediate render updates, normalized rotations, loss holds, and unchanged expiry. The ARM64 build, SDK-enabled and SDK-free APK builds, and packaged-runtime freshness checks pass. Installed APK: `build/DroidDeck-hand-cadence.apk`, SHA-256 `7df588f0eacdbf4f5ddaae91c511e415b01fe39f37100c012e8f8da3fced52e0`. Driver SHA-256: `195261f27043b260052873a3b083c436457763caa5ebb7bcd1f52708e426bf11`. The graphics layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. Built, packaged, and deployed binaries match; the spatial channel remains version 2, 176 bytes. Installation preserved CPU selection and verified calibration across processes 13863 and 14542. Evidence: `build/hand-cadence-{native,stage,build,sdkfree,install,audit}.log` and corresponding install/audit JSON.
|
||||
|
||||
A saved-pair comparison ran under SteamVR in one Android process/session using the previously authorized pair `64390821246320`. Each configuration had four warmup and twenty measured pairs. Times cover stereo inference, not acquisition, reconstruction, publication, or display latency.
|
||||
|
||||
| CPU configuration, in test order | Median inference | p95 inference | Valid left palms |
|
||||
|---|---:|---:|---:|
|
||||
| Full, sequential before | 202.74 ms | 286.60 ms | 20/20 |
|
||||
| Full, concurrent | 133.19 ms | 162.33 ms | 20/20 |
|
||||
| Lite landmarks, concurrent | 104.18 ms | 165.52 ms | 20/20 |
|
||||
| Full, sequential after | 176.31 ms | 208.47 ms | 20/20 |
|
||||
|
||||
Concurrent full inference had a 24–34% lower median than the surrounding sequential runs. Its first palm position matched the full sequential result exactly; the lite result differed by about 1.9 mm. All repetitions retained released pinch/grip validity. These are repeated static-pair results, not physical accuracy, dynamic coverage, or clicking acceptance. Android thermal status was 4 throughout, with current HAL skin temperature increasing from 48.8 to 50.6 C. The ordered test was not temperature controlled. The lite model's lower median did not improve its p95 over concurrent full inference; the full model remains the default. Evidence: `build/hand-reliability-benchmark-steamvr-parallel.json` and `build/hand-cadence-benchmark.log`.
|
||||
|
||||
For the live tests, the user confirmed an open left palm approximately 12–18 inches away, moving slowly side to side. Each sample lasted 16 seconds. All coherent head samples were fresh. The sequential baseline used session 50; both concurrent-input samples used process 14542 and session 51 without a restart.
|
||||
|
||||
| Live configuration | Coherent reads | Reads with fresh left hand | Distinct left observations | Left arrival age, median / p95 | Sampled inference median |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Previous sequential CPU, 12.5 Hz input | 2,959 | 29.47% | 113 | 160.16 / 196.23 ms | 117.03 ms |
|
||||
| Concurrent CPU, 12.5 Hz input | 2,941 | 91.77% | 186 | 74.94 / 84.81 ms | 66.06 ms |
|
||||
| Concurrent CPU, 25 Hz input | 2,932 | 64.36% | 152 | 91.93 / 109.69 ms | 62.55 ms |
|
||||
|
||||
The freshness percentages measure sampled shared-channel availability; they are not the acceptance target's percentage of valid camera observations and cannot pass or fail that 95% gate. Arrival age measures the original observation's age when first seen by the channel monitor, not externally measured physical-motion-to-visible-controller latency. Natural hand motion, detected hand count, and temperature were not held constant. Both cameras often reported two hands during the requested left-palm test, so right-channel availability is not evidence of verified right-hand identity or aiming. Thermal status was 4, with HAL skin temperatures of 47.5–47.7 C for the baseline, 50.5 C for limited input, and 50.7–50.8 C for full input. These short, ordered samples support further testing of the candidate, not a causal attribution of the entire improvement or a general conclusion that full-rate input is worse. Full-rate input did not improve observed availability in this comparison; the original 12.5 Hz input cap was restored. The SDK cameras continue at approximately 25 Hz.
|
||||
|
||||
The user reported improved but still choppy motion, a reversed right-controller pointing direction while the left behaved as expected, and less flicker with both hands detected than with one. Right-hand orientation correction and investigation of the one-hand/two-hand difference are in progress; neither is validated by this candidate's measurements. No new camera images were saved. Session 51 later suspended and exited with status 0; a subsequent audit found session 52 READY, CPU/full/concurrent inference, the 12.5 Hz cap, verified calibration, and no temporary graphics overrides. Evidence: `build/spatial-live-hands-choppy-baseline-confirmed.json`, `build/spatial-live-hands-cadence-{half,full}.json`, `build/hand-input-comparison.json`, `build/hand-cadence-summary.json`, and `build/hand-cadence-audit.json`.
|
||||
|
||||
## Right-hand orientation and camera-clock continuity (2026-10-16)
|
||||
|
||||
The user's report of a reversed right controller exposed a handedness-independent palm basis. The right hand now reverses its lateral axis before constructing the rotation, giving mirrored palms the same normal convention while preserving the existing left mapping and wrist-to-finger direction. Both live reconstruction and saved-pair replay pass an explicit hand side. Tests cover mirrored palms, 180-degree and arbitrary rotations, camera-to-world transforms, and unchanged gesture values and timestamps. These tests verify the mathematical correction; the user subsequently confirmed that the right-controller orientation is correct on the installed build. Measured aiming accuracy across poses remains pending.
|
||||
|
||||
The one-hand/two-hand flicker difference has a plausible processing explanation: the [upstream MediaPipe video graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259) repeats palm detection until the previous tracked hand count reaches the configured maximum. DroidDeck requests two hands. Per-camera inference durations and current/previous raw hand counts are now reported separately from confidence-filtered counts so this hypothesis can be measured. No hand-count restriction, confidence reduction, or synthetic second hand was introduced.
|
||||
|
||||
A separate clock defect reset camera synchronization whenever the worker skipped more than 200 ms of source frames, even when host and source clocks advanced consistently. A processing stall could therefore add another 16 observations of warmup (roughly 1.3 seconds at 12.5 Hz). The mapping now survives an aligned long gap; offset discrepancies above 20 ms, invalid timestamps, or nonmonotonic source/receipt times still require warmup. Existing short-gap rolling-minimum/p90 jitter handling is retained. Six regressions cover 240 ms and one-second gaps, resets, offset changes in both directions, jitter boundaries, and recovery. Hand observation expiry remains 200 ms and head expiry remains 100 ms.
|
||||
|
||||
All 271 selected Android tests pass without failures, errors, or skips. SDK-enabled and SDK-free builds and packaged-runtime freshness checks pass. Native code is unchanged from the tested cadence build. Installed APK: `build/DroidDeck-hand-continuity.apk`, SHA-256 `46b99d939e5c7063840ae669128b40ba60a794e62a30c1a14f38da27362d2855`; driver remains `195261f27043b260052873a3b083c436457763caa5ebb7bcd1f52708e426bf11` and the graphics layer remains `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. The installed APK hash matches, and processes 29580 and 30045 both initialized CPU inference and loaded the verified calibration. CPU/full/concurrent inference and the 12.5 Hz input cap remain selected. No new images were captured. Evidence: `build/hand-continuity-{build,sdkfree,stop,install}.log` and `build/hand-continuity-install.json`. Subsequent live results follow; sustained smoothness acceptance remains pending.
|
||||
|
||||
SteamVR reopened as session 54 and reached READY with valid head tracking. A live audit verified the built, packaged, and deployed driver/layer hashes, the version-2 spatial channel, saved calibration, CPU/full/concurrent inference, the 12.5 Hz cap, and the new per-camera diagnostic fields. This is an installation/startup check, not the pending deliberate one-hand/two-hand comparison or a visual orientation pass. Evidence: `build/hand-continuity-launch.log`, `build/hand-continuity-audit.log`, and `build/hand-continuity-audit.json`.
|
||||
|
||||
## Live hand-count comparison and remaining startup delay (2026-10-16)
|
||||
|
||||
The subsequent 44-second live comparison retained Android process 30045 and SteamVR session `2026-10-16-54-steam`. All 8,146 coherent head reads were fresh; the left hand was fresh in 97.30% of channel reads. The right hand was introduced after the left, and both cameras reported two hands from approximately 13 seconds. Consequently, the nominal first-20-second/remaining-test stages are not clean one-hand/two-hand windows. The whole-run right-hand availability of 71.00% includes the period when it was absent and must not be interpreted as right-hand detection accuracy.
|
||||
|
||||
Per-camera timings were grouped by each detector's previous raw hand count, which is the count relevant to the upstream video graph's palm-search decision. The smaller count group contained eight diagnostic snapshots per camera, and the two-hand group contained nineteen.
|
||||
|
||||
| Camera | Previous raw hands | Diagnostic samples | Median inference | p95 inference |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Left camera | 1 | 8 | 73.36 ms | 80.27 ms |
|
||||
| Left camera | 2 | 19 | 55.96 ms | 69.10 ms |
|
||||
| Right camera | 1 | 8 | 71.94 ms | 87.04 ms |
|
||||
| Right camera | 2 | 19 | 53.15 ms | 65.69 ms |
|
||||
|
||||
The observed per-camera medians were approximately 17–19 ms lower with two previously tracked hands. This is consistent with the extra palm-search work described above, but the ordered, natural-motion test does not isolate that work from pose, visibility, or workload changes. Thermal status remained 4, with HAL skin temperature 50.7 C at both endpoints. These internal inference timings do not measure physical-motion-to-visible-controller latency. The 97.30% sampled left-channel availability is also distinct from the required 95% valid camera-observation rate; neither acceptance gate is established by this run. No new camera images were saved. Evidence: `build/hand-count-live.json`, `build/hand-count-live.log`, and `build/hand-count-summary.json`.
|
||||
|
||||
The user confirmed the right-controller orientation correction and described single-hand tracking as smoother and more reliable, but said it takes a while after SteamVR starts to become smooth. Startup choppiness remains unresolved. This measurement did not isolate startup behavior, and it does not establish a warmup, thermal, synchronization, or graphics cause for that remaining delay. Longer controlled runs, an observation-based coverage measure, external latency measurement, and deliberate optical interaction checks are still required.
|
||||
|
||||
The user clarified that startup choppiness lasts about a minute or longer after the dashboard appears. Both CPU detectors, camera synchronization, and verified calibration were already active in process 30045 before SteamVR launched, so initial model loading alone does not explain the delay. Session-54 compositor logs, aligned to their Denver timestamps, record 138 shader-pipeline creations during seconds 20–69 after the launch request and another 12 during seconds 120–129. This identifies ongoing graphics initialization as a plausible source of CPU contention, not a measured causal explanation of all startup stutter. The current session remained READY; its later presenter counters were about 600 frames per ten seconds. No matched `vkerror`, device-loss, fatal, or segmentation messages appeared in the filtered session window, although a `WaitForAcquire` timeout remains. No startup scheduling, cache, graphics, or loading-screen changes were made. A correlated startup trace of hand latency and CPU/GPU workload is still needed. Evidence: `build/hand-startup-review.json` and `build/hand-continuity-install.json`.
|
||||
|
||||
## Gesture, single-hand scheduling, and motion update (2026-10-16)
|
||||
|
||||
The next candidate keeps CPU/full-model/concurrent stereo inference. Calibrated stereo palm reconstruction is unchanged; pinch and finger closure now use agreement between each camera's image landmarks. Invalid or conflicting finger geometry clears the affected gesture, and the driver still requires an observed release before rearming it. This avoids making a valid palm depend on successful triangulation of every fingertip, without claiming that a lost finger is released.
|
||||
|
||||
CPU inference now retains separate one-hand and two-hand video trackers for each camera. Three consistent stereo observations enable the one-hand path; a two-hand search is due after 240 ms, and loss or ambiguous handedness returns to search. The selected bank's raw counts and hand limit are reported. Both per-camera tasks finish before images or detectors are reused or destroyed. GPU remains optional and serial. The adaptive copy gate starts at 500 permille of source callbacks (12.5 Hz with 25 Hz cameras), with 600/800 permille available only after eight consecutive healthy samples meet processing and age thresholds. Slow discovery samples can prevent an upshift; the presence of this controller is not evidence of achieving 15 or 20 Hz. Camera capture and vendor head tracking retain their normal rate.
|
||||
|
||||
The driver removes the additional observation-level low-pass and retains one 15 ms rendering correction. Stable translation advances from the time a result reaches the driver, bounded by 80 ms and 2 cm, instead of exhausting prediction during inference. Capture timestamps still control velocity and expiry. Explicit loss freezes the displayed pose, immediately releases invalid inputs, and preserves the 200 ms hand expiry and 100 ms head expiry. This is bounded motion estimation, not additional measured hand observations.
|
||||
|
||||
Validation: all 306 selected Android tracking/UI tests pass, with no failures, errors, or skips. The ARM64 VR build and native regressions pass, including delayed/jittered observations at 4 ms driver ticks, stops, reversals, outliers, quaternion normalization, exact loss freeze, expiry, and origin/connection changes. Fractional camera-gate regressions pass with strict compiler warnings. SDK-enabled and SDK-free Android builds, 13 runtime-preparation tests, and runtime checksums pass. The loading/first-frame flow and graphics color path were not changed in this update.
|
||||
|
||||
Installed on the existing Fold7 as `build/DroidDeck-hand-smoothness.apk`, final SHA-256 `9508370c42870e60a0070abf61d99792df581436f802049b068b96eba95fa027`; driver SHA-256 `52a96238afedf68c38866588db8225d2bd82450ed13635a5673ca7f9e0c6835b`. Android requested USB access after the first candidate installation; the user accepted it. Final verification matched the installed APK hash and confirmed CPU selection, automatic rate selection, scheduled single-hand tracking, and the existing verified calibration in two separate app processes (6139 and 6989). CPU inference initialized in 6139; process 6989 was initially waiting for valid head tracking, so its configured CPU delegate was not yet initialized. The saved 25-view calibration and distance check were retained. Evidence: `build/hand-smoothness-{native,stage,final-build,sdkfree,runtime-tests,install}.log` and `build/hand-smoothness-install.json`. Deliberate moving-hand, grab/release, external latency, and sustained game-performance checks remain required.
|
||||
|
||||
Saved-pair replay of the previously authorized capture `64390821246320` compared parallel full-model search, scheduled full-model tracking, scheduled lite-model tracking, then a repeated full-model search baseline. Each row measured 20 frames after four warmup frames. The scheduled rows used the one-hand path for 13 of those frames. All rows retained a valid left palm and valid released pinch/grip in all 20 measured frames; the source image is a stationary open palm, so this does not validate moving hands or pressing gestures.
|
||||
|
||||
| Workload | Full search median / p95 | Scheduled full median / p95 | Repeated full search median / p95 |
|
||||
|---|---:|---:|---:|
|
||||
| App idle | 70.93 / 76.19 ms | 36.74 / 74.04 ms | 71.28 / 75.77 ms |
|
||||
| SteamVR | 91.48 / 99.26 ms | 46.92 / 94.71 ms | 86.39 / 100.62 ms |
|
||||
|
||||
Scheduled lite-model medians/p95 were 25.41/67.04 ms idle and 40.26/86.77 ms under SteamVR. The full model remains selected because a repeated static image cannot establish the lighter model's live tracking or gesture accuracy. The SteamVR comparison retained process 333 and session `2026-10-16-57-steam`; thermal status was 4 and HAL skin temperature rose from 49.5 to 51.0 C. These short ordered runs show lower typical inference cost on this saved pair, while discovery-frame tail latency remains. They do not establish external motion-to-controller latency, sustained game frame time, or reduced thermal throttling. No new camera images were captured. Evidence: `build/hand-reliability-benchmark-smoothness-{idle,steamvr}.json`.
|
||||
|
||||
The final adaptive policy uses average processing cost for normal budget changes so occasional discovery frames do not veto otherwise sustained headroom. Any single processing cost at least 80 ms, observation age at least 140 ms, or invalid tracking falls back to 12.5 Hz. Regression tests verify that a repeating 36/36/74 ms pattern can select 15 Hz but not 20 Hz, while continuous 70 ms work stays at 12.5 Hz. This change does not assert that the observed SteamVR workload qualifies for a higher rate.
|
||||
|
||||
Final deployment audit: SteamVR session `2026-10-16-58-steam` reached READY with a first frame and valid head tracking. Built, packaged, and deployed driver hashes match; the unchanged graphics-layer hash is `b94d5598131883586fa1d428178f5048669edc40529f3900ec1d0955a5378f89`. The version-2 spatial channel and verified calibration were present, with no temporary environment overrides. CPU/full/concurrent inference, scheduled discovery, and adaptive input were active. The audit snapshot showed one accepted palm with valid pinch/grip geometry, the one-hand inference bank, 47.10 ms stereo inference, approximately 25 Hz source cameras, and a 500-permille input limit. A single snapshot does not demonstrate reliable movement or actual pinch/grip presses. Evidence: `build/hand-smoothness-{launch,audit}.log` and `build/hand-smoothness-audit.json`.
|
||||
|
||||
## Stationary jitter and persistent input release (2026-10-16)
|
||||
|
||||
The user reported stationary controller bounce and a dashboard drag that remained attached after a controller flickered and the hand returned open. The original 16-second reproduction recorded 2,905 coherent channel reads with fresh head tracking throughout. The right hand was fresh in 812 reads; 64 distinct poses were recorded. Right-hand transitions included five samples above the trigger-click threshold, 63 valid released-grip observations, and no pressed grip. Both hands were invalid at the end of the sample. This does not by itself locate the missing dashboard release: the original driver lacked per-input publication diagnostics. Evidence: `build/spatial-live-hands-sticky-grab-baseline.json`.
|
||||
|
||||
Inspection of the installed Vive compositor binding showed that dashboard left-click/drag uses the trigger, with grip mapped to Back. The installed dashboard `GrabHandle` attaches the panel to the initiating hand and clears that attachment on a window mouse-up event; it has no tracking-loss cleanup in that component. A missing mouse-up is consistent with the report, but the exposed code does not prove that native laser-mouse processing discards releases when a pose becomes invalid. Resource evidence: `resources/config/vrcompositor_bindings_vive_controller.json`, SHA-256 `16f19a170608c9b3450e7fc980644220a5bd3a34e1bb88050e65f51c7600a2d1`, and `resources/webinterface/dashboard/chunk~8012d0c89.js`, SHA-256 `8a529bd96141480a18efa44b912a81a1e27ac1bf8bae6b2ab8945ebaf6da2e73`. No SteamVR resource was modified.
|
||||
|
||||
The driver now stabilizes translation and rotation independently against fixed stationary anchors, holding within 3 mm and 1 degree. Breakaway uses the existing 15 ms rendering correction; it does not subtract a dead-zone distance from every observation. Returning to a hold requires 240 ms inside tighter 1.5 mm / 0.5-degree bounds and a check against sustained directional drift. Position prediction stops while anchored. Native tests cover still jitter, quaternion sign flips, accumulated slow translation/rotation, fast breakaway, settling, and the existing delayed-motion and loss behavior. These thresholds are candidates; stationary jitter and fine aiming still need hardware confirmation.
|
||||
|
||||
A confirmed latest-snapshot race could hide a transient invalid hand/head observation from the driver. Protocol version 3 retains the 176-byte layout and uses the offset-36 word for four independent 8-bit loss counters: left trigger/grip, then right trigger/grip. Loss remains visible in subsequent snapshots even if the invalid snapshot was overwritten. Head gaps clear cached hands, captures preceding head loss cannot rearm inputs, and expiry/invalid-gesture transitions update only the affected counters. The driver clears matching input state on counter changes, rejects old or duplicate observations for rearming, and detects observation gaps beyond 200 ms. It publishes released trigger scalar/click and grip before pose loss, clears inputs before deactivation, and logs state/error transitions for hardware verification. No synthetic presses or fresh poses were introduced. Versions 1 and 2 are rejected, so the Android and driver binaries must be updated together.
|
||||
|
||||
All 317 selected Android tests and the native regressions pass. New tests deliberately skip invalid snapshots, preserve independent trigger/grip state, reject stale release observations, verify publication ordering, and retry rejected OpenVR component writes. SDK-enabled and SDK-free Android builds and runtime checksum verification pass. Installed APK: `build/DroidDeck-hand-stability.apk`, SHA-256 `0e3f4a140e36eb48e91f5937748b0e10afe3491a0236484b21e9a3faa3ca6766`; driver SHA-256 `cca5723189a77cde606d11afd55d80567e7f72398e3f50e706bee164cb91eb5f`. Calibration survived two app processes (17259 and 17867); CPU initialized in the latter. SteamVR session `2026-10-16-60-steam` reached READY with a first frame. Evidence: `build/hand-stability-{native,stage,build,sdkfree,install,launch,audit}.log` and `build/hand-stability-{install,audit}.json`. End-to-end dashboard release and visible stationary improvement remain pending the deliberate post-installation check.
|
||||
|
||||
## Outstanding acceptance
|
||||
|
||||
### Investigation and shared base calibration (2026-10-16)
|
||||
|
||||
The phone's current verified 640×480 pinhole profile was exported directly over authorized ADB. Its 1.1209395390660726-px fit, lens coefficients, stereo transform, and camera-to-head transform are now bundled unchanged as `app/src/main/assets/calibration/luma-ultra-640x480.json`, with the source device identifier removed. Asset SHA-256: `d160f0c81d0328255f9afa69bc61505e5aa2988a9bb7a9faaeb611598eee7faa`. A matching personal profile takes precedence; new or unidentified Luma Ultras use this base for experimental tracking. Shared geometry is not falsely marked validated on other devices. UI and diagnostic status identify the shared base separately from individually verified profiles.
|
||||
|
||||
Hand tracking now displays **Experimental** throughout the relevant controls and setup status. Per-observation diagnostics distinguish detection/association failures, missing palm landmarks, palm width/basis/motion rejection, stale data, and tracking loss. Connection-lifetime counters retain independent palm, trigger, and grip acceptance/loss evidence without counting repeated status polls. Tracking and input thresholds are unchanged. The historical trace analysis and prioritized improvement experiments are recorded in `luma-hand-tracking-investigation.md`; no new moving-hand or game-usability pass is claimed.
|
||||
|
||||
All 388 focused glasses and setup/UI tests pass, including the new calibration selection and diagnostic-counter regressions. Exact geometry comparison against the phone export passes. The SDK-enabled APK builds with runtime freshness verification enabled. After the concurrent upstream Joy-Con merge, the already rebuilt VR outputs were validated against current source/header/configuration hashes and restaged with the updated helper scripts; unrelated prepared binaries retained their hashes. All 150 packaged runtime outputs match the resulting manifest. The packaged base calibration matches its source exactly, and the APK includes the new diagnostics. Artifact: `build/DroidDeckXR-luma-experimental.apk`, SHA-256 `11235040ad8c34abfd23482beb552fb09db4d9429703fbc8d7c5090b98da940e`. Runtime provenance is in `build/luma-hand-runtime-restage-20261016/provenance.json`. This change has not been installed on the phone. Logs: `build/luma-investigation-tests.log`, `build/luma-investigation-build.log`; metadata: `build/luma-investigation-validation.json`. The source phone profile backup stays in ignored `build/luma-ultra-phone-calibration-20261016.json`.
|
||||
|
||||
| Gate | Current evidence or blocker |
|
||||
|---|---|
|
||||
| Measured head axes and meter scale | Pending measured translation, leaning and crouching |
|
||||
| Floor and separate origins on hardware | Automated math passes; physical crouched recenter and picture switching remain pending |
|
||||
| Screen anchoring and plane crossing | Shader and parameter tests pass; visual movement and plane-crossing checks remain pending |
|
||||
| Camera image orientation | Upright 640 by 480 images and visible palms verified in one authorized diagnostic pair |
|
||||
| Calibration quality | 25-view set passes fit/alignment checks with 1.12094 px held-out reprojection error. Repeated distance check passed: 12.13 inches measured against 12.00 selected; SDK-identified verified profile survives app restarts |
|
||||
| Hand pose and aiming | User confirms corrected right-controller orientation and smoother, more reliable single-hand tracking. Latest 44-second comparison had 97.30% sampled left-channel availability; startup choppiness, measured aiming accuracy, and sustained continuity remain pending |
|
||||
| At least 95% valid hands at 30–80 cm | Camera-observation acceptance rate not measured; sampled channel availability above is a different metric |
|
||||
| At most 5 cm hand error and 2 cm jitter | Not measured |
|
||||
| External p95 controller latency below 100 ms | Not measured |
|
||||
| Hardware input release and reacquisition | State-machine tests pass; user-requested 200 ms hand expiry / 500 ms reacquisition not measured physically |
|
||||
| Twenty-minute game performance | No controlled baseline or CPU/GPU comparison |
|
||||
| Dashboard and Angry Birds VR interactions | Not validated with optical controllers |
|
||||
| Occlusion, crossed hands, lighting and thermal load | Pending |
|
||||
| Physical USB reconnect during inference | Generation/lifecycle guards implemented; no calibrated-inference unplug test |
|
||||
|
||||
The user confirmed corrected colors, optical controllers appearing without a freeze, corrected right-controller orientation, and smoother, more reliable single-hand tracking. The latest timing comparison is consistent with additional palm-search work when only one hand is tracked. Startup choppiness remains unresolved, with no established cause. A confirmed open/pinch/release sequence is still needed to distinguish missing/expired observations from gesture recognition or input-path failures. Full palm detection with the lite landmark model remains an experiment; the current full model stays selected. Controlled temperature-matched SteamVR measurements, right-hand correspondence across poses, and measured aiming validation remain necessary. The compositor guard passed the reproduced gamepad-controller visibility trigger; long-duration and GPU-inference stability remain unvalidated. Delegate and process identity must be checked across each measurement because graphics recovery recreates the hand engine; it now selects CPU by default and discards any temporary GPU override. The saved calibration no longer needs repeating after an identified-device reconnect. The combined feature must remain experimental until hand tracking and the remaining acceptance measurements pass. Native 6DoF availability alone is insufficient for release.
|
||||
|
||||
## Fork migration and latest feedback (2026-10-16)
|
||||
|
||||
The post-installation check did not establish a fix: the user could not start the dashboard drag and reported stationary controller jitter about the same. Dashboard interaction, release after tracking loss, and visible stationary stability remain unresolved.
|
||||
|
||||
At the user's request, the 63 uncommitted XR files were moved from `C:/Users/conno/Desktop/DroidDeck` to `C:/Users/conno/Desktop/DroidDeckXR` on branch `feat/3dfixes-6dof`. A three-way merge preserved the fork's resolution and menu changes; 13 non-overlapping fork files remained unchanged. Original files and patches from both checkouts are preserved in `build/xr-migration-20261016`. The source checkout is clean after hash-verified removal of the migrated edits.
|
||||
|
||||
The merged fork passes 373 selected Android tests and 13 packaging tests, plus SDK-enabled and SDK-free builds. The existing tested ARM64 VR outputs were reused after validating their source, SDK-header, configuration and binary checksums; they were not rebuilt during migration. All 133 prepared runtime outputs match the SDK-enabled APK. The new APK is `build/DroidDeckXR-migrated-sdk.apk`; it has not been installed. See `build/xr-migration-verification.json` for hashes and the retained historical runtime-provenance boundary. Subsequent work belongs in DroidDeckXR.
|
||||
|
||||
## Connected-device isolation review (2026-10-16)
|
||||
|
||||
Reviewed the outstanding DroidDeckXR changes together, including optical and Joy-Con tracking, calibration, display-mode routing, recovery overlays, graphics fixes, RGB diagnostics, and build/runtime provenance. Ultra controls now require a controllable, connected device whose Carina capability was confirmed by the VITURE SDK. Model names and saved Ultra preferences alone do not enable them. The experimental position/hand/Joy-Con controls, floor/eye-height setup, 6DoF label, calibration activity, recovery overlay, and RGB diagnostic entry follow this capability. Calibration closes when the connection disappears, and launching it directly without an Ultra is rejected.
|
||||
|
||||
Saved Ultra controller preferences no longer suppress the phone-camera permission or controller path for other glasses. Routing is refreshed on device changes as well as setting changes. Opening a non-Carina link clears the retained spatial-channel mode; temporary Ultra tracking loss still invalidates poses without silently switching to fixed-height legacy tracking. Tests cover disconnect, reconnect, unconfirmed model names, non-Ultra VITURE, XREAL and RayNeo states, rejected mode changes, and the native-channel flags.
|
||||
|
||||
The optional UVC directory now accepts an explicitly empty Gradle override, requires its license files before packaging, and is included in the local native-source preparation audit. All 456 selected Android tests pass with no failures, errors or skips. An earlier Compose calibration timeout passed in isolation and in the final complete selection without changing the test or product behavior. The ARM64 VR rebuild and native regressions pass. Linux packaging/helper tooling reports 47 passing tests; two shader compilation tests were skipped because glslangValidator is unavailable (this change set does not modify shader source or its generated header).
|
||||
|
||||
Both the SDK/UVC-enabled and SDK/UVC-free Android builds pass, and all 150 prepared runtime outputs match each APK. Package inspection confirms the enabled build includes the optional libraries and UVC notices, while the disabled build contains none of those optional libraries. Artifacts: `build/DroidDeckXR-ultra-reviewed.apk` (SHA-256 `a01706d56b876c2fec7b9386d33e9dc66854ead66b48a1392c492bde7c3591d3`) and `build/DroidDeckXR-ultra-reviewed-sdkfree.apk` (`6ce45e225a37c74355eb7d65e76b2ac63be7064ec24cbe389e006452e0e27390`). Results and logs are retained under `build/ultra-review-*`, including `build/ultra-review-verification.json`.
|
||||
|
||||
This review did not install a new package or repeat hardware acceptance. The existing dashboard interaction and stationary jitter reports remain unresolved; the combined tracking feature remains experimental.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Luma Ultra spatial tracking
|
||||
|
||||
For supported glasses across brands, feature levels, and app-wide setup, see the
|
||||
[smart-glasses support matrix](../smart-glasses-support.md).
|
||||
|
||||
This is an experimental implementation of native Luma Ultra head tracking, positional virtual screens, and optical SteamVR controllers. Positional tracking, bare-hand controllers, and Luma Joy-Con tracking all default to off and are grouped under Experimental. Combined release acceptance remains pending until the calibration, accuracy, latency, gameplay, and sustained performance targets below have been measured on hardware.
|
||||
|
||||
## Build and setup
|
||||
|
||||
Luma Ultra now includes a shared starting calibration exported from the user's verified Fold7 profile on 2026-10-16. The bundled `calibration/luma-ultra-640x480.json` contains the original lens, stereo, and camera-to-head geometry without the source glasses' serial identifier. It applies only to the SDK-confirmed Luma Ultra primary `left0/right0` pair at 640 by 480. Matching saved device calibration takes precedence, including an unfinished personal fit that still requires distance validation. Missing, corrupt, or foreign profiles fall back to the model base; unknown serials can also use the base.
|
||||
|
||||
The base enables experimental hand tracking without requiring every user to repeat board collection. It is not marked individually verified on other units. Tracking and setup show **Shared Luma Ultra calibration** while it is in use; diagnostics distinguish `calibrationSource` (`luma-ultra-base`, `device`, or `none`) from `calibrationVerified`. The existing `calibrated` availability field is true for either the base or a verified personal profile. The following board steps are optional refinement for a new device using the base; a newly calculated personal profile still needs the unchanged distance check before use. A successful distance check saves the profile for the connected identified glasses. Source validation does not establish manufacturing consistency or aiming accuracy across other units.
|
||||
|
||||
Use the existing optional `droiddeck.vitureSdk` local property with the external VITURE Android SDK 2.4.0. No vendor headers or libraries are checked into the repository. `-PvitureSdkDir=` explicitly builds without the SDK. The ordinary runtime freshness checks remain enabled. Rebuild the ARM64 VR components with `tools/vr/build_windows.py`; its manifest includes `spatialpose_shm.h`.
|
||||
|
||||
The hand pipeline pins MediaPipe Tasks Vision 1.0.0, OpenCV 4.13.0, and the model and SHA-256 recorded in `app/src/main/assets/models/hand_landmarker.provenance.json`. Both Gradle and the inference worker verify the model checksum. CPU inference, the full model, concurrent camera inference, scheduled hand discovery, and automatic input-rate adjustment are the defaults, including after app restarts, reconnects, and provider restarts. **Show diagnostics → GPU hand inference** temporarily enables GPU until the next tracking restart; GPU remains experimental because SteamVR restarts and Vulkan device loss were observed during testing. Diagnostics report the selection as `handGpuEnabled` and the initialized detectors as `handInferenceBackend`; `not initialized` means detector startup has not completed. Sustained performance acceptance remains unresolved.
|
||||
|
||||
Hand input starts by copying 500 permille of stereo callbacks, approximately 12.5 pairs per second at the SDK's documented 25 Hz camera rate. On CPU, automatic adjustment can raise this to 600 or 800 permille (about 15 or 20 Hz) only after eight consecutive valid observations have sufficient processing and freshness headroom. Promotion requires average processing at most 55 ms and observation age at most 110 ms for 15 Hz, then at most 40 ms and 90 ms for 20 Hz. A moving average allows occasional scheduled discovery frames while preserving a sustained processing budget. An average above the selected period reduces the rate; any single processing cost of 80 ms or observation age of 140 ms immediately returns it to 500 permille. Tracking failure and detector resets also return it to 500. This policy requests more input only when processing can accommodate it; it does not guarantee that many completed or valid hand updates.
|
||||
|
||||
Skipping happens before copying camera pixels into DroidDeck, so skipped pairs also avoid JNI transfer, preprocessing, and inference. The SDK exposes no tracking-camera rate setter: sensor capture, vendor VIO, pose callbacks, and head polling retain their normal rates. Calibration collection, distance checks, explicit diagnostic capture, and preview-only operation use full-rate input. Diagnostics keep incoming `cameraHz` distinct from `cameraCopyPermille`, `rateLimitedFrames` (intentional skips), and `dropped` (eligible pairs replaced while the worker is busy). The compatibility field `cameraFrameDivisor` is 2 at 500 permille, 1 at 1000, and 0 for intermediate fractions. The bounded latest-pair queue and original capture timestamps remain intact. **Automatic hand input rate** can be disabled to expose the manual half/full-rate switch; both choices reset with tracking. GPU uses that manual preference and does not adapt. Thermal, latency, and gameplay benefits require hardware comparison.
|
||||
|
||||
1. Connect a Luma Ultra and grant its existing USB permission. Spatial controls appear only after the SDK identifies the device as Carina.
|
||||
2. Enable **Positional tracking — experimental**. This restarts the provider in 6DoF mode. Switching picture modes does not restart it.
|
||||
3. Enter the current seated or standing eye height (shown in centimeters and feet/inches) and choose **Set floor from current height**. Normal VR recentering then preserves that floor, including while crouched. After reconnecting or explicitly restarting tracking, accept the new origin through setup/recenter. A recenter request during lost/stale tracking or recovery remains pending until 250 ms of fresh, steady poses arrive; repeated taps are unnecessary. The pending request is canceled on disconnect.
|
||||
4. Open **Stereo camera setup and diagnostics → Open setup**. The page uses the selected DroidDeck theme and app scale, with a single compact Back/title row so the controls have room to scroll in landscape. Check head tracking and camera synchronization first; capture controls become available when both are ready. Expand **Show diagnostics** for frame rate, frame age, dropped frames, processing time, and recovery tools.
|
||||
5. Print the 5 by 7 ChArUco board at actual size. Measure a square: it must be 30 mm (approximately 1.181 inches). Keep the board stationary and capture at least 24 sharp views by moving and rotating the glasses, covering the image center and edges. Include both yaw and pitch, roll, and varied distances. If the underlying tracking frame changes, start a new capture set; software recentering in an unchanged frame preserves the set.
|
||||
6. Calculate calibration. Accepted captures confirm board detection, not alignment quality. The fitter compares pinhole and fisheye models on held-out views. It rejects poor stereo reprojection, an implausible baseline, insufficient rotational diversity, and inconsistent stationary-board alignment. Turning and tilting the glasses is required; pause before each capture. If the board stayed fixed but alignment varies beyond 5 cm (about 2 inches), the head-tracking frame may have shifted during collection. Wait for steady tracking and start a new set; do not remove the quality gate to accept the old one.
|
||||
7. Start with **12 in (30.48 cm)** or **30 cm (11.81 in)** under **Measured distance**; **20 in** and **50 cm** remain available. The closer settings make the small printed markers easier to resolve in the 640 by 480 tracking cameras. Show the complete board with its center exactly the selected distance from the left camera, then choose **Validate distance**. You can hold or support the glasses without wearing them. Progress and the result appear beside this button. The count refers to detected checkerboard intersections, not the paper's edges; a fully visible but small or blurred board can still have too few detections. Validation detects corners in the original images, then corrects those coordinates for lens distortion before triangulating, avoiding marker blur from resampling the images. Rejected stereo geometry reports that depth could not be measured; a valid reconstruction shows measured and selected distances in both units. Each imperial or metric choice uses its exact distance. The complete 24-corner board and physical-distance check within 5 cm (about 2 inches) are still required before a newly fitted personal profile becomes usable. Profiles are keyed to the SDK board-serial SHA-256 identifier (falling back to a nonblank Android USB serial), dimensions, and format version. Verified profiles reload automatically after an app restart or reconnect to the same identified glasses. If neither identifier is available, profiles remain connection-local.
|
||||
8. Enable **Hand controllers — experimental**. Present an open hand first so inputs can arm. Aim the virtual controller at the target, pinch thumb and index finger together to press the trigger/click, then separate them to release. Moving the SteamVR dashboard by its handle uses this same pinch-and-release action. Keep the other fingers open when only clicking; curling the middle, ring, and little fingers activates the separate grip input, which the installed Vive dashboard binding maps to Back. Games can use grip for holding objects. After tracking loss, open the hand again before another press. Gamepad sticks supply locomotion and turning; buttons retain the Vive-compatible mapping. External VR controllers retain precedence.
|
||||
|
||||
The diagnostic capture button saves one stereo pair and metadata to the app-private `files/spatial/captures` directory. Merely opening diagnostics does not save images. Calibration profiles stay in `files/spatial`. A brief missed hand detection holds the last observed controller transform only until its original observation reaches 350 ms old. Input freshness remains limited to 200 ms. Pinch and grip release immediately on a detected miss and require an observed release before rearming. The hold does not extrapolate motion, refresh timestamps, or substitute a gamepad pose. Head tracking loss, origin/connection changes, disconnects, and controller-mode changes clear it. Gamepad menus and locomotion remain available during optical hand loss. Switch hand controllers off explicitly to return to gamepad-positioned controllers.
|
||||
|
||||
Accepted board landmarks and their capture-time head poses are saved separately in app-private `files/spatial/*-captures.json`, without camera images. A matching SDK or USB device identity and image geometry allow automatic recovery for recalculation. Restored observations cannot be appended to a new tracking frame; start a new set to capture again. Devices without either stable identifier keep a session-labelled diagnostic backup and require explicit developer recovery onto the same physical device. They do not automatically reuse another connection's unidentified capture set or personal calibration; they can still use the bundled model base.
|
||||
|
||||
The vendor SDK working directory is `noBackupFilesDir/viture`. Android can evict `cacheDir` during a live session, so active tracking files must not depend on that location. The SDK initializes its own files in the new directory on the next provider start; existing printed-board calibration profiles remain in `files/spatial`.
|
||||
|
||||
## Coordinates and lifecycle
|
||||
|
||||
`SpatialPose` uses meters and right/up/back OpenGL coordinates. Quaternions are normalized xyzw internally; SDK arrays are position followed by wxyz. Legacy orientation conversion and the 128-byte head-pose channel remain intact. The session coordinator survives `HeadTracker` recreation and owns separate full-pose screen and yaw/floor VR origins. The camera worker uses a generation guard, a bounded latest-frame queue, and no SDK lifecycle operations in callbacks.
|
||||
|
||||
Head polling is approximately every 4 ms with prediction disabled; this is not a claim of a 250 Hz sensor. Camera-rate callbacks provide a liveness check. Stale or unstable poses retain the last rendered transform and produce an invalid SteamVR pose. While a positional session is active, DroidDeck covers the resulting grey view with a themed recovery screen on the phone and glasses, including one readable copy per eye in SBS. It distinguishes tracking loss, USB disconnection, required recentering, and stabilization after recentering. Tracking-loss instructions ask the user to look around slowly in a well-lit room with visible detail, keep the cameras clear, and open the setup diagnostics to restart tracking if recovery stalls. Phone controls remain accessible, and the screen closes automatically when tracking is valid. Freshness is checked even if pose callbacks stop. Repeated USB timeouts require connection recovery rather than a synthetic valid pose. Large pose discontinuities latch a setup requirement. Provider shutdown invalidates queued work before closing the vendor provider and Java USB connection.
|
||||
|
||||
The 176-byte `droiddeck-spatialpose` snapshot uses protocol version 4 and sequence-counter publication. The driver validates file length, version, finite normalized poses, and monotonic age. Head and both hands use one snapshot and one VR origin. Hand pose flags independently identify valid palm tracking (bit 0), pinch (bit 1), and grip (bit 2); head flags use only bit 0. The formerly reserved 32-bit word at byte offset 36 is now `input_epochs`, containing independent 8-bit loss counters for left trigger, left grip, right trigger, and right grip, from least to most significant byte. The layout size and pose offsets are unchanged, but versions 1 through 3 are rejected; app and driver must be rebuilt together.
|
||||
|
||||
Snapshot flags select positional tracking (bit 0), bare-hand controllers (bit 1), or Luma Joy-Con
|
||||
controllers (bit 2). The latter two selections are mutually exclusive; either enables stereo inference.
|
||||
The Joy-Con mode consumes only wrist landmark 0 and the original capture timestamp; rotation and inputs
|
||||
come from the paired Joy-Con. `luma_joycon.h` selects fresh stereo positions or a 6DoF-head-anchored arm
|
||||
model independently per hand, rejects observations over 1.5 m from the head, and blends source switches
|
||||
with a 180 ms time constant. It never integrates accelerometer position or uses the phone-camera
|
||||
correction in this mode. Stale head poses invalidate output and release inputs. Selected mode flags
|
||||
survive disconnect so a lost Luma cannot silently switch to a fixed-origin controller. Origin/connection
|
||||
changes reset both positional blending and controller yaw alignment. This is hand-assisted tracking,
|
||||
not a detector for the controller shell; see [Joy-Con setup and limits](../cardboard-vr.md#joy-con-controllers).
|
||||
|
||||
Each counter advances when its gesture becomes ineligible, including invalid finger data, hand/head loss, and observation expiry. It survives later valid snapshots, allowing the driver to notice a short loss even if it never read the intervening invalid snapshot. Counters wrap independently within their bytes. Connection/origin changes reset driver input state, and a gap beyond 200 ms from its last accepted observation also releases inputs. Every hand retains its original camera observation timestamp: experimental hand poses and inputs expire after 200 ms, while head poses expire after 100 ms. A changed loss counter disarms only its affected gesture. Duplicate or older observations cannot rearm an input or replay a press; a new, valid released observation is required after loss before another press. Missing finger data never counts as a release gesture. Head updates cannot extend a hand observation's lifetime. Diagnostics expose the hand limit as `handMaxAgeMs`.
|
||||
|
||||
The driver submits trigger scalar, trigger click, and grip updates before submitting the controller pose, including when the pose is becoming invalid. Disconnect and deactivation also clear inputs first. Changes in input state, loss counters, or API error codes produce concise driver log records with raw gestures, output values, armed state, pose/input validity, observation age, connection, and origin. Successful input API calls confirm submission to SteamVR; they do not prove that the dashboard received its mouse-up event. The installed dashboard grab handler retains a hand attachment until mouse-up and has no tracking-loss cleanup in that JavaScript component. Recovery of the reported stuck dashboard drag therefore still requires hardware validation; the implementation does not manufacture another press or a valid pose to force release.
|
||||
|
||||
Controller display smoothing has no stationary anchors or dead zones. Small intentional motion remains visible; smoothing and prediction do not change the measured pose used for gesture validation.
|
||||
|
||||
The displayed controller converges toward each measured palm pose on every driver update with one 15 ms correction time constant and normalized, hemisphere-corrected quaternions. There is no additional per-observation smoothing stage. After three consecutive observations with consistent motion, translation advances from the result's receipt time for at most 80 ms and 2 cm; capture timestamps still determine velocity and freshness. Receipt-relative prediction can continue between delayed inference results instead of consuming its entire horizon before a result arrives. Prediction is disabled above 1.5 m/s or 6 m/s², on a missed/invalid observation, head loss, or changed origin/connection. A missed observation freezes the last displayed transform through duplicate old samples until a newer observation arrives or the original 350 ms visual-hold expiry is reached. Fast motion reaches the 2 cm bound sooner, and correction adds a small settling delay. Stabilization and prediction do not alter gesture values or observation timestamps and cannot replace measured latency validation.
|
||||
|
||||
The screen shader puts translation in `extra.yzw`, rotates the eye offset, and rejects intersections behind the eye. SBS and flat branches ignore translation. The push-constant allocation stays unchanged. A screen reference can include pitch and roll, preserving placement while reclining.
|
||||
|
||||
## Stereo reconstruction and limitations
|
||||
|
||||
Skipped camera pairs do not require clock warmup again when the source/host offset remains consistent. A source gap above 200 ms checks that offset against the existing 20 ms bound instead of unconditionally resetting synchronization. Nonmonotonic or invalid timestamps and inconsistent long gaps require 16 fresh observations; normal short-gap jitter retains the existing rolling-minimum and p90 policy. This preserves synchronization through a processing stall without publishing expired hand observations.
|
||||
|
||||
Palm orientation uses an explicit hand side. The lateral axis reverses for the right hand so mirrored palms have the same normal and finger-direction convention; the existing left-hand mapping is retained. This corrects the mathematical right-hand inversion, while physical controller aiming still needs confirmation.
|
||||
|
||||
The stock two-hand video graph can repeat whole-image palm detection when fewer than two hands were tracked previously. With both hands tracked it can skip that search; see the [MediaPipe graph](https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/cc/vision/hand_landmarker/hand_landmarker_graph.cc#L259). CPU scheduling maintains two persistent detector banks, each containing one detector per camera: one bank permits one hand, and the other permits two. After three observations agree on exactly one confident hand in both cameras, the one-hand bank handles intervening frames, and a two-hand search becomes due after 240 ms of camera time, running on the next processed frame. Both hands, a miss, handedness disagreement, nonmonotonic time, or a gap above 400 ms return selection to the two-hand bank. Second-hand discovery therefore remains enabled but can wait for the next scheduled search and processing. Banks retain their video tracking state between selections; they are not rebuilt per frame. GPU retains the ordinary two-hand bank. Diagnostics expose the active bank as `inferenceHandLimit`, alongside raw and filtered hand counts and per-camera timing. These choices reduce unnecessary searches by design; their benefit and extra model-memory cost require device measurement.
|
||||
|
||||
The primary `left0/right0` images are copied only for diagnostics, calibration, or enabled hands. At most one pair is pending while the worker processes another. Automatic exposure is retained because the cameras also serve native VIO. SDK 2.4.0 documents the tracking-camera rate as 25 Hz and provides no stereo frame-rate setter. Polling more frequently does not create additional camera frames. The camera-rate readout accounts for skipped source-frame sequence numbers instead of treating processed-pair throughput as sensor rate. The selected video-mode Hand Landmarker pair receives matching timestamps. CPU inference runs the cameras concurrently with separate image buffers; each bank's right detector is created, used, and closed on one dedicated worker. Both calls must finish before pixels or detectors can be reused, including after failure or shutdown interruption. GPU inference remains serial on its owning thread. The pipeline detects landmarks in the original grayscale images replicated into RGB channels, without image normalization or fisheye remapping, corrects landmark coordinates for stereo reconstruction, matches unambiguous same-handed candidates, rejects excessive reprojection or implausible depth, and derives the palm pose from triangulated landmarks. Gesture recognition uses agreement between the two image-space finger estimates. MediaPipe world landmarks are not treated as headset-relative positions.
|
||||
|
||||
The public header does not document the coordinate basis of the camera-rate pose callback. Its pose values are therefore not used as canonical head transforms. A bounded history of polled OpenGL poses supplies the capture-time transform. Source-to-host clock offsets must first stabilize over at least 16 observations; callback timestamp agreement and a nearby canonical head sample are required. Hand inference starts without waiting for the matching VIO callback, then checks for that exact camera timestamp before reconstruction can publish a valid pose. Missing confirmation, changed connections/tracking frames, invalid head tracking, and hand observations older than 200 ms still discard the result. Calibration and preview-only acquisition retain their bounded pose wait. Receipt-based offset estimation cannot establish absolute camera latency. External motion-to-visible-controller measurement is still required. Increasing expiry can reduce disappearing controllers at the cost of displaying older poses; it does not resolve GPU device loss or establish the latency/performance gates.
|
||||
|
||||
Stereo validation retains the original 3-pixel reprojection, positive-depth, and 15–150 cm depth checks per landmark. Bare-hand controller position and orientation require a valid triangulated wrist and all four palm knuckles (0, 5, 9, 13, 17). Joy-Con mode reconstructs wrist landmark 0 only and takes orientation and inputs from the paired controller. Live pinch and grip no longer require trustworthy fingertip depth: each image estimates palm-normalized pinch and middle/ring/little finger curl, then both views must agree. Grip combines fingertip reach and finger-chain bending, with separate press/release thresholds and three curled fingers to press versus two to retain a press. Invalid, degenerate, or conflicting image evidence releases only its affected gesture and requires an observed release before rearming. A valid 3D palm is never sufficient by itself to invent a grab or click. Calibration and distance validation still require every board point to pass. Conservative stereo association rejects ambiguous hands rather than inventing depth. Crossed hands, edge-on palms, and occlusion may reduce availability. The 2D gesture consensus and aiming still require physical testing through grabs, pulls, and releases; they do not establish an accuracy or latency pass. No skeleton output, desktop gestures, fast-action guarantee, or hands-only locomotion is provided.
|
||||
|
||||
Diagnostics include automatic or manual half/full input rates and a Lite-model comparison. Choices reset with the tracking session; the default is full-model parallel CPU with scheduled discovery and an adaptive rate starting at 500 permille. The Lite bundle uses Google's Lite palm detector and Lite landmark model with matching Tasks metadata. Both bundles are checksum-pinned in Gradle and at runtime; `tools/build_hand_lite.py` reproduces the candidate from pinned inputs. Saved-pair CPU replay compares full-model two-hand inference, scheduled full, scheduled lite, then full-model two-hand inference again, all with concurrent cameras. Rows identify `parallel` and `scheduled`; samples identify `inferenceHandLimit`. Its timings exclude model construction and cannot establish live detection coverage, external latency, second-hand discovery delay, or game performance.
|
||||
|
||||
## Combined release gates
|
||||
|
||||
| Area | Required evidence |
|
||||
|---|---|
|
||||
| Translation | Measured right/up/back movement and approximately 1:1 meter scale; lean and crouch; no double application |
|
||||
| Origins | Crouched recenter preserves floor; picture switching preserves independent screen and VR references |
|
||||
| Screens | Anchoring while moving sideways, toward and away; stable output after crossing the screen plane |
|
||||
| Hands | At least 95% valid slow unobstructed observations at 30–80 cm; position error at most 5 cm; stationary jitter at most 2 cm |
|
||||
| Latency | External physical-motion-to-visible-controller p95 below 100 ms |
|
||||
| Recovery | Immediate release on detected loss; observation expiry extended to 200 ms by user request; clear-hand reacquisition within 500 ms |
|
||||
| Performance | At most 10% regression in p95 game frame time over 20 minutes, with baseline frame rate maintained |
|
||||
| Gameplay | Dashboard selection and Angry Birds VR aim, grip, pull, release |
|
||||
| Robustness | Crossed hands, partial occlusion, head motion, lighting, USB reconnect, thermal load |
|
||||
|
||||
Record results in `luma-spatial-evidence.md`. A build or head-tracking pass alone does not satisfy this combined gate.
|
||||
@@ -187,7 +187,7 @@ Checked against bionic `android16-release`:
|
||||
|---|---|---|
|
||||
| `rseq`, `set_robust_list` | **blocked** | glibc, at the start of every thread, while all signals are blocked |
|
||||
| `faccessat2` | **blocked** | glibc's `faccessat()`, which tries it on **every** call before falling back, so 2 round trips (`access()` calls the plain `faccessat` syscall directly) |
|
||||
| `fchmodat2`, `openat2` | **blocked** | newer glibc, Flatpak/libglnx, FEX's rootfs lookups (`libblsession.so` turns `openat2` into `openat`) |
|
||||
| `fchmodat2`, `openat2` | **blocked** | newer glibc, Flatpak/libglnx, FEX's rootfs lookups (`libblsession.so` turns `openat2` into `openat`, resolving `RESOLVE_IN_ROOT` paths component by component inside the root) |
|
||||
| `setuid`, `setgid` and the rest of the family | **blocked** | Xwayland (hence `-i`) |
|
||||
| `futex_waitv`, `io_uring_*`, `landlock_*` | **blocked** | Proton's fsync (droiddeck-fsync answers `futex_waitv` in `libblsession.so`), some engines |
|
||||
| `clone3` | blocked before Android 15 | glibc `pthread_create` |
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
# RayNeo Air / GT XR integration
|
||||
|
||||
Built on the current `feat/viture-xr` PR #732 at bdf876d47cc9bb6fa810d014370da5bbb3c28039.
|
||||
The supplied historical hashes a127e66, 7f9dd75, e8e3e6b and f0b3297 were not
|
||||
resolvable in the fetched branch; no native projection or synchronization files are changed.
|
||||
|
||||
## Discovery and transport
|
||||
|
||||
`Glasses.kt` owns USB permission, attachment/detachment and presentation-display
|
||||
selection. There is no manifest USB VID filter: discovery uses `UsbManager.deviceList`.
|
||||
VITURE discovery is retained. `RayNeoModel` accepts the scaffold's confirmed
|
||||
`1bbb:af50` and verncat's confirmed GT identity `3941:af50`; other products under `1bbb` or bridge vendor `22d9` must identify
|
||||
as RayNeo/TCL Air or GT in their USB product name. This covers named Air 2, 2s,
|
||||
3, 4 Pro and GT devices without claiming arbitrary TCL phones or bridge chips.
|
||||
Additional unnamed VID/PID pairs require captures before being allowlisted.
|
||||
Other products under `3941` are rejected. Without a meaningful USB product name,
|
||||
`3941:af50` is labeled "RayNeo GT series"; a supplied GT MAX name is preserved.
|
||||
GT Max was observed on the Samsung SM-F976U with this USB identity on 2026-10-14;
|
||||
USB permission and the command handshake succeeded.
|
||||
EDID/product display names containing RayNeo, Ray Neo, or TCL Air/GT use the existing
|
||||
external-screen path. Android's supported modes, not a synthetic refresh value,
|
||||
determine whether 120 Hz can be selected. Stereo modes may have a lower maximum rate.
|
||||
GT Max's observed display identity is `SmartGlasses`, EDID manufacturer `TCL`,
|
||||
product `980` (1920x1080, 60/120 Hz in 2D). This exact metadata combination is
|
||||
recognized even though neither display name contains RayNeo. Other generic
|
||||
SmartGlasses/TCL products are not accepted by this rule.
|
||||
|
||||
Game output uses the same `GlassesScreen` presentation as VITURE, with the phone
|
||||
retaining the session controls; no screen-capture permission is needed. Discovery
|
||||
checks both Android's display name and its product name. On Android 12 and newer,
|
||||
a missing product-info object is treated like an unnamed panel: a recognized
|
||||
glasses USB device can identify a single presentation display. Unrelated named
|
||||
monitors and multiple unnamed displays are not automatically selected by that
|
||||
fallback. `Game on glasses` remains enabled by default.
|
||||
The presentation holder is selected before `show()`, and the phone SurfaceView
|
||||
is hidden only after the compositor attaches to the glasses holder. Late phone
|
||||
surface callbacks cannot reset the external output.
|
||||
|
||||
`RayNeoLink` claims a paired IN/OUT interface and translates the controller's
|
||||
internal commands to RayNeo packets. For `3941:af50`, selection prefers interface
|
||||
number 5 (not array index 5), as tested in verncat's GT contribution. Only endpoints
|
||||
that can carry 64-byte packets qualify. Firmware with a different interface number
|
||||
can use a unique qualifying IN/OUT pair; ambiguous alternatives are rejected.
|
||||
It requests device info (0x66/0), enables IMU
|
||||
(0x66/1) and waits for acknowledgments, warms up for two seconds, and stops IMU
|
||||
(0x66/2) when no pose consumer remains or the link closes. SBS uses commands 6/7.
|
||||
Unsupported VITURE brightness, volume and electrochromic commands are never sent;
|
||||
the VITURE hardware-control UI is hidden for RayNeo.
|
||||
GT spatial-button notification 0x08 and volume/display ACKs remain non-device-info
|
||||
responses and cannot satisfy unrelated startup commands. The spatial button is
|
||||
left to the glasses' firmware; no undocumented spatial-mode command is sent.
|
||||
|
||||
The parser/filter are adapted from the user's scaffold at
|
||||
581dbf712ed601ece4cf5e4a7c1f52e780299c88. Sensor packets are framed with 0x99/0x65,
|
||||
with little-endian acceleration at byte 4, gyro (degrees/sec) at 16 and a uint32
|
||||
100us device tick at 40. Fragmented and coalesced reports are assembled before decoding.
|
||||
Board 0x3a (Air 4 Pro) receives the scaffold's -20-degree X IMU mounting correction.
|
||||
GT Max (`3941:af50`, board `0x41`) uses the mapping measured on 2026-10-14:
|
||||
gyro and acceleration `(-y, z, -x)`, then +35 degrees around X.
|
||||
At the user's level pose the raw acceleration was approximately
|
||||
`(5.70, 0.00, 7.95)` m/s². A directed left-turn-and-hold capture produced positive
|
||||
X/Z gyro (e.g. `(20.5, 2.0, 28.2)` deg/s), and upward looks produced negative Y
|
||||
gyro. The 35-degree mounting estimate comes from that level capture,
|
||||
not manufacturer calibration; frame fit and other GT variants remain unverified.
|
||||
Gravity initializes orientation before the first published pose. Subsequent
|
||||
body-to-world fusion uses `measuredUp × predictedUp`, correcting tilt toward gravity
|
||||
instead of amplifying the error. These fusion corrections also apply to Air;
|
||||
the GT Max sensor remapping does not apply to Air or VITURE.
|
||||
|
||||
The non-Max GT shares USB `3941:af50` but reports board `0x40`, confirmed by
|
||||
switching between both physical pairs on the same phone. Its sensor axes use
|
||||
the ordinary XYZ basis with a measured -25-degree X mounting correction, rather
|
||||
than the Max permutation. Its confirmed upside-down resting acceleration was
|
||||
approximately `(0.020, -8.819, -4.106)` m/s²; the worn level capture was near
|
||||
`(0.09, 9.26, 3.11)`. Left turns produced positive Y/Z gyro. The mounting estimate
|
||||
uses the resting surface; per-device calibration refines its tilt and gyro bias.
|
||||
Both the pose filter and calibration collector select the same board profile.
|
||||
Non-Max calibration keys have a `:gt-standard-v1` suffix so they cannot replace
|
||||
existing Max profiles when USB identity fields coincide. Calibration loading
|
||||
waits for the device-info response; detached-device callbacks cannot change the
|
||||
replacement device's calibration key. Existing Max keys remain compatible.
|
||||
After installing this correction, the non-Max completed the upside-down
|
||||
calibration on the phone with `saved=true`, phase `SAVED`, progress 100.
|
||||
All 186 glasses/shared-pose/SteamVR-launch/UI regression checks passed and the
|
||||
installed APK's packaged runtime inputs verified. After calibration and the
|
||||
recenter check, the wearer confirmed on 2026-10-14 that the non-Max GT view
|
||||
looks and feels good.
|
||||
|
||||
## Existing pose and SteamVR pipeline
|
||||
|
||||
The scaffold fuses gyro/accel into wxyz. The Android adapter converts its Y-up
|
||||
coordinate basis and component order into VITURE's Z-up xyzw convention before
|
||||
calling the same `Glasses.PoseSink` used by VITURE. `HeadTracker` and `HeadPose`
|
||||
retain their existing recenter/seated-reference behavior. The path continues through
|
||||
`WaylandCompositor.nativeSetCardboardPose`, the compositor's `headpose_shm.h`
|
||||
shared-memory record and `tools/vr/src/driver_droiddeck.cpp`'s pose-file reader.
|
||||
The OpenVR projection, presenter, Vulkan wait logic and vblank counters are untouched.
|
||||
An explicit Play SteamVR request polls the guest runtime independently of display
|
||||
attachment and USB-control readiness. Both RayNeo and VITURE call the same
|
||||
`droiddeck-steamvr start` helper. Stopping cancels the pending launch request;
|
||||
automatic hardware 3D switching still respects the shared Auto 3D setting.
|
||||
|
||||
## Optional stationary calibration
|
||||
|
||||
Glasses → Tracking → Tracking calibration guides the user to rest the connected
|
||||
RayNeo upside down on a level surface. It allows three seconds to settle, then
|
||||
requires five continuous seconds of stationary samples (at least 200 reports).
|
||||
Invalid gravity, the wrong orientation, motion/variance, or gaps longer than 250 ms
|
||||
restart the window. A 30-second deadline fails without replacing a saved profile;
|
||||
USB stream loss also terminates the operation. Cancellation and link replacement
|
||||
invalidate queued callbacks by operation revision and connection generation.
|
||||
|
||||
The profile contains a raw gyro bias and a small mounting tilt correction, applied
|
||||
to both sensor vectors before fusion. It does not estimate heading, full accelerometer
|
||||
scale/bias, or translation. Profiles persist by vendor/product/serial; when a serial
|
||||
is unavailable the product and model name share a fallback profile. Reset deletes
|
||||
only the current profile and restores the model mapping. VITURE's fused SDK poses
|
||||
do not expose the raw inputs this measurement needs, so this control is RayNeo-only.
|
||||
|
||||
While measuring, live pose publication pauses. On completion the user wears the
|
||||
glasses, looks ahead and taps Done to recenter the active session. Dismissing the
|
||||
dialog or leaving the tracking panel cancels an in-progress measurement. Agent
|
||||
diagnostics expose the calibration state through `glasses`; operations are
|
||||
`calibrate`, `cancel-calibration`, and `reset-calibration`.
|
||||
The calibration build passed 157 targeted sensor, lifecycle, pose, launch and
|
||||
Compose UI tests. Installing this feature and validating a real upside-down capture
|
||||
on the phone remain pending a debugging connection.
|
||||
|
||||
## Regression and hardware validation
|
||||
|
||||
### SteamVR library launch routing (2026-10-14)
|
||||
|
||||
DroidDeck's SteamVR library entry (appid 250820), including its game shortcut,
|
||||
now sets the same `EXTRA_STEAMVR` request as Play SteamVR instead of sending a
|
||||
normal Steam game URL. Existing sessions resume with that request as well.
|
||||
Other games retain their normal launch path and arguments.
|
||||
|
||||
Inside Steam's library, the existing SharedJSContext bridge wraps
|
||||
`SteamClient.Apps.RunGame` for exactly 250820 and dispatches it to
|
||||
`droiddeck-steamvr launch`. This calls the same install/start helper used by the
|
||||
button, including OpenVR registration, the ARM64 runtime and Vulkan layer setup.
|
||||
It does not change Steam's saved launch options. The library bridge is installed
|
||||
for both desktop and Big Picture. It relies on the live Steam UI API and needs
|
||||
verification against the actual phone's Steam version after installation.
|
||||
|
||||
The helper serializes installation/startup and avoids restarting an active
|
||||
DroidDeck SteamVR session. Library launch diagnostics are in
|
||||
`${XDG_RUNTIME_DIR:-/tmp}/droiddeck-steamvr/library-launch.log`; normal runtime
|
||||
logs remain in `vrstartup.log` and Steam's logs directory. It does not intercept
|
||||
arbitrary external `steam://` URLs or change VR game launch arguments.
|
||||
|
||||
Only VR-only games (`vr_only`) wait for SteamVR and start it. Games where VR is
|
||||
optional, such as No Man's Sky, launch normally unless SteamVR is already
|
||||
running; before this they started (and if needed downloaded) SteamVR on every
|
||||
Play, so they opened in VR without the user asking for it.
|
||||
|
||||
Validation: seven bridge/dispatch tests passed under Node/Python, five Android
|
||||
SteamVR launch tests passed, shell syntax passed and the debug APK built with
|
||||
runtime input verification. The broader Linux compatibility suite cannot run
|
||||
unmodified on Windows because it requires `fcntl`; no Linux runtime was available
|
||||
for the helper's lock/process integration tests. Installation and live library
|
||||
Play verification are pending while the phone charges.
|
||||
|
||||
Run `./gradlew -PskipRust=true :app:testDebugUnitTest --tests 'com.droiddeck.launcher.glasses.*'`.
|
||||
Protocol tests cover malformed/fragmented packets, unsigned timestamps, model gating,
|
||||
quaternion normalization, yaw basis and tick rollover. USB integration tests cover
|
||||
permissions, GT interface preference/fallback, interleaved GT notifications and
|
||||
ACKs during startup, 2D/3D writes, and stale pose callbacks. Hardware validation is still required per model:
|
||||
USB enumeration, acknowledgment/stream startup, axes and recenter, hot unplug/replug,
|
||||
SBS negotiation, Android's actual 120Hz mode, and SteamVR pose/frame pacing.
|
||||
GT Max was verified on SM-F976U with USB `3941:af50` and the TCL/980 display:
|
||||
the compositor attached to its separate 1920x1080 display, the Play SteamVR
|
||||
request started native `vrserver`/`vrcompositor` through the shared helper, and
|
||||
the presenter logged roughly 60 frames/s while the IMU packet count increased.
|
||||
The test retained the user's disabled Auto 3D setting. After installing the corrected
|
||||
GT sensor mapping, the user confirmed an upright starting view and correct movement
|
||||
in all directions. SteamVR and the glasses IMU remained running. Stereo negotiation,
|
||||
long-term drift, and alternate bridge firmware variants still need their own
|
||||
hardware validation. The targeted glasses, Cardboard pose, and SteamVR launch
|
||||
suite passed 134 tests after the tracking correction. Tracking regression coverage
|
||||
checks initial gravity alignment, tilt convergence, the recorded GT Max level pose,
|
||||
and yaw/pitch/roll through the shared VITURE basis conversion and recenter path.
|
||||
|
||||
Attribution and the scaffold's Apache-2.0 license are in `docs/third-party/rayneo/`.
|
||||
@@ -0,0 +1,140 @@
|
||||
# VITURE Android SDK integration
|
||||
|
||||
For the cross-brand feature matrix and user setup, see [smart-glasses support](../smart-glasses-support.md).
|
||||
|
||||
The optional adapter is built against the user-supplied VITURE XR Glasses SDK
|
||||
2.4.0. Set `droiddeck.vitureSdk=C:/path/to/VITURE_XR_Glasses_SDK_for_Android`
|
||||
in the ignored `local.properties`, or pass `-PvitureSdkDir=/path/to/sdk`.
|
||||
The directory must contain `include/` and `android/arm64-v8a/`.
|
||||
|
||||
Gradle compiles `droiddeck_viture`, packages `libglasses`, `libcarina_vio`, and
|
||||
`libcloud_protocol`, and includes the SDK's LICENSE and NOTICE under
|
||||
`assets/licenses/viture`. Vendor headers and binaries stay outside the repository.
|
||||
A build without the optional SDK still supports the existing legacy VITURE and
|
||||
RayNeo transports, but does not advertise control for SDK-dependent glasses.
|
||||
|
||||
## Supported app functions
|
||||
|
||||
| Device | Transport and tracking | App controls |
|
||||
| --- | --- | --- |
|
||||
| Pro 2 XR | SDK Gen2 pose callbacks, observed OpenGL axes converted to NWU; negotiated pose frequency | 1080p 2D through 120 Hz, SBS 60 Hz, brightness 0–8, volume 0–8 |
|
||||
| Luma Ultra | SDK Carina USB transport; OpenGL poses converted to NWU | 1200p 2D through 120 Hz, SBS 60/90 Hz, brightness, volume, binary dimming |
|
||||
| Beast | SDK Gen2; native/bypass display commands selected by capability and current mode | 1200p output, SBS, brightness, volume 0–15, dimming toggle |
|
||||
|
||||
Pro 2 identification uses the SDK's PID validation and market name, not a guessed
|
||||
USB ID. Pro 2 has no electrochromic film control. Its display profile uses the
|
||||
[official 50° FOV and 1080p/SBS limits](https://www.viture.com/academy/product/viture-pro-2-xr-glasses).
|
||||
Device firmware can reject individual modes; Android's physical display size
|
||||
must confirm SBS before the app considers the switch successful.
|
||||
|
||||
All three feed the existing `Glasses.PoseSink`, head recentering, compositor and
|
||||
SteamVR pipeline. Ultra selects 3DoF or the opt-in positional 6DoF mode before SDK
|
||||
initialization. Its positional poses and stereo cameras feed the
|
||||
[Luma spatial pipeline](luma-spatial-tracking.md); camera passthrough is not implemented.
|
||||
The SDK's optional statistics reporter is not enabled.
|
||||
|
||||
The SDK owns USB interface selection and I/O. `UsbDeviceConnection` remains open
|
||||
until stop/shutdown/destroy finish. A generation token rejects callbacks from an
|
||||
old connection. Device brightness, volume and dimming events update the app.
|
||||
Beast native and standard display codes have separate meanings; they are mapped
|
||||
explicitly. Host tracking suspends native DOF to prevent applying rotation twice
|
||||
and restores the previous native DOF on stop. Older firmware rejecting a bypass
|
||||
mode can fall back to the native table, with rollback on failure.
|
||||
|
||||
`Game on glasses` uses Android's existing Presentation output. USB control alone
|
||||
does not create a video display: the phone, cable and adapter must expose a usable
|
||||
external display. Enabling 3D also enables and saves `Game on glasses` before
|
||||
requesting the stereo mode, so the picture uses the glasses' presentation instead
|
||||
of the mirrored phone output. Returning to 2D keeps the game on the glasses.
|
||||
Discovery checks both display name and product metadata, with
|
||||
a single unnamed-display fallback when recognized glasses USB is present.
|
||||
|
||||
Screen width and distance apply to **Virtual screen** rendering. In **Automatic**,
|
||||
ordinary 2D games stay flat. On an SBS display, the complete flat frame is fitted
|
||||
into each eye without cropping or head tracking. A running SteamVR session uses
|
||||
the direct stereo view. Games or videos that already contain separate eye images
|
||||
use the explicit **Side-by-side game** picture setting. Switching SBS on does not
|
||||
activate a virtual screen or head tracking for ordinary games.
|
||||
The direct view uses SteamVR field of view instead of virtual-screen dimensions
|
||||
while SteamVR is running. A plain side-by-side game fills each eye and does not
|
||||
use the SteamVR field-of-view or recenter controls.
|
||||
|
||||
All supported glasses with an available head-tracking connection have a separate
|
||||
**3DoF virtual screen** toggle, including VITURE, RayNeo and XREAL 1S. Availability
|
||||
comes from the connected transport's IMU capability rather than a model whitelist.
|
||||
The app renders this screen without enabling a built-in 3DoF mode or changing
|
||||
brightness. Enabling it selects Virtual screen, head tracking and Game on glasses,
|
||||
and saves those preferences without changing the glasses' 2D/SBS mode. It works
|
||||
with flat games in either display mode. Disabling it returns the picture to Automatic; switching
|
||||
SBS on or off preserves the 3DoF selection. Virtual screens center on the current
|
||||
viewing direction when tracking starts or the user recenters, including the
|
||||
up/down angle. Direct SteamVR views retain their gravity-aligned recentering.
|
||||
|
||||
The settings UI, menu presentation, and session renderer share this mode selection
|
||||
through `GlassesRendering`. Setup → Smart glasses configures DroidDeck menus,
|
||||
non-Steam launchers, desktop sessions, and games. Tracked menus use the app's own
|
||||
glasses presentation while phone controls remain available; see
|
||||
[tracked menus](glasses-menu-tracking.md). Regular refresh rate applies to 2D
|
||||
output; SBS uses the model's separate 3D refresh rate.
|
||||
Both the home menu and session drawer use **Display**, **VR**, **Tracking**, and
|
||||
**Device** tabs. Display groups connection and output controls, 2D/3D resolution
|
||||
and active refresh rate, then picture and virtual-screen geometry. VR contains
|
||||
SteamVR controls, automatic 3D, VR resolution, and field of view. Tracking groups
|
||||
head orientation, position and floor, and hands with camera setup. Device contains
|
||||
brightness, volume, and sunblock. Unsupported hardware tabs are hidden. Recenter
|
||||
is the first action on every session tab while head tracking is enabled for a
|
||||
3DoF/6DoF virtual screen or SteamVR.
|
||||
|
||||
Display saves **2D resolution** and **3D resolution**; VR saves **VR resolution**. Values are per eye: 1920 x 1080 or, on tall VITURE
|
||||
models, 1920 x 1200. Automatic keeps the existing model default; VR defaults to
|
||||
following the active 2D/3D preference. Home settings expose all three profiles,
|
||||
while a running session shows the active profile next to its refresh control.
|
||||
Switching resolution retains the refresh preference and does not change brightness,
|
||||
color processing, or the session's game-rendering resolution. Android selects an
|
||||
advertised mode with matching dimensions and the closest available refresh rate,
|
||||
without crossing the 2D/SBS layout. VITURE's USB command selects the matching
|
||||
resolution family first. A saved resolution unsupported by another attached model
|
||||
falls back to its automatic behavior. Resolution controls wait for a pending SBS
|
||||
transition to finish before accepting another change.
|
||||
|
||||
Both lists offer 30 Hz only when Android advertises a matching native display mode at the current
|
||||
resolution and **Game on glasses** is enabled. Android's presentation selects that mode without a
|
||||
vendor command. A saved 30 Hz choice falls back to 60 Hz if the display or eye layout lacks it;
|
||||
mirrored output never substitutes a 30 fps rendering limit.
|
||||
Software picture controls remain available while a display is connected even if
|
||||
USB control is lost; hardware controls and head tracking still require USB.
|
||||
Presentation refresh preferences are revalidated on display changes even when
|
||||
the requested Hz is unchanged, and keep both dimensions of the active mode.
|
||||
|
||||
## Validation
|
||||
|
||||
Run `:app:externalNativeBuildDebug` and
|
||||
`:app:testDebugUnitTest --tests 'com.droiddeck.launcher.glasses.*'` with the SDK
|
||||
configured. Compile `app/src/test/cpp/viture_display_modes_test.cpp` with the NDK
|
||||
C++17 compiler and the SDK include directory to check native/standard mappings
|
||||
against the vendor constants. The Kotlin tests cover model capabilities, axis
|
||||
conversion, invalid poses, rejected tracking starts and stale callbacks.
|
||||
|
||||
Physical acceptance checks remain: connect each model, grant USB permission,
|
||||
verify the Play SteamVR control, verify game output moves to the glasses,
|
||||
test head rotation/recenter and SBS, then unplug/reconnect. Building and unit
|
||||
testing do not establish hardware compatibility for every firmware revision.
|
||||
|
||||
### Pro 2 pose-axis correction, 2026-10-14
|
||||
|
||||
The connected Pro 2 delivered approximately 200 normalized poses/sec. The wearer
|
||||
reported yaw moving the view vertically, pitch rolling it, and roll turning it
|
||||
horizontally. Directed captures showed a left turn primarily increasing quaternion
|
||||
Y (approximately 0.004 to 0.253), while repeated upward looks varied quaternion X
|
||||
(approximately -0.187 to +0.082) with Y/Z remaining near zero. This is the OpenGL
|
||||
right/up/back basis, despite the SDK 2.4.0 generic Gen2 callback documentation
|
||||
describing NWU.
|
||||
|
||||
The adapter therefore selects the existing OpenGL-to-shared-glasses conversion
|
||||
for the SDK-identified Pro 2 family as well as Carina. Other Gen2 models retain
|
||||
their previous mapping. Tests exercise model selection and yaw/pitch/roll through
|
||||
the complete shared `HeadPose` conversion and recenter path. This fixes a model
|
||||
coordinate mismatch; it does not claim to run a vendor sensor-calibration command
|
||||
or save a RayNeo-style calibration for VITURE. All 188 glasses/shared-pose/
|
||||
SteamVR-launch/UI regression checks passed, packaged runtime inputs verified,
|
||||
and the corrected build was installed on the phone. Visual verification is pending.
|
||||
@@ -0,0 +1,451 @@
|
||||
# x86-64 Steam client
|
||||
|
||||
Valve's x86-64 Linux Steam client, run unchanged under FEX beside the default arm64 client. It is
|
||||
opt-in: Steam settings, **Steam client: x86-64** (`steamClientArch`, passed to the
|
||||
session as `BL_STEAM_CLIENT_ARCH=x86_64`, applies from the next session), or the desktop menu entry
|
||||
**Steam x86-64 (FEX, experimental)**. The Games tab lists the active client's games and tags its
|
||||
header "x86-64" while that client's list is shown. This page records why each workaround exists and what was seen
|
||||
on devices. The timeline is in `docs/progress-log.md`.
|
||||
|
||||
**Why.** The arm64 Linux client never runs VAC: Left 4 Dead 2 and Team Fortress 2 are refused by
|
||||
VAC-secured servers ("Connection Blocked ... VAC secured server") and no client log mentions VAC. VAC
|
||||
modules are x86 code. Under FEX the x86-64 client has a module it can run, as SteamLite does in Wine.
|
||||
|
||||
**Layout.** Everything lives in its own home, `$X64_HOME` (`BL_X64_HOME`, default `~/.droiddeck-x64`):
|
||||
its own `~/.steam` links, config, libraries, x86 rootfs and log (`steam-x64.log`). Shared with the arm64
|
||||
client: the sign-in, and its `config/uioverrides` (Steam's startup movie overrides), which the x86 client
|
||||
links. One client runs at a time, since both use the same account; the standalone launcher shuts the
|
||||
arm64 client down first.
|
||||
|
||||
| Command | Does |
|
||||
| --- | --- |
|
||||
| `droiddeck-steam-x64 [steam:// URL or client arguments]` | standalone run (the desktop menu entry) |
|
||||
| `droiddeck-steam-x64 --prepare` | install and set up, then exit |
|
||||
| `droiddeck-steam-x64 --exec ARGS...` | run the client binary in its environment (how the session starts it) |
|
||||
| `droiddeck-steam-x64 --sync-login` | copy the newer sign-in to the other client |
|
||||
| `droiddeck-steam-x64 --map-direct [--live]` | map Linux titles to DroidDeck direct |
|
||||
| `droiddeck-steam-x64 --default-tool` | print the default ARM64 Proton wrapper's tool name |
|
||||
| `droiddeck-steam-x64 --gpu` / `--webhelper-gpu` | exit status: client / x86 web helper on the GPU |
|
||||
|
||||
## Client install and update
|
||||
|
||||
- The first run downloads FEX (release `FEX_TAG` in Droid-Deck/DroidDeck-Components, sha256-pinned), FEX's Ubuntu 24.04 x86
|
||||
rootfs (about 525 MB, 1.5 GB unpacked, sha256-pinned; the same repository's copy of FEX-Emu's 2026-08-11 image) and the client (about 600 MB) from Valve's CDN,
|
||||
part by part with sha256 checks. A part-done rootfs download resumes; a finished one answers 416,
|
||||
which is not a failure. `unsquashfs -no-xattrs -ignore-errors`: ownership and xattrs mean nothing in
|
||||
the guest, and device nodes cannot be made and are not needed (FEX passes `/dev` through).
|
||||
- Channel `steamdeck_publicbeta`, the `-clientbeta` the session starts it with. `package/beta` must
|
||||
match `-clientbeta` before every start, or the client loops on rc 42 (its restart request, which the
|
||||
standalone launcher honours up to five times, as `steam.sh` does).
|
||||
- Before each start `x64_sync_client` brings the client to Valve's current build, downloading only the
|
||||
changed parts (9 of 28 on a device). If that fails, the client updates itself.
|
||||
- `.steam-enable-steamrt64-client` in the client folder opts into the SteamRT3 build
|
||||
(`steamrt64/steam`). Without it the client re-executes itself in the scout environment through
|
||||
`steam.sh`, whose start-up checks need what proot cannot give; the session stalled before the
|
||||
interface. The client is started directly, as the session starts `steamrtarm64/steam`:
|
||||
`STEAM_RUNTIME=1`, its own `steamrt64` libraries first, and the `~/.steam` links `steam.sh` makes.
|
||||
- **User namespaces.** Unlike the arm64 client, this one carries pressure-vessel and refuses to start
|
||||
("Steam now requires user namespaces") when `steam-runtime-check-requirements`' bwrap probe cannot
|
||||
create a user namespace, which no Android app can. `STEAM_RUNTIME_STEAMRT` points at
|
||||
`steamrt-shim`: Valve's runtime linked file by file with only the probe replaced (`exit 0`). proot
|
||||
is the container, as for the arm64 client.
|
||||
- **Re-exec after an update.** The client re-executes itself inside the SteamRT container through
|
||||
`pressure-vessel-wrap` (bwrap). `STEAM_STEAMRT_RECURSION_GUARD=1` says it is already inside one; the
|
||||
FEX rootfs with the client's `steamrt64` libraries is that container.
|
||||
- **Starting games.** Believing itself inside the container, the client starts games and system
|
||||
commands "alongside Steam" through `steam-runtime-launch-client`. Valve's asks a launcher service on
|
||||
the session bus (there is none: the game ended at once), and with none on `PATH` the launch fails
|
||||
(No such file). The shim's version applies `--directory`, `--env` and `--unset-env` and runs the
|
||||
command.
|
||||
- **Libraries.** The rootfs lacks what the SteamRT container gives the client (GTK 2 for
|
||||
`steamui.so`, ...). Valve's steamrt3c platform libraries are linked into `steamrt-libs`; they ship
|
||||
without soname links (pressure-vessel makes those when it builds the container), so the rootfs's
|
||||
`ldconfig` adds them there. `LD_LIBRARY_PATH` beats the system folders whatever its order, so only
|
||||
libraries the rootfs lacks are kept: the platform's older glibc must not shadow the rootfs's.
|
||||
- **System V IPC.** Android kernels have none, and the client's tier0 gave up on start ("Function not
|
||||
implemented" from `semget`; traced with a `-DDEBUG_STRACE` FEX). The session shim (`preload/*.c`) is
|
||||
also built for x86-64 and i386 and preloaded through the rootfs's own `/etc/ld.so.preload`, one copy
|
||||
per architecture under the multiarch folder named through glibc's `$LIB`. Without a list in the
|
||||
rootfs, FEX falls through to the guest's, which names the session's arm64 preloads.
|
||||
- **Tracer.** proot is a tracer, so tier0 believes a debugger is attached and breaks into it with int3
|
||||
on the first failed assertion: the web helper died of SIGTRAP a second after every start.
|
||||
`preload/tracer.c` exempts web helpers on purpose (Chromium traps on a status fd that is not
|
||||
procfs); with `BL_WEBHELPER_HIDE_TRACER=1` only opens called from `libtier0_s.so` get the copy
|
||||
without the tracer.
|
||||
- **FEXServer owns the rootfs.** A FEX process takes its rootfs from FEXServer, and a server started
|
||||
earlier without `FEX_ROOTFS` (any FEX run in the guest) serves none: every x86 library lookup fell
|
||||
through to the arm64 guest. It is stopped while no x86 client runs.
|
||||
- **`/dev/dri`.** x86 programs enumerated `/dev/dri`, found the session's KGSL node and spun on it (x86
|
||||
Turnip "failed to query kernel driver version", KMS dumb-buffer ioctls in a loop); the web helper
|
||||
never got a browser up. FEX looks in the rootfs first, so an empty `rootfs/dev/dri` hides the node.
|
||||
- **Scout runtime.** Valve ships it as `ubuntu12_32/steam-runtime.tar.xz` and `steam.sh` unpacks it.
|
||||
Since `steam.sh` is not used, `--prepare` unpacks it when its checksum changes.
|
||||
- **Repair.** Repair Steam (`BL_X64_REPAIR=1`), or two starts in a row that never showed the client
|
||||
(`package/droiddeck-unconfirmed`; the app clears the count once the client is on screen), removes
|
||||
the install stamp so the client is reinstalled. Counts from before this was tracked are dropped
|
||||
rather than taken as failed starts. `droiddeck-steam-install` removes the repair request when it is
|
||||
done, so the session reads it first and both clients are repaired.
|
||||
- **Stale state.** A killed client leaves `.crash`, which opens the update UI on its spinner, and
|
||||
`htmlcache/Singleton*` locks, which can leave the UI blank. Both are removed while no client runs.
|
||||
- **Network.** The client's connection header and Settings > Internet come from NetworkManager on the
|
||||
system bus; without it the client shows no connection while downloads run. The standalone launcher
|
||||
starts the session's system bus configuration and the `droiddeck-netmanager` stand-in when no
|
||||
system bus is up (a desktop has none).
|
||||
- **FEX settings.** The app's PERFORMANCE_TSO preset (`core/FexPreset.kt`, WinNative's default). FEX's
|
||||
own defaults keep half-barrier TSO and full-precision x87, which cost every x86 program time. Values
|
||||
already in the environment win.
|
||||
- **Fast path.** libblfastpath, preloaded into every guest process and so into FEX, broke FEX's 32-bit
|
||||
mode (SIGILL at start). It stands down when FEX is about to run a 32-bit program; 64-bit programs
|
||||
keep it. `BL_X64_FP_OFF=1` turns it off for everything.
|
||||
- **Diagnostics.** `BL_FEX` runs another FEX build (a `DEBUG_STRACE` one, say). `BL_FEXSERVER_LOG`
|
||||
starts a FEXServer in the foreground logging there; every FEX process that logs to "server"
|
||||
(`FEX_OUTPUTLOG` or an AppConfig) then has its lines tagged with its pid (file logs overwrite each
|
||||
other).
|
||||
|
||||
## Web helper on the GPU, QAM/WSI and flicker
|
||||
|
||||
The client's web helper draws its whole UI. There are two:
|
||||
|
||||
- **The arm64 client's helper** (`droiddeck-x64-arm-webhelper`), used when both clients are the same
|
||||
Steam build (unless `BL_X64_X86_WEBHELPER=1`). FEX hands the arm64 program to the CPU, so it runs
|
||||
natively with the session's preloads (KGSL poll fix, fast path) and the arm64 session's GPU
|
||||
settings. It talks to the client over IPC, and a helper of another build never comes up: the client
|
||||
restarted it every 10 s (device: 1790721607 against 1790904859). The session therefore updates the
|
||||
arm64 client's parts on the same channel first (`droiddeck-steam-install --sync
|
||||
steamdeck_publicbeta`). With this helper the client itself gets the GPU: it shows every Big Picture
|
||||
frame the helper draws, and in software (llvmpipe under FEX) that held the UI at about 20 fps
|
||||
(`BL_X64_CLIENT_SOFTWARE=1` keeps it in software). Device: 144 fps menus.
|
||||
- **The client's own x86 helper on the GPU** (`droiddeck-x64-webhelper`), otherwise: ANGLE's GL
|
||||
backend through FEX's GL thunk to zink and Turnip. On ANGLE's Vulkan backend the x86 compositor
|
||||
thread was the bottleneck (16 fps against 60). Never `--in-process-gpu`, the preload's default: a
|
||||
GPU crash there took the whole helper down (102 restarts). `BL_X64_CEF_GPU=0` keeps it in software.
|
||||
|
||||
The session shim (`preload/net.c`) turns `steamrt64/steamwebhelper.sh`'s exec of `./steamwebhelper`
|
||||
into `BL_X64_ARM_WEBHELPER`. Nothing in the client's folder is edited: the client checksums its files
|
||||
as it starts and reinstalled itself over an edited launcher (2-3 minutes) on every start.
|
||||
|
||||
Quick Access Menu and flicker, x86 helper:
|
||||
|
||||
- Steam's compositor in the helper draws each browser window (main UI, QAM) from that window's X
|
||||
pixmap. gamescope's WSI layer took the helper's zink swapchains and sent the frames to gamescope
|
||||
instead: the QAM opened blank until a d-pad press and the screen behind it went black. The helper
|
||||
runs with `DISABLE_GAMESCOPE_WSI=1`; games keep the layer. Not FEX: a texture-from-pixmap test
|
||||
behaved the same under FEX and natively.
|
||||
- gamescope sets `vk_xwayland_wait_ready=false` for its children because its layer does the waiting.
|
||||
Without the layer, unfinished frames reached the compositor as black flashes (3 in 10 QAM opens,
|
||||
screen-recorded and counted frame by frame). The helper sets `vk_xwayland_wait_ready=true`.
|
||||
- zink's kopper swapchain still flashed on about 1 in 20 opens with the wait. With
|
||||
`LIBGL_KOPPER_DISABLE=true` none did in 60 opens, at 60 fps instead of about 140 (Xwayland's
|
||||
per-window timing for child windows). No flashes was chosen over frame rate.
|
||||
|
||||
Software path (no host thunks, or `BL_X64_SOFTWARE=1`): x86 Mesa has no driver for this GPU, so
|
||||
llvmpipe for GL (the client's own window needs GLX) and the rootfs's own Vulkan ICDs (lavapipe). GLX
|
||||
first failed because the desktop exports zink over the arm64 Turnip ICD; those variables are cleared.
|
||||
`LIBGL_DRI3_DISABLE=1`: over DRI3 the X server hands out the GPU node and Mesa's software path then
|
||||
tries KMS dumb buffers on it in a loop.
|
||||
|
||||
Decky's loader talks to the client's CEF debugging port. Its marker
|
||||
(`.cef-enable-remote-debugging`) is kept in both clients' roots, because the session reads it from the
|
||||
client it runs. The app creates it in the x86-64 root only once that client is installed: the folder
|
||||
alone would then read as an installed client.
|
||||
|
||||
## FEX build and patches
|
||||
|
||||
- `tools/fex/build-in-arch.sh` runs in an Arch Linux ARM container (`menci/archlinuxarm:base-devel`)
|
||||
on an arm64 runner: Arch Linux ARM has no FEX package and FEX publishes no Linux binaries. It packs
|
||||
FEX and the runtime's own `unsquashfs` as `fex.tzst` (`usr/local/...`), with `fex-needed.txt` (every
|
||||
needed soname must resolve in a stock guest) and `fex-files.txt`. `TUNE_CPU=none`: the default
|
||||
(native) would tune for the runner's Neoverse cores, not the device.
|
||||
- **Host thunks** (`FEX_THUNKS=1`): an x86 program's libvulkan, libGL, libX11, ... calls go to the
|
||||
runtime's arm64 libraries (Turnip, Mesa). The host half needs the graphics development packages. The
|
||||
guest half is cross-compiled with clang against an x86 sysroot of Arch x86_64 and multilib i686
|
||||
packages, unpacked by pacman into their own root and never run (no scriptlets, no signature checks).
|
||||
FEX's x86 toolchain files assume an x86 build machine, so on arm64 the sysroot also supplies headers,
|
||||
C runtime objects and libgcc.
|
||||
- **Stack protector.** Arch's clang builds the guest thunks with stack protection. The canary load in
|
||||
`CallHostFunction`'s prologue (`mov %fs:0x28, %r11`) overwrote r11, which carries the host function's
|
||||
address, and FEX called the canary. 9 of 521 Vulkan and 66 of 736 GL host calls (those with 7 or
|
||||
more arguments) were affected, `vkCmdBindDescriptorSets` from TF2's DXVK among them; 32-bit games pass
|
||||
the host address in mm0. The toolchain files add `-fno-stack-protector` (`fex-2610-r2`).
|
||||
- **`vkGetDeviceProcAddr`.** FEX-2609's guest version passed a lookup of `vkGetDeviceProcAddr` itself
|
||||
to the host and handed back the raw host function, so every pointer fetched through it was unlinked.
|
||||
TF2's native DXVK (2.3) makes that lookup. FEX-2610 answers self-lookups in the guest (FEX-Emu/FEX
|
||||
3397eb9b).
|
||||
- Patches in `tools/fex/patches`:
|
||||
- `0001-openat2-enosys-fallback`: our proot answers `openat2` with ENOSYS (`tools/proot/PATCHES.md`,
|
||||
0008). FEX opened every rootfs path with `openat2(RESOLVE_IN_ROOT)` and fell through to the arm64
|
||||
guest on anything but EXDEV; x86 bash could not find libtinfo. Since r4 the fallback resolves the
|
||||
path inside the rootfs itself (absolute symlinks restart at the rootfs, as `RESOLVE_IN_ROOT` does);
|
||||
before, they resolved against the arm64 guest. `libblsession.so` (`preload/robust.c`) answers
|
||||
`openat2` first in practice and does the same walk for `RESOLVE_IN_ROOT` callers, so Valve's FEX-Emu
|
||||
on the arm64 client gets it too. Each path component is then its own system call.
|
||||
- `0002-thunkgen-newer-clang`: the container's newer clang has no `FileManager::clearStatCache()`,
|
||||
which the thunk generator called after each run.
|
||||
- `0003-allocator32-map32bit-exhaustion`: Counter-Strike 2 reserves the low 2 GB with `MAP_32BIT`
|
||||
in shrinking sizes. FEX's `MemAllocator32Bit` met EEXIST, stepped down one page per retry and
|
||||
restarted from the top forever (found with `perf_event_open` IP sampling, since proot holds
|
||||
ptrace). Such a range is now skipped whole and a second exhausted scan returns `-ENOMEM`, as the
|
||||
kernel does (`fex-2610-r3`).
|
||||
- **Building and publishing.** Droid-Deck/DroidDeck-Components' `build-fex.yml` builds `tools/fex` from
|
||||
this repository at a chosen ref (default `main`) with host thunks, in the same Arch Linux ARM container
|
||||
with a read-only token, then a separate job replaces `fex.tzst` on Components' single `fex` release,
|
||||
checks the digest GitHub stored, and records the build's version, sha256 and size in its `fex.json`
|
||||
(in that order: a device that reads the old `fex.json` meanwhile gets a mismatching archive and keeps
|
||||
its FEX). This repository's `build-fex.yml` only builds `tools/fex` when it changes (and on dispatch,
|
||||
for diagnostic builds such as `-DDEBUG_STRACE`); nothing is published from it.
|
||||
- Releases so far: `fex-2609-r2` (no thunks), `fex-2609-r3` (thunks), `fex-2610-r1`, `fex-2610-r2`,
|
||||
`fex-2610-r3`, `fex-2610-r4` (in-root resolution when openat2 is missing, exact MAP_32BIT
|
||||
marking), `fex-2610-r5` (current: bundles libxxhash, which the Linux runtime does not ship; the
|
||||
build fails when a program in the package needs a library that is neither bundled nor listed in
|
||||
`tools/fex/runtime-libs.txt`). They were built and published in The412Banner/DroidDeck; up to r3 their tags point at that
|
||||
repository's `main`, from r4 at the commit each was built from. Later builds are versions on the single
|
||||
`fex` release; `fex-2610-r5` stays published as the build `droiddeck-steam-x64` pins.
|
||||
- **Where the app gets it.** The app downloads FEX from Droid-Deck/DroidDeck-Components, where nothing is
|
||||
uploaded by hand. The app follows its `fex.json`: while the x86-64 client is selected it reads it from
|
||||
Components' `main` (directly, at most every ten minutes) and keeps the last copy, and a session passes
|
||||
that build to `droiddeck-steam-x64` (`BL_FEX_TAG`, `BL_FEX_SHA256`, `BL_FEX_URLS` with the download
|
||||
source's mirrors). The script installs a build only after its archive matches the sha256 and every
|
||||
program in it loads (`ld-linux --list`); the files it replaces are kept aside until the installed copy
|
||||
loads too and put back otherwise, and the tag is written last. Otherwise it keeps the installed one while that still loads,
|
||||
then falls back to `FEX_TAG`/`FEX_SHA256` built in, which stay the first-start default when no copy
|
||||
of `fex.json` has been read. Publishing a FEX build therefore takes a Components run only; a
|
||||
replaced archive cannot be pointed back to, so undoing a build means building an earlier ref. The rootfs is
|
||||
FEX-Emu's own image, unmodified: `fex-rootfs.json` there names its rootfs.fex-emu.gg link, sha256 and
|
||||
size, and `mirror-upstream.yml` fetches it into the catalog's release only when both match. A newer
|
||||
image is a new catalog release; `ROOTFS_URL` and `ROOTFS_SHA256` then move to it, and a device that
|
||||
already unpacked one keeps it.
|
||||
|
||||
- **What a session log shows.** `FEX ready (<tag>, <source>)` names the build and where it came from
|
||||
(`catalog`, `built-in`, or `kept` when a newer one did not install, with the reason a line above);
|
||||
`x86-64 rootfs:` names the image; `x86-64 client:` the client's ubuntu12 and steamrt64 builds at start
|
||||
(it may update itself after). The rootfs download reports every tenth of its size as a step. When the
|
||||
web helper's GPU process keeps exiting until Chromium gives up on the GPU (the menus then never appear),
|
||||
one line says so with the exit codes, followed by x86 `glxinfo -B` and `vulkaninfo --summary` through
|
||||
FEX's thunks.
|
||||
|
||||
## DroidDeck direct for Linux games and runtimes
|
||||
|
||||
- Steam runs native Linux games in the Steam Linux Runtime container (pressure-vessel), which needs
|
||||
bwrap and user namespaces. **DroidDeck direct** (`compatibilitytools.d/droiddeck-direct`, linux to
|
||||
linux) runs them under FEX with no container. Scout titles run through Valve's scout library runtime
|
||||
(`ubuntu12_32/steam-runtime/run.sh`). Titles Valve gives a newer runtime (sniper, soldier; listed in
|
||||
`direct-sniper`) are built for newer libraries than scout's and run on the rootfs (Ubuntu 24.04) as
|
||||
it is.
|
||||
- **Mapping** (`x64_map_direct`): every game the client's appinfo cache shows with a Linux build (type
|
||||
game, oslist with linux; runtime from `config/app_mappings`: SteamLinuxRuntime_sniper or _soldier,
|
||||
none means scout), installed or not, so Steam downloads the Linux depots. A title on a tool chosen in
|
||||
Properties keeps it, except `BL_X64_DIRECT_APPS` (Left 4 Dead 2, 550, by default), which is always
|
||||
taken back. `direct-auto.json` lists what was mapped. One pass over the mapping: a 2000-title fixture
|
||||
went from 16.5 s to 0.2 s with the same output.
|
||||
- A game whose Windows build is installed (it ran on a Proton before) stays a Windows game on the
|
||||
default Proton. Steam does not swap an installed build when the tool changes: DroidDeck direct was
|
||||
handed `Launcher.exe` and the game ended at once (Borderlands 2), and getting the Linux build back
|
||||
means downloading the game again.
|
||||
- The client rewrites `config.vdf` from memory as it exits, so the mapping is written only while no
|
||||
x86 client runs: by `--prepare`, and by the session after each exit, when the appinfo cache also
|
||||
knows titles bought during the run. `--live` (the session's pass while the client runs) writes
|
||||
nothing; installed Linux titles it would map go to the session's pending list.
|
||||
- A new user's client has no `config.vdf` before its first start, so Steam Play's default and the
|
||||
Linux titles would wait a session. `--prepare` writes the `CompatToolMapping` block alone, which the
|
||||
client keeps.
|
||||
- **Sniper check.** `tf.sh` reads `VERSION_CODENAME` from `/etc/os-release` and stops unless it is
|
||||
`sniper`. For `direct-sniper` titles the run script runs such a script through `bash -c` with that
|
||||
line replaced by `VERSION_CODENAME=sniper`.
|
||||
- **HarfBuzz.** The rootfs's HarfBuzz needs a newer FreeType than a game may bring. Counter-Strike 2
|
||||
ships an older `libfreetype.so.6` and no HarfBuzz: its menu text library failed to load, the game
|
||||
flagged its files corrupt (a verify each start) and waited behind an error box that could not open.
|
||||
Newer-runtime titles get Valve's runtime HarfBuzz (2.7) preloaded from `sniper-compat`, as they would
|
||||
in its container.
|
||||
- **GPU.** Games get FEX's host thunks and the session's GPU settings (Turnip, Zink), saved by the
|
||||
launcher as `game-gpu.env` and rewritten whenever it runs; `game-gpu.local` is the user's, never
|
||||
rewritten, and is read after it. `BL_X64_GAME_SOFTWARE=1` keeps a game in software, with
|
||||
`mesa_glthread=false` (GL threading is no help with software GL under FEX).
|
||||
- Source games (`hl2.sh`, `hl2_linux`, `srcds`) get `-condebug` for the VAC diagnostics.
|
||||
- **Controllers.** `libfakeinput` is built for x86-64 and preloaded in the rootfs; Big Picture works
|
||||
with it. The 32-bit copy crashed Left 4 Dead 2: its fake devices reach the game's own SDL2 through
|
||||
udev with a missing attribute and SDL `strdup()`s the NULL (fault backtrace: strlen in libc <- `__strdup`
|
||||
<- libSDL2 <- `udev_device_get_sysattr_value`). 32-bit programs get only the session shim. The
|
||||
preload line names the x86_64 copy by full path, since with `$LIB` FEX falls through to the
|
||||
session's own `/usr/local/lib/i386-linux-gnu` copy.
|
||||
- Device: TF2 main menu at 358 fps on the GPU and about 84 fps in a match; Half-Life (32-bit) at
|
||||
101 fps; Left 4 Dead 2 at 51 fps (about 11 in software).
|
||||
- The arm64 client's route to x86 Linux titles, `droiddeck-fex`, is not wrapped for this client. Its
|
||||
wrapper ran when the client probed its tools after logon; the arm64 client took that for an x86
|
||||
program wanting FEX and installed Valve's FEX.
|
||||
- `droiddeck-fex` itself (arm64 client): Valve's FEX-Emu keeps its server socket in
|
||||
`$STEAM_COMPAT_DATA_PATH/fex-emu/Server` whatever `FEX_APP_DATA_LOCATION` or `FEX_SERVERSOCKETPATH`
|
||||
say. Under the app's data folder that path passed the 108 bytes a socket path may have (Half-Life:
|
||||
136), FEXServer never listened and the game ended at once. FEX's data folder in the runtime directory
|
||||
is short enough; the one under HOME is not, so `/tmp/fex` stands in for it.
|
||||
|
||||
## Libraries, SD card and sharing between clients
|
||||
|
||||
- **Separate libraries.** With folders shared, each client could update, verify or switch the
|
||||
other's games. A title with a Linux build runs as that in the x86 client and as the Windows build in
|
||||
the arm64 one, so opening it in the other client switched the files and the first client downloaded
|
||||
its version again (Left 4 Dead 2, 8.6 GB). Hard links do not help: switching platform makes Steam
|
||||
download depot 551 anyway. A game wanted in both clients is installed in both. Folders an earlier
|
||||
version shared are taken out of the x86 client's lists (written only while no x86 client runs); the
|
||||
games stay in the arm64 library.
|
||||
- **SD card.** The x86 client's card library is a folder inside the arm64 one (`/mnt/droiddeck-sd/x86-64`,
|
||||
`GameStorage.X64_CARD_DIR`, passed as `BL_X64_STEAM_LIBRARIES`), so the install dialog offers Internal
|
||||
Storage and SD card. compatdata, shadercache and runtime tools stay private, as in the arm64 card
|
||||
library. FEX runs x86 programs from the noexec exFAT card. The folder appears with the client's first
|
||||
session on the card and its `steamapps` with the first library write. Desktop sessions prepare it as
|
||||
well: desktop mode offers the x86 client whichever is selected, and `droiddeck-steam-x64` drops
|
||||
library folders it is not given. The arm64 card library may hold the arm64 copy of the same title,
|
||||
so the session's installed check looks only in the x86 card folder.
|
||||
- **Card downloads.** `storage.c` (no zero-filled reservation where `fallocate` is refused) and
|
||||
`dircache.c` recognised only `steamrtarm64/steam`, and the x86 client wrote each download in full as
|
||||
zeros first (Audiosurf: 415 MB written, 0 downloaded). After the fix: FlatOut, 1.19 GB, to the card
|
||||
in 51 s.
|
||||
- **Sign-in.** Before either client starts, the newer sign-in (`config/loginusers.vdf`, the login token
|
||||
in `local.vdf`, `AutoLoginUser` and `RememberPassword` in `registry.vdf`) is copied to the other, so
|
||||
signing in or switching accounts in one counts for both. Steam renews its token every few days and
|
||||
refuses an old copy (a 3-day-old one asked for the password), so the newest is copied at every start,
|
||||
never while either client runs, since each rewrites these files. A client that refused its token
|
||||
rewrites `local.vdf` with an empty `ConnectCache`: newest, but copied over the other it would sign
|
||||
that one out too (seen after the x86 client's "Invalid Password"), so it is never the source. The copy keeps
|
||||
the time stamp, so both match until one client signs in again. The rest of `registry.vdf`
|
||||
(language, skin) stays each client's.
|
||||
- **App side.** The library, art picker, grid sync and boot movie follow the running client, else the
|
||||
selected one; a changed client applies from the next session. Offline Mode is written to both
|
||||
clients' `loginusers.vdf`; until the x86 client's first start its sign-in is the arm64 one. The log
|
||||
redactor learns account names from both clients. The startup movie is staged into the x86 client's
|
||||
config only once its first start has created it; `--prepare` links the arm64 `config/uioverrides`
|
||||
and copies the `StartupMovie` choice.
|
||||
- **Storage page.** It lists both clients' games, the x86 ones labelled; a game can be installed in
|
||||
both and an id left over in several libraries. `steam://uninstall` reaches only the session's client,
|
||||
so the other client's copy is uninstalled after switching clients. Tools in the x86 client's
|
||||
libraries are shown as required (it runs its Linux games with Valve's runtimes). Proton tool files
|
||||
live only in the arm64 root; the x86 client has wrappers. An x86 client without `config.vdf` has
|
||||
never mapped a game.
|
||||
|
||||
## Proton for Windows titles in the x86 client
|
||||
|
||||
- Windows games run on the arm64 client's Protons; x86 Proton would need pressure-vessel. For each
|
||||
arm64 client tool (droiddeck-proton-arm64, DroidDeck-Proton, GE-Proton, CachyOS) `--prepare` makes a
|
||||
wrapper in the x86 client's `compatibilitytools.d` with the same folder and display name: Components
|
||||
swaps and per-title mappings are keyed by them. In the tool name `arm64` and `aarch64` become `a64`,
|
||||
because the x86 client ignores a tool named for arm64 ("for a different target platform linux
|
||||
arm64"). The session runs `--prepare` again after it installs a new Proton.
|
||||
- The wrapper is an x86 script under FEX. It starts `droiddeck-x64-arm-proton` through an arm64 binary
|
||||
(a copy of `/usr/bin/env`), which FEX hands straight to the CPU, so Wine and the game run as from the
|
||||
arm64 client: same Proton, launcher, Components swaps, per-game environment and Steam client library
|
||||
(`~/.steam/sdk64`).
|
||||
- `droiddeck-x64-arm-proton` drops FEX's settings and the x86 library paths and preloads, then restores
|
||||
the arm64 session's environment that `droiddeck-steam-x64` saved before it built the x86 one
|
||||
(`arm-session.env`: HOME, PATH, GPU, audio, input, fast path). Steam's per-game variables are not in
|
||||
it. It keeps the game display Steam chose: the saved `DISPLAY=:0` hid the game window and made
|
||||
DiRT Showdown exit after about 7 s. Proton takes its Steam client parts from the client's install
|
||||
folder, and the x86 client's are x86, so `STEAM_COMPAT_CLIENT_INSTALL_PATH` points at the arm64 root
|
||||
and `BL_X64_STEAM_ROOT` keeps the x86 one for install-script lookups.
|
||||
- **Game overlay.** The x86 client preloads its own x86 overlay; the arm64 one takes its place. The
|
||||
overlay marks the game's windows with the Steam app ID, and without it gamescope never shows them
|
||||
(FlatOut ran and played sound behind Steam's launch screen, "steam app id: 0"). The agent also tags
|
||||
untagged `steam_app_<id>` windows with `STEAM_GAME` on the game display when the x86 client runs
|
||||
(DiRT 3, an added game, then reached its title screen at 144 fps).
|
||||
- The x86 client's tools are not adopted as arm64 Protons: `adopt_extras` on its
|
||||
`compatibilitytools.d` rewrote DroidDeck direct as a Windows tool, and Steam installed Half-Life's
|
||||
Windows files. The arm64 Proton is adopted on demand at an x86 launch.
|
||||
- **Default.** The newest DroidDeck-Proton, then the newest GE-Proton (aarch64), else
|
||||
`droiddeck-proton-a64`. FlatOut from the card ran on DroidDeck-Proton 11-9.1 and exited on
|
||||
GE-Proton11-7 (no log). With droiddeck-proton-arm64 (Proton Experimental) Xwayland never got its
|
||||
window buffers back and FlatOut took the device's RAM with it (10.5 GB of dma-buf in 35 s);
|
||||
GE-Proton, same game and client, stayed flat. The session asks for GE-Proton when none is installed.
|
||||
`default-tool` records the default the script set; when it changes, the titles still on the old one
|
||||
move along (device: 113 titles off GE-Proton11-7). A default or title the user set to another tool
|
||||
keeps it. Before `default-tool` existed, the default the script set was an `-a64` wrapper at
|
||||
priority 75.
|
||||
- `steam-compatibility --x64` repoints entries the client aimed at a Proton it cannot start (Valve's
|
||||
`proton_*` names, and `droiddeck-proton-arm64` left by an arm64 compat pass) to the default wrapper.
|
||||
Tools in its own `compatibilitytools.d` are kept: the wrappers include `proton-cachyos-*-a64`, which
|
||||
starts with "proton" like Valve's.
|
||||
- **Added games** are written by the client the session runs and pointed at its default wrapper. The
|
||||
shortcuts pass keeps the x86 client's own default and state (`droiddeck-compat` beside its Steam
|
||||
folder), because the arm64 client's `state.json` lists which of that client's titles follow its
|
||||
default.
|
||||
- **Install scripts.** A game's own installers without a `HasRunKey` (Borderlands 2: VC++ 2005, 2008
|
||||
and 2010, DirectX, .NET 4.0) are marked under `Software\Valve\Steam\Apps\<appid>` (and the
|
||||
Wow6432Node view), so Steam skips them; this applies to both clients. Windows components install into
|
||||
the x86 client's prefix (DiRT Showdown and DiRT 3 needed OpenAL).
|
||||
|
||||
## VAC findings
|
||||
|
||||
- x86-64 client with Linux builds: secure. Left 4 Dead 2 joined a live Quick Match game, the server
|
||||
type that refuses the arm64 client; on the GPU an online server answered the A2S info query as
|
||||
"Valve Left4Dead 2 US East Server", VAC=1. TF2 Casual matchmaking put it on a VAC-secured Valve
|
||||
server. Half-Life's native build joined a VAC-secured server.
|
||||
- x86-64 client with a Windows build (TF2 on DroidDeck-Proton): insecure. Proton reaches Steam through
|
||||
the arm64 client's `steamclient.so`.
|
||||
- arm64 client: Left 4 Dead 2 was blocked on both Protons; Half-Life (Windows build) joined a vac=1
|
||||
GoldSrc server. Why the arm64 client fails is not established; it has no VAC module of its own to run.
|
||||
- Headless VAC testing is not possible: current Steam starts the saved login from its UI.
|
||||
- **VAC diagnostics** (Setup page). A Source game decides secure or insecure once, as it starts, and
|
||||
says so only in its own console ("You are in insecure mode. You must restart before you can connect
|
||||
to secure servers."). That console reaches a file only with `-condebug`: `<game>/<mod>/console.log`,
|
||||
or `<game>/game/<mod>/console.log` for Source 2. At session start the app writes
|
||||
`~/.droiddeck-vac-diagnostics` (start time, client, each console.log's length, since `-condebug`
|
||||
appends). The launch wrapper (`bl_vacdiag`) then adds `-condebug` to Source titles (found by
|
||||
`gameinfo.txt` or `gameinfo.gi`), flags `-insecure` in launch options and notes each launch. At the
|
||||
end the session copies what each console.log gained into `vac/`, scrubbed, and writes `vac.txt`: the
|
||||
game's secure/insecure lines, VAC lines in the client's logs, and which client files are present. A
|
||||
stopped session keeps its own copy of the marker and launch list; a new session waits for earlier
|
||||
collections, because it replaces the marker and Steam's logs.
|
||||
- "Connecting to" is only an attempt. L4D2 logs it, retries, and then refuses a VAC server with a
|
||||
dialog ("Connection Blocked: Please remove plug-ins before attempting to connect to a VAC secured
|
||||
server") that never reaches the console. Only "Connected to" means the game got in.
|
||||
- SteamLite's "INSECURE - VAC may kick" warning was our own parser reading the wrong log while the
|
||||
client was secure all along. `vac.txt` therefore quotes the lines it found and says "inconclusive"
|
||||
when the game never reached a server.
|
||||
|
||||
## Session integration, restarts and mapping
|
||||
|
||||
- With `BL_STEAM_CLIENT_ARCH=x86_64`, `droiddeck-session` updates the arm64 client's parts
|
||||
(`droiddeck-steam-install`, then `--sync steamdeck_publicbeta`), runs `droiddeck-steam-x64 --prepare`
|
||||
and starts the client with `droiddeck-steam-x64 --exec`. Steps only for the arm64 client (library
|
||||
registration, the ARM64 Proton fetch, the live compat helper) are skipped; added games,
|
||||
`steam-compatibility --x64`, UI scale and redistributables run for the x86 client.
|
||||
- **Deck mode.** With `-steamos3` the client's SteamOS audio manager passed libpulse the NULL
|
||||
operation of a failed request, and libpulse aborted the client right after start ("Assertion 'o'
|
||||
failed at ../src/pulse/operation.c:136 pa_operation_get_state()", the runtime's PulseAudio 13).
|
||||
`preload/pulseguard.c` (x86 builds only; the arm64 client does not hit it) answers a missing operation
|
||||
as cancelled and ignores its unref and cancel. The session passes `-steamdeck -steamos3` unless
|
||||
`BL_X64_DECK=0`. Switch to Desktop is expected to need no change (an x86 shell under FEX runs
|
||||
`steamos-session-select`); the full switch has not been tested on a device.
|
||||
- **Process names.** Under FEX a process's argv[0] is FEX and the program is its first argument. The
|
||||
session's game detection, CPU pinning and the zRAM watcher read that argument. Steam pins itself and
|
||||
its web helper to a few cores, so the session's CPU set is applied again every 5 s.
|
||||
- **Live mapping and restarts.** While the client runs, the session's background loop runs
|
||||
`steam-compatibility --x64 --live` and `droiddeck-steam-x64 --map-direct --live`, every 15 s and
|
||||
only in the session's first hour (720 beats of 5 s). A title installed after that is mapped when the
|
||||
client exits and takes effect from the next session. Steam reads a
|
||||
mapping changed while it runs only at its next start, so changed titles go to `.bl-compat-pending`.
|
||||
Once they have installed (`StateFlags` 4 in the client's root or its card folder) and no game runs,
|
||||
the client is restarted once per title, at most 3 times a session.
|
||||
- **After exit.** Added games, `steam-compatibility --x64` and `--map-direct` run again: the client
|
||||
wrote its mapping from memory as it exited, over what the live passes set.
|
||||
- The Lossless Scaling ownership probe and the Steam import watcher reach the x86 client through the
|
||||
arm64 client's `steamclient.so` (`BL_STEAMCLIENT`); the watcher restarts the client with
|
||||
`droiddeck-steam-x64 --exec -shutdown`.
|
||||
- Steam logs and the Steam language come from the x86 client's home. On the desktop, `steam` and
|
||||
`steamdeck-steam` start the selected client, and the desktop game entries list its libraries.
|
||||
- The standalone launcher (desktop menu) shuts the arm64 client down first, starts the system bus and
|
||||
the NetworkManager stand-in when missing, clears stale locks, pins CPUs and restarts the client on
|
||||
rc 42.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Experimental and opt-in; one client runs at a time.
|
||||
- Windows titles in the x86 client are not VAC-secure: Proton reaches Steam through the arm64
|
||||
`steamclient.so`.
|
||||
- 32-bit games get no controller reader (the i386 `libfakeinput` is left out).
|
||||
- 32-bit Linux titles through the arm64 client's `droiddeck-fex` still fail (not pursued).
|
||||
- The x86 web helper's menus run at 60 fps without kopper.
|
||||
- Counter-Strike 2 reaches its menu (and loaded Dust II at 78 fps), then Android's `lmkd` kills
|
||||
DroidDeck. The game's GPU memory is pinned: KGSL `page_alloc` followed Shmem through the load, to
|
||||
about 3.2 GB at the kill, with Unevictable just above it. This device's `lmkd` runs in minfree mode,
|
||||
which does not count swap, while MemAvailable counted that memory as available. Paging the game out
|
||||
with a root tool (`process_madvise(MADV_PAGEOUT)`) let it draw its menu, but `lmkd` still killed the
|
||||
session. Not solved.
|
||||
- Sharing libraries between the clients is parked.
|
||||
@@ -0,0 +1,242 @@
|
||||
# XREAL 1S / One Pro feasibility — 2026-10-14
|
||||
|
||||
The user owns XREAL 1S and also wants One Pro support. Supporting these in DroidDeck
|
||||
is feasible based on existing open drivers, but the supplied Monado Air driver is
|
||||
not their transport. The initial investigation below preceded implementation.
|
||||
An experimental native 1S driver is now enabled; see the hardware validation
|
||||
section at the end for what has actually been verified.
|
||||
|
||||
## Supplied Monado source
|
||||
|
||||
Inspected `C:/Users/conno/Downloads/monado-main/monado-main/src/xrt/drivers/xreal_air/`
|
||||
and `src/xrt/targets/common/target_builder_xreal_air.c`.
|
||||
|
||||
| Model | USB vendor:product | Sensor/control interface | Sensor report |
|
||||
| --- | --- | --- | --- |
|
||||
| Air | 3318:0424 | 3 / 4 | 64 bytes |
|
||||
| Air 2 | 3318:0428 | 3 / 4 | 64 bytes |
|
||||
| Air 2 Pro | 3318:0432 | 3 / 4 | 64 bytes |
|
||||
| Air 2 Ultra | 3318:0426 | 2 / 0 | 512 bytes |
|
||||
|
||||
This driver implements HID IMU input, segmented factory-calibration JSON reads,
|
||||
gravity/gyro fusion, brightness 0–7 and 2D/3D mode commands. It does not register
|
||||
1S or One Pro. Copying those Air HID commands to the new products is unjustified.
|
||||
Its source carries Tobias Frisch's BSL-1.0 notice.
|
||||
|
||||
## Applicable One-family references
|
||||
|
||||
* [XRLinuxDriver](https://github.com/wheaney/XRLinuxDriver), inspected commit
|
||||
`36e2139dd482f5984112d0c3ebafcc07a78f99c0`. `src/devices/xreal.c` lists vendor
|
||||
`3318`, One Pro products `0435/0436`, and 1S products `043d/043e`. These use the
|
||||
non-HID transport. Its README explicitly lists both models and requires current
|
||||
firmware with glasses stabilization/anchor disabled when using external tracking.
|
||||
* [xrealOneDeviceKit](https://github.com/wheaney/xrealOneDeviceKit), inspected commit
|
||||
`f5833f611265f418b1ea549da20dc4764d86a4ec`. Its IMU adapter consumes the One TCP
|
||||
driver, interprets gyro as radians/sec and acceleration as m/s², then feeds fusion.
|
||||
Its factory-calibration JSON hook returns unsupported. The maintained Linux
|
||||
integration disables its SBS-control capability when no MCU connection is open;
|
||||
a static model table alone is not proof of working display switching.
|
||||
* [xreal_one_driver](https://github.com/rohitsangwan01/xreal_one_driver), MIT,
|
||||
inspected commit `3e912315ec16e94e9a47688770d649cfef293624`. Opens TCP
|
||||
`169.254.2.1:52998` with a two-second timeout, assembles 84-byte IMU records,
|
||||
reads a timestamp at byte 14, gyro floats at 34 and acceleration floats at 46.
|
||||
This is a transport/parser reference, not a drop-in Android glasses controller.
|
||||
|
||||
The source check confirms a practical path for both requested models. It does not
|
||||
establish that this Samsung phone exposes the glasses' USB Ethernet interface to
|
||||
ordinary Android apps, or establish the physical tracking axes on this hardware.
|
||||
|
||||
## DroidDeck integration path
|
||||
|
||||
1. Attach the 1S and inspect its USB descriptors, presentation display identity,
|
||||
Ethernet network/link properties and whether the IMU endpoint is reachable.
|
||||
Obtain the same evidence for One Pro before claiming hardware validation.
|
||||
2. Add exact model discovery and external-display selection. Preserve each panel's
|
||||
Android-advertised dimensions and modes; do not assume the Air's resolution.
|
||||
3. Implement a bounded One-family stream parser and cancellable socket transport.
|
||||
On Android, bind the socket to the relevant Ethernet `Network` when available,
|
||||
leaving Steam's Internet route alone. If Android does not expose that link,
|
||||
a USB network transport is additional work; no root requirement is assumed.
|
||||
4. Establish gyro units, timestamp deltas and axis mapping using stationary,
|
||||
directed yaw, pitch and roll captures. Route the validated fused pose through
|
||||
the same glasses/HeadTracker/SteamVR pipeline as VITURE and RayNeo. Avoid layering
|
||||
DroidDeck tracking on top of the glasses' own anchor/stabilization.
|
||||
5. Gate brightness and automatic stereo switching separately on a verified control
|
||||
protocol. IMU streaming alone does not justify showing those controls as working.
|
||||
6. Test malformed/fragmented/coalesced records, missing network, timeout/reconnect,
|
||||
USB detach, screen handoff, SteamVR launch and each physical movement direction.
|
||||
|
||||
At this initial investigation stage, no external SDK was installed, firmware was
|
||||
modified, or XREAL commands were sent.
|
||||
|
||||
## Further source investigation while the phone charges
|
||||
|
||||
Inspected [xr-tools](https://github.com/0xcaff/xr-tools), commit
|
||||
`65c11105434a998f09ac4a5f8e4de8736bb4d36c`, particularly
|
||||
`packages/xreal_one_driver/src/proto/net/`, `src/proto/usb/mod.rs`, and the IMU
|
||||
example. This describes considerably more of the One protocol than the simple
|
||||
TCP IMU reader above:
|
||||
|
||||
* Reports use TCP `169.254.2.1:52998`; controls use `169.254.2.1:52999`.
|
||||
Both have a two-byte message identifier followed by a four-byte **big-endian**
|
||||
body length. Control messages additionally contain a big-endian transaction ID
|
||||
and protobuf payload. A port must bound lengths, use unique transaction IDs,
|
||||
enforce deadlines, and close both sockets on detach.
|
||||
* Sensor messages `28 36` have a **128-byte body**, hence 134 bytes including the
|
||||
header. The older reader's 84-byte extraction is not the complete frame format.
|
||||
Body offsets: device ID 0, device timestamp 8, report type 24, gyro 28,
|
||||
acceleration 40, magnetometer 52, temperature 64 and IMU ID 68. Numeric sensor
|
||||
fields are little-endian; timestamp is documented as nanoseconds, gyro rad/s,
|
||||
acceleration m/s². Type 11 is IMU; type 4 is magnetometer. Do not interpret every
|
||||
report as gyro data or use arrival time as the device's integration interval.
|
||||
* Control `27 1f` reads factory JSON: IMU bias, temperature-dependent gyro bias,
|
||||
and display-to-IMU transforms. Its example subtracts the factory biases, fuses
|
||||
gyro/acceleration and applies the display rotation. This is preferable to
|
||||
guessing a fixed mounting angle, but the example's frame convention still
|
||||
needs conversion and directed-motion verification in DroidDeck.
|
||||
* `27 1c` sets brightness 0–9; `27 27` sets dimming; `28 22` selects regular/SBS
|
||||
input; `27 5e/5f` reads/sets display configuration; `28 a6/a7` reads/sets EIS.
|
||||
Documented display configurations are 1920×1080 at 60/90/120 Hz and 3840×1080
|
||||
at 60 Hz. These are evidence for One/One Pro, not confirmed 1S modes. Preserve
|
||||
the Android-advertised modes and verify controls separately on 1S firmware.
|
||||
* `28 29` is labelled buttons-enabled/buttons-disabled scene mode. Do not assume
|
||||
that command disables stabilization; the separate EIS control exists.
|
||||
* This project's USB detection labels `3318:0436` One Pro and `3318:0438` One as
|
||||
normal devices, and `0435/0437` as **recovery** devices. XRLinuxDriver accepting
|
||||
both IDs does not justify opening recovery products as ordinary glasses or
|
||||
sending firmware commands. xr-tools does not register 1S; its applicability
|
||||
there is a protocol-family inference, not hardware validation.
|
||||
|
||||
No license file or Cargo license declaration was found in this inspected xr-tools
|
||||
revision. No implementation was copied into DroidDeck. Use protocol observations
|
||||
for an independently written adapter, or resolve source licensing before a port.
|
||||
The MIT TCP driver and GPL XRLinuxDriver remain explicitly licensed references.
|
||||
|
||||
Also inspected XRLinuxDriver commit
|
||||
[`05d0cfb`](https://github.com/wheaney/XRLinuxDriver/commit/05d0cfb86abaacbf073e1a285ad4eabf681ccb05).
|
||||
It moves per-device pitch adjustment into the core pose path, after coordinate
|
||||
conversion; it does not add an Android transport or new One control protocol.
|
||||
Its pitch constants should not replace the GT mapping already verified by the user.
|
||||
|
||||
Next hardware check remains read-only: enumerate the 1S USB interfaces, display
|
||||
and Ethernet link; attempt bounded reads from both TCP services, save the factory
|
||||
configuration, then capture stationary/yaw/pitch/roll data. No XREAL support is
|
||||
advertised in this build. Android Ethernet availability and 1S control compatibility
|
||||
are still unverified.
|
||||
|
||||
## Official XREAL documentation and SDK inspection
|
||||
|
||||
Checked the user's [official documentation](https://docs.xreal.com/) on
|
||||
2026-10-14. This establishes an additional integration candidate alongside the
|
||||
direct TCP driver; it does not make that candidate a drop-in Android controller.
|
||||
|
||||
* The [device page](https://docs.xreal.com/XREALDevices/XREAL%20Glasses) explicitly
|
||||
includes One, One Pro and 1S in the X1-based One family with rotational tracking.
|
||||
* [SDK 3.1.0 release notes](https://docs.xreal.com/Release%20Note/XREAL%20SDK%203.1.0)
|
||||
add positional tracking for the One series with XREAL Eye and updated firmware.
|
||||
Beam Pro and Samsung S25 are its listed tested hosts. This does not establish
|
||||
6DoF on a camera-less 1S or validation on the user's SM-F976U.
|
||||
* [ControlGlasses](https://docs.xreal.com/Tools/ControlGlasses) is a required server
|
||||
for official SDK apps on compatible Android phones. It supplies device services,
|
||||
3D switching and dynamically loaded algorithm libraries. Beam Pro uses MyGlasses
|
||||
instead. An SDK integration would have a companion-app/version dependency.
|
||||
* The [quickstart](https://docs.xreal.com/Getting%20Started%20with%20XREAL%20SDK)
|
||||
describes Unity XR Plugin integration, ARM64 and OpenGL ES3. The
|
||||
[dual-display guide](https://docs.xreal.com/Tools/Dual%20Screen%20Display)
|
||||
confirms separate phone/glasses views and continued glasses output while the
|
||||
phone runs another app. This demonstrates the desired behavior in the SDK;
|
||||
it does not itself supply a native Kotlin implementation.
|
||||
* The [compatibility page](https://docs.xreal.com/XREALDevices/Compatibility)
|
||||
currently flags unresolved Android 16 compatibility. The
|
||||
[FAQ](https://docs.xreal.com/Frequently%20Asked%20Questions) also says DeX can
|
||||
interfere with SDK display operation, and Android 16 USB permission may require
|
||||
camera permission first when the device exposes an enabled UVC interface. Apply
|
||||
that permission requirement only when relevant; ordinary tracking must not
|
||||
start camera capture or enable UVC automatically.
|
||||
|
||||
Downloaded, but did not install or bundle, the official release archive linked
|
||||
from the 3.1.0 release notes:
|
||||
`https://public-resource.xreal.com/download/XREALSDK_Release_3.1.0.20251125/com.xreal.xr.tar.gz`.
|
||||
SHA-256: `fd7d0fce103712e7acc85bfcbb8b127007e1d84f64596efb6c11580bea058d90`.
|
||||
Size: 248,169,074 bytes.
|
||||
|
||||
Package inspection found Android AARs including `nr_api` and `nr_loader`, ARM64
|
||||
`libXREALXRPlugin.so` / `libXREALNativeSessionManager.so`, and Java
|
||||
`com.xreal.sdk.display.GlassesDisplay` plus display/surface listeners. The C# layer
|
||||
calls native exports for tracking type, resolution, refresh rate, component poses
|
||||
and session lifecycle; the loader also integrates Unity subsystems. Presence of
|
||||
these libraries does not establish a supported standalone JNI API or permission
|
||||
to redistribute a stripped subset. No C headers were present in the archive file
|
||||
listing inspected. No SDK source or binaries were added to the app.
|
||||
|
||||
Decision: retain both candidates. Before choosing the official native-library
|
||||
route, establish non-Unity initialization, head-pose access, ControlGlasses
|
||||
compatibility, display-surface ownership and distribution terms. Before choosing
|
||||
direct TCP, establish the Android network link and 1S protocol behavior. Compare
|
||||
both against DroidDeck's existing compositor instead of replacing its rendering
|
||||
architecture merely to fit the SDK's Unity sample.
|
||||
|
||||
## Native 1S implementation and hardware validation
|
||||
|
||||
The pre-implementation checkpoint is `4b045519`. The new driver uses independently
|
||||
written Kotlin TCP framing, control transactions, sensor parsing and gyro/gravity
|
||||
fusion. It does not bundle XREAL SDK binaries or copy the unlicensed xr-tools
|
||||
source. Protocol observations came from the references above and
|
||||
[xr-tools](https://github.com/0xcaff/xr-tools), inspected commit
|
||||
`65c11105434a998f09ac4a5f8e4de8736bb4d36c`.
|
||||
|
||||
Verified on the connected SM-F976U (Android reports release 17):
|
||||
|
||||
* Normal 1S USB identity is `3318:043e`, product name `XREAL 1S`. Only that exact
|
||||
product enables this driver; Air, One Pro and alternate/recovery IDs are excluded.
|
||||
* Android exposes NCM/ECM Ethernet networks at `169.254.2.10/24` and
|
||||
`169.254.1.10/24`. Individual sockets bound to the first network reach the
|
||||
glasses at `169.254.2.1`, control port 52999 and report port 52998. The process
|
||||
default network remains unchanged, preserving Steam's Internet connectivity.
|
||||
* Requests require the high bit of the transaction ID; responses clear that bit.
|
||||
Replies must match both command and transaction before values are published.
|
||||
* GET_DISPLAY returned payload `22 02 10 1d`: mode **29**, with Android reporting
|
||||
**1920 x 1200 at 60 Hz**. This differs from the older One's mode 2 (1080 lines).
|
||||
The 1S default 60 Hz mapping preserves mode 29.
|
||||
* A two-second capture delivered **2,000 valid IMU poses (1,000 Hz)**. Reports
|
||||
use the 128-byte body, type 11, little-endian device nanoseconds and float
|
||||
radians/sec / m/s² format. The framing handles partial and coalesced TCP reads.
|
||||
|
||||
The driver plugs into the shared glasses display and head-pose paths, including
|
||||
SteamVR. The FOV calculation now uses the actual per-eye aspect ratio, including
|
||||
the 1S's 16:10 display. Controls use acknowledged commands, bounded queues and
|
||||
timeouts; connection failures clear tracking state and retry. Volume is hidden
|
||||
because no volume protocol has been established. Missing Ethernet is reported
|
||||
without requesting HID permission or claiming USB interfaces from Android.
|
||||
|
||||
Directed wearer captures on the same device established the 1S sensor basis:
|
||||
level acceleration is approximately `(0, -9.65, -1.07)` m/s²; turning left gives
|
||||
negative gyro Y; looking up gives positive gyro X; tilting toward the right
|
||||
shoulder gives positive gyro Z with negative acceleration X. Both vectors are
|
||||
mapped as `(x, -y, -z)` into the shared right/up/back basis before initial gravity
|
||||
alignment and fusion. No RayNeo GT mounting angle is reused. Regression tests
|
||||
exercise those measured signs through `HeadPose`, check an upright first pose
|
||||
before recenter, and preserve the device's radians/sec and nanosecond units.
|
||||
|
||||
Stereo switching was verified at 3840 x 1080 / 60 Hz, returning to native
|
||||
1920 x 1200. On 1S, SET_INPUT performs the EDID switch itself; immediately sending
|
||||
SET_DISPLAY returns busy status 5001 (`22 03 08 89 27`). The driver now polls
|
||||
GET_DISPLAY across the temporary empty reply (`22 00`) and only sends a second
|
||||
mode command if the settled mode differs. Hardware retesting kept the connection
|
||||
and sensor stream running through the switch. The busy status is never accepted
|
||||
as a successful acknowledgement.
|
||||
|
||||
After installing the corrected build on 2026-10-14, the wearer confirmed that
|
||||
the view and motion feel good in response to the upright-view, yaw, pitch and
|
||||
head-tilt check. The SteamVR-requested session was running with a rendered first
|
||||
frame, stereo output and active tracking. All 181 glasses, shared-pose, SteamVR
|
||||
launch and glasses-UI regression checks passed; packaged runtime inputs verified.
|
||||
|
||||
The wearer also confirmed on 2026-10-14 that XREAL 1S brightness control works.
|
||||
Higher refresh rates were not tested.
|
||||
|
||||
Still experimental: dimmer, 90/120 Hz compatibility modes and
|
||||
extended gameplay stability need hardware validation. One successful view/motion
|
||||
check does not establish long-session stability. Users must select Follow mode and disable the glasses' own
|
||||
stabilization. Factory calibration and XREAL Eye/6DoF are not implemented.
|
||||
One Pro is not enabled or claimed as tested.
|
||||
Reference in new issue
Block a user