DroidDeck/docs/development/cardboard-vr-learnings.md

Cardboard VR and SteamVR on Turnip: what we learned

Written 2026-10-13, after the first stretch of Cardboard mode work on the AYN Thor (Snapdragon 8 Gen 2, Adreno 740, Turnip on KGSL). docs/cardboard-vr.md describes how the feature works. This page records what it took to get there: the traps, why each fix looks the way it does, how to debug a live session, and what is still open. Read it before the next round of VR work.

Where it stands

What	Result
Cardboard mode (lens pass, 3DoF head tracking, recentre)	Works on any picture; picture follows SteamVR on and off
SteamVR 2.17.10 ARM64 (the Steam Frame's deckard-steamvr)	Starts from the drawer; driver-direct at 60 Hz, dashboard and Steam library in VR
Thumper (356400, D3D11, VR launch option)	Menus and levels at 60 Hz, 7 ms CPU / 13 ms GPU, 23 of 6485 frames dropped
The Thrill of the Fight (494150, Unity)	Gym loads, 60 Hz, 6.5 ms CPU / 8.5 ms GPU
The Lab (450390)	Loads, then Android's low-memory killer takes the whole app
SUPERHOT VR (617830)	Reaches its EULA in the dashboard (not accepted in testing)
SteamVR Home	Blocked: Proton's vrclient_x64 lacks IVROverlayView_002
helloxr -g Vulkan2 (OpenXR)	Session focused and submitting, picture shows as torn tiles

Screenshots from the Thor: Thumper gameplay, Thumper title, The Thrill of the Fight, the SteamVR dashboard with a game's EULA.

Games run as x86-64 Windows builds through Proton ARM64 (FEX, DXVK). Proton's vrclient_x64 forwards OpenVR to SteamVR's ARM64 vrclient.so, so no x86 SteamVR is involved.

Turnip on KGSL: the four things SteamVR trips over

1. VK_KHR_display makes the GPU disappear. Turnip on KGSL enumerates no physical device when an instance enables it ("I can't KHR_display"). vrcompositor always asks for it. The layer takes the display extensions off the instance and fakes one display, mode and plane backed by a headless surface. Headless surfaces report a UINT32_MAX extent, so the swapchain is sized from the mode.

2. No exportable semaphores. SteamVR shares a timeline semaphore with every shared image as an opaque fd, and Turnip/KGSL cannot export one. The layer creates the semaphore unshared and hands out a memfd (droiddeck-shared-sem) holding a 64-bit counter. A poller thread in each process carries values both ways every 300 µs. Dropping the semaphores outright, the first attempt, gave torn dashboard tiles.

3. Wait-before-signal blocks the submitting thread. Turnip does not defer a submit that waits on a timeline value nobody has signalled yet; vkQueueSubmit sleeps until it is. SteamVR's client waits on a compositor value that the compositor signals only after the client has submitted and said so, so every client deadlocked before its first frame:

   • helloxr blocked in a wait-idle in vrclient.so;
   • DXVK's dxvk-queue thread blocked in Proton games.

   The layer now gives a submit's wait on an imported shared value 3 ms, then drops it, at once when the shared counter has not moved for 100 ms. It is kept when the value is already reached, when this process has queued the signal itself, or while a Quest session's frame socket exists (checked at most once a second). Three details mattered:

   • the waits are judged before the submit's own signals count, because an acquire waits on one value and signals the next on the same semaphore;
   • SteamVR's client interop device is Vulkan 1.0, so only the KHR entry points resolve there;
   • a Proton game waits on two values a frame that are never signalled (5,500 dropped, none reached within 200 ms afterwards). With the grace on each, Angry Birds VR's submit thread lost 6 ms a frame and the game ran at 60 fps; without it, 120.

4. UBWC layouts must agree. An imported image has to be created with the same VkImageFormatListCreateInfo (sRGB + UNORM) as its creator, or Turnip lays out the compression differently and the reader sees garbage. The presenter's eye-texture import does this.

Other traps, in the order they cost time

• A still headset goes to standby. A phone on a table looks unworn, SteamVR enters standby after about 15 s, and the compositor stops submitting. The driver's proximity sensor reads "worn" while Cardboard is on.
• Emulated SysV semaphores never set sem_otime. SteamVR's web helper waits for it to become non-zero before using a semaphore set, so the dashboard never appeared. sysv.c now stamps sem_otime and sem_ctime.
• Steam's web helper needs the layer too. It shares Steam's in-VR overlays with SteamVR and crashed without the layer; after a few crashes Steam quits. Its GPU process outlives SteamVR runs, so steamwebhelper gets the semaphore bridge whenever SteamVR is installed, not only while SteamVR's active marker exists. It never gets the display stand-in.
• Keep the layer out of ordinary sessions. As an implicit layer it loads everywhere. Display extensions are only taken off while SteamVR runs, a new session clears a leftover active marker, and without SteamVR installed every process gets the driver's functions unchanged. Review feedback on the first version pointed out that gamescope asking for a display extension would have been rewritten in an ordinary session.
• The active marker race. Clearing a stale marker whenever vrserver is not running also cleared it during startup, before vrserver appeared, because the app polls status every 3 s. Steam quit as a result. Only a marker older than 2 minutes is cleared now.
• Signal handlers in Proton games. The layer's crash tracer replaced the SIGSEGV handler that FEX and Wine rely on, and The Lab crashed in FEX's JIT. It now installs only where nothing handles the signal.
• mangoapp owns gamescope's single external overlay slot, so an X11 presenter has to use STEAM_OVERLAY. The default presenter is now a Wayland window on DroidDeck's own compositor, kept on top by app id.
• Memory. Steam, SteamVR and one VR game fit in the Thor's 12 GB. A large game such as The Lab, or two games at once, push the device into critical pressure, and the low-memory killer takes DroidDeck itself (ended-without-teardown.txt in the session logs, lowmemorykiller in logcat).
• Adreno 840 forced sysmem. On the Fold with VITURE Luma Ultra, launching SteamVR from the library inherited TU_DEBUG=sysmem and produced vkerror=-4, triggering recovery. Adding noubwc still failed; replacing the flags with noubwc or an empty value worked. The shared launch helper removes only sysmem on Adreno840*, leaving other flags, the parent Steam session and other GPUs unchanged. The user confirmed improved landing-area brightness and colors with this configuration and asked to preserve them.
• Adreno 840 with DD-Turnip v0.2.0. SteamVR reproduced vkerror=-4 with sysmem removed, including on the previous app build and with the presenter disabled. After a phone reboot, the unchanged fcf8c588 baseline rendered for 109 seconds without observed device loss. TU_DEBUG=flushall initially appeared promising: a global run rendered for over a minute, and a SteamVR-only run rendered for about 32 seconds before background suspension. The packaged SteamVR-only change then reproduced device loss with flushall confirmed in vrcompositor, so the automatic flag was removed. These bounded runs do not establish the underlying cause or a reliable workaround.
• Live files are not cache. Android's installd evicted cache/shm/u10528-* while SteamVR was running, followed by Chromium shared-memory failures and lost IPC. Shared memory, emulated device files, sandbox trees and PRoot temporary files now use no_backup/guest-runtime, retaining their existing cleanup lifetimes. Session state must survive cache eviction.
• One host signal owner. The submit-wait path must not mirror a shared timeline value itself: the poller can signal it between the submit thread's counter read and signal, producing an invalid duplicate signal. The poller now owns mirroring; a forced-interleaving test covers this race.
• Lost hand tracking is not a disconnect. Publishing a disconnected pose and then a connected, untracked pose on every optical update floods SteamVR with device-change notifications. With uncalibrated hands this grew its web buffer to about 34 MB and triggered the Web thread watchdog. Release inputs and publish only the connected, untracked pose; reserve disconnection for an actual controller removal. The controller regression exercises repeated loss and reacquisition.
• Failed realtime scheduling can still install a CPU limit. On the Fold, SteamVR's SDL RTKit fallback set RLIMIT_RTTIME to 100/200 ms despite realtime priority being refused. The render thread later received kernel SIGXCPU during IOCTL_KGSL_GPU_COMMAND, with policy SCHED_OTHER and the soft limit advanced to 1100 ms, confirming the realtime watchdog fired. The VR scheduling preload rejects lowering this limit only in vrcompositor when RLIMIT_RTPRIO is zero and CAP_SYS_NICE is absent. It preserves inherited limits, other processes and other resources; it does not intercept signals. SDL's scheduling hint cannot cover SteamVR's direct SDL_LinuxSetThreadPriorityAndPolicy call. A native preload regression covers rejection and forwarding. The exact Android kernel boost/accounting path was not established.
• Frames shown out of order. Video from the glasses jumped back a few frames in bursts while the head turned. Two causes, each found by stamping a frame counter into the presenter's output:
  • the presenter copied SteamVR's eye texture a fixed 4 ms after Present, with no fence. Under GPU load vrcompositor had not drawn it yet, so the copy got that texture's previous frame. The layer now fences vrcompositor's submits and the presenter waits for the frame's count;
  • DroidDeck's compositor copies a Wayland client's dma-buf without waiting for its implicit fence, and KGSL does not order the copy after the client's GPU work. The presenter's blit sat behind the game's frame, so the compositor showed the swapchain image's older picture. The presenter now waits for its blit before it presents.
• Flat games under SteamVR. With SteamVR running, Steam lists a flat game after its own window in gamescope's base layer, so Big Picture keeps the focus and the game never shows, theatre setting or not. SteamVR is therefore started only by Play SteamVR, never with the glasses alone. A flat game launched while SteamVR runs now opens as a SteamVR panel instead (droiddeck-steamvr-flat). Games launched from the SteamVR dashboard do not pass through a SteamClient.Apps.RunGame hook installed later, so the launcher asks Steam at run time rather than marking the click.
• A new projection needs an event. SetDisplayProjectionRaw changed what vrserver reports, but vrcompositor kept the frustum it started with, so the glasses' field of view did nothing until SteamVR restarted. The driver now follows it with VREvent_LensDistortionChanged.
• Head-tracking axes. With the display at ROTATION_90, screen x is minus device y and screen y is device x. Getting this wrong made a turn at the hips tilt the view. Recentring now keeps the world vertical and resets the heading only.

Library launch validation (2026-10-15)

Two hands-off launches through the Steam library's Launch button completed 308 seconds and 188 seconds, with an intentional stop between runs. Each retained its compositor process and session, sustained 60 fps, showed dashboard/world transitions, and kept USB control and valid 6DoF head tracking. Neither run reported device loss, a watchdog abort, recovery, or a session restart. Hand controllers remained enabled but uncalibrated. Brightness stayed at 8; no shader color changes or temporary graphics overrides were used.

These timed runs used APK SHA-256 4ce7a623ceb6b1a100c0ed1184530ece2ea8e13a1cb9ee6c0ab24666a8249004. A newer calibration/recenter APK was subsequently installed and verified; see luma-spatial-evidence.md. The timed results do not establish sustained gameplay or calibrated-hand acceptance for that newer build.

Debugging a live session

• Logs: ~/.local/share/Steam/logs/{vrserver,vrcompositor,xrclient_*}.txt and /run/droiddeck/droiddeck-steamvr/vrstartup.log. Lines from the driver start with droiddeck:, lines from the layer with [droiddeck-vr-layer]. A game's layer messages land in the session log.

• Markers beside active in /run/droiddeck/droiddeck-steamvr/:

  • crashtrace prints backtraces on fatal signals;
  • semtrace logs each bridged semaphore's local and shared value every 2 s, named by the memfd inode so the same semaphore can be matched across processes, plus the first waits, signals and shared image create infos.

  Both are read when a process starts.

• Frame stats: in /opt/steamvr/bin/linuxarm64, LD_LIBRARY_PATH=. ./vrcmd --stats reports submits, drops and app CPU/GPU times. ./vrcmd --hidedashboard closes the menu SteamVR opens over a new game.

• Where a process is stuck: a stack scan of /proc/<tid>/mem from the saved stack pointer (a small Python helper) gives module+offset frames. That is how the wait-idle in vrclient.so and the blocked dxvk-queue thread were found.

• VR launch options: SteamClient.Apps.GetLaunchOptionsForApp(id) lists them, and SteamClient.Apps.RunGame("<id>", "", <index>, 100) starts one, through droiddeckctl cdp eval.

• Quitting: droiddeckctl quit <appid>. Without the id the game keeps running.

• Pitfalls when testing:

  • never pkill -f or pgrep -f a pattern inside a guest command, because it matches the command's own shell; use pgrep -x;
  • replace a loaded .so by copying to a temporary name and mv, never cp over it;
  • a session start restages the driver and layer from the APK assets, so a hand-copied build is lost on the next start.

Open items

• helloxr picture. Both sides create the swapchain images with identical parameters, so this is not the UBWC import issue. Reading an image mid-render (the dropped wait) or binning tiles is the next suspicion.
• SteamVR Home needs IVROverlayView_002 in Proton's vrclient, or an ARM64 build of Home.
• The Lab and other large games need memory headroom. Candidates: stop Steam's own web helpers while a VR game runs, or lower SteamVR's footprint.
• Wait drops are a heuristic. A 3 ms grace keeps the common case in sync, and an idle counter skips it. A submit thread in the layer, deferring submits until their waits are reachable, would be exact.
• Layers: only the first layer of each frame is presented. No reprojection.
• Controllers are a gaze pointer driven by the pad. Pad presses did not close the SteamVR menu in testing. Untested: whether that came from gamescope focus being on Steam, or from the system button mapping.