mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-11 02:51:10 +00:00
Kurt Himebauch
22a675a664
## Problem and intent Desktop sessions need configurable mouse, keyboard and application actions from a controller. On dual-screen devices, a lower-screen trackpad may already handle the pointer, leaving the physical controls available for other assignments. This PR gives physical controllers and local touch controls a shared profile/action model instead of separate configuration paths. This follows the centralization direction discussed in [#1202](https://github.com/Droid-Deck/DroidDeck/pull/1202#issuecomment-6299647180) and addresses the controller-input portion of #860 and #1356. Those discussions are context, **not prior maintainer approval** of this implementation. I would appreciate architectural feedback and guidance on splitting or coordinating this work if that would make review easier. ## Profiles and configuration - Controller stays within the existing Setup tabs. Game and Desktop each have a default-profile selector, followed by one profile list and device settings. - Named profiles reference three independently reusable parts: Physical controller, Main screen and Second screen. Layout geometry and action bindings are stored separately. Profiles can share physical controls while assigning different touch actions. - The session menu selects a complete profile, or expands to mix the three parts. A mixed selection can be saved as a new profile; Game and Desktop retain separate selections. - Both screens use the same full-size editor, with add, move, resize, snapping, undo and shared properties. The main-screen editor has movable, collapsible tools and is transparent above a running session. When editing the secondary display, its entire area is the canvas and the main display hosts tools/properties. - The old fixed gamepad, fixed Deck controls and separate Screen keys editor are replaced with editable presets and the shared renderer. Existing positions, bindings, shapes, symbols, imported images and toggle behavior migrate; legacy Screen keys become additional profiles without replacing the user's selected profile. Migration retains a backup before removing old preferences. - Saved layouts have thumbnails and screen targeting. Actions include compatible native output/remapping, keys/combinations, mouse buttons, pointer movement, vertical/horizontal scrolling, directional keys, existing menu/keyboard commands, and disabled. Buttons can use symbols, shapes and toggle behavior. ## Architecture and latency ```mermaid flowchart LR Library[Profiles + reusable layouts and bindings] --> Router[InputRouter] Physical[Physical controller adapters] --> Router Touch[Main + secondary InputSurfaceView] --> Sources[Owned pad / Deck sources] Sources --> Router BLE[Steam Controller BLE adapter] --> Router Router --> Native[Existing controller publication] Native --> Steam[Steam Input / guest controller] Router --> Held[Shared key and mouse ownership] Held --> Guest[Existing guest input outputs] Router --> UI[Existing menu / keyboard hosts] ``` Bindings resolve when the profile changes, outside event dispatch. Native forwarding has no additional frame/timer queue; identity routes reuse existing state/reports. Only synthesized continuous stick pointer/scroll motion uses a frame callback. One source releasing a key/button does not release another source's hold. Menu, profile, suspension and removal transitions release old outputs and gate held controls until neutral. Steam continues to own game-specific mappings and action sets. Steam Mobile Touch and VR retain their specialized paths. Raw device adapters remain responsible for transport/calibration and must join routing before publishing consumed controls; this does not claim that every adapter has been unified. Historical Thor measurements on the native Game path found kernel-event-to-guest-observer upper-bound medians of **3.061 ms baseline versus 2.999–3.403 ms candidate**, with a subsequent installed-APK check at **3.039 ms / 300 matched edges**. These are relative software-path measurements, not button-to-photon latency. Some tails increased, simultaneous BLE contention remains unmeasured, and **this UI revision has not had a new latency measurement**. The previous cursor fix changed visual update intervals from roughly 33.3 ms to 16.7 ms without changing sensitivity or adding smoothing. [Full design, migration and boundaries](https://github.com/Delete228/DroidDeck/blob/codex/unified-controller-input/docs/development/unified-input-proposal.md). ## Validation Updated through upstream 542bbf89c; current PR head08526f828. The latest CodeRabbit review generated no actionable comments. - **1,017 tests passed in 133 suites after the latest upstream merge**, with no failures, errors or skips: all input/UI tests, global reset, component updates, PiP policy, runtime auto-update and Proton menu auto-selection. The pre-merge review-fix run passed **967 input/UI/global-reset tests**. The subsequent second-screen refresh fix passed **35 focused surface/presentation tests**, including three new state-preservation regressions. - Full local run before the final review follow-ups: **3,598 tests, 3,573 passed, 25 failed**. The 25 are unchanged upstream macOS limitations: log-export tests requiring SecureDirectoryStream (24) and a case-sensitive WoW add-on test (1). The global-reset regression also passes. Earlier disk-exhausted attempts were discarded; the completed run bounded temporary resources with one worker and JVM recycling, without excluding tests or changing repository test configuration. - APK build, runtime/session assets, native dependencies, alignment and v1/v2/v3 signatures passed. Installed over the separate Input Test app on **AYN Thor** with its environment retained. Local packaging omits Rust store engines; the [full Build APK workflow for08526f828passed](https://github.com/Droid-Deck/DroidDeck/actions/runs/42295403480), including Android/JVM, Python/native, Rust store-engine and Xbox checks. - Review follow-ups keep migration IDs stable, preserve unreadable supported-version libraries before recovery, retain forward-version protection, release keyboard owners on focus loss, handle unavailable presentation displays, and allow controller confirmation/cancellation after key capture. Deck router cleanup checks ownership; BLE teardown synchronously blocks late routing/publication, and shared Triton slot ownership protects a replacement controller from stale instances. Legacy Select-to-Steam migrates to Desktop without replacing saved bindings; mouse backend transitions are serialized with publication. Profile refresh now updates custom controls in place, preserving terminal/dashboard state, the field keyboard and WoW chat. No physical BLE-controller test or new latency measurement was performed. - The latest08526f828APK is installed on Thor. Profile preferences stayed byte-for-byte identical and the retained Steam environment reached READY at 1280×720 / 60 Hz. The earlier capture-dialog fix was also exercised on Thor. APK SHA-256: `ed6ef888648468b4ebeb94e8f1ae74a195bb1f0a3f27e4fb3646d5462a4beb6b`. - Upstream integration preserves the new WoW chat panel, platform-pointer and PiP fixes. The fast-double-tap stick-click rule and its six regressions were adapted to the shared renderer; WoW field selection also preserves Ctrl held by a profile. - Latest Thor UI pass: existing Setup navigation, profile/layout previews, action persistence, centered element creation, main-screen editing over Steam, secondary canvas with primary tools/properties, and X returning from the docked menu to controls. - Across the documented device sessions, the user tested native physical buttons/sticks/triggers, Deck pads/grips, desktop pointer/clicks/scrolling/arrows, Android keyboard with physical input, Ctrl+F and Ctrl+A/X/V/S, simultaneous physical/touch holds, menu release and Home/resume. These historical checks are distinguished from the latest UI-only pass. [Revision-specific results, APK hashes and limitations](https://github.com/Delete228/DroidDeck/blob/codex/unified-controller-input/docs/development/unified-input-validation.md). ## Coordination and remaining work - #1202 (jmarti326): shortcut capture/device identity and the shared-assignment discussion informed this work. - #1305 (kingokksa): shared PC-key definitions were adapted from that PR; this does not absorb its entire feature set. - #1414, #581, #1226 and #1150 remain relevant for external-device input, calibration, keyboard routing and DualSense integration. The DualSense coordination discussion in this PR describes adapter boundaries and duplicate-stream suppression. Import/export, automatic per-app selection, macros, multi-button activation chords and automatic text-field keyboard detection remain out of scope. A single binding emitting a key combination is implemented. The configuration UI is now implemented and exercised on Thor. I am happy to continue UX/accessibility/localization refinements here before merge, or in follow-up PRs, according to maintainer feedback. The main review questions remain the shared model, source ownership/lifecycle contract, migration policy and integration with other input work. ## UI walkthrough on AYN Thor Actual captures from the installed Input Test build. In paired images, the main display is above the second display. The test application name and local build badge are not production branding changes. <details> <summary>Controller settings: separate defaults for Desktop and Game, followed by the profile library.</summary> <img width="960" alt="Controller settings: separate defaults for Desktop and Game, followed by the profile library." src="https://github.com/user-attachments/assets/a4c156d6-9ded-43df-b0f8-e5f34028806d" /> </details> <details> <summary>Physical controller: bindings grouped by control type, with each current action visible.</summary> <img width="960" alt="Physical controller: bindings grouped by control type, with each current action visible." src="https://github.com/user-attachments/assets/a8a840e3-19ac-473f-9f55-b06ae9c637b5" /> </details> <details> <summary>Reusable layouts: bounded previews and separate screen targets.</summary> <img width="960" alt="Reusable layouts: bounded previews and separate screen targets." src="https://github.com/user-attachments/assets/cf0cf7d1-42f3-46cd-b50b-f01e1fa8ed41" /> </details> <details> <summary>Main-screen editor over the live Steam UI, with the movable tools expanded. The background is Steam artwork, not a running game.</summary> <img width="960" alt="Main-screen editor over the live Steam UI, with the movable tools expanded. The background is Steam artwork, not a running game." src="https://github.com/user-attachments/assets/0781f7c8-fd10-47e5-8a23-817f945a6289" /> </details> <details> <summary>Dual-screen editor: tools and selection on the main display, full-size placement canvas on the second display.</summary> <img width="720" alt="Dual-screen editor: tools and selection on the main display, full-size placement canvas on the second display." src="https://github.com/user-attachments/assets/f13d4ef6-be6d-4d95-9d42-e047e0a63b8a" /> </details> <details> <summary>The selected lower-screen trackpad stays visible while its properties open on the main display.</summary> <img width="720" alt="The selected lower-screen trackpad stays visible while its properties open on the main display." src="https://github.com/user-attachments/assets/26baf597-f037-419e-9601-e48f086c0d4f" /> </details> <details> <summary>Session menu: expand Profile to select the physical, main-screen and second-screen parts independently.</summary> <img width="720" alt="Session menu: expand Profile to select the physical, main-screen and second-screen parts independently." src="https://github.com/user-attachments/assets/27ac4d9c-68b3-4257-8353-23104e2343de" /> </details>
DroidDeck brings the SteamOS experience to Android: Steam Gaming Mode, full Linux desktop, integration with GOG, Epic Games, Xbox, and other storefronts, Steam VR, and more!
100% open source. No ads. No telemetry. No source withheld.
DroidDeck has no website. Only download it from this repository's Releases.
Install
- Android 9 or newer on Adreno 730 or newer, or any 8xx. Adreno 6xx is experimental. Mali, Xclipse, PowerVR and Adreno 710 are unsupported. No root needed.
- About 3 GB free for the runtime, plus 1.1 GB for the desktop and emulators.
- Install the APK, install the Linux runtime, press Play and sign in to Steam.
- On Android 14 and newer, turn on Disable child process restrictions in Developer options. On Android 12 and 13, Steam's first launch shows a Fix it for me button that walks you through it.
Smart glasses setup is in the support matrix. Release notes are in docs/releases.
Help, Preview builds and device reports are on Discord. For bugs, post in #bug-reports with the zip from Share logs.
Build
Needs Docker, Java 17, the Android SDK and NDK, Rust via rustup, GitHub CLI, Python 3 and zstd.
tools/build_local.sh
The APK lands in app/build/outputs/apk/release/. Install it with tools/deploy_local.sh. After the first run, ./gradlew :app:assembleRelease reuses the prepared inputs. Unit tests run with ./gradlew -PskipRust=true :app:testDebugUnitTest, and the helper and preload suite with bash tools/test_local.sh.
On Windows, see tools/run_windows.ps1 and tools/prepare_windows_runtime.py.
Limits
Compatibility and performance vary by device. Desktop compositing is software rendered, and Firefox sandboxing is reduced under proot.
Credits and licence
DroidDeck is GPL-3.0. Steam and Proton belong to Valve Corporation. VITURE, RayNeo, XREAL, Rokid, REDMAGIC, Nintendo Switch and Joy-Con are trademarks of their owners. This project is not affiliated with any of them.
Runtime, Wine and Proton
- Runtime, shim, input and controller work build on WinNative and Bannerlator by maxjivi05.
- mmap ring buffer controller input by ribbit384 (WinNative #345).
- FEX (MIT) runs x86 code. DroidDeck's build puts the platform name under
AT_PLATFORMso sharun AppImages by VHSgunzo (MIT) start through its userland-execve loader (MIT). FEX's code disk cache was checked in its source. - x86 AppImages, and arm64 ones whose runtime can't unpack on Android, are unpacked with uruntime by VHSgunzo (MIT), shipped unmodified with its licence.
- x86 programs linking OpenSSL 3 use the copy in the Steam Linux Runtime 4.0. x86 Linux games get the
/etc/asound.confthat pressure-vessel writes. WriteProcessMemoryhooks fall back to/proc/<pid>/memthe way Wine'sserver/ptrace.cdid before Wine 11.- Wine write watches use userfaultfd support in Proton's
dlls/ntdll/unix/virtual.c(LGPL-2.1-or-later). libblsession supplies the arm64 syscall number and answersPAGEMAP_SCANfrom/proc/self/pagemap. - libblsession runs Wine's arm64
SIGUSR2context handler (dlls/ntdll/unix/signal_arm64.c, LGPL-2.1-or-later) in place on a frame laid out like Linux'sarch/arm64/kernel/signal.c, and servesfill_vm_countersindlls/ntdll/unix/process.cthe kernel's own/proc/<pid>/status. - droiddeck-ntsync follows the Linux ntsync semantics in
Documentation/userspace-api/ntsync.rst(GPL-2.0). - The proot fast path orders bindings like PRoot's
path/binding.c(GPL-2.0-or-later). - DiRT 3 and DiRT Showdown keep their telemetry socket the way RoccoC's DiRTTelemetryErrorFix (MIT) does, based on the fix by Carsten Gottschling.
- Games use Mesa Turnip's per render pass memory choice. The Steam client keeps
TU_DEBUG=sysmem. - Black Ops III and Dark Souls Remastered start without umu-protonfixes (BSD-2-Clause), whose
311210.pyand570940.pyfail under ARM64 Proton. - ReSkate (GPL-3.0) by Dingo Shenanigans uses the launch option from its README and D3D12 feature level 12_1.
- The arm64 client's sign in was checked by reading Steam's saved token with the
ConnectCacheformat from deadlock-api-ingest.
Stores
- Epic Games and GOG build on Phobos's adaptation of Legendary and gogdl. GOG install script registry entries are read the way Heroic Games Launcher (GPL-3.0) reads them.
- Epic's official Linux launcher preview runs through FEX. That setup is outside Epic's listed Linux support.
- Battle.net is by @kayakyakr. It runs Blizzard's own client in Proton (downloaded from Blizzard, not included) and shows key art from the Battle.net Shop. Without Steam, its games see the pad as an Xbox 360 controller via
PROTON_SPOOF_STEAMINPUT_VIDPIDin Proton'sdlls/winebus.sys. - Battle.net installs drive Blizzard's Agent the way @barncastle's Battle.Net-Installer does, with requests from wowdev.wiki. Read without taking code: @mubaraknumann's unifideck, @djmittens's neo-mittens
bnetswitch, @BCD1210's soju and Lutris'sproduct.dbreader. - World of Warcraft addon imports replace an addon's folder from a temporary copy, as WowUp does, and read flavour folders the way WoWDeck does. No code from either is used.
- Xbox games load @domsvoboda's xodus-nexus modules (LGPL-2.1-or-later), including
wintypes.dll. Minecraft's GameInput setup was checked against @Wyze3306's BedrockOnLinux (MIT). - The Xbox Cloud Gaming player uses the WebRTC client of @unknownskl's Greenlight (MIT), and its offer tweaks and sharpening follow @redphx's Better xCloud (MIT). See
tools/xcloud-player/NOTICE.md. - The itch.io butler pack ships butler's
7z.so, itch.io's build of 7-Zip by Igor Pavlov (LGPL-2.1, unRAR restriction on RAR code), unmodified. Other archives use Apache Commons Compress and the runtime's bsdtar (libarchive). - The Store uses Flatpak (LGPL-2.1) through DroidDeck's own bubblewrap shim. Chromium and Electron apps start directly, since zypak by @refi64 (BSD-3-Clause) always uses Flatpak's spawn portal.
- Folder sync runs Syncthing (MPL-2.0), built from @researchxxl's syncthing-android fork (MPL-2.0).
Audio, video and HUD
- Steam audio goes through @The412Banner's DirectAudio PulseAudio sink (LGPL-2.1-or-later), built from
tools/directaudio/sink. - The DroidDeck HUD is @The412Banner's FusionHUD (GPL-3.0 with an attribution term). Its readings match Valve's MangoHud build (MIT), with frame times from gamescope patch 0125.
- The HUD's CPU, GPU load, thermal and battery readers come from @xXJSONDeruloXx's work.
- LSFG frame generation comes from Camille LaVey and the Eden project, following lsfg-vk, ported by @maxjivi05. It needs your own copy of Lossless Scaling and ships none of its shaders.
- Remote Play decodes through Android's MediaCodec or the FFmpeg (LGPL-2.1-or-later) in Steam's runtime. Qualcomm low latency settings follow Moonlight (GPL-3.0).
- PipeWire (MIT) sockets use the settings in its
module-access.c, and WirePlumber (MIT) keeps those clients at full permissions.
Desktop
- The KDE pacman repository snapshots Arch Linux ARM and installs with pacman (GPL-2.0-or-later).
- KWin (GPL-2.0-or-later) is patched to pass on the scroll data its
wayland_backend.cppleaves as a TODO. - Read without taking code: Qt's
qwaylandinputdevice.cppandqguiapplication.cpp, Konsole'sTerminalDisplay.cpp, and Mesa'su_threaded_context.candzink_resource.c. - The Steam Keyboard window rule follows Valve's
steamdeck-kde-presetswith Bazzite's0001-steam-virtual-keyboard.patch. - Steam's Browse buttons open a GTK 3 file chooser via PyGObject (LGPL-2.1-or-later), shown over Steam by gamescope (BSD-2-Clause) through
GetAppIdFromReaper. - The AYN Odin 2 phantom second display was confirmed by @SapphireRhodonite's WatermelonDS 0.0.5 notes and iiSU #363, without taking their code.
Smart glasses
- Builds on WinNative's VITURE work. Protocol checked against @wheaney's XRLinuxDriver, @badicsalex's ar-drivers-rs and go-macos/iokit. The optional VITURE SDK is not included.
- The VITURE protocol V2 frames adapt @elasticjava's viture-v2 (MIT) by Holger Bartnick. See
docs/third-party/viture-v2. - RayNeo adapts @informalTechCode's RayNeo-Air-Series-3DoF-Scaffold (Apache-2.0) and references @verncat's RayNeo-Air-3S-Pro-OpenVR (MIT) and Monado (BSL-1.0). See docs/third-party/rayneo.
- XREAL 1S references XRLinuxDriver and xrealOneDeviceKit by @wheaney, @rohitsangwan01's xreal_one_driver (MIT) and @0xcaff's xr-tools.
- Rokid follows ar-drivers-rs, @badicsalex's Void Computing write-ups and Monado's Rokid driver.
- REDMAGIC display findings draw on @mekhontsev's MagicDesk investigation and @Coding-BR's decompiled Nubia kernel.
VR, Joy-Con and hand tracking
- Joy-Con VR ports the arm model from Google's GVR SDK for Unity (Apache-2.0) and adapts the hand skeleton from Valve's OpenVR
handskeletonsimulationsample (BSD-3-Clause), which follows Moshi Turner's work in Monado. Index offsets come from Monado'svive_poses.c(BSL-1.0). - Joy-Con research: Linux's hid-nintendo by Daniel J. Ogorchock, @dekuNukem's Nintendo_Switch_Reverse_Engineering, SDL and AOSP. Checked against ALVR, OpenComposite, PSMoveSteamVRBridge, @F8LEFT's JoyConVr, @krnelpan1c's Daydream-Controller-SteamVR, @yoyobuae's joyconvr, Moonlight and Dolphin. See docs/third-party/joycon.
- Camera hand tracking runs Google's MediaPipe Pose Landmarker (Lite), BlazePose GHUM 3D and Hand Landmarker (Apache-2.0) on CameraX frames (Apache-2.0), with usage reports off. Checked against JoyConVr's camera app (MIT).
- Tracking math: the GVR arm model's accelerometer mode, Eric Foxlin's zero velocity updates, Frosio, Pedersini and Borghese's accelerometer autocalibration, a Kalman filter (Kalman, 1960), Horn's quaternion method (1987) and the 1€ filter (Casiez, Roussel and Vogel, 2012).
- Gestures follow ALVR's
hand_gestures.rs(MIT) and Meta Quest's menu pinch, tuned on HaGRID photos (CC BY-SA 4.0, not included). - The glove model follows OpenVR's render model reference, is meshed with surface nets (Gibson, 1998; Mikola Lysenko), and uses Inigo Quilez's smooth minimum, distance field ambient occlusion and polygon distance (MIT). See docs/third-party/handtracking.