DroidDeck/docs/development/unified-input-proposal.md

Shared input profiles and configurable control surfaces

This change implements a shared routing model for physical controllers and local screen controls. It is intended for review of the architectural direction, with a functional configuration UI and hardware validation on AYN Thor. The related maintainer discussions motivate this work; they are not approval of this implementation.

The configuration UI keeps the existing Setup tabs. Controller contains mode defaults, one profile list and device settings. Reusable layouts and actions are stored separately but are managed within each profile source. Direct editing on the target display has been exercised on AYN Thor, including editing above a live Steam session and keeping the secondary canvas separate from the main-screen tools. Revision-specific checks and remaining limits are recorded in the validation report.

Why a shared system

Desktop sessions need mouse, keyboard and application actions from a controller. A dual-screen device may already have a comfortable lower-screen mouse, leaving its physical sticks available for other actions. The same logical button or pad can appear on the main screen, lower screen or a physical device; its location should not dictate which actions it can perform.

The model separates three concerns:

• Source role: physical controller, game overlay (primary) or auxiliary panel (secondary). Swapping the game and auxiliary panel between displays preserves the panel's role and bindings.
• Profile: references to a physical binding preset and a layout/binding pair for each touch role. Game and Desktop are editable bases; named profiles can share parts. Auto has separate Steam and Desktop assignments. Opening a profile for editing does not activate it.
• Surface layout: stable element IDs, types, names, positions and sizes. Named custom layouts can be used on either touch role. Actions belong to a separate binding preset associated with that layout. Two desktop profiles can share their physical bindings and layout while using different touch actions.

A binding selects native controller output, a DroidDeck action, or disabled. Native output continues through the existing game/Steam path, optionally with a compatible gamepad remap. A local action consumes that control so, for example, R3 assigned to Ctrl+F does not also press the guest's R3 button.

flowchart TD
    Physical[Android controller events] --> PadBridge
    Primary[Primary on-screen controls] --> PadBridge
    Secondary[Secondary custom controls] --> PadBridge
    Deck[Deck pads and grips] --> DeckControls
    BLE[Steam Controller BLE reports] --> TritonInputRouter
    Library[Saved profiles, layouts and binding presets] --> Profiles[Resolved active profile by source role]
    Profiles --> Router[InputRouter]
    PadBridge --> Router
    DeckControls --> Router
    TritonInputRouter --> Router
    Router --> Native[Existing pad / Deck report publication]
    Router --> Held[Shared key and mouse ownership]
    Router --> Commands[Existing menu and keyboard hosts]
    Router --> Motion[InputMotion: active stick pointer / scroll only]
    Motion --> Held
    Held --> Guest[Existing guest keyboard and mouse output]
    Native --> Steam[Steam Input or guest controller consumer]
    Layout[Independent surface geometry and element IDs] --> Secondary
    Layout --> Primary

These are routing relationships, not additional asynchronous queues. Native publication is performed by the existing source adapters and writers.

Implementation

Component	Responsibility
InputProfile	Logical controls, source roles and supported actions
InputLibraries / InputLibrary	Versioned reusable parts, validated references, selection and migration
InputProfiles	Compatibility API for the compiled active profile and legacy migration
InputSurfaceLayout	Bounded normalized touch geometry and legacy layout migration
InputSurfaceSource	Independently owned per-element routing through existing pad/Deck publishers
InputRouter	Resolve channels on profile changes, route events, track held outputs
PadBridge	Existing player slots plus independently owned local touch sources
DeckControls	Source ownership for Deck grips, contacts, coordinates and clicks
TritonInputRouter	Apply routing before Steam Controller BLE report publication
PhysicalInputBindings	Capture additional Android keys with device identity
HeldInput	Coordinate router outputs with existing keyboard and gesture outputs
InputMotion	Frame callback for active synthesized stick pointer/scroll movement
InputTiming	Optional bounded dispatch-to-publication diagnostics

Supported actions are native output, compatible native remapping, disabled, keyboard keys/combinations, mouse buttons, pointer movement, scrolling, four keyboard directions, and existing menu/keyboard/Steam/quick-access commands. Analog controls used as buttons have separate activation and release thresholds. Key combinations press modifiers first and release in reverse order.

Standard controller assignments are shared by the physical role. Additional captured keys match a device identity plus Android keycode/scancode, rather than persisting a transient Android device ID. System-reserved keys remain with Android. This is not a per-device calibration system.

Controller navigation

Setup keeps its existing top-level tabs. Controller starts with Desktop and Game mode defaults, followed by one profile list and inline device settings. Opening or editing a profile does not activate it; the session menu owns the active profile override. Quick-menu overrides are called out beside the default selectors.

Each profile opens Physical controller, Main screen or Second screen. Control cards and touch element rows open the same large properties editor. Layout editing uses the full-size canvas and one draggable, expandable Tools palette. Back returns to the source and profile being edited, rather than the library root. Saved layouts and reuse commands live inside the relevant source editor; there are no parallel top-level layout, binding and profile catalogs.

New profiles copy actions and custom geometry independently by default. Users can explicitly keep physical controls shared at creation, or copy/link a source from another profile later. Reusing a saved layout can link geometry without linking actions. Make independent copies both geometry and actions for that source. Reset actions affects only the selected profile, preserving its layouts and detaching shared actions before resetting them.

Main-screen visibility remains a global device setting. Color and opacity belong to each saved touch layout. The standard gamepad is an ordinary editable layout: its legacy positions, appearance and stick options are migrated into elements. Other saved resolutions remain selectable layouts. The previous fixed gamepad renderer and its separate editor are removed; only its placement calculation is retained for migration. Vibration, radial stick dead zone, Joy-Con VR, global shortcuts and timing diagnostics are inline below the profile list. This adds no settings reads or queues to input dispatch.

Session profile composition

Both the session drawer and launcher controls sheet expose Profile, Physical controller, Main screen and Second screen. Choosing a profile loads its three saved parts; choosing one part creates a mode-specific override without editing the source profile. Game and Desktop remember their choices separately. Auto resolves the corresponding default profile. Selecting a complete profile clears part overrides. Save as new profile creates independent copies of all three parts, including geometry, appearance and actions.

Fixed lower-screen tabs, side selectors and click bars are removed. Deck pads and grips, one or two Deck pads, grips alone, the mouse trackpad with keyboard, and a compact gamepad are ordinary editable layout/binding presets. Their menu, keyboard and click buttons are ordinary elements too. The compact gamepad reuses the previous panel geometry calculation. Existing fixed-panel selections migrate to profile controls on first use. Schema 4 also preserves version-3 custom layouts and converts legacy fixed layout references without changing physical mappings. Active quick-menu parts cannot be deleted.

Mouse touchpad gestures reuse TouchpadGestures for relative movement, tap, hold-drag, two-finger scrolling and secondary clicks. A tap uses the element's configured click action; it does not reposition the pointer. Gesture outputs use owned router channels and are suppressed while input is suspended or until a held contact returns to neutral after a profile change. Native Deck pad mode retains contact/position reporting and click behavior for Steam Input.

Reuse and migration

Existing FakeInputWriter transports, player assignment, guest keyboard/mouse output, KeyboardHost and second-screen presentation are reused. The existing button, stick, d-pad and trackpad views now live in InputSurfaceView; there is no parallel fixed controller panel. The custom editor arranges the existing button, stick, d-pad and trackpad views; it does not create another emulated controller. Trigger elements provide press/release input. A new element is unassigned until an action is explicitly saved. Native sticks and pads retain their existing click gestures. Movement/contact and click are separate assignments for sticks and trackpads. Both use the same properties editor and router ownership rules; migrated click bindings remain editable, and a custom mouse pad can assign its click separately.

The editor uses the target display at actual size. On a dual-screen device, Add, Undo, Snap, Save and element properties stay on the activity display; the auxiliary display is a full-size canvas with no toolbar inset. On one display, a draggable Tools bubble expands into those same tools, including element properties, and collapses to leave the canvas accessible. Properties replace the canvas and return to the same draft. Snapping aligns centers and edges with other elements and the display, including during resize. Done persists geometry and actions together; Cancel discards the draft after confirmation. The preview has no input sources and cannot send input to the running session.

Primary custom layouts follow the existing on-screen-controls visibility setting; Steam's touch-controller mode retains priority. Secondary layouts use Profile controls mode. Terminal, desktop and dashboard remain separate panel modes. Steam-managed touch controls keep their own editor; a new editable layout can start empty or copy a saved custom layout.

Legacy Game/Desktop mappings, key captures, active selection and lower-screen geometry migrate to the library together. Legacy element IDs preserve their existing logical bindings. New elements use stable independent IDs, so two buttons can have different actions or hold the same output without releasing one another. Referenced parts cannot be deleted, and editing an unsupported library version cannot overwrite it. Shared edits show their scope; copying a binding preset from a profile relinks only that profile's selected role.

Library decoding, validation and preset resolution happen during configuration or profile activation. Dispatch uses preconfigured router channels, with no preference reads, profile lookups or new asynchronous queues. The physical native identity path and frame-based stick motion remain in place. These properties do not substitute for a fresh latency comparison on hardware.

The old touch-only mapping and Select-as-Steam settings migrate once into Game's primary-screen and physical bindings respectively. Their redundant UI is removed. PadBridge retains raw SELECT state; only the profile maps it to GUIDE. The old flag is read for migration, not applied again during event handling, so explicit native/disabled/local bindings and profile resets remain authoritative. Attaching or removing a router cannot change which raw button a SELECT release clears. Invalid stored entries fall back to defaults. The global settings reset clears both profile and surface settings without reimporting the old mappings.

The shared PC-key definitions are adapted from kingokksa's #1305 at 76fc9b31. Device capture and identity handling build on the approach in jmarti326's #1202 at 42db788f. The initial implementation adapted these parts rather than merging either PR wholesale. Upstream subsequently merged #1202; the October 30 integration retains its global physical shortcuts, saved assignments and configuration UI alongside the profile bindings.

The November 4 integration reuses upstream's common bottom-screen bar, extracted trackpad/terminal panels and screen-swap presentation. Custom controls coexist with the dashboard and KDE desktop panels. Routed on-screen stick updates retain upstream's throttled visual redraws without deferring pad publication. Upstream hardware-key translation and mouse capture use the existing output ownership and release capture while the input-profile editor is open.

Explicit captured bindings in the active profile take precedence over a global physical shortcut for the same key. Otherwise the global shortcut precedes the default Back/QAM handling and logical controller routing. A consumed shortcut does not also reach the guest, and its release remains consumed across menu changes. Shortcut capture remains available from Input profiles, the session menu and mode settings. Profile editing gates execution of these shortcuts.

Ownership and lifecycle

Each source retains its own contribution. Releasing a physical mouse button does not release an identical button still held on the lower screen. Closing a surface releases that surface, including the final release on the existing Epic launcher mouse route. Existing gesture and keyboard outputs participate in shared key and mouse-button ownership.

Native pad merging uses OR for buttons, the maximum trigger value, and the strongest complete stick vector. A native Deck trackpad retains one source's coordinates until that contact ends, then accepts a waiting contact. Concurrent surfaces do not overwrite its coordinates during the same contact.

Profile changes, suspension and menu entry release old local outputs. Held controls must return to neutral before activating their new bindings. Device removal and surface disposal remove their owned state. The existing menu navigation path remains available independently of gameplay assignments.

When an adapter attaches retained input state to a replacement router, it seeds the new channels without emitting presses and requires held controls to return to neutral. Already-open pad rings receive the released state immediately; attaching a router, changing a profile or enabling input does not open an unused ring. Steam-menu and VR stick-switch requests use separate system-owned button pulses, so primary-surface bindings cannot suppress or remap those commands.

The menu and keyboard commands currently open their respective UI; they are not toggles. Back/B closes the menu. This can be reconsidered with the UX work.

Hot path and scheduling

Profile JSON, preferences and layout geometry are processed outside event routing. Channels resolve their actions when configuration changes. The current router uses mutable per-source state under one synchronized monitor; it is not an immutable compiled plan. This keeps shared ownership consistent, but contention with simultaneous BLE input has not yet been measured on hardware.

For an unchanged Game binding, PadSource returns the original PadState. The BLE identity route returns its original report array. Modified reports use reusable buffers and preserve unrelated timestamp, motion, pressure and extension fields. Normal controller events do not wait for a frame, coroutine or timer.

Only synthesized stick-to-pointer/scroll motion uses Choreographer, because a held stick must keep moving without new device events. Trackpad deltas and button changes publish synchronously on the normal UI input path. Pointer output arriving from an off-main-thread adapter is marshalled to the existing UI cursor path; application commands also use the main handler. Neither schedules native forwarding.

The mapped cursor reuses the existing compositor coordinates and Android cursor layer. Trackpad contact starts from the current pointer position, preventing jumps when switching sources. Publishing pending Compose snapshot notifications before drawing avoids the observed every-other-frame cursor update without changing the stick response curve or adding smoothing.

See validation for measurements and their limits. The evidence supports no clear typical latency regression in the tested native path, not zero overhead or a complete input-to-display latency guarantee.

Steam ownership and boundaries

Steam remains responsible for game mappings, action sets, layers, radial menus and its cloud profiles. Native/Steam-managed here means forwarding the controller through the existing path; it is not a new integration with Steam's profile API. Steam Mobile Touch retains its own Steam-managed layout and protocol. Its widgets do not pass through this local remapper. The VR-specific input path also remains separate. Existing gestures retain their recognizers and share output ownership.

The local UI revision includes named profiles, reusable binding presets and custom layouts edited on either display. Import/export, automatic per-app selection, macros, configurable tap/hold behavior and multi-button activation chords remain deferred. One button emitting Ctrl+F is supported; pressing multiple buttons to activate a new binding is not implemented. Built-in Steam touch layouts retain their separate Steam-managed editor.

Physical native remaps stay within compatible report families. Generic custom buttons choose a supported button output explicitly. Additional hardware adapters must join before publishing consumed controls; an Activity-only interceptor cannot cover every raw device. Future adapters should use this contract without replacing device-specific calibration or protocol handling.

Related upstream work

Reviewed on 2026-10-24. Several related proposals remain open; this is not a claim that no other controller work exists or that this implementation supersedes it.

• #1202: physical shortcuts and capture. The maintainer's centralization feedback motivates one shared assignment model. Questions there about overlapping actions inform consumption/ownership; multi-button activation remains deferred.
• #1305: screen key mappings, combinations and layout management. This change shares the action model across physical and screen sources, but does not implement that PR's full layout UX.
• #1414: external-controller button and touchpad-press capture, including devices exposing a touchpad press as pointer input. Our key capture does not yet implement that pointer-contact adapter. It is a concrete area for coordination.
• #860 and #1356: desktop controller input. This implements controller-driven pointer/keyboard actions, not every gesture or automatic keyboard behavior requested in #1356.
• #581: hardware calibration and device profiles remain a separate responsibility from action profiles.
• #1226: keyboard host selection; the local action reuses the existing host rather than implementing another IME.
• #1150: raw DualSense input. That proposed adapter will need integration before publication; it is not covered merely by the current Android key capture.

Review focus

The main questions are whether source roles and profile boundaries match the project's direction, whether adapters join at the right point, and whether the shared ownership/lifecycle contract is suitable for future input sources. The validation document distinguishes device-tested behavior, host regressions and remaining coverage. UX/UI refinement can follow agreement on those decisions.

Settings polish and display targeting

The session Profile row expands to show the three saved parts and the editor. Choosing the profile remains a separate selector. Session editing opens its settings on the spare display, when available; element properties and editor commands stay on the activity display. Main-screen placement over a running session uses a transparent canvas so controls can be arranged around game UI. Existing visible controls are suspended while editing.

Schema 5 adds an explicit main/secondary/both target to saved layouts. Migration infers the target from existing profile and quick-menu references, preserving layouts already used on both screens. Layout rows render a miniature with the same InputSurfaceView renderer; unused legacy copies are retained under Other saved layouts. Newly created layouts belong to the source being edited.

Action editing retains a draft for each route and action type. Sticks assigned to a digital action press above 55% on either axis and release below 35%; a trackpad holds a digital action during contact. These semantics are explained next to the action. Horizontal scroll uses the same router and existing evdev REL_HWHEEL / Wayland axis transport, with independent fractional accumulation. No additional event thread or queue is introduced.

Screen-key consolidation

Schema 6 imports the retired screen-key overlay into normal saved profiles, layouts and element bindings. The active old set and each named saved set become separate imported profiles. Current Game/Desktop defaults and quick selections remain unchanged; the settings page explains where to find the imported profiles. The old editor, overlay View, guest-key dispatch path and session-menu entries are removed. Only a one-time reader of the old preferences remains.

Migration converts normalized centers and density-based sizes into the shared layout geometry. Key chords, four-way directions, separate eight-way stick bindings, toggle buttons, built-in symbols, custom image references, shape and opacity are retained. Blank stick diagonals keep their previous behavior of pressing neighboring cardinal directions. Imported images are decoded once per view, sampled to at most 512 pixels, and read only from the app's existing image directory. New buttons can choose text or an existing built-in key symbol.

A converted library and the original raw preferences are committed together before old preference entries are removed. Invalid data or capacity failures do not erase the old library or keys, and settings show a pending-import notice. The backup is recovery data, not a second runtime input configuration.

Toggle buttons use InputSurfaceSource and the same router ownership as every other element. Releasing one owner cannot release another owner's held key. Menu suspension, profile changes, cancellation and detach release latched output. Physical native input continues through its existing fast path; no queue, timer or additional event thread is introduced for these features.