mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-21 22:05:59 +00:00
183 lines
12 KiB
Markdown
183 lines
12 KiB
Markdown
# Steam's touch controller in a DroidDeck session
|
|
|
|
Research notes, started 2026-10-07. The question: instead of DroidDeck drawing its own on-screen
|
|
pad and keeping its own profiles, can the session use the touch controller Steam already has for
|
|
the Steam Link app, so that touch layouts are per game, synced to the Steam account and shared
|
|
through Steam's community configs?
|
|
|
|
This page records what the Steam client contains, the candidate approaches, and the experiments
|
|
run on the Thor to decide between them. Findings are appended as they come in; see
|
|
[Experiment log](#experiment-log).
|
|
|
|
## How Steam Link's touch controller works
|
|
|
|
Sources: the arm64 client DroidDeck installs (`steam_client_steamdeck_publicbeta_linuxarm64`,
|
|
manifest version 1791249696), the protobuf definitions in
|
|
[SteamDatabase/Protobufs](https://github.com/SteamDatabase/Protobufs)
|
|
(`steam/steammessages_remoteplay.proto`, `steam/steammessages_virtualcontroller.proto`) and Valve's
|
|
[touch controller guide](https://steamcommunity.com/app/353380/discussions/4/1735462352489233412).
|
|
|
|
The work is split between the streaming client (what runs on the phone) and the Steam client being
|
|
streamed from (the host):
|
|
|
|
- **Client: drawing and editing.** `streaming_client` holds `CVirtualController`, built from
|
|
`streaming/client/virtualcontroller.cpp`: it draws the controls over the video, runs the layout
|
|
editor (tray, colour picker, move/scale), and sends raw touches to the host as
|
|
`CInputTouchFingerDownMsg` / `Motion` / `Up` (finger id, x and y normalised to the video).
|
|
- **Host: the controller.** `steamui.so` holds the stream server (`CStreamServer`,
|
|
`IStreamServerInputDelegate`). `steamclient.so` holds `CMobileTouchControllerAbstraction`, a
|
|
Steam Input controller backend next to `CSteamOSHandheldAbstraction` (the Deck),
|
|
`CSteamControllerAbstraction` and `CGenericGamepadControllerAbstraction`. Its display name is
|
|
"Steam Link Touch Controller" and its config type is `controller_mobile_touch`.
|
|
- **Host: the config.** The host owns the touch config and pushes it to the client:
|
|
`k_EStreamControlTouchConfigActive` (appid, revision, creator),
|
|
`GetTouchConfigData`/`SetTouchConfigData` (config data and layout blobs),
|
|
`SaveTouchConfigLayout` (the client sends back an edited layout), `Get`/`SetTouchIconData`,
|
|
`TouchActionSetActive` and `TouchActionSetLayerAdded`/`Removed`. Touch configs are Steam Input
|
|
configs like any other: per game, saved to the account, publishable. SteamInputDB counted
|
|
10,540 community configs for Mobile Touch in March 2026.
|
|
|
|
Every piece of this is in the arm64 package DroidDeck already installs: the host side in
|
|
`steamrtarm64/steamclient.so` and `steamui.so`, and the full renderer and editor in
|
|
`steamrtarm64/streaming_client`. The package also carries an `androidarm64/libsteamclient.so`
|
|
with the same touch strings.
|
|
|
|
## Where DroidDeck is today
|
|
|
|
The session's pad is presented to the Steam client as a Steam Deck controller (28de:1205 through a
|
|
served `hidraw` node and a written sysfs, see `SteamDeckPad.kt` and `fakeinput_steam.cpp`), so Deck
|
|
configs, per-game bindings and Deck community configs already apply. What Steam's touch controller
|
|
would add: control placement, icons, touch menus and on-device layout editing defined by Steam, and
|
|
the touch-specific community configs.
|
|
|
|
## Candidate approaches
|
|
|
|
| | How | For | Against |
|
|
|---|---|---|---|
|
|
| **A. Local Remote Play** | DroidDeck streams from its own Steam client the way the Steam Link app does (pairing, start stream, touch input negotiated), using Valve's `streaming_client` or a minimal client of our own. | Valve's controller, editor, sync and sharing unchanged. Leans on a protocol Valve keeps compatible with shipped Steam Link apps. | Host captures and encodes video nobody needs; the client may refuse to stream to itself. |
|
|
| **B. In-process injection** | Feed touches straight into the path between the stream server (`steamui.so`) and `CMobileTouchControllerAbstraction` (`steamclient.so`), with no stream. | No video cost. | Reverse engineering stripped arm64 binaries; breaks when Steam updates; layout drawing is still ours. |
|
|
| **C. Data only** | Read Steam's touch configs and layouts, draw them ourselves, keep feeding the Deck pad. | Simple. | Touch configs bind straight to keys, mouse and actions; honouring them means reimplementing Steam Input. |
|
|
|
|
## Experiments
|
|
|
|
Run in order; each one either rules an approach out or narrows the next.
|
|
|
|
1. **Host side alive?** Does the arm64 client in a DroidDeck session run the stream server, answer
|
|
discovery on the network and accept a Steam Link pairing?
|
|
2. **Touch controller instantiated?** Stream to the official Steam Link app (same phone or another
|
|
device) and check that the touch controller appears, takes a game config and drives a game.
|
|
3. **Loopback.** Can a client on the same device (127.0.0.1 / the device's own address) stream from
|
|
the session's Steam, and what does it cost (capture, encode)?
|
|
4. **Input without video.** Does input keep flowing with the stream paused or captured tiny
|
|
(`k_EStreamControlPause`, `SetCaptureSize`), or with a client that never decodes video?
|
|
5. **Valve's renderer as an overlay.** Can `streaming_client` itself run inside the session as an
|
|
overlay over the game, rather than the phone app?
|
|
6. **Injection points.** How the stream server hands touches to `steamclient.so`, to size approach B.
|
|
|
|
## Experiment log
|
|
|
|
### 1. Host side alive: yes (2026-10-07, Thor, Steam 1791249696)
|
|
|
|
- With a Steam session READY, the client listens on TCP 27036 and UDP 27036 (`/proc/net`, app
|
|
uid). A discovery probe built from `steammessages_remoteclient_discovery.proto` sent to
|
|
`127.0.0.1:27036` from the device gets a `CMsgRemoteClientBroadcastStatus` back: hostname
|
|
`DroidDeck`, `enabled_services` 98 (game streaming on), the signed-in user, `steam_deck`
|
|
unset, `gaming_device_type` 541.
|
|
- The official Steam Link app (Play Store, on the Thor's bottom screen while the session runs on
|
|
the top one) finds "DroidDeck" by itself, shows a PIN, and the session's Big Picture shows
|
|
**Authorize Device**. After the PIN the app runs its network test and lists DroidDeck as
|
|
paired. Pairing to the session's own client over loopback works with no changes.
|
|
- Launching Steam Link on display 0 instead puts DroidDeck's session activity into PiP; use
|
|
`am start --display 4`. On first start Steam Link sits on its splash until BLUETOOTH_CONNECT
|
|
is answered, and the permission dialog does not render on the second screen (granted with
|
|
`pm grant`).
|
|
|
|
### 2. Touch controller over the stream: works end to end
|
|
|
|
Starting a stream (`streaming_log.txt`, `console_log.txt` in the session's Steam `logs/`):
|
|
|
|
```
|
|
CLIENT: Sending HID device 2020/0112/-1 Xbox Wireless Controller at sdl://3
|
|
CLIENT: Sending HID device 0000/11fb/-1 Mobile Touch Control at touch://0
|
|
...
|
|
Remote Device Found type: 0000 11fb path: touch://0 Product: Mobile Touch Control
|
|
!! Steam controller device opened for index 1.
|
|
Controller device closed after hid_read failure
|
|
Controller 1 disconnected
|
|
```
|
|
|
|
- **The touch controller reaches the host as a remote HID device, 0000:11fb**, opened by the
|
|
same controller code that opens the Deck pad, and turned into a virtual controller. The
|
|
streaming client is what makes the reports; Steam on the host only sees a HID device.
|
|
- Video fails on the host: `PipeWire: Could not connect PipeWire context` (the session runs no
|
|
PipeWire), so the app gets audio but a black screen and keeps its loading spinner.
|
|
- The app then re-sends only the Thor's built-in pad and drops the touch device; it seems to
|
|
withdraw touch controls when a physical controller is present, or until video arrives.
|
|
|
|
With video fixed (below), **Steam's touch controller runs against the session's own client**:
|
|
|
|
- Steam Link withholds its touch controls while a physical pad is attached (the Thor's own pad
|
|
shows up as "Xbox Wireless Controller"). A four-finger tap opens its stream menu, which has
|
|
**Enable Touch Controls** (injected with `sendevent` on `/dev/input/event5`, the bottom panel).
|
|
- The client then re-sends `touch://0`; the host opens it as controller type 43, product 0x11fb,
|
|
serial `MT-<Steam Link device id>`, capabilities `0000007f83045bff`, creates a virtual controller
|
|
for it, and sends the client `TouchConfigActive`, `TouchActionSetActive` and
|
|
`SetTouchConfigData`. The app draws Big Picture's touch layout (d-pad, ABXY, Steam button,
|
|
keyboard, `...`).
|
|
- Touching the controls drives the session: the touch Steam button opens Big Picture's main
|
|
menu on the Thor's top screen, touch B closes it, and the session's glyphs switch to the touch
|
|
controller's. (D-pad taps from `input tap` did not register; not chased.)
|
|
- Cost: the host encodes with libx264 in software (`/dev/video-enc0` missing), 4 threads,
|
|
1240x698. CPU sat at 72-85 % in the overlay and `k_EStreamControlVideoOverflow` flooded the log;
|
|
the app's picture froze within a minute while input kept working. **Video loopback is not a
|
|
shippable path**; only the input half of the stream is useful.
|
|
|
|
Video fix used for this test only: PipeWire and WirePlumber are in the rootfs and gamescope is
|
|
built with PipeWire, so a `gamescope` wrapper on `PATH` (via `Download/droiddeck-env`) that starts
|
|
`pipewire` and `wireplumber` first gives gamescope a capture node (`stream available on node ID`)
|
|
and the client a picture. (Editing `droiddeck-session` on the device does not stick: the app
|
|
re-syncs the overlay each session.)
|
|
|
|
### Where Steam keeps touch configs
|
|
|
|
In the session's Steam, `steamapps/common/Steam Controller Configs/<account id>/config/`:
|
|
|
|
- `<appid>/controller_mobile_touch.vdf` for each game with a saved touch config (dozens already
|
|
there for this account, synced down from Steam Cloud), plus named copies like
|
|
`<appid>/default touch_0.vdf`;
|
|
- `configset_controller_mobile_touch.vdf`: which config each app uses (`autosave` = the per-app
|
|
file above, otherwise a template or workshop id);
|
|
- `configset_MT-<serial>.vdf`, `preferences_MT-<serial>.vdf` and `config/MT-<serial>_gyro.vdf`:
|
|
per touch device.
|
|
|
|
A touch config is an ordinary `controller_mappings` VDF (`controller_type` `controller_mobile_touch`,
|
|
bindings per action set and layer) with one extra key, **`touch_layout`: the on-screen layout as
|
|
a hex-encoded `CVirtualControllerLayouts` protobuf**: one layout per action set, each a list of
|
|
elements (`EControllerElementType`, visible, x/y position normalised to the screen, x/y scale)
|
|
and a colour, plus input mode, mouse mode and pinch-zoom settings. Bindings and layout travel
|
|
together in the one file that syncs and is shared as a community config.
|
|
|
|
### The touch device's HID protocol
|
|
|
|
From Steam Link for Android (`libmain.so`, exported symbols; `CVirtualController` is both the
|
|
renderer and the HID device behind `touch://0`):
|
|
|
|
- **Input report: 40 bytes** (`CVirtualController::Read`), kept at `this+200` and marked dirty
|
|
at `this+240` by every setter. Bytes 0-7: button/touch bits (e.g. bit 27, 19, 20: trackpads
|
|
0, 1, 2 touched). Bytes 16, 20, 24: trackpads 0-2 as int16 x (centred, `x*65535 ^ 0x8000`) and
|
|
y (inverted). Remaining fields (sticks, triggers, gyro) not yet mapped.
|
|
`CHIDDeviceReportGenerator::BInjectGamepadStateMobileTouch` is a stub; the report is the
|
|
controller's own format, not the generic SDL gamepad state.
|
|
- **Output reports from Steam** (`Write` → `QueueFeatureReport` → `HandleFeatureReports`),
|
|
at least 17 bytes, first byte the type:
|
|
- 1: rumble (u16 low, u16 high, u32 duration);
|
|
- 3: setting `0x30` with a u16 value (a flag at `this+1488`, likely gyro);
|
|
- **4: action set change** (u32 appid, u32 action set id): switches the layout shown;
|
|
- **5 / 6: action set layer added / removed** (u32 appid, u32 layer id).
|
|
- **Feature report 2**: battery (level, from `SDL_GetPowerInfo`), 17 bytes.
|
|
|
|
So Steam drives action sets, layers and rumble over HID. The only things that come solely over
|
|
the stream are the config and layout blobs (`SetTouchConfigData`), the active config's
|
|
appid/revision/creator (`TouchConfigActive`) and custom icons (`SetTouchIconData`); the first two
|
|
are on disk in the VDF above.
|