mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-20 05:16:01 +00:00
feat: agent bridge wayland and gamescope coverage (#406)
* feat: run a guest agent in every session for commands, focus and the Steam client's DevTools * feat: give the agent bridge game, input, UI and experiment commands behind a Setup toggle * feat: droiddeckctl commands for the new bridge, a scenario runner and contract tests * feat: run cleanup steps after every scenario run so a failure cannot leak into the next * feat: restart a session that died between scenario runs with start --reuse * fix: give droiddeckctl stop its own, longer wait for the session's logs * fix: harden debugging tools access and output
This commit is contained in:
1 parent
37e89c5b4d
commit
d231564f8e
42 files changed
+3787
-168
No files matched your search
+176
-26
@@ -1,41 +1,191 @@
|
||||
# Agent control
|
||||
|
||||
The debug APK exposes a ContentProvider at `content://com.droiddeck.launcher.agent` and an Activity for starting sessions. Android's shell-only `DUMP` permission protects both. They are absent from release APKs.
|
||||
`tools/droiddeckctl` drives DroidDeck on a real device over ADB, for agents and scripts that test
|
||||
the app end to end: start and stop sessions, launch and quit Steam games, wait for a game to have
|
||||
focus, send input, inspect the app's UI and gamescope's windows, run commands inside the session,
|
||||
and collect logs, screenshots and recordings. `tools/droiddeck-scenario` repeats a scripted test
|
||||
and reports every run.
|
||||
|
||||
Build and install the debug APK:
|
||||
## How it works
|
||||
|
||||
- **The provider** `content://com.droiddeck.launcher.agent` (`agent/AgentBridgeProvider.kt`)
|
||||
answers each command with JSON. It and the session-start Activity (`AgentStartActivity`) are in
|
||||
every build and require `android.permission.DUMP`, held by the ADB shell and privileged or
|
||||
explicitly granted callers. The provider checks the calling permission before decoding requests.
|
||||
- **The guest agent** `droiddeck-agent` runs inside every session (started by
|
||||
`droiddeck-session`). It runs commands with the session's own environment, publishes gamescope's
|
||||
focus, and talks to the Steam client's DevTools, which only accepts guest processes. The app and
|
||||
the agent exchange request and answer files under the session root (`files/session/agent`).
|
||||
- **The UI dump** reads the app's Compose semantics from inside the app (`agent/AgentUi.kt`), so it
|
||||
works on every display and window. UI Automator, by contrast, dumps whichever window has focus,
|
||||
which on a dual-screen device is often another app's.
|
||||
|
||||
## Access
|
||||
|
||||
Player controls and observations work over authorized ADB on any build: `state`, `start`, `stop`,
|
||||
`resume`, `wait`, `launch`, `quit`, `focus`, `input`, `ui`, `logs`, `screenshot`, `record`, `displays`,
|
||||
`wake`, and `access`.
|
||||
|
||||
**Setup > Session > Debugging tools** is off by default in every build, including debug builds.
|
||||
It enables `run`, `guest`, `cdp`, all `env` operations, `override`, and `prefs`. The toggle switches
|
||||
directly without a confirmation dialog. The bridge refuses to change it through `ui` or preference
|
||||
writes. The existing preference key, `setting-agent-commands` test tag, JSON command names, and
|
||||
`AGENT_COMMANDS_DISABLED` error code are unchanged. Refusals exit 7; `droiddeckctl access` reports
|
||||
whether commands are enabled.
|
||||
|
||||
Disabling the toggle clears queued and persistent agent environment settings and installed binary
|
||||
overrides. Disabled settings cannot apply at the next session start. Already running commands and
|
||||
effects already applied to a session are not undone; stop and restart that session to discard them.
|
||||
|
||||
Authorized command execution runs inside the runtime containing Steam's saved login. CDP evaluates
|
||||
JavaScript in the authenticated Steam client, and preferences can contain private values. These
|
||||
capabilities remain unrestricted when enabled: the toggle is an explicit opt-in, not credential
|
||||
isolation or a sandbox against the authorized caller. Main already supported arbitrary guest
|
||||
programs through `run`; restricting credential access would require addressing that route too.
|
||||
ADB access remains a trust boundary, and ADB input can operate the device's own toggle.
|
||||
|
||||
The helper exchanges private files with the app and adds no network listener. The guarded Steam
|
||||
DevTools port is bound to loopback and accepts only peers with a fresh private registration for a
|
||||
live socket descriptor, matching socket inode, destination port, and address family. The existing
|
||||
x86_64 Decky loader exception remains: that mode does not enable the guest-only acceptance guard.
|
||||
|
||||
Requests are limited to 256 KiB of encoded JSON. Guest responses and DevTools messages are limited
|
||||
to 1 MiB; provider replies above 384 Ki characters return `RESPONSE_TOO_LARGE` to avoid overflowing
|
||||
Binder. Command output retains at most 256 KiB per stream while continuing to drain both streams.
|
||||
Timeouts must be finite, positive, and no greater than 600 seconds, and terminate the command's
|
||||
process group. Override installation refuses path traversal and symbolic links.
|
||||
|
||||
Automatic process logs record environment keys and argument counts rather than values or full
|
||||
argument lists. UI nodes marked as passwords redact text, editable text, descriptions, and state
|
||||
text. This does not redact arbitrary authorized command results, CDP data, preferences, or screen
|
||||
captures. Scenario directories are owner-only and JSON reports are written with mode 0600. Review
|
||||
artifacts before sharing; a live `logs` pull copies the current folder without a fresh export scrub.
|
||||
|
||||
## Device selection and output
|
||||
|
||||
The CLI resolves one authorized device using `ADB_SERIAL` or `ANDROID_SERIAL` when set. Otherwise it
|
||||
de-duplicates transports that report the same device serial and asks for an explicit serial if
|
||||
multiple devices remain. Pass `--serial` before the command to choose directly; set `ADB` (or pass
|
||||
`--adb`) to choose an adb executable. `tools/deploy_local.sh` uses the same resolver.
|
||||
|
||||
Every command writes one JSON object to stdout and its resolved serial to stderr. Exit codes: 0
|
||||
success, 2 invalid or rejected command, 3 ADB or device error, 4 session error, 5 timeout, 6
|
||||
artifact or file error, 7 debugging tools disabled.
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
./gradlew assembleDebug --console=plain
|
||||
adb install -r app/build/outputs/apk/debug/app-debug.apk
|
||||
```
|
||||
|
||||
The provider can also be called directly for state, stop, and resume:
|
||||
|
||||
```sh
|
||||
adb shell content call --uri content://com.droiddeck.launcher.agent --method state
|
||||
adb shell content call --uri content://com.droiddeck.launcher.agent --method stop
|
||||
```
|
||||
|
||||
Start requests go through `tools/droiddeckctl`, which launches the protected debug Activity from the ADB shell. Calling the provider's `start` method directly returns an error because a background provider cannot reliably open the session screen.
|
||||
|
||||
The host CLI resolves one authorized device using `ADB_SERIAL` or `ANDROID_SERIAL` when set. Otherwise it deduplicates transports that report the same device serial and asks for an explicit serial if multiple devices remain.
|
||||
|
||||
```sh
|
||||
tools/droiddeckctl state --json
|
||||
tools/droiddeckctl start steam
|
||||
tools/droiddeckctl wait ready --timeout 90
|
||||
tools/droiddeckctl state # schema 2, below
|
||||
tools/droiddeckctl start steam --wait # --ui desktop, --url steam://..., --timeout
|
||||
tools/droiddeckctl start steam --wait --reuse # keep a Steam session that is already up
|
||||
tools/droiddeckctl start desktop
|
||||
tools/droiddeckctl run /usr/bin/foo -- arg1 arg2
|
||||
tools/droiddeckctl wait ready|idle|failed # --timeout (default 90 s)
|
||||
tools/droiddeckctl stop
|
||||
tools/droiddeckctl resume
|
||||
|
||||
tools/droiddeckctl launch 489830 --wait # Steam game by app id; --wait = until it has focus
|
||||
tools/droiddeckctl wait focused 489830 # gamescope focuses it AND Steam lists it focusable
|
||||
tools/droiddeckctl wait game # any app but the Steam client (769) has focus
|
||||
tools/droiddeckctl quit 489830 # Steam's TerminateApp, then SIGTERM/SIGKILL
|
||||
tools/droiddeckctl focus # focus properties and every Xwayland server's windows
|
||||
|
||||
tools/droiddeckctl input key ENTER # names (ESC, F1, UP, A...) or evdev codes; --mod CTRL
|
||||
tools/droiddeckctl input text 'hello world' # as US-layout keys; --commit for text-input
|
||||
tools/droiddeckctl input tap 640 360 # session output pixels; --space fraction for 0..1
|
||||
tools/droiddeckctl input swipe 100 600 100 100 --duration 400
|
||||
tools/droiddeckctl input pad a # a b x y lb rb start select guide qam up down left right
|
||||
tools/droiddeckctl input stick left 0 -1 --hold 500
|
||||
tools/droiddeckctl input trigger right 1.0
|
||||
|
||||
tools/droiddeckctl ui dump # tagged, labelled and clickable nodes; --all for every node
|
||||
tools/droiddeckctl ui click rail-steam # by test tag
|
||||
tools/droiddeckctl ui click --match "Play" # by text or description; --index for the nth match
|
||||
tools/droiddeckctl ui set-text some-field --text abc
|
||||
|
||||
tools/droiddeckctl guest -- xprop -root GAMESCOPE_FOCUSED_APP # --timeout --env K=V --cwd --stdin --check
|
||||
tools/droiddeckctl cdp targets
|
||||
tools/droiddeckctl cdp eval 'SteamClient.Apps.GetMyRunningApps?.()' # --target, --no-await
|
||||
tools/droiddeckctl env set DXVK_HUD=fps # next session only; --persistent until cleared
|
||||
tools/droiddeckctl env list
|
||||
tools/droiddeckctl env clear [next|persistent|all]
|
||||
tools/droiddeckctl override gamescope ./gamescope # first on the next session's PATH, removed after it
|
||||
tools/droiddeckctl prefs get session
|
||||
tools/droiddeckctl prefs set session some_key true --type bool # returns the previous value
|
||||
|
||||
tools/droiddeckctl logs latest ./session-artifacts
|
||||
tools/droiddeckctl screenshot ./screen.png
|
||||
tools/droiddeckctl screenshot ./screen.png --display 4
|
||||
tools/droiddeckctl record ./clip.mp4 --seconds 8 --display 0
|
||||
tools/droiddeckctl displays # logical ids, names, sizes, capture ids
|
||||
tools/droiddeckctl wake # wake the screen, dismiss the keyguard
|
||||
```
|
||||
|
||||
Every command writes JSON to stdout and reports its resolved ADB serial to stderr. Exit codes are 0 for success, 2 for invalid or rejected commands, 3 for ADB/device errors, 4 for session failures, 5 for timeouts, and 6 for artifact or file errors. Set `ADB` to select an adb executable (or pass `--adb`); pass `--serial` before the command to select a device directly.
|
||||
A sleeping device suspends the session and hides the app's windows from `ui dump`; `wake` first.
|
||||
|
||||
`start` accepts `steam` or `desktop`. Steam starts in Big Picture by default; `--ui desktop` selects the client's desktop UI, and `--url steam://...` passes a client URL. Use `--wait` to wait for `READY` as part of `start`. `run` accepts a program path and optional guest arguments.
|
||||
`input` needs a session that has drawn its first frame. Pad input goes through the session screen's
|
||||
virtual pad, the same one the on-screen controls drive, so games that ignore touch still respond.
|
||||
`launch`, `quit` and `cdp` need a Steam session; `focus` and `wait focused` need gamescope (every
|
||||
mode except the desktop).
|
||||
|
||||
The `state` response uses schema 1 and includes the build label, runtime version, session ID and phase, mode, requested program, suspend and first-frame state, output size, guest PID, failure details, and artifact paths. Phases are `IDLE`, `PREPARING`, `INSTALLING_RUNTIME`, `STARTING_COMPOSITOR`, `STARTING_GUEST`, `STARTING_STEAM`, `READY`, `SUSPENDED`, `STOPPING`, and `FAILED`. Each session also writes `events.jsonl` alongside its existing artifacts.
|
||||
## State (schema 2)
|
||||
|
||||
`tools/deploy_local.sh` uses the same device resolver. If multiple distinct devices are connected, select one with `ADB_SERIAL` or `ANDROID_SERIAL`.
|
||||
`state` returns `schema`, `build`, `appVersion`, `runtime {installed, version}`,
|
||||
`agent {guest, guestFocus, commands}` and `session`:
|
||||
|
||||
- `id`, `phase`, `running`, `mode`, `program`, `steamUi`, `steamUrl`, `suspended`, `pip`,
|
||||
`firstFrame`, `output` [w, h], `refreshHz`, `lastTransitionAt`, `guestPid`, `installing`
|
||||
- `logDir`, `eventsFile`, `artifactsAvailable`, `artifactsComplete`, `failure {code, message, status}`
|
||||
- `focus`: gamescope's focus as the guest agent last published it, or null:
|
||||
`focusedApp`, `focusedWindow`, `focusableApps`, `baselayerAppIds`, `seq`, `t`.
|
||||
|
||||
Phases: `IDLE`, `PREPARING`, `INSTALLING_RUNTIME`, `STARTING_COMPOSITOR`, `STARTING_GUEST`,
|
||||
`STARTING_STEAM`, `READY`, `SUSPENDED`, `STOPPING`, `FAILED`. Schema 2 only added `agent` and
|
||||
`session.focus`; every schema 1 field is unchanged.
|
||||
|
||||
A game the player can see is `focusedApp == appId` **and** `appId in focusableApps`.
|
||||
`GAMESCOPE_FOCUSABLE_WINDOWS` is what the Steam client orders the base layer from: a game missing
|
||||
there stays behind the client's loading screen however healthy its window looks
|
||||
(docs/development/steam-focus-debugging.md). `focus` returns the full snapshot, including each
|
||||
server's windows with map state, override-redirect, app id, `WM_STATE`, pid, class, opacity and
|
||||
`_NET_WM_STATE`.
|
||||
|
||||
## Events
|
||||
|
||||
Each session writes `events.jsonl` beside its other artifacts: the lifecycle (`session.created`,
|
||||
`compositor.started`, `guest.exited`, ...) plus `agent.ready` (the guest agent is up),
|
||||
`focus.changed` (the focused app, focusable apps or base-layer order changed) and
|
||||
`agent.launch_requested`.
|
||||
|
||||
## UI test tags
|
||||
|
||||
Compose test tags are exposed as resource ids (`ui/TestTags.kt`), so both `ui dump` and UI
|
||||
Automator show them:
|
||||
|
||||
| Tag | Where |
|
||||
|---|---|
|
||||
| `rail-steam`, `rail-games`, `rail-desktop`, `rail-store`, `rail-components`, `rail-android-apps`, `rail-setup`, `rail-updates` | the front end's rail |
|
||||
| `play-steam` | Play on the Steam and Games pages |
|
||||
| `setting-<key>` | every toggle and choice row, by its key (`setting-logs`, `setting-agent-commands`, ...) |
|
||||
| `tab-<n>` | a page's tab strip (Setup's Overview, Controller, Session, Launcher...), by position |
|
||||
| `drawer-tab-controller`, `-second_screen`, `-display`, `-effects`, `-games`, `-session` | the session drawer's pages |
|
||||
| `drawer-stop`, `drawer-stop-confirm`, `drawer-stop-cancel` | stopping a session from the drawer |
|
||||
|
||||
## Scenarios
|
||||
|
||||
`tools/droiddeck-scenario` runs a JSON scenario: `setup` once, `steps` `repeat` times, `teardown`
|
||||
once. Each run starts with `before` (typically `start steam --wait --reuse`, so a session that died
|
||||
in one run, to a crash or the low-memory killer, is started again for the next) and ends with
|
||||
`after`, whether it passed or not, so a failed run cannot leave a game running into the next. A list step is a droiddeckctl command line; `{"assert": "dotted.path", "equals"|"in"|
|
||||
"contains"|"not"|"exists": ...}` checks the last answer; `{"sleep"}`, `{"screenshot"}` and
|
||||
`{"record"}` do what they say. A failing run keeps a screenshot, the focus snapshot and the
|
||||
session's logs in its folder, and `report.json` sums up the runs.
|
||||
|
||||
```sh
|
||||
tools/droiddeck-scenario tools/scenarios/steam-game-focus.json --var appId=489830 --repeat 10 --keep-going
|
||||
```
|
||||
|
||||
`tools/scenarios/steam-game-focus.json` launches a game, checks it has focus the way a player
|
||||
sees it, and quits it: the test that catches a game left behind Steam's loading screen, which only
|
||||
shows up in some fraction of launches. A game with a launcher of its own needs the launcher pressed
|
||||
through first, since the launcher window carries the game's app id too:
|
||||
`tools/scenarios/skyrim-launcher-focus.json` taps Skyrim's Play before checking.
|
||||
Reference in new issue
Block a user