Merge org main 5aa245cf into the non-Adreno test branch

Conflicts resolved keeping both sides: main's comment cleanup (#764) taken
where only comments differed; the test branch's code kept everywhere else.
ensureAuto runs main's Quest auto drivers before the Mali paths;
vkp_update_map keeps the Quest layer path and the completion-unknown check
(pass_begin goes through it); gamescope backend selection plus main's
steam-mode cursor composite; droiddeck-session's watcher is main's shared
loop with the Venus GL probe after it; SessionFiles/runtime_inputs/build.yml
carry both the Xclipse layer and main's VR assets. Main's GPU detection
test follows the non-Adreno support text; the Steam presentation test
finds its block by code, not by a removed comment.
This commit is contained in:
The412Banner committed 2026-10-15 18:40:47 -04:00
commit 75612da44b
1016 files changed
+95104 -48259

No files matched your search

+13 -4
View File
@@ -66,10 +66,19 @@ value holds on both sides.
Automatic: the game folder's own images, then Steam (the appid the files name, else a
store search whose result has the same name), then SteamGridDB, then the exe's icon. The
art chooser (an Artwork row) shows the game folder's images, Steam's official piece and
SteamGridDB's best few (fetched when it opens); a tap applies one, "Pick image…" takes a
file through the same picker (png, jpg, jpeg, webp, ico), "Reset to automatic" drops the
choice. A choice is a copy in `files/added-art/<appid>/chosen-<slot>.<ext>`, its source in
the preferences.
SteamGridDB's best few (fetched when it opens); a tap applies one, "Search SteamGridDB…"
opens a search (it starts with the game's name and opens the match at once when one is
exact; "Show more" pages through the images), "Pick image…" takes a file through the same
picker (png, jpg, jpeg, webp, ico), "Reset to automatic" drops the choice. A choice is a
copy in `files/added-art/<appid>/chosen-<slot>.<ext>`, its source in the preferences.
SteamGridDB failures are told apart (`SteamGridDb.Status`: no key, key refused, rate limit,
offline, server, no match for the game) and shown under the SteamGridDB images instead of
leaving the row out. The SteamGridDB game id is remembered per game in
`added-art/<appid>/sgdb-id`; "none" (SteamGridDB has no such game) is kept for a week, but
a failed request is never written as "none", a "none" older than the saved key is asked
again, and the chooser always asks again when the user opens it. A background pass stops
asking SteamGridDB after its first failure.
Setup › Launcher › Games holds the Artwork switch (automatic fetching) and the
SteamGridDB API key: the user's own, sealed with `CredentialCipher`, wins over the build's
+175
View File
@@ -0,0 +1,175 @@
# Amazon Games module: clean-room rewrite
The Amazon Games integration (the Rust `amazon` module in `libdroiddeckstores.so` and the Kotlin
package `com.droiddeck.launcher.stores.amazon`) was rewritten so that it carries no code from the
earlier implementation. This page records how that was done and every judgement call the new code
makes.
## Process
1. **Spec, written separately.** A functional specification (six documents: summary and unknowns,
service protocol, manifest format, install behaviour, on-disk compatibility, and the app-facing
interface) was written from the old code by a separate pass. It describes behaviour and data
formats only, not code structure.
2. **Implementation from the spec only.** The implementer worked from those documents and from
code that is not Amazon-specific:
- read: the spec; the crate's `Cargo.toml`, `Cargo.lock`, `lib.rs`, `stores_jni.rs`,
`priority.rs`, `fetch_tuning.rs` and the shared `fetch_core.rs`; the clean-room Epic module
(Rust and Kotlin) as the structural template; `JNI.md`; the shared Kotlin store code (account
store, install records, download queue, login activity, catalog and state); the signatures and
call sites of `StoreSizes`' Amazon branch (to keep them compiling); build files and CI;
- not read: anything under `stores/amazon/` (Kotlin, Java, tests) or `src/amazon*` (Rust) of the
old tree, other branches or checkouts, or any Amazon client source. The old files were deleted
with `git rm` without being opened.
3. **Fresh code.** Every file has its own structure, names and comments, with a plain description
and `SPDX-License-Identifier: GPL-3.0-or-later` (the crate's licence). Where the spec left a fact
open, the choice was made from general knowledge of Amazon's public services or from
unauthenticated probes of the live endpoints (below), and is listed here.
## Layout
Rust, `app/src/main/rust/stores/src/amazon/`:
| File | Role |
|---|---|
| `net.rs` | blocking HTTP (reqwest + rustls), the three User-Agent identities, hex / percent / base64url helpers, URL scrubbing for logs |
| `auth.rs` | per-attempt device serial, client id, PKCE verifier and challenge; sign-in URL; code from the return URL; registration, refresh, deregistration |
| `distribution.rs` | the distribution service call shape (`X-Amz-Target`, `x-amzn-token`, `Content-Encoding: amz-1.0`), refusal detection, the download lookup |
| `library.rs` | entitlement pages, field fallbacks, de-duplication, add-on filter, sort |
| `cdn.rs` | path insertion before the signed query, file URLs, the manifest GET |
| `codec.rs` | manifest framing (big-endian header length) and xz / LZMA-alone decompression (`lzma-rs`) |
| `proto.rs` | a strict protobuf wire reader (and test writers) |
| `manifest.rs` | packages → files, path normalisation and safety, duplicate handling |
| `install/` | `mod.rs` (the run), `check.rs` (completeness, stray `.tmp`, free space), `writer.rs` (streamed download, SHA-256, atomic placement), `tests.rs` (loopback end-to-end) |
| `launch.rs` | `fuel.json`, exe scoring, the FuelPump environment |
| `service.rs`, `ffi.rs` | named operations and the JNI boundary |
Kotlin, `app/src/main/java/com/droiddeck/launcher/stores/amazon/`: `AmazonBackend` (the
`StoreBackend`, installs the login flow), `AmazonLogin` (the WebView flow), `AmazonCredentialStore`
(the account through `StoreAccounts`, single-flight refresh), `AmazonPrefs`, `AmazonLibrary` (sync,
cache, posters, publishing), `AmazonInstaller` (the queue job), `AmazonNative` (JNI), and two thin
wrappers kept for `StoreSizes`: `AmazonApiClient` (`getGameDownload`, `getBytes`, `appendPath`) and
`AmazonManifest` (`parse(bytes).totalInstallSize`).
The JNI surface (`nativeConfigure`, `nativeCall`, `nativeFetch`, `nativeInspect`, `nativeInstall`,
`nativeCancel`, `nativeRelease`) is documented in `app/src/main/rust/stores/JNI.md`. The old
`nativeStart` symbol is gone. The Java xz library stays in `app/build.gradle`: commons-compress
needs it for the `.tar.xz` Windows components, so only Amazon stopped using it.
## Compatibility kept exactly
- Credentials (`stores/amazon/credentials.json` through `StoreAccounts`): the same keys
(`access_token`, `refresh_token`, `device_serial`, `client_id`, `expires_at`, `display_name`). A
refresh merges `access_token`, `device_serial`, `client_id`, `expires_at` into the stored record
and keeps every other key; a new sign-in writes the whole record. Registration and refresh keep
the launcher's device constants, so existing refresh tokens keep working.
- Preferences `stores.amazon`: `library_cache` (same element keys), `library_synced_at`,
`vcover_<id>`, `size_<id>`, `version_<id>`; sign-out removes only the first two.
- Install folders, the sidecar keys (`extra.entitlementId`, `extra.sku`, no `cloud` key), the
launcher name `.droiddeck-launch.bat`, the `.tmp` naming and the two marker files.
- The catalog item's `extra` (`entitlementId`, `sku`, `version` = `""`) so the shared size cache and
the saved queue keep working.
## User-Agents and what the probes showed
Unauthenticated probes of the live services (2026-10-14):
- **Distribution service** (`gaming.amazon.com`): `GetEntitlements`, `GetGameDownload` and
`GetLiveVersionIds` answered identically for `AGSLauncher/1.0.0`, `DroidDeck/…`, a curl UA, with
and without the `UserAgent` header. A bad token gives HTTP 401 `<NotAuthorizedException/>`; no
token gives HTTP 400 with `com.amazon.coral.service#NotAuthorizedException`. The probes cannot
show whether the UA matters for an authorised call, so the launcher's `User-Agent` and
`UserAgent` values are kept (Amazon's own client identifiers, one constant each in `net.rs`).
- **Identity API** (`api.amazon.com`): register, token and deregister gave the same validation
errors whatever the UA (Firefox, `DroidDeck/…`, none). The desktop Firefox string is kept for
these calls, the same as the WebView: it is a generic browser label, not a client name, and
existing refresh tokens are known to work with it.
- **Sign-in page** (`www.amazon.com/ap/signin`): every UA got Amazon's bot check from the probe
host, so nothing could be learned; the WebView keeps the desktop Firefox string (a sign-in page
that misbehaves would lock every user out).
- **CDN**: no URL is reachable without a token. Manifest and file requests now send
`DroidDeck/<versionName>`, the only third-party label the old integration sent; this needs the
device test below.
Refusal detection follows the probes: 401, 403, and 400 whose body names `NotAuthorizedException`
mean "sign in again".
## Judgement calls
Recommended fixes from the spec, adopted:
1. **Complete = size and SHA-256.** One test for the space check and the download plan; a
same-size corrupt file is repaired. Files without a SHA-256 (another algorithm) are judged by
size. No cross-run "verified" cache: every present file is hashed on each run.
2. **A failed library page fails the sync** and keeps the old cache; a refused token is reported
with the shared "session expired: sign in again" line, also when a cache is shown.
3. **Single-flight refresh** (`AmazonCredentialStore`); a forced refresh after a failed library call
skips the refresh when another caller already replaced the token.
4. **Mark updating** (`beforeReplace` → `StoreInstalls.markUpdating`) once, before the first file
is put in place (new or replacing; the shared step only acts on an installed record).
5. **Manifest path safety**: absolute paths, any `:` (drive letters and stream names; Windows file
names cannot contain it), NUL, `.` / `..` and empty segments, and DroidDeck's own root files
(`.droiddeck-store.json` and its `.tmp`, `.droiddeck-launch.bat`, the two markers, compared
ignoring case) fail the install naming the path. A manifest that lists both `X` and `X.tmp` is
refused (the second would collide with the first's temporary file). The protobuf reader rejects
truncation, over-long varints, groups and unknown wire types, field number 0, wrong wire types
on the fields used, and a non-empty file without a hash or a SHA-256 that is not 32 bytes.
6. **Exe scoring on relative paths.** The Unreal-folder rule is tested against `"/" + relative
path` so a top-level `Binaries/Win64/…` still matches. The "first five letters" rule compares the
title with the file name **without** `.exe` (the extension's letters would otherwise count). An
empty title never matches the "path contains the title" rule. Ties: larger file, then the earlier
path. Only regular files are candidates; symlinks are not followed; depth is capped at 16.
7. **CDN User-Agent** is `DroidDeck/<versionName>` (see above).
8. **Empty files** whose hash is absent or the SHA-256 of empty input are created locally; stray
`.tmp` siblings of manifest files are removed at run start (other `.tmp` files are untouched).
9. **Markers** `.amazon_download_in_progress` / `.amazon_download_complete` are still written.
10. **Duplicate paths**: the later entry wins with a log warning; files sharing one hash are fetched
once per path (no copy optimisation).
Other choices:
- **Install size** is the sum over distinct paths (equal to the spec's "duplicates included" sum
for every manifest without duplicate paths, which none have been seen to have); it is the
progress total, the disk size and `size_<id>`.
- **Free space** is measured in Rust (`statvfs` on the folder or its nearest existing parent),
which is what the shared probe measures; unknown never blocks, a measured zero does.
- **Pre-download check** is reported in the queue's Manifest stage ("Checking files…" with byte
progress), like GOG and Epic: the Downloads UI orders stages Manifest → Download → Verify →
Install, so a Verify stage before Download would read backwards.
- **Concurrency**: one streamed GET per file through the shared fetch core in stream mode (the
speed tier's window clamped to 1–128, per-host cap `max(6, ceil(window / hosts))`, 120 s to the
response headers, the core's 30 s idle cap on the body, its five attempts with back-off and its
stall and 429 handling). A piece at offset 0 restarts the file (a retry); an out-of-order piece,
a body longer than the manifest size, a short body or a hash mismatch deletes the `.tmp` and is
retried; a write error, a refused `beforeReplace` or a failed rename fails the run.
- **Decompression** uses the `lzma-rs` crate (pure Rust; xz with any check type and LZMA-alone),
with a 256 MiB cap on the decompressed manifest.
- **Live version lookup** (`GetLiveVersionIds`) is not implemented; nothing used it.
- **DLC filter**: add-ons are hidden only when they also name a parent product (as the spec's sync
contract says), unless that would hide everything; the Rust call returns the filtered, sorted list
plus the unfiltered count.
- **Sign-in code**: the authorization code is read from the return URL's query (`%XX` decoded, `+`
kept literally; codes are URL-safe). The WebView flow passes the URL to Rust and never logs it.
- **Randomness** for the serial and verifier comes from `/dev/urandom`.
- **`fuel.json`** is found by exact name, else a case-insensitive match at the folder root; a UTF-8
BOM is accepted. `Main.Command` is accepted after stripping leading `/` and `./`, and must be a
safe relative path naming a regular file.
- **Logs**: the Rust side strips the query string and fragment of every URL in a line before it
reaches `onLog`; the shared redaction runs after that as before.
- **No engine, no store**: without the native library, sign-in, sync and install report
"Amazon Games: the store engine is not available"; there is no Java fallback.
## What needs device testing
- An existing signed-in user stays signed in after the update: the library shows from cache at once,
and a forced sync refreshes the token with the stored refresh token and lists the games.
- A fresh sign-in through the WebView (registration, display name), and sign-out (deregistration
fired, library cache cleared).
- A library with many pages; add-ons hidden; posters.
- Install, pause, resume, cancel (folder removed); repair of an installed game (nothing downloaded
when intact; a corrupted file fetched again; "Resume install" offered when a repair is
interrupted); an SD-card install; out-of-space refusal.
- Every endpoint with the chosen User-Agents, the CDN with `DroidDeck/…` in particular.
- Launch of an existing install through its existing Steam shortcut (same shortcut, same prefix)
and of a fresh install (exe choice, `fuel.json` arguments, the five FuelPump variables in
`.droiddeck-launch.bat`); uninstall.
+131
View File
@@ -0,0 +1,131 @@
# Cardboard VR and SteamVR on Turnip: what we learned
Written 2026-10-13, after the first stretch of Cardboard mode work on the AYN Thor (Snapdragon 8 Gen 2,
Adreno 740, Turnip on KGSL). `docs/cardboard-vr.md` describes how the feature works. This page records
what it took to get there: the traps, why each fix looks the way it does, how to debug a live session,
and what is still open. Read it before the next round of VR work.
## Where it stands
| What | Result |
| --- | --- |
| Cardboard mode (lens pass, 3DoF head tracking, recentre) | Works on any picture; picture follows SteamVR on and off |
| SteamVR 2.17.10 ARM64 (the Steam Frame's `deckard-steamvr`) | Starts from the drawer; driver-direct at 60 Hz, dashboard and Steam library in VR |
| Thumper (356400, D3D11, VR launch option) | Menus and levels at 60 Hz, 7 ms CPU / 13 ms GPU, 23 of 6485 frames dropped |
| The Thrill of the Fight (494150, Unity) | Gym loads, 60 Hz, 6.5 ms CPU / 8.5 ms GPU |
| The Lab (450390) | Loads, then Android's low-memory killer takes the whole app |
| SUPERHOT VR (617830) | Reaches its EULA in the dashboard (not accepted in testing) |
| SteamVR Home | Blocked: Proton's `vrclient_x64` lacks `IVROverlayView_002` |
| `helloxr -g Vulkan2` (OpenXR) | Session focused and submitting, picture shows as torn tiles |
Screenshots from the Thor: [Thumper gameplay](evidence/cardboard-thumper-vr-gameplay-2026-10-13.png),
[Thumper title](evidence/cardboard-thumper-vr-cardboard-2026-10-13.png),
[The Thrill of the Fight](evidence/cardboard-thrill-of-the-fight-cardboard-2026-10-13.png),
[the SteamVR dashboard with a game's EULA](evidence/cardboard-steamvr-dashboard-2026-10-13.png).
Games run as x86-64 Windows builds through Proton ARM64 (FEX, DXVK). Proton's `vrclient_x64` forwards
OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
## Turnip on KGSL: the four things SteamVR trips over
1. **`VK_KHR_display` makes the GPU disappear.** Turnip on KGSL enumerates no physical device when an
instance enables it ("I can't KHR_display"). `vrcompositor` always asks for it. The layer takes the
display extensions off the instance and fakes one display, mode and plane backed by a headless
surface. Headless surfaces report a `UINT32_MAX` extent, so the swapchain is sized from the mode.
2. **No exportable semaphores.** SteamVR shares a timeline semaphore with every shared image as an
opaque fd, and Turnip/KGSL cannot export one. The layer creates the semaphore unshared and hands out
a memfd (`droiddeck-shared-sem`) holding a 64-bit counter. A poller thread in each process carries
values both ways every 300 µs. Dropping the semaphores outright, the first attempt, gave torn
dashboard tiles.
3. **Wait-before-signal blocks the submitting thread.** Turnip does not defer a submit that waits on
a timeline value nobody has signalled yet; `vkQueueSubmit` sleeps until it is. SteamVR's client
waits on a compositor value that the compositor signals only after the client has submitted and
said so, so every client deadlocked before its first frame:
- `helloxr` blocked in a wait-idle in `vrclient.so`;
- DXVK's `dxvk-queue` thread blocked in Proton games.
The layer now gives a submit's wait on an imported shared value 3 ms, then drops it. It is kept
when the value is already reached, or when this process has queued the signal itself. Two details
mattered:
- the waits are judged before the submit's own signals count, because an acquire waits on one
value and signals the next on the same semaphore;
- SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there.
4. **UBWC layouts must agree.** An imported image has to be created with the same
`VkImageFormatListCreateInfo` (sRGB + UNORM) as its creator, or Turnip lays out the compression
differently and the reader sees garbage. The presenter's eye-texture import does this.
## Other traps, in the order they cost time
- **A still headset goes to standby.** A phone on a table looks unworn, SteamVR enters standby after
about 15 s, and the compositor stops submitting. The driver's proximity sensor reads "worn" while
Cardboard is on.
- **Emulated SysV semaphores never set `sem_otime`.** SteamVR's web helper waits for it to become
non-zero before using a semaphore set, so the dashboard never appeared. `sysv.c` now stamps
`sem_otime` and `sem_ctime`.
- **Steam's web helper needs the layer too.** It shares Steam's in-VR overlays with SteamVR and crashed
without the layer; after a few crashes Steam quits. Its GPU process outlives SteamVR runs, so
`steamwebhelper` gets the semaphore bridge whenever SteamVR is installed, not only while SteamVR's
`active` marker exists. It never gets the display stand-in.
- **Keep the layer out of ordinary sessions.** As an implicit layer it loads everywhere. Display
extensions are only taken off while SteamVR runs, a new session clears a leftover `active` marker,
and without SteamVR installed every process gets the driver's functions unchanged. Review feedback
on the first version pointed out that gamescope asking for a display extension would have been
rewritten in an ordinary session.
- **The `active` marker race.** Clearing a stale marker whenever `vrserver` is not running also
cleared it during startup, before `vrserver` appeared, because the app polls status every 3 s. Steam
quit as a result. Only a marker older than 2 minutes is cleared now.
- **Signal handlers in Proton games.** The layer's crash tracer replaced the `SIGSEGV` handler that FEX
and Wine rely on, and The Lab crashed in FEX's JIT. It now installs only where nothing handles the
signal.
- **mangoapp owns gamescope's single external overlay slot**, so an X11 presenter has to use
`STEAM_OVERLAY`. The default presenter is now a Wayland window on DroidDeck's own compositor, kept
on top by app id.
- **Memory.** Steam, SteamVR and one VR game fit in the Thor's 12 GB. A large game such as The Lab, or
two games at once, push the device into critical pressure, and the low-memory killer takes
DroidDeck itself (`ended-without-teardown.txt` in the session logs, `lowmemorykiller` in logcat).
- **Head-tracking axes.** With the display at `ROTATION_90`, screen x is minus device y and screen y is
device x. Getting this wrong made a turn at the hips tilt the view. Recentring now keeps the world
vertical and resets the heading only.
## Debugging a live session
- **Logs:** `~/.local/share/Steam/logs/{vrserver,vrcompositor,xrclient_*}.txt` and
`/run/droiddeck/droiddeck-steamvr/vrstartup.log`. Lines from the driver start with `droiddeck:`,
lines from the layer with `[droiddeck-vr-layer]`. A game's layer messages land in the session log.
- **Markers beside `active`** in `/run/droiddeck/droiddeck-steamvr/`:
- `crashtrace` prints backtraces on fatal signals;
- `semtrace` logs each bridged semaphore's local and shared value every 2 s, named by the memfd
inode so the same semaphore can be matched across processes, plus the first waits, signals and
shared image create infos.
Both are read when a process starts.
- **Frame stats:** in `/opt/steamvr/bin/linuxarm64`, `LD_LIBRARY_PATH=. ./vrcmd --stats` reports
submits, drops and app CPU/GPU times. `./vrcmd --hidedashboard` closes the menu SteamVR opens over a
new game.
- **Where a process is stuck:** a stack scan of `/proc/<tid>/mem` from the saved stack pointer (a small
Python helper) gives module+offset frames. That is how the wait-idle in `vrclient.so` and the
blocked `dxvk-queue` thread were found.
- **VR launch options:** `SteamClient.Apps.GetLaunchOptionsForApp(id)` lists them, and
`SteamClient.Apps.RunGame("<id>", "", <index>, 100)` starts one, through `droiddeckctl cdp eval`.
- **Quitting:** `droiddeckctl quit <appid>`. Without the id the game keeps running.
- **Pitfalls when testing:**
- never `pkill -f` or `pgrep -f` a pattern inside a guest command, because it matches the
command's own shell; use `pgrep -x`;
- replace a loaded `.so` by copying to a temporary name and `mv`, never `cp` over it;
- a session start restages the driver and layer from the APK assets, so a hand-copied build is
lost on the next start.
## Open items
- **`helloxr` picture.** Both sides create the swapchain images with identical parameters, so this is
not the UBWC import issue. Reading an image mid-render (the dropped wait) or binning tiles is the
next suspicion.
- **SteamVR Home** needs `IVROverlayView_002` in Proton's `vrclient`, or an ARM64 build of Home.
- **The Lab and other large games** need memory headroom. Candidates: stop Steam's own web helpers
while a VR game runs, or lower SteamVR's footprint.
- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync. A submit thread in the
layer, deferring submits until their waits are reachable, would be exact.
- **Layers:** only the first layer of each frame is presented. No reprojection.
- **Controllers** are a gaze pointer driven by the pad. Pad presses did not close the SteamVR menu in
testing. Untested: whether that came from gamescope focus being on Steam, or from the system button
mapping.
@@ -0,0 +1,29 @@
# Discord build channels
Use the same names as the app's Updates page:
| App | Discord | Builds | Existing webhook secret |
| --- | --- | --- | --- |
| Preview | [#preview-builds](https://discord.com/channels/1552707793247539220/1554202932965015583) | Signed builds of `main`, including merged fixes | `DISCORD_BUILDS_WEBHOOK` |
| Test builds | [#test-builds](https://discord.com/channels/1552707793247539220/1557605185365217340) | Signed builds of individual open pull requests | `DISCORD_PR_BUILDS_WEBHOOK` |
Stable contains tested releases published from version tags. Preview contains the newest
merged fixes before Stable; Test builds let users try an unmerged fix by its PR number.
## Renaming safely
Rename the existing Discord channels in place. Keep their IDs, permissions, and webhooks.
The webhook URLs used by `build.yml`, `pr-release-publish.yml`, and `pr-release-cleanup.yml`
route by ID, so changing a channel's name or topic needs no secret rotation or routing change.
Keep the existing GitHub secret names even though they use the old channel terminology.
PR publication saves `webhook_id` and `message_id` in the signed build report comment.
Republishing edits that message; closing or merging the PR retires it. Neither operation
looks up a Discord channel by name. Keep the existing webhook so those saved references
continue to work. Recreating a channel or webhook is a different operation and needs a
separate migration.
For future renames, check the channel and webhook IDs before and after, verify that the
existing webhook token can still read an earlier build message, and inspect the next
publish/update/retirement runs. Separately hosted bots may have their own configuration;
this repository only controls the GitHub Actions build announcements.
+144
View File
@@ -0,0 +1,144 @@
# Epic Games Store module: clean-room rewrite
The Epic integration (the Rust `epic` module in `libdroiddeckstores.so` and the Kotlin package
`com.droiddeck.launcher.stores.epic`) was rewritten so that it carries no code from the earlier
implementation, whose Rust module stated derivation from third-party Epic clients. This page
records how that was done and every judgement call the new code makes.
## Process
1. **Spec, written separately.** A functional specification (six documents: summary and unknowns,
service protocol, manifest and chunk formats, install behaviour, on-disk compatibility, and the
app-facing interface) was written from the old code by a separate pass. It describes behaviour
and data formats only, not code structure.
2. **Implementation from the spec only.** The implementer worked from those documents and from
code that is not Epic-specific:
- read: the spec; the crate's `Cargo.toml`, `lib.rs`, `stores_jni.rs`, `priority.rs`,
`fetch_tuning.rs`, the shared `fetch_core.rs` and the GOG module (for crate, threading and JNI
conventions); `JNI.md`; the Kotlin code outside `stores/epic/` that calls Epic (to keep its API);
build files and CI;
- not read: anything under `stores/epic/` (Kotlin, Java, tests) or `src/epic*` (Rust) of the old
tree, the body of the old `EpicCloud` class and of `CloudSavePaths.epic`, other branches or
checkouts, or any Epic client source. The old files were deleted with `git rm` without being
opened. (A search over shared files incidentally printed two lines of the old `EpicCloud`, its
base URL and User-Agent constant; both values are in the spec and only the URL is used.)
3. **Fresh code.** Every file of the new module has its own structure, names and comments, with a
plain description and `SPDX-License-Identifier: GPL-3.0-or-later` (the crate's licence). Where
the spec left a fact open, the choice was made from general knowledge of Epic's public services
and is listed below.
## Layout
| Rust (`app/src/main/rust/stores/src/epic/`) | |
|---|---|
| `net.rs` | blocking HTTP helper, the one User-Agent, percent/base64 encoders |
| `account.rs` | sign-in with an authorization code, refresh, exchange codes, redirect-page parsing |
| `library.rs`, `catalog.rs` | owned items (all pages), catalog details and attributes |
| `build_info.rs` | live build lookup, CDN list, manifest download |
| `manifest.rs`, `chunk.rs` | binary and JSON manifests, chunk files |
| `locale.rs` | device language → install tags, tag selection |
| `install/` | the install engine: `staging`, `delta`, `space`, `cache`, `schedule`, `writer`, `mod` (orchestration), end-to-end `tests` against a loopback server |
| `shop.rs`, `achievements.rs`, `savesync.rs`, `launch.rs`, `timefmt.rs` | shelves, achievements, cloud saves, launch arguments, ISO dates |
| `service.rs`, `ffi.rs` | the JSON operations and the JNI entry points |
| Kotlin (`stores/epic/`) | |
|---|---|
| `EpicBackend` | `StoreBackend` + `EpicLaunchSupport`; installs the login flow |
| `EpicLogin` | the `StoreLoginActivity.Flow` |
| `EpicCredentialStore`, `EpicPrefs` | account (shared Keystore store), `stores.epic` prefs |
| `EpicLibrary`, `EpicShelves` | library sync and cache, shop shelves |
| `EpicInstaller` | the download-queue job |
| `EpicApiClient`, `EpicDownloadManager`, `EpicInstallTags`, `EpicAchievements` | API kept for callers outside the package |
| `EpicSaveSync`, `EpicSaveTemplate` | cloud-save transfers for `EpicCloud`, template parsing for `CloudSavePaths.epic` |
| `EpicNative` | JNI wrapper (see `rust/stores/JNI.md`) |
## Judgement calls
Numbers in brackets refer to the spec's list of unknowns.
**Fixed where clearly correct**
1. OAuth form values (authorization code, refresh token) are percent-encoded [9].
2. A library page that fails ends the sync with an error and keeps the cached library instead of
caching a partial one; an HTTP 401 is reported as "session expired: sign in again" rather than
an empty library [7].
**Chosen where the spec left it open**
3. **User-Agent** [1]: every HTTP request (account, library, catalog, build info, CDNs, store
GraphQL, promotions, cloud saves) sends `DroidDeck/<versionName>`, defined once in
`epic/net.rs` (`USER_AGENT_PRODUCT`); changing that constant is the one place to adjust.
Achievements are the exception: Epic's achievements service rejects other agents, so
`epic/achievements.rs` sends Epic's launcher agent (`net.rs::EPIC_LAUNCHER_USER_AGENT`). The sign-in **WebView** keeps the launcher-style UA it used
before, because a sign-in page that rejects the UA would lock every user out; that string lives
in `epic/account.rs` (`WEBVIEW_USER_AGENT`).
4. **Shared fetch core** [14]: chunk downloads reuse the crate's store-agnostic `fetch_core` (also
used by GOG and Amazon; it came from Bannerlator's own crate and, unlike the old Epic module,
states no derivation from an Epic client). It already provides the adaptive window, CDN
rotation, retries with back-off, 429 and stall cooldowns the spec asks for. If its lineage is
ever in doubt, it is the one shared dependency to replace.
5. **Out-of-package Epic code** [14]: `EpicCloud` and `CloudSavePaths.epic` were rewritten to call
the new module. `StoreLaunch`'s exchange-code handling, `EpicOptions`, `WinCompSources.epic` and
`SessionPrefs.storesEpicMode` are shared code and were left as they are.
6. **Save-path template** parsing is Kotlin (`EpicSaveTemplate`), not Rust: the prefix walk it feeds
is shared Kotlin code and its existing unit test runs on the JVM without the native library. The
Epic walk matches folder names case- and punctuation-insensitively (`MyGame` finds `My-Game`), as
that test expects.
7. **One engine** [12, 13]: the hidden `storesEpicMode` switch is not read and there is no Java
fallback; without the native library the Epic store reports itself unavailable. When the full
space plan does not fit, the engine runs in a narrow mode (one file staged at a time, no chunk
cache), which needs only the net peak of the pending files.
8. **Schedule and cache**: the cache budget is enforced by the plan (eviction of the chunk needed
furthest in the future). At run time files are admitted up to 48 downloads ahead of the
furthest dispatched one, so the cache can briefly hold a few entries more than the plan's peak.
A chunk is first written into every admitted file that uses it; it touches the cache only when a
not-yet-admitted file still needs it.
9. **Space probing** uses `statvfs` (available blocks × fragment size, the same figure as Android's
`StatFs.getAvailableBytes`) from Rust; a failed probe is "unknown". A scratch directory on the same
filesystem as the game is treated as "beside the game".
10. **File hashing** is done in file order while parts land; parts that arrive early are read back
from the stage file. Files without a manifest hash are not hashed.
11. **Staging folder recovery**: an *empty* staging folder without its marker (a run stopped between
creating the folder and writing the marker) is adopted and the marker written; any other missing
or foreign marker fails the run without deleting.
12. **Unsafe manifest paths** (absolute, `..`, empty segments) fail the install.
13. **Encrypted manifests** (stored-as bit 1) are refused with an explicit message.
14. **Meta section prerequisite ids** are all read (no cap at 64), so the following fields stay aligned.
15. **Store GraphQL shelves** are requested with their arguments written into the query text
(country reduced to two letters first) instead of typed GraphQL variables, so the query does not
depend on the server's variable type names.
16. **Achievements**: when an account id is stored and the player-progress query gets no answer, the
result is "unreachable" rather than a list with everything locked.
17. **Token refresh** is single-flight (one refresh at a time; a waiting caller re-reads the stored
account first), so a rotated refresh token is never spent twice. A fresh sign-in replaces the
stored account instead of merging with it.
18. **Download progress** reported to the queue never moves backwards, although samples come from
several native threads.
19. **Network window** is the speed tier's value clamped to 1–128; the per-host cap is
`max(6, ceil(window / hosts))`.
**Kept at parity (known weak spots)**
- Chunk requests do not carry the CDN's signing query; only the manifest request does; the
`cloudflare.epicgamescdn.com` CDN stays excluded [2].
- The manifest file name comes from the first manifest location and is requested from every CDN [3].
- The manifest header SHA-1 and the chunk header GUID/hash are not checked (chunk size and chunk
SHA-1 are) [4].
- Per-file flags, symlink targets and custom fields are ignored [5]; JSON manifests carry no hashes
as read, so their files are rewritten on every run [6].
- The refresh token's own expiry is not tracked; a failed refresh still returns the stored token [8].
- The cloud `hash` is compared as if it were an MD5 [10].
- `AdditionalCommandLine` is one argument, frozen at install [11].
- Shop search, an unused "wants an exchange code" helper and per-game uninstall work are not
ported; DLC stay hidden and are never installed separately; optional (non-language) install tags
are never selected; resume restarts a partially written file from zero [15–18].
- Library details are fetched one game at a time, as before.
## What needs device testing
- Sign-in through the WebView, and that an **existing** saved account (sealed `credentials.json`)
stays signed in after the update.
- Every endpoint with the DroidDeck User-Agent: library, catalog, build info, CDN downloads, store
GraphQL and promotions, cloud saves (achievements use the launcher agent, see above).
- Library sync and shelves; install, pause/resume, cancel, update and repair; a card install with
the internal scratch cache; launch with an exchange code; cloud-save up/down.
Binary file not shown.

After

Width:  |  Height:  |  Size: 251 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 256 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 644 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 614 KiB

@@ -228,3 +228,11 @@ flag for an experiment.
Device check (Thor): Alan Wake's American Nightmare with its 1280x720 Game Resolution runs on `:1`
at 1280x720 with no `--force-windows-fullscreen` in the session, 2 frame-size changes, no crash
in logcat. The Games tab and the drawer render without the switch.
## Steam picture position
The launcher's Execute Steam settings, under Display, offer Picture position: Top, Bottom, Center (default). The selection applies to the entire Steam session, including menus and games, and is loaded again on resume. No focus-based switching or in-game toggle is used. PiP and VR retain their own presentation.
Positioning keeps the selected resolution, fit scale, aspect ratio and full surface intact. It distributes outer letterboxing above or below the picture; it does not crop game-authored black bars inside the surface. A surface that already fills the panel has no vertical space to move. Use a 16:9 resolution for fixed-16:9 games when a separate controls area is desired.
The compositor publishes actual bounds for touch mapping and ignores touches starting outside the picture. The mapping harness covers top, bottom and center across tall, 16:9 and wide surfaces, including inverse input mapping and legacy half-screen mode compatibility.
+126 -16
View File
@@ -24,26 +24,112 @@ scanned (`StoreInstallRoot.roots`): internal, every card, and the Steam library'
installs an earlier build put there. `<Store>` is `GOG`, `Epic` or `Amazon`; `<title>` is the title
with unsafe characters dropped, at most 60 characters (`StoreInstallRoot.folderName`), chosen once;
a rerun (repair, update) lands on the folder whose sidecar carries the game's id, wherever it is.
An Epic install on a card keeps its in-flight chunks in `cacheDir/stores/epic/<id>/`: the
`chunkCacheDir` of both native calls (`EpicNative.run` fetches into it, `EpicNative.assemble`
writes the files from it and drops each chunk after its last use, then the folder), and of the
manager's own loops when the engine is not there; `""` keeps the cache beside the game, as an
internal install has it. The scratch folder is removed on cancel-with-delete and on uninstall.
The cache holds whole ~1 MiB chunk windows, shared with files this device does not install, so it
is often larger than the game (Metalstorm: 9.1 GB of chunks for 4.4 GB of files); the free-space
check, made after the delta pass, counts the missing chunks and the missing files on their own
volumes. A successful run removes the cache.
An Epic install on a card keeps its in-flight chunks in `cacheDir/stores/epic/<id>/`;
`""` keeps the cache beside the game, as an internal install has it. The scratch folder is
removed on cancel-with-delete and on uninstall.
Preflight resolves an existing game's folder before selecting its target root, so a resume or
repair on a card checks and remembers that volume rather than internal storage. Fresh GOG and
Amazon installs can reject insufficient catalog-size space immediately; existing installs defer
to manifest checks of the files still needing work (GOG MD5, Amazon SHA-256 where supplied).
Free-space display probes the nearest existing ancestor without creating the target folder; if
the probe cannot determine space, it reports **Free space unknown** rather than blocking an install.
`StoreInstallRoot` uses `-1` for unknown and preserves a measured `StatFs` zero as **0 B free**.
Fresh GOG/Amazon preflight and the backends' own space checks (GOG in Kotlin, Epic and Amazon in
the Rust engine) reject known zero when bytes remain. The read-only probe walks existing ancestors without creating directories; a failed
`StatFs` probe can use a positive `File.usableSpace` fallback, but that API's ambiguous zero is
unknown, not a measurement of a full disk.
Preflight failures remain failed Downloads rows with Clear and Retry. Retry reruns preflight
outside the worker lifecycle and then enqueues the real backend; it never occupies a queue slot
with a placeholder job. Failures are kept in the Downloads UI, notification and logs rather than
duplicated as transient failure toasts. Active queue persistence and ordinary backend pause/retry
are unchanged.
Epic installs run in the Rust `epic` module of `libdroiddeckstores.so` (`epic/install/`), driven by
`EpicInstaller` through `EpicNative.install`. There is one engine; the old hidden `storesEpicMode`
switch is no longer read. Without the native library the Epic store cannot install (the facade
reports "the store engine is not available"). The design, and how it was written from a
functional spec only, is in [epic-cleanroom.md](epic-cleanroom.md).
A run, in order: CDN list from the build information (`EpicApiClient.getManifestApiJson`) →
manifest from the first CDN that serves it → parse (binary or JSON) → files selected by the device
language's install tags (`EpicInstallTags`) → staging recovery → the delta pass (a selected file is
up to date when it has the manifest size and SHA-1; a file without a manifest hash is always
rewritten) → space plan → chunk cache preparation → fetch schedule → download and write → the
installed record (`StoreInstalls.complete`). Resume, update and repair are the same rerun.
- **Schedule** (`epic/install/schedule.rs`). The pending files' parts form one timeline. A chunk is
downloaded at its first use and feeds every later use in the same file directly. If it is used
again in a later file it is kept, decompressed and verified, in the chunk cache, which has a byte
budget; when keeping a chunk would exceed it, the cached chunk needed furthest in the future is
dropped and downloaded again when needed. Chunks a previous run left in the cache are reused,
soonest-needed first, as far as the budget allows.
- **Writing** (`epic/install/writer.rs`). Files are admitted strictly in order, a little ahead of
the download, and a download is dispatched only once the file of its first use is admitted, so
the number of half-written files stays small. Each admitted file is written into a unique
`epic-<pid>-<seq>-<nanos>.stage` file in `<game>/.droiddeck-epic-staging/`; the file SHA-1 is
computed in file order as parts land. A complete file is synced, size- and hash-checked, and
renamed over its destination. Right before the first rename of the run `StoreInstalls.markUpdating`
switches an installed record to `updating`; a refusal fails the run with nothing replaced.
- **Fetching** uses the shared `fetch_core` (adaptive window, CDN rotation, retries with back-off,
stall and 429 cooldowns) with the speed tier's window and decode workers. Every chunk is checked
(header, zlib stream end, exact size, SHA-1) before any byte reaches a file or the cache.
- **Space** (`epic/install/space.rs`). Free space is probed per volume (`statvfs`; unknown is never
"no space"). The plan: net peak of the pending files in order (each old copy credited at its own
swap), a cache budget of half the remaining space (at most 1 GiB, 256 MiB when unknown, 0 is
allowed), an overlap budget that limits how many bytes of files replacing existing data are
staged at once on an update (at least the largest replaced file), and a 64 MiB transient
allowance. A card install keeps its cache in `cacheDir/stores/epic/<id>/` when internal storage
can hold a useful cache and the game has no non-empty `.chunks` yet; otherwise the cache is
`<game>/.chunks`. When the full plan does not fit, the run falls back to one file at a time with
no cache (needs only the net peak); when even that does not fit the install is refused before any
chunk request, naming the storage, the need, the free space and the shortfall.
- **Progress**: the bar counts pending-file bytes written (not compressed chunk bytes), at most every
50 ms plus once per finished file. The run logs `epic: plan …` and `epic: summary mode=adaptive|narrow
files=… bytes_written=… avg_mbps=… fetches=… refetches=… peak_cache_mib=… hosts=…`.
- **Cancel / pause / failure**: the cancel flag is forwarded within 250 ms; the run stops its
requests, joins its threads, removes every unfinished stage file and `.part` cache entry, and only
then returns. Swapped files stay; unswapped destinations keep their old content; the chunk cache
stays for the next run. Success removes the cache and the staging folder (when only its marker is
left).
The staging folder is recognised by its `owner` marker (`DroidDeck Epic staging v1` + LF). Before
the delta pass the run checks that the folder is a real directory at the expected canonical path,
that the marker matches and that every other entry is a plain `.stage` file directly inside it; only
then are those orphans deleted. Anything unexpected fails the run without deleting. A manifest path
naming the staging folder, an absolute path or a `..` segment is refused. `.stage` / `.part` files
elsewhere in the game folder are never touched.
Amazon installs run in the Rust `amazon` module (`amazon/install/`), driven by `AmazonInstaller`
through `AmazonNative.install`; sign-in, the library sync and the size lookup call the same module
through `AmazonNative.call`. The design, and how it was written from a functional spec only, is in
[amazon-cleanroom.md](amazon-cleanroom.md).
An Amazon run, in order: download lookup for the entitlement (`GetGameDownload`: a CDN base URL with
a signed query, and the version id) → `manifest.proto` from that base (a 4-byte big-endian header
length, an opaque header, then an xz or LZMA-alone compressed protobuf) → parse (every file of every
package; paths refused when absolute, with a drive letter, `..`, empty segments, or naming
DroidDeck's own root files) → leftover `<file>.tmp` siblings removed → the check of files on disk (a
file is complete when it has the manifest size and SHA-256; the same test drives the space check and
the download plan) → space check (`Not enough free space: need …, only … free`; unknown free space
never blocks) → empty files created locally → every other pending file fetched from
`files/<sha256 hex>` under the base, one streamed GET each through the shared fetch core (the speed
tier's window; at least half the CPU cores verifying and writing), hashed as it arrives, written to
`<file>.tmp` and renamed over the destination after the size and hash checks → the installed record.
Before the first file is put in place the record is switched to `updating`, so an interrupted repair
is offered as "Resume install". There is no partial-file resume: a file is complete or fetched again
from its first byte. Files the manifest does not list are left alone.
## The sidecar
```json
{
"version": 1,
"store": "epic", // gog | epic | amazon
"id": "Samorost3", // the store's own id: GOG product id, Epic app name, Amazon product id
"store": "epic",
"id": "Samorost3",
"title": "Samorost 3",
"exe": "Samorost3.exe", // relative to the folder, forward slashes; what the icon is read from
"launcher": ".droiddeck-launch.bat",// optional; what the shortcut runs instead of exe
"exe": "Samorost3.exe",
"launcher": ".droiddeck-launch.bat",
"args": ["-EpicPortal", "-epicusername=\"Name\"", "..."],
"env": {"FUEL_DIR": "C:\\ProgramData\\Amazon Games Services\\Legacy"},
"installVersion": "1.4.0",
@@ -59,7 +145,18 @@ store root whose sidecar says `installing`, or that has no sidecar at all, is an
install: `AddedGames.scan` leaves it out (it is neither a game nor a Custom folder), and the Stores
card and page offer **Resume install**, which reuses that folder - and an Epic `.chunks` beside it -
instead of starting a fresh one (`StoreInstallRoot.existingFolder`). A finished install being
repaired keeps its finished sidecar. The launcher and the finished sidecar are written before the
repaired keeps its finished sidecar until Epic reaches the first verified file swap. Immediately
before that swap Epic atomically changes the record to `"state": "updating"`, preserves launch
choices and identifiers, and clears `installVersion`. Network/hash failures or cancellation before
that point leave the old installed record and files intact. A later failure can leave a mixture of
verified new files and untouched old files: replacement is atomic **per file**, not a transactional
whole-game update or rollback. The updating folder is excluded from installed scans and offered
for resume. Cancellation with deletion clears caches but retains that folder and its verified
swaps; retry preserves the updating record's metadata. If marking or swapping fails, no destructive
fallback is attempted (the conservative updating state can remain even when a swap fails).
Completion writes the installed record with the new version only after all selected files succeed.
An already registered live Steam shortcut is not revoked at each file swap; do not launch an
in-progress update. The launcher and the finished sidecar are written before the
art and the Steam registration; a failure writing them fails the download with its message.
`StoreGameSidecar.parse` refuses anything whose `exe` or `launcher` would point outside the folder.
@@ -106,6 +203,15 @@ An Epic game's launch choices are the Epic card in the Games tab's launch settin
read the same thing - and "Resolve Epic sign-in", which opens Epic's account page behind its
sign-in in Android's browser.
An Amazon game's exe and arguments come from its `fuel.json` (`Main.Command`, used when it names
an existing file; `Main.Args`, used even when it does not), else from a score over every `.exe`
below the folder (Unreal shipping binaries and `Binaries/Win64` first, crash handlers, setups,
launchers and tiny stubs last; the rules see paths relative to the install folder), else the shared
picker. Its launcher sets five FuelPump variables frozen at install time: `FUEL_DIR`,
`AMAZON_GAMES_SDK_PATH`, `AMAZON_GAMES_FUEL_ENTITLEMENT_ID`, `AMAZON_GAMES_FUEL_PRODUCT_SKU` and
`AMAZON_GAMES_FUEL_DISPLAY_NAME` (the account name, `Player` without one). No Amazon Games SDK files
are deployed into the prefix and `fuel.json` `PostInstall` steps are not run.
A game's web links reach Android: `droiddeck-store-launch` sets the prefix's
`HKCU\Software\Wine\WineBrowser` `Browsers` to `/usr/local/bin/droiddeck-open-url`, which is also
`xdg-open` first on the game's PATH, and hands http(s) addresses to the app, which opens them in
@@ -146,7 +252,9 @@ the prefix's user folder appears, within a minute. Every path logs
has no cloud saves.
- **The services:** GOG `cloudstorage.gog.com/v1/<user>/<client>` with a token issued to the game's
own client (the client secret is kept at install; the Galaxy token is the fallback); Epic's
savesync data storage with signed read and write links.
savesync data storage with signed read and write links (`EpicCloud` → `EpicSaveSync` → the Rust
`epic/savesync.rs`; the template's root token and `{EpicId}` / `{AppName}` are read by
`EpicSaveTemplate`).
- **Which launches:** `droiddeck-store-launch` recognises a store game by the launcher .bat Steam runs
or, for a game without one (most GOG games), by its exe - the folders above it are searched for
the sidecar.
@@ -172,7 +280,9 @@ keeps the files; every store's install skips complete, verified files on the rer
rerun. Cancel deletes the folder. `StoreDownloadService` holds a foreground notification while
anything runs. The native engine (`libdroiddeckstores.so`, `StoresNative`, `GogNative`, `EpicNative`,
`AmazonNative`; contract in `app/src/main/rust/stores/JNI.md`) is loaded on first use; without it
each manager runs its Java fetch loop.
the GOG manager runs its Java fetch loop, and the Epic and Amazon stores (all of whose service calls
and installs run in the Rust `epic` and `amazon` modules) are unavailable: sign-in, sync and install
report "the store engine is not available".
## Settings