mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-19 21:06:09 +00:00
feat: rewrite the Epic and Amazon store modules (#802)
Both store modules are rewritten from a protocol-level spec: new Rust engines behind a small JNI surface (nativeCall/nativeInstall) and thin Kotlin facades, with the launcher's own user agent. Epic: sign-in carries over, library, manifest download with byte-exact verification, assembly, uninstall, launch arguments, achievements. Amazon: sign-in carries over, library, manifest download with per-file hash checks, path checks, executable detection from fuel.json, uninstall. On-disk install layout is unchanged, so existing installs keep working. Store game pages now take controller focus on their lead hero action (Install, Uninstall, Pause, Resume, Get, Buy), fall back to Back, and keep D-pad focus inside the page while it covers the library.
This commit is contained in:
1 parent
a849556f3f
commit
7ad3c60efe
105 files changed
+10975
-13233
No files matched your search
@@ -0,0 +1,175 @@
|
||||
# Amazon Games module: clean-room rewrite
|
||||
|
||||
The Amazon Games integration (the Rust `amazon` module in `libdroiddeckstores.so` and the Kotlin
|
||||
package `com.droiddeck.launcher.stores.amazon`) was rewritten so that it carries no code from the
|
||||
earlier implementation. This page records how that was done and every judgement call the new code
|
||||
makes.
|
||||
|
||||
## Process
|
||||
|
||||
1. **Spec, written separately.** A functional specification (six documents: summary and unknowns,
|
||||
service protocol, manifest format, install behaviour, on-disk compatibility, and the app-facing
|
||||
interface) was written from the old code by a separate pass. It describes behaviour and data
|
||||
formats only, not code structure.
|
||||
2. **Implementation from the spec only.** The implementer worked from those documents and from
|
||||
code that is not Amazon-specific:
|
||||
- read: the spec; the crate's `Cargo.toml`, `Cargo.lock`, `lib.rs`, `stores_jni.rs`,
|
||||
`priority.rs`, `fetch_tuning.rs` and the shared `fetch_core.rs`; the clean-room Epic module
|
||||
(Rust and Kotlin) as the structural template; `JNI.md`; the shared Kotlin store code (account
|
||||
store, install records, download queue, login activity, catalog and state); the signatures and
|
||||
call sites of `StoreSizes`' Amazon branch (to keep them compiling); build files and CI;
|
||||
- not read: anything under `stores/amazon/` (Kotlin, Java, tests) or `src/amazon*` (Rust) of the
|
||||
old tree, other branches or checkouts, or any Amazon client source. The old files were deleted
|
||||
with `git rm` without being opened.
|
||||
3. **Fresh code.** Every file has its own structure, names and comments, with a plain description
|
||||
and `SPDX-License-Identifier: GPL-3.0-or-later` (the crate's licence). Where the spec left a fact
|
||||
open, the choice was made from general knowledge of Amazon's public services or from
|
||||
unauthenticated probes of the live endpoints (below), and is listed here.
|
||||
|
||||
## Layout
|
||||
|
||||
Rust, `app/src/main/rust/stores/src/amazon/`:
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `net.rs` | blocking HTTP (reqwest + rustls), the three User-Agent identities, hex / percent / base64url helpers, URL scrubbing for logs |
|
||||
| `auth.rs` | per-attempt device serial, client id, PKCE verifier and challenge; sign-in URL; code from the return URL; registration, refresh, deregistration |
|
||||
| `distribution.rs` | the distribution service call shape (`X-Amz-Target`, `x-amzn-token`, `Content-Encoding: amz-1.0`), refusal detection, the download lookup |
|
||||
| `library.rs` | entitlement pages, field fallbacks, de-duplication, add-on filter, sort |
|
||||
| `cdn.rs` | path insertion before the signed query, file URLs, the manifest GET |
|
||||
| `codec.rs` | manifest framing (big-endian header length) and xz / LZMA-alone decompression (`lzma-rs`) |
|
||||
| `proto.rs` | a strict protobuf wire reader (and test writers) |
|
||||
| `manifest.rs` | packages → files, path normalisation and safety, duplicate handling |
|
||||
| `install/` | `mod.rs` (the run), `check.rs` (completeness, stray `.tmp`, free space), `writer.rs` (streamed download, SHA-256, atomic placement), `tests.rs` (loopback end-to-end) |
|
||||
| `launch.rs` | `fuel.json`, exe scoring, the FuelPump environment |
|
||||
| `service.rs`, `ffi.rs` | named operations and the JNI boundary |
|
||||
|
||||
Kotlin, `app/src/main/java/com/droiddeck/launcher/stores/amazon/`: `AmazonBackend` (the
|
||||
`StoreBackend`, installs the login flow), `AmazonLogin` (the WebView flow), `AmazonCredentialStore`
|
||||
(the account through `StoreAccounts`, single-flight refresh), `AmazonPrefs`, `AmazonLibrary` (sync,
|
||||
cache, posters, publishing), `AmazonInstaller` (the queue job), `AmazonNative` (JNI), and two thin
|
||||
wrappers kept for `StoreSizes`: `AmazonApiClient` (`getGameDownload`, `getBytes`, `appendPath`) and
|
||||
`AmazonManifest` (`parse(bytes).totalInstallSize`).
|
||||
|
||||
The JNI surface (`nativeConfigure`, `nativeCall`, `nativeFetch`, `nativeInspect`, `nativeInstall`,
|
||||
`nativeCancel`, `nativeRelease`) is documented in `app/src/main/rust/stores/JNI.md`. The old
|
||||
`nativeStart` symbol is gone. The Java xz library stays in `app/build.gradle`: commons-compress
|
||||
needs it for the `.tar.xz` Windows components, so only Amazon stopped using it.
|
||||
|
||||
## Compatibility kept exactly
|
||||
|
||||
- Credentials (`stores/amazon/credentials.json` through `StoreAccounts`): the same keys
|
||||
(`access_token`, `refresh_token`, `device_serial`, `client_id`, `expires_at`, `display_name`). A
|
||||
refresh merges `access_token`, `device_serial`, `client_id`, `expires_at` into the stored record
|
||||
and keeps every other key; a new sign-in writes the whole record. Registration and refresh keep
|
||||
the launcher's device constants, so existing refresh tokens keep working.
|
||||
- Preferences `stores.amazon`: `library_cache` (same element keys), `library_synced_at`,
|
||||
`vcover_<id>`, `size_<id>`, `version_<id>`; sign-out removes only the first two.
|
||||
- Install folders, the sidecar keys (`extra.entitlementId`, `extra.sku`, no `cloud` key), the
|
||||
launcher name `.droiddeck-launch.bat`, the `.tmp` naming and the two marker files.
|
||||
- The catalog item's `extra` (`entitlementId`, `sku`, `version` = `""`) so the shared size cache and
|
||||
the saved queue keep working.
|
||||
|
||||
## User-Agents and what the probes showed
|
||||
|
||||
Unauthenticated probes of the live services (2026-10-14):
|
||||
|
||||
- **Distribution service** (`gaming.amazon.com`): `GetEntitlements`, `GetGameDownload` and
|
||||
`GetLiveVersionIds` answered identically for `AGSLauncher/1.0.0`, `DroidDeck/…`, a curl UA, with
|
||||
and without the `UserAgent` header. A bad token gives HTTP 401 `<NotAuthorizedException/>`; no
|
||||
token gives HTTP 400 with `com.amazon.coral.service#NotAuthorizedException`. The probes cannot
|
||||
show whether the UA matters for an authorised call, so the launcher's `User-Agent` and
|
||||
`UserAgent` values are kept (Amazon's own client identifiers, one constant each in `net.rs`).
|
||||
- **Identity API** (`api.amazon.com`): register, token and deregister gave the same validation
|
||||
errors whatever the UA (Firefox, `DroidDeck/…`, none). The desktop Firefox string is kept for
|
||||
these calls, the same as the WebView: it is a generic browser label, not a client name, and
|
||||
existing refresh tokens are known to work with it.
|
||||
- **Sign-in page** (`www.amazon.com/ap/signin`): every UA got Amazon's bot check from the probe
|
||||
host, so nothing could be learned; the WebView keeps the desktop Firefox string (a sign-in page
|
||||
that misbehaves would lock every user out).
|
||||
- **CDN**: no URL is reachable without a token. Manifest and file requests now send
|
||||
`DroidDeck/<versionName>`, the only third-party label the old integration sent; this needs the
|
||||
device test below.
|
||||
|
||||
Refusal detection follows the probes: 401, 403, and 400 whose body names `NotAuthorizedException`
|
||||
mean "sign in again".
|
||||
|
||||
## Judgement calls
|
||||
|
||||
Recommended fixes from the spec, adopted:
|
||||
|
||||
1. **Complete = size and SHA-256.** One test for the space check and the download plan; a
|
||||
same-size corrupt file is repaired. Files without a SHA-256 (another algorithm) are judged by
|
||||
size. No cross-run "verified" cache: every present file is hashed on each run.
|
||||
2. **A failed library page fails the sync** and keeps the old cache; a refused token is reported
|
||||
with the shared "session expired: sign in again" line, also when a cache is shown.
|
||||
3. **Single-flight refresh** (`AmazonCredentialStore`); a forced refresh after a failed library call
|
||||
skips the refresh when another caller already replaced the token.
|
||||
4. **Mark updating** (`beforeReplace` → `StoreInstalls.markUpdating`) once, before the first file
|
||||
is put in place (new or replacing; the shared step only acts on an installed record).
|
||||
5. **Manifest path safety**: absolute paths, any `:` (drive letters and stream names; Windows file
|
||||
names cannot contain it), NUL, `.` / `..` and empty segments, and DroidDeck's own root files
|
||||
(`.droiddeck-store.json` and its `.tmp`, `.droiddeck-launch.bat`, the two markers, compared
|
||||
ignoring case) fail the install naming the path. A manifest that lists both `X` and `X.tmp` is
|
||||
refused (the second would collide with the first's temporary file). The protobuf reader rejects
|
||||
truncation, over-long varints, groups and unknown wire types, field number 0, wrong wire types
|
||||
on the fields used, and a non-empty file without a hash or a SHA-256 that is not 32 bytes.
|
||||
6. **Exe scoring on relative paths.** The Unreal-folder rule is tested against `"/" + relative
|
||||
path` so a top-level `Binaries/Win64/…` still matches. The "first five letters" rule compares the
|
||||
title with the file name **without** `.exe` (the extension's letters would otherwise count). An
|
||||
empty title never matches the "path contains the title" rule. Ties: larger file, then the earlier
|
||||
path. Only regular files are candidates; symlinks are not followed; depth is capped at 16.
|
||||
7. **CDN User-Agent** is `DroidDeck/<versionName>` (see above).
|
||||
8. **Empty files** whose hash is absent or the SHA-256 of empty input are created locally; stray
|
||||
`.tmp` siblings of manifest files are removed at run start (other `.tmp` files are untouched).
|
||||
9. **Markers** `.amazon_download_in_progress` / `.amazon_download_complete` are still written.
|
||||
10. **Duplicate paths**: the later entry wins with a log warning; files sharing one hash are fetched
|
||||
once per path (no copy optimisation).
|
||||
|
||||
Other choices:
|
||||
|
||||
- **Install size** is the sum over distinct paths (equal to the spec's "duplicates included" sum
|
||||
for every manifest without duplicate paths, which none have been seen to have); it is the
|
||||
progress total, the disk size and `size_<id>`.
|
||||
- **Free space** is measured in Rust (`statvfs` on the folder or its nearest existing parent),
|
||||
which is what the shared probe measures; unknown never blocks, a measured zero does.
|
||||
- **Pre-download check** is reported in the queue's Manifest stage ("Checking files…" with byte
|
||||
progress), like GOG and Epic: the Downloads UI orders stages Manifest → Download → Verify →
|
||||
Install, so a Verify stage before Download would read backwards.
|
||||
- **Concurrency**: one streamed GET per file through the shared fetch core in stream mode (the
|
||||
speed tier's window clamped to 1–128, per-host cap `max(6, ceil(window / hosts))`, 120 s to the
|
||||
response headers, the core's 30 s idle cap on the body, its five attempts with back-off and its
|
||||
stall and 429 handling). A piece at offset 0 restarts the file (a retry); an out-of-order piece,
|
||||
a body longer than the manifest size, a short body or a hash mismatch deletes the `.tmp` and is
|
||||
retried; a write error, a refused `beforeReplace` or a failed rename fails the run.
|
||||
- **Decompression** uses the `lzma-rs` crate (pure Rust; xz with any check type and LZMA-alone),
|
||||
with a 256 MiB cap on the decompressed manifest.
|
||||
- **Live version lookup** (`GetLiveVersionIds`) is not implemented; nothing used it.
|
||||
- **DLC filter**: add-ons are hidden only when they also name a parent product (as the spec's sync
|
||||
contract says), unless that would hide everything; the Rust call returns the filtered, sorted list
|
||||
plus the unfiltered count.
|
||||
- **Sign-in code**: the authorization code is read from the return URL's query (`%XX` decoded, `+`
|
||||
kept literally; codes are URL-safe). The WebView flow passes the URL to Rust and never logs it.
|
||||
- **Randomness** for the serial and verifier comes from `/dev/urandom`.
|
||||
- **`fuel.json`** is found by exact name, else a case-insensitive match at the folder root; a UTF-8
|
||||
BOM is accepted. `Main.Command` is accepted after stripping leading `/` and `./`, and must be a
|
||||
safe relative path naming a regular file.
|
||||
- **Logs**: the Rust side strips the query string and fragment of every URL in a line before it
|
||||
reaches `onLog`; the shared redaction runs after that as before.
|
||||
- **No engine, no store**: without the native library, sign-in, sync and install report
|
||||
"Amazon Games: the store engine is not available"; there is no Java fallback.
|
||||
|
||||
## What needs device testing
|
||||
|
||||
- An existing signed-in user stays signed in after the update: the library shows from cache at once,
|
||||
and a forced sync refreshes the token with the stored refresh token and lists the games.
|
||||
- A fresh sign-in through the WebView (registration, display name), and sign-out (deregistration
|
||||
fired, library cache cleared).
|
||||
- A library with many pages; add-ons hidden; posters.
|
||||
- Install, pause, resume, cancel (folder removed); repair of an installed game (nothing downloaded
|
||||
when intact; a corrupted file fetched again; "Resume install" offered when a repair is
|
||||
interrupted); an SD-card install; out-of-space refusal.
|
||||
- Every endpoint with the chosen User-Agents, the CDN with `DroidDeck/…` in particular.
|
||||
- Launch of an existing install through its existing Steam shortcut (same shortcut, same prefix)
|
||||
and of a fresh install (exe choice, `fuel.json` arguments, the five FuelPump variables in
|
||||
`.droiddeck-launch.bat`); uninstall.
|
||||
@@ -0,0 +1,143 @@
|
||||
# Epic Games Store module: clean-room rewrite
|
||||
|
||||
The Epic integration (the Rust `epic` module in `libdroiddeckstores.so` and the Kotlin package
|
||||
`com.droiddeck.launcher.stores.epic`) was rewritten so that it carries no code from the earlier
|
||||
implementation, whose Rust module stated derivation from third-party Epic clients. This page
|
||||
records how that was done and every judgement call the new code makes.
|
||||
|
||||
## Process
|
||||
|
||||
1. **Spec, written separately.** A functional specification (six documents: summary and unknowns,
|
||||
service protocol, manifest and chunk formats, install behaviour, on-disk compatibility, and the
|
||||
app-facing interface) was written from the old code by a separate pass. It describes behaviour
|
||||
and data formats only, not code structure.
|
||||
2. **Implementation from the spec only.** The implementer worked from those documents and from
|
||||
code that is not Epic-specific:
|
||||
- read: the spec; the crate's `Cargo.toml`, `lib.rs`, `stores_jni.rs`, `priority.rs`,
|
||||
`fetch_tuning.rs`, the shared `fetch_core.rs` and the GOG module (for crate, threading and JNI
|
||||
conventions); `JNI.md`; the Kotlin code outside `stores/epic/` that calls Epic (to keep its API);
|
||||
build files and CI;
|
||||
- not read: anything under `stores/epic/` (Kotlin, Java, tests) or `src/epic*` (Rust) of the old
|
||||
tree, the body of the old `EpicCloud` class and of `CloudSavePaths.epic`, other branches or
|
||||
checkouts, or any Epic client source. The old files were deleted with `git rm` without being
|
||||
opened. (A search over shared files incidentally printed two lines of the old `EpicCloud`, its
|
||||
base URL and User-Agent constant; both values are in the spec and only the URL is used.)
|
||||
3. **Fresh code.** Every file of the new module has its own structure, names and comments, with a
|
||||
plain description and `SPDX-License-Identifier: GPL-3.0-or-later` (the crate's licence). Where
|
||||
the spec left a fact open, the choice was made from general knowledge of Epic's public services
|
||||
and is listed below.
|
||||
|
||||
## Layout
|
||||
|
||||
| Rust (`app/src/main/rust/stores/src/epic/`) | |
|
||||
|---|---|
|
||||
| `net.rs` | blocking HTTP helper, the one User-Agent, percent/base64 encoders |
|
||||
| `account.rs` | sign-in with an authorization code, refresh, exchange codes, redirect-page parsing |
|
||||
| `library.rs`, `catalog.rs` | owned items (all pages), catalog details and attributes |
|
||||
| `build_info.rs` | live build lookup, CDN list, manifest download |
|
||||
| `manifest.rs`, `chunk.rs` | binary and JSON manifests, chunk files |
|
||||
| `locale.rs` | device language → install tags, tag selection |
|
||||
| `install/` | the install engine: `staging`, `delta`, `space`, `cache`, `schedule`, `writer`, `mod` (orchestration), end-to-end `tests` against a loopback server |
|
||||
| `shop.rs`, `achievements.rs`, `savesync.rs`, `launch.rs`, `timefmt.rs` | shelves, achievements, cloud saves, launch arguments, ISO dates |
|
||||
| `service.rs`, `ffi.rs` | the JSON operations and the JNI entry points |
|
||||
|
||||
| Kotlin (`stores/epic/`) | |
|
||||
|---|---|
|
||||
| `EpicBackend` | `StoreBackend` + `EpicLaunchSupport`; installs the login flow |
|
||||
| `EpicLogin` | the `StoreLoginActivity.Flow` |
|
||||
| `EpicCredentialStore`, `EpicPrefs` | account (shared Keystore store), `stores.epic` prefs |
|
||||
| `EpicLibrary`, `EpicShelves` | library sync and cache, shop shelves |
|
||||
| `EpicInstaller` | the download-queue job |
|
||||
| `EpicApiClient`, `EpicDownloadManager`, `EpicInstallTags`, `EpicAchievements` | API kept for callers outside the package |
|
||||
| `EpicSaveSync`, `EpicSaveTemplate` | cloud-save transfers for `EpicCloud`, template parsing for `CloudSavePaths.epic` |
|
||||
| `EpicNative` | JNI wrapper (see `rust/stores/JNI.md`) |
|
||||
|
||||
## Judgement calls
|
||||
|
||||
Numbers in brackets refer to the spec's list of unknowns.
|
||||
|
||||
**Fixed where clearly correct**
|
||||
|
||||
1. OAuth form values (authorization code, refresh token) are percent-encoded [9].
|
||||
2. A library page that fails ends the sync with an error and keeps the cached library instead of
|
||||
caching a partial one; an HTTP 401 is reported as "session expired: sign in again" rather than
|
||||
an empty library [7].
|
||||
|
||||
**Chosen where the spec left it open**
|
||||
|
||||
3. **User-Agent** [1]: every HTTP request (account, library, catalog, build info, CDNs, store
|
||||
GraphQL, promotions, achievements, cloud saves) sends `DroidDeck/<versionName>`, defined once in
|
||||
`epic/net.rs` (`USER_AGENT_PRODUCT`). Whether every endpoint accepts it is untested; changing that
|
||||
constant is the one place to adjust. The sign-in **WebView** keeps the launcher-style UA it used
|
||||
before, because a sign-in page that rejects the UA would lock every user out; that string lives
|
||||
in `epic/account.rs` (`WEBVIEW_USER_AGENT`).
|
||||
4. **Shared fetch core** [14]: chunk downloads reuse the crate's store-agnostic `fetch_core` (also
|
||||
used by GOG and Amazon; it came from Bannerlator's own crate and, unlike the old Epic module,
|
||||
states no derivation from an Epic client). It already provides the adaptive window, CDN
|
||||
rotation, retries with back-off, 429 and stall cooldowns the spec asks for. If its lineage is
|
||||
ever in doubt, it is the one shared dependency to replace.
|
||||
5. **Out-of-package Epic code** [14]: `EpicCloud` and `CloudSavePaths.epic` were rewritten to call
|
||||
the new module. `StoreLaunch`'s exchange-code handling, `EpicOptions`, `WinCompSources.epic` and
|
||||
`SessionPrefs.storesEpicMode` are shared code and were left as they are.
|
||||
6. **Save-path template** parsing is Kotlin (`EpicSaveTemplate`), not Rust: the prefix walk it feeds
|
||||
is shared Kotlin code and its existing unit test runs on the JVM without the native library. The
|
||||
Epic walk matches folder names case- and punctuation-insensitively (`MyGame` finds `My-Game`), as
|
||||
that test expects.
|
||||
7. **One engine** [12, 13]: the hidden `storesEpicMode` switch is not read and there is no Java
|
||||
fallback; without the native library the Epic store reports itself unavailable. When the full
|
||||
space plan does not fit, the engine runs in a narrow mode (one file staged at a time, no chunk
|
||||
cache), which needs only the net peak of the pending files.
|
||||
8. **Schedule and cache**: the cache budget is enforced by the plan (eviction of the chunk needed
|
||||
furthest in the future). At run time files are admitted up to 48 downloads ahead of the
|
||||
furthest dispatched one, so the cache can briefly hold a few entries more than the plan's peak.
|
||||
A chunk is first written into every admitted file that uses it; it touches the cache only when a
|
||||
not-yet-admitted file still needs it.
|
||||
9. **Space probing** uses `statvfs` (available blocks × fragment size, the same figure as Android's
|
||||
`StatFs.getAvailableBytes`) from Rust; a failed probe is "unknown". A scratch directory on the same
|
||||
filesystem as the game is treated as "beside the game".
|
||||
10. **File hashing** is done in file order while parts land; parts that arrive early are read back
|
||||
from the stage file. Files without a manifest hash are not hashed.
|
||||
11. **Staging folder recovery**: an *empty* staging folder without its marker (a run stopped between
|
||||
creating the folder and writing the marker) is adopted and the marker written; any other missing
|
||||
or foreign marker fails the run without deleting.
|
||||
12. **Unsafe manifest paths** (absolute, `..`, empty segments) fail the install.
|
||||
13. **Encrypted manifests** (stored-as bit 1) are refused with an explicit message.
|
||||
14. **Meta section prerequisite ids** are all read (no cap at 64), so the following fields stay aligned.
|
||||
15. **Store GraphQL shelves** are requested with their arguments written into the query text
|
||||
(country reduced to two letters first) instead of typed GraphQL variables, so the query does not
|
||||
depend on the server's variable type names.
|
||||
16. **Achievements**: when an account id is stored and the player-progress query gets no answer, the
|
||||
result is "unreachable" rather than a list with everything locked.
|
||||
17. **Token refresh** is single-flight (one refresh at a time; a waiting caller re-reads the stored
|
||||
account first), so a rotated refresh token is never spent twice. A fresh sign-in replaces the
|
||||
stored account instead of merging with it.
|
||||
18. **Download progress** reported to the queue never moves backwards, although samples come from
|
||||
several native threads.
|
||||
19. **Network window** is the speed tier's value clamped to 1–128; the per-host cap is
|
||||
`max(6, ceil(window / hosts))`.
|
||||
|
||||
**Kept at parity (known weak spots)**
|
||||
|
||||
- Chunk requests do not carry the CDN's signing query; only the manifest request does; the
|
||||
`cloudflare.epicgamescdn.com` CDN stays excluded [2].
|
||||
- The manifest file name comes from the first manifest location and is requested from every CDN [3].
|
||||
- The manifest header SHA-1 and the chunk header GUID/hash are not checked (chunk size and chunk
|
||||
SHA-1 are) [4].
|
||||
- Per-file flags, symlink targets and custom fields are ignored [5]; JSON manifests carry no hashes
|
||||
as read, so their files are rewritten on every run [6].
|
||||
- The refresh token's own expiry is not tracked; a failed refresh still returns the stored token [8].
|
||||
- The cloud `hash` is compared as if it were an MD5 [10].
|
||||
- `AdditionalCommandLine` is one argument, frozen at install [11].
|
||||
- Shop search, an unused "wants an exchange code" helper and per-game uninstall work are not
|
||||
ported; DLC stay hidden and are never installed separately; optional (non-language) install tags
|
||||
are never selected; resume restarts a partially written file from zero [15–18].
|
||||
- Library details are fetched one game at a time, as before.
|
||||
|
||||
## What needs device testing
|
||||
|
||||
- Sign-in through the WebView, and that an **existing** saved account (sealed `credentials.json`)
|
||||
stays signed in after the update.
|
||||
- Every endpoint with the DroidDeck User-Agent: library, catalog, build info, CDN downloads, store
|
||||
GraphQL and promotions, achievements, cloud saves.
|
||||
- Library sync and shelves; install, pause/resume, cancel, update and repair; a card install with
|
||||
the internal scratch cache; launch with an exchange code; cloud-save up/down.
|
||||
+86
-136
@@ -35,8 +35,8 @@ to manifest checks of the files still needing work (GOG MD5, Amazon SHA-256 wher
|
||||
Free-space display probes the nearest existing ancestor without creating the target folder; if
|
||||
the probe cannot determine space, it reports **Free space unknown** rather than blocking an install.
|
||||
`StoreInstallRoot` uses `-1` for unknown and preserves a measured `StatFs` zero as **0 B free**.
|
||||
Fresh GOG/Amazon preflight and all three Java backend space checks reject known zero when bytes
|
||||
remain. The read-only probe walks existing ancestors without creating directories; a failed
|
||||
Fresh GOG/Amazon preflight and the backends' own space checks (GOG in Kotlin, Epic and Amazon in
|
||||
the Rust engine) reject known zero when bytes remain. The read-only probe walks existing ancestors without creating directories; a failed
|
||||
`StatFs` probe can use a positive `File.usableSpace` fallback, but that API's ambiguous zero is
|
||||
unknown, not a measurement of a full disk.
|
||||
Preflight failures remain failed Downloads rows with Clear and Retry. Retry reruns preflight
|
||||
@@ -45,143 +45,80 @@ with a placeholder job. Failures are kept in the Downloads UI, notification and
|
||||
duplicated as transient failure toasts. Active queue persistence and ordinary backend pause/retry
|
||||
are unchanged.
|
||||
|
||||
Epic installs have four writers, all with the same per-file safety rules (below). The hidden
|
||||
`storesEpicMode` string in the `session` preferences picks one: `adaptive` (default), `lookahead`,
|
||||
`batch` or `java`. There is no UI for it; set it on a test device to compare the modes on the same
|
||||
game. Every mode logs an `epic: summary mode=<mode>` line (files, bytes written, elapsed, average
|
||||
MB/s). When `libdroiddeckstores.so` does not load, every mode runs the Java writer. Once it is
|
||||
loaded, an engine failure (refused plan, ABI mismatch, network, hash or I/O failure, unsupported
|
||||
atomic swap) fails the install; it never silently reruns as Java downloads.
|
||||
Epic installs run in the Rust `epic` module of `libdroiddeckstores.so` (`epic/install/`), driven by
|
||||
`EpicInstaller` through `EpicNative.install`. There is one engine; the old hidden `storesEpicMode`
|
||||
switch is no longer read. Without the native library the Epic store cannot install (the facade
|
||||
reports "the store engine is not available"). The design, and how it was written from a
|
||||
functional spec only, is in [epic-cleanroom.md](epic-cleanroom.md).
|
||||
|
||||
**Adaptive (default)** - `EpicNative.stream`, `epic/stream.rs`. Measured on device: about
|
||||
83 MB/s on a 71 GB install. The chunks go through the shared fetch core in the order their files
|
||||
need them, with its full adaptivity (window ramp, per-host caps, in-flight byte budget, retries,
|
||||
host rotation, stall handoff). A chunk only one file part uses is verified and written straight
|
||||
into that file's staging file at the part's offset, in whatever order it arrives; it is never
|
||||
cached, so each byte is written once. A chunk several parts use is written into every part whose
|
||||
file the download has reached and kept, decompressed, in the cache until its last part is written.
|
||||
The cache has a byte budget; the plan simulates the install in file order and, when the budget
|
||||
would be exceeded, drops the cached chunk needed furthest ahead and downloads it again later.
|
||||
Files are opened only once the download reaches them, so the set of half-written files stays
|
||||
within the fetch window. All writes happen in the fetch core's process pool, whose completion
|
||||
frees the in-flight byte budget: a slow card throttles the download instead of growing memory.
|
||||
Each file's SHA-1 is computed as its parts are written (parts that arrived out of order are read
|
||||
back once the bytes before them are in). Stage files go into the owned staging namespace below.
|
||||
The run logs `stream-plan` (direct vs shared chunks, planned refetches, budget) and
|
||||
`stream-summary mode=adaptive` (peak cache, peak staging, refetches, bytes written from the
|
||||
download vs from the cache).
|
||||
A run, in order: CDN list from the build information (`EpicApiClient.getManifestApiJson`) →
|
||||
manifest from the first CDN that serves it → parse (binary or JSON) → files selected by the device
|
||||
language's install tags (`EpicInstallTags`) → staging recovery → the delta pass (a selected file is
|
||||
up to date when it has the manifest size and SHA-1; a file without a manifest hash is always
|
||||
rewritten) → space plan → chunk cache preparation → fetch schedule → download and write → the
|
||||
installed record (`StoreInstalls.complete`). Resume, update and repair are the same rerun.
|
||||
|
||||
On an update, a staged file that replaces an existing one sits beside its old copy until its swap,
|
||||
so the engine paces replacements with an overlap budget: files are admitted in order, a replaced
|
||||
file only while the planned sizes of admitted, not-yet-swapped replaced files fit the budget (one
|
||||
is always admitted, so the run progresses), and a chunk is fetched only once its file is admitted.
|
||||
Fresh files are never held back, and admission only delays files, never reorders them, so the
|
||||
cache plan stays exact. The run logs `overlap_budget=` in `stream-plan` and
|
||||
`peak_replace_overlap_bytes=` in `stream-summary`. With room to spare the budget is large and
|
||||
updates run as wide as fresh installs; on a nearly full volume, replaced files are staged one
|
||||
after another and the fetch pauses at each replaced-file boundary until the previous one is
|
||||
swapped in.
|
||||
- **Schedule** (`epic/install/schedule.rs`). The pending files' parts form one timeline. A chunk is
|
||||
downloaded at its first use and feeds every later use in the same file directly. If it is used
|
||||
again in a later file it is kept, decompressed and verified, in the chunk cache, which has a byte
|
||||
budget; when keeping a chunk would exceed it, the cached chunk needed furthest in the future is
|
||||
dropped and downloaded again when needed. Chunks a previous run left in the cache are reused,
|
||||
soonest-needed first, as far as the budget allows.
|
||||
- **Writing** (`epic/install/writer.rs`). Files are admitted strictly in order, a little ahead of
|
||||
the download, and a download is dispatched only once the file of its first use is admitted, so
|
||||
the number of half-written files stays small. Each admitted file is written into a unique
|
||||
`epic-<pid>-<seq>-<nanos>.stage` file in `<game>/.droiddeck-epic-staging/`; the file SHA-1 is
|
||||
computed in file order as parts land. A complete file is synced, size- and hash-checked, and
|
||||
renamed over its destination. Right before the first rename of the run `StoreInstalls.markUpdating`
|
||||
switches an installed record to `updating`; a refusal fails the run with nothing replaced.
|
||||
- **Fetching** uses the shared `fetch_core` (adaptive window, CDN rotation, retries with back-off,
|
||||
stall and 429 cooldowns) with the speed tier's window and decode workers. Every chunk is checked
|
||||
(header, zlib stream end, exact size, SHA-1) before any byte reaches a file or the cache.
|
||||
- **Space** (`epic/install/space.rs`). Free space is probed per volume (`statvfs`; unknown is never
|
||||
"no space"). The plan: net peak of the pending files in order (each old copy credited at its own
|
||||
swap), a cache budget of half the remaining space (at most 1 GiB, 256 MiB when unknown, 0 is
|
||||
allowed), an overlap budget that limits how many bytes of files replacing existing data are
|
||||
staged at once on an update (at least the largest replaced file), and a 64 MiB transient
|
||||
allowance. A card install keeps its cache in `cacheDir/stores/epic/<id>/` when internal storage
|
||||
can hold a useful cache and the game has no non-empty `.chunks` yet; otherwise the cache is
|
||||
`<game>/.chunks`. When the full plan does not fit, the run falls back to one file at a time with
|
||||
no cache (needs only the net peak); when even that does not fit the install is refused before any
|
||||
chunk request, naming the storage, the need, the free space and the shortfall.
|
||||
- **Progress**: the bar counts pending-file bytes written (not compressed chunk bytes), at most every
|
||||
50 ms plus once per finished file. The run logs `epic: plan …` and `epic: summary mode=adaptive|narrow
|
||||
files=… bytes_written=… avg_mbps=… fetches=… refetches=… peak_cache_mib=… hosts=…`.
|
||||
- **Cancel / pause / failure**: the cancel flag is forwarded within 250 ms; the run stops its
|
||||
requests, joins its threads, removes every unfinished stage file and `.part` cache entry, and only
|
||||
then returns. Swapped files stay; unswapped destinations keep their old content; the chunk cache
|
||||
stays for the next run. Success removes the cache and the staging folder (when only its marker is
|
||||
left).
|
||||
|
||||
Its space plan (`EpicDownloadManager.planAdaptiveSpace`, unit-tested in `EpicAdaptivePlanTest`)
|
||||
checks each volume for what lands on it, with the lookahead planner's rule (the need must not
|
||||
exceed the free reading). The game volume: the net growth at its worst point (`settledNetPeak`,
|
||||
the running Σ(new − old) with each old copy released at its swap), plus the overlap budget, plus
|
||||
the cache when it is beside the game. The overlap budget is whatever the game volume has left
|
||||
after the net growth and the cache, never less than the largest replaced file; when even that
|
||||
does not fit, the plan hands over to the lookahead engine. The cache volume: the budget plus 64 MiB of
|
||||
chunks in transit. The cache budget is half the cache volume's free space after those needs, at most
|
||||
1 GiB (256 MiB when free space is unknown; 0 is allowed and means every reuse is a refetch). On a
|
||||
card install the cache stays on internal scratch unless internal storage cannot hold at least
|
||||
64 MiB of it; then the cache goes beside the game (still released chunk by chunk) instead of
|
||||
failing. A plan that does not fit logs the short storage and the shortfall and hands over to the
|
||||
lookahead engine, which narrows its window to the space there is (or fails its own preflight with
|
||||
an explicit error). Running out of space in the middle of an adaptive run is not destructive: a
|
||||
failed write fails the run, unfinished staging files are removed, the old files stay.
|
||||
The staging folder is recognised by its `owner` marker (`DroidDeck Epic staging v1` + LF). Before
|
||||
the delta pass the run checks that the folder is a real directory at the expected canonical path,
|
||||
that the marker matches and that every other entry is a plain `.stage` file directly inside it; only
|
||||
then are those orphans deleted. Anything unexpected fails the run without deleting. A manifest path
|
||||
naming the staging folder, an absolute path or a `..` segment is refused. `.stage` / `.part` files
|
||||
elsewhere in the game folder are never touched.
|
||||
|
||||
**Lookahead** - `EpicNative.installAdaptive`, `epic/adaptive.rs`; also the adaptive engine's
|
||||
fallback when its space plan does not fit. The Java delta pass sends a versioned snapshot of
|
||||
pending files, chunk metadata, verified cache lengths, old output lengths and both measured
|
||||
free-space probes to Rust. The native planner and execution share one ordered moving-window
|
||||
policy. Before any chunk request, Rust chooses the largest safe lookahead (128, 64, 32, 16, 8, 4,
|
||||
2 or 1 part references) and reusable-cache budget (64 MiB, 16 MiB or zero) whose simulated
|
||||
staged-file and cache peaks fit. Native device IDs also detect an explicitly named cache on the
|
||||
game volume, so two separate paths cannot incorrectly turn one filesystem's requirement into two
|
||||
independent checks.
|
||||
Amazon installs run in the Rust `amazon` module (`amazon/install/`), driven by `AmazonInstaller`
|
||||
through `AmazonNative.install`; sign-in, the library sync and the size lookup call the same module
|
||||
through `AmazonNative.call`. The design, and how it was written from a functional spec only, is in
|
||||
[amazon-cleanroom.md](amazon-cleanroom.md).
|
||||
|
||||
| Storage situation | Lookahead policy |
|
||||
| --- | --- |
|
||||
| Plenty of room | Up to 128 references ahead, up to 32 network requests, 64 MiB reusable data |
|
||||
| Tight | Smaller lookahead and/or reusable budget; parallel fetching remains where space permits |
|
||||
| Barely fits | One-reference lookahead, one request, no reusable retention if needed |
|
||||
| Does not fit even the minimal schedule | Explicit preflight error before requests or replacement |
|
||||
| Free space unknown | Conservative bounded lookahead (at most eight), not an invented zero |
|
||||
|
||||
The lookahead fetch coordinator reuses `fetch_core`'s host scheduling, retries, adaptive network
|
||||
window and decode pool. Each chunk is cached, then its parts are copied into staged files by a
|
||||
writer that runs concurrently, waiting only for its next required chunk, through the native
|
||||
assembler's streaming copy primitive. GUIDs in the lookahead are reserved once before submission,
|
||||
at their full decoded size. Chunks are released at last use on **both** cache layouts; distant
|
||||
reusable data is oldest-first bounded and may be refetched. At most the selected reusable budget
|
||||
plus distinct lookahead decoded sizes is resident/reserved. New request admission re-probes both
|
||||
volumes and can shrink unsent lookahead or retention if headroom disappears; already in-flight
|
||||
reservations remain protected. If even existing reservations cannot fit, the run fails and joins
|
||||
its workers rather than waiting indefinitely or deleting old game files. An install to SD retains
|
||||
its internal scratch cache, so assembled output is the only game data written to the card.
|
||||
|
||||
**Java** - without the native library, or with `storesEpicMode=java`. Eight Java workers prefetch
|
||||
the next eight chunk-part references, across file boundaries, while one writer assembles and
|
||||
verifies files in manifest order. Shared GUIDs within that window are fetched only once. Chunks
|
||||
are deleted after their last use; other reusable chunks are retained up to 64 MiB. At most eight
|
||||
additional decoded chunks (complete or in flight) coexist with that retained cache. The size
|
||||
bound is 64 MiB plus the decoded sizes of the window's distinct GUIDs, not a fixed eight-MiB
|
||||
allowance: manifests can supply larger chunk windows. Evicted distant shared chunks may be
|
||||
downloaded again. No worker downloads outside the moving window; cancellation and failure
|
||||
disconnect and join all workers before returning or removing the cache. Its summary line is
|
||||
`mode=java-bounded`.
|
||||
|
||||
**Batch** - the earlier native design, kept for A/B benchmarks: every chunk fetched into the
|
||||
cache in parallel, then the files assembled from it (`EpicNative.run` + `EpicNative.assemble`).
|
||||
It needs the cache and the files at once (about twice the game) and checks exactly that before
|
||||
starting; it writes files in place, so an existing install is marked `updating` before assembly.
|
||||
`epic/assemble.rs::build_assemble_plan` enables `scratch_cache` only for a nonempty cache path:
|
||||
`run_assemble` then removes chunks at their last use; the default beside-game cache is removed at
|
||||
the end (`default_cache_assembles_like_the_kotlin_loop_and_removes_the_cache_at_the_end` and the
|
||||
scratch-cache assembly tests cover that distinction).
|
||||
|
||||
The Epic download bar, measured rate and ETA (adaptive, lookahead and Java) count pending file bytes written, not unique
|
||||
compressed chunks fetched. Shared chunks therefore cannot complete the bar before later files
|
||||
are written, and an evicted chunk's refetch is reflected in the time spent doing that work.
|
||||
Streaming stays in the Download stage until all files are written and verified, then enters
|
||||
Install for registration; it does not bounce between stages for each file.
|
||||
|
||||
For the lookahead and Java writers, the space check follows the manifest delta pass and uses the
|
||||
same pure moving-window policy as execution, reserving all incoming decoded chunk sizes before writing parts and simulating
|
||||
last-use releases and unpinned oldest-first evictions. Existing verified cache bytes count against
|
||||
current free space and are credited only at their scheduled release. This avoids adding the entire
|
||||
cache to the entire game without undercounting concurrent downloads. Each file is written to a
|
||||
unique file in an owned staging namespace on its output volume (never the literal `<filename>.part`, which can
|
||||
be a legitimate manifest file), size/hash verified, synced and atomically renamed over its output.
|
||||
The old file remains until that swap succeeds. The planner credits its bytes only after that
|
||||
file's complete staging write, not by subtracting all old outputs up front. Both volume checks
|
||||
must accept this safe staging peak; insufficient space is rejected explicitly, with no destructive
|
||||
delete-first fallback. An unsupported/failed atomic rename fails the install and retains the old
|
||||
destination. Completed files and
|
||||
verified cached chunks are accounted for on resume. On a card, game and cache requirements
|
||||
are checked separately. File SHA-1 hashes are checked where supplied by the manifest; failed
|
||||
or cancelled staging files are removed without deleting old outputs. A successful run removes the
|
||||
cache. A resume within a large incomplete file restarts that file, not its individual parts.
|
||||
All three staged writers (adaptive, lookahead, Java) use `<installDir>/.droiddeck-epic-staging/`, whose `owner` marker identifies this
|
||||
namespace. A successful install removes it when it holds nothing but the marker (same no-follow
|
||||
and canonical-path checks); the next run recreates it. Before the delta pass and free-space probes, the manager removes orphan `.stage`
|
||||
entries only from this owned namespace. A manifest naming that directory or any descendant
|
||||
fails without deletion; an unowned directory or unexpected entry also fails conservatively.
|
||||
The namespace must be a real directory at the expected canonical child of the install root.
|
||||
Ownership markers and staging entries must be no-follow regular files; symlink directories,
|
||||
markers and entries are rejected before recovery deletes anything.
|
||||
Ordinary game files with `.stage` or `.part` suffixes elsewhere are never recovery targets.
|
||||
Abrupt process termination is therefore recoverable on the next run without inferring ownership
|
||||
from a loose filename prefix. Cancellation/failure waits for all native/Java fetch writers before
|
||||
returning; completed verified cache chunks remain usable on resume.
|
||||
An Amazon run, in order: download lookup for the entitlement (`GetGameDownload`: a CDN base URL with
|
||||
a signed query, and the version id) → `manifest.proto` from that base (a 4-byte big-endian header
|
||||
length, an opaque header, then an xz or LZMA-alone compressed protobuf) → parse (every file of every
|
||||
package; paths refused when absolute, with a drive letter, `..`, empty segments, or naming
|
||||
DroidDeck's own root files) → leftover `<file>.tmp` siblings removed → the check of files on disk (a
|
||||
file is complete when it has the manifest size and SHA-256; the same test drives the space check and
|
||||
the download plan) → space check (`Not enough free space: need …, only … free`; unknown free space
|
||||
never blocks) → empty files created locally → every other pending file fetched from
|
||||
`files/<sha256 hex>` under the base, one streamed GET each through the shared fetch core (the speed
|
||||
tier's window; at least half the CPU cores verifying and writing), hashed as it arrives, written to
|
||||
`<file>.tmp` and renamed over the destination after the size and hash checks → the installed record.
|
||||
Before the first file is put in place the record is switched to `updating`, so an interrupted repair
|
||||
is offered as "Resume install". There is no partial-file resume: a file is complete or fetched again
|
||||
from its first byte. Files the manifest does not list are left alone.
|
||||
|
||||
## The sidecar
|
||||
|
||||
@@ -266,6 +203,15 @@ An Epic game's launch choices are the Epic card in the Games tab's launch settin
|
||||
read the same thing - and "Resolve Epic sign-in", which opens Epic's account page behind its
|
||||
sign-in in Android's browser.
|
||||
|
||||
An Amazon game's exe and arguments come from its `fuel.json` (`Main.Command`, used when it names
|
||||
an existing file; `Main.Args`, used even when it does not), else from a score over every `.exe`
|
||||
below the folder (Unreal shipping binaries and `Binaries/Win64` first, crash handlers, setups,
|
||||
launchers and tiny stubs last; the rules see paths relative to the install folder), else the shared
|
||||
picker. Its launcher sets five FuelPump variables frozen at install time: `FUEL_DIR`,
|
||||
`AMAZON_GAMES_SDK_PATH`, `AMAZON_GAMES_FUEL_ENTITLEMENT_ID`, `AMAZON_GAMES_FUEL_PRODUCT_SKU` and
|
||||
`AMAZON_GAMES_FUEL_DISPLAY_NAME` (the account name, `Player` without one). No Amazon Games SDK files
|
||||
are deployed into the prefix and `fuel.json` `PostInstall` steps are not run.
|
||||
|
||||
A game's web links reach Android: `droiddeck-store-launch` sets the prefix's
|
||||
`HKCU\Software\Wine\WineBrowser` `Browsers` to `/usr/local/bin/droiddeck-open-url`, which is also
|
||||
`xdg-open` first on the game's PATH, and hands http(s) addresses to the app, which opens them in
|
||||
@@ -306,7 +252,9 @@ the prefix's user folder appears, within a minute. Every path logs
|
||||
has no cloud saves.
|
||||
- **The services:** GOG `cloudstorage.gog.com/v1/<user>/<client>` with a token issued to the game's
|
||||
own client (the client secret is kept at install; the Galaxy token is the fallback); Epic's
|
||||
savesync data storage with signed read and write links.
|
||||
savesync data storage with signed read and write links (`EpicCloud` → `EpicSaveSync` → the Rust
|
||||
`epic/savesync.rs`; the template's root token and `{EpicId}` / `{AppName}` are read by
|
||||
`EpicSaveTemplate`).
|
||||
- **Which launches:** `droiddeck-store-launch` recognises a store game by the launcher .bat Steam runs
|
||||
or, for a game without one (most GOG games), by its exe - the folders above it are searched for
|
||||
the sidecar.
|
||||
@@ -332,7 +280,9 @@ keeps the files; every store's install skips complete, verified files on the rer
|
||||
rerun. Cancel deletes the folder. `StoreDownloadService` holds a foreground notification while
|
||||
anything runs. The native engine (`libdroiddeckstores.so`, `StoresNative`, `GogNative`, `EpicNative`,
|
||||
`AmazonNative`; contract in `app/src/main/rust/stores/JNI.md`) is loaded on first use; without it
|
||||
each manager runs its Java fetch loop.
|
||||
the GOG manager runs its Java fetch loop, and the Epic and Amazon stores (all of whose service calls
|
||||
and installs run in the Rust `epic` and `amazon` modules) are unavailable: sign-in, sync and install
|
||||
report "the store engine is not available".
|
||||
|
||||
## Settings
|
||||
|
||||
|
||||
Reference in new issue
Block a user