Stores: GOG, Epic Games and Amazon Games libraries, installs and cloud saves (#476)

A Stores section (rail, under Games; Setup › Stores gate) with each store's sign-in,
library, storefront shelves and a download queue. Each store has two tabs, Store and
Library, and Library filters to All or Installed; libraries open from a saved copy
and refresh in the background, with install sizes on every card. Games install to internal storage,
or a card chosen at Install time, and are registered in the Steam client as non-Steam
games, so they launch through Steam and droiddeck-proton from either the front end or
the client.

- Downloads: GOG, Epic and Amazon engines in Rust (app/src/main/rust/stores, built by
  cargo-ndk, nothing compiled in the tree). One adaptive fetch core with a latency
  brake, stalled-host handoff and every CDN a store offers. Per-stage progress,
  pause/resume, two-step cancel that deletes, failed downloads kept for retry, a
  foreground service with Wi-Fi and wake locks and notification controls.
- Games tab: store source chips (Steam, GOG, Epic, Amazon, Custom) and store art.
- Epic: per-launch exchange code from the app and from Steam's Play button,
  "Epic sign-in (EOS)", "Launch offline", and a game's web links opened in Android's
  browser for EOS consent. The EOS overlay crashed games and is left out.
- Cloud saves (GOG, Epic): pulled before launch, pushed after the game or the session
  ends, no upload before a first download, conflicts never overwritten, backups both
  ways.
- Logs: every store line redacted; the engine log goes to daily files.
This commit is contained in:
The412Banner authored and GitHub committed 2026-10-09 11:08:17 -04:00
1 parent fe71084f23
commit e99d5f9505
141 files changed
+25539 -73

No files matched your search

+1729
View File
File diff suppressed because it is too large. Load diff
+41
View File
@@ -0,0 +1,41 @@
[package]
name = "droiddeckstores"
version = "0.1.0"
edition = "2021"
# The engines are a port of Bannerlator's bl-steam-client store modules (GPL-3.0-or-later, derived
# from WinNative's wn-steam-client; the Epic selection/verify rules trace back to GameNative and
# Legendary, both GPL-3.0). DroidDeck is GPL-3.0, so the crate keeps the licence it came with.
license = "GPL-3.0-or-later"
publish = false
[lib]
name = "droiddeckstores"
crate-type = ["cdylib"]
[dependencies]
jni = "0.21"
# setpriority/gettid: every engine thread runs at nice +10 (see src/priority.rs). Already in the
# dependency tree through tokio, at the version Cargo.lock pins.
libc = "0.2"
# zlib inflate for GOG gen2 chunks and Epic chunk bodies. The pure-Rust backend keeps the .so free
# of a C zlib and of the NDK sysroot's libz, which the apk does not ship.
flate2 = { version = "1", default-features = false, features = ["rust_backend"] }
# The fetch core drives every request through ONE tokio current-thread runtime and feeds a sync
# process pool; rustls (via ring) is the TLS stack so there is no OpenSSL to cross-compile. `gzip`
# transparently inflates a CDN answer that comes back content-encoded. No `blocking` client: the
# three engines are async-fetch only.
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "gzip"] }
tokio = { version = "1", default-features = false, features = ["rt", "net", "time", "sync"] }
futures-util = { version = "0.3", default-features = false, features = ["std"] }
# preserve_order keeps JSON object keys in source order, as in Bannerlator, so the GOG depot plan
# walks a manifest's files in the order the manifest lists them (largest-first sorting happens
# after, explicitly).
serde_json = { version = "1", features = ["preserve_order"] }
# Epic chunks are SHA-1 verified, Amazon files SHA-256; GOG's MD5 is the in-tree md5_small.
sha1 = "0.10"
sha2 = "0.10"
[profile.release]
codegen-units = 1
lto = "thin"
strip = "symbols"
+272
View File
@@ -0,0 +1,272 @@
# `libdroiddeckstores.so` — the JNI contract
The Kotlin side of the store download engines mirrors this file exactly. Every symbol the library
exports is listed here with the class it binds to, the JNI signature, when it is called and on
which thread. The engines came from Bannerlator's `bl-steam-client` crate; the method names and
signatures are identical to Bannerlator's `BlGogDownload` / `BlEpicDownload` / `BlAmazonDownload`,
only the package and class names changed.
Load it once with `System.loadLibrary("droiddeckstores")`. The library is built by Gradle
(`buildRustStores`, see `app/build.gradle`) and is absent from a `-PskipRust=true` build, so every
call site must be ready for `UnsatisfiedLinkError` on `loadLibrary` or on the first `external`
call and degrade to "store unavailable".
## Conventions that hold for all three engines
- The `external` functions are declared `@JvmStatic` in a Kotlin `object` (a `JClass` arrives as
the second JNI argument). Declaring them in a plain `object` without `@JvmStatic` also binds —
the symbol name is the same — but keep `@JvmStatic` so the contract matches Bannerlator's.
- No overloads: the exported names are the short JNI form (`Java_<package>_<Class>_<method>`), so
adding a second `nativeStart` with a different parameter list would NOT bind.
- Listener objects are only ever reached through `CallVoidMethod` by name + signature on the
object that was passed to `nativeStart`. The library never does `FindClass`, so the listener's
class may be any class (interface, nested interface, anonymous object) that declares the listed
methods with exactly these signatures. The names suggested below are what the Rust doc comments
use.
- Every listener method runs on a native thread (the run's worker thread, a process-pool thread
or the throughput reporter) that is attached to the JVM as a daemon. Never touch Views directly;
post to the main thread. A Java exception thrown by a listener method is cleared and ignored by
the native side — it does not stop the run.
- `nativeStart` returns an opaque handle (`long`). `0` means the run did not start. The handle is
a heap pointer: call `nativeRelease(handle)` exactly once per non-zero handle (after
`onComplete`), never twice, never with a made-up value.
- `nativeCancel(handle)` is idempotent and safe at any time between `nativeStart` and
`nativeRelease`. The run stops at its next scheduling point; `onComplete` still fires.
- `nativeRelease(handle)` is safe even while a cancelled run is still winding down: the worker
thread owns its own references (cancel flag, listener `GlobalRef`).
- `caBundlePath`: a PEM bundle the HTTPS client adds as extra root certificates; `""` = the roots
rustls ships with (webpki). Bannerlator extracts the system CA store to a file for this; on
DroidDeck `""` is the sensible default unless a CDN fails TLS verification.
- `maxWorkers` is the ceiling of the adaptive in-flight request window; `processWorkers` is the
number of sync inflate+hash+write threads. Bannerlator passes its "download threads" preference
for both (8 by default).
## `com.droiddeck.launcher.stores.StoresNative`
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeVersion(): String` | `()Ljava/lang/String;` | `Java_com_droiddeck_launcher_stores_StoresNative_nativeVersion` |
Returns the crate version from `Cargo.toml` (`"0.1.0"`). Call it from Setup, after
`System.loadLibrary("droiddeckstores")`, to prove the library loads and binds; show the string
next to the other component versions. May return `null` only if the JVM cannot allocate a string.
```kotlin
package com.droiddeck.launcher.stores
object StoresNative {
@JvmStatic external fun nativeVersion(): String
}
```
## `com.droiddeck.launcher.stores.gog.GogNative`
GOG gen2 chunk engine (depot manifests → chunk fetch + zlib inflate + MD5) and gen1 (build manifest
→ per-file HTTP Range GET streamed to disk). One `nativeStart` = one download loop (base install,
DLC install, or a dependency-redist assembly). The Kotlin manager owns everything before (token,
builds, manifest head, product + language depot selection, secure-link resolution, disk guard) and
after it (marker files, exe resolution, DLC markers, redist orchestration).
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeProbe(): Int` | `()I` | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeProbe` |
| `nativeStart(...)`: Long | `(I[Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;IIZLjava/lang/String;Ljava/lang/Object;)J` (last parameter is the listener; its static type is the app's choice) | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeStart` |
| `nativeCancel(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeCancel` |
| `nativeRelease(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeRelease` |
`nativeProbe()` always returns `1`; it exists so `isAvailable()` can catch `UnsatisfiedLinkError`
once at startup instead of mid-download.
`nativeStart` parameters, in order:
| # | Parameter | Meaning |
|---|---|---|
| 1 | `kind: Int` | `0` = `KIND_GEN2_CHUNKS`, `1` = `KIND_GEN1_RANGES` (anything else → gen2) |
| 2 | `depotManifests: Array<String>` | gen2: the inflated depot-manifest JSON strings, in fetch order, already filtered by product + language; gen1: the inflated build manifest (one element) |
| 3 | `cdnBases: Array<String>` | gen2: **every** CDN base of the secure-link answer — one per `urls[]` entry, each resolved the way `parseCdnUrl` resolves `urls[0]` today (query string intact), in GOG's order — or the one unauthenticated dependency store base (an array of one); gen1: `emptyArray()` (file URLs are in the manifest). Blank entries are ignored and a base whose host already appeared is dropped (the first wins). The fetch core spreads the window across the distinct hosts with a per-host cap of `max(6, ceil(maxWorkers / hosts))`, rotating a failed chunk to another CDN; one base behaves exactly like the old single `cdnBase`. The start log line lists them as `hosts=a,b,c`. |
| 4 | `installDir: String` | absolute install directory |
| 5 | `skipPaths: Array<String>` | relative paths already completed by an earlier run of this same download (secure-link refresh re-run): counted done without re-hashing, no progress event |
| 6 | `caBundlePath: String` | see conventions |
| 7 | `maxWorkers: Int` | window ceiling for this loop (Bannerlator: its download-threads preference, 8, or 1 for redists) |
| 8 | `processWorkers: Int` | inflate/hash/write threads |
| 9 | `sortLargestFirst: Boolean` | base install = `true` (largest files first); DLC / dependency = `false` |
| 10 | `label: String` | log prefix, e.g. `"gog base"`; `""` → `"gog"` |
| 11 | `listener` | see below |
Returns `0` without any callback when: the listener is null, the JavaVM cannot be obtained, the
install dir or manifest list is empty, `cdnBases` holds no usable base for gen2, or the worker
thread cannot be spawned. Otherwise the run is on a thread named `gog-dl` and `onComplete` fires exactly once.
Suggested listener: `com.droiddeck.launcher.stores.gog.GogNative.Listener` (an interface).
| Listener method | JNI signature | When |
|---|---|---|
| `onProgress(bytesDone: Long, bytesTotal: Long, filesDone: Int, filesTotal: Int, file: String, fileBytes: Long, verified: Boolean)` | `(JJIILjava/lang/String;JZ)V` | one file reached its final state. `verified = true` = resume-skip (existing file passed size+MD5; no bytes credited, `fileBytes` = its size); `false` = freshly assembled, size+MD5-verified and renamed (`fileBytes` = its decompressed size). `filesDone` counts both, `bytesDone` counts assembled bytes only. Process-pool / worker thread. |
| `onBytes(bytesDone: Long, bytesTotal: Long)` | `(JJ)V` | byte progress between file completions: every 250 ms while the run is fetching, only when the value moved, plus once when the fetch ends. `bytesDone` = bytes of assembled files + bytes of files still in flight (gen2: their finished chunks, decompressed; gen1: what the current attempt has written; a retried attempt starts over, so nothing is counted twice); `bytesTotal` is the same total `onProgress` reports. It never goes past the next `onProgress.bytesDone` by more than the in-flight files and is equal to it once they finish. Drive the progress bar and speed from this; keep `onProgress` for file counts and per-file events. Called from the engine's ticker thread. Optional: a listener without it still works (the call fails quietly). |
| `onLog(line: String)` | `(Ljava/lang/String;)V` | engine diagnostics; the same line already went to logcat under the tag `GogNative`. Any native thread. |
| `onComplete(success: Boolean, cancelled: Boolean, linkExpiry: Boolean, error: String, bytesWritten: Long, filesDone: Int)` | `(ZZZLjava/lang/String;JI)V` | exactly once, from the worker thread, last. `linkExpiry = true` = the run died on an HTTP 401/403/404/500, the codes the GOG secure link returns once it expired: refresh the link and call `nativeStart` again with the finished files in `skipPaths`. `error` is `""` on success. |
On-disk protocol (what a resumed run, by either engine, relies on): a file is assembled into
`<file>.bhtmp`, then renamed over `<file>` once its size and MD5 match the manifest; an existing
`<file>` whose size+MD5 match is skipped. The engine writes nothing else.
## `com.droiddeck.launcher.stores.epic.EpicNative`
Two runs make up the native half of an Epic install:
1. **Fetch** (`nativeStart`): fills the chunk cache with verified, decompressed chunks for the
files the manager says are pending. The Kotlin manager still fetches and parses the manifest
API JSON, downloads the manifest, selects files (install tags), runs the delta/verify pass and
does every post-install step. The engine re-parses the same manifest bytes (ChunksV4 binary
or legacy JSON), rebuilds the chunk plan for the pending files, skips chunks already in the
cache, and writes verified chunks with a `.part` + rename protocol, so the on-disk state is the
same whichever side fetched.
2. **Assemble** (`nativeAssemble`): writes the pending files from the cache, one sequential write
per file, SHA-1 checked as the bytes go out when the manifest hashes the file. This is the
same loop the Kotlin manager runs today (`for part in file.parts: write(chunk[offset..+size])`,
same "Missing chunk X for Y" wording); the Kotlin loop remains a valid fallback.
**The chunk cache directory** is the `chunkCacheDir` parameter of both calls (always pass the
same value to both for one install):
- `""` — today's behaviour: the cache is `<installDir>/.chunks`. The fetch leaves it in place;
`nativeAssemble` removes it whole at the end of a successful run (as the Kotlin loop's
`deleteDir(.chunks)` does), and never deletes a chunk early.
- any other absolute path — a **scratch cache**, meant for internal storage when the game
installs to an SD card (each byte then hits the card once instead of twice; the card's write
speed stops starving the fetchers). The fetch fills it exactly like `.chunks`. The assembler
reference-counts chunk GUIDs across the files it was given and **deletes each cached chunk as
soon as the last file that uses it has been written and verified**, then removes the directory
at the end of a successful run. A cancelled or failed run (fetch or assembly) leaves the cache
exactly as it stands, for resume: the next run's delta pass skips finished files and the next
fetch skips chunks still cached. The directory is created by the fetch if it does not exist;
give each install its own (e.g. `<cacheRoot>/epic/<appName>`), since a successful assembly
removes it.
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeStart(...)`: Long | `([BLjava/lang/String;Ljava/lang/String;[Ljava/lang/String;[IIJLjava/lang/String;IILjava/lang/Object;)J` (last parameter is the listener) | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeStart` |
| `nativeAssemble(...)`: Long | `([BLjava/lang/String;Ljava/lang/String;[ILjava/lang/Object;)J` (last parameter is the listener) | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeAssemble` |
| `nativeCancel(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeCancel` |
| `nativeRelease(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeRelease` |
`nativeCancel` / `nativeRelease` take the handle of either run. There is no `nativeProbe`;
Bannerlator probes by catching the `Throwable` from `loadLibrary` / the first `nativeStart`.
```kotlin
@JvmStatic private external fun nativeStart(
manifest: ByteArray, installDir: String, chunkCacheDir: String, cdnPrefixes: Array<String>,
pendingFileIdx: IntArray, expectedChunks: Int, expectedBytes: Long, caBundlePath: String,
maxWorkers: Int, processWorkers: Int, listener: Listener,
): Long
@JvmStatic private external fun nativeAssemble(
manifest: ByteArray, installDir: String, chunkCacheDir: String, fileIdx: IntArray, listener: Listener,
): Long
```
### `nativeStart` parameters, in order
| # | Parameter | Meaning |
|---|---|---|
| 1 | `manifest: ByteArray` | the raw manifest bytes the manager downloaded (binary ChunksV4 or JSON) |
| 2 | `installDir: String` | absolute install directory |
| 3 | `chunkCacheDir: String` | `""` = `<installDir>/.chunks`; else the scratch cache directory (see above) |
| 4 | `cdnPrefixes: Array<String>` | the CDN base URLs from the manifest API answer (`.../ChunksV4/<n>/` style); one or many, distinct hosts widen the window |
| 5 | `pendingFileIdx: IntArray` | indices into the manifest's file list of the files still to fetch (the manager's delta/verify result) |
| 6 | `expectedChunks: Int` | the manager's own needed-chunk count for a cross-check; `-1` = no cross-check |
| 7 | `expectedBytes: Long` | the manager's `Σ max(fileSize, 1)` for a cross-check; `-1` = no cross-check |
| 8 | `caBundlePath: String` | see conventions |
| 9 | `maxWorkers: Int` | window ceiling (Bannerlator: 8) |
| 10 | `processWorkers: Int` | inflate/verify/write threads |
| 11 | `listener` | see below |
Planning (manifest parse + plan + cross-check) runs synchronously on the calling thread, so a
plan failure returns `0` BEFORE any fetch; the manager then runs its own fallback loop. On `0`,
`onComplete(false, reason, 0)` has already been called synchronously (except for a null listener).
On success, `onPlan` has been called synchronously before `nativeStart` returns, then the fetch
runs on a new thread. The engine's `plan` log line names the cache:
`plan chunk_dir=ChunksV4 cache_dir=<dir> scratch=<bool> files_pending=… chunks=… bytes=… …`.
### `nativeAssemble` parameters, in order
| # | Parameter | Meaning |
|---|---|---|
| 1 | `manifest: ByteArray` | the same manifest bytes as the fetch |
| 2 | `installDir: String` | absolute install directory (files are written at `installDir/<filename with \\ → />`) |
| 3 | `chunkCacheDir: String` | the same value the fetch was given (`""` or the scratch directory) |
| 4 | `fileIdx: IntArray` | indices into the manifest's file list of the files to write — the fetch's `pendingFileIdx`, in one call |
| 5 | `listener` | see below |
Parse + plan run on the calling thread: an unparsable manifest, an index out of range or an
empty `installDir` returns `0` after a synchronous `onComplete(false, reason, 0)` and nothing is
written. Otherwise the assembly runs on a new thread. Each file is created fresh and written in
part order; on any error (missing chunk, short chunk, I/O error, SHA-1 mismatch) the file being
written is deleted, the run fails with that error, no further files are touched and the cache is
left as it is. A manifest SHA-1 of all zeros (and a JSON manifest, which has none) means no hash
check — the Kotlin delta pass treats those the same way. The run's log lines:
`assemble cache_dir=… scratch=… files=… bytes=… chunks_referenced=…`, then `cache removed …` /
`cache not removed …: <why>` (not fatal) and `assembled files=… bytes=… chunks_removed_early=…`.
### Listener
Suggested: `com.droiddeck.launcher.stores.epic.EpicNative.Listener` (nested interface, as in
Bannerlator). One interface serves both runs.
| Listener method | JNI signature | When |
|---|---|---|
| `onPlan(chunksTotal: Int, bytesTotal: Long, chunkDir: String)` | `(IJLjava/lang/String;)V` | fetch only: once, synchronously inside `nativeStart`, before any fetch: the plan the engine derived (`chunkDir` is the manifest's, e.g. `ChunksV4`, not the cache directory) |
| `onProgress(bytesDone: Long, bytesTotal: Long, done: Int, total: Int)` | `(JJII)V` | fetch: per accounted chunk (cached-skip or fetched), `done`/`total` = chunks; assembly: after each file written, `bytesDone` = bytes written so far, `done`/`total` = files. Native threads. |
| `onLog(line: String)` | `(Ljava/lang/String;)V` | engine log line, already in logcat under `EpicNative`; any native thread (also synchronously from `nativeStart`/`nativeAssemble` for the start line) |
| `onComplete(success: Boolean, error: String, bytes: Long)` | `(ZLjava/lang/String;J)V` | terminal, exactly once (worker thread; or synchronously when the call returns `0`). Fetch: `bytes` = credited bytes (cached-skip + fetched). Assembly: `bytes` = bytes written. `error` is `"cancelled"` on cancel. |
Bannerlator's `BlEpicDownload.run(...)` wraps `nativeStart` in a blocking call that polls an
`AtomicBoolean` cancel flag every 250 ms, calls `nativeCancel` on cancel, waits up to 5 s for
`onComplete`, then `nativeRelease` in `finally`. The same wrapper shape fits `nativeAssemble`
(cancel is honoured between files; a multi-GB file finishes before the run stops).
## `com.droiddeck.launcher.stores.amazon.AmazonNative`
The file-fetch pool of an Amazon Games install: one streamed HTTP GET per manifest file, SHA-256
checked, written as `<dest>.tmp` and renamed. The manager keeps manifest/auth/markers/post-install
and hands the engine a JSON plan.
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeStart(...)`: Long | `(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;IILjava/lang/Object;)J` (last parameter is the listener) | `Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeStart` |
| `nativeCancel(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeCancel` |
| `nativeRelease(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeRelease` |
`nativeStart` parameters, in order:
| # | Parameter | Meaning |
|---|---|---|
| 1 | `planJson: String` | a JSON array, one object per manifest file: `{"relPath": "...", "url": "https://...", "size": <long>, "sha256hex": "..."}` (`sha256hex` may be `""` = no hash check). Every manifest file goes in: the engine applies the size-based resume-skip itself. |
| 2 | `installDir: String` | absolute install directory |
| 3 | `caBundlePath: String` | see conventions |
| 4 | `maxWorkers: Int` | window size; `<= 0` = the engine default (8, Amazon's own `MAX_PARALLEL`) |
| 5 | `processWorkers: Int` | sync verify+write threads (`<= 0` → 1) |
| 6 | `listener` | see below |
Returns `0` when the listener is null (no callback), or when `planJson` / `installDir` is empty
(`onComplete(false, "empty plan or install dir", 0)` fires synchronously first), or when the
listener `GlobalRef` cannot be taken (no callback). The engine keeps the `JavaVM` it takes from
the first `nativeStart`.
Suggested listener: `com.droiddeck.launcher.stores.amazon.AmazonNative.Listener` (an interface).
| Listener method | JNI signature | When |
|---|---|---|
| `onProgress(bytesDone: Long, bytesTotal: Long, filesDone: Long, filesTotal: Long)` | `(JJJJ)V` | once up front with the resume-skipped credit, then after each committed file (and as pieces stream in); `bytesDone` includes skipped files. Native threads. |
| `onLog(line: String)` | `(Ljava/lang/String;)V` | one diagnostic line (`engine=rust …`, `fetch-window …`, `summary …`). The Amazon engine does NOT write to logcat itself: the Kotlin side is the one place that logs these, so each line appears once. |
| `onComplete(success: Boolean, error: String, bytesWritten: Long)` | `(ZLjava/lang/String;J)V` | exactly once, when the run succeeds, fails or is cancelled (`error = "cancelled"`). |
Bannerlator's `BlAmazonDownload.runBlocking(...)` polls a cancel probe every 100 ms, calls
`nativeCancel` once, and `nativeRelease` in `finally` after `onComplete`; on thread interrupt it
cancels and waits up to 30 s for the run to wind down before releasing.
## Log tags
Lines go to logcat under `GogNative` and `EpicNative` (the Amazon engine logs only through
`onLog`). Bannerlator used `BL_GOG_DL` / `BL_EPIC_DL` / `BL_AMAZON_DL`; the line grammar
(`fetch-start`, `fetch-window`, `throughput`, `fetch-end`, `summary`) is unchanged, so its tuning
notes still read against a DroidDeck log.
+990
View File
@@ -0,0 +1,990 @@
//! Amazon Games download adapter over the shared fetch core.
//!
//! Replaces ONLY the byte-fetching pool of `AmazonDownloadManager.install()` (Step 4).
//! Java still resolves the download spec, fetches and parses `manifest.proto`, and hands
//! this module the resolved file list (one JSON entry per manifest file with the exact
//! URL the Java loop would have opened). The adapter mirrors the Java rules 1:1 where the
//! core allows it — see Bannerlator's `docs/RUST_AMAZON_PARITY.md` §4 / §9 for the table and the
//! documented deviations:
//!
//! * one `FetchItem` per WHOLE file, streamed (`FetchOptions.stream = true`): pieces are
//! appended to `<dest>.tmp` as they arrive, 8 files wide like Java's pool, memory bounded
//! to the pieces not yet written;
//! * resume-skip = `st_size == manifest size`, no hash check on skip (Java `:250`);
//! * SHA-256 is accumulated while streaming and checked in `on_finish` when the manifest
//! carries a hash; a mismatch deletes the tmp and is a retryable failure (Java `:269`);
//! * `<dest>.tmp` → delete existing dest → rename (Java `:280`); rename failure is fatal
//! without retry (Java `:281`);
//! * a piece credits its wire length to progress as it is written (Java `:347`), and pieces
//! of an attempt that later fails stay counted (Java never rolls `totalDownloaded` back).
pub mod jni;
use crate::fetch_core::{run_fetch, FetchItem, FetchOptions, FetchSink, SinkError};
use sha2::{Digest, Sha256};
use std::collections::HashMap;
use std::fs::{self, File};
use std::io::Write;
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
/// Java `AmazonDownloadManager.MAX_PARALLEL`.
pub const MAX_PARALLEL: usize = 8;
/// Java `AmazonDownloadManager.DOWNLOAD_USER_AGENT`.
pub const DOWNLOAD_USER_AGENT: &str = "nile/0.1 Amazon";
/// Java `conn.setReadTimeout(120000)`; in stream mode the core applies this to the response
/// headers and then as an idle deadline per received piece — the same shape as a read timeout.
pub const REQUEST_TIMEOUT: Duration = Duration::from_secs(120);
/// Java `new File(installDir, file.unixPath() + ".tmp")`.
pub const TMP_SUFFIX: &str = ".tmp";
/// Log label used in the core's `fetch-window` lines.
pub const LABEL: &str = "amazon";
/// One resolved manifest file as serialised by Java (`AmazonDownloadManager.buildRustPlan`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PlanEntry {
/// `ManifestFile.unixPath()` — relative to the install dir, forward slashes.
pub rel_path: String,
/// `AmazonApiClient.appendPath(downloadUrl, "files/" + hashHex)` — signed, whole file.
pub url: String,
/// `ManifestFile.size`.
pub size: u64,
/// SHA-256 to verify against; EMPTY = Java would not verify (`hashAlgorithm != 0` or
/// no hash bytes).
pub sha256: Vec<u8>,
}
/// Parse the JSON array `[{relPath, url, size, sha256hex}]`. `relPath` is normalised the same
/// way as `ManifestFile.unixPath()` so a caller that forgot is still byte-identical.
pub fn parse_plan(json: &str) -> Result<Vec<PlanEntry>, String> {
let value: serde_json::Value =
serde_json::from_str(json).map_err(|err| format!("plan json: {err}"))?;
let Some(array) = value.as_array() else {
return Err("plan json: expected an array".to_string());
};
let mut entries = Vec::with_capacity(array.len());
for (index, item) in array.iter().enumerate() {
let rel_path = item
.get("relPath")
.and_then(|v| v.as_str())
.map(unix_path)
.unwrap_or_default();
if rel_path.is_empty() {
return Err(format!("plan json: entry {index} has no relPath"));
}
let url = item
.get("url")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
if url.is_empty() {
return Err(format!("plan json: entry {index} ({rel_path}) has no url"));
}
let size = item
.get("size")
.and_then(|v| v.as_u64().or_else(|| v.as_i64().map(|s| s.max(0) as u64)))
.unwrap_or(0);
let sha_hex = item
.get("sha256hex")
.and_then(|v| v.as_str())
.unwrap_or_default();
let sha256 = if sha_hex.is_empty() {
Vec::new()
} else {
hex_decode(sha_hex)
.ok_or_else(|| format!("plan json: entry {index} ({rel_path}) bad sha256hex"))?
};
entries.push(PlanEntry {
rel_path,
url,
size,
sha256,
});
}
Ok(entries)
}
/// `ManifestFile.unixPath()`.
pub fn unix_path(path: &str) -> String {
path.replace('\\', "/")
}
/// Lowercase/uppercase hex → bytes; `None` on odd length or a non-hex digit.
pub fn hex_decode(hex: &str) -> Option<Vec<u8>> {
let bytes = hex.as_bytes();
if bytes.len() % 2 != 0 {
return None;
}
let nibble = |b: u8| -> Option<u8> {
match b {
b'0'..=b'9' => Some(b - b'0'),
b'a'..=b'f' => Some(b - b'a' + 10),
b'A'..=b'F' => Some(b - b'A' + 10),
_ => None,
}
};
let mut out = Vec::with_capacity(bytes.len() / 2);
for pair in bytes.chunks(2) {
out.push((nibble(pair[0])? << 4) | nibble(pair[1])?);
}
Some(out)
}
/// `new File(installDir, rel)` — Java resolves the child textually under the parent, so
/// build the path by concatenation (never `Path::join`, which would let a leading `/` escape).
pub fn dest_path(install_dir: &str, rel_path: &str) -> PathBuf {
let base = install_dir.trim_end_matches('/');
PathBuf::from(format!("{base}/{rel_path}"))
}
/// `new File(installDir, rel + ".tmp")`.
pub fn tmp_path(install_dir: &str, rel_path: &str) -> PathBuf {
dest_path(install_dir, &format!("{rel_path}{TMP_SUFFIX}"))
}
/// Java `:250`: `destFile.exists() && destFile.length() == file.size`. `File.length()` is
/// `st_size`, which is what `Metadata::len()` returns — including for a directory, so the
/// (degenerate) behaviours match too. No hash is checked on skip, on purpose.
pub fn is_present_and_complete(install_dir: &str, entry: &PlanEntry) -> bool {
fs::metadata(dest_path(install_dir, &entry.rel_path))
.map(|meta| meta.len() == entry.size)
.unwrap_or(false)
}
/// `scheme://host[:port]` of a URL — the per-host cap key. Amazon serves every file from the
/// one signed CDN base, so this yields a single host.
pub fn host_key(url: &str) -> String {
let (scheme, rest) = match url.find("://") {
Some(idx) => (&url[..idx], &url[idx + 3..]),
None => ("https", url),
};
let end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
format!("{scheme}://{}", &rest[..end])
}
/// Improvements round 1: `max(6, ceil(max_workers / distinct_hosts))` — a single-host store
/// (Amazon's one signed a2z base) gets the whole ceiling on its one host; the core clamps the
/// window to `hosts × per_host_cap`, so anything smaller would silently cut the ceiling.
pub fn per_host_cap_for(max_workers: usize, distinct_hosts: usize) -> usize {
let hosts = distinct_hosts.max(1);
let per_host = max_workers.max(1).div_ceil(hosts);
per_host.max(crate::fetch_tuning::PER_HOST_CAP)
}
/// The resolved work for one run.
#[derive(Debug, Default)]
pub struct Plan {
/// Items to fetch (index into `entries` via `FetchItem.id`).
pub items: Vec<FetchItem>,
/// Every manifest file, in manifest order (fetched and skipped alike).
pub entries: Vec<PlanEntry>,
/// Σ size of all entries = `ParsedManifest.totalInstallSize`.
pub total_bytes: u64,
/// Bytes credited up front for files that passed the resume check.
pub skipped_bytes: u64,
/// Number of files that passed the resume check.
pub skipped_files: u64,
/// Distinct host keys in fetch order.
pub hosts: Vec<String>,
}
/// Apply the Java resume-skip rule to every entry and turn the rest into whole-file items.
pub fn build_plan(entries: Vec<PlanEntry>, install_dir: &str) -> Plan {
let mut plan = Plan {
total_bytes: entries.iter().map(|e| e.size).sum(),
..Plan::default()
};
for (index, entry) in entries.iter().enumerate() {
if is_present_and_complete(install_dir, entry) {
plan.skipped_bytes = plan.skipped_bytes.saturating_add(entry.size);
plan.skipped_files += 1;
continue;
}
let host = host_key(&entry.url);
if !plan.hosts.contains(&host) {
plan.hosts.push(host);
}
plan.items.push(FetchItem {
id: index as u64,
urls: vec![entry.url.clone()],
reserve: entry.size,
range: None,
});
}
if plan.hosts.is_empty() {
plan.hosts.push("https://amazon".to_string());
}
plan.entries = entries;
plan
}
/// One `<dest>.tmp` being streamed: the open handle, the running hash and the bytes written
/// by the current attempt.
struct OpenTmp {
file: File,
hasher: Sha256,
written: u64,
}
/// Sink: stream pieces into `<dest>.tmp` (SHA-256 accumulated on the fly), verify and rename
/// in `on_finish`. `process` (whole-body mode) does the same in one step.
pub struct AmazonSink {
install_dir: String,
entries: Vec<PlanEntry>,
cancel: Arc<AtomicBool>,
files_done: AtomicU64,
/// Per-item open tmp files. The map lock is held only to look the entry up; the write
/// itself happens under the per-item lock so 8 streams never serialise on one mutex.
open: Mutex<HashMap<u64, Arc<Mutex<OpenTmp>>>>,
}
impl AmazonSink {
pub fn new(install_dir: &str, entries: Vec<PlanEntry>, cancel: Arc<AtomicBool>) -> Self {
Self {
install_dir: install_dir.to_string(),
entries,
cancel,
files_done: AtomicU64::new(0),
open: Mutex::new(HashMap::new()),
}
}
pub fn files_done(&self) -> u64 {
self.files_done.load(Ordering::Relaxed)
}
fn entry(&self, item: &FetchItem) -> Result<&PlanEntry, SinkError> {
self.entries
.get(item.id as usize)
.ok_or_else(|| SinkError::Fatal(format!("plan index {} out of range", item.id)))
}
/// Java `:260` — `destFile.getParentFile().mkdirs()` before each attempt; a failure
/// surfaces on the create/write below, like Java's.
fn ensure_parent(&self, entry: &PlanEntry) {
if let Some(parent) = dest_path(&self.install_dir, &entry.rel_path).parent() {
let _ = fs::create_dir_all(parent);
}
}
/// Java `:280-284`: delete an existing dest, rename tmp → dest; rename failure fails the
/// file with NO retry (→ whole install fails).
fn rename_into_place(&self, entry: &PlanEntry) -> Result<(), SinkError> {
let dest = dest_path(&self.install_dir, &entry.rel_path);
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
if dest.exists() {
let _ = fs::remove_file(&dest);
}
if let Err(err) = fs::rename(&tmp, &dest) {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Fatal(format!(
"Failed to rename tmp → {} ({err})",
dest.display()
)));
}
self.files_done.fetch_add(1, Ordering::Relaxed);
Ok(())
}
/// Java `:267-277`: SHA-256 check when the manifest carries a hash; a mismatch deletes the
/// tmp and is retried.
fn verify(&self, entry: &PlanEntry, digest: &[u8]) -> Result<(), SinkError> {
if !entry.sha256.is_empty() && digest != entry.sha256.as_slice() {
let _ = fs::remove_file(tmp_path(&self.install_dir, &entry.rel_path));
return Err(SinkError::Retry(format!(
"SHA-256 mismatch for: {}",
entry.rel_path
)));
}
Ok(())
}
/// Whole-body path (`FetchOptions.stream = false`): verify → write tmp → rename.
pub fn commit(&self, entry: &PlanEntry, body: &[u8]) -> Result<u64, SinkError> {
if !entry.sha256.is_empty() {
let digest = Sha256::digest(body);
if digest.as_slice() != entry.sha256.as_slice() {
return Err(SinkError::Retry(format!(
"SHA-256 mismatch for: {}",
entry.rel_path
)));
}
}
self.ensure_parent(entry);
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
if let Err(err) = fs::write(&tmp, body) {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!("write {}: {err}", tmp.display())));
}
self.rename_into_place(entry)?;
Ok(body.len() as u64)
}
/// After the run: close and delete every tmp that never reached `on_finish` (a stream that
/// failed after max attempts, or was cut by cancel / a fatal error). Java deletes the tmp
/// of every failed or cancelled attempt (`:271`, `:288`, `:293`, `:297`), so no `.tmp`
/// may survive a run on either engine. Returns how many were removed.
pub fn cleanup_partials(&self) -> usize {
let leftovers: Vec<(u64, Arc<Mutex<OpenTmp>>)> = match self.open.lock() {
Ok(mut open) => open.drain().collect(),
Err(_) => return 0,
};
let mut removed = 0;
for (id, open) in leftovers {
drop(open);
if let Some(entry) = self.entries.get(id as usize) {
if fs::remove_file(tmp_path(&self.install_dir, &entry.rel_path)).is_ok() {
removed += 1;
}
}
}
removed
}
}
impl FetchSink for AmazonSink {
fn process(&self, item: &FetchItem, body: Vec<u8>) -> Result<u64, SinkError> {
if self.cancel.load(Ordering::Relaxed) {
return Err(SinkError::Fatal("cancelled".to_string()));
}
let entry = self.entry(item)?;
self.commit(entry, &body)
}
fn on_chunk(&self, item: &FetchItem, offset: u64, data: &[u8]) -> Result<(), SinkError> {
if self.cancel.load(Ordering::Relaxed) {
// Java `:342-344`: cancel inside the read loop → tmp deleted (cleanup_partials).
return Err(SinkError::Fatal("cancelled".to_string()));
}
let entry = self.entry(item)?;
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
let open = if offset == 0 {
// (Re)start of an attempt: Java opens a fresh FileOutputStream (truncating).
self.ensure_parent(entry);
let file = File::create(&tmp).map_err(|err| {
let _ = fs::remove_file(&tmp);
SinkError::Retry(format!("create {}: {err}", tmp.display()))
})?;
let open = Arc::new(Mutex::new(OpenTmp {
file,
hasher: Sha256::new(),
written: 0,
}));
match self.open.lock() {
Ok(mut map) => {
map.insert(item.id, Arc::clone(&open));
}
Err(_) => return Err(SinkError::Fatal("sink map poisoned".to_string())),
}
open
} else {
let found = match self.open.lock() {
Ok(map) => map.get(&item.id).cloned(),
Err(_) => return Err(SinkError::Fatal("sink map poisoned".to_string())),
};
match found {
Some(open) => open,
None => {
return Err(SinkError::Retry(format!(
"piece at offset {offset} without a stream start for: {}",
entry.rel_path
)))
}
}
};
let Ok(mut state) = open.lock() else {
return Err(SinkError::Fatal("sink item poisoned".to_string()));
};
if state.written != offset {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!(
"piece out of order ({} != {offset}) for: {}",
state.written, entry.rel_path
)));
}
if let Err(err) = state.file.write_all(data) {
// Java: IOException inside downloadFile → tmp deleted → next attempt.
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!("write {}: {err}", tmp.display())));
}
state.hasher.update(data);
state.written = state.written.saturating_add(data.len() as u64);
Ok(())
}
fn on_finish(&self, item: &FetchItem, total_len: u64) -> Result<u64, SinkError> {
let entry = self.entry(item)?;
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
let taken = match self.open.lock() {
Ok(mut map) => map.remove(&item.id),
Err(_) => return Err(SinkError::Fatal("sink map poisoned".to_string())),
};
let (written, digest) = match taken {
Some(open) => {
let Ok(mut state) = open.lock() else {
return Err(SinkError::Fatal("sink item poisoned".to_string()));
};
if let Err(err) = state.file.flush() {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!("flush {}: {err}", tmp.display())));
}
let written = state.written;
let digest = std::mem::replace(&mut state.hasher, Sha256::new()).finalize();
(written, digest)
}
None => {
// Zero-byte body: no piece ever arrived. Java would have created an empty tmp.
if total_len != 0 {
return Err(SinkError::Retry(format!(
"finish without a stream for: {}",
entry.rel_path
)));
}
self.ensure_parent(entry);
if let Err(err) = File::create(&tmp) {
return Err(SinkError::Retry(format!("create {}: {err}", tmp.display())));
}
(0, Sha256::new().finalize())
}
};
if written != total_len {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!(
"stream length {written} != {total_len} for: {}",
entry.rel_path
)));
}
self.verify(entry, digest.as_slice())?;
self.rename_into_place(entry)?;
Ok(0)
}
}
/// Outcome of one `run_download`.
#[derive(Clone, Debug, Default)]
pub struct RunResult {
pub success: bool,
pub cancelled: bool,
pub error: String,
/// Bytes fetched by this run (skipped files excluded; failed attempts' pieces included,
/// like Java's `totalDownloaded`).
pub bytes_written: u64,
/// Files committed by this run plus files skipped by the resume check.
pub files_done: u64,
pub files_total: u64,
}
/// Throughput bookkeeping for the final summary line (avg over the run, peak over ≥500 ms
/// progress intervals — the same sampling window the Java speed label uses).
struct SpeedMeter {
started: Instant,
last_at: Instant,
last_bytes: u64,
peak_bps: f64,
}
impl SpeedMeter {
fn new() -> Self {
let now = Instant::now();
Self {
started: now,
last_at: now,
last_bytes: 0,
peak_bps: 0.0,
}
}
fn observe(&mut self, bytes: u64) {
let now = Instant::now();
let dt = now.duration_since(self.last_at).as_secs_f64();
if dt >= 0.5 {
let bps = bytes.saturating_sub(self.last_bytes) as f64 / dt;
if bps > self.peak_bps {
self.peak_bps = bps;
}
self.last_at = now;
self.last_bytes = bytes;
}
}
fn summary(&self, bytes: u64) -> (f64, f64, f64) {
let elapsed = self.started.elapsed().as_secs_f64().max(0.001);
let avg = bytes as f64 / elapsed;
(elapsed, avg * 8.0 / 1_000_000.0, self.peak_bps * 8.0 / 1_000_000.0)
}
}
/// Blocking: plan → stream-fetch → verify/rename, calling `progress(bytes_done, bytes_total,
/// files_done, files_total)` after each written piece and each finished file (skipped files
/// are pre-credited) and `log` for every diagnostic line. `bytes_done` includes skipped bytes,
/// matching Java's `totalDownloaded` (which credits skipped files at `:251`).
pub fn run_download(
plan_json: &str,
install_dir: &str,
ca_bundle_path: &str,
max_workers: usize,
process_workers: usize,
cancel: Arc<AtomicBool>,
progress: &(dyn Fn(u64, u64, u64, u64) + Sync),
log: &(dyn Fn(&str) + Sync),
) -> RunResult {
let max_workers = if max_workers == 0 { MAX_PARALLEL } else { max_workers };
let process_workers = process_workers.max(1);
let entries = match parse_plan(plan_json) {
Ok(entries) => entries,
Err(err) => {
log(&format!("engine=rust plan error: {err}"));
return RunResult {
error: err,
..RunResult::default()
};
}
};
let plan = build_plan(entries, install_dir);
let files_total = plan.entries.len() as u64;
let fetch_bytes: u64 = plan.items.iter().map(|item| item.reserve).sum();
let per_host_cap = per_host_cap_for(max_workers, plan.hosts.len());
log(&format!(
"engine=rust mode=stream plan={} files skip={} ({} bytes) fetch={} ({} bytes) hosts={} workers={} per_host_cap={} process={} dir={} {}",
files_total,
plan.skipped_files,
plan.skipped_bytes,
plan.items.len(),
fetch_bytes,
plan.hosts.len(),
max_workers,
per_host_cap,
process_workers,
install_dir,
crate::priority::log_field()
));
progress(plan.skipped_bytes, plan.total_bytes, plan.skipped_files, files_total);
if plan.items.is_empty() {
log("summary bytes=0 elapsed=0.000 avg_mbps=0.0 peak_mbps=0.0 files=all-present");
return RunResult {
success: true,
files_done: plan.skipped_files,
files_total,
..RunResult::default()
};
}
if cancel.load(Ordering::Relaxed) {
return RunResult {
cancelled: true,
error: "cancelled".to_string(),
files_done: plan.skipped_files,
files_total,
..RunResult::default()
};
}
let opts = FetchOptions {
max_workers,
// Every Amazon file comes from the one signed CDN base: give that host the whole
// ceiling (`per_host_cap_for`), or the core would clamp the window to hosts × cap.
per_host_cap,
timeout: REQUEST_TIMEOUT,
headers: vec![("User-Agent".to_string(), DOWNLOAD_USER_AGENT.to_string())],
ca_bundle_path: ca_bundle_path.to_string(),
process_workers,
label: LABEL.to_string(),
// Whole files, streamed into the tmp as they arrive: 8 wide like Java's pool, memory
// bounded to the pieces not yet written.
stream: true,
..FetchOptions::default()
};
let sink = AmazonSink::new(install_dir, plan.entries.clone(), Arc::clone(&cancel));
let meter = Mutex::new(SpeedMeter::new());
let skipped_bytes = plan.skipped_bytes;
let skipped_files = plan.skipped_files;
let total_bytes = plan.total_bytes;
let progress_cb = |credited: u64, items_ok: u64| {
if let Ok(mut meter) = meter.lock() {
meter.observe(credited);
}
progress(
skipped_bytes.saturating_add(credited),
total_bytes,
skipped_files.saturating_add(items_ok),
files_total,
);
};
let outcome = run_fetch(
plan.items,
&plan.hosts,
&opts,
&sink,
cancel.as_ref(),
&progress_cb,
log,
);
let leftovers = sink.cleanup_partials();
let (elapsed, avg_mbps, peak_mbps) = meter
.lock()
.map(|meter| meter.summary(outcome.bytes_credited))
.unwrap_or((0.0, 0.0, 0.0));
let files_done = skipped_files.saturating_add(sink.files_done());
log(&format!(
"summary bytes={} elapsed={:.3} avg_mbps={:.1} peak_mbps={:.1} files={}/{} items_ok={} tmp_removed={} cancelled={} error={}",
outcome.bytes_credited,
elapsed,
avg_mbps,
peak_mbps,
files_done,
files_total,
outcome.items_ok,
leftovers,
outcome.cancelled,
outcome.error.as_deref().unwrap_or("")
));
let cancelled = outcome.cancelled || cancel.load(Ordering::Relaxed);
let error = if cancelled {
outcome.error.unwrap_or_else(|| "cancelled".to_string())
} else {
outcome.error.unwrap_or_default()
};
RunResult {
success: !cancelled && error.is_empty(),
cancelled,
error,
bytes_written: outcome.bytes_credited,
files_done,
files_total,
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::AtomicUsize;
static COUNTER: AtomicUsize = AtomicUsize::new(0);
fn scratch_dir() -> String {
let n = COUNTER.fetch_add(1, Ordering::Relaxed);
let dir = std::env::temp_dir().join(format!(
"amazon-test-{}-{}",
std::process::id(),
n
));
let _ = fs::remove_dir_all(&dir);
fs::create_dir_all(&dir).unwrap();
dir.to_string_lossy().into_owned()
}
fn sha_hex(data: &[u8]) -> String {
crate::gog::to_hex(Sha256::digest(data).as_slice())
}
fn item_for(entry: &PlanEntry, id: u64) -> FetchItem {
FetchItem {
id,
urls: vec![entry.url.clone()],
reserve: entry.size,
range: None,
}
}
const PLAN: &str = r#"[
{"relPath":"Binaries\\Win64\\Game.exe","url":"https://d1.cdn.example/base/files/aa?Sig=1","size":4,"sha256hex":"9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"},
{"relPath":"data/level1.pak","url":"https://d1.cdn.example/base/files/bb?Sig=1","size":2,"sha256hex":""},
{"relPath":"readme.txt","url":"https://d2.cdn.example/base/files/cc?Sig=1","size":0,"sha256hex":""}
]"#;
#[test]
fn parses_plan_and_normalises_paths() {
let entries = parse_plan(PLAN).unwrap();
assert_eq!(entries.len(), 3);
assert_eq!(entries[0].rel_path, "Binaries/Win64/Game.exe");
assert_eq!(entries[0].size, 4);
assert_eq!(entries[0].sha256.len(), 32);
assert!(entries[1].sha256.is_empty());
assert_eq!(entries[2].size, 0);
}
#[test]
fn rejects_bad_plan() {
assert!(parse_plan("{}").is_err());
assert!(parse_plan(r#"[{"url":"x","size":1}]"#).is_err());
assert!(parse_plan(r#"[{"relPath":"a","size":1}]"#).is_err());
assert!(parse_plan(r#"[{"relPath":"a","url":"u","size":1,"sha256hex":"zz"}]"#).is_err());
}
#[test]
fn hex_roundtrip() {
assert_eq!(hex_decode("00ff10"), Some(vec![0x00, 0xff, 0x10]));
assert_eq!(hex_decode("00FF"), Some(vec![0x00, 0xff]));
assert_eq!(hex_decode("0"), None);
assert_eq!(hex_decode("0g"), None);
}
#[test]
fn per_host_cap_gives_single_host_the_whole_ceiling() {
assert_eq!(per_host_cap_for(32, 1), 32);
assert_eq!(per_host_cap_for(32, 3), 11);
assert_eq!(per_host_cap_for(8, 1), 8);
assert_eq!(per_host_cap_for(4, 1), 6);
assert_eq!(per_host_cap_for(0, 0), 6);
}
#[test]
fn host_key_strips_path_and_query() {
assert_eq!(
host_key("https://d1.cdn.example/base/files/aa?Sig=1"),
"https://d1.cdn.example"
);
assert_eq!(host_key("http://h:8080/x"), "http://h:8080");
assert_eq!(host_key("nohost"), "https://nohost");
}
#[test]
fn dest_path_never_escapes_install_dir() {
assert_eq!(
dest_path("/inst/", "a/b.txt"),
PathBuf::from("/inst/a/b.txt")
);
assert_eq!(dest_path("/inst", "/abs.txt"), PathBuf::from("/inst//abs.txt"));
assert_eq!(tmp_path("/inst", "a.bin"), PathBuf::from("/inst/a.bin.tmp"));
}
#[test]
fn build_plan_skips_only_size_matched_files() {
let dir = scratch_dir();
let entries = parse_plan(PLAN).unwrap();
// Entry 1 present with the right size → skipped (no hash check even if wrong bytes).
fs::create_dir_all(format!("{dir}/data")).unwrap();
fs::write(format!("{dir}/data/level1.pak"), b"zz").unwrap();
// Entry 0 present with the wrong size → fetched.
fs::create_dir_all(format!("{dir}/Binaries/Win64")).unwrap();
fs::write(format!("{dir}/Binaries/Win64/Game.exe"), b"tes").unwrap();
let plan = build_plan(entries, &dir);
assert_eq!(plan.total_bytes, 6);
assert_eq!(plan.skipped_files, 1);
assert_eq!(plan.skipped_bytes, 2);
assert_eq!(plan.items.len(), 2);
assert_eq!(plan.items[0].id, 0);
assert_eq!(plan.items[0].reserve, 4);
assert!(plan.items[0].range.is_none());
assert_eq!(plan.items[1].id, 2);
assert_eq!(
plan.hosts,
vec!["https://d1.cdn.example".to_string(), "https://d2.cdn.example".to_string()]
);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn zero_size_missing_file_is_fetched_and_present_zero_is_skipped() {
let dir = scratch_dir();
let entry = PlanEntry {
rel_path: "empty.txt".into(),
url: "https://h/x".into(),
size: 0,
sha256: Vec::new(),
};
assert!(!is_present_and_complete(&dir, &entry));
fs::write(format!("{dir}/empty.txt"), b"").unwrap();
assert!(is_present_and_complete(&dir, &entry));
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn whole_body_sink_verifies_writes_and_renames() {
let dir = scratch_dir();
let body = b"test".to_vec();
let entry = PlanEntry {
rel_path: "sub/Game.exe".into(),
url: "https://h/x".into(),
size: 4,
sha256: hex_decode(&sha_hex(&body)).unwrap(),
};
let cancel = Arc::new(AtomicBool::new(false));
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::clone(&cancel));
let item = item_for(&entry, 0);
// Mismatch → Retry, nothing on disk.
match sink.process(&item, b"nope".to_vec()) {
Err(SinkError::Retry(msg)) => assert!(msg.contains("SHA-256 mismatch")),
Err(SinkError::Fatal(msg)) => panic!("expected Retry, got Fatal({msg})"),
Ok(_) => panic!("expected Retry, got Ok"),
}
assert!(!dest_path(&dir, "sub/Game.exe").exists());
assert!(!tmp_path(&dir, "sub/Game.exe").exists());
// Match → written, renamed, tmp gone, credited body length.
assert_eq!(sink.process(&item, body.clone()).unwrap(), 4);
assert_eq!(fs::read(dest_path(&dir, "sub/Game.exe")).unwrap(), body);
assert!(!tmp_path(&dir, "sub/Game.exe").exists());
assert_eq!(sink.files_done(), 1);
// Existing dest is replaced (Java `:280`).
assert_eq!(sink.process(&item, body.clone()).unwrap(), 4);
// Cancel → Fatal, no write.
cancel.store(true, Ordering::Relaxed);
assert!(matches!(
sink.process(&item, body),
Err(SinkError::Fatal(_))
));
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_appends_verifies_and_renames() {
let dir = scratch_dir();
let body = b"hello world".to_vec();
let entry = PlanEntry {
rel_path: "sub/big.bin".into(),
url: "https://h/x".into(),
size: body.len() as u64,
sha256: hex_decode(&sha_hex(&body)).unwrap(),
};
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::new(AtomicBool::new(false)));
let item = item_for(&entry, 0);
sink.on_chunk(&item, 0, b"hello").unwrap();
assert!(tmp_path(&dir, "sub/big.bin").exists());
sink.on_chunk(&item, 5, b" wor").unwrap();
sink.on_chunk(&item, 9, b"ld").unwrap();
assert_eq!(sink.on_finish(&item, 11).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "sub/big.bin")).unwrap(), body);
assert!(!tmp_path(&dir, "sub/big.bin").exists());
assert_eq!(sink.files_done(), 1);
assert_eq!(sink.cleanup_partials(), 0);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_restart_truncates_and_mismatch_retries() {
let dir = scratch_dir();
let body = b"abcdef".to_vec();
let entry = PlanEntry {
rel_path: "f.bin".into(),
url: "https://h/x".into(),
size: 6,
sha256: hex_decode(&sha_hex(&body)).unwrap(),
};
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::new(AtomicBool::new(false)));
let item = item_for(&entry, 0);
// Attempt 1 delivers garbage, then the core restarts at offset 0.
sink.on_chunk(&item, 0, b"zzzz").unwrap();
sink.on_chunk(&item, 0, b"abc").unwrap();
sink.on_chunk(&item, 3, b"xyz").unwrap();
match sink.on_finish(&item, 6) {
Err(SinkError::Retry(msg)) => assert!(msg.contains("SHA-256 mismatch")),
Err(SinkError::Fatal(msg)) => panic!("expected Retry, got Fatal({msg})"),
Ok(_) => panic!("expected Retry, got Ok"),
}
assert!(!tmp_path(&dir, "f.bin").exists());
assert!(!dest_path(&dir, "f.bin").exists());
// Out-of-order piece → Retry.
sink.on_chunk(&item, 0, b"abc").unwrap();
assert!(matches!(
sink.on_chunk(&item, 5, b"f"),
Err(SinkError::Retry(_))
));
// Length mismatch at finish → Retry.
sink.on_chunk(&item, 0, b"abc").unwrap();
assert!(matches!(sink.on_finish(&item, 6), Err(SinkError::Retry(_))));
// Piece without a start → Retry.
assert!(matches!(
sink.on_chunk(&item, 3, b"def"),
Err(SinkError::Retry(_))
));
// Good attempt.
sink.on_chunk(&item, 0, b"abc").unwrap();
sink.on_chunk(&item, 3, b"def").unwrap();
assert_eq!(sink.on_finish(&item, 6).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "f.bin")).unwrap(), body);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_cleanup_removes_unfinished_tmp_and_zero_len_finishes() {
let dir = scratch_dir();
let big = PlanEntry {
rel_path: "cut.bin".into(),
url: "https://h/x".into(),
size: 10,
sha256: Vec::new(),
};
let empty = PlanEntry {
rel_path: "empty.bin".into(),
url: "https://h/y".into(),
size: 0,
sha256: Vec::new(),
};
let cancel = Arc::new(AtomicBool::new(false));
let sink = AmazonSink::new(&dir, vec![big.clone(), empty.clone()], Arc::clone(&cancel));
sink.on_chunk(&item_for(&big, 0), 0, b"12345").unwrap();
assert!(tmp_path(&dir, "cut.bin").exists());
// Zero-byte body: on_finish with no pieces creates and commits an empty file.
assert_eq!(sink.on_finish(&item_for(&empty, 1), 0).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "empty.bin")).unwrap(), b"");
// Cancel mid-stream → Fatal; cleanup deletes the partial tmp like Java does.
cancel.store(true, Ordering::Relaxed);
assert!(matches!(
sink.on_chunk(&item_for(&big, 0), 5, b"6"),
Err(SinkError::Fatal(_))
));
assert_eq!(sink.cleanup_partials(), 1);
assert!(!tmp_path(&dir, "cut.bin").exists());
assert!(!dest_path(&dir, "cut.bin").exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_skips_hash_when_manifest_has_none() {
let dir = scratch_dir();
let entry = PlanEntry {
rel_path: "a.bin".into(),
url: "https://h/x".into(),
size: 3,
sha256: Vec::new(),
};
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::new(AtomicBool::new(false)));
let item = item_for(&entry, 0);
sink.on_chunk(&item, 0, b"abc").unwrap();
assert_eq!(sink.on_finish(&item, 3).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "a.bin")).unwrap(), b"abc");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn run_download_all_present_short_circuits() {
let dir = scratch_dir();
fs::write(format!("{dir}/a.bin"), b"abc").unwrap();
let plan = r#"[{"relPath":"a.bin","url":"https://h/x","size":3,"sha256hex":""}]"#;
let seen = Mutex::new(Vec::new());
let logs = Mutex::new(Vec::new());
let result = run_download(
plan,
&dir,
"",
0,
1,
Arc::new(AtomicBool::new(false)),
&|done, total, fd, ft| seen.lock().unwrap().push((done, total, fd, ft)),
&|line| logs.lock().unwrap().push(line.to_string()),
);
assert!(result.success);
assert_eq!(result.files_done, 1);
assert_eq!(result.files_total, 1);
assert_eq!(seen.lock().unwrap().as_slice(), &[(3, 3, 1, 1)]);
assert!(logs.lock().unwrap()[0].starts_with("engine=rust mode=stream"));
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn run_download_reports_plan_error() {
let result = run_download(
"not json",
"/nonexistent",
"",
8,
2,
Arc::new(AtomicBool::new(false)),
&|_, _, _, _| {},
&|_| {},
);
assert!(!result.success);
assert!(result.error.contains("plan json"));
}
}
+229
View File
@@ -0,0 +1,229 @@
//! JNI facade for the Amazon download adapter.
//!
//! Kotlin side: `com.droiddeck.launcher.stores.amazon.AmazonNative` (`nativeStart` /
//! `nativeCancel` / `nativeRelease`) with an `AmazonNative.Listener`
//! (`onProgress(JJJJ)V`, `onLog(String)V`, `onComplete(ZString;J)V`). Mirrors the shape of
//! the GOG and Epic facades: the run happens on a spawned native
//! thread, callbacks attach the JavaVM as daemon threads and hold a `GlobalRef` to the
//! listener; cancel flips an `AtomicBool`. This module keeps its own `JavaVM` handle (taken
//! from the calling `JNIEnv` on first use) so it depends on nothing outside this module.
use crate::amazon::{run_download, MAX_PARALLEL};
use jni::objects::{GlobalRef, JClass, JObject, JString, JValue};
use jni::sys::{jint, jlong, JNI_FALSE, JNI_TRUE};
use jni::{JNIEnv, JavaVM};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
use std::thread;
/// Logcat tag (`adb logcat -s AmazonNative`). Every engine line reaches logcat exactly ONCE:
/// this module only forwards lines to the listener (`onLog`), and the Java manager is the one
/// place that writes them to `android.util.Log` + its debug file. (Round 1 fix: the native side
/// used to also call `__android_log_write`, so each line appeared twice.)
pub const LOG_TAG: &str = "AmazonNative";
static JVM: OnceLock<JavaVM> = OnceLock::new();
fn clear_pending_exception(env: &mut JNIEnv) {
if env.exception_check().unwrap_or(false) {
let _ = env.exception_clear();
}
}
fn jstring_to_string(env: &mut JNIEnv, value: &JString) -> Option<String> {
env.get_string(value)
.ok()
.map(|s| s.to_string_lossy().into_owned())
}
/// One live run; boxed and handed to Kotlin as an opaque `jlong`.
struct AmazonRun {
cancel: Arc<AtomicBool>,
}
fn to_handle(run: Box<AmazonRun>) -> jlong {
Box::into_raw(run) as jlong
}
unsafe fn from_handle<'a>(handle: jlong) -> Option<&'a AmazonRun> {
if handle == 0 {
None
} else {
Some(&*(handle as *const AmazonRun))
}
}
fn call_log(env: &mut JNIEnv, listener: &JObject, line: &str) {
let Ok(text) = env.new_string(line) else {
clear_pending_exception(env);
return;
};
let text_obj = JObject::from(text);
let _ = env.call_method(
listener,
"onLog",
"(Ljava/lang/String;)V",
&[JValue::Object(&text_obj)],
);
clear_pending_exception(env);
}
fn call_progress(
env: &mut JNIEnv,
listener: &JObject,
bytes_done: u64,
bytes_total: u64,
files_done: u64,
files_total: u64,
) {
let _ = env.call_method(
listener,
"onProgress",
"(JJJJ)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
JValue::Long(files_done as jlong),
JValue::Long(files_total as jlong),
],
);
clear_pending_exception(env);
}
fn call_complete(
env: &mut JNIEnv,
listener: &JObject,
success: bool,
error: &str,
bytes_written: u64,
) {
let Ok(error) = env.new_string(error) else {
clear_pending_exception(env);
return;
};
let error_obj = JObject::from(error);
let _ = env.call_method(
listener,
"onComplete",
"(ZLjava/lang/String;J)V",
&[
JValue::Bool(if success { JNI_TRUE } else { JNI_FALSE }),
JValue::Object(&error_obj),
JValue::Long(bytes_written as jlong),
],
);
clear_pending_exception(env);
}
/// Attach the current (native) thread and run `f` with the listener object.
fn with_attached_env(listener: &GlobalRef, f: impl FnOnce(&mut JNIEnv, &JObject)) {
let Some(vm) = JVM.get() else {
return;
};
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
f(&mut env, listener.as_obj());
}
/// `AmazonNative.nativeStart(planJson, installDir, caBundlePath, maxWorkers,
/// processWorkers, listener) -> handle`. Returns 0 (and fires `onComplete(false, …)`) when the
/// inputs are unusable. The run itself happens on a spawned thread.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeStart(
mut env: JNIEnv,
_class: JClass,
plan_json: JString,
install_dir: JString,
ca_bundle_path: JString,
max_workers: jint,
process_workers: jint,
listener: JObject,
) -> jlong {
if listener.is_null() {
return 0;
}
if JVM.get().is_none() {
if let Ok(vm) = env.get_java_vm() {
let _ = JVM.set(vm);
}
}
let plan_json = jstring_to_string(&mut env, &plan_json).unwrap_or_default();
let install_dir = jstring_to_string(&mut env, &install_dir).unwrap_or_default();
let ca_bundle_path = jstring_to_string(&mut env, &ca_bundle_path).unwrap_or_default();
if plan_json.is_empty() || install_dir.is_empty() {
call_complete(&mut env, &listener, false, "empty plan or install dir", 0);
return 0;
}
let Ok(listener) = env.new_global_ref(&listener) else {
clear_pending_exception(&mut env);
return 0;
};
let max_workers = if max_workers <= 0 {
MAX_PARALLEL
} else {
max_workers as usize
};
let process_workers = process_workers.max(1) as usize;
let cancel = Arc::new(AtomicBool::new(false));
let run = Box::new(AmazonRun {
cancel: Arc::clone(&cancel),
});
thread::spawn(move || {
crate::priority::background();
let log_listener = listener.clone();
let log = move |line: &str| {
with_attached_env(&log_listener, |env, obj| call_log(env, obj, line));
};
let progress_listener = listener.clone();
let progress = move |bytes_done: u64, bytes_total: u64, files_done: u64, files_total: u64| {
with_attached_env(&progress_listener, |env, obj| {
call_progress(env, obj, bytes_done, bytes_total, files_done, files_total)
});
};
let result = run_download(
&plan_json,
&install_dir,
&ca_bundle_path,
max_workers,
process_workers,
cancel,
&progress,
&log,
);
with_attached_env(&listener, |env, obj| {
call_complete(env, obj, result.success, &result.error, result.bytes_written)
});
});
to_handle(run)
}
/// `AmazonNative.nativeCancel(handle)` — flips the run's cancel flag; the core stops
/// issuing requests, in-flight bodies are dropped, `onComplete(false, "cancelled", …)` follows.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeCancel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if let Some(run) = unsafe { from_handle(handle) } {
run.cancel.store(true, Ordering::Relaxed);
}
}
/// `AmazonNative.nativeRelease(handle)` — frees the handle. Safe at any time: the worker
/// thread owns its own `Arc` of the cancel flag and the listener `GlobalRef`.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeRelease(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle != 0 {
unsafe {
drop(Box::from_raw(handle as *mut AmazonRun));
}
}
}
+46
View File
@@ -0,0 +1,46 @@
//! Epic Games Store download adapter for the shared fetch core (`crate::fetch_core`).
//!
//! Scope = the chunk-fetch inner loop of `EpicDownloadManager.java` (the fixed 8-thread pool that
//! fills `<installDir>/.chunks/<GUID>`), nothing more. The Java manager still parses the manifest
//! API JSON, downloads the manifest, selects files (install tags), runs the delta/verify pass,
//! assembles files from the chunk cache and does every post-install step. The adapter re-parses
//! the same manifest bytes, rebuilds the same chunk plan for the pending files Java hands it,
//! skips chunks already in the cache and writes verified chunks with the same `.part` + rename
//! protocol, so the on-disk state is identical whichever engine ran (see
//! Bannerlator's `docs/RUST_EPIC_PARITY.md` for the rule-by-rule table).
//!
//! Submodules:
//! - [`manifest`] — ChunksV4 binary manifest + legacy JSON manifest parsers (1:1 with the Java
//! `parseManifest` / `parseJsonManifest`).
//! - [`plan`] — install-tag selection, unique-chunk plan, chunk paths / CDN URLs.
//! - [`chunk`] — chunk header parse, zlib inflate, SHA-1 verify, cache write.
//! - [`driver`] — plan → `FetchItem`s → `fetch_core::run_fetch` with the chunk-cache sink.
//! - [`assemble`] — the files from the cache, one sequential write each, SHA-1 checked as they
//! are written; with a scratch cache (one the caller placed apart from the install) each chunk
//! is deleted as soon as the last file that uses it is done.
//! - [`jni`] — `Java_com_droiddeck_launcher_stores_epic_EpicNative_native*` exports.
//!
//! The selective-install-tag, delta/resume and chunk verification rules mirrored here were ported
//! into the Java manager from GameNative (`service/epic/EpicDownloadManager.kt`,
//! `manifest/ManifestUtils.kt`), itself derived from Legendary; both are GPL-3.0 and this crate
//! carries the same licence (see `Cargo.toml`).
pub mod assemble;
pub mod chunk;
pub mod driver;
pub mod jni;
pub mod manifest;
pub mod plan;
/// User-Agent the Java manager sends on every chunk request (`EpicDownloadManager.UA`).
pub const USER_AGENT: &str =
"UELauncher/11.0.1-14907503+++Portal+Release-Live Windows/10.0.19041.1.256.64bit";
/// Name of the chunk cache directory under the install dir (`new File(installDir, ".chunks")`).
pub const CHUNK_CACHE_DIR: &str = ".chunks";
/// Fixed pool width of the Java chunk downloader (`Executors.newFixedThreadPool(8)`).
pub const JAVA_POOL_THREADS: usize = 8;
/// logcat tag for every engine line (the Kotlin manager mirrors it into its own debug log).
pub const LOG_TAG: &str = "EpicNative";
@@ -0,0 +1,525 @@
//! Files from the chunk cache: the last stage of an Epic install, as a native run.
//!
//! The Kotlin manager's own loop does this too (`EpicDownloadManager.install`, after the chunk
//! fetch): for every pending file, open it, and for every part read the cached chunk and write
//! `chunk[offset..offset+size]`. This module does the same thing, in the same order, with the
//! same wording on a missing chunk, plus what the Kotlin loop cannot do without re-reading the
//! finished file from a slow card:
//!
//! - the file's SHA-1 is computed from the bytes as they are written (no extra I/O) and compared
//! with the manifest's, when the manifest has one; a mismatch deletes the file and fails the
//! run, so a bad assembly can never pass the next delta pass by size alone;
//! - with a scratch cache — one the caller placed apart from the install directory — every chunk
//! is reference-counted across the pending files and deleted the moment the last file that
//! needs it is written and verified, so the cache never holds more than what is still owed.
//!
//! The cache directory itself goes at the end of a successful run, scratch or not, which is what
//! the Kotlin loop does with `<installDir>/.chunks`. Cancellation and failure leave the cache
//! exactly as it stands, so the next run's delta pass skips the finished files and the fetch
//! skips the chunks still cached — on either engine.
//!
//! Why the cache may live elsewhere at all: an install to an SD card wrote every byte twice to
//! the card (chunk cache, then assembled file). With the cache on internal storage the card sees
//! each byte once, and the fetchers stop waiting on the card's write speed.
use std::collections::HashMap;
use std::fs::{self, File};
use std::io::{BufWriter, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use sha1::{Digest, Sha1};
use super::manifest::{parse_manifest, FileInfo, Manifest};
use super::plan::resolve_cache_dir;
/// Everything the JNI layer hands over for one assembly run.
pub struct AssembleRequest {
pub manifest_bytes: Vec<u8>,
pub install_dir: String,
/// `""` = `<installDir>/.chunks` (no scratch semantics); anything else = scratch cache.
pub chunk_cache_dir: String,
/// Indices into `manifest.files` of the files to assemble, in order.
pub file_indices: Vec<usize>,
}
/// Terminal result of an assembly run.
#[derive(Clone, Debug, Default)]
pub struct AssembleOutcome {
pub success: bool,
pub cancelled: bool,
pub error: String,
/// Bytes written into assembled files this run.
pub bytes_written: u64,
pub files_done: u64,
pub files_total: u64,
pub bytes_total: u64,
/// Chunk files deleted early because no remaining file needed them (scratch cache only).
pub chunks_removed: u64,
}
/// The parsed plan: manifest, the files to write, and how many of them still need each chunk.
/// Built up front so a bad index fails before anything is written.
#[derive(Debug)]
pub struct AssemblePlan {
pub manifest: Manifest,
pub install_dir: PathBuf,
pub cache_dir: PathBuf,
pub scratch_cache: bool,
pub files: Vec<usize>,
pub bytes_total: u64,
/// GUID → number of files in `files` whose parts reference it (a file counts once per GUID,
/// however many of its parts use the chunk).
pub refs: HashMap<String, u32>,
}
/// Parse + plan. `Err` = nothing was written.
pub fn build_assemble_plan(req: &AssembleRequest) -> Result<AssemblePlan, String> {
if req.install_dir.is_empty() {
return Err("assemble: empty install dir".to_string());
}
let manifest = parse_manifest(&req.manifest_bytes).map_err(|e| format!("assemble: {e}"))?;
for &i in &req.file_indices {
if i >= manifest.files.len() {
return Err(format!(
"assemble: file index {i} out of range ({} files)",
manifest.files.len()
));
}
}
let refs = chunk_refs(&manifest, &req.file_indices);
let bytes_total = req
.file_indices
.iter()
.map(|&i| manifest.files[i].file_size())
.sum();
Ok(AssemblePlan {
install_dir: PathBuf::from(&req.install_dir),
cache_dir: resolve_cache_dir(&req.install_dir, &req.chunk_cache_dir),
scratch_cache: !req.chunk_cache_dir.is_empty(),
files: req.file_indices.clone(),
bytes_total,
refs,
manifest,
})
}
/// How many of `file_indices` reference each chunk GUID.
pub fn chunk_refs(m: &Manifest, file_indices: &[usize]) -> HashMap<String, u32> {
let mut refs: HashMap<String, u32> = HashMap::new();
for &fi in file_indices {
let Some(f) = m.files.get(fi) else {
continue;
};
for g in distinct_guids(f) {
*refs.entry(g).or_insert(0) += 1;
}
}
refs
}
/// The GUIDs a file's parts reference, each once, in first-seen order.
fn distinct_guids(f: &FileInfo) -> Vec<String> {
let mut out: Vec<String> = Vec::with_capacity(f.parts.len());
for p in &f.parts {
let g = p.guid_str();
if !out.contains(&g) {
out.push(g);
}
}
out
}
/// The manifest's relative path with Windows separators normalised, as the Kotlin loop does
/// (`filename.replace("\\", "/")`).
pub fn relative_path(f: &FileInfo) -> String {
f.filename.replace('\\', "/")
}
/// A manifest SHA-1 that is all zero means "none" (the Kotlin delta pass treats it the same way).
fn expected_sha1(f: &FileInfo) -> Option<&[u8; 20]> {
f.sha1.as_ref().filter(|h| h.iter().any(|&b| b != 0))
}
/// Chunk bytes, read whole, with the last chunk kept: consecutive parts of one file usually come
/// from the same chunk, and a file's parts are the only place a chunk is read.
struct ChunkReader {
cache_dir: PathBuf,
last: Option<(String, Vec<u8>)>,
}
impl ChunkReader {
fn read(&mut self, guid: &str, rel_path: &str) -> Result<&[u8], String> {
if self.last.as_ref().map(|(g, _)| g.as_str()) != Some(guid) {
let path = self.cache_dir.join(guid);
if !path.exists() {
return Err(format!("Missing chunk {guid} for {rel_path}"));
}
let data = fs::read(&path).map_err(|e| format!("read chunk {guid}: {e}"))?;
self.last = Some((guid.to_string(), data));
}
Ok(self.last.as_ref().map(|(_, d)| d.as_slice()).unwrap_or(&[]))
}
/// Drop the kept copy of a chunk that is about to be deleted.
fn forget(&mut self, guid: &str) {
if self.last.as_ref().map(|(g, _)| g.as_str()) == Some(guid) {
self.last = None;
}
}
}
/// Write one file from the cache. Returns the bytes written. The output is removed on any error,
/// so a file only ever exists complete (and, when the manifest hashes it, verified) or not at all.
fn write_file(reader: &mut ChunkReader, f: &FileInfo, out_path: &Path) -> Result<u64, String> {
let rel_path = relative_path(f);
if let Some(parent) = out_path.parent() {
fs::create_dir_all(parent).map_err(|e| format!("mkdirs {}: {e}", parent.display()))?;
}
let result = (|| {
let file = File::create(out_path).map_err(|e| format!("create {rel_path}: {e}"))?;
let mut out = BufWriter::with_capacity(1024 * 1024, file);
let mut sha = expected_sha1(f).map(|_| Sha1::new());
let mut written: u64 = 0;
for part in &f.parts {
let guid = part.guid_str();
let data = reader.read(&guid, &rel_path)?;
let start = usize::try_from(part.offset).map_err(|_| {
format!("Chunk {guid} part offset {} invalid for {rel_path}", part.offset)
})?;
let len = part.size as u32 as usize;
let end = start
.checked_add(len)
.filter(|&e| e <= data.len())
.ok_or_else(|| {
format!(
"Chunk {guid} too short for {rel_path}: need {start}+{len}, have {}",
data.len()
)
})?;
let slice = &data[start..end];
out.write_all(slice).map_err(|e| format!("write {rel_path}: {e}"))?;
if let Some(sha) = sha.as_mut() {
sha.update(slice);
}
written += len as u64;
}
out.flush().map_err(|e| format!("flush {rel_path}: {e}"))?;
drop(out);
if let (Some(expected), Some(sha)) = (expected_sha1(f), sha) {
if sha.finalize().as_slice() != &expected[..] {
return Err(format!("SHA-1 mismatch for {rel_path}"));
}
}
Ok(written)
})();
if result.is_err() {
let _ = fs::remove_file(out_path);
}
result
}
/// Run the plan. `progress(bytes_done, bytes_total, files_done, files_total)` fires after every
/// file; `log` receives the start line, failures and the terminal summary.
pub fn run_assemble(
plan: &AssemblePlan,
cancel: &AtomicBool,
progress: &(dyn Fn(u64, u64, u64, u64) + Sync),
log: &(dyn Fn(&str) + Sync),
) -> AssembleOutcome {
let files_total = plan.files.len() as u64;
let mut outcome = AssembleOutcome {
files_total,
bytes_total: plan.bytes_total,
..AssembleOutcome::default()
};
log(&format!(
"assemble cache_dir={} scratch={} {} files={files_total} bytes={} chunks_referenced={}",
plan.cache_dir.display(),
plan.scratch_cache,
crate::priority::log_field(),
plan.bytes_total,
plan.refs.len()
));
let mut refs = plan.refs.clone();
let mut reader = ChunkReader {
cache_dir: plan.cache_dir.clone(),
last: None,
};
for &fi in &plan.files {
if cancel.load(Ordering::Relaxed) {
outcome.cancelled = true;
outcome.error = "cancelled".to_string();
log(&format!(
"cancelled during assembly ({}/{files_total} files)",
outcome.files_done
));
return outcome;
}
let f = &plan.manifest.files[fi];
let out_path = plan.install_dir.join(relative_path(f));
match write_file(&mut reader, f, &out_path) {
Ok(written) => outcome.bytes_written += written,
Err(err) => {
outcome.error = err;
log(&format!(
"FAIL {} ({}/{files_total} files ok)",
outcome.error, outcome.files_done
));
return outcome;
}
}
// Scratch cache: this file was the last user of a chunk → the chunk goes now, not at the
// end, so the cache holds only what is still owed to files not yet written. Released
// before the progress event, so "file done" also means "its chunks are freed".
if plan.scratch_cache {
for guid in distinct_guids(f) {
let Some(count) = refs.get_mut(&guid) else {
continue;
};
*count = count.saturating_sub(1);
if *count == 0 {
reader.forget(&guid);
if fs::remove_file(plan.cache_dir.join(&guid)).is_ok() {
outcome.chunks_removed += 1;
}
}
}
}
outcome.files_done += 1;
progress(
outcome.bytes_written,
plan.bytes_total,
outcome.files_done,
files_total,
);
}
outcome.success = true;
// Done with the cache: remove it whole (a scratch cache is already down to the chunks no
// pending file referenced; the default one is what the Kotlin loop deleted at this point).
match fs::remove_dir_all(&plan.cache_dir) {
Ok(()) => log(&format!("cache removed {}", plan.cache_dir.display())),
Err(e) => log(&format!(
"cache not removed {}: {e}",
plan.cache_dir.display()
)),
}
log(&format!(
"assembled files={} bytes={} chunks_removed_early={}",
outcome.files_done, outcome.bytes_written, outcome.chunks_removed
));
outcome
}
#[cfg(test)]
mod tests {
use super::super::chunk::test_support::temp_dir;
use super::super::manifest::test_support::*;
use super::*;
use std::sync::Mutex;
const A: [u32; 4] = [1, 2, 3, 4];
const B: [u32; 4] = [5, 6, 7, 8];
fn chunk_a() -> Vec<u8> {
(0..300u32).map(|i| (i % 251) as u8).collect()
}
fn chunk_b() -> Vec<u8> {
(0..100u32).map(|i| (200 - i) as u8).collect()
}
fn sha(bytes: &[u8]) -> [u8; 20] {
let mut out = [0u8; 20];
out.copy_from_slice(&Sha1::digest(bytes));
out
}
/// file 0 = A[0..100] ++ B[10..60]; file 1 = A[100..300]. A is shared, B is file 0's alone.
fn expected_files() -> (Vec<u8>, Vec<u8>) {
let (a, b) = (chunk_a(), chunk_b());
let mut f0 = a[0..100].to_vec();
f0.extend_from_slice(&b[10..60]);
(f0, a[100..300].to_vec())
}
fn manifest_bytes(file0_sha: [u8; 20]) -> Vec<u8> {
let (_, f1) = expected_files();
let chunks = vec![
TestChunk { guid: A, hash: 1, sha1: [0x11; 20], group: 1, window: 300, file_size: 300 },
TestChunk { guid: B, hash: 2, sha1: [0x22; 20], group: 2, window: 100, file_size: 100 },
];
let files = vec![
TestFile {
name: "Game\\Binaries\\Game.exe".to_string(),
sha1: file0_sha,
tags: vec![],
parts: vec![(A, 0, 100), (B, 10, 50)],
},
TestFile {
name: "Game/Content/data.pak".to_string(),
sha1: sha(&f1),
tags: vec![],
parts: vec![(A, 100, 200)],
},
];
build_manifest(&chunks, &files, 21, false)
}
fn guid(g: [u32; 4]) -> String {
super::super::manifest::guid_to_string(&g)
}
fn fill_cache(cache: &Path) {
fs::create_dir_all(cache).unwrap();
fs::write(cache.join(guid(A)), chunk_a()).unwrap();
fs::write(cache.join(guid(B)), chunk_b()).unwrap();
}
fn request(install: &Path, cache: &str) -> AssembleRequest {
let (f0, _) = expected_files();
AssembleRequest {
manifest_bytes: manifest_bytes(sha(&f0)),
install_dir: install.to_str().unwrap().to_string(),
chunk_cache_dir: cache.to_string(),
file_indices: vec![0, 1],
}
}
fn run(plan: &AssemblePlan, cancel: bool) -> (AssembleOutcome, Vec<(u64, u64, u64, u64)>) {
let calls = Mutex::new(Vec::new());
let progress = |b: u64, bt: u64, f: u64, ft: u64| calls.lock().unwrap().push((b, bt, f, ft));
let log = |_: &str| {};
let out = run_assemble(plan, &AtomicBool::new(cancel), &progress, &log);
(out, calls.into_inner().unwrap())
}
#[test]
fn default_cache_assembles_like_the_kotlin_loop_and_removes_the_cache_at_the_end() {
let dir = temp_dir("asm-default");
let install = dir.join("install");
let cache = install.join(".chunks");
fill_cache(&cache);
let plan = build_assemble_plan(&request(&install, "")).unwrap();
assert!(!plan.scratch_cache);
assert_eq!(plan.refs[&guid(A)], 2);
assert_eq!(plan.refs[&guid(B)], 1);
let (out, calls) = run(&plan, false);
assert!(out.success, "{}", out.error);
let (f0, f1) = expected_files();
assert_eq!(fs::read(install.join("Game/Binaries/Game.exe")).unwrap(), f0);
assert_eq!(fs::read(install.join("Game/Content/data.pak")).unwrap(), f1);
assert_eq!(out.bytes_written, 350);
assert_eq!(calls, vec![(150, 350, 1, 2), (350, 350, 2, 2)]);
assert_eq!(out.chunks_removed, 0, "no early deletion without a scratch cache");
assert!(!cache.exists(), "the cache goes at the end, as the Kotlin loop's deleteDir");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn scratch_cache_frees_each_chunk_when_its_last_file_is_done() {
let dir = temp_dir("asm-scratch");
let install = dir.join("install");
let cache = dir.join("cache");
fill_cache(&cache);
let plan = build_assemble_plan(&request(&install, cache.to_str().unwrap())).unwrap();
assert!(plan.scratch_cache);
// After file 0: B (only file 0 used it) must be gone, A (file 1 still needs it) must stay.
let seen = Mutex::new(Vec::new());
let progress = |_: u64, _: u64, files_done: u64, _: u64| {
if files_done == 1 {
seen.lock().unwrap().push((cache.join(guid(A)).exists(), cache.join(guid(B)).exists()));
}
};
let log = |_: &str| {};
let out = run_assemble(&plan, &AtomicBool::new(false), &progress, &log);
assert!(out.success, "{}", out.error);
assert_eq!(*seen.lock().unwrap(), vec![(true, false)]);
assert_eq!(out.chunks_removed, 2);
assert!(!cache.exists());
let (f0, f1) = expected_files();
assert_eq!(fs::read(install.join("Game/Binaries/Game.exe")).unwrap(), f0);
assert_eq!(fs::read(install.join("Game/Content/data.pak")).unwrap(), f1);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn a_missing_chunk_fails_with_the_kotlin_wording_and_leaves_the_cache() {
let dir = temp_dir("asm-missing");
let install = dir.join("install");
let cache = dir.join("cache");
fill_cache(&cache);
fs::remove_file(cache.join(guid(B))).unwrap();
let plan = build_assemble_plan(&request(&install, cache.to_str().unwrap())).unwrap();
let (out, calls) = run(&plan, false);
assert!(!out.success && !out.cancelled);
assert_eq!(out.error, format!("Missing chunk {} for Game/Binaries/Game.exe", guid(B)));
assert!(calls.is_empty());
assert!(!install.join("Game/Binaries/Game.exe").exists(), "no partial file survives");
assert!(cache.join(guid(A)).exists(), "failure leaves the cache for the next run");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn a_sha1_mismatch_deletes_the_file_and_fails() {
let dir = temp_dir("asm-sha");
let install = dir.join("install");
let cache = dir.join("cache");
fill_cache(&cache);
let mut req = request(&install, cache.to_str().unwrap());
req.manifest_bytes = manifest_bytes([0x99; 20]);
let plan = build_assemble_plan(&req).unwrap();
let (out, _) = run(&plan, false);
assert!(!out.success);
assert_eq!(out.error, "SHA-1 mismatch for Game/Binaries/Game.exe");
assert!(!install.join("Game/Binaries/Game.exe").exists());
assert!(cache.join(guid(B)).exists(), "nothing was freed for a file that failed");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn an_all_zero_manifest_hash_means_no_check() {
let dir = temp_dir("asm-nohash");
let install = dir.join("install");
let cache = install.join(".chunks");
fill_cache(&cache);
let mut req = request(&install, "");
req.manifest_bytes = manifest_bytes([0u8; 20]);
let plan = build_assemble_plan(&req).unwrap();
let (out, _) = run(&plan, false);
assert!(out.success, "{}", out.error);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn cancel_before_the_first_file_writes_nothing_and_keeps_the_cache() {
let dir = temp_dir("asm-cancel");
let install = dir.join("install");
let cache = dir.join("cache");
fill_cache(&cache);
let plan = build_assemble_plan(&request(&install, cache.to_str().unwrap())).unwrap();
let (out, calls) = run(&plan, true);
assert!(out.cancelled && !out.success);
assert!(calls.is_empty());
assert!(cache.join(guid(A)).exists() && cache.join(guid(B)).exists());
assert!(!install.join("Game").exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn bad_inputs_fail_before_anything_is_written() {
let dir = temp_dir("asm-inputs");
let mut req = request(&dir, "");
req.file_indices = vec![0, 2];
assert!(build_assemble_plan(&req).unwrap_err().contains("out of range"));
req.file_indices = vec![0];
req.install_dir.clear();
assert!(build_assemble_plan(&req).unwrap_err().contains("empty install dir"));
req.install_dir = dir.to_str().unwrap().to_string();
req.manifest_bytes = vec![1, 2, 3];
assert!(build_assemble_plan(&req).is_err());
let _ = fs::remove_dir_all(&dir);
}
}
+375
View File
@@ -0,0 +1,375 @@
//! One chunk: header parse → (zlib inflate | copy) → SHA-1 → `.part` → rename into the cache.
//!
//! Port of `EpicDownloadManager.downloadChunkStreaming` minus the HTTP part (the fetch core
//! hands us the whole response body). Java's protocol, kept exactly:
//! - 41-byte header: magic(4) headerVersion(4) headerSize(4) compressedSize(4) GUID(16) hash(8)
//! storedAs(1); payload starts at `max(headerSize, 41)` (Java never seeks back);
//! - payload = the next `compressedSize` bytes, or whatever the body still has (a short body
//! is NOT an error — the SHA-1 check decides);
//! - `storedAs & 1` → zlib; corrupt zlib = failure; a stream that ends early = partial output;
//! - the decompressed bytes go to `<final>.part`; the SHA-1 (when the manifest has one) must
//! match or the `.part` is deleted; then an atomic rename publishes `<final>`.
//! A cache file therefore exists under its final name ONLY if it fully downloaded and verified,
//! on either engine.
use std::fs::{self, File};
use std::io::{self, BufWriter, Write};
use std::path::{Path, PathBuf};
use sha1::{Digest, Sha1};
/// Chunk file magic (`0xB1FE3AA2`).
pub const CHUNK_MAGIC: u32 = 0xB1FE_3AA2;
/// Bytes Java reads before looking at the header (`byte[] hdrBuf = new byte[41]`).
pub const CHUNK_HEADER_PREFIX: usize = 41;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ChunkHeader {
pub header_version: i32,
pub header_size: i32,
pub compressed_size: i32,
pub stored_as: u8,
}
impl ChunkHeader {
pub fn is_compressed(&self) -> bool {
(self.stored_as & 1) != 0
}
/// Where the payload begins in the body (`if (headerSize > 41) skipFully(headerSize - 41)`).
pub fn payload_start(&self) -> usize {
if self.header_size > CHUNK_HEADER_PREFIX as i32 {
self.header_size as usize
} else {
CHUNK_HEADER_PREFIX
}
}
}
/// `readFully(hdrBuf)` + header decode. `Err` = the Java `continue`/exception paths
/// (short stream, bad magic).
pub fn parse_chunk_header(body: &[u8]) -> Result<ChunkHeader, String> {
if body.len() < CHUNK_HEADER_PREFIX {
return Err(format!(
"Stream ended after {}/{} bytes",
body.len(),
CHUNK_HEADER_PREFIX
));
}
let le = |o: usize| i32::from_le_bytes([body[o], body[o + 1], body[o + 2], body[o + 3]]);
let magic = le(0) as u32;
if magic != CHUNK_MAGIC {
return Err(format!("Bad chunk magic (streaming): 0x{magic:x}"));
}
Ok(ChunkHeader {
header_version: le(4),
header_size: le(8),
compressed_size: le(12),
stored_as: body[40],
})
}
/// The payload slice Java's read loop would consume: `compressedSize` bytes after the header,
/// truncated to what the body holds (`if (n <= 0) break;`). `Err` when the header skip runs
/// past the body (`skipFully` throws).
pub fn chunk_payload<'a>(body: &'a [u8], hdr: &ChunkHeader) -> Result<&'a [u8], String> {
let start = hdr.payload_start();
if start > body.len() {
return Err("Stream ended during skip".to_string());
}
let want = if hdr.compressed_size <= 0 {
0
} else {
hdr.compressed_size as usize
};
let avail = body.len() - start;
let take = want.min(avail);
Ok(&body[start..start + take])
}
/// `Writer` that mirrors `fos.write(...)` + `sha.update(...)` on every block.
struct HashingWriter<W: Write> {
inner: W,
sha: Option<Sha1>,
written: u64,
}
impl<W: Write> Write for HashingWriter<W> {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.inner.write(buf)?;
if let Some(sha) = self.sha.as_mut() {
sha.update(&buf[..n]);
}
self.written += n as u64;
Ok(n)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
/// `new File(outFile.getPath() + ".part")`.
pub fn part_path(final_path: &Path) -> PathBuf {
let mut s = final_path.as_os_str().to_os_string();
s.push(".part");
PathBuf::from(s)
}
/// Process one downloaded body into `final_path`. Returns the number of DECOMPRESSED bytes
/// written. `Err(reason)` = Java's "try the next CDN" outcome; the `.part` never survives an
/// error and `final_path` is only ever created by the rename after verification.
pub fn write_verified_chunk(
body: &[u8],
expected_sha1: Option<&[u8; 20]>,
final_path: &Path,
) -> Result<u64, String> {
let tmp = part_path(final_path);
// `tmp.delete()` — clear any stale partial from a prior interrupted attempt.
let _ = fs::remove_file(&tmp);
let result = write_part(body, expected_sha1, &tmp, final_path);
if result.is_err() {
let _ = fs::remove_file(&tmp);
}
result
}
fn write_part(
body: &[u8],
expected_sha1: Option<&[u8; 20]>,
tmp: &Path,
final_path: &Path,
) -> Result<u64, String> {
let hdr = parse_chunk_header(body)?;
let payload = chunk_payload(body, &hdr)?;
let file = File::create(tmp).map_err(|e| format!("open {}: {e}", tmp.display()))?;
let mut out = HashingWriter {
inner: BufWriter::with_capacity(128 * 1024, file),
sha: expected_sha1.map(|_| Sha1::new()),
written: 0,
};
if hdr.is_compressed() {
// Java: inflate until the stream finishes or the input runs out; a corrupt stream throws
// (DataFormatException) → this CDN attempt fails. flate2 behaves the same way: EOF on a
// truncated stream is a clean end (partial output), corruption is an error.
let mut dec = flate2::read::ZlibDecoder::new(payload);
io::copy(&mut dec, &mut out).map_err(|e| format!("inflate: {e}"))?;
} else {
out.write_all(payload).map_err(|e| format!("write: {e}"))?;
}
out.flush().map_err(|e| format!("flush: {e}"))?;
let written = out.written;
let digest = out.sha.take().map(|s| s.finalize());
drop(out); // close the file before the rename
if let (Some(expected), Some(actual)) = (expected_sha1, digest) {
if actual.as_slice() != &expected[..] {
return Err("Chunk SHA-1 mismatch (streaming)".to_string());
}
}
// Verified → publish atomically (same directory → same filesystem).
fs::rename(tmp, final_path).map_err(|e| format!("Chunk rename failed: {e}"))?;
Ok(written)
}
#[cfg(test)]
pub(crate) mod test_support {
use super::*;
/// Build a chunk body the way the CDN serves it.
pub fn build_chunk_body(
decompressed: &[u8],
compress: bool,
extra_header: usize,
declared_compressed_size: Option<i32>,
) -> Vec<u8> {
let payload = if compress {
let mut enc =
flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::default());
enc.write_all(decompressed).unwrap();
enc.finish().unwrap()
} else {
decompressed.to_vec()
};
let header_size = (CHUNK_HEADER_PREFIX + extra_header) as i32;
let mut out = Vec::new();
out.extend_from_slice(&CHUNK_MAGIC.to_le_bytes());
out.extend_from_slice(&3i32.to_le_bytes()); // headerVersion
out.extend_from_slice(&header_size.to_le_bytes());
out.extend_from_slice(
&declared_compressed_size
.unwrap_or(payload.len() as i32)
.to_le_bytes(),
);
out.extend_from_slice(&[0xAB; 16]); // GUID (unused by the reader)
out.extend_from_slice(&[0xCD; 8]); // hash (unused by the reader)
out.push(if compress { 1 } else { 0 });
out.extend(std::iter::repeat(0x5A).take(extra_header));
assert_eq!(out.len(), header_size as usize);
out.extend_from_slice(&payload);
out
}
pub fn sha1_of(data: &[u8]) -> [u8; 20] {
let mut h = Sha1::new();
h.update(data);
let d = h.finalize();
let mut out = [0u8; 20];
out.copy_from_slice(&d);
out
}
pub fn temp_dir(tag: &str) -> PathBuf {
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
let dir = std::env::temp_dir().join(format!(
"epic-{tag}-{}-{nanos}",
std::process::id()
));
fs::create_dir_all(&dir).unwrap();
dir
}
}
#[cfg(test)]
mod tests {
use super::test_support::*;
use super::*;
fn sample_data() -> Vec<u8> {
(0..200_000u32).map(|i| (i % 251) as u8).collect()
}
#[test]
fn header_parse_matches_java_fields() {
let body = build_chunk_body(b"hello", true, 5, None);
let h = parse_chunk_header(&body).unwrap();
assert_eq!(h.header_version, 3);
assert_eq!(h.header_size, 46);
assert!(h.is_compressed());
assert_eq!(h.payload_start(), 46);
let short = ChunkHeader {
header_version: 0,
header_size: 20,
compressed_size: 1,
stored_as: 0,
};
assert_eq!(short.payload_start(), 41, "headerSize < 41 → payload still starts at 41");
}
#[test]
fn short_or_wrong_magic_bodies_are_rejected() {
assert!(parse_chunk_header(&[0u8; 40]).is_err());
let mut body = build_chunk_body(b"x", false, 0, None);
body[0] ^= 0xFF;
assert!(parse_chunk_header(&body).unwrap_err().contains("Bad chunk magic"));
}
#[test]
fn payload_is_truncated_to_the_body_not_an_error() {
let body = build_chunk_body(b"abcdef", false, 0, Some(100));
let h = parse_chunk_header(&body).unwrap();
assert_eq!(chunk_payload(&body, &h).unwrap(), b"abcdef");
let neg = build_chunk_body(b"abcdef", false, 0, Some(-1));
let h = parse_chunk_header(&neg).unwrap();
assert_eq!(chunk_payload(&neg, &h).unwrap(), b"");
// headerSize past the body → skipFully throws.
let mut far = build_chunk_body(b"abcdef", false, 0, None);
far[8..12].copy_from_slice(&5000i32.to_le_bytes());
let h = parse_chunk_header(&far).unwrap();
assert!(chunk_payload(&far, &h).is_err());
}
#[test]
fn compressed_chunk_is_inflated_verified_and_renamed() {
let dir = temp_dir("ok");
let data = sample_data();
let body = build_chunk_body(&data, true, 3, None);
let final_path = dir.join("00000001000000020000000300000004");
let n = write_verified_chunk(&body, Some(&sha1_of(&data)), &final_path).unwrap();
assert_eq!(n, data.len() as u64);
assert_eq!(fs::read(&final_path).unwrap(), data);
assert!(!part_path(&final_path).exists(), ".part renamed away");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stored_chunk_is_copied_as_is() {
let dir = temp_dir("stored");
let data = b"stored-as-is payload".to_vec();
let body = build_chunk_body(&data, false, 0, None);
let final_path = dir.join("chunk");
let n = write_verified_chunk(&body, Some(&sha1_of(&data)), &final_path).unwrap();
assert_eq!(n, data.len() as u64);
assert_eq!(fs::read(&final_path).unwrap(), data);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn sha1_mismatch_leaves_nothing_behind() {
let dir = temp_dir("sha");
let data = sample_data();
let body = build_chunk_body(&data, true, 0, None);
let final_path = dir.join("chunk");
let err = write_verified_chunk(&body, Some(&[0x42; 20]), &final_path).unwrap_err();
assert!(err.contains("SHA-1 mismatch"), "{err}");
assert!(!final_path.exists());
assert!(!part_path(&final_path).exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn unverifiable_chunk_is_accepted_without_a_hash() {
let dir = temp_dir("nosha");
let data = sample_data();
let body = build_chunk_body(&data, true, 0, None);
let final_path = dir.join("chunk");
assert_eq!(
write_verified_chunk(&body, None, &final_path).unwrap(),
data.len() as u64
);
assert_eq!(fs::read(&final_path).unwrap(), data);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn corrupt_zlib_fails_and_truncated_zlib_fails_the_hash() {
let dir = temp_dir("zlib");
let data = sample_data();
let mut body = build_chunk_body(&data, true, 0, None);
let final_path = dir.join("chunk");
// Truncate the body: inflate ends early → partial output → SHA-1 mismatch.
let cut = body.len() - 1000;
let err = write_verified_chunk(&body[..cut], Some(&sha1_of(&data)), &final_path).unwrap_err();
assert!(err.contains("SHA-1 mismatch") || err.contains("inflate"), "{err}");
assert!(!final_path.exists());
// Corrupt the deflate stream itself.
for b in body.iter_mut().skip(60).take(64) {
*b ^= 0x55;
}
let err = write_verified_chunk(&body, Some(&sha1_of(&data)), &final_path).unwrap_err();
assert!(err.contains("SHA-1 mismatch") || err.contains("inflate"), "{err}");
assert!(!final_path.exists());
assert!(!part_path(&final_path).exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stale_part_file_is_cleared_before_the_attempt() {
let dir = temp_dir("stale");
let final_path = dir.join("chunk");
fs::write(part_path(&final_path), b"stale").unwrap();
let body = [0u8; 10]; // too short → header failure
assert!(write_verified_chunk(&body, None, &final_path).is_err());
assert!(!part_path(&final_path).exists());
assert!(!final_path.exists());
let _ = fs::remove_dir_all(&dir);
}
}
+480
View File
@@ -0,0 +1,480 @@
//! Plan → `FetchItem`s → `crate::fetch_core::run_fetch` with a sink that fills the chunk cache.
//!
//! This is the replacement for the body of the Java pool block in `EpicDownloadManager.install`
//! ("Download unique chunks — 8 parallel threads"). Inputs are exactly what that block sees:
//! the parsed manifest (re-parsed here from the same bytes), the pending file set (indices Java
//! computed after its delta/verify pass), the CDN prefixes (`baseUrl + cloudDir`, cloudflare
//! already skipped) and the chunk cache dir. Output = `<cache>/<GUID>` for every needed chunk,
//! where `<cache>` is `<installDir>/.chunks` unless the caller named another directory; the
//! files are assembled from it afterwards (by the Kotlin manager, or by [`super::assemble`]).
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use crate::fetch_core::{run_fetch, FetchItem, FetchOptions, FetchSink, SinkError};
use super::manifest::{parse_manifest, Manifest};
use super::plan::{
cached_chunk_path, chunk_url, distinct_prefixes, per_host_cap, resolve_cache_dir,
total_credit_bytes, unique_chunks_for_files,
};
/// Java `conn.setReadTimeout(60000)` — the longer of the two Java timeouts (connect was 30 s).
pub const REQUEST_TIMEOUT: Duration = Duration::from_secs(60);
/// Everything the JNI layer hands over for one run.
pub struct EpicRequest {
pub manifest_bytes: Vec<u8>,
pub install_dir: String,
/// Chunk cache directory; `""` = `<installDir>/.chunks`. See [`resolve_cache_dir`].
pub chunk_cache_dir: String,
/// `cdn.baseUrl + cdn.cloudDir` per manifest-API CDN entry, in Java's order.
pub cdn_prefixes: Vec<String>,
/// Indices into `manifest.files` of Java's `pendingFiles` (post delta/verify), in order.
pub pending_file_indices: Vec<usize>,
/// Java's `neededChunks.size()` / `totalBytes` for the plan cross-check (`None` = skip).
pub expected_chunks: Option<u64>,
pub expected_bytes: Option<u64>,
pub ca_bundle_path: String,
/// Window ceiling: the Steam speed tier's network window (Fast = 32) since improvements
/// round 1; the Java fallback pool keeps its fixed 8 (`super::JAVA_POOL_THREADS`).
pub max_workers: usize,
pub process_workers: usize,
pub label: String,
}
/// The chunk plan for one run.
#[derive(Debug)]
pub struct EpicPlan {
pub manifest: Manifest,
pub cache_dir: PathBuf,
/// The cache is a directory the caller named, apart from the install: scratch space that
/// the assembler empties as it goes and removes when it is done.
pub scratch_cache: bool,
/// Chunk indices (into `manifest.unique_chunks`) in Java's submission order.
pub needed: Vec<usize>,
/// `Σ max(fileSize, 1)` over `needed`.
pub total_bytes: u64,
/// Distinct CDN prefixes = fetch-core host keys.
pub hosts: Vec<String>,
}
/// Terminal result of a run, in the shape Java's pool block needs to reproduce its own exit
/// paths (`CANCELLED during chunk download (n/total chunks)`, `N chunks failed`, `chunksOK=`).
#[derive(Clone, Debug, Default)]
pub struct EpicOutcome {
pub success: bool,
pub cancelled: bool,
pub error: String,
/// Credited (manifest `max(fileSize,1)`) bytes: skipped-cached + fetched.
pub bytes_credited: u64,
/// Chunks accounted for (skipped-cached + fetched), Java's `completedCount`.
pub chunks_done: u64,
pub chunks_total: u64,
pub bytes_total: u64,
/// Decompressed bytes actually written to the cache this run.
pub decompressed_written: u64,
}
/// Parse + plan (no I/O beyond reading the cache dir). `Err` = "engine could not start"; Java
/// then runs its own pool, since nothing has been fetched yet.
pub fn build_plan(req: &EpicRequest) -> Result<EpicPlan, String> {
let manifest = parse_manifest(&req.manifest_bytes).map_err(|e| format!("plan: {e}"))?;
for &i in &req.pending_file_indices {
if i >= manifest.files.len() {
return Err(format!(
"plan: pending file index {i} out of range ({} files)",
manifest.files.len()
));
}
}
let needed = unique_chunks_for_files(&manifest, &req.pending_file_indices);
let total_bytes = total_credit_bytes(&manifest, &needed);
if let Some(expected) = req.expected_chunks {
if expected != needed.len() as u64 {
return Err(format!(
"plan: chunk count mismatch java={expected} rust={}",
needed.len()
));
}
}
if let Some(expected) = req.expected_bytes {
if expected != total_bytes {
return Err(format!(
"plan: byte total mismatch java={expected} rust={total_bytes}"
));
}
}
let hosts = distinct_prefixes(&req.cdn_prefixes);
if hosts.is_empty() {
return Err("plan: no CDN prefixes".to_string());
}
let cache_dir = resolve_cache_dir(&req.install_dir, &req.chunk_cache_dir);
std::fs::create_dir_all(&cache_dir)
.map_err(|e| format!("plan: mkdirs {}: {e}", cache_dir.display()))?;
Ok(EpicPlan {
manifest,
cache_dir,
scratch_cache: !req.chunk_cache_dir.is_empty(),
needed,
total_bytes,
hosts,
})
}
/// Sink: one downloaded body → verified cache file. `id` indexes `fetch_chunks`.
struct ChunkCacheSink<'a> {
plan: &'a EpicPlan,
/// Chunk indices (into `manifest.unique_chunks`) of the items actually being fetched.
fetch_chunks: &'a [usize],
decompressed: AtomicU64,
}
impl<'a> FetchSink for ChunkCacheSink<'a> {
fn process(&self, item: &FetchItem, body: Vec<u8>) -> Result<u64, SinkError> {
let Some(&ci) = self.fetch_chunks.get(item.id as usize) else {
return Err(SinkError::Fatal(format!("item id {} out of range", item.id)));
};
let chunk = &self.plan.manifest.unique_chunks[ci];
let final_path = cached_chunk_path(&self.plan.cache_dir, chunk);
match super::chunk::write_verified_chunk(&body, chunk.verifiable_sha1(), &final_path) {
Ok(written) => {
self.decompressed.fetch_add(written, Ordering::Relaxed);
Ok(chunk.credit_bytes())
}
// Every per-attempt failure in Java is "try the next CDN"; the core's Retry rotates
// hosts and backs off the same way (bounded at its attempt cap).
Err(reason) => Err(SinkError::Retry(format!("{} {reason}", chunk.guid_str()))),
}
}
}
/// Rolling speed sampler for the end-of-run summary: samples at ≥500 ms like the Java pool's
/// `lastSpeedMs` logic, keeps the peak.
struct SpeedMeter {
start: Instant,
last_at: Instant,
last_bytes: u64,
peak_bps: f64,
}
impl SpeedMeter {
fn new() -> Self {
let now = Instant::now();
Self {
start: now,
last_at: now,
last_bytes: 0,
peak_bps: 0.0,
}
}
fn sample(&mut self, bytes_now: u64) {
let now = Instant::now();
let dt = now.duration_since(self.last_at);
if dt >= Duration::from_millis(500) {
let delta = bytes_now.saturating_sub(self.last_bytes) as f64;
let bps = delta / dt.as_secs_f64();
if bps > self.peak_bps {
self.peak_bps = bps;
}
self.last_at = now;
self.last_bytes = bytes_now;
}
}
fn summary(&self, bytes: u64, decompressed: u64, skipped: u64) -> String {
let secs = self.start.elapsed().as_secs_f64().max(0.001);
let avg_bps = bytes as f64 / secs;
format!(
"summary bytes={bytes} decompressed={decompressed} skipped_chunks={skipped} elapsed={secs:.1}s avg_mbps={:.2} peak_mbps={:.2} avg_MBps={:.2}",
avg_bps * 8.0 / 1_000_000.0,
self.peak_bps * 8.0 / 1_000_000.0,
avg_bps / (1024.0 * 1024.0)
)
}
}
/// Run the plan. `progress(bytes_done, chunks_done)` fires once per accounted chunk (cached-skip
/// or fetched), exactly the cadence of the Java pool's per-task `progress(...)` call. `log`
/// receives engine lines (`fetch-window`, `summary`, failures).
pub fn run_plan(
plan: &EpicPlan,
req: &EpicRequest,
cancel: &AtomicBool,
progress: &(dyn Fn(u64, u64) + Sync),
log: &(dyn Fn(&str) + Sync),
) -> EpicOutcome {
let chunks_total = plan.needed.len() as u64;
let mut outcome = EpicOutcome {
chunks_total,
bytes_total: plan.total_bytes,
..EpicOutcome::default()
};
let host_cap = per_host_cap(req.max_workers, plan.hosts.len());
log(&format!(
"plan chunk_dir={} cache_dir={} scratch={} {} files_pending={} chunks={} bytes={} hosts={} workers={} per_host_cap={host_cap} process_workers={}",
plan.manifest.chunk_dir,
plan.cache_dir.display(),
plan.scratch_cache,
crate::priority::log_field(),
req.pending_file_indices.len(),
chunks_total,
plan.total_bytes,
plan.hosts.len(),
req.max_workers,
req.process_workers
));
// Java: each pool task first checks the cancel flag, then `cachedFile.exists()` → credit
// without fetching. Account for the cached chunks up front (same credit, same callback).
let mut fetch_chunks: Vec<usize> = Vec::with_capacity(plan.needed.len());
let mut pre_bytes: u64 = 0;
let mut pre_chunks: u64 = 0;
for &ci in &plan.needed {
if cancel.load(Ordering::Relaxed) {
outcome.cancelled = true;
outcome.error = "cancelled".to_string();
outcome.bytes_credited = pre_bytes;
outcome.chunks_done = pre_chunks;
log(&format!(
"cancelled before fetch ({pre_chunks}/{chunks_total} chunks)"
));
return outcome;
}
let chunk = &plan.manifest.unique_chunks[ci];
if cached_chunk_path(&plan.cache_dir, chunk).exists() {
pre_bytes += chunk.credit_bytes();
pre_chunks += 1;
progress(pre_bytes, pre_chunks);
} else {
fetch_chunks.push(ci);
}
}
for (i, h) in plan.hosts.iter().enumerate() {
log(&format!("host[{i}]={h}"));
}
log(&format!(
"skip cached={pre_chunks} bytes={pre_bytes} to_fetch={}",
fetch_chunks.len()
));
if fetch_chunks.is_empty() {
outcome.success = true;
outcome.bytes_credited = pre_bytes;
outcome.chunks_done = pre_chunks;
log("summary bytes=0 decompressed=0 elapsed=0.0s avg_mbps=0.00 peak_mbps=0.00 (nothing to fetch)");
return outcome;
}
let items: Vec<FetchItem> = fetch_chunks
.iter()
.enumerate()
.map(|(id, &ci)| {
let chunk = &plan.manifest.unique_chunks[ci];
FetchItem {
id: id as u64,
urls: plan
.hosts
.iter()
.map(|h| chunk_url(h, &plan.manifest.chunk_dir, chunk))
.collect(),
reserve: if chunk.file_size > 0 {
chunk.file_size as u64
} else {
0
},
range: None,
}
})
.collect();
let opts = FetchOptions {
max_workers: req.max_workers.max(1),
// Improvements round 1: the tier ceiling split across the distinct CDNs (floor 6), so
// the whole window is reachable on 1 host or on all 3.
per_host_cap: host_cap,
timeout: REQUEST_TIMEOUT,
headers: vec![("User-Agent".to_string(), super::USER_AGENT.to_string())],
ca_bundle_path: req.ca_bundle_path.clone(),
process_workers: req.process_workers.max(1),
label: req.label.clone(),
// Whole-body mode (Epic chunks are ≤ ~1 MiB compressed; the core's byte budget scales
// with the window); `stream` and any future field keep the core's defaults.
..FetchOptions::default()
};
let sink = ChunkCacheSink {
plan,
fetch_chunks: &fetch_chunks,
decompressed: AtomicU64::new(0),
};
let meter = Mutex::new(SpeedMeter::new());
let progress_adapter = |bytes: u64, items_ok: u64| {
if let Ok(mut m) = meter.lock() {
m.sample(bytes);
}
progress(pre_bytes + bytes, pre_chunks + items_ok);
};
let fetched = run_fetch(
items,
&plan.hosts,
&opts,
&sink,
cancel,
&progress_adapter,
log,
);
let decompressed = sink.decompressed.load(Ordering::Relaxed);
outcome.bytes_credited = pre_bytes + fetched.bytes_credited;
outcome.chunks_done = pre_chunks + fetched.items_ok;
outcome.decompressed_written = decompressed;
if let Ok(m) = meter.lock() {
log(&m.summary(fetched.bytes_credited, decompressed, pre_chunks));
}
if fetched.cancelled {
outcome.cancelled = true;
outcome.error = "cancelled".to_string();
log(&format!(
"cancelled during chunk download ({}/{chunks_total} chunks)",
outcome.chunks_done
));
return outcome;
}
if let Some(err) = fetched.error {
outcome.error = err;
log(&format!(
"FAIL {} ({}/{chunks_total} chunks ok)",
outcome.error, outcome.chunks_done
));
return outcome;
}
if fetched.items_ok != fetch_chunks.len() as u64 {
outcome.error = format!(
"fetch ended with {}/{} chunks",
fetched.items_ok,
fetch_chunks.len()
);
log(&format!("FAIL {}", outcome.error));
return outcome;
}
outcome.success = true;
log(&format!("chunksOK={}", outcome.chunks_done));
outcome
}
#[cfg(test)]
mod tests {
use super::super::manifest::test_support::*;
use super::*;
fn request(dir: &str, pending: Vec<usize>) -> EpicRequest {
EpicRequest {
manifest_bytes: build_manifest(&sample_chunks(), &sample_files(), 21, true),
install_dir: dir.to_string(),
chunk_cache_dir: String::new(),
cdn_prefixes: vec![
"https://fastly-download.epicgames.com/Builds/o/x/default".to_string(),
"https://download.epicgames.com/Builds/o/x/default".to_string(),
"https://fastly-download.epicgames.com/Builds/o/x/default".to_string(),
],
pending_file_indices: pending,
expected_chunks: None,
expected_bytes: None,
ca_bundle_path: String::new(),
max_workers: super::super::JAVA_POOL_THREADS,
process_workers: 2,
label: "epic test".to_string(),
}
}
#[test]
fn plan_matches_java_for_the_pending_set() {
let dir = super::super::chunk::test_support::temp_dir("plan");
let req = request(dir.to_str().unwrap(), vec![0, 1]);
let plan = build_plan(&req).unwrap();
assert_eq!(plan.needed, vec![0, 1]);
assert_eq!(plan.total_bytes, 700_000 + 1);
assert_eq!(plan.hosts.len(), 2, "duplicate CDN prefix collapsed");
assert!(plan.cache_dir.ends_with(".chunks"));
assert!(plan.cache_dir.is_dir());
assert!(!plan.scratch_cache);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_named_cache_dir_is_created_and_marked_scratch() {
let dir = super::super::chunk::test_support::temp_dir("scratch");
let cache = dir.join("cache").join("Game");
let mut req = request(dir.join("install").to_str().unwrap(), vec![0, 1]);
req.chunk_cache_dir = cache.to_str().unwrap().to_string();
let plan = build_plan(&req).unwrap();
assert_eq!(plan.cache_dir, cache);
assert!(plan.cache_dir.is_dir());
assert!(plan.scratch_cache);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn plan_cross_check_rejects_mismatches() {
let dir = super::super::chunk::test_support::temp_dir("xcheck");
let mut req = request(dir.to_str().unwrap(), vec![0, 1]);
req.expected_chunks = Some(3);
assert!(build_plan(&req).unwrap_err().contains("chunk count mismatch"));
req.expected_chunks = Some(2);
req.expected_bytes = Some(5);
assert!(build_plan(&req).unwrap_err().contains("byte total mismatch"));
req.expected_bytes = Some(700_001);
assert!(build_plan(&req).is_ok());
req.pending_file_indices = vec![7];
assert!(build_plan(&req).unwrap_err().contains("out of range"));
req.pending_file_indices = vec![0];
req.cdn_prefixes.clear();
assert!(build_plan(&req).unwrap_err().contains("no CDN"));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn fully_cached_plan_completes_without_fetching() {
let dir = super::super::chunk::test_support::temp_dir("cached");
let req = request(dir.to_str().unwrap(), vec![0]);
let plan = build_plan(&req).unwrap();
for &ci in &plan.needed {
std::fs::write(
cached_chunk_path(&plan.cache_dir, &plan.manifest.unique_chunks[ci]),
b"cached",
)
.unwrap();
}
let cancel = AtomicBool::new(false);
let calls = Mutex::new(Vec::new());
let progress = |b: u64, c: u64| calls.lock().unwrap().push((b, c));
let log = |_: &str| {};
let out = run_plan(&plan, &req, &cancel, &progress, &log);
assert!(out.success);
assert_eq!(out.chunks_done, 2);
assert_eq!(out.bytes_credited, 700_001);
assert_eq!(*calls.lock().unwrap(), vec![(700_000, 1), (700_001, 2)]);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn cancel_before_fetch_reports_cancelled() {
let dir = super::super::chunk::test_support::temp_dir("cancel");
let req = request(dir.to_str().unwrap(), vec![0]);
let plan = build_plan(&req).unwrap();
let cancel = AtomicBool::new(true);
let progress = |_: u64, _: u64| {};
let log = |_: &str| {};
let out = run_plan(&plan, &req, &cancel, &progress, &log);
assert!(out.cancelled);
assert!(!out.success);
let _ = std::fs::remove_dir_all(&dir);
}
}
+487
View File
@@ -0,0 +1,487 @@
//! JNI exports for `com.droiddeck.launcher.stores.epic.EpicNative`.
//!
//! ```text
//! nativeStart(manifest: byte[], installDir: String, chunkCacheDir: String, cdnPrefixes: String[],
//! pendingFileIdx: int[], expectedChunks: int, expectedBytes: long,
//! caBundlePath: String, maxWorkers: int, processWorkers: int,
//! listener: EpicNative.Listener) -> long handle (0 = not started)
//! nativeAssemble(manifest: byte[], installDir: String, chunkCacheDir: String, fileIdx: int[],
//! listener: EpicNative.Listener) -> long handle (0 = not started)
//! nativeCancel(handle) — flips the run's AtomicBool; fetch or assembly stops promptly
//! nativeRelease(handle) — frees the handle (after onComplete)
//! ```
//! `chunkCacheDir` = "" keeps the cache at `<installDir>/.chunks`; any other directory is a
//! scratch cache (see `assemble.rs`). Listener (all methods run on native threads):
//! `onPlan(int chunksTotal, long bytesTotal, String chunkDir)` once before any fetch,
//! `onProgress(long bytesDone, long bytesTotal, int done, int total)` per chunk fetched (or, in
//! an assembly run, per file written), `onLog(String line)`,
//! `onComplete(boolean success, String error, long bytes)`.
//!
//! Planning (manifest parse + cross-check) runs synchronously on the calling thread so a plan
//! failure returns 0 BEFORE any fetch and Java can fall back to its own pool; the fetch runs on
//! a new thread. JavaVM attach / GlobalRef handling is the same shape as the GOG facade.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
use jni::objects::{GlobalRef, JByteArray, JClass, JIntArray, JObject, JObjectArray, JString, JValue};
use jni::sys::{jint, jlong, JNI_FALSE, JNI_TRUE};
use jni::{JNIEnv, JavaVM};
use super::assemble::{build_assemble_plan, run_assemble, AssembleRequest};
use super::driver::{build_plan, run_plan, EpicRequest};
#[cfg(target_os = "android")]
#[link(name = "log")]
unsafe extern "C" {
fn __android_log_write(prio: i32, tag: *const i8, text: *const i8) -> i32;
}
fn android_log(message: &str) {
#[cfg(target_os = "android")]
{
use std::ffi::CString;
let Ok(tag) = CString::new(super::LOG_TAG) else {
return;
};
let sanitized = message.replace('\0', " ");
let Ok(text) = CString::new(sanitized) else {
return;
};
unsafe {
let _ = __android_log_write(4, tag.as_ptr().cast(), text.as_ptr().cast());
}
}
#[cfg(not(target_os = "android"))]
{
let _ = message;
}
}
/// Per-run native state behind the `jlong` handle — the same type for a fetch run and an
/// assembly run, so one `nativeCancel` / `nativeRelease` pair serves both.
pub struct EpicDownloadHandle {
cancel: Arc<AtomicBool>,
}
fn clear_pending_exception(env: &mut JNIEnv) {
if env.exception_check().unwrap_or(false) {
let _ = env.exception_clear();
}
}
fn jstring_to_string(env: &mut JNIEnv, value: &JString) -> Option<String> {
env.get_string(value)
.ok()
.map(|s| s.to_string_lossy().into_owned())
}
fn string_array_to_vec(env: &mut JNIEnv, array: &JObjectArray) -> Vec<String> {
let len = env.get_array_length(array).unwrap_or(0);
let mut out = Vec::with_capacity(len.max(0) as usize);
let mut i = 0;
while i < len {
match env.get_object_array_element(array, i) {
Ok(obj) => {
if obj.is_null() {
out.push(String::new());
} else {
let js = JString::from(obj);
out.push(jstring_to_string(env, &js).unwrap_or_default());
}
}
Err(_) => {
clear_pending_exception(env);
out.push(String::new());
}
}
i += 1;
}
out
}
fn int_array_to_vec(env: &JNIEnv, array: &JIntArray) -> Vec<i32> {
let len = env.get_array_length(array).unwrap_or(0);
if len <= 0 {
return Vec::new();
}
let mut values = vec![0i32; len as usize];
if env.get_int_array_region(array, 0, &mut values).is_err() {
return Vec::new();
}
values
}
fn call_on_log(env: &mut JNIEnv, listener: &JObject, line: &str) {
if listener.is_null() {
return;
}
let Ok(s) = env.new_string(line) else {
clear_pending_exception(env);
return;
};
let obj = JObject::from(s);
let _ = env.call_method(
listener,
"onLog",
"(Ljava/lang/String;)V",
&[JValue::Object(&obj)],
);
clear_pending_exception(env);
}
fn call_on_plan(env: &mut JNIEnv, listener: &JObject, chunks_total: u64, bytes_total: u64, chunk_dir: &str) {
if listener.is_null() {
return;
}
let Ok(s) = env.new_string(chunk_dir) else {
clear_pending_exception(env);
return;
};
let obj = JObject::from(s);
let _ = env.call_method(
listener,
"onPlan",
"(IJLjava/lang/String;)V",
&[
JValue::Int(chunks_total.min(i32::MAX as u64) as jint),
JValue::Long(bytes_total as jlong),
JValue::Object(&obj),
],
);
clear_pending_exception(env);
}
fn call_on_progress(
env: &mut JNIEnv,
listener: &JObject,
bytes_done: u64,
bytes_total: u64,
chunks_done: u64,
chunks_total: u64,
) {
if listener.is_null() {
return;
}
let _ = env.call_method(
listener,
"onProgress",
"(JJII)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
JValue::Int(chunks_done.min(i32::MAX as u64) as jint),
JValue::Int(chunks_total.min(i32::MAX as u64) as jint),
],
);
clear_pending_exception(env);
}
fn call_on_complete(env: &mut JNIEnv, listener: &JObject, success: bool, error: &str, bytes: u64) {
if listener.is_null() {
return;
}
let Ok(s) = env.new_string(error) else {
clear_pending_exception(env);
return;
};
let obj = JObject::from(s);
let _ = env.call_method(
listener,
"onComplete",
"(ZLjava/lang/String;J)V",
&[
JValue::Bool(if success { JNI_TRUE } else { JNI_FALSE }),
JValue::Object(&obj),
JValue::Long(bytes as jlong),
],
);
clear_pending_exception(env);
}
/// Log to logcat AND to the listener (the Kotlin manager mirrors it into its own debug log).
fn log_both(env: &mut JNIEnv, listener: &JObject, line: &str) {
android_log(line);
call_on_log(env, listener, line);
}
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeStart(
mut env: JNIEnv,
_class: JClass,
manifest: JByteArray,
install_dir: JString,
chunk_cache_dir: JString,
cdn_prefixes: JObjectArray,
pending_file_idx: JIntArray,
expected_chunks: jint,
expected_bytes: jlong,
ca_bundle_path: JString,
max_workers: jint,
process_workers: jint,
listener: JObject,
) -> jlong {
if listener.is_null() {
android_log("nativeStart: null listener");
return 0;
}
let Ok(manifest_bytes) = env.convert_byte_array(manifest) else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "manifest bytes unreadable", 0);
return 0;
};
let install_dir = jstring_to_string(&mut env, &install_dir).unwrap_or_default();
let chunk_cache_dir = jstring_to_string(&mut env, &chunk_cache_dir).unwrap_or_default();
let cdn_prefixes = string_array_to_vec(&mut env, &cdn_prefixes);
let pending_file_indices: Vec<usize> = int_array_to_vec(&env, &pending_file_idx)
.into_iter()
.map(|i| i.max(0) as usize)
.collect();
let ca_bundle_path = jstring_to_string(&mut env, &ca_bundle_path).unwrap_or_default();
let max_workers = max_workers.max(1) as usize;
let process_workers = process_workers.max(1) as usize;
let label = {
let base = std::path::Path::new(&install_dir)
.file_name()
.map(|s| s.to_string_lossy().into_owned())
.unwrap_or_default();
format!("epic app={base}")
};
let host_count = super::plan::distinct_prefixes(&cdn_prefixes).len();
let host_cap = super::plan::per_host_cap(max_workers, host_count);
log_both(
&mut env,
&listener,
&format!(
"engine=rust label=\"{label}\" install_dir={install_dir} cache_dir={} cdns={} hosts={host_count} pending_files={} workers={max_workers} per_host_cap={host_cap} process_workers={process_workers} manifest_bytes={}",
if chunk_cache_dir.is_empty() { "<installDir>/.chunks" } else { chunk_cache_dir.as_str() },
cdn_prefixes.len(),
pending_file_indices.len(),
manifest_bytes.len()
),
);
let req = EpicRequest {
manifest_bytes,
install_dir,
chunk_cache_dir,
cdn_prefixes,
pending_file_indices,
expected_chunks: if expected_chunks >= 0 {
Some(expected_chunks as u64)
} else {
None
},
expected_bytes: if expected_bytes >= 0 {
Some(expected_bytes as u64)
} else {
None
},
ca_bundle_path,
max_workers,
process_workers,
label,
};
let plan = match build_plan(&req) {
Ok(plan) => plan,
Err(err) => {
log_both(&mut env, &listener, &format!("not started: {err}"));
call_on_complete(&mut env, &listener, false, &err, 0);
return 0;
}
};
call_on_plan(
&mut env,
&listener,
plan.needed.len() as u64,
plan.total_bytes,
&plan.manifest.chunk_dir,
);
let Ok(vm) = env.get_java_vm() else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "JavaVM unavailable", 0);
return 0;
};
let Ok(listener) = env.new_global_ref(&listener) else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "listener ref failed", 0);
return 0;
};
let cancel = Arc::new(AtomicBool::new(false));
let thread_cancel = Arc::clone(&cancel);
let handle = Box::new(EpicDownloadHandle { cancel });
thread::spawn(move || {
crate::priority::background();
run_on_thread(vm, listener, plan, req, thread_cancel);
});
Box::into_raw(handle) as jlong
}
fn run_on_thread(
vm: JavaVM,
listener: GlobalRef,
plan: super::driver::EpicPlan,
req: EpicRequest,
cancel: Arc<AtomicBool>,
) {
let bytes_total = plan.total_bytes;
let chunks_total = plan.needed.len() as u64;
// Progress fires from the core's process-pool threads; attach each as a daemon like the
// Steam engine's `dispatch_download_progress` does.
let progress = |bytes_done: u64, chunks_done: u64| {
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_progress(
&mut env,
listener.as_obj(),
bytes_done,
bytes_total,
chunks_done,
chunks_total,
);
};
let log = |line: &str| {
android_log(line);
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_log(&mut env, listener.as_obj(), line);
};
let outcome = run_plan(&plan, &req, cancel.as_ref(), &progress, &log);
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_complete(
&mut env,
listener.as_obj(),
outcome.success,
&outcome.error,
outcome.bytes_credited,
);
}
/// `EpicNative.nativeAssemble(manifest, installDir, chunkCacheDir, fileIdx, listener)`: write
/// the files `fileIdx` names from the chunk cache (see `assemble.rs`). Parse + plan run on the
/// calling thread, so bad inputs return 0 (after `onComplete(false, reason, 0)`) before anything
/// is written; the assembly runs on a new thread and reports per file through `onProgress(bytes,
/// bytesTotal, filesDone, filesTotal)`, then `onComplete(success, error, bytesWritten)`.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeAssemble(
mut env: JNIEnv,
_class: JClass,
manifest: JByteArray,
install_dir: JString,
chunk_cache_dir: JString,
file_idx: JIntArray,
listener: JObject,
) -> jlong {
if listener.is_null() {
android_log("nativeAssemble: null listener");
return 0;
}
let Ok(manifest_bytes) = env.convert_byte_array(manifest) else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "manifest bytes unreadable", 0);
return 0;
};
let req = AssembleRequest {
manifest_bytes,
install_dir: jstring_to_string(&mut env, &install_dir).unwrap_or_default(),
chunk_cache_dir: jstring_to_string(&mut env, &chunk_cache_dir).unwrap_or_default(),
file_indices: int_array_to_vec(&env, &file_idx)
.into_iter()
.map(|i| i.max(0) as usize)
.collect(),
};
let plan = match build_assemble_plan(&req) {
Ok(plan) => plan,
Err(err) => {
log_both(&mut env, &listener, &format!("not started: {err}"));
call_on_complete(&mut env, &listener, false, &err, 0);
return 0;
}
};
let Ok(vm) = env.get_java_vm() else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "JavaVM unavailable", 0);
return 0;
};
let Ok(listener) = env.new_global_ref(&listener) else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "listener ref failed", 0);
return 0;
};
let cancel = Arc::new(AtomicBool::new(false));
let thread_cancel = Arc::clone(&cancel);
let handle = Box::new(EpicDownloadHandle { cancel });
thread::spawn(move || {
crate::priority::background();
let progress = |bytes_done: u64, bytes_total: u64, files_done: u64, files_total: u64| {
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_progress(&mut env, listener.as_obj(), bytes_done, bytes_total, files_done, files_total);
};
let log = |line: &str| {
android_log(line);
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_log(&mut env, listener.as_obj(), line);
};
let outcome = run_assemble(&plan, thread_cancel.as_ref(), &progress, &log);
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_complete(
&mut env,
listener.as_obj(),
outcome.success,
&outcome.error,
outcome.bytes_written,
);
});
Box::into_raw(handle) as jlong
}
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeCancel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle == 0 {
return;
}
let handle = unsafe { &*(handle as *const EpicDownloadHandle) };
handle.cancel.store(true, Ordering::Relaxed);
android_log("cancel requested");
}
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeRelease(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle == 0 {
return;
}
// The worker thread owns its own Arc clone of the cancel flag and its own GlobalRef, so
// freeing the handle is safe even if a (cancelled) run is still winding down.
drop(unsafe { Box::from_raw(handle as *mut EpicDownloadHandle) });
}
@@ -0,0 +1,987 @@
//! Epic build manifest parsers — a 1:1 port of `EpicDownloadManager.parseManifest` (binary,
//! magic `0x44BEC00C`) and `parseJsonManifest` (legacy JSON manifests).
//!
//! Every quirk of the Java reader is kept on purpose (the Java manager stays the reference and
//! the two engines must derive the same plan from the same bytes): section skips by declared
//! size, `readFString` UTF-16LE/ASCII decoding, duplicate-GUID handling (last value, first
//! position), the JSON `Long.parseLong(.., 16)` / `parseInt` fallbacks, and "any exception →
//! parse failed".
use std::collections::HashMap;
use std::io::Read;
/// Binary manifest magic (`EpicDownloadManager.parseManifest`).
pub const MANIFEST_MAGIC: u32 = 0x44BE_C00C;
/// One entry of the ChunkDataList section (`EpicDownloadManager.ChunkInfo`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ChunkInfo {
/// Four uint32 in binary read order.
pub guid: [u32; 4],
/// uint64 "hash" that forms the first half of the chunk file name.
pub hash: u64,
/// SHA-1 of the DECOMPRESSED chunk; `None` for JSON manifests (Java: `sha1 == null`).
pub sha1: Option<[u8; 20]>,
/// Subfolder number (DECIMAL `%02d`). `i32` so the JSON parser's `Integer.parseInt` range is
/// representable exactly (binary manifests always yield 0..=255).
pub group_num: i32,
/// Uncompressed size (0 for JSON manifests).
pub window_size: i32,
/// Compressed (wire) size. Signed like the Java `long`; JSON manifests may yield anything.
pub file_size: i64,
}
impl ChunkInfo {
/// `String.format("%08X%08X%08X%08X", guid[0..4])`.
pub fn guid_str(&self) -> String {
guid_to_string(&self.guid)
}
/// `ChunkInfo.getPath`: `"<chunkDir>/<%02d groupNum>/<%016X hash>_<GUID>.chunk"`.
pub fn path(&self, chunk_dir: &str) -> String {
format!(
"{}/{:02}/{:016X}_{}.chunk",
chunk_dir,
self.group_num,
self.hash,
self.guid_str()
)
}
/// Java `chunk.sha1 != null && length == 20 && !allZero` — the condition under which the
/// streaming downloader verifies the decompressed chunk.
pub fn verifiable_sha1(&self) -> Option<&[u8; 20]> {
match &self.sha1 {
Some(h) if h.iter().any(|b| *b != 0) => Some(h),
_ => None,
}
}
/// `Math.max(chunk.fileSize, 1)` — the byte credit the Java pool uses for totals/progress.
pub fn credit_bytes(&self) -> u64 {
if self.file_size < 1 {
1
} else {
self.file_size as u64
}
}
}
/// One file part referencing a chunk (`EpicDownloadManager.ChunkPart`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ChunkPart {
pub guid: [u32; 4],
/// Offset into the DECOMPRESSED chunk (Java `int`).
pub offset: i32,
/// Part length (Java `int`; summed as `size & 0xFFFFFFFFL`).
pub size: i32,
}
impl ChunkPart {
pub fn guid_str(&self) -> String {
guid_to_string(&self.guid)
}
}
/// One entry of the FileManifestList section (`EpicDownloadManager.FileInfo`).
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct FileInfo {
pub filename: String,
pub parts: Vec<ChunkPart>,
/// Install-tag names; empty = required/base file.
pub install_tags: Vec<String>,
/// Full-file SHA-1; `None` for JSON manifests.
pub sha1: Option<[u8; 20]>,
}
impl FileInfo {
/// `FileInfo.fileSize()` = Σ `(part.size & 0xFFFFFFFFL)`.
pub fn file_size(&self) -> u64 {
self.parts.iter().map(|p| p.size as u32 as u64).sum()
}
}
/// Parsed manifest (`EpicDownloadManager.EpicManifest.ParsedManifest`, minus the CDN list which
/// the Java side still owns).
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct Manifest {
pub chunk_dir: String,
pub unique_chunks: Vec<ChunkInfo>,
pub files: Vec<FileInfo>,
}
impl Manifest {
/// `byGuid` map as `uniqueChunksForFiles` builds it: `LinkedHashMap.put` per chunk, so a
/// duplicate GUID resolves to the LAST `ChunkInfo`.
pub fn chunk_index_by_guid(&self) -> HashMap<String, usize> {
let mut map = HashMap::with_capacity(self.unique_chunks.len());
for (i, c) in self.unique_chunks.iter().enumerate() {
map.insert(c.guid_str(), i);
}
map
}
}
pub fn guid_to_string(guid: &[u32; 4]) -> String {
format!("{:08X}{:08X}{:08X}{:08X}", guid[0], guid[1], guid[2], guid[3])
}
/// `chunkDir` from the binary header version (`parseManifest`).
pub fn chunk_dir_for_binary_version(version: i32) -> &'static str {
if version >= 15 {
"ChunksV4"
} else if version >= 6 {
"ChunksV3"
} else if version >= 3 {
"ChunksV2"
} else {
"Chunks"
}
}
/// `chunkDir` from `ManifestFileVersion` (`parseJsonManifest`; note the different `< 3` default).
pub fn chunk_dir_for_json_version(version: i32) -> &'static str {
if version >= 15 {
"ChunksV4"
} else if version >= 6 {
"ChunksV3"
} else if version >= 3 {
"ChunksV2"
} else {
"ChunksV4"
}
}
// ── little-endian cursor with ByteBuffer-like failure semantics ─────────────────────────────
/// Mirrors the `ByteBuffer` calls the Java parser makes: every under-run or out-of-range
/// `position()` is an exception there, which the Java code turns into "parse failed" (null).
struct Cursor<'a> {
buf: &'a [u8],
pos: usize,
}
impl<'a> Cursor<'a> {
fn new(buf: &'a [u8]) -> Self {
Self { buf, pos: 0 }
}
fn position(&self) -> usize {
self.pos
}
/// `ByteBuffer.position(p)` — `p` must be within `0..=limit`.
fn set_position(&mut self, p: i64) -> Result<(), String> {
if p < 0 || p as u64 > self.buf.len() as u64 {
return Err(format!("position {p} out of range (limit {})", self.buf.len()));
}
self.pos = p as usize;
Ok(())
}
fn take(&mut self, n: usize) -> Result<&'a [u8], String> {
let end = self
.pos
.checked_add(n)
.ok_or_else(|| "buffer overflow".to_string())?;
if end > self.buf.len() {
return Err(format!(
"buffer underflow: need {n} at {} (limit {})",
self.pos,
self.buf.len()
));
}
let out = &self.buf[self.pos..end];
self.pos = end;
Ok(out)
}
fn u8(&mut self) -> Result<u8, String> {
Ok(self.take(1)?[0])
}
fn i32(&mut self) -> Result<i32, String> {
let b = self.take(4)?;
Ok(i32::from_le_bytes([b[0], b[1], b[2], b[3]]))
}
fn u32(&mut self) -> Result<u32, String> {
Ok(self.i32()? as u32)
}
fn u64(&mut self) -> Result<u64, String> {
let b = self.take(8)?;
Ok(u64::from_le_bytes([
b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7],
]))
}
fn sha1(&mut self) -> Result<[u8; 20], String> {
let b = self.take(20)?;
let mut out = [0u8; 20];
out.copy_from_slice(b);
Ok(out)
}
/// `ByteBuffer.position(position + n)` with the same range check.
fn skip(&mut self, n: i64) -> Result<(), String> {
let target = self.pos as i64 + n;
self.set_position(target)
}
fn remaining(&self) -> &'a [u8] {
&self.buf[self.pos..]
}
/// `EpicDownloadManager.readFString`: int32 length; `0` → ""; negative → `(-len)-1` UTF-16LE
/// code units + 2-byte terminator; positive → `len-1` US-ASCII bytes + 1-byte terminator.
fn fstring(&mut self) -> Result<String, String> {
let length = self.i32()?;
if length == 0 {
return Ok(String::new());
}
if length < 0 {
// Java: `int chars = (-length) - 1;` — `-Integer.MIN_VALUE` wraps to itself, then -1
// gives Integer.MAX_VALUE; `new byte[chars * 2]` overflows negative → exception.
let chars = length.wrapping_neg().wrapping_sub(1);
let byte_len = chars.checked_mul(2).ok_or("fstring length overflow")?;
if byte_len < 0 {
return Err("negative fstring length".to_string());
}
let bytes = self.take(byte_len as usize)?;
self.take(2)?; // null terminator
let units: Vec<u16> = bytes
.chunks_exact(2)
.map(|p| u16::from_le_bytes([p[0], p[1]]))
.collect();
Ok(String::from_utf16_lossy(&units))
} else {
let bytes = self.take((length - 1) as usize)?;
self.take(1)?; // null terminator
// `new String(bytes, US_ASCII)`: every byte > 0x7F decodes to U+FFFD.
Ok(bytes
.iter()
.map(|&b| if b < 0x80 { b as char } else { '\u{FFFD}' })
.collect())
}
}
}
// ── binary manifest ──────────────────────────────────────────────────────────────────────
/// `EpicDownloadManager.parseManifest(bytes)`: binary when the magic matches, otherwise the JSON
/// parser. `Err` wherever Java returns `null`.
pub fn parse_manifest(bytes: &[u8]) -> Result<Manifest, String> {
let mut cur = Cursor::new(bytes);
let magic = cur.u32()?;
if magic != MANIFEST_MAGIC {
return parse_json_manifest(bytes);
}
parse_binary_after_magic(&mut cur)
}
fn parse_binary_after_magic(cur: &mut Cursor<'_>) -> Result<Manifest, String> {
let header_size = cur.i32()?;
let size_uncompressed = cur.i32()?;
let _size_compressed = cur.i32()?;
cur.skip(20)?; // SHA-1 of the body, not checked by the Java parser
let stored_as = cur.u8()?;
let version = cur.i32()?;
let chunk_dir = chunk_dir_for_binary_version(version);
cur.set_position(header_size as i64)?;
let body_slice = cur.remaining();
let inflated;
let body: &[u8] = if (stored_as & 1) != 0 {
if size_uncompressed < 0 {
return Err("negative sizeUncompressed".to_string());
}
// Java: `inflater.inflate(new byte[sizeUncompressed])` — a single call that fills at most
// `sizeUncompressed` bytes; anything else (short, corrupt) is a size mismatch → null.
let mut out = Vec::with_capacity(size_uncompressed as usize);
let mut dec = flate2::read::ZlibDecoder::new(body_slice).take(size_uncompressed as u64);
dec.read_to_end(&mut out)
.map_err(|e| format!("manifest inflate failed: {e}"))?;
if out.len() != size_uncompressed as usize {
return Err(format!(
"decomp size mismatch: expected {size_uncompressed} got {}",
out.len()
));
}
inflated = out;
&inflated
} else {
body_slice
};
let mut b = Cursor::new(body);
// ManifestMeta: skipped by its declared size (the size field counts itself).
let meta_size = b.i32()?;
let meta_end = b.position() as i64 - 4 + meta_size as i64;
b.set_position(meta_end)?;
// ChunkDataList
let cdl_start = b.position();
let cdl_size = b.i32()?;
let _cdl_version = b.u8()?;
let chunk_count = b.i32()?;
if chunk_count < 0 {
return Err("negative chunkCount".to_string()); // `new ArrayList<>(negative)` throws
}
let chunk_count = chunk_count as usize;
let mut chunks: Vec<ChunkInfo> = (0..chunk_count)
.map(|_| ChunkInfo {
guid: [0; 4],
hash: 0,
sha1: None,
group_num: 0,
window_size: 0,
file_size: 0,
})
.collect();
for c in chunks.iter_mut() {
c.guid = [b.u32()?, b.u32()?, b.u32()?, b.u32()?];
}
for c in chunks.iter_mut() {
c.hash = b.u64()?;
}
for c in chunks.iter_mut() {
c.sha1 = Some(b.sha1()?);
}
for c in chunks.iter_mut() {
c.group_num = b.u8()? as i32;
}
for c in chunks.iter_mut() {
c.window_size = b.i32()?;
}
for c in chunks.iter_mut() {
c.file_size = b.u64()? as i64;
}
b.set_position(cdl_start as i64 + cdl_size as i64)?;
// FileManifestList
let fml_start = b.position();
let fml_size = b.i32()?;
let _fml_version = b.u8()?;
let file_count = b.i32()?;
if file_count < 0 {
return Err("negative fileCount".to_string());
}
let file_count = file_count as usize;
let mut files: Vec<FileInfo> = (0..file_count).map(|_| FileInfo::default()).collect();
for f in files.iter_mut() {
f.filename = b.fstring()?;
}
for _ in 0..file_count {
b.fstring()?; // symlink targets
}
for f in files.iter_mut() {
f.sha1 = Some(b.sha1()?);
}
b.skip(file_count as i64)?; // per-file flags byte
for f in files.iter_mut() {
let tag_count = b.i32()?;
let mut j = 0;
while j < tag_count {
let tag = b.fstring()?;
if !tag.is_empty() {
f.install_tags.push(tag);
}
j += 1;
}
}
for f in files.iter_mut() {
let part_count = b.i32()?;
let mut j = 0;
while j < part_count {
let part_start = b.position();
let part_struct_size = b.i32()?;
let guid = [b.u32()?, b.u32()?, b.u32()?, b.u32()?];
let offset = b.i32()?;
let size = b.i32()?;
f.parts.push(ChunkPart { guid, offset, size });
b.set_position(part_start as i64 + part_struct_size as i64)?;
j += 1;
}
}
b.set_position(fml_start as i64 + fml_size as i64)?;
Ok(Manifest {
chunk_dir: chunk_dir.to_string(),
unique_chunks: dedup_last_wins_first_position(chunks),
files,
})
}
/// `LinkedHashMap<String, ChunkInfo>` keyed by GUID string: a repeated key keeps its FIRST
/// position but takes the LAST value.
fn dedup_last_wins_first_position(chunks: Vec<ChunkInfo>) -> Vec<ChunkInfo> {
let mut order: Vec<ChunkInfo> = Vec::with_capacity(chunks.len());
let mut index: HashMap<String, usize> = HashMap::with_capacity(chunks.len());
for c in chunks {
let key = c.guid_str();
match index.get(&key).copied() {
Some(i) => order[i] = c,
None => {
index.insert(key, order.len());
order.push(c);
}
}
}
order
}
// ── JSON manifest (older games) ──────────────────────────────────────────────────────────
/// `org.json` value → string coercion as `optString` / `getString` do on Android (numbers and
/// booleans print as-is, JSON null prints "null", containers print their JSON).
fn json_to_string(v: &serde_json::Value) -> String {
match v {
serde_json::Value::String(s) => s.clone(),
serde_json::Value::Null => "null".to_string(),
other => other.to_string(),
}
}
/// `obj.optString(key, fallback)`.
fn opt_string(obj: &serde_json::Map<String, serde_json::Value>, key: &str, fallback: &str) -> String {
match obj.get(key) {
Some(v) => json_to_string(v),
None => fallback.to_string(),
}
}
/// `(int) Long.parseLong(hex8, 16)` — the Java GUID quarter parse; invalid hex throws out of the
/// whole parse.
fn parse_guid_quarter(s: &str) -> Result<u32, String> {
let v = i64::from_str_radix(s, 16).map_err(|e| format!("bad guid hex '{s}': {e}"))?;
Ok(v as i32 as u32)
}
fn parse_guid_from_hex(s: &str) -> Result<[u32; 4], String> {
// Java substrings by UTF-16 unit and `parseLong` throws on anything non-hex; a non-ASCII
// prefix can never parse, so refuse it before slicing (keeps the slice on char boundaries).
if s.len() < 32 || !s.as_bytes()[..32].is_ascii() {
return Err(format!("bad guid hex '{s}'"));
}
Ok([
parse_guid_quarter(&s[0..8])?,
parse_guid_quarter(&s[8..16])?,
parse_guid_quarter(&s[16..24])?,
parse_guid_quarter(&s[24..32])?,
])
}
/// `EpicDownloadManager.parseJsonManifest`.
pub fn parse_json_manifest(bytes: &[u8]) -> Result<Manifest, String> {
let text = String::from_utf8_lossy(bytes);
let root: serde_json::Value =
serde_json::from_str(&text).map_err(|e| format!("JSON manifest: {e}"))?;
let root = root
.as_object()
.ok_or_else(|| "JSON manifest: root is not an object".to_string())?;
let manifest_version: i32 = opt_string(root, "ManifestFileVersion", "0")
.parse::<i32>()
.unwrap_or(0);
let chunk_dir = chunk_dir_for_json_version(manifest_version);
let chunk_hash_list = root
.get("ChunkHashList")
.and_then(|v| v.as_object())
.ok_or_else(|| "JSON manifest: no ChunkHashList".to_string())?;
let data_group_list = root.get("DataGroupList").and_then(|v| v.as_object());
let chunk_filesize_list = root.get("ChunkFilesizeList").and_then(|v| v.as_object());
// `LinkedHashMap<String, ChunkInfo>` keyed by the RAW key string (not the normalized GUID).
let mut chunk_order: Vec<ChunkInfo> = Vec::new();
let mut chunk_index: HashMap<String, usize> = HashMap::new();
for (guid_hex, hash_value) in chunk_hash_list.iter() {
if guid_hex.len() < 32 || !guid_hex.is_char_boundary(32) {
continue;
}
let hash_hex = json_to_string(hash_value);
let guid = parse_guid_from_hex(guid_hex)?;
let hash = if hash_hex.len() >= 16 && hash_hex.is_char_boundary(16) {
u64::from_str_radix(&hash_hex[0..16], 16).unwrap_or(0)
} else {
0
};
let group_num = match data_group_list {
Some(dgl) => opt_string(dgl, guid_hex, "0").parse::<i32>().unwrap_or(0),
None => 0,
};
let file_size = match chunk_filesize_list {
Some(cfl) => i64::from_str_radix(&opt_string(cfl, guid_hex, "0"), 16).unwrap_or(0),
None => 0,
};
let c = ChunkInfo {
guid,
hash,
sha1: None,
group_num,
window_size: 0,
file_size,
};
match chunk_index.get(guid_hex).copied() {
Some(i) => chunk_order[i] = c,
None => {
chunk_index.insert(guid_hex.clone(), chunk_order.len());
chunk_order.push(c);
}
}
}
let file_list = root
.get("FileManifestList")
.and_then(|v| v.as_array())
.ok_or_else(|| "JSON manifest: no FileManifestList".to_string())?;
let mut files = Vec::with_capacity(file_list.len());
for entry in file_list {
let file_obj = entry
.as_object()
.ok_or_else(|| "JSON manifest: FileManifestList entry is not an object".to_string())?;
let mut fi = FileInfo {
filename: opt_string(file_obj, "Filename", ""),
..FileInfo::default()
};
if let Some(tags) = file_obj.get("InstallTags").and_then(|v| v.as_array()) {
for t in tags {
let tag = json_to_string(t);
if !tag.is_empty() {
fi.install_tags.push(tag);
}
}
}
if let Some(parts) = file_obj.get("FileChunkParts").and_then(|v| v.as_array()) {
for p in parts {
let part_obj = p.as_object().ok_or_else(|| {
"JSON manifest: FileChunkParts entry is not an object".to_string()
})?;
let mut part = ChunkPart {
guid: [0; 4],
offset: 0,
size: 0,
};
let part_guid = opt_string(part_obj, "Guid", "");
if part_guid.len() >= 32 && part_guid.is_char_boundary(32) {
part.guid = parse_guid_from_hex(&part_guid)?;
}
part.offset = opt_string(part_obj, "Offset", "0").parse::<i32>().unwrap_or(0);
part.size = opt_string(part_obj, "Size", "0").parse::<i32>().unwrap_or(0);
fi.parts.push(part);
}
}
files.push(fi);
}
Ok(Manifest {
chunk_dir: chunk_dir.to_string(),
unique_chunks: chunk_order,
files,
})
}
// ── tests ────────────────────────────────────────────────────────────────────────────────
#[cfg(test)]
pub(crate) mod test_support {
//! Synthetic binary-manifest builder used by this module's tests and by `plan`/`driver`
//! tests. Emits exactly the layout `parseManifest` walks (meta / CDL / FML with sizes).
use super::*;
use std::io::Write;
pub struct TestChunk {
pub guid: [u32; 4],
pub hash: u64,
pub sha1: [u8; 20],
pub group: u8,
pub window: i32,
pub file_size: u64,
}
pub struct TestFile {
pub name: String,
pub sha1: [u8; 20],
pub tags: Vec<String>,
/// (guid, offset, size)
pub parts: Vec<([u32; 4], i32, i32)>,
}
pub fn fstring_ascii(s: &str) -> Vec<u8> {
let mut v = Vec::new();
if s.is_empty() {
v.extend_from_slice(&0i32.to_le_bytes());
return v;
}
v.extend_from_slice(&((s.len() as i32) + 1).to_le_bytes());
v.extend_from_slice(s.as_bytes());
v.push(0);
v
}
pub fn fstring_utf16(s: &str) -> Vec<u8> {
let units: Vec<u16> = s.encode_utf16().collect();
let mut v = Vec::new();
v.extend_from_slice(&(-((units.len() as i32) + 1)).to_le_bytes());
for u in units {
v.extend_from_slice(&u.to_le_bytes());
}
v.extend_from_slice(&[0, 0]);
v
}
/// Body (meta + CDL + FML) without the outer header.
pub fn build_body(chunks: &[TestChunk], files: &[TestFile], meta_padding: usize) -> Vec<u8> {
let mut body = Vec::new();
// ManifestMeta: size field counts itself.
let meta_size = 4 + meta_padding;
body.extend_from_slice(&(meta_size as i32).to_le_bytes());
body.extend(std::iter::repeat(0xEEu8).take(meta_padding));
// ChunkDataList
let mut cdl = Vec::new();
cdl.push(0u8); // version
cdl.extend_from_slice(&(chunks.len() as i32).to_le_bytes());
for c in chunks {
for g in c.guid {
cdl.extend_from_slice(&g.to_le_bytes());
}
}
for c in chunks {
cdl.extend_from_slice(&c.hash.to_le_bytes());
}
for c in chunks {
cdl.extend_from_slice(&c.sha1);
}
for c in chunks {
cdl.push(c.group);
}
for c in chunks {
cdl.extend_from_slice(&c.window.to_le_bytes());
}
for c in chunks {
cdl.extend_from_slice(&c.file_size.to_le_bytes());
}
// Trailing padding the parser must skip via cdlSize.
cdl.extend_from_slice(&[0xAA, 0xBB, 0xCC]);
body.extend_from_slice(&((cdl.len() + 4) as i32).to_le_bytes());
body.extend_from_slice(&cdl);
// FileManifestList
let mut fml = Vec::new();
fml.push(0u8); // version
fml.extend_from_slice(&(files.len() as i32).to_le_bytes());
for f in files {
if f.name.is_ascii() {
fml.extend_from_slice(&fstring_ascii(&f.name));
} else {
fml.extend_from_slice(&fstring_utf16(&f.name));
}
}
for _ in files {
fml.extend_from_slice(&fstring_ascii("")); // symlink target
}
for f in files {
fml.extend_from_slice(&f.sha1);
}
for _ in files {
fml.push(0); // flags
}
for f in files {
fml.extend_from_slice(&(f.tags.len() as i32).to_le_bytes());
for t in &f.tags {
fml.extend_from_slice(&fstring_ascii(t));
}
}
for f in files {
fml.extend_from_slice(&(f.parts.len() as i32).to_le_bytes());
for (guid, offset, size) in &f.parts {
// struct size 4 + 16 + 4 + 4 = 28, plus 2 bytes of padding the parser skips.
fml.extend_from_slice(&30i32.to_le_bytes());
for g in guid {
fml.extend_from_slice(&g.to_le_bytes());
}
fml.extend_from_slice(&offset.to_le_bytes());
fml.extend_from_slice(&size.to_le_bytes());
fml.extend_from_slice(&[0x11, 0x22]);
}
}
body.extend_from_slice(&((fml.len() + 4) as i32).to_le_bytes());
body.extend_from_slice(&fml);
body
}
/// Full manifest bytes: header + (optionally zlib-compressed) body.
pub fn build_manifest(
chunks: &[TestChunk],
files: &[TestFile],
version: i32,
compress: bool,
) -> Vec<u8> {
let body = build_body(chunks, files, 7);
let payload = if compress {
let mut enc =
flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::default());
enc.write_all(&body).unwrap();
enc.finish().unwrap()
} else {
body.clone()
};
let header_size: i32 = 41 + 3; // 3 bytes of extra header the parser must jump over
let mut out = Vec::new();
out.extend_from_slice(&MANIFEST_MAGIC.to_le_bytes());
out.extend_from_slice(&header_size.to_le_bytes());
out.extend_from_slice(&(body.len() as i32).to_le_bytes());
out.extend_from_slice(&(payload.len() as i32).to_le_bytes());
out.extend_from_slice(&[0u8; 20]);
out.push(if compress { 1 } else { 0 });
out.extend_from_slice(&version.to_le_bytes());
out.extend_from_slice(&[0xF0, 0xF1, 0xF2]);
assert_eq!(out.len(), header_size as usize);
out.extend_from_slice(&payload);
out
}
pub fn sample_chunks() -> Vec<TestChunk> {
vec![
TestChunk {
guid: [0x0000_0001, 0x0000_0002, 0x0000_0003, 0x0000_0004],
hash: 0x0123_4567_89AB_CDEF,
sha1: [0x11; 20],
group: 5,
window: 1_048_576,
file_size: 700_000,
},
TestChunk {
guid: [0xDEAD_BEEF, 0xFFFF_FFFF, 0x0000_0000, 0x8000_0000],
hash: 0xFFFF_FFFF_FFFF_FFFE,
sha1: [0x22; 20],
group: 123,
window: 4096,
file_size: 0,
},
TestChunk {
guid: [0x0000_00AA, 0x0000_00BB, 0x0000_00CC, 0x0000_00DD],
hash: 0x10,
sha1: [0u8; 20],
group: 0,
window: 10,
file_size: 9,
},
]
}
pub fn sample_files() -> Vec<TestFile> {
let c = sample_chunks();
vec![
TestFile {
name: "Game/Binaries/Win64/Game.exe".to_string(),
sha1: [0x51; 20],
tags: vec![],
parts: vec![(c[0].guid, 0, 1_048_576), (c[1].guid, 100, 4000)],
},
TestFile {
name: "Game\\Content\\Localization\\de.pak".to_string(),
sha1: [0x52; 20],
tags: vec!["German".to_string(), "de-DE".to_string()],
parts: vec![(c[1].guid, 0, 4096)],
},
TestFile {
name: "Game/Content/Ünïcode.pak".to_string(),
sha1: [0x53; 20],
tags: vec!["French".to_string()],
parts: vec![(c[2].guid, 0, 10), (c[0].guid, 5, 7)],
},
]
}
}
#[cfg(test)]
mod tests {
use super::test_support::*;
use super::*;
#[test]
fn binary_manifest_round_trips_uncompressed() {
let bytes = build_manifest(&sample_chunks(), &sample_files(), 21, false);
let m = parse_manifest(&bytes).expect("parse");
assert_eq!(m.chunk_dir, "ChunksV4");
assert_eq!(m.unique_chunks.len(), 3);
assert_eq!(m.files.len(), 3);
let c0 = &m.unique_chunks[0];
assert_eq!(c0.guid_str(), "00000001000000020000000300000004");
assert_eq!(c0.hash, 0x0123_4567_89AB_CDEF);
assert_eq!(c0.sha1, Some([0x11; 20]));
assert_eq!(c0.group_num, 5);
assert_eq!(c0.window_size, 1_048_576);
assert_eq!(c0.file_size, 700_000);
assert_eq!(
c0.path("ChunksV4"),
"ChunksV4/05/0123456789ABCDEF_00000001000000020000000300000004.chunk"
);
let c1 = &m.unique_chunks[1];
assert_eq!(c1.guid_str(), "DEADBEEFFFFFFFFF0000000080000000");
assert_eq!(
c1.path("ChunksV4"),
"ChunksV4/123/FFFFFFFFFFFFFFFE_DEADBEEFFFFFFFFF0000000080000000.chunk"
);
assert_eq!(c1.credit_bytes(), 1, "Math.max(fileSize, 1)");
assert!(c1.verifiable_sha1().is_some());
assert!(m.unique_chunks[2].verifiable_sha1().is_none(), "all-zero SHA-1 → unverifiable");
let f0 = &m.files[0];
assert_eq!(f0.filename, "Game/Binaries/Win64/Game.exe");
assert!(f0.install_tags.is_empty());
assert_eq!(f0.sha1, Some([0x51; 20]));
assert_eq!(f0.parts.len(), 2);
assert_eq!(f0.parts[1].offset, 100);
assert_eq!(f0.parts[1].size, 4000);
assert_eq!(f0.file_size(), 1_048_576 + 4000);
let f1 = &m.files[1];
assert_eq!(f1.filename, "Game\\Content\\Localization\\de.pak");
assert_eq!(f1.install_tags, vec!["German".to_string(), "de-DE".to_string()]);
let f2 = &m.files[2];
assert_eq!(f2.filename, "Game/Content/Ünïcode.pak", "UTF-16LE fstring");
assert_eq!(f2.parts[1].guid_str(), c0.guid_str());
}
#[test]
fn binary_manifest_round_trips_compressed() {
let bytes = build_manifest(&sample_chunks(), &sample_files(), 17, true);
let m = parse_manifest(&bytes).expect("parse");
assert_eq!(m.unique_chunks.len(), 3);
assert_eq!(m.files.len(), 3);
assert_eq!(m.files[2].filename, "Game/Content/Ünïcode.pak");
}
#[test]
fn chunk_dir_follows_version_thresholds() {
for (v, dir) in [(2, "Chunks"), (3, "ChunksV2"), (6, "ChunksV3"), (15, "ChunksV4")] {
let bytes = build_manifest(&sample_chunks(), &sample_files(), v, false);
assert_eq!(parse_manifest(&bytes).unwrap().chunk_dir, dir, "version {v}");
}
assert_eq!(chunk_dir_for_json_version(1), "ChunksV4");
assert_eq!(chunk_dir_for_json_version(4), "ChunksV2");
}
#[test]
fn duplicate_guid_keeps_first_position_last_value() {
let mut chunks = sample_chunks();
let dup = TestChunk {
guid: chunks[0].guid,
hash: 0x99,
sha1: [0x77; 20],
group: 9,
window: 1,
file_size: 1,
};
chunks.push(dup);
let bytes = build_manifest(&chunks, &sample_files(), 21, false);
let m = parse_manifest(&bytes).unwrap();
assert_eq!(m.unique_chunks.len(), 3);
assert_eq!(m.unique_chunks[0].hash, 0x99);
assert_eq!(m.unique_chunks[0].group_num, 9);
}
#[test]
fn truncated_manifest_is_a_parse_failure() {
let bytes = build_manifest(&sample_chunks(), &sample_files(), 21, false);
assert!(parse_manifest(&bytes[..bytes.len() - 40]).is_err());
assert!(parse_manifest(&bytes[..20]).is_err());
}
#[test]
fn bad_uncompressed_size_is_a_parse_failure() {
let good = build_manifest(&sample_chunks(), &sample_files(), 21, true);
let real = i32::from_le_bytes([good[8], good[9], good[10], good[11]]);
// Claim MORE than the body inflates to: Java's single `inflate()` returns fewer bytes
// than `sizeUncompressed` → "Decomp size mismatch" → null.
let mut bigger = good.clone();
bigger[8..12].copy_from_slice(&(real + 1).to_le_bytes());
assert!(parse_manifest(&bigger).is_err());
// Claim LESS: Java fills exactly `sizeUncompressed` bytes (silently truncating), then the
// FileManifestList end position lands past the limit → exception → null.
let mut smaller = good;
smaller[8..12].copy_from_slice(&(real - 1).to_le_bytes());
assert!(parse_manifest(&smaller).is_err());
}
#[test]
fn fstring_decoding_matches_java() {
let mut v = fstring_ascii("abc");
v.extend_from_slice(&fstring_utf16("dé"));
v.extend_from_slice(&fstring_ascii(""));
v.extend_from_slice(&[4, 0, 0, 0, b'a', 0xFF, b'b', 0]); // non-ASCII byte → U+FFFD
let mut c = Cursor::new(&v);
assert_eq!(c.fstring().unwrap(), "abc");
assert_eq!(c.fstring().unwrap(), "dé");
assert_eq!(c.fstring().unwrap(), "");
assert_eq!(c.fstring().unwrap(), "a\u{FFFD}b");
}
#[test]
fn json_manifest_parses_like_java() {
let json = r#"{
"ManifestFileVersion": "013",
"ChunkHashList": {
"0000000100000002000000030000000ff": "0123456789ABCDEF00",
"short": "00",
"DEADBEEFFFFFFFFF0000000080000000": "zz"
},
"DataGroupList": { "0000000100000002000000030000000ff": "7", "DEADBEEFFFFFFFFF0000000080000000": "x" },
"ChunkFilesizeList": { "0000000100000002000000030000000ff": "FF" },
"FileManifestList": [
{ "Filename": "a.bin", "InstallTags": ["German", ""],
"FileChunkParts": [ { "Guid": "0000000100000002000000030000000ff", "Offset": "12", "Size": "34" },
{ "Guid": "tooshort", "Offset": "x", "Size": "-5" } ] },
{ "Filename": "b.bin" }
]
}"#;
let m = parse_manifest(json.as_bytes()).expect("json parse");
assert_eq!(m.chunk_dir, "ChunksV3");
assert_eq!(m.unique_chunks.len(), 2, "keys shorter than 32 chars are skipped");
let c0 = &m.unique_chunks[0];
assert_eq!(c0.guid, [1, 2, 3, 0x0000_000F]);
assert_eq!(c0.hash, 0x0123_4567_89AB_CDEF);
assert_eq!(c0.group_num, 7);
assert_eq!(c0.file_size, 0xFF);
assert_eq!(c0.window_size, 0);
assert!(c0.sha1.is_none());
let c1 = &m.unique_chunks[1];
assert_eq!(c1.hash, 0, "unparseable hash → 0");
assert_eq!(c1.group_num, 0, "unparseable group → 0");
assert_eq!(c1.file_size, 0, "missing size → 0");
assert_eq!(m.files.len(), 2);
assert_eq!(m.files[0].install_tags, vec!["German".to_string()]);
assert_eq!(m.files[0].parts.len(), 2);
assert_eq!(m.files[0].parts[0].offset, 12);
assert_eq!(m.files[0].parts[0].size, 34);
assert_eq!(m.files[0].parts[1].guid, [0; 4]);
assert_eq!(m.files[0].parts[1].offset, 0);
assert_eq!(m.files[0].parts[1].size, -5);
assert!(m.files[0].sha1.is_none());
assert!(m.files[1].parts.is_empty());
}
#[test]
fn json_manifest_failures_match_java_nulls() {
assert!(parse_manifest(br#"{"FileManifestList": []}"#).is_err(), "no ChunkHashList");
assert!(parse_manifest(br#"{"ChunkHashList": {}}"#).is_err(), "no FileManifestList");
assert!(
parse_manifest(br#"{"ChunkHashList": {"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz": "0"}, "FileManifestList": []}"#)
.is_err(),
"invalid GUID hex throws out of the parse"
);
assert!(parse_manifest(b"not json at all").is_err());
}
}
+289
View File
@@ -0,0 +1,289 @@
//! Chunk plan — the part of `EpicDownloadManager.install` between "manifest parsed" and "submit
//! to the pool", ported 1:1: install-tag file selection (`resolveInstallFiles` /
//! `getRequiredInstallFiles`), the unique-chunk list for a file set (`uniqueChunksForFiles`),
//! the byte total (`Σ max(fileSize, 1)`), chunk cache paths and CDN URLs.
//!
//! At runtime the Java manager passes the INDICES of its `pendingFiles` (post delta/verify), so
//! the file selection functions here are exercised by the unit tests (and available to a future
//! caller) while the driver only runs `unique_chunks_for_files` on Java's pending set — the same
//! function Java runs, on the same manifest, in the same order.
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf};
use super::manifest::{ChunkInfo, Manifest};
/// `getRequiredInstallFiles`: files with no install tags; if NONE carries a tag, every file.
pub fn required_install_files(m: &Manifest) -> Vec<usize> {
let required: Vec<usize> = m
.files
.iter()
.enumerate()
.filter(|(_, f)| f.install_tags.is_empty())
.map(|(i, _)| i)
.collect();
if required.is_empty() {
(0..m.files.len()).collect()
} else {
required
}
}
/// `resolveInstallFiles(manifest, selectedTags)`:
/// - `None` → every file (legacy / DLC / overlay path);
/// - `Some([])` → required(base) files only;
/// - otherwise → base files + any file carrying at least one selected tag, falling back to
/// required-only when nothing matched.
pub fn resolve_install_files(m: &Manifest, selected_tags: Option<&[String]>) -> Vec<usize> {
let Some(tags) = selected_tags else {
return (0..m.files.len()).collect();
};
if tags.is_empty() {
return required_install_files(m);
}
let want: HashSet<&str> = tags.iter().map(String::as_str).collect();
let mut out = Vec::new();
for (i, f) in m.files.iter().enumerate() {
if f.install_tags.is_empty() {
out.push(i);
continue;
}
if f.install_tags.iter().any(|t| want.contains(t.as_str())) {
out.push(i);
}
}
if out.is_empty() {
required_install_files(m)
} else {
out
}
}
/// `uniqueChunksForFiles(manifest, files)`: chunk indices (into `unique_chunks`) in first-seen
/// order over `files[*].parts`, each GUID once, GUIDs absent from the chunk list dropped.
pub fn unique_chunks_for_files(m: &Manifest, file_indices: &[usize]) -> Vec<usize> {
let by_guid: HashMap<String, usize> = m.chunk_index_by_guid();
let mut seen: HashSet<String> = HashSet::new();
let mut chunks = Vec::new();
for &fi in file_indices {
let Some(f) = m.files.get(fi) else {
continue;
};
for p in &f.parts {
let g = p.guid_str();
if seen.insert(g.clone()) {
if let Some(&ci) = by_guid.get(&g) {
chunks.push(ci);
}
}
}
}
chunks
}
/// `totalBytes += Math.max(chunk.fileSize, 1)` over the needed chunks.
pub fn total_credit_bytes(m: &Manifest, chunk_indices: &[usize]) -> u64 {
chunk_indices
.iter()
.filter_map(|&i| m.unique_chunks.get(i))
.map(ChunkInfo::credit_bytes)
.sum()
}
/// `<installDir>/.chunks`.
pub fn chunk_cache_dir(install_dir: &str) -> PathBuf {
Path::new(install_dir).join(super::CHUNK_CACHE_DIR)
}
/// Where this run's chunks go: `cache_dir` when the caller named one, else the default
/// `<installDir>/.chunks`. A separate directory exists so the cache can sit on internal storage
/// while the game installs to an SD card: a chunk is written once into the cache and once into
/// the assembled file, and with both on a 26 MB/s card the second write starved the fetchers.
pub fn resolve_cache_dir(install_dir: &str, cache_dir: &str) -> PathBuf {
if cache_dir.is_empty() {
chunk_cache_dir(install_dir)
} else {
PathBuf::from(cache_dir)
}
}
/// Final cache file for a chunk: `new File(chunkCacheDir, chunk.guidStr())`.
pub fn cached_chunk_path(cache_dir: &Path, chunk: &ChunkInfo) -> PathBuf {
cache_dir.join(chunk.guid_str())
}
/// The Java pool's skip rule: a chunk whose cache file EXISTS (any size) is not fetched.
pub fn is_chunk_cached(cache_dir: &Path, chunk: &ChunkInfo) -> bool {
cached_chunk_path(cache_dir, chunk).exists()
}
/// One CDN entry as the Java manager passes it: `cdn.baseUrl + cdn.cloudDir` (auth params are
/// never used on chunk URLs). Chunk URL = `prefix + "/" + chunk.getPath(chunkDir)`.
pub fn chunk_url(cdn_prefix: &str, chunk_dir: &str, chunk: &ChunkInfo) -> String {
format!("{}/{}", cdn_prefix, chunk.path(chunk_dir))
}
/// Undo the JSON string escaping the Java CDN scanner passes through. `EpicApiClient` re-serializes
/// the manifest API response with Android's `org.json`, which escapes every `/` as `\/`, and
/// `parseCdnUrls` takes the raw substrings — so a base URL arrives as `https:\/\/host\` and a cloud
/// dir as `/Builds\/Org\/…\/default\`. Java's `HttpURLConnection` (OkHttp) treats `\` as `/` and
/// only ever reached the first CDN, so it never noticed; reqwest's URL parser does the same
/// mapping, which leaves empty `//` path segments that Fastly/Akamai tolerate but CloudFront
/// rejects outright (immediate 4xx). Produce the URL Java intended: `\/` → `/`, any other `\`
/// dropped, and no trailing slash (the chunk path is joined with one).
pub fn normalize_prefix(raw: &str) -> String {
let unescaped = raw.replace("\\/", "/").replace('\\', "");
unescaped.trim_end_matches('/').to_string()
}
/// Distinct CDN prefixes in first-seen order, normalized (the fetch core wants one host key per
/// distinct endpoint; Java would simply try an identical URL twice, which changes nothing).
pub fn distinct_prefixes(cdn_prefixes: &[String]) -> Vec<String> {
let mut seen = HashSet::new();
let mut out = Vec::new();
for p in cdn_prefixes {
let n = normalize_prefix(p);
if n.is_empty() {
continue;
}
if seen.insert(n.clone()) {
out.push(n);
}
}
out
}
/// Improvements round 1: split the window across the distinct CDNs so the whole ceiling is
/// reachable whatever the host count — `max(6, ceil(max_workers / hosts))` (3 Epic CDNs at a
/// ceiling of 32 → 11 each; a single host gets all 32).
pub fn per_host_cap(max_workers: usize, host_count: usize) -> usize {
let hosts = host_count.max(1);
let workers = max_workers.max(1);
let split = (workers + hosts - 1) / hosts;
split.max(6)
}
#[cfg(test)]
mod tests {
use super::super::manifest::test_support::*;
use super::super::manifest::parse_manifest;
use super::*;
fn manifest() -> Manifest {
parse_manifest(&build_manifest(&sample_chunks(), &sample_files(), 21, false)).unwrap()
}
#[test]
fn install_tag_rules_match_java() {
let m = manifest();
// null → all
assert_eq!(resolve_install_files(&m, None), vec![0, 1, 2]);
// empty → required only (file 0 has no tags)
assert_eq!(resolve_install_files(&m, Some(&[])), vec![0]);
// German → base + the German file
let de = vec!["German".to_string(), "de-DE".to_string()];
assert_eq!(resolve_install_files(&m, Some(&de)), vec![0, 1]);
// unknown tag → nothing extra matched, but base files matched, so base only
let xx = vec!["Klingon".to_string()];
assert_eq!(resolve_install_files(&m, Some(&xx)), vec![0]);
}
#[test]
fn manifest_without_tags_treats_every_file_as_required() {
let mut files = sample_files();
for f in files.iter_mut() {
f.tags.clear();
}
let m = parse_manifest(&build_manifest(&sample_chunks(), &files, 21, false)).unwrap();
assert_eq!(required_install_files(&m), vec![0, 1, 2]);
assert_eq!(resolve_install_files(&m, Some(&[])), vec![0, 1, 2]);
}
#[test]
fn manifest_where_every_file_is_tagged_falls_back_to_all() {
let mut files = sample_files();
files[0].tags = vec!["English".to_string()];
let m = parse_manifest(&build_manifest(&sample_chunks(), &files, 21, false)).unwrap();
// No untagged file, unknown tag selected → matched nothing → required → (none) → all.
let xx = vec!["Klingon".to_string()];
assert_eq!(resolve_install_files(&m, Some(&xx)), vec![0, 1, 2]);
}
#[test]
fn unique_chunks_follow_first_seen_part_order() {
let m = manifest();
// file 2 first: chunk c[2] then c[0]; then file 0: c[0] (dup), c[1].
assert_eq!(unique_chunks_for_files(&m, &[2, 0]), vec![2, 0, 1]);
assert_eq!(unique_chunks_for_files(&m, &[1]), vec![1]);
assert!(unique_chunks_for_files(&m, &[]).is_empty());
assert!(unique_chunks_for_files(&m, &[99]).is_empty(), "out-of-range index ignored");
// total = 700000 + max(0,1) + 9
assert_eq!(total_credit_bytes(&m, &[0, 1, 2]), 700_000 + 1 + 9);
}
#[test]
fn parts_referencing_unknown_guids_are_dropped() {
let mut files = sample_files();
files[0]
.parts
.push(([0x1234, 0x5678, 0x9ABC, 0xDEF0], 0, 1));
let m = parse_manifest(&build_manifest(&sample_chunks(), &files, 21, false)).unwrap();
assert_eq!(unique_chunks_for_files(&m, &[0]), vec![0, 1]);
}
#[test]
fn urls_and_cache_paths_match_java() {
let m = manifest();
let c = &m.unique_chunks[0];
assert_eq!(
chunk_url("https://fastly-download.epicgames.com/Builds/Org/o-x/y/default", "ChunksV4", c),
"https://fastly-download.epicgames.com/Builds/Org/o-x/y/default/ChunksV4/05/0123456789ABCDEF_00000001000000020000000300000004.chunk"
);
let cache = chunk_cache_dir("/data/x/imagefs/epic_games/Game");
assert_eq!(cache, PathBuf::from("/data/x/imagefs/epic_games/Game/.chunks"));
assert_eq!(resolve_cache_dir("/data/x/imagefs/epic_games/Game", ""), cache);
assert_eq!(
resolve_cache_dir("/sdcard/Games/Game", "/data/x/cache/epic/Game"),
PathBuf::from("/data/x/cache/epic/Game")
);
assert_eq!(
cached_chunk_path(&cache, c),
PathBuf::from("/data/x/imagefs/epic_games/Game/.chunks/00000001000000020000000300000004")
);
let prefixes = vec!["a".to_string(), "b".to_string(), "a".to_string()];
assert_eq!(distinct_prefixes(&prefixes), vec!["a".to_string(), "b".to_string()]);
}
#[test]
fn prefixes_are_unescaped_from_the_java_json_scan() {
// Exactly what Bannerlator's Epic debug log shows its Java manager extracting for a real title.
let raw = "https:\\/\\/egdownload.fastly-edge.com\\/Builds\\/Org\\/o-83e8\\/04b7\\/default\\";
assert_eq!(
normalize_prefix(raw),
"https://egdownload.fastly-edge.com/Builds/Org/o-83e8/04b7/default"
);
// Already-clean input is untouched (minus a trailing slash).
assert_eq!(
normalize_prefix("https://download.epicgames.com/Builds/x/default/"),
"https://download.epicgames.com/Builds/x/default"
);
let m = manifest();
let c = &m.unique_chunks[0];
assert_eq!(
chunk_url(&normalize_prefix(raw), "ChunksV4", c),
"https://egdownload.fastly-edge.com/Builds/Org/o-83e8/04b7/default/ChunksV4/05/0123456789ABCDEF_00000001000000020000000300000004.chunk"
);
let two = vec![raw.to_string(), "https://egdownload.fastly-edge.com/Builds/Org/o-83e8/04b7/default".to_string(), String::new()];
assert_eq!(distinct_prefixes(&two).len(), 1, "escaped and clean forms collapse; empty dropped");
}
#[test]
fn per_host_cap_splits_the_window_with_a_floor_of_six() {
assert_eq!(per_host_cap(32, 3), 11);
assert_eq!(per_host_cap(32, 1), 32);
assert_eq!(per_host_cap(32, 2), 16);
assert_eq!(per_host_cap(8, 3), 6, "floor");
assert_eq!(per_host_cap(8, 1), 8);
assert_eq!(per_host_cap(0, 0), 6, "degenerate inputs");
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,211 @@
//! The numbers and the two small types [`fetch_core`](crate::fetch_core) shares with the Steam
//! depot writer it was carved out of.
//!
//! In Bannerlator these live in `depot_writer.rs` and `cdn_client.rs`, next to the Steam CM and
//! depot code, so that the Steam engine and the store engines run on ONE set of tunables and their
//! `fetch-window` log lines stay A/B-comparable. DroidDeck ships no Steam depot engine (the Steam
//! client runs natively in the Linux runtime), so only this slice came across: the values are
//! copied verbatim and keep their names, which is what lets a GOG/Epic/Amazon log from either app
//! be read against the same tuning notes.
//!
//! The important behaviours are structural (slow-start ramp -> hold at plateau -> back off on REAL
//! errors, per-host cap, hard byte budget), not the exact numbers. The r3/r4 notes below record
//! why each number is what it is, because every one of them was arrived at by a device run that
//! went wrong the other way.
use std::time::Duration;
/// Default `User-Agent` of the pooled client. Every store engine overrides it per request with
/// the agent its own launcher sends (`FetchOptions::headers`), so this value is only what a
/// request without an explicit header would carry.
pub const USER_AGENT: &str = "Valve/Steam HTTP Client 1.0";
/// TCP connect deadline for every request the pooled client makes.
pub fn connect_timeout() -> Duration {
Duration::from_secs(15)
}
/// Classified fetch failure so the fetch driver can back off proportionally (429/5xx/timeout/reset
/// shrink the window + cool the host harder than a one-off protocol error).
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum FetchFailKind {
Timeout,
RateLimited,
ServerFault,
Connect,
Other,
}
#[derive(Clone, Debug)]
pub struct AsyncFetchError {
pub message: String,
pub kind: FetchFailKind,
}
/// Attempts per item before the run gives up on it (fetch failures and sink retries both count).
pub const MAX_CHUNK_ATTEMPTS: u32 = 5;
/// Floor for the in-flight byte budget — the pre-r4 fixed value, so no run ever gets *less*
/// buffered memory than it had before the budget started scaling. See [`inflight_budget_bytes`].
pub const FETCH_INFLIGHT_BUDGET_FLOOR_BYTES: u64 = 24 * 1024 * 1024;
/// Hard ceiling for the in-flight byte budget. This is THE OOM guard: whatever the tier, the host
/// count or the chunk size, the RAW bytes in memory for one run can never exceed this. 96 MiB = 4×
/// the old fixed budget and still a small fraction of the heap an Android game installer runs in,
/// and it is only ever reached by a fast-tier run whose CDN pool is large enough to support a 64+
/// window. Raising this is the one change in the fetch layer that can reintroduce OOM, so it stays
/// fixed and small.
pub const FETCH_INFLIGHT_BUDGET_HARD_CAP_BYTES: u64 = 96 * 1024 * 1024;
/// Assumed RAW (on-the-wire, still-compressed) size of one chunk when sizing the budget. Store
/// chunks are ~1 MiB; this is the same nominal [`NOMINAL_CHUNK_RESERVE_BYTES`] uses at dispatch.
pub const TYPICAL_CHUNK_BYTES: u64 = NOMINAL_CHUNK_RESERVE_BYTES;
/// Headroom the budget carries over "one chunk per window slot" (3/2 = 1.5×), as a rational so the
/// arithmetic stays integer. The window is a count of concurrent *requests*; a slot's bytes are held
/// from DISPATCH until its WRITE completes, so at any instant some slots hold a fetched-but-not-yet-
/// written chunk while others hold a reservation for a fetch still on the wire. 1.5× covers that
/// overlap (and the occasional above-nominal chunk) without doubling the memory bound.
pub const BUDGET_HEADROOM_NUM: u64 = 3;
pub const BUDGET_HEADROOM_DEN: u64 = 2;
/// Byte budget for the *compressed* bytes buffered in the fetch→process channel, scaled to the
/// window ceiling this run can actually reach.
///
/// The decouple channel is bounded by BYTES, not chunk count: a count cap makes in-flight memory
/// swing wildly with chunk size (a 4-chunk cap is ~4 MB for 1 MB chunks but ~120 MB for 30 MB
/// chunks). Budgeting bytes keeps the heap bounded regardless of game size while giving the fetch
/// pool bandwidth-delay-product headroom so the socket stays busy while process workers
/// decompress+write.
///
/// r4: a device run proved a fixed 24 MiB budget becomes the binding constraint the moment the
/// adaptive window works — on a 1 Gbps line the window grew to 42 but the budget admitted only ~24
/// chunks, so `in_flight` stalled at 23–28 with `budget_stalls` in the hundreds and
/// `host_stalls=0`. A budget that refuses slots the window has already decided are safe is not a
/// memory guard, it is a throttle. So the budget scales WITH the effective ceiling (tier ∧
/// hosts×per-host-cap ∧ hard cap) and is clamped between the old fixed value and a hard cap. Low
/// tiers are unaffected; a fast link gets a budget that stops binding before the window does.
///
/// Budget only the RAW bytes actually queued (reserved at DISPATCH, released on WRITE-COMPLETE); the
/// decoded expansion is transient inside a process worker and never accumulates. See
/// [`budget_admits`] for the single-chunk deadlock guard.
pub fn inflight_budget_bytes(effective_ceiling: usize) -> u64 {
(effective_ceiling as u64)
.saturating_mul(TYPICAL_CHUNK_BYTES)
.saturating_mul(BUDGET_HEADROOM_NUM)
.saturating_div(BUDGET_HEADROOM_DEN)
.clamp(
FETCH_INFLIGHT_BUDGET_FLOOR_BYTES,
FETCH_INFLIGHT_BUDGET_HARD_CAP_BYTES,
)
}
// ─── Adaptive-window tuning ────────────────────────────────────────────────────────────────────
// r3 retune: the first device run pinned the window at the floor on a 1 Gbps link. The controller
// shrank on throughput dips and latency jitter (normal CDN noise) as well as on errors, shrank a flat
// −6 on every single error, and only grew when throughput rose >10% in a 700 ms sample. The shrinks
// dominated, so the window ratcheted to the floor and stayed there. r3 makes growth eager and makes
// ERRORS the only shrink signal.
/// Max concurrent requests to any single CDN host. CDNs throttle/reset per host, so a big window
/// is reached by spreading across MORE hosts, never by piling onto one.
pub const PER_HOST_CAP: usize = 6;
/// Initial in-flight window. Slow-start doubles from here, so this only sets how fast the first
/// couple of probes get useful — the ceiling still comes from the tier × hosts clamp.
pub const BOOTSTRAP_WINDOW: usize = 8;
/// Hard safety ceiling on the window regardless of tier (also clamped to distinct-hosts × per-host).
pub const WINDOW_HARD_CAP: usize = 256;
/// The window never shrinks below this (keeps a little pipelining even on a rough link).
pub const WINDOW_MIN_FLOOR: usize = 2;
/// Additive growth step once slow-start has ended (after the first real back-off).
pub const WINDOW_STEP_UP: usize = 4;
/// Slow-start multiplier: until the first back-off the window DOUBLES each healthy probe, so a fast
/// link reaches a useful window in seconds instead of never.
pub const WINDOW_SLOW_START_FACTOR: usize = 2;
/// Proportional shrink on a real error signal (never a flat step: a flat −6 slammed a small window
/// straight to the floor). Always at least −1 while above the floor.
pub const WINDOW_SHRINK_FACTOR: f64 = 0.75;
/// How often the adaptive window re-evaluates. Long enough that one jittery sample cannot drive a
/// decision; decisions use the EWMA, not the raw sample.
pub const WINDOW_PROBE_INTERVAL_MS: u64 = 2_000;
/// After a back-off, hold the window (no growth) for this long — hysteresis against oscillation.
pub const WINDOW_COOLDOWN_MS: u64 = 3_000;
/// EWMA smoothing for the measured throughput samples (higher = more responsive).
pub const WINDOW_BPS_EWMA_ALPHA: f64 = 0.4;
/// Throughput above `best × (1 + this)` counts as a genuine improvement (keeps slow-start climbing).
pub const WINDOW_IMPROVE_EPS: f64 = 0.05;
/// Throughput below `best × (1 − this)` counts as falling — growth STOPS (hold). It never shrinks:
/// only real errors shrink.
pub const WINDOW_DECLINE_EPS: f64 = 0.15;
/// Error-rate thresholds over one probe window: above HIGH shrinks; growth needs at/below LOW
/// (in between = hold).
pub const WINDOW_ERR_RATE_HIGH: f64 = 0.10;
pub const WINDOW_ERR_RATE_LOW: f64 = 0.05;
/// Errors inside ONE probe interval that trigger an immediate (don't-wait-for-the-probe) shrink. A
/// single stray timeout no longer moves the window; a burst does. A 429 always shrinks immediately.
pub const WINDOW_ERR_BURST_IMMEDIATE: u32 = 3;
/// Consecutive non-improving probes still allowed to grow before declaring a plateau — throughput
/// lags a window change by a probe or two, so one flat sample must not stop the ramp.
pub const WINDOW_PLATEAU_PATIENCE: u32 = 2;
/// Once plateaued, re-arm the ramp this long later (doubling, capped) to re-probe for headroom if the
/// link improved. Bounded re-probing, not a creep: a weak link answers with errors and shrinks back.
pub const WINDOW_PLATEAU_REARM_MS: u64 = 30_000;
pub const WINDOW_PLATEAU_REARM_MAX_MS: u64 = 300_000;
/// Even when the window does not change, emit one `fetch-window` line every N probes so a stuck
/// window explains itself in the log (`reason=` says why it is not moving).
pub const WINDOW_LOG_EVERY_PROBES: u32 = 5;
/// Every Nth exploit dispatch is instead an exploration pick (epsilon-greedy ≈ 1/N) so a
/// demoted-but-recovered host is retried rather than starved forever.
pub const SERVER_EXPLORE_EVERY: u64 = 8;
/// Reservation for an item whose compressed size the manifest didn't carry (keeps the byte budget
/// honest at dispatch time).
pub const NOMINAL_CHUNK_RESERVE_BYTES: u64 = 1024 * 1024;
/// Extra host cooldown floor when a host answers 429 (rate-limited) — back off harder than a
/// one-off timeout.
pub const RATE_LIMIT_COOLDOWN_MS: u64 = 5_000;
/// Exponential back-off between attempts of one item: 0, 300, 600, 1200, 2400 ms … capped at 4 s.
pub fn retry_backoff_millis(attempt: u32) -> u64 {
if attempt == 0 {
return 0;
}
(300u64 << (attempt - 1)).min(4000)
}
/// Byte-budget admission test for the fetch→process channel. Always admits when nothing is in
/// flight (`in_flight == 0`) so a single item LARGER than the whole budget can never deadlock — it
/// simply waits until the channel drains, then goes through alone.
#[inline]
pub fn budget_admits(in_flight: u64, raw_len: u64, budget: u64) -> bool {
in_flight == 0 || in_flight.saturating_add(raw_len) <= budget
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn backoff_is_exponential_and_capped() {
assert_eq!(retry_backoff_millis(0), 0);
assert_eq!(retry_backoff_millis(1), 300);
assert_eq!(retry_backoff_millis(2), 600);
assert_eq!(retry_backoff_millis(4), 2400);
assert_eq!(retry_backoff_millis(5), 4000);
assert_eq!(retry_backoff_millis(12), 4000);
}
#[test]
fn budget_scales_between_floor_and_cap() {
assert_eq!(inflight_budget_bytes(1), FETCH_INFLIGHT_BUDGET_FLOOR_BYTES);
assert_eq!(inflight_budget_bytes(8), FETCH_INFLIGHT_BUDGET_FLOOR_BYTES);
assert_eq!(inflight_budget_bytes(32), 48 * 1024 * 1024);
assert_eq!(inflight_budget_bytes(1024), FETCH_INFLIGHT_BUDGET_HARD_CAP_BYTES);
}
#[test]
fn single_oversized_item_is_always_admitted() {
assert!(budget_admits(0, u64::MAX, 1));
assert!(budget_admits(10, 5, 15));
assert!(!budget_admits(10, 6, 15));
}
}
+121
View File
@@ -0,0 +1,121 @@
//! GOG gen2 chunk download engine — the Rust replacement for the byte-fetching inner loop of
//! `GogDownloadManager.java` (`assembleDepotFile` / `fetchChunkVerified` driven from the
//! per-file thread pool). Everything around that loop — token/builds/manifest head, base-product
//! + language depot selection, secure-link resolution, disk guard, `_gog_manifest.json` marker,
//! prefs, exe resolution, DLC markers, redist orchestration — stays in Java and is untouched.
//!
//! Parity contract: Bannerlator's `docs/RUST_GOG_PARITY.md`. The adapter is fed the SAME inflated depot-manifest
//! JSON strings Java already parsed, the SAME resolved CDN base (secure link) and the SAME
//! install dir, and produces the SAME on-disk layout (`<file>.bhtmp` staging, atomic rename,
//! size+MD5-verified finals) so a download can be resumed by either engine.
//!
//! Layout:
//! - [`plan`] — depot-manifest JSON → file/chunk plan (mirrors `parseDepotManifest`), CDN path
//! and chunk-URL builders, largest-first ordering.
//! - [`engine`] — resume/skip pass (mirrors `fileVerified`), `FetchItem` list, the `FetchSink`
//! (compressed size/MD5 → inflate → decompressed size/MD5 → positioned write →
//! whole-file size/MD5 → rename), summary logging.
//! - [`jni`] — `GogNative` JNI exports (`nativeStart` / `nativeCancel` / `nativeRelease`).
pub mod engine;
pub mod jni;
pub mod plan;
use std::fs::File;
use std::io::Read;
use std::path::Path;
/// Lowercase hex of `bytes` (matches `GogDownloadManager.toHex`).
pub fn to_hex(bytes: &[u8]) -> String {
const DIGITS: &[u8; 16] = b"0123456789abcdef";
let mut out = String::with_capacity(bytes.len() * 2);
for b in bytes {
out.push(DIGITS[(b >> 4) as usize] as char);
out.push(DIGITS[(b & 0x0f) as usize] as char);
}
out
}
/// Lowercase hex MD5 of an in-memory buffer (`GogDownloadManager.md5Hex`).
pub fn md5_hex(data: &[u8]) -> String {
let mut hasher = crate::md5_small::Md5::new();
hasher.update(data);
to_hex(&hasher.finalize())
}
/// Streaming lowercase hex MD5 of a file, never buffering the whole file
/// (`GogDownloadManager.md5HexFile`). `None` when the file cannot be read.
pub fn md5_hex_file(path: &Path) -> Option<String> {
let mut file = File::open(path).ok()?;
let mut hasher = crate::md5_small::Md5::new();
let mut buf = vec![0u8; 128 * 1024];
loop {
let n = file.read(&mut buf).ok()?;
if n == 0 {
break;
}
hasher.update(&buf[..n]);
}
Some(to_hex(&hasher.finalize()))
}
/// zlib-inflate a chunk body (`GogDownloadManager.inflateZlib`): `None` when the body is not a
/// zlib stream (first byte != 0x78) or fails to inflate — the caller then treats the body as a
/// stored (non-compressed) chunk, exactly like Java's `if (inflated == null) inflated = raw`.
/// A truncated stream yields the partial output (Java's loop breaks on `n == 0` the same way);
/// the decompressed-size check that follows rejects it.
pub fn inflate_zlib(data: &[u8]) -> Option<Vec<u8>> {
if data.len() < 2 || data[0] != 0x78 {
return None;
}
let mut out = Vec::with_capacity(data.len().saturating_mul(3));
let mut decoder = flate2::read::ZlibDecoder::new(data);
match decoder.read_to_end(&mut out) {
Ok(_) => Some(out),
Err(_) => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn hex_is_lowercase_and_padded() {
assert_eq!(to_hex(&[0x00, 0x0a, 0xff, 0x10]), "000aff10");
}
#[test]
fn inflate_roundtrip_and_stored_fallback() {
use flate2::write::ZlibEncoder;
use flate2::Compression;
use std::io::Write;
let payload = b"hello gog chunk payload".repeat(64);
let mut enc = ZlibEncoder::new(Vec::new(), Compression::default());
enc.write_all(&payload).unwrap();
let compressed = enc.finish().unwrap();
assert_eq!(compressed[0], 0x78);
assert_eq!(inflate_zlib(&compressed).unwrap(), payload);
// Not zlib → None (stored chunk path).
assert!(inflate_zlib(b"\x1f\x8bnot zlib").is_none());
assert!(inflate_zlib(b"\x78").is_none());
// Truncated zlib → partial output, like Java's `while (!inf.finished()) { if (n == 0) break; }`
// loop; the decompressed-size check downstream rejects it either way.
let mut truncated = compressed.clone();
let mid = truncated.len() / 2;
truncated.truncate(mid);
assert!(inflate_zlib(&truncated).map(|out| out != payload).unwrap_or(true));
// Corrupt zlib body → None (Java: DataFormatException → null → raw fallback).
let mut corrupt = compressed.clone();
for b in corrupt.iter_mut().skip(2) {
*b = !*b;
}
assert!(inflate_zlib(&corrupt).is_none() || inflate_zlib(&corrupt).unwrap() != payload);
}
#[test]
fn md5_hex_matches_rfc1321_vector() {
assert_eq!(md5_hex(b""), "d41d8cd98f00b204e9800998ecf8427e");
assert_eq!(md5_hex(b"abc"), "900150983cd24fb0d6963f7d28e17f72");
}
}
File diff suppressed because it is too large. Load diff
+321
View File
@@ -0,0 +1,321 @@
//! JNI facade for the GOG Rust download engine — the native half of
//! `com.droiddeck.launcher.stores.gog.GogNative` (Kotlin). Symbol names are bound to that
//! package/class; the `@JvmStatic external` declarations there are the other side of this file.
//!
//! Shape mirrors the Steam download JNI (`jni.rs` `nativeDownloadApp`): `nativeStart` validates
//! its inputs, takes a `GlobalRef` on the listener, spawns the worker thread and returns a handle
//! immediately; callbacks are delivered from the worker / process-pool threads through a
//! `JavaVM` attach; `nativeCancel` flips an `AtomicBool`; `nativeRelease` drops the Java side's
//! reference to the handle (the worker keeps its own until it exits).
//!
//! Log tag `GogNative`: every engine log line goes to logcat here AND to the listener's `onLog`
//! (which the Kotlin manager folds into its own debug log).
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
use jni::objects::{GlobalRef, JClass, JObject, JObjectArray, JString, JValue};
use jni::sys::{jboolean, jint, jlong, JNI_FALSE, JNI_TRUE};
use jni::{JNIEnv, JavaVM};
use super::engine::{self, GogEvents, GogRequest, GogRunResult, PlanKind};
pub const LOG_TAG: &str = "GogNative";
#[cfg(target_os = "android")]
#[link(name = "log")]
unsafe extern "C" {
fn __android_log_write(prio: i32, tag: *const i8, text: *const i8) -> i32;
}
/// `android.util.Log.i(LOG_TAG, message)`; a no-op on non-Android hosts (cargo test).
pub fn android_log(message: &str) {
#[cfg(target_os = "android")]
{
use std::ffi::CString;
let Ok(tag) = CString::new(LOG_TAG) else {
return;
};
let sanitized = message.replace('\0', " ");
let Ok(text) = CString::new(sanitized) else {
return;
};
unsafe {
let _ = __android_log_write(4, tag.as_ptr().cast(), text.as_ptr().cast());
}
}
#[cfg(not(target_os = "android"))]
{
let _ = message;
}
}
/// One in-flight run. The Java side owns a `Box<Arc<..>>` (the `jlong`); the worker thread owns a
/// second `Arc` so `nativeRelease` is safe at any time after `nativeStart` returned.
pub struct GogRunHandle {
pub cancel: AtomicBool,
}
struct JniEvents {
vm: JavaVM,
listener: GlobalRef,
}
impl JniEvents {
fn with_env(&self, f: impl FnOnce(&mut JNIEnv)) {
let Ok(mut env) = self.vm.attach_current_thread_as_daemon() else {
return;
};
f(&mut env);
}
fn clear_exception(env: &mut JNIEnv) {
if env.exception_check().unwrap_or(false) {
let _ = env.exception_clear();
}
}
fn complete(&self, result: &GogRunResult) {
self.with_env(|env| {
let Ok(error) = env.new_string(&result.error) else {
return;
};
let error = JObject::from(error);
let _ = env.call_method(
self.listener.as_obj(),
"onComplete",
"(ZZZLjava/lang/String;JI)V",
&[
JValue::Bool(if result.success { JNI_TRUE } else { JNI_FALSE }),
JValue::Bool(if result.cancelled { JNI_TRUE } else { JNI_FALSE }),
JValue::Bool(if result.link_expiry { JNI_TRUE } else { JNI_FALSE }),
JValue::Object(&error),
JValue::Long(result.bytes_written as jlong),
JValue::Int(result.files_done as jint),
],
);
Self::clear_exception(env);
let _ = env.delete_local_ref(error);
});
}
}
impl GogEvents for JniEvents {
fn on_file_done(
&self,
rel_path: &str,
file_bytes: u64,
verified: bool,
files_done: u32,
files_total: u32,
bytes_done: u64,
bytes_total: u64,
) {
self.with_env(|env| {
let Ok(path) = env.new_string(rel_path) else {
return;
};
let path = JObject::from(path);
let _ = env.call_method(
self.listener.as_obj(),
"onProgress",
"(JJIILjava/lang/String;JZ)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
JValue::Int(files_done as jint),
JValue::Int(files_total as jint),
JValue::Object(&path),
JValue::Long(file_bytes as jlong),
JValue::Bool(if verified { JNI_TRUE } else { JNI_FALSE }),
],
);
Self::clear_exception(env);
let _ = env.delete_local_ref(path);
});
}
fn on_bytes(&self, bytes_done: u64, bytes_total: u64) {
// A listener without `onBytes` (an app built against the older contract) makes this a
// NoSuchMethodError, which clear_exception drops: the run is unaffected.
self.with_env(|env| {
let _ = env.call_method(
self.listener.as_obj(),
"onBytes",
"(JJ)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
],
);
Self::clear_exception(env);
});
}
fn on_log(&self, line: &str) {
android_log(line);
self.with_env(|env| {
let Ok(text) = env.new_string(line) else {
return;
};
let text = JObject::from(text);
let _ = env.call_method(
self.listener.as_obj(),
"onLog",
"(Ljava/lang/String;)V",
&[JValue::Object(&text)],
);
Self::clear_exception(env);
let _ = env.delete_local_ref(text);
});
}
}
fn jstring_to_string(env: &mut JNIEnv, value: &JString) -> Option<String> {
env.get_string(value)
.ok()
.map(|s| s.to_string_lossy().into_owned())
}
fn string_array_to_vec(env: &mut JNIEnv, array: &JObjectArray) -> Vec<String> {
if array.is_null() {
return Vec::new();
}
let len = env.get_array_length(array).unwrap_or(0);
let mut out = Vec::with_capacity(len.max(0) as usize);
for i in 0..len {
let Ok(element) = env.get_object_array_element(array, i) else {
continue;
};
if element.is_null() {
continue;
}
let text = JString::from(element);
if let Some(s) = jstring_to_string(env, &text) {
out.push(s);
}
let _ = env.delete_local_ref(text);
}
out
}
unsafe fn handle_ref(handle: jlong) -> Option<&'static Arc<GogRunHandle>> {
let ptr = handle as *const Arc<GogRunHandle>;
if ptr.is_null() {
None
} else {
Some(unsafe { &*ptr })
}
}
/// Symbol-binding probe (see `GogNative.isAvailable`). Returns 1.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeProbe(
_env: JNIEnv,
_class: JClass,
) -> jint {
1
}
/// Starts one download loop on a worker thread and returns a handle (0 = could not start; the
/// listener then receives NO callbacks). Inputs are exactly what `GogDownloadManager` holds after
/// manifest + secure-link resolution — see `GogRequest`.
#[no_mangle]
#[allow(clippy::too_many_arguments)]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeStart(
mut env: JNIEnv,
_class: JClass,
kind: jint,
depot_manifests: JObjectArray,
cdn_bases: JObjectArray,
install_dir: JString,
skip_paths: JObjectArray,
ca_bundle_path: JString,
max_workers: jint,
process_workers: jint,
sort_largest_first: jboolean,
label: JString,
listener: JObject,
) -> jlong {
if listener.is_null() {
android_log("nativeStart: null listener");
return 0;
}
let Ok(vm) = env.get_java_vm() else {
android_log("nativeStart: JavaVM unavailable");
return 0;
};
let Ok(listener) = env.new_global_ref(&listener) else {
android_log("nativeStart: listener ref failed");
return 0;
};
let request = GogRequest {
kind: PlanKind::from_i32(kind),
depot_manifests: string_array_to_vec(&mut env, &depot_manifests),
cdn_bases: string_array_to_vec(&mut env, &cdn_bases),
install_dir: jstring_to_string(&mut env, &install_dir).unwrap_or_default(),
skip_paths: string_array_to_vec(&mut env, &skip_paths),
ca_bundle_path: jstring_to_string(&mut env, &ca_bundle_path).unwrap_or_default(),
max_workers: max_workers.max(1) as usize,
process_workers: process_workers.max(1) as usize,
sort_largest_first: sort_largest_first != JNI_FALSE,
label: jstring_to_string(&mut env, &label).unwrap_or_else(|| "gog".to_string()),
};
let needs_base = request.kind == PlanKind::Gen2Chunks;
if (needs_base && super::engine::distinct_bases(&request.cdn_bases).is_empty())
|| request.install_dir.is_empty()
|| request.depot_manifests.is_empty()
{
android_log("nativeStart: invalid request (no cdn base / empty install dir / no manifests)");
return 0;
}
let handle = Arc::new(GogRunHandle {
cancel: AtomicBool::new(false),
});
let worker = Arc::clone(&handle);
let spawned = thread::Builder::new()
.name("gog-dl".to_string())
.spawn(move || {
crate::priority::background();
let events = JniEvents { vm, listener };
let result = engine::run(&request, &worker.cancel, &events);
events.complete(&result);
});
if spawned.is_err() {
android_log("nativeStart: worker thread spawn failed");
return 0;
}
Box::into_raw(Box::new(handle)) as jlong
}
/// Requests cancellation; the run stops at its next scheduling point and `onComplete` fires with
/// `cancelled = true`. Idempotent.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeCancel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if let Some(run) = unsafe { handle_ref(handle) } {
run.cancel.store(true, Ordering::Relaxed);
}
}
/// Drops the Java side's reference. Call once, after `onComplete` (or after cancelling — the worker
/// keeps the run alive until it exits either way).
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeRelease(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
let ptr = handle as *mut Arc<GogRunHandle>;
if ptr.is_null() {
return;
}
unsafe {
drop(Box::from_raw(ptr));
}
}
+370
View File
@@ -0,0 +1,370 @@
//! Depot-manifest JSON → download plan. Mirrors `GogDownloadManager.parseDepotManifest` rule for
//! rule (see Bannerlator's `docs/RUST_GOG_PARITY.md` §3):
//! - root `depot.items[]`; `path` with `\` → `/` and one leading `/` stripped;
//! - a file is kept only when `path` is non-empty AND `chunks[]` is non-empty;
//! - per chunk: CDN hash = `compressedMd5` when non-empty, else `md5`; a chunk with neither is
//! dropped; `compressedSize` / `size` default 0;
//! - file `md5` kept when non-empty; `total_size` = Σ chunk `size` (decompressed);
//! - a file whose chunks all dropped is not added.
//! `org.json` coercion (`optString` turns numbers/bools into their text, `optLong` parses numeric
//! strings) is reproduced by [`opt_string`] / [`opt_u64`].
//!
//! Additionally each chunk carries `offset` = prefix-sum of the preceding chunks' decompressed
//! sizes — Java appends sequentially so it never needs one; the Rust sink writes chunks at that
//! offset as they arrive so the final bytes are identical.
use serde_json::Value;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ChunkRef {
/// CDN-path hash (`compressedMd5` preferred, else `md5`).
pub hash: String,
/// Verify compressed bytes before inflate (may be empty).
pub compressed_md5: String,
/// Verify decompressed bytes after inflate (may be empty).
pub md5: String,
/// Expected compressed size, 0 if absent.
pub compressed_size: u64,
/// Expected decompressed size, 0 if absent.
pub size: u64,
/// Decompressed offset inside the file (prefix-sum of preceding `size`s).
pub offset: u64,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PlannedFile {
/// Install-dir-relative path, forward slashes, no leading slash.
pub relative_path: String,
/// Whole-file MD5 from the manifest ("" when absent).
pub md5: String,
/// Σ chunk decompressed sizes (0 if unknown).
pub total_size: u64,
pub chunks: Vec<ChunkRef>,
}
/// `JSONObject.optString(key, "")` semantics: strings as-is, numbers/bools rendered, null/absent
/// → "". (`org.json` renders `JSONObject.NULL` as "null"; GOG manifests never carry JSON nulls in
/// these fields, and treating one as absent is the only safe reading.)
pub fn opt_string(obj: &Value, key: &str) -> String {
match obj.get(key) {
Some(Value::String(s)) => s.clone(),
Some(Value::Number(n)) => n.to_string(),
Some(Value::Bool(b)) => b.to_string(),
_ => String::new(),
}
}
/// `JSONObject.optLong(key, 0)` semantics for a size field (numbers, or numeric strings; a
/// negative/float value that Java would truncate is clamped to 0 — sizes are never negative).
pub fn opt_u64(obj: &Value, key: &str) -> u64 {
match obj.get(key) {
Some(Value::Number(n)) => n
.as_u64()
.or_else(|| n.as_i64().map(|v| v.max(0) as u64))
.or_else(|| n.as_f64().map(|v| if v > 0.0 { v as u64 } else { 0 }))
.unwrap_or(0),
Some(Value::String(s)) => s.trim().parse::<i64>().map(|v| v.max(0) as u64).unwrap_or(0),
_ => 0,
}
}
/// Parses one inflated gen2 depot manifest and appends its files to `out`
/// (`GogDownloadManager.parseDepotManifest`). Malformed JSON adds nothing, like Java's catch-all.
pub fn parse_depot_manifest(json: &str, out: &mut Vec<PlannedFile>) {
let Ok(root) = serde_json::from_str::<Value>(json) else {
return;
};
let Some(items) = root
.get("depot")
.and_then(|d| d.get("items"))
.and_then(Value::as_array)
else {
return;
};
for entry in items {
if !entry.is_object() {
continue;
}
let mut path = opt_string(entry, "path").replace('\\', "/");
if let Some(stripped) = path.strip_prefix('/') {
path = stripped.to_string();
}
let chunks = entry.get("chunks").and_then(Value::as_array);
let Some(chunks) = chunks else {
continue;
};
if path.is_empty() || chunks.is_empty() {
continue;
}
let file_md5 = opt_string(entry, "md5");
let mut planned = PlannedFile {
relative_path: path,
md5: file_md5,
total_size: 0,
chunks: Vec::with_capacity(chunks.len()),
};
let mut offset = 0u64;
for chunk in chunks {
if !chunk.is_object() {
continue;
}
let compressed_md5 = opt_string(chunk, "compressedMd5");
let dec_md5 = opt_string(chunk, "md5");
let compressed_size = opt_u64(chunk, "compressedSize");
let dec_size = opt_u64(chunk, "size");
let hash = if !compressed_md5.is_empty() {
compressed_md5.clone()
} else {
dec_md5.clone()
};
if hash.is_empty() {
continue;
}
planned.chunks.push(ChunkRef {
hash,
compressed_md5,
md5: dec_md5,
compressed_size,
size: dec_size,
offset,
});
offset = offset.saturating_add(dec_size);
}
planned.total_size = offset;
if !planned.chunks.is_empty() {
out.push(planned);
}
}
}
/// One gen1 file: a byte range of a (pre-signed) depot blob (`GogDownloadManager.Gen1File`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Gen1File {
/// Install-dir-relative path, used verbatim (Java's gen1 path does not normalise slashes).
pub path: String,
pub url: String,
pub offset: u64,
pub size: u64,
}
/// Parses a gen1 build manifest (`runGen1`): every `depot[]` entry that is not `support: true`,
/// its `files[]` → (`path`, `url`, `offset` default 0, `size` default 0); an entry with
/// `size == 0` is dropped (Java's `path == null || url == null` checks never fire — `optString`
/// returns "" — so `size == 0` is the only effective filter, mirrored here).
pub fn parse_gen1_manifest(json: &str, out: &mut Vec<Gen1File>) {
let Ok(root) = serde_json::from_str::<Value>(json) else {
return;
};
let Some(depots) = root.get("depot").and_then(Value::as_array) else {
return;
};
for depot in depots {
if !depot.is_object() {
continue;
}
if depot.get("support").and_then(Value::as_bool).unwrap_or(false) {
continue;
}
let Some(files) = depot.get("files").and_then(Value::as_array) else {
continue;
};
for f in files {
if !f.is_object() {
continue;
}
let path = opt_string(f, "path");
let offset = opt_u64(f, "offset");
let size = opt_u64(f, "size");
let url = opt_string(f, "url");
if size == 0 {
continue;
}
out.push(Gen1File { path, url, offset, size });
}
}
}
/// Parses every manifest in order (the order Java fetched the depots) and optionally applies the
/// base-install largest-first (LPT) ordering — a STABLE sort by descending `total_size`, exactly
/// `Collections.sort(files, (a, b) -> Long.compare(b.totalSize, a.totalSize))`. The DLC and
/// dependency paths in Java do not sort, so they pass `sort_largest_first = false`.
pub fn build_plan(manifests: &[String], sort_largest_first: bool) -> Vec<PlannedFile> {
let mut files = Vec::new();
for manifest in manifests {
parse_depot_manifest(manifest, &mut files);
}
if sort_largest_first {
files.sort_by(|a, b| b.total_size.cmp(&a.total_size));
}
files
}
/// `GogDownloadManager.buildCdnPath`: `"ab/cd/abcdef..."`. A hash shorter than 4 chars would
/// throw in Java (and fail that file); here it is returned unchanged and fails at fetch time.
pub fn build_cdn_path(hash: &str) -> String {
if !hash.is_char_boundary(2) || !hash.is_char_boundary(4) || hash.len() < 4 {
return hash.to_string();
}
format!("{}/{}/{}", &hash[0..2], &hash[2..4], hash)
}
/// `GogDownloadManager.buildChunkUrl`: appends the chunk path BEFORE any query string so the
/// secure-link token survives.
pub fn build_chunk_url(base: &str, chunk_path: &str) -> String {
match base.find('?') {
Some(q) => format!("{}/{}{}", &base[..q], chunk_path, &base[q..]),
None => format!("{base}/{chunk_path}"),
}
}
/// Per-host key for the fetch core's host semaphores: `scheme://host[:port]` of `url` (lowercased
/// host). Java has one resolved CDN base per run, so the host list has exactly one entry.
pub fn host_key(url: &str) -> String {
let (scheme, rest) = match url.find("://") {
Some(i) => (&url[..i], &url[i + 3..]),
None => ("https", url),
};
let end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
format!("{}://{}", scheme.to_ascii_lowercase(), rest[..end].to_ascii_lowercase())
}
#[cfg(test)]
mod tests {
use super::*;
const MANIFEST: &str = r#"{
"depot": { "items": [
{ "path": "\\bin\\game.exe", "md5": "AABBCCDDEEFF00112233445566778899",
"chunks": [
{ "compressedMd5": "c1c1c1c1", "md5": "d1d1d1d1", "compressedSize": 100, "size": 250 },
{ "compressedMd5": "c2c2c2c2", "md5": "d2d2d2d2", "compressedSize": "50", "size": 75 }
] },
{ "path": "/data/pak0.pak",
"chunks": [
{ "md5": "onlydec1", "size": 1000 },
{ "compressedSize": 5, "size": 5 }
] },
{ "path": "empty.txt", "chunks": [] },
{ "path": "", "chunks": [ { "compressedMd5": "zz", "size": 1 } ] },
{ "path": "nohash.bin", "chunks": [ { "size": 9 } ] },
{ "path": "dir", "type": "DepotDirectory" },
"not-an-object"
] } }"#;
#[test]
fn parse_mirrors_java_rules() {
let mut files = Vec::new();
parse_depot_manifest(MANIFEST, &mut files);
assert_eq!(files.len(), 2, "empty-chunk, empty-path, no-hash and directory entries drop");
let exe = &files[0];
assert_eq!(exe.relative_path, "bin/game.exe");
assert_eq!(exe.md5, "AABBCCDDEEFF00112233445566778899");
assert_eq!(exe.total_size, 325);
assert_eq!(exe.chunks.len(), 2);
assert_eq!(exe.chunks[0].hash, "c1c1c1c1");
assert_eq!(exe.chunks[0].offset, 0);
assert_eq!(exe.chunks[1].hash, "c2c2c2c2");
assert_eq!(exe.chunks[1].compressed_size, 50, "numeric string coerces like optLong");
assert_eq!(exe.chunks[1].offset, 250);
let pak = &files[1];
assert_eq!(pak.relative_path, "data/pak0.pak");
assert_eq!(pak.md5, "", "absent file md5 → empty (no whole-file verify)");
assert_eq!(pak.chunks.len(), 1, "chunk without any hash is dropped");
assert_eq!(pak.chunks[0].hash, "onlydec1", "hash falls back to md5");
assert_eq!(pak.chunks[0].compressed_md5, "");
assert_eq!(pak.total_size, 1000);
}
#[test]
fn malformed_manifest_adds_nothing() {
let mut files = Vec::new();
parse_depot_manifest("{not json", &mut files);
parse_depot_manifest(r#"{"depot": {}}"#, &mut files);
parse_depot_manifest(r#"{"items": []}"#, &mut files);
assert!(files.is_empty());
}
#[test]
fn plan_orders_largest_first_stably_only_when_asked() {
let m1 = r#"{"depot":{"items":[
{"path":"a","chunks":[{"md5":"a1","size":10}]},
{"path":"b","chunks":[{"md5":"b1","size":30}]}]}}"#
.to_string();
let m2 = r#"{"depot":{"items":[
{"path":"c","chunks":[{"md5":"c1","size":30}]},
{"path":"d","chunks":[{"md5":"d1","size":20}]}]}}"#
.to_string();
let unsorted = build_plan(&[m1.clone(), m2.clone()], false);
let order: Vec<&str> = unsorted.iter().map(|f| f.relative_path.as_str()).collect();
assert_eq!(order, ["a", "b", "c", "d"], "depot order, file order within depot");
let sorted = build_plan(&[m1, m2], true);
let order: Vec<&str> = sorted.iter().map(|f| f.relative_path.as_str()).collect();
assert_eq!(order, ["b", "c", "d", "a"], "stable: b (30) before c (30) keeps source order");
}
#[test]
fn gen1_parse_mirrors_run_gen1() {
let json = r#"{"installDirectory":"Game","depot":[
{"support":true,"files":[{"path":"support.bin","url":"u","offset":0,"size":5}]},
{"files":[
{"path":"a.bin","url":"https://cdn.gog.com/blob?t=1","offset":10,"size":20},
{"path":"zero.bin","url":"u","offset":0,"size":0},
{"path":"b.bin","url":"https://cdn.gog.com/blob?t=1","size":"7"},
"junk"
]},
{"nofiles":true}
]}"#;
let mut files = Vec::new();
parse_gen1_manifest(json, &mut files);
assert_eq!(
files,
vec![
Gen1File { path: "a.bin".into(), url: "https://cdn.gog.com/blob?t=1".into(), offset: 10, size: 20 },
Gen1File { path: "b.bin".into(), url: "https://cdn.gog.com/blob?t=1".into(), offset: 0, size: 7 },
]
);
let mut none = Vec::new();
parse_gen1_manifest(r#"{"depots":[]}"#, &mut none);
assert!(none.is_empty());
}
#[test]
fn cdn_path_and_chunk_url() {
assert_eq!(build_cdn_path("abcdef0123"), "ab/cd/abcdef0123");
assert_eq!(build_cdn_path("abc"), "abc");
assert_eq!(
build_chunk_url("https://cdn.gog.com/content-system/v2/store/1?token=x&y=1", "ab/cd/abcdef"),
"https://cdn.gog.com/content-system/v2/store/1/ab/cd/abcdef?token=x&y=1"
);
assert_eq!(
build_chunk_url("https://gog-cdn-fastly.gog.com/content-system/v2/dependencies/store", "ab/cd/abcdef"),
"https://gog-cdn-fastly.gog.com/content-system/v2/dependencies/store/ab/cd/abcdef"
);
}
#[test]
fn host_key_strips_path_and_query() {
assert_eq!(host_key("https://CDN.gog.com/a/b?c=1"), "https://cdn.gog.com");
assert_eq!(host_key("http://host:8080/x"), "http://host:8080");
assert_eq!(host_key("nohost"), "https://nohost");
}
#[test]
fn opt_helpers_coerce_like_org_json() {
let v: Value = serde_json::from_str(r#"{"n": 12, "s": "34", "b": true, "z": null, "f": 2.5, "neg": -4}"#).unwrap();
assert_eq!(opt_string(&v, "n"), "12");
assert_eq!(opt_string(&v, "s"), "34");
assert_eq!(opt_string(&v, "b"), "true");
assert_eq!(opt_string(&v, "z"), "");
assert_eq!(opt_string(&v, "missing"), "");
assert_eq!(opt_u64(&v, "n"), 12);
assert_eq!(opt_u64(&v, "s"), 34);
assert_eq!(opt_u64(&v, "f"), 2);
assert_eq!(opt_u64(&v, "neg"), 0);
assert_eq!(opt_u64(&v, "b"), 0);
assert_eq!(opt_u64(&v, "missing"), 0);
}
}
+29
View File
@@ -0,0 +1,29 @@
//! DroidDeck's native store download engines (`libdroiddeckstores.so`).
//!
//! The GOG, Epic Games Store and Amazon Games download loops, ported from Bannerlator's
//! `bl-steam-client` crate (GPL-3.0-or-later, itself derived from WinNative's `wn-steam-client`).
//! Only the store engines and the fetch core they run on came across: DroidDeck's Steam client is
//! the real one, running in the Linux runtime, so none of the Steam CM / depot code is here.
//!
//! Each store module owns its manifest → plan → `FetchItem` mapping, its `FetchSink` (inflate +
//! hash + write in the exact output layout of the Kotlin manager that drives it) and its JNI facade.
//! The JNI contract — classes, method names, signatures, threading — is written down in `JNI.md`
//! next to `Cargo.toml`; the Kotlin side mirrors it exactly.
//!
//! What a run looks like: Kotlin hands over a manifest it already parsed (or, for Amazon, the
//! plan it built from one), the resolved CDN base(s) and the install directory, calls `nativeStart`
//! and gets a handle back at once. Everything after that happens on native threads: the fetch core
//! keeps an adaptive window of HTTP requests in flight on one tokio runtime, hands bodies to a sync
//! process pool that inflates, hashes and writes them, and reports progress through the listener
//! object Kotlin passed in. `nativeCancel` flips a flag; `onComplete` fires exactly once.
#![allow(clippy::missing_safety_doc, clippy::result_large_err)]
pub mod amazon;
pub mod epic;
pub mod fetch_core;
pub mod fetch_tuning;
pub mod gog;
pub mod md5_small;
pub mod priority;
pub mod stores_jni;
+197
View File
@@ -0,0 +1,197 @@
//! Minimal MD5 (RFC 1321) for the store download engines (GOG chunk / file verification).
//!
//! In-crate on purpose: the crate is built `--locked` in CI with no way to add a dependency from
//! this environment, and the algorithm is 100 lines. Not for anything security-sensitive — it only
//! mirrors the checksum the Java managers already compare against.
/// One-shot digest.
pub fn md5(data: &[u8]) -> [u8; 16] {
let mut h = Md5::new();
h.update(data);
h.finalize()
}
/// Lower-case hex of a 16-byte digest.
pub fn md5_hex(digest: &[u8; 16]) -> String {
const HEX: &[u8; 16] = b"0123456789abcdef";
let mut out = String::with_capacity(32);
for b in digest {
out.push(HEX[(b >> 4) as usize] as char);
out.push(HEX[(b & 0x0f) as usize] as char);
}
out
}
/// Streaming MD5: `update` any number of times, then `finalize`.
#[derive(Clone, Debug)]
pub struct Md5 {
state: [u32; 4],
buffer: [u8; 64],
buffer_len: usize,
total_len: u64,
}
impl Default for Md5 {
fn default() -> Self {
Self::new()
}
}
const S: [u32; 64] = [
7, 12, 17, 22, 7, 12, 17, 22, 7, 12, 17, 22, 7, 12, 17, 22, 5, 9, 14, 20, 5, 9, 14, 20, 5, 9,
14, 20, 5, 9, 14, 20, 4, 11, 16, 23, 4, 11, 16, 23, 4, 11, 16, 23, 4, 11, 16, 23, 6, 10, 15,
21, 6, 10, 15, 21, 6, 10, 15, 21, 6, 10, 15, 21,
];
const K: [u32; 64] = [
0xd76aa478, 0xe8c7b756, 0x242070db, 0xc1bdceee, 0xf57c0faf, 0x4787c62a, 0xa8304613, 0xfd469501,
0x698098d8, 0x8b44f7af, 0xffff5bb1, 0x895cd7be, 0x6b901122, 0xfd987193, 0xa679438e, 0x49b40821,
0xf61e2562, 0xc040b340, 0x265e5a51, 0xe9b6c7aa, 0xd62f105d, 0x02441453, 0xd8a1e681, 0xe7d3fbc8,
0x21e1cde6, 0xc33707d6, 0xf4d50d87, 0x455a14ed, 0xa9e3e905, 0xfcefa3f8, 0x676f02d9, 0x8d2a4c8a,
0xfffa3942, 0x8771f681, 0x6d9d6122, 0xfde5380c, 0xa4beea44, 0x4bdecfa9, 0xf6bb4b60, 0xbebfbc70,
0x289b7ec6, 0xeaa127fa, 0xd4ef3085, 0x04881d05, 0xd9d4d039, 0xe6db99e5, 0x1fa27cf8, 0xc4ac5665,
0xf4292244, 0x432aff97, 0xab9423a7, 0xfc93a039, 0x655b59c3, 0x8f0ccc92, 0xffeff47d, 0x85845dd1,
0x6fa87e4f, 0xfe2ce6e0, 0xa3014314, 0x4e0811a1, 0xf7537e82, 0xbd3af235, 0x2ad7d2bb, 0xeb86d391,
];
impl Md5 {
pub fn new() -> Self {
Self {
state: [0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476],
buffer: [0u8; 64],
buffer_len: 0,
total_len: 0,
}
}
pub fn update(&mut self, mut data: &[u8]) {
self.total_len = self.total_len.wrapping_add(data.len() as u64);
if self.buffer_len > 0 {
let take = (64 - self.buffer_len).min(data.len());
self.buffer[self.buffer_len..self.buffer_len + take].copy_from_slice(&data[..take]);
self.buffer_len += take;
data = &data[take..];
if self.buffer_len == 64 {
let block = self.buffer;
self.compress(&block);
self.buffer_len = 0;
}
}
while data.len() >= 64 {
let mut block = [0u8; 64];
block.copy_from_slice(&data[..64]);
self.compress(&block);
data = &data[64..];
}
if !data.is_empty() {
self.buffer[..data.len()].copy_from_slice(data);
self.buffer_len = data.len();
}
}
pub fn finalize(mut self) -> [u8; 16] {
let bit_len = self.total_len.wrapping_mul(8);
let mut pad = [0u8; 72];
pad[0] = 0x80;
// Pad to 56 mod 64, then append the 64-bit little-endian bit length.
let pad_len = if self.buffer_len < 56 {
56 - self.buffer_len
} else {
120 - self.buffer_len
};
let total_len_before = self.total_len;
self.update(&pad[..pad_len]);
self.update(&bit_len.to_le_bytes());
debug_assert_eq!(self.buffer_len, 0);
self.total_len = total_len_before;
let mut out = [0u8; 16];
for (i, word) in self.state.iter().enumerate() {
out[i * 4..i * 4 + 4].copy_from_slice(&word.to_le_bytes());
}
out
}
fn compress(&mut self, block: &[u8; 64]) {
let mut m = [0u32; 16];
for (i, word) in m.iter_mut().enumerate() {
*word = u32::from_le_bytes([
block[i * 4],
block[i * 4 + 1],
block[i * 4 + 2],
block[i * 4 + 3],
]);
}
let [mut a, mut b, mut c, mut d] = self.state;
for i in 0..64 {
let (f, g) = match i / 16 {
0 => ((b & c) | (!b & d), i),
1 => ((d & b) | (!d & c), (5 * i + 1) % 16),
2 => (b ^ c ^ d, (3 * i + 5) % 16),
_ => (c ^ (b | !d), (7 * i) % 16),
};
let f = f.wrapping_add(a).wrapping_add(K[i]).wrapping_add(m[g]);
a = d;
d = c;
c = b;
b = b.wrapping_add(f.rotate_left(S[i]));
}
self.state[0] = self.state[0].wrapping_add(a);
self.state[1] = self.state[1].wrapping_add(b);
self.state[2] = self.state[2].wrapping_add(c);
self.state[3] = self.state[3].wrapping_add(d);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn hex(data: &[u8]) -> String {
md5_hex(&md5(data))
}
#[test]
fn rfc_1321_vectors() {
assert_eq!(hex(b""), "d41d8cd98f00b204e9800998ecf8427e");
assert_eq!(hex(b"a"), "0cc175b9c0f1b6a831c399e269772661");
assert_eq!(hex(b"abc"), "900150983cd24fb0d6963f7d28e17f72");
assert_eq!(hex(b"message digest"), "f96b697d7cb7938d525a2f31aaf161d0");
assert_eq!(
hex(b"abcdefghijklmnopqrstuvwxyz"),
"c3fcd3d76192e4007dfb496cca67e13b"
);
assert_eq!(
hex(b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"),
"d174ab98d277d9f5a5611c2c9f419d9f"
);
assert_eq!(
hex(b"12345678901234567890123456789012345678901234567890123456789012345678901234567890"),
"57edf4a22be3c955ac49da2e2107b67a"
);
}
#[test]
fn streaming_matches_one_shot_across_block_boundaries() {
let data: Vec<u8> = (0..1000u32).map(|i| (i * 7 % 251) as u8).collect();
let whole = md5(&data);
for split in [0usize, 1, 55, 56, 63, 64, 65, 127, 128, 500, 999, 1000] {
let mut h = Md5::new();
h.update(&data[..split]);
h.update(&data[split..]);
assert_eq!(h.finalize(), whole, "split={split}");
}
let mut h = Md5::new();
for b in &data {
h.update(std::slice::from_ref(b));
}
assert_eq!(h.finalize(), whole);
}
#[test]
fn padding_edge_lengths() {
// 55 / 56 / 64 bytes exercise the "pad fits in one block" vs "needs a second block" paths.
assert_eq!(hex(&[b'a'; 55]), "ef1772b6dff9a122358552954ad0df65");
assert_eq!(hex(&[b'a'; 56]), "3b0c8ac703f828b04c6c197006d17218");
assert_eq!(hex(&[b'a'; 64]), "014842d480b571495a4a0363793f7367");
}
}
+55
View File
@@ -0,0 +1,55 @@
//! Background priority for every thread the store engines create.
//!
//! A download keeps several threads busy at once (the run's worker, the tokio runtime and its
//! blocking pool for DNS, the inflate/hash/write pool, the throughput reporter, GOG's verify pass,
//! the Epic assembler). At the default nice 0 they compete as equals with the app's UI and input
//! threads, and on the device the menus' animations and controller navigation slowed visibly for
//! as long as a store download ran. At nice +10 the scheduler hands them only what the UI leaves,
//! which for a download bound by network and storage costs next to nothing.
//!
//! Linux applies `setpriority(PRIO_PROCESS, tid, …)` to the one thread `tid`, not the process, and
//! a new thread inherits its creator's nice value; every thread is still lowered explicitly at its
//! start so the guarantee does not depend on who spawned it. Only threads this library creates are
//! touched: the JNI calling thread belongs to the app and keeps its priority.
/// The nice value store-engine threads run at.
pub const BACKGROUND_NICE: i32 = 10;
/// Lower the calling thread to [`BACKGROUND_NICE`]. Best effort: a refusal leaves the thread at
/// its current priority and the download runs as before.
pub fn background() {
#[cfg(any(target_os = "android", target_os = "linux"))]
unsafe {
let tid = libc::gettid();
let _ = libc::setpriority(libc::PRIO_PROCESS, tid as libc::id_t, BACKGROUND_NICE);
}
}
/// The calling thread's nice value as a log field, e.g. `priority=nice10`.
pub fn log_field() -> String {
#[cfg(any(target_os = "android", target_os = "linux"))]
{
let nice = unsafe { libc::getpriority(libc::PRIO_PROCESS, libc::gettid() as libc::id_t) };
format!("priority=nice{nice}")
}
#[cfg(not(any(target_os = "android", target_os = "linux")))]
{
"priority=default".to_string()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_lowered_thread_reports_nice10() {
let field = std::thread::spawn(|| {
background();
log_field()
})
.join()
.unwrap();
assert_eq!(field, "priority=nice10");
}
}
@@ -0,0 +1,26 @@
//! The one export that is not a store engine: `com.droiddeck.launcher.stores.StoresNative`.
//!
//! `nativeVersion()` exists so the app can prove, at Setup time, that `libdroiddeckstores.so`
//! loaded and its JNI exports bind — before a user is mid-download and an
//! `UnsatisfiedLinkError` would surface as a failed install. It returns the crate version from
//! `Cargo.toml`, which the app can show next to the other component versions.
use jni::objects::JClass;
use jni::sys::jstring;
use jni::JNIEnv;
/// The crate version, as `Cargo.toml` declares it.
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
/// `StoresNative.nativeVersion(): String` — the crate version (`"0.1.0"`). A null return means the
/// JVM could not allocate the string, which the caller treats like a failed load.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_StoresNative_nativeVersion(
env: JNIEnv,
_class: JClass,
) -> jstring {
match env.new_string(VERSION) {
Ok(s) => s.into_raw(),
Err(_) => std::ptr::null_mut(),
}
}