Store download engines: the GOG, Epic and Amazon Rust crate

Bannerlator's bl-steam-client store modules (store_dl/{gog,epic,amazon}) and the
adaptive fetch core they run on, as their own cdylib: libdroiddeckstores.so.
None of the Steam CM/depot code comes along; the handful of tunables and the two
error types fetch_core borrowed from the depot writer live in fetch_tuning.rs,
value for value, so a fetch-window log line still reads against the same notes.

JNI exports keep every method name and signature and bind to
com.droiddeck.launcher.stores.{gog.GogNative,epic.EpicNative,amazon.AmazonNative};
StoresNative.nativeVersion() is the load probe for Setup. JNI.md is the contract
the Kotlin side mirrors. Log tags follow the class names (GogNative, EpicNative).

The Cargo.lock is Bannerlator's as is; cargo prunes it to this crate's closure on
the first build and CI reports the trimmed file.
This commit is contained in:
The412Banner committed 2026-10-08 16:17:20 -04:00
1 parent 134fa167fe
commit 401a650b5b
20 files changed
+11219

No files matched your search

+2104
View File
File diff suppressed because it is too large. Load diff
+38
View File
@@ -0,0 +1,38 @@
[package]
name = "droiddeckstores"
version = "0.1.0"
edition = "2021"
# The engines are a port of Bannerlator's bl-steam-client store modules (GPL-3.0-or-later, derived
# from WinNative's wn-steam-client; the Epic selection/verify rules trace back to GameNative and
# Legendary, both GPL-3.0). DroidDeck is GPL-3.0, so the crate keeps the licence it came with.
license = "GPL-3.0-or-later"
publish = false
[lib]
name = "droiddeckstores"
crate-type = ["cdylib"]
[dependencies]
jni = "0.21"
# zlib inflate for GOG gen2 chunks and Epic chunk bodies. The pure-Rust backend keeps the .so free
# of a C zlib and of the NDK sysroot's libz, which the apk does not ship.
flate2 = { version = "1", default-features = false, features = ["rust_backend"] }
# The fetch core drives every request through ONE tokio current-thread runtime and feeds a sync
# process pool; rustls (via ring) is the TLS stack so there is no OpenSSL to cross-compile. `gzip`
# transparently inflates a CDN answer that comes back content-encoded. No `blocking` client: the
# three engines are async-fetch only.
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "gzip"] }
tokio = { version = "1", default-features = false, features = ["rt", "net", "time", "sync"] }
futures-util = { version = "0.3", default-features = false, features = ["std"] }
# preserve_order keeps JSON object keys in source order, as in Bannerlator, so the GOG depot plan
# walks a manifest's files in the order the manifest lists them (largest-first sorting happens
# after, explicitly).
serde_json = { version = "1", features = ["preserve_order"] }
# Epic chunks are SHA-1 verified, Amazon files SHA-256; GOG's MD5 is the in-tree md5_small.
sha1 = "0.10"
sha2 = "0.10"
[profile.release]
codegen-units = 1
lto = "thin"
strip = "symbols"
+211
View File
@@ -0,0 +1,211 @@
# `libdroiddeckstores.so` — the JNI contract
The Kotlin side of the store download engines mirrors this file exactly. Every symbol the library
exports is listed here with the class it binds to, the JNI signature, when it is called and on
which thread. The engines came from Bannerlator's `bl-steam-client` crate; the method names and
signatures are identical to Bannerlator's `BlGogDownload` / `BlEpicDownload` / `BlAmazonDownload`,
only the package and class names changed.
Load it once with `System.loadLibrary("droiddeckstores")`. The library is built by Gradle
(`buildRustStores`, see `app/build.gradle`) and is absent from a `-PskipRust=true` build, so every
call site must be ready for `UnsatisfiedLinkError` on `loadLibrary` or on the first `external`
call and degrade to "store unavailable".
## Conventions that hold for all three engines
- The `external` functions are declared `@JvmStatic` in a Kotlin `object` (a `JClass` arrives as
the second JNI argument). Declaring them in a plain `object` without `@JvmStatic` also binds —
the symbol name is the same — but keep `@JvmStatic` so the contract matches Bannerlator's.
- No overloads: the exported names are the short JNI form (`Java_<package>_<Class>_<method>`), so
adding a second `nativeStart` with a different parameter list would NOT bind.
- Listener objects are only ever reached through `CallVoidMethod` by name + signature on the
object that was passed to `nativeStart`. The library never does `FindClass`, so the listener's
class may be any class (interface, nested interface, anonymous object) that declares the listed
methods with exactly these signatures. The names suggested below are what the Rust doc comments
use.
- Every listener method runs on a native thread (the run's worker thread, a process-pool thread
or the throughput reporter) that is attached to the JVM as a daemon. Never touch Views directly;
post to the main thread. A Java exception thrown by a listener method is cleared and ignored by
the native side — it does not stop the run.
- `nativeStart` returns an opaque handle (`long`). `0` means the run did not start. The handle is
a heap pointer: call `nativeRelease(handle)` exactly once per non-zero handle (after
`onComplete`), never twice, never with a made-up value.
- `nativeCancel(handle)` is idempotent and safe at any time between `nativeStart` and
`nativeRelease`. The run stops at its next scheduling point; `onComplete` still fires.
- `nativeRelease(handle)` is safe even while a cancelled run is still winding down: the worker
thread owns its own references (cancel flag, listener `GlobalRef`).
- `caBundlePath`: a PEM bundle the HTTPS client adds as extra root certificates; `""` = the roots
rustls ships with (webpki). Bannerlator extracts the system CA store to a file for this; on
DroidDeck `""` is the sensible default unless a CDN fails TLS verification.
- `maxWorkers` is the ceiling of the adaptive in-flight request window; `processWorkers` is the
number of sync inflate+hash+write threads. Bannerlator passes its "download threads" preference
for both (8 by default).
## `com.droiddeck.launcher.stores.StoresNative`
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeVersion(): String` | `()Ljava/lang/String;` | `Java_com_droiddeck_launcher_stores_StoresNative_nativeVersion` |
Returns the crate version from `Cargo.toml` (`"0.1.0"`). Call it from Setup, after
`System.loadLibrary("droiddeckstores")`, to prove the library loads and binds; show the string
next to the other component versions. May return `null` only if the JVM cannot allocate a string.
```kotlin
package com.droiddeck.launcher.stores
object StoresNative {
@JvmStatic external fun nativeVersion(): String
}
```
## `com.droiddeck.launcher.stores.gog.GogNative`
GOG gen2 chunk engine (depot manifests → chunk fetch + zlib inflate + MD5) and gen1 (build manifest
→ per-file HTTP Range GET streamed to disk). One `nativeStart` = one download loop (base install,
DLC install, or a dependency-redist assembly). The Kotlin manager owns everything before (token,
builds, manifest head, product + language depot selection, secure-link resolution, disk guard) and
after it (marker files, exe resolution, DLC markers, redist orchestration).
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeProbe(): Int` | `()I` | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeProbe` |
| `nativeStart(...)`: Long | `(I[Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;IIZLjava/lang/String;Ljava/lang/Object;)J` (last parameter is the listener; its static type is the app's choice) | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeStart` |
| `nativeCancel(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeCancel` |
| `nativeRelease(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_gog_GogNative_nativeRelease` |
`nativeProbe()` always returns `1`; it exists so `isAvailable()` can catch `UnsatisfiedLinkError`
once at startup instead of mid-download.
`nativeStart` parameters, in order:
| # | Parameter | Meaning |
|---|---|---|
| 1 | `kind: Int` | `0` = `KIND_GEN2_CHUNKS`, `1` = `KIND_GEN1_RANGES` (anything else → gen2) |
| 2 | `depotManifests: Array<String>` | gen2: the inflated depot-manifest JSON strings, in fetch order, already filtered by product + language; gen1: the inflated build manifest (one element) |
| 3 | `cdnBase: String` | gen2: the resolved secure-link base (query string intact) or the unauthenticated dependency store base; gen1: `""` (file URLs are in the manifest) |
| 4 | `installDir: String` | absolute install directory |
| 5 | `skipPaths: Array<String>` | relative paths already completed by an earlier run of this same download (secure-link refresh re-run): counted done without re-hashing, no progress event |
| 6 | `caBundlePath: String` | see conventions |
| 7 | `maxWorkers: Int` | window ceiling for this loop (Bannerlator: its download-threads preference, 8, or 1 for redists) |
| 8 | `processWorkers: Int` | inflate/hash/write threads |
| 9 | `sortLargestFirst: Boolean` | base install = `true` (largest files first); DLC / dependency = `false` |
| 10 | `label: String` | log prefix, e.g. `"gog base"`; `""` → `"gog"` |
| 11 | `listener` | see below |
Returns `0` without any callback when: the listener is null, the JavaVM cannot be obtained, the
install dir or manifest list is empty, `cdnBase` is empty for gen2, or the worker thread cannot be
spawned. Otherwise the run is on a thread named `gog-dl` and `onComplete` fires exactly once.
Suggested listener: `com.droiddeck.launcher.stores.gog.GogNative.Listener` (an interface).
| Listener method | JNI signature | When |
|---|---|---|
| `onProgress(bytesDone: Long, bytesTotal: Long, filesDone: Int, filesTotal: Int, file: String, fileBytes: Long, verified: Boolean)` | `(JJIILjava/lang/String;JZ)V` | one file reached its final state. `verified = true` = resume-skip (existing file passed size+MD5; no bytes credited, `fileBytes` = its size); `false` = freshly assembled, size+MD5-verified and renamed (`fileBytes` = its decompressed size). `filesDone` counts both, `bytesDone` counts assembled bytes only. Process-pool / worker thread. |
| `onLog(line: String)` | `(Ljava/lang/String;)V` | engine diagnostics; the same line already went to logcat under the tag `GogNative`. Any native thread. |
| `onComplete(success: Boolean, cancelled: Boolean, linkExpiry: Boolean, error: String, bytesWritten: Long, filesDone: Int)` | `(ZZZLjava/lang/String;JI)V` | exactly once, from the worker thread, last. `linkExpiry = true` = the run died on an HTTP 401/403/404/500, the codes the GOG secure link returns once it expired: refresh the link and call `nativeStart` again with the finished files in `skipPaths`. `error` is `""` on success. |
On-disk protocol (what a resumed run, by either engine, relies on): a file is assembled into
`<file>.bhtmp`, then renamed over `<file>` once its size and MD5 match the manifest; an existing
`<file>` whose size+MD5 match is skipped. The engine writes nothing else.
## `com.droiddeck.launcher.stores.epic.EpicNative`
The chunk-fetch inner loop of an Epic install: fills `<installDir>/.chunks/<GUID>` with verified,
decompressed chunks for the files the manager says are pending. The Kotlin manager still fetches
and parses the manifest API JSON, downloads the manifest, selects files (install tags), runs the
delta/verify pass, assembles files from the chunk cache and does every post-install step. The
engine re-parses the same manifest bytes (ChunksV4 binary or legacy JSON), rebuilds the chunk plan
for the pending files, skips chunks already in the cache, and writes verified chunks with a
`.part` + rename protocol, so the on-disk state is the same whichever side fetched.
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeStart(...)`: Long | `([BLjava/lang/String;[Ljava/lang/String;[IIJLjava/lang/String;IILjava/lang/Object;)J` (last parameter is the listener) | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeStart` |
| `nativeCancel(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeCancel` |
| `nativeRelease(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeRelease` |
There is no `nativeProbe`; Bannerlator probes by catching the `Throwable` from `loadLibrary` /
the first `nativeStart`.
`nativeStart` parameters, in order:
| # | Parameter | Meaning |
|---|---|---|
| 1 | `manifest: ByteArray` | the raw manifest bytes the manager downloaded (binary ChunksV4 or JSON) |
| 2 | `installDir: String` | absolute install directory; the chunk cache is `<installDir>/.chunks` |
| 3 | `cdnPrefixes: Array<String>` | the CDN base URLs from the manifest API answer (`.../ChunksV4/<n>/` style); one or many, distinct hosts widen the window |
| 4 | `pendingFileIdx: IntArray` | indices into the manifest's file list of the files still to fetch (the manager's delta/verify result) |
| 5 | `expectedChunks: Int` | the manager's own needed-chunk count for a cross-check; `-1` = no cross-check |
| 6 | `expectedBytes: Long` | the manager's `Σ max(fileSize, 1)` for a cross-check; `-1` = no cross-check |
| 7 | `caBundlePath: String` | see conventions |
| 8 | `maxWorkers: Int` | window ceiling (Bannerlator: 8) |
| 9 | `processWorkers: Int` | inflate/verify/write threads |
| 10 | `listener` | see below |
Planning (manifest parse + plan + cross-check) runs synchronously on the calling thread, so a
plan failure returns `0` BEFORE any fetch; the manager then runs its own fallback loop. On `0`,
`onComplete(false, reason, 0)` has already been called synchronously (except for a null listener).
On success, `onPlan` has been called synchronously before `nativeStart` returns, then the fetch
runs on a new thread.
Suggested listener: `com.droiddeck.launcher.stores.epic.EpicNative.Listener` (nested interface,
as in Bannerlator).
| Listener method | JNI signature | When |
|---|---|---|
| `onPlan(chunksTotal: Int, bytesTotal: Long, chunkDir: String)` | `(IJLjava/lang/String;)V` | once, synchronously inside `nativeStart`, before any fetch: the plan the engine derived |
| `onProgress(bytesDone: Long, bytesTotal: Long, chunksDone: Int, chunksTotal: Int)` | `(JJII)V` | per accounted chunk (cached-skip or fetched); process-pool thread |
| `onLog(line: String)` | `(Ljava/lang/String;)V` | engine log line, already in logcat under `EpicNative`; any native thread (also synchronously from `nativeStart` for the start line) |
| `onComplete(success: Boolean, error: String, bytesCredited: Long)` | `(ZLjava/lang/String;J)V` | terminal, exactly once (worker thread; or synchronously when `nativeStart` returns `0`) |
Bannerlator's `BlEpicDownload.run(...)` wraps this in a blocking call that polls an
`AtomicBoolean` cancel flag every 250 ms, calls `nativeCancel` on cancel, waits up to 5 s for
`onComplete`, then `nativeRelease` in `finally`. That shape is worth keeping.
## `com.droiddeck.launcher.stores.amazon.AmazonNative`
The file-fetch pool of an Amazon Games install: one streamed HTTP GET per manifest file, SHA-256
checked, written as `<dest>.tmp` and renamed. The manager keeps manifest/auth/markers/post-install
and hands the engine a JSON plan.
| Method | JNI signature | Symbol |
|---|---|---|
| `nativeStart(...)`: Long | `(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;IILjava/lang/Object;)J` (last parameter is the listener) | `Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeStart` |
| `nativeCancel(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeCancel` |
| `nativeRelease(handle: Long)` | `(J)V` | `Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeRelease` |
`nativeStart` parameters, in order:
| # | Parameter | Meaning |
|---|---|---|
| 1 | `planJson: String` | a JSON array, one object per manifest file: `{"relPath": "...", "url": "https://...", "size": <long>, "sha256hex": "..."}` (`sha256hex` may be `""` = no hash check). Every manifest file goes in: the engine applies the size-based resume-skip itself. |
| 2 | `installDir: String` | absolute install directory |
| 3 | `caBundlePath: String` | see conventions |
| 4 | `maxWorkers: Int` | window size; `<= 0` = the engine default (8, Amazon's own `MAX_PARALLEL`) |
| 5 | `processWorkers: Int` | sync verify+write threads (`<= 0` → 1) |
| 6 | `listener` | see below |
Returns `0` when the listener is null (no callback), or when `planJson` / `installDir` is empty
(`onComplete(false, "empty plan or install dir", 0)` fires synchronously first), or when the
listener `GlobalRef` cannot be taken (no callback). The engine keeps the `JavaVM` it takes from
the first `nativeStart`.
Suggested listener: `com.droiddeck.launcher.stores.amazon.AmazonNative.Listener` (an interface).
| Listener method | JNI signature | When |
|---|---|---|
| `onProgress(bytesDone: Long, bytesTotal: Long, filesDone: Long, filesTotal: Long)` | `(JJJJ)V` | once up front with the resume-skipped credit, then after each committed file (and as pieces stream in); `bytesDone` includes skipped files. Native threads. |
| `onLog(line: String)` | `(Ljava/lang/String;)V` | one diagnostic line (`engine=rust …`, `fetch-window …`, `summary …`). The Amazon engine does NOT write to logcat itself: the Kotlin side is the one place that logs these, so each line appears once. |
| `onComplete(success: Boolean, error: String, bytesWritten: Long)` | `(ZLjava/lang/String;J)V` | exactly once, when the run succeeds, fails or is cancelled (`error = "cancelled"`). |
Bannerlator's `BlAmazonDownload.runBlocking(...)` polls a cancel probe every 100 ms, calls
`nativeCancel` once, and `nativeRelease` in `finally` after `onComplete`; on thread interrupt it
cancels and waits up to 30 s for the run to wind down before releasing.
## Log tags
Lines go to logcat under `GogNative` and `EpicNative` (the Amazon engine logs only through
`onLog`). Bannerlator used `BL_GOG_DL` / `BL_EPIC_DL` / `BL_AMAZON_DL`; the line grammar
(`fetch-start`, `fetch-window`, `throughput`, `fetch-end`, `summary`) is unchanged, so its tuning
notes still read against a DroidDeck log.
+989
View File
@@ -0,0 +1,989 @@
//! Amazon Games download adapter over the shared fetch core.
//!
//! Replaces ONLY the byte-fetching pool of `AmazonDownloadManager.install()` (Step 4).
//! Java still resolves the download spec, fetches and parses `manifest.proto`, and hands
//! this module the resolved file list (one JSON entry per manifest file with the exact
//! URL the Java loop would have opened). The adapter mirrors the Java rules 1:1 where the
//! core allows it — see Bannerlator's `docs/RUST_AMAZON_PARITY.md` §4 / §9 for the table and the
//! documented deviations:
//!
//! * one `FetchItem` per WHOLE file, streamed (`FetchOptions.stream = true`): pieces are
//! appended to `<dest>.tmp` as they arrive, 8 files wide like Java's pool, memory bounded
//! to the pieces not yet written;
//! * resume-skip = `st_size == manifest size`, no hash check on skip (Java `:250`);
//! * SHA-256 is accumulated while streaming and checked in `on_finish` when the manifest
//! carries a hash; a mismatch deletes the tmp and is a retryable failure (Java `:269`);
//! * `<dest>.tmp` → delete existing dest → rename (Java `:280`); rename failure is fatal
//! without retry (Java `:281`);
//! * a piece credits its wire length to progress as it is written (Java `:347`), and pieces
//! of an attempt that later fails stay counted (Java never rolls `totalDownloaded` back).
pub mod jni;
use crate::fetch_core::{run_fetch, FetchItem, FetchOptions, FetchSink, SinkError};
use sha2::{Digest, Sha256};
use std::collections::HashMap;
use std::fs::{self, File};
use std::io::Write;
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
/// Java `AmazonDownloadManager.MAX_PARALLEL`.
pub const MAX_PARALLEL: usize = 8;
/// Java `AmazonDownloadManager.DOWNLOAD_USER_AGENT`.
pub const DOWNLOAD_USER_AGENT: &str = "nile/0.1 Amazon";
/// Java `conn.setReadTimeout(120000)`; in stream mode the core applies this to the response
/// headers and then as an idle deadline per received piece — the same shape as a read timeout.
pub const REQUEST_TIMEOUT: Duration = Duration::from_secs(120);
/// Java `new File(installDir, file.unixPath() + ".tmp")`.
pub const TMP_SUFFIX: &str = ".tmp";
/// Log label used in the core's `fetch-window` lines.
pub const LABEL: &str = "amazon";
/// One resolved manifest file as serialised by Java (`AmazonDownloadManager.buildRustPlan`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PlanEntry {
/// `ManifestFile.unixPath()` — relative to the install dir, forward slashes.
pub rel_path: String,
/// `AmazonApiClient.appendPath(downloadUrl, "files/" + hashHex)` — signed, whole file.
pub url: String,
/// `ManifestFile.size`.
pub size: u64,
/// SHA-256 to verify against; EMPTY = Java would not verify (`hashAlgorithm != 0` or
/// no hash bytes).
pub sha256: Vec<u8>,
}
/// Parse the JSON array `[{relPath, url, size, sha256hex}]`. `relPath` is normalised the same
/// way as `ManifestFile.unixPath()` so a caller that forgot is still byte-identical.
pub fn parse_plan(json: &str) -> Result<Vec<PlanEntry>, String> {
let value: serde_json::Value =
serde_json::from_str(json).map_err(|err| format!("plan json: {err}"))?;
let Some(array) = value.as_array() else {
return Err("plan json: expected an array".to_string());
};
let mut entries = Vec::with_capacity(array.len());
for (index, item) in array.iter().enumerate() {
let rel_path = item
.get("relPath")
.and_then(|v| v.as_str())
.map(unix_path)
.unwrap_or_default();
if rel_path.is_empty() {
return Err(format!("plan json: entry {index} has no relPath"));
}
let url = item
.get("url")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
if url.is_empty() {
return Err(format!("plan json: entry {index} ({rel_path}) has no url"));
}
let size = item
.get("size")
.and_then(|v| v.as_u64().or_else(|| v.as_i64().map(|s| s.max(0) as u64)))
.unwrap_or(0);
let sha_hex = item
.get("sha256hex")
.and_then(|v| v.as_str())
.unwrap_or_default();
let sha256 = if sha_hex.is_empty() {
Vec::new()
} else {
hex_decode(sha_hex)
.ok_or_else(|| format!("plan json: entry {index} ({rel_path}) bad sha256hex"))?
};
entries.push(PlanEntry {
rel_path,
url,
size,
sha256,
});
}
Ok(entries)
}
/// `ManifestFile.unixPath()`.
pub fn unix_path(path: &str) -> String {
path.replace('\\', "/")
}
/// Lowercase/uppercase hex → bytes; `None` on odd length or a non-hex digit.
pub fn hex_decode(hex: &str) -> Option<Vec<u8>> {
let bytes = hex.as_bytes();
if bytes.len() % 2 != 0 {
return None;
}
let nibble = |b: u8| -> Option<u8> {
match b {
b'0'..=b'9' => Some(b - b'0'),
b'a'..=b'f' => Some(b - b'a' + 10),
b'A'..=b'F' => Some(b - b'A' + 10),
_ => None,
}
};
let mut out = Vec::with_capacity(bytes.len() / 2);
for pair in bytes.chunks(2) {
out.push((nibble(pair[0])? << 4) | nibble(pair[1])?);
}
Some(out)
}
/// `new File(installDir, rel)` — Java resolves the child textually under the parent, so
/// build the path by concatenation (never `Path::join`, which would let a leading `/` escape).
pub fn dest_path(install_dir: &str, rel_path: &str) -> PathBuf {
let base = install_dir.trim_end_matches('/');
PathBuf::from(format!("{base}/{rel_path}"))
}
/// `new File(installDir, rel + ".tmp")`.
pub fn tmp_path(install_dir: &str, rel_path: &str) -> PathBuf {
dest_path(install_dir, &format!("{rel_path}{TMP_SUFFIX}"))
}
/// Java `:250`: `destFile.exists() && destFile.length() == file.size`. `File.length()` is
/// `st_size`, which is what `Metadata::len()` returns — including for a directory, so the
/// (degenerate) behaviours match too. No hash is checked on skip, on purpose.
pub fn is_present_and_complete(install_dir: &str, entry: &PlanEntry) -> bool {
fs::metadata(dest_path(install_dir, &entry.rel_path))
.map(|meta| meta.len() == entry.size)
.unwrap_or(false)
}
/// `scheme://host[:port]` of a URL — the per-host cap key. Amazon serves every file from the
/// one signed CDN base, so this yields a single host.
pub fn host_key(url: &str) -> String {
let (scheme, rest) = match url.find("://") {
Some(idx) => (&url[..idx], &url[idx + 3..]),
None => ("https", url),
};
let end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
format!("{scheme}://{}", &rest[..end])
}
/// Improvements round 1: `max(6, ceil(max_workers / distinct_hosts))` — a single-host store
/// (Amazon's one signed a2z base) gets the whole ceiling on its one host; the core clamps the
/// window to `hosts × per_host_cap`, so anything smaller would silently cut the ceiling.
pub fn per_host_cap_for(max_workers: usize, distinct_hosts: usize) -> usize {
let hosts = distinct_hosts.max(1);
let per_host = max_workers.max(1).div_ceil(hosts);
per_host.max(crate::fetch_tuning::PER_HOST_CAP)
}
/// The resolved work for one run.
#[derive(Debug, Default)]
pub struct Plan {
/// Items to fetch (index into `entries` via `FetchItem.id`).
pub items: Vec<FetchItem>,
/// Every manifest file, in manifest order (fetched and skipped alike).
pub entries: Vec<PlanEntry>,
/// Σ size of all entries = `ParsedManifest.totalInstallSize`.
pub total_bytes: u64,
/// Bytes credited up front for files that passed the resume check.
pub skipped_bytes: u64,
/// Number of files that passed the resume check.
pub skipped_files: u64,
/// Distinct host keys in fetch order.
pub hosts: Vec<String>,
}
/// Apply the Java resume-skip rule to every entry and turn the rest into whole-file items.
pub fn build_plan(entries: Vec<PlanEntry>, install_dir: &str) -> Plan {
let mut plan = Plan {
total_bytes: entries.iter().map(|e| e.size).sum(),
..Plan::default()
};
for (index, entry) in entries.iter().enumerate() {
if is_present_and_complete(install_dir, entry) {
plan.skipped_bytes = plan.skipped_bytes.saturating_add(entry.size);
plan.skipped_files += 1;
continue;
}
let host = host_key(&entry.url);
if !plan.hosts.contains(&host) {
plan.hosts.push(host);
}
plan.items.push(FetchItem {
id: index as u64,
urls: vec![entry.url.clone()],
reserve: entry.size,
range: None,
});
}
if plan.hosts.is_empty() {
plan.hosts.push("https://amazon".to_string());
}
plan.entries = entries;
plan
}
/// One `<dest>.tmp` being streamed: the open handle, the running hash and the bytes written
/// by the current attempt.
struct OpenTmp {
file: File,
hasher: Sha256,
written: u64,
}
/// Sink: stream pieces into `<dest>.tmp` (SHA-256 accumulated on the fly), verify and rename
/// in `on_finish`. `process` (whole-body mode) does the same in one step.
pub struct AmazonSink {
install_dir: String,
entries: Vec<PlanEntry>,
cancel: Arc<AtomicBool>,
files_done: AtomicU64,
/// Per-item open tmp files. The map lock is held only to look the entry up; the write
/// itself happens under the per-item lock so 8 streams never serialise on one mutex.
open: Mutex<HashMap<u64, Arc<Mutex<OpenTmp>>>>,
}
impl AmazonSink {
pub fn new(install_dir: &str, entries: Vec<PlanEntry>, cancel: Arc<AtomicBool>) -> Self {
Self {
install_dir: install_dir.to_string(),
entries,
cancel,
files_done: AtomicU64::new(0),
open: Mutex::new(HashMap::new()),
}
}
pub fn files_done(&self) -> u64 {
self.files_done.load(Ordering::Relaxed)
}
fn entry(&self, item: &FetchItem) -> Result<&PlanEntry, SinkError> {
self.entries
.get(item.id as usize)
.ok_or_else(|| SinkError::Fatal(format!("plan index {} out of range", item.id)))
}
/// Java `:260` — `destFile.getParentFile().mkdirs()` before each attempt; a failure
/// surfaces on the create/write below, like Java's.
fn ensure_parent(&self, entry: &PlanEntry) {
if let Some(parent) = dest_path(&self.install_dir, &entry.rel_path).parent() {
let _ = fs::create_dir_all(parent);
}
}
/// Java `:280-284`: delete an existing dest, rename tmp → dest; rename failure fails the
/// file with NO retry (→ whole install fails).
fn rename_into_place(&self, entry: &PlanEntry) -> Result<(), SinkError> {
let dest = dest_path(&self.install_dir, &entry.rel_path);
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
if dest.exists() {
let _ = fs::remove_file(&dest);
}
if let Err(err) = fs::rename(&tmp, &dest) {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Fatal(format!(
"Failed to rename tmp → {} ({err})",
dest.display()
)));
}
self.files_done.fetch_add(1, Ordering::Relaxed);
Ok(())
}
/// Java `:267-277`: SHA-256 check when the manifest carries a hash; a mismatch deletes the
/// tmp and is retried.
fn verify(&self, entry: &PlanEntry, digest: &[u8]) -> Result<(), SinkError> {
if !entry.sha256.is_empty() && digest != entry.sha256.as_slice() {
let _ = fs::remove_file(tmp_path(&self.install_dir, &entry.rel_path));
return Err(SinkError::Retry(format!(
"SHA-256 mismatch for: {}",
entry.rel_path
)));
}
Ok(())
}
/// Whole-body path (`FetchOptions.stream = false`): verify → write tmp → rename.
pub fn commit(&self, entry: &PlanEntry, body: &[u8]) -> Result<u64, SinkError> {
if !entry.sha256.is_empty() {
let digest = Sha256::digest(body);
if digest.as_slice() != entry.sha256.as_slice() {
return Err(SinkError::Retry(format!(
"SHA-256 mismatch for: {}",
entry.rel_path
)));
}
}
self.ensure_parent(entry);
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
if let Err(err) = fs::write(&tmp, body) {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!("write {}: {err}", tmp.display())));
}
self.rename_into_place(entry)?;
Ok(body.len() as u64)
}
/// After the run: close and delete every tmp that never reached `on_finish` (a stream that
/// failed after max attempts, or was cut by cancel / a fatal error). Java deletes the tmp
/// of every failed or cancelled attempt (`:271`, `:288`, `:293`, `:297`), so no `.tmp`
/// may survive a run on either engine. Returns how many were removed.
pub fn cleanup_partials(&self) -> usize {
let leftovers: Vec<(u64, Arc<Mutex<OpenTmp>>)> = match self.open.lock() {
Ok(mut open) => open.drain().collect(),
Err(_) => return 0,
};
let mut removed = 0;
for (id, open) in leftovers {
drop(open);
if let Some(entry) = self.entries.get(id as usize) {
if fs::remove_file(tmp_path(&self.install_dir, &entry.rel_path)).is_ok() {
removed += 1;
}
}
}
removed
}
}
impl FetchSink for AmazonSink {
fn process(&self, item: &FetchItem, body: Vec<u8>) -> Result<u64, SinkError> {
if self.cancel.load(Ordering::Relaxed) {
return Err(SinkError::Fatal("cancelled".to_string()));
}
let entry = self.entry(item)?;
self.commit(entry, &body)
}
fn on_chunk(&self, item: &FetchItem, offset: u64, data: &[u8]) -> Result<(), SinkError> {
if self.cancel.load(Ordering::Relaxed) {
// Java `:342-344`: cancel inside the read loop → tmp deleted (cleanup_partials).
return Err(SinkError::Fatal("cancelled".to_string()));
}
let entry = self.entry(item)?;
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
let open = if offset == 0 {
// (Re)start of an attempt: Java opens a fresh FileOutputStream (truncating).
self.ensure_parent(entry);
let file = File::create(&tmp).map_err(|err| {
let _ = fs::remove_file(&tmp);
SinkError::Retry(format!("create {}: {err}", tmp.display()))
})?;
let open = Arc::new(Mutex::new(OpenTmp {
file,
hasher: Sha256::new(),
written: 0,
}));
match self.open.lock() {
Ok(mut map) => {
map.insert(item.id, Arc::clone(&open));
}
Err(_) => return Err(SinkError::Fatal("sink map poisoned".to_string())),
}
open
} else {
let found = match self.open.lock() {
Ok(map) => map.get(&item.id).cloned(),
Err(_) => return Err(SinkError::Fatal("sink map poisoned".to_string())),
};
match found {
Some(open) => open,
None => {
return Err(SinkError::Retry(format!(
"piece at offset {offset} without a stream start for: {}",
entry.rel_path
)))
}
}
};
let Ok(mut state) = open.lock() else {
return Err(SinkError::Fatal("sink item poisoned".to_string()));
};
if state.written != offset {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!(
"piece out of order ({} != {offset}) for: {}",
state.written, entry.rel_path
)));
}
if let Err(err) = state.file.write_all(data) {
// Java: IOException inside downloadFile → tmp deleted → next attempt.
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!("write {}: {err}", tmp.display())));
}
state.hasher.update(data);
state.written = state.written.saturating_add(data.len() as u64);
Ok(())
}
fn on_finish(&self, item: &FetchItem, total_len: u64) -> Result<u64, SinkError> {
let entry = self.entry(item)?;
let tmp = tmp_path(&self.install_dir, &entry.rel_path);
let taken = match self.open.lock() {
Ok(mut map) => map.remove(&item.id),
Err(_) => return Err(SinkError::Fatal("sink map poisoned".to_string())),
};
let (written, digest) = match taken {
Some(open) => {
let Ok(mut state) = open.lock() else {
return Err(SinkError::Fatal("sink item poisoned".to_string()));
};
if let Err(err) = state.file.flush() {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!("flush {}: {err}", tmp.display())));
}
let written = state.written;
let digest = std::mem::replace(&mut state.hasher, Sha256::new()).finalize();
(written, digest)
}
None => {
// Zero-byte body: no piece ever arrived. Java would have created an empty tmp.
if total_len != 0 {
return Err(SinkError::Retry(format!(
"finish without a stream for: {}",
entry.rel_path
)));
}
self.ensure_parent(entry);
if let Err(err) = File::create(&tmp) {
return Err(SinkError::Retry(format!("create {}: {err}", tmp.display())));
}
(0, Sha256::new().finalize())
}
};
if written != total_len {
let _ = fs::remove_file(&tmp);
return Err(SinkError::Retry(format!(
"stream length {written} != {total_len} for: {}",
entry.rel_path
)));
}
self.verify(entry, digest.as_slice())?;
self.rename_into_place(entry)?;
Ok(0)
}
}
/// Outcome of one `run_download`.
#[derive(Clone, Debug, Default)]
pub struct RunResult {
pub success: bool,
pub cancelled: bool,
pub error: String,
/// Bytes fetched by this run (skipped files excluded; failed attempts' pieces included,
/// like Java's `totalDownloaded`).
pub bytes_written: u64,
/// Files committed by this run plus files skipped by the resume check.
pub files_done: u64,
pub files_total: u64,
}
/// Throughput bookkeeping for the final summary line (avg over the run, peak over ≥500 ms
/// progress intervals — the same sampling window the Java speed label uses).
struct SpeedMeter {
started: Instant,
last_at: Instant,
last_bytes: u64,
peak_bps: f64,
}
impl SpeedMeter {
fn new() -> Self {
let now = Instant::now();
Self {
started: now,
last_at: now,
last_bytes: 0,
peak_bps: 0.0,
}
}
fn observe(&mut self, bytes: u64) {
let now = Instant::now();
let dt = now.duration_since(self.last_at).as_secs_f64();
if dt >= 0.5 {
let bps = bytes.saturating_sub(self.last_bytes) as f64 / dt;
if bps > self.peak_bps {
self.peak_bps = bps;
}
self.last_at = now;
self.last_bytes = bytes;
}
}
fn summary(&self, bytes: u64) -> (f64, f64, f64) {
let elapsed = self.started.elapsed().as_secs_f64().max(0.001);
let avg = bytes as f64 / elapsed;
(elapsed, avg * 8.0 / 1_000_000.0, self.peak_bps * 8.0 / 1_000_000.0)
}
}
/// Blocking: plan → stream-fetch → verify/rename, calling `progress(bytes_done, bytes_total,
/// files_done, files_total)` after each written piece and each finished file (skipped files
/// are pre-credited) and `log` for every diagnostic line. `bytes_done` includes skipped bytes,
/// matching Java's `totalDownloaded` (which credits skipped files at `:251`).
pub fn run_download(
plan_json: &str,
install_dir: &str,
ca_bundle_path: &str,
max_workers: usize,
process_workers: usize,
cancel: Arc<AtomicBool>,
progress: &(dyn Fn(u64, u64, u64, u64) + Sync),
log: &(dyn Fn(&str) + Sync),
) -> RunResult {
let max_workers = if max_workers == 0 { MAX_PARALLEL } else { max_workers };
let process_workers = process_workers.max(1);
let entries = match parse_plan(plan_json) {
Ok(entries) => entries,
Err(err) => {
log(&format!("engine=rust plan error: {err}"));
return RunResult {
error: err,
..RunResult::default()
};
}
};
let plan = build_plan(entries, install_dir);
let files_total = plan.entries.len() as u64;
let fetch_bytes: u64 = plan.items.iter().map(|item| item.reserve).sum();
let per_host_cap = per_host_cap_for(max_workers, plan.hosts.len());
log(&format!(
"engine=rust mode=stream plan={} files skip={} ({} bytes) fetch={} ({} bytes) hosts={} workers={} per_host_cap={} process={} dir={}",
files_total,
plan.skipped_files,
plan.skipped_bytes,
plan.items.len(),
fetch_bytes,
plan.hosts.len(),
max_workers,
per_host_cap,
process_workers,
install_dir
));
progress(plan.skipped_bytes, plan.total_bytes, plan.skipped_files, files_total);
if plan.items.is_empty() {
log("summary bytes=0 elapsed=0.000 avg_mbps=0.0 peak_mbps=0.0 files=all-present");
return RunResult {
success: true,
files_done: plan.skipped_files,
files_total,
..RunResult::default()
};
}
if cancel.load(Ordering::Relaxed) {
return RunResult {
cancelled: true,
error: "cancelled".to_string(),
files_done: plan.skipped_files,
files_total,
..RunResult::default()
};
}
let opts = FetchOptions {
max_workers,
// Every Amazon file comes from the one signed CDN base: give that host the whole
// ceiling (`per_host_cap_for`), or the core would clamp the window to hosts × cap.
per_host_cap,
timeout: REQUEST_TIMEOUT,
headers: vec![("User-Agent".to_string(), DOWNLOAD_USER_AGENT.to_string())],
ca_bundle_path: ca_bundle_path.to_string(),
process_workers,
label: LABEL.to_string(),
// Whole files, streamed into the tmp as they arrive: 8 wide like Java's pool, memory
// bounded to the pieces not yet written.
stream: true,
..FetchOptions::default()
};
let sink = AmazonSink::new(install_dir, plan.entries.clone(), Arc::clone(&cancel));
let meter = Mutex::new(SpeedMeter::new());
let skipped_bytes = plan.skipped_bytes;
let skipped_files = plan.skipped_files;
let total_bytes = plan.total_bytes;
let progress_cb = |credited: u64, items_ok: u64| {
if let Ok(mut meter) = meter.lock() {
meter.observe(credited);
}
progress(
skipped_bytes.saturating_add(credited),
total_bytes,
skipped_files.saturating_add(items_ok),
files_total,
);
};
let outcome = run_fetch(
plan.items,
&plan.hosts,
&opts,
&sink,
cancel.as_ref(),
&progress_cb,
log,
);
let leftovers = sink.cleanup_partials();
let (elapsed, avg_mbps, peak_mbps) = meter
.lock()
.map(|meter| meter.summary(outcome.bytes_credited))
.unwrap_or((0.0, 0.0, 0.0));
let files_done = skipped_files.saturating_add(sink.files_done());
log(&format!(
"summary bytes={} elapsed={:.3} avg_mbps={:.1} peak_mbps={:.1} files={}/{} items_ok={} tmp_removed={} cancelled={} error={}",
outcome.bytes_credited,
elapsed,
avg_mbps,
peak_mbps,
files_done,
files_total,
outcome.items_ok,
leftovers,
outcome.cancelled,
outcome.error.as_deref().unwrap_or("")
));
let cancelled = outcome.cancelled || cancel.load(Ordering::Relaxed);
let error = if cancelled {
outcome.error.unwrap_or_else(|| "cancelled".to_string())
} else {
outcome.error.unwrap_or_default()
};
RunResult {
success: !cancelled && error.is_empty(),
cancelled,
error,
bytes_written: outcome.bytes_credited,
files_done,
files_total,
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::AtomicUsize;
static COUNTER: AtomicUsize = AtomicUsize::new(0);
fn scratch_dir() -> String {
let n = COUNTER.fetch_add(1, Ordering::Relaxed);
let dir = std::env::temp_dir().join(format!(
"amazon-test-{}-{}",
std::process::id(),
n
));
let _ = fs::remove_dir_all(&dir);
fs::create_dir_all(&dir).unwrap();
dir.to_string_lossy().into_owned()
}
fn sha_hex(data: &[u8]) -> String {
crate::gog::to_hex(Sha256::digest(data).as_slice())
}
fn item_for(entry: &PlanEntry, id: u64) -> FetchItem {
FetchItem {
id,
urls: vec![entry.url.clone()],
reserve: entry.size,
range: None,
}
}
const PLAN: &str = r#"[
{"relPath":"Binaries\\Win64\\Game.exe","url":"https://d1.cdn.example/base/files/aa?Sig=1","size":4,"sha256hex":"9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"},
{"relPath":"data/level1.pak","url":"https://d1.cdn.example/base/files/bb?Sig=1","size":2,"sha256hex":""},
{"relPath":"readme.txt","url":"https://d2.cdn.example/base/files/cc?Sig=1","size":0,"sha256hex":""}
]"#;
#[test]
fn parses_plan_and_normalises_paths() {
let entries = parse_plan(PLAN).unwrap();
assert_eq!(entries.len(), 3);
assert_eq!(entries[0].rel_path, "Binaries/Win64/Game.exe");
assert_eq!(entries[0].size, 4);
assert_eq!(entries[0].sha256.len(), 32);
assert!(entries[1].sha256.is_empty());
assert_eq!(entries[2].size, 0);
}
#[test]
fn rejects_bad_plan() {
assert!(parse_plan("{}").is_err());
assert!(parse_plan(r#"[{"url":"x","size":1}]"#).is_err());
assert!(parse_plan(r#"[{"relPath":"a","size":1}]"#).is_err());
assert!(parse_plan(r#"[{"relPath":"a","url":"u","size":1,"sha256hex":"zz"}]"#).is_err());
}
#[test]
fn hex_roundtrip() {
assert_eq!(hex_decode("00ff10"), Some(vec![0x00, 0xff, 0x10]));
assert_eq!(hex_decode("00FF"), Some(vec![0x00, 0xff]));
assert_eq!(hex_decode("0"), None);
assert_eq!(hex_decode("0g"), None);
}
#[test]
fn per_host_cap_gives_single_host_the_whole_ceiling() {
assert_eq!(per_host_cap_for(32, 1), 32);
assert_eq!(per_host_cap_for(32, 3), 11);
assert_eq!(per_host_cap_for(8, 1), 8);
assert_eq!(per_host_cap_for(4, 1), 6);
assert_eq!(per_host_cap_for(0, 0), 6);
}
#[test]
fn host_key_strips_path_and_query() {
assert_eq!(
host_key("https://d1.cdn.example/base/files/aa?Sig=1"),
"https://d1.cdn.example"
);
assert_eq!(host_key("http://h:8080/x"), "http://h:8080");
assert_eq!(host_key("nohost"), "https://nohost");
}
#[test]
fn dest_path_never_escapes_install_dir() {
assert_eq!(
dest_path("/inst/", "a/b.txt"),
PathBuf::from("/inst/a/b.txt")
);
assert_eq!(dest_path("/inst", "/abs.txt"), PathBuf::from("/inst//abs.txt"));
assert_eq!(tmp_path("/inst", "a.bin"), PathBuf::from("/inst/a.bin.tmp"));
}
#[test]
fn build_plan_skips_only_size_matched_files() {
let dir = scratch_dir();
let entries = parse_plan(PLAN).unwrap();
// Entry 1 present with the right size → skipped (no hash check even if wrong bytes).
fs::create_dir_all(format!("{dir}/data")).unwrap();
fs::write(format!("{dir}/data/level1.pak"), b"zz").unwrap();
// Entry 0 present with the wrong size → fetched.
fs::create_dir_all(format!("{dir}/Binaries/Win64")).unwrap();
fs::write(format!("{dir}/Binaries/Win64/Game.exe"), b"tes").unwrap();
let plan = build_plan(entries, &dir);
assert_eq!(plan.total_bytes, 6);
assert_eq!(plan.skipped_files, 1);
assert_eq!(plan.skipped_bytes, 2);
assert_eq!(plan.items.len(), 2);
assert_eq!(plan.items[0].id, 0);
assert_eq!(plan.items[0].reserve, 4);
assert!(plan.items[0].range.is_none());
assert_eq!(plan.items[1].id, 2);
assert_eq!(
plan.hosts,
vec!["https://d1.cdn.example".to_string(), "https://d2.cdn.example".to_string()]
);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn zero_size_missing_file_is_fetched_and_present_zero_is_skipped() {
let dir = scratch_dir();
let entry = PlanEntry {
rel_path: "empty.txt".into(),
url: "https://h/x".into(),
size: 0,
sha256: Vec::new(),
};
assert!(!is_present_and_complete(&dir, &entry));
fs::write(format!("{dir}/empty.txt"), b"").unwrap();
assert!(is_present_and_complete(&dir, &entry));
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn whole_body_sink_verifies_writes_and_renames() {
let dir = scratch_dir();
let body = b"test".to_vec();
let entry = PlanEntry {
rel_path: "sub/Game.exe".into(),
url: "https://h/x".into(),
size: 4,
sha256: hex_decode(&sha_hex(&body)).unwrap(),
};
let cancel = Arc::new(AtomicBool::new(false));
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::clone(&cancel));
let item = item_for(&entry, 0);
// Mismatch → Retry, nothing on disk.
match sink.process(&item, b"nope".to_vec()) {
Err(SinkError::Retry(msg)) => assert!(msg.contains("SHA-256 mismatch")),
Err(SinkError::Fatal(msg)) => panic!("expected Retry, got Fatal({msg})"),
Ok(_) => panic!("expected Retry, got Ok"),
}
assert!(!dest_path(&dir, "sub/Game.exe").exists());
assert!(!tmp_path(&dir, "sub/Game.exe").exists());
// Match → written, renamed, tmp gone, credited body length.
assert_eq!(sink.process(&item, body.clone()).unwrap(), 4);
assert_eq!(fs::read(dest_path(&dir, "sub/Game.exe")).unwrap(), body);
assert!(!tmp_path(&dir, "sub/Game.exe").exists());
assert_eq!(sink.files_done(), 1);
// Existing dest is replaced (Java `:280`).
assert_eq!(sink.process(&item, body.clone()).unwrap(), 4);
// Cancel → Fatal, no write.
cancel.store(true, Ordering::Relaxed);
assert!(matches!(
sink.process(&item, body),
Err(SinkError::Fatal(_))
));
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_appends_verifies_and_renames() {
let dir = scratch_dir();
let body = b"hello world".to_vec();
let entry = PlanEntry {
rel_path: "sub/big.bin".into(),
url: "https://h/x".into(),
size: body.len() as u64,
sha256: hex_decode(&sha_hex(&body)).unwrap(),
};
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::new(AtomicBool::new(false)));
let item = item_for(&entry, 0);
sink.on_chunk(&item, 0, b"hello").unwrap();
assert!(tmp_path(&dir, "sub/big.bin").exists());
sink.on_chunk(&item, 5, b" wor").unwrap();
sink.on_chunk(&item, 9, b"ld").unwrap();
assert_eq!(sink.on_finish(&item, 11).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "sub/big.bin")).unwrap(), body);
assert!(!tmp_path(&dir, "sub/big.bin").exists());
assert_eq!(sink.files_done(), 1);
assert_eq!(sink.cleanup_partials(), 0);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_restart_truncates_and_mismatch_retries() {
let dir = scratch_dir();
let body = b"abcdef".to_vec();
let entry = PlanEntry {
rel_path: "f.bin".into(),
url: "https://h/x".into(),
size: 6,
sha256: hex_decode(&sha_hex(&body)).unwrap(),
};
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::new(AtomicBool::new(false)));
let item = item_for(&entry, 0);
// Attempt 1 delivers garbage, then the core restarts at offset 0.
sink.on_chunk(&item, 0, b"zzzz").unwrap();
sink.on_chunk(&item, 0, b"abc").unwrap();
sink.on_chunk(&item, 3, b"xyz").unwrap();
match sink.on_finish(&item, 6) {
Err(SinkError::Retry(msg)) => assert!(msg.contains("SHA-256 mismatch")),
Err(SinkError::Fatal(msg)) => panic!("expected Retry, got Fatal({msg})"),
Ok(_) => panic!("expected Retry, got Ok"),
}
assert!(!tmp_path(&dir, "f.bin").exists());
assert!(!dest_path(&dir, "f.bin").exists());
// Out-of-order piece → Retry.
sink.on_chunk(&item, 0, b"abc").unwrap();
assert!(matches!(
sink.on_chunk(&item, 5, b"f"),
Err(SinkError::Retry(_))
));
// Length mismatch at finish → Retry.
sink.on_chunk(&item, 0, b"abc").unwrap();
assert!(matches!(sink.on_finish(&item, 6), Err(SinkError::Retry(_))));
// Piece without a start → Retry.
assert!(matches!(
sink.on_chunk(&item, 3, b"def"),
Err(SinkError::Retry(_))
));
// Good attempt.
sink.on_chunk(&item, 0, b"abc").unwrap();
sink.on_chunk(&item, 3, b"def").unwrap();
assert_eq!(sink.on_finish(&item, 6).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "f.bin")).unwrap(), body);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_cleanup_removes_unfinished_tmp_and_zero_len_finishes() {
let dir = scratch_dir();
let big = PlanEntry {
rel_path: "cut.bin".into(),
url: "https://h/x".into(),
size: 10,
sha256: Vec::new(),
};
let empty = PlanEntry {
rel_path: "empty.bin".into(),
url: "https://h/y".into(),
size: 0,
sha256: Vec::new(),
};
let cancel = Arc::new(AtomicBool::new(false));
let sink = AmazonSink::new(&dir, vec![big.clone(), empty.clone()], Arc::clone(&cancel));
sink.on_chunk(&item_for(&big, 0), 0, b"12345").unwrap();
assert!(tmp_path(&dir, "cut.bin").exists());
// Zero-byte body: on_finish with no pieces creates and commits an empty file.
assert_eq!(sink.on_finish(&item_for(&empty, 1), 0).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "empty.bin")).unwrap(), b"");
// Cancel mid-stream → Fatal; cleanup deletes the partial tmp like Java does.
cancel.store(true, Ordering::Relaxed);
assert!(matches!(
sink.on_chunk(&item_for(&big, 0), 5, b"6"),
Err(SinkError::Fatal(_))
));
assert_eq!(sink.cleanup_partials(), 1);
assert!(!tmp_path(&dir, "cut.bin").exists());
assert!(!dest_path(&dir, "cut.bin").exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stream_sink_skips_hash_when_manifest_has_none() {
let dir = scratch_dir();
let entry = PlanEntry {
rel_path: "a.bin".into(),
url: "https://h/x".into(),
size: 3,
sha256: Vec::new(),
};
let sink = AmazonSink::new(&dir, vec![entry.clone()], Arc::new(AtomicBool::new(false)));
let item = item_for(&entry, 0);
sink.on_chunk(&item, 0, b"abc").unwrap();
assert_eq!(sink.on_finish(&item, 3).unwrap(), 0);
assert_eq!(fs::read(dest_path(&dir, "a.bin")).unwrap(), b"abc");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn run_download_all_present_short_circuits() {
let dir = scratch_dir();
fs::write(format!("{dir}/a.bin"), b"abc").unwrap();
let plan = r#"[{"relPath":"a.bin","url":"https://h/x","size":3,"sha256hex":""}]"#;
let seen = Mutex::new(Vec::new());
let logs = Mutex::new(Vec::new());
let result = run_download(
plan,
&dir,
"",
0,
1,
Arc::new(AtomicBool::new(false)),
&|done, total, fd, ft| seen.lock().unwrap().push((done, total, fd, ft)),
&|line| logs.lock().unwrap().push(line.to_string()),
);
assert!(result.success);
assert_eq!(result.files_done, 1);
assert_eq!(result.files_total, 1);
assert_eq!(seen.lock().unwrap().as_slice(), &[(3, 3, 1, 1)]);
assert!(logs.lock().unwrap()[0].starts_with("engine=rust mode=stream"));
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn run_download_reports_plan_error() {
let result = run_download(
"not json",
"/nonexistent",
"",
8,
2,
Arc::new(AtomicBool::new(false)),
&|_, _, _, _| {},
&|_| {},
);
assert!(!result.success);
assert!(result.error.contains("plan json"));
}
}
+228
View File
@@ -0,0 +1,228 @@
//! JNI facade for the Amazon download adapter.
//!
//! Kotlin side: `com.droiddeck.launcher.stores.amazon.AmazonNative` (`nativeStart` /
//! `nativeCancel` / `nativeRelease`) with an `AmazonNative.Listener`
//! (`onProgress(JJJJ)V`, `onLog(String)V`, `onComplete(ZString;J)V`). Mirrors the shape of
//! the GOG and Epic facades: the run happens on a spawned native
//! thread, callbacks attach the JavaVM as daemon threads and hold a `GlobalRef` to the
//! listener; cancel flips an `AtomicBool`. This module keeps its own `JavaVM` handle (taken
//! from the calling `JNIEnv` on first use) so it depends on nothing outside this module.
use crate::amazon::{run_download, MAX_PARALLEL};
use jni::objects::{GlobalRef, JClass, JObject, JString, JValue};
use jni::sys::{jint, jlong, JNI_FALSE, JNI_TRUE};
use jni::{JNIEnv, JavaVM};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
use std::thread;
/// Logcat tag (`adb logcat -s AmazonNative`). Every engine line reaches logcat exactly ONCE:
/// this module only forwards lines to the listener (`onLog`), and the Java manager is the one
/// place that writes them to `android.util.Log` + its debug file. (Round 1 fix: the native side
/// used to also call `__android_log_write`, so each line appeared twice.)
pub const LOG_TAG: &str = "AmazonNative";
static JVM: OnceLock<JavaVM> = OnceLock::new();
fn clear_pending_exception(env: &mut JNIEnv) {
if env.exception_check().unwrap_or(false) {
let _ = env.exception_clear();
}
}
fn jstring_to_string(env: &mut JNIEnv, value: &JString) -> Option<String> {
env.get_string(value)
.ok()
.map(|s| s.to_string_lossy().into_owned())
}
/// One live run; boxed and handed to Kotlin as an opaque `jlong`.
struct AmazonRun {
cancel: Arc<AtomicBool>,
}
fn to_handle(run: Box<AmazonRun>) -> jlong {
Box::into_raw(run) as jlong
}
unsafe fn from_handle<'a>(handle: jlong) -> Option<&'a AmazonRun> {
if handle == 0 {
None
} else {
Some(&*(handle as *const AmazonRun))
}
}
fn call_log(env: &mut JNIEnv, listener: &JObject, line: &str) {
let Ok(text) = env.new_string(line) else {
clear_pending_exception(env);
return;
};
let text_obj = JObject::from(text);
let _ = env.call_method(
listener,
"onLog",
"(Ljava/lang/String;)V",
&[JValue::Object(&text_obj)],
);
clear_pending_exception(env);
}
fn call_progress(
env: &mut JNIEnv,
listener: &JObject,
bytes_done: u64,
bytes_total: u64,
files_done: u64,
files_total: u64,
) {
let _ = env.call_method(
listener,
"onProgress",
"(JJJJ)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
JValue::Long(files_done as jlong),
JValue::Long(files_total as jlong),
],
);
clear_pending_exception(env);
}
fn call_complete(
env: &mut JNIEnv,
listener: &JObject,
success: bool,
error: &str,
bytes_written: u64,
) {
let Ok(error) = env.new_string(error) else {
clear_pending_exception(env);
return;
};
let error_obj = JObject::from(error);
let _ = env.call_method(
listener,
"onComplete",
"(ZLjava/lang/String;J)V",
&[
JValue::Bool(if success { JNI_TRUE } else { JNI_FALSE }),
JValue::Object(&error_obj),
JValue::Long(bytes_written as jlong),
],
);
clear_pending_exception(env);
}
/// Attach the current (native) thread and run `f` with the listener object.
fn with_attached_env(listener: &GlobalRef, f: impl FnOnce(&mut JNIEnv, &JObject)) {
let Some(vm) = JVM.get() else {
return;
};
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
f(&mut env, listener.as_obj());
}
/// `AmazonNative.nativeStart(planJson, installDir, caBundlePath, maxWorkers,
/// processWorkers, listener) -> handle`. Returns 0 (and fires `onComplete(false, …)`) when the
/// inputs are unusable. The run itself happens on a spawned thread.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeStart(
mut env: JNIEnv,
_class: JClass,
plan_json: JString,
install_dir: JString,
ca_bundle_path: JString,
max_workers: jint,
process_workers: jint,
listener: JObject,
) -> jlong {
if listener.is_null() {
return 0;
}
if JVM.get().is_none() {
if let Ok(vm) = env.get_java_vm() {
let _ = JVM.set(vm);
}
}
let plan_json = jstring_to_string(&mut env, &plan_json).unwrap_or_default();
let install_dir = jstring_to_string(&mut env, &install_dir).unwrap_or_default();
let ca_bundle_path = jstring_to_string(&mut env, &ca_bundle_path).unwrap_or_default();
if plan_json.is_empty() || install_dir.is_empty() {
call_complete(&mut env, &listener, false, "empty plan or install dir", 0);
return 0;
}
let Ok(listener) = env.new_global_ref(&listener) else {
clear_pending_exception(&mut env);
return 0;
};
let max_workers = if max_workers <= 0 {
MAX_PARALLEL
} else {
max_workers as usize
};
let process_workers = process_workers.max(1) as usize;
let cancel = Arc::new(AtomicBool::new(false));
let run = Box::new(AmazonRun {
cancel: Arc::clone(&cancel),
});
thread::spawn(move || {
let log_listener = listener.clone();
let log = move |line: &str| {
with_attached_env(&log_listener, |env, obj| call_log(env, obj, line));
};
let progress_listener = listener.clone();
let progress = move |bytes_done: u64, bytes_total: u64, files_done: u64, files_total: u64| {
with_attached_env(&progress_listener, |env, obj| {
call_progress(env, obj, bytes_done, bytes_total, files_done, files_total)
});
};
let result = run_download(
&plan_json,
&install_dir,
&ca_bundle_path,
max_workers,
process_workers,
cancel,
&progress,
&log,
);
with_attached_env(&listener, |env, obj| {
call_complete(env, obj, result.success, &result.error, result.bytes_written)
});
});
to_handle(run)
}
/// `AmazonNative.nativeCancel(handle)` — flips the run's cancel flag; the core stops
/// issuing requests, in-flight bodies are dropped, `onComplete(false, "cancelled", …)` follows.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeCancel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if let Some(run) = unsafe { from_handle(handle) } {
run.cancel.store(true, Ordering::Relaxed);
}
}
/// `AmazonNative.nativeRelease(handle)` — frees the handle. Safe at any time: the worker
/// thread owns its own `Arc` of the cancel flag and the listener `GlobalRef`.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_amazon_AmazonNative_nativeRelease(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle != 0 {
unsafe {
drop(Box::from_raw(handle as *mut AmazonRun));
}
}
}
+42
View File
@@ -0,0 +1,42 @@
//! Epic Games Store download adapter for the shared fetch core (`crate::fetch_core`).
//!
//! Scope = the chunk-fetch inner loop of `EpicDownloadManager.java` (the fixed 8-thread pool that
//! fills `<installDir>/.chunks/<GUID>`), nothing more. The Java manager still parses the manifest
//! API JSON, downloads the manifest, selects files (install tags), runs the delta/verify pass,
//! assembles files from the chunk cache and does every post-install step. The adapter re-parses
//! the same manifest bytes, rebuilds the same chunk plan for the pending files Java hands it,
//! skips chunks already in the cache and writes verified chunks with the same `.part` + rename
//! protocol, so the on-disk state is identical whichever engine ran (see
//! Bannerlator's `docs/RUST_EPIC_PARITY.md` for the rule-by-rule table).
//!
//! Submodules:
//! - [`manifest`] — ChunksV4 binary manifest + legacy JSON manifest parsers (1:1 with the Java
//! `parseManifest` / `parseJsonManifest`).
//! - [`plan`] — install-tag selection, unique-chunk plan, chunk paths / CDN URLs.
//! - [`chunk`] — chunk header parse, zlib inflate, SHA-1 verify, cache write.
//! - [`driver`] — plan → `FetchItem`s → `fetch_core::run_fetch` with the chunk-cache sink.
//! - [`jni`] — `Java_com_droiddeck_launcher_stores_epic_EpicNative_native*` exports.
//!
//! The selective-install-tag, delta/resume and chunk verification rules mirrored here were ported
//! into the Java manager from GameNative (`service/epic/EpicDownloadManager.kt`,
//! `manifest/ManifestUtils.kt`), itself derived from Legendary; both are GPL-3.0 and this crate
//! carries the same licence (see `Cargo.toml`).
pub mod chunk;
pub mod driver;
pub mod jni;
pub mod manifest;
pub mod plan;
/// User-Agent the Java manager sends on every chunk request (`EpicDownloadManager.UA`).
pub const USER_AGENT: &str =
"UELauncher/11.0.1-14907503+++Portal+Release-Live Windows/10.0.19041.1.256.64bit";
/// Name of the chunk cache directory under the install dir (`new File(installDir, ".chunks")`).
pub const CHUNK_CACHE_DIR: &str = ".chunks";
/// Fixed pool width of the Java chunk downloader (`Executors.newFixedThreadPool(8)`).
pub const JAVA_POOL_THREADS: usize = 8;
/// logcat tag for every engine line (the Kotlin manager mirrors it into its own debug log).
pub const LOG_TAG: &str = "EpicNative";
+375
View File
@@ -0,0 +1,375 @@
//! One chunk: header parse → (zlib inflate | copy) → SHA-1 → `.part` → rename into the cache.
//!
//! Port of `EpicDownloadManager.downloadChunkStreaming` minus the HTTP part (the fetch core
//! hands us the whole response body). Java's protocol, kept exactly:
//! - 41-byte header: magic(4) headerVersion(4) headerSize(4) compressedSize(4) GUID(16) hash(8)
//! storedAs(1); payload starts at `max(headerSize, 41)` (Java never seeks back);
//! - payload = the next `compressedSize` bytes, or whatever the body still has (a short body
//! is NOT an error — the SHA-1 check decides);
//! - `storedAs & 1` → zlib; corrupt zlib = failure; a stream that ends early = partial output;
//! - the decompressed bytes go to `<final>.part`; the SHA-1 (when the manifest has one) must
//! match or the `.part` is deleted; then an atomic rename publishes `<final>`.
//! A cache file therefore exists under its final name ONLY if it fully downloaded and verified,
//! on either engine.
use std::fs::{self, File};
use std::io::{self, BufWriter, Write};
use std::path::{Path, PathBuf};
use sha1::{Digest, Sha1};
/// Chunk file magic (`0xB1FE3AA2`).
pub const CHUNK_MAGIC: u32 = 0xB1FE_3AA2;
/// Bytes Java reads before looking at the header (`byte[] hdrBuf = new byte[41]`).
pub const CHUNK_HEADER_PREFIX: usize = 41;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ChunkHeader {
pub header_version: i32,
pub header_size: i32,
pub compressed_size: i32,
pub stored_as: u8,
}
impl ChunkHeader {
pub fn is_compressed(&self) -> bool {
(self.stored_as & 1) != 0
}
/// Where the payload begins in the body (`if (headerSize > 41) skipFully(headerSize - 41)`).
pub fn payload_start(&self) -> usize {
if self.header_size > CHUNK_HEADER_PREFIX as i32 {
self.header_size as usize
} else {
CHUNK_HEADER_PREFIX
}
}
}
/// `readFully(hdrBuf)` + header decode. `Err` = the Java `continue`/exception paths
/// (short stream, bad magic).
pub fn parse_chunk_header(body: &[u8]) -> Result<ChunkHeader, String> {
if body.len() < CHUNK_HEADER_PREFIX {
return Err(format!(
"Stream ended after {}/{} bytes",
body.len(),
CHUNK_HEADER_PREFIX
));
}
let le = |o: usize| i32::from_le_bytes([body[o], body[o + 1], body[o + 2], body[o + 3]]);
let magic = le(0) as u32;
if magic != CHUNK_MAGIC {
return Err(format!("Bad chunk magic (streaming): 0x{magic:x}"));
}
Ok(ChunkHeader {
header_version: le(4),
header_size: le(8),
compressed_size: le(12),
stored_as: body[40],
})
}
/// The payload slice Java's read loop would consume: `compressedSize` bytes after the header,
/// truncated to what the body holds (`if (n <= 0) break;`). `Err` when the header skip runs
/// past the body (`skipFully` throws).
pub fn chunk_payload<'a>(body: &'a [u8], hdr: &ChunkHeader) -> Result<&'a [u8], String> {
let start = hdr.payload_start();
if start > body.len() {
return Err("Stream ended during skip".to_string());
}
let want = if hdr.compressed_size <= 0 {
0
} else {
hdr.compressed_size as usize
};
let avail = body.len() - start;
let take = want.min(avail);
Ok(&body[start..start + take])
}
/// `Writer` that mirrors `fos.write(...)` + `sha.update(...)` on every block.
struct HashingWriter<W: Write> {
inner: W,
sha: Option<Sha1>,
written: u64,
}
impl<W: Write> Write for HashingWriter<W> {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.inner.write(buf)?;
if let Some(sha) = self.sha.as_mut() {
sha.update(&buf[..n]);
}
self.written += n as u64;
Ok(n)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
/// `new File(outFile.getPath() + ".part")`.
pub fn part_path(final_path: &Path) -> PathBuf {
let mut s = final_path.as_os_str().to_os_string();
s.push(".part");
PathBuf::from(s)
}
/// Process one downloaded body into `final_path`. Returns the number of DECOMPRESSED bytes
/// written. `Err(reason)` = Java's "try the next CDN" outcome; the `.part` never survives an
/// error and `final_path` is only ever created by the rename after verification.
pub fn write_verified_chunk(
body: &[u8],
expected_sha1: Option<&[u8; 20]>,
final_path: &Path,
) -> Result<u64, String> {
let tmp = part_path(final_path);
// `tmp.delete()` — clear any stale partial from a prior interrupted attempt.
let _ = fs::remove_file(&tmp);
let result = write_part(body, expected_sha1, &tmp, final_path);
if result.is_err() {
let _ = fs::remove_file(&tmp);
}
result
}
fn write_part(
body: &[u8],
expected_sha1: Option<&[u8; 20]>,
tmp: &Path,
final_path: &Path,
) -> Result<u64, String> {
let hdr = parse_chunk_header(body)?;
let payload = chunk_payload(body, &hdr)?;
let file = File::create(tmp).map_err(|e| format!("open {}: {e}", tmp.display()))?;
let mut out = HashingWriter {
inner: BufWriter::with_capacity(128 * 1024, file),
sha: expected_sha1.map(|_| Sha1::new()),
written: 0,
};
if hdr.is_compressed() {
// Java: inflate until the stream finishes or the input runs out; a corrupt stream throws
// (DataFormatException) → this CDN attempt fails. flate2 behaves the same way: EOF on a
// truncated stream is a clean end (partial output), corruption is an error.
let mut dec = flate2::read::ZlibDecoder::new(payload);
io::copy(&mut dec, &mut out).map_err(|e| format!("inflate: {e}"))?;
} else {
out.write_all(payload).map_err(|e| format!("write: {e}"))?;
}
out.flush().map_err(|e| format!("flush: {e}"))?;
let written = out.written;
let digest = out.sha.take().map(|s| s.finalize());
drop(out); // close the file before the rename
if let (Some(expected), Some(actual)) = (expected_sha1, digest) {
if actual.as_slice() != &expected[..] {
return Err("Chunk SHA-1 mismatch (streaming)".to_string());
}
}
// Verified → publish atomically (same directory → same filesystem).
fs::rename(tmp, final_path).map_err(|e| format!("Chunk rename failed: {e}"))?;
Ok(written)
}
#[cfg(test)]
pub(crate) mod test_support {
use super::*;
/// Build a chunk body the way the CDN serves it.
pub fn build_chunk_body(
decompressed: &[u8],
compress: bool,
extra_header: usize,
declared_compressed_size: Option<i32>,
) -> Vec<u8> {
let payload = if compress {
let mut enc =
flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::default());
enc.write_all(decompressed).unwrap();
enc.finish().unwrap()
} else {
decompressed.to_vec()
};
let header_size = (CHUNK_HEADER_PREFIX + extra_header) as i32;
let mut out = Vec::new();
out.extend_from_slice(&CHUNK_MAGIC.to_le_bytes());
out.extend_from_slice(&3i32.to_le_bytes()); // headerVersion
out.extend_from_slice(&header_size.to_le_bytes());
out.extend_from_slice(
&declared_compressed_size
.unwrap_or(payload.len() as i32)
.to_le_bytes(),
);
out.extend_from_slice(&[0xAB; 16]); // GUID (unused by the reader)
out.extend_from_slice(&[0xCD; 8]); // hash (unused by the reader)
out.push(if compress { 1 } else { 0 });
out.extend(std::iter::repeat(0x5A).take(extra_header));
assert_eq!(out.len(), header_size as usize);
out.extend_from_slice(&payload);
out
}
pub fn sha1_of(data: &[u8]) -> [u8; 20] {
let mut h = Sha1::new();
h.update(data);
let d = h.finalize();
let mut out = [0u8; 20];
out.copy_from_slice(&d);
out
}
pub fn temp_dir(tag: &str) -> PathBuf {
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
let dir = std::env::temp_dir().join(format!(
"epic-{tag}-{}-{nanos}",
std::process::id()
));
fs::create_dir_all(&dir).unwrap();
dir
}
}
#[cfg(test)]
mod tests {
use super::test_support::*;
use super::*;
fn sample_data() -> Vec<u8> {
(0..200_000u32).map(|i| (i % 251) as u8).collect()
}
#[test]
fn header_parse_matches_java_fields() {
let body = build_chunk_body(b"hello", true, 5, None);
let h = parse_chunk_header(&body).unwrap();
assert_eq!(h.header_version, 3);
assert_eq!(h.header_size, 46);
assert!(h.is_compressed());
assert_eq!(h.payload_start(), 46);
let short = ChunkHeader {
header_version: 0,
header_size: 20,
compressed_size: 1,
stored_as: 0,
};
assert_eq!(short.payload_start(), 41, "headerSize < 41 → payload still starts at 41");
}
#[test]
fn short_or_wrong_magic_bodies_are_rejected() {
assert!(parse_chunk_header(&[0u8; 40]).is_err());
let mut body = build_chunk_body(b"x", false, 0, None);
body[0] ^= 0xFF;
assert!(parse_chunk_header(&body).unwrap_err().contains("Bad chunk magic"));
}
#[test]
fn payload_is_truncated_to_the_body_not_an_error() {
let body = build_chunk_body(b"abcdef", false, 0, Some(100));
let h = parse_chunk_header(&body).unwrap();
assert_eq!(chunk_payload(&body, &h).unwrap(), b"abcdef");
let neg = build_chunk_body(b"abcdef", false, 0, Some(-1));
let h = parse_chunk_header(&neg).unwrap();
assert_eq!(chunk_payload(&neg, &h).unwrap(), b"");
// headerSize past the body → skipFully throws.
let mut far = build_chunk_body(b"abcdef", false, 0, None);
far[8..12].copy_from_slice(&5000i32.to_le_bytes());
let h = parse_chunk_header(&far).unwrap();
assert!(chunk_payload(&far, &h).is_err());
}
#[test]
fn compressed_chunk_is_inflated_verified_and_renamed() {
let dir = temp_dir("ok");
let data = sample_data();
let body = build_chunk_body(&data, true, 3, None);
let final_path = dir.join("00000001000000020000000300000004");
let n = write_verified_chunk(&body, Some(&sha1_of(&data)), &final_path).unwrap();
assert_eq!(n, data.len() as u64);
assert_eq!(fs::read(&final_path).unwrap(), data);
assert!(!part_path(&final_path).exists(), ".part renamed away");
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stored_chunk_is_copied_as_is() {
let dir = temp_dir("stored");
let data = b"stored-as-is payload".to_vec();
let body = build_chunk_body(&data, false, 0, None);
let final_path = dir.join("chunk");
let n = write_verified_chunk(&body, Some(&sha1_of(&data)), &final_path).unwrap();
assert_eq!(n, data.len() as u64);
assert_eq!(fs::read(&final_path).unwrap(), data);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn sha1_mismatch_leaves_nothing_behind() {
let dir = temp_dir("sha");
let data = sample_data();
let body = build_chunk_body(&data, true, 0, None);
let final_path = dir.join("chunk");
let err = write_verified_chunk(&body, Some(&[0x42; 20]), &final_path).unwrap_err();
assert!(err.contains("SHA-1 mismatch"), "{err}");
assert!(!final_path.exists());
assert!(!part_path(&final_path).exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn unverifiable_chunk_is_accepted_without_a_hash() {
let dir = temp_dir("nosha");
let data = sample_data();
let body = build_chunk_body(&data, true, 0, None);
let final_path = dir.join("chunk");
assert_eq!(
write_verified_chunk(&body, None, &final_path).unwrap(),
data.len() as u64
);
assert_eq!(fs::read(&final_path).unwrap(), data);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn corrupt_zlib_fails_and_truncated_zlib_fails_the_hash() {
let dir = temp_dir("zlib");
let data = sample_data();
let mut body = build_chunk_body(&data, true, 0, None);
let final_path = dir.join("chunk");
// Truncate the body: inflate ends early → partial output → SHA-1 mismatch.
let cut = body.len() - 1000;
let err = write_verified_chunk(&body[..cut], Some(&sha1_of(&data)), &final_path).unwrap_err();
assert!(err.contains("SHA-1 mismatch") || err.contains("inflate"), "{err}");
assert!(!final_path.exists());
// Corrupt the deflate stream itself.
for b in body.iter_mut().skip(60).take(64) {
*b ^= 0x55;
}
let err = write_verified_chunk(&body, Some(&sha1_of(&data)), &final_path).unwrap_err();
assert!(err.contains("SHA-1 mismatch") || err.contains("inflate"), "{err}");
assert!(!final_path.exists());
assert!(!part_path(&final_path).exists());
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn stale_part_file_is_cleared_before_the_attempt() {
let dir = temp_dir("stale");
let final_path = dir.join("chunk");
fs::write(part_path(&final_path), b"stale").unwrap();
let body = [0u8; 10]; // too short → header failure
assert!(write_verified_chunk(&body, None, &final_path).is_err());
assert!(!part_path(&final_path).exists());
assert!(!final_path.exists());
let _ = fs::remove_dir_all(&dir);
}
}
+455
View File
@@ -0,0 +1,455 @@
//! Plan → `FetchItem`s → `crate::fetch_core::run_fetch` with a sink that fills the chunk cache.
//!
//! This is the replacement for the body of the Java pool block in `EpicDownloadManager.install`
//! ("Download unique chunks — 8 parallel threads"). Inputs are exactly what that block sees:
//! the parsed manifest (re-parsed here from the same bytes), the pending file set (indices Java
//! computed after its delta/verify pass), the CDN prefixes (`baseUrl + cloudDir`, cloudflare
//! already skipped) and the chunk cache dir. Output = `<installDir>/.chunks/<GUID>` for every
//! needed chunk; Java assembles the files afterwards exactly as before.
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use crate::fetch_core::{run_fetch, FetchItem, FetchOptions, FetchSink, SinkError};
use super::manifest::{parse_manifest, Manifest};
use super::plan::{
cached_chunk_path, chunk_cache_dir, chunk_url, distinct_prefixes, per_host_cap,
total_credit_bytes, unique_chunks_for_files,
};
/// Java `conn.setReadTimeout(60000)` — the longer of the two Java timeouts (connect was 30 s).
pub const REQUEST_TIMEOUT: Duration = Duration::from_secs(60);
/// Everything the JNI layer hands over for one run.
pub struct EpicRequest {
pub manifest_bytes: Vec<u8>,
pub install_dir: String,
/// `cdn.baseUrl + cdn.cloudDir` per manifest-API CDN entry, in Java's order.
pub cdn_prefixes: Vec<String>,
/// Indices into `manifest.files` of Java's `pendingFiles` (post delta/verify), in order.
pub pending_file_indices: Vec<usize>,
/// Java's `neededChunks.size()` / `totalBytes` for the plan cross-check (`None` = skip).
pub expected_chunks: Option<u64>,
pub expected_bytes: Option<u64>,
pub ca_bundle_path: String,
/// Window ceiling: the Steam speed tier's network window (Fast = 32) since improvements
/// round 1; the Java fallback pool keeps its fixed 8 (`super::JAVA_POOL_THREADS`).
pub max_workers: usize,
pub process_workers: usize,
pub label: String,
}
/// The chunk plan for one run.
#[derive(Debug)]
pub struct EpicPlan {
pub manifest: Manifest,
pub cache_dir: PathBuf,
/// Chunk indices (into `manifest.unique_chunks`) in Java's submission order.
pub needed: Vec<usize>,
/// `Σ max(fileSize, 1)` over `needed`.
pub total_bytes: u64,
/// Distinct CDN prefixes = fetch-core host keys.
pub hosts: Vec<String>,
}
/// Terminal result of a run, in the shape Java's pool block needs to reproduce its own exit
/// paths (`CANCELLED during chunk download (n/total chunks)`, `N chunks failed`, `chunksOK=`).
#[derive(Clone, Debug, Default)]
pub struct EpicOutcome {
pub success: bool,
pub cancelled: bool,
pub error: String,
/// Credited (manifest `max(fileSize,1)`) bytes: skipped-cached + fetched.
pub bytes_credited: u64,
/// Chunks accounted for (skipped-cached + fetched), Java's `completedCount`.
pub chunks_done: u64,
pub chunks_total: u64,
pub bytes_total: u64,
/// Decompressed bytes actually written to the cache this run.
pub decompressed_written: u64,
}
/// Parse + plan (no I/O beyond reading the cache dir). `Err` = "engine could not start"; Java
/// then runs its own pool, since nothing has been fetched yet.
pub fn build_plan(req: &EpicRequest) -> Result<EpicPlan, String> {
let manifest = parse_manifest(&req.manifest_bytes).map_err(|e| format!("plan: {e}"))?;
for &i in &req.pending_file_indices {
if i >= manifest.files.len() {
return Err(format!(
"plan: pending file index {i} out of range ({} files)",
manifest.files.len()
));
}
}
let needed = unique_chunks_for_files(&manifest, &req.pending_file_indices);
let total_bytes = total_credit_bytes(&manifest, &needed);
if let Some(expected) = req.expected_chunks {
if expected != needed.len() as u64 {
return Err(format!(
"plan: chunk count mismatch java={expected} rust={}",
needed.len()
));
}
}
if let Some(expected) = req.expected_bytes {
if expected != total_bytes {
return Err(format!(
"plan: byte total mismatch java={expected} rust={total_bytes}"
));
}
}
let hosts = distinct_prefixes(&req.cdn_prefixes);
if hosts.is_empty() {
return Err("plan: no CDN prefixes".to_string());
}
let cache_dir = chunk_cache_dir(&req.install_dir);
std::fs::create_dir_all(&cache_dir)
.map_err(|e| format!("plan: mkdirs {}: {e}", cache_dir.display()))?;
Ok(EpicPlan {
manifest,
cache_dir,
needed,
total_bytes,
hosts,
})
}
/// Sink: one downloaded body → verified cache file. `id` indexes `fetch_chunks`.
struct ChunkCacheSink<'a> {
plan: &'a EpicPlan,
/// Chunk indices (into `manifest.unique_chunks`) of the items actually being fetched.
fetch_chunks: &'a [usize],
decompressed: AtomicU64,
}
impl<'a> FetchSink for ChunkCacheSink<'a> {
fn process(&self, item: &FetchItem, body: Vec<u8>) -> Result<u64, SinkError> {
let Some(&ci) = self.fetch_chunks.get(item.id as usize) else {
return Err(SinkError::Fatal(format!("item id {} out of range", item.id)));
};
let chunk = &self.plan.manifest.unique_chunks[ci];
let final_path = cached_chunk_path(&self.plan.cache_dir, chunk);
match super::chunk::write_verified_chunk(&body, chunk.verifiable_sha1(), &final_path) {
Ok(written) => {
self.decompressed.fetch_add(written, Ordering::Relaxed);
Ok(chunk.credit_bytes())
}
// Every per-attempt failure in Java is "try the next CDN"; the core's Retry rotates
// hosts and backs off the same way (bounded at its attempt cap).
Err(reason) => Err(SinkError::Retry(format!("{} {reason}", chunk.guid_str()))),
}
}
}
/// Rolling speed sampler for the end-of-run summary: samples at ≥500 ms like the Java pool's
/// `lastSpeedMs` logic, keeps the peak.
struct SpeedMeter {
start: Instant,
last_at: Instant,
last_bytes: u64,
peak_bps: f64,
}
impl SpeedMeter {
fn new() -> Self {
let now = Instant::now();
Self {
start: now,
last_at: now,
last_bytes: 0,
peak_bps: 0.0,
}
}
fn sample(&mut self, bytes_now: u64) {
let now = Instant::now();
let dt = now.duration_since(self.last_at);
if dt >= Duration::from_millis(500) {
let delta = bytes_now.saturating_sub(self.last_bytes) as f64;
let bps = delta / dt.as_secs_f64();
if bps > self.peak_bps {
self.peak_bps = bps;
}
self.last_at = now;
self.last_bytes = bytes_now;
}
}
fn summary(&self, bytes: u64, decompressed: u64, skipped: u64) -> String {
let secs = self.start.elapsed().as_secs_f64().max(0.001);
let avg_bps = bytes as f64 / secs;
format!(
"summary bytes={bytes} decompressed={decompressed} skipped_chunks={skipped} elapsed={secs:.1}s avg_mbps={:.2} peak_mbps={:.2} avg_MBps={:.2}",
avg_bps * 8.0 / 1_000_000.0,
self.peak_bps * 8.0 / 1_000_000.0,
avg_bps / (1024.0 * 1024.0)
)
}
}
/// Run the plan. `progress(bytes_done, chunks_done)` fires once per accounted chunk (cached-skip
/// or fetched), exactly the cadence of the Java pool's per-task `progress(...)` call. `log`
/// receives engine lines (`fetch-window`, `summary`, failures).
pub fn run_plan(
plan: &EpicPlan,
req: &EpicRequest,
cancel: &AtomicBool,
progress: &(dyn Fn(u64, u64) + Sync),
log: &(dyn Fn(&str) + Sync),
) -> EpicOutcome {
let chunks_total = plan.needed.len() as u64;
let mut outcome = EpicOutcome {
chunks_total,
bytes_total: plan.total_bytes,
..EpicOutcome::default()
};
let host_cap = per_host_cap(req.max_workers, plan.hosts.len());
log(&format!(
"plan chunk_dir={} files_pending={} chunks={} bytes={} hosts={} workers={} per_host_cap={host_cap} process_workers={}",
plan.manifest.chunk_dir,
req.pending_file_indices.len(),
chunks_total,
plan.total_bytes,
plan.hosts.len(),
req.max_workers,
req.process_workers
));
// Java: each pool task first checks the cancel flag, then `cachedFile.exists()` → credit
// without fetching. Account for the cached chunks up front (same credit, same callback).
let mut fetch_chunks: Vec<usize> = Vec::with_capacity(plan.needed.len());
let mut pre_bytes: u64 = 0;
let mut pre_chunks: u64 = 0;
for &ci in &plan.needed {
if cancel.load(Ordering::Relaxed) {
outcome.cancelled = true;
outcome.error = "cancelled".to_string();
outcome.bytes_credited = pre_bytes;
outcome.chunks_done = pre_chunks;
log(&format!(
"cancelled before fetch ({pre_chunks}/{chunks_total} chunks)"
));
return outcome;
}
let chunk = &plan.manifest.unique_chunks[ci];
if cached_chunk_path(&plan.cache_dir, chunk).exists() {
pre_bytes += chunk.credit_bytes();
pre_chunks += 1;
progress(pre_bytes, pre_chunks);
} else {
fetch_chunks.push(ci);
}
}
for (i, h) in plan.hosts.iter().enumerate() {
log(&format!("host[{i}]={h}"));
}
log(&format!(
"skip cached={pre_chunks} bytes={pre_bytes} to_fetch={}",
fetch_chunks.len()
));
if fetch_chunks.is_empty() {
outcome.success = true;
outcome.bytes_credited = pre_bytes;
outcome.chunks_done = pre_chunks;
log("summary bytes=0 decompressed=0 elapsed=0.0s avg_mbps=0.00 peak_mbps=0.00 (nothing to fetch)");
return outcome;
}
let items: Vec<FetchItem> = fetch_chunks
.iter()
.enumerate()
.map(|(id, &ci)| {
let chunk = &plan.manifest.unique_chunks[ci];
FetchItem {
id: id as u64,
urls: plan
.hosts
.iter()
.map(|h| chunk_url(h, &plan.manifest.chunk_dir, chunk))
.collect(),
reserve: if chunk.file_size > 0 {
chunk.file_size as u64
} else {
0
},
range: None,
}
})
.collect();
let opts = FetchOptions {
max_workers: req.max_workers.max(1),
// Improvements round 1: the tier ceiling split across the distinct CDNs (floor 6), so
// the whole window is reachable on 1 host or on all 3.
per_host_cap: host_cap,
timeout: REQUEST_TIMEOUT,
headers: vec![("User-Agent".to_string(), super::USER_AGENT.to_string())],
ca_bundle_path: req.ca_bundle_path.clone(),
process_workers: req.process_workers.max(1),
label: req.label.clone(),
// Whole-body mode (Epic chunks are ≤ ~1 MiB compressed; the core's byte budget scales
// with the window); `stream` and any future field keep the core's defaults.
..FetchOptions::default()
};
let sink = ChunkCacheSink {
plan,
fetch_chunks: &fetch_chunks,
decompressed: AtomicU64::new(0),
};
let meter = Mutex::new(SpeedMeter::new());
let progress_adapter = |bytes: u64, items_ok: u64| {
if let Ok(mut m) = meter.lock() {
m.sample(bytes);
}
progress(pre_bytes + bytes, pre_chunks + items_ok);
};
let fetched = run_fetch(
items,
&plan.hosts,
&opts,
&sink,
cancel,
&progress_adapter,
log,
);
let decompressed = sink.decompressed.load(Ordering::Relaxed);
outcome.bytes_credited = pre_bytes + fetched.bytes_credited;
outcome.chunks_done = pre_chunks + fetched.items_ok;
outcome.decompressed_written = decompressed;
if let Ok(m) = meter.lock() {
log(&m.summary(fetched.bytes_credited, decompressed, pre_chunks));
}
if fetched.cancelled {
outcome.cancelled = true;
outcome.error = "cancelled".to_string();
log(&format!(
"cancelled during chunk download ({}/{chunks_total} chunks)",
outcome.chunks_done
));
return outcome;
}
if let Some(err) = fetched.error {
outcome.error = err;
log(&format!(
"FAIL {} ({}/{chunks_total} chunks ok)",
outcome.error, outcome.chunks_done
));
return outcome;
}
if fetched.items_ok != fetch_chunks.len() as u64 {
outcome.error = format!(
"fetch ended with {}/{} chunks",
fetched.items_ok,
fetch_chunks.len()
);
log(&format!("FAIL {}", outcome.error));
return outcome;
}
outcome.success = true;
log(&format!("chunksOK={}", outcome.chunks_done));
outcome
}
#[cfg(test)]
mod tests {
use super::super::manifest::test_support::*;
use super::*;
fn request(dir: &str, pending: Vec<usize>) -> EpicRequest {
EpicRequest {
manifest_bytes: build_manifest(&sample_chunks(), &sample_files(), 21, true),
install_dir: dir.to_string(),
cdn_prefixes: vec![
"https://fastly-download.epicgames.com/Builds/o/x/default".to_string(),
"https://download.epicgames.com/Builds/o/x/default".to_string(),
"https://fastly-download.epicgames.com/Builds/o/x/default".to_string(),
],
pending_file_indices: pending,
expected_chunks: None,
expected_bytes: None,
ca_bundle_path: String::new(),
max_workers: super::super::JAVA_POOL_THREADS,
process_workers: 2,
label: "epic test".to_string(),
}
}
#[test]
fn plan_matches_java_for_the_pending_set() {
let dir = super::super::chunk::test_support::temp_dir("plan");
let req = request(dir.to_str().unwrap(), vec![0, 1]);
let plan = build_plan(&req).unwrap();
assert_eq!(plan.needed, vec![0, 1]);
assert_eq!(plan.total_bytes, 700_000 + 1);
assert_eq!(plan.hosts.len(), 2, "duplicate CDN prefix collapsed");
assert!(plan.cache_dir.ends_with(".chunks"));
assert!(plan.cache_dir.is_dir());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn plan_cross_check_rejects_mismatches() {
let dir = super::super::chunk::test_support::temp_dir("xcheck");
let mut req = request(dir.to_str().unwrap(), vec![0, 1]);
req.expected_chunks = Some(3);
assert!(build_plan(&req).unwrap_err().contains("chunk count mismatch"));
req.expected_chunks = Some(2);
req.expected_bytes = Some(5);
assert!(build_plan(&req).unwrap_err().contains("byte total mismatch"));
req.expected_bytes = Some(700_001);
assert!(build_plan(&req).is_ok());
req.pending_file_indices = vec![7];
assert!(build_plan(&req).unwrap_err().contains("out of range"));
req.pending_file_indices = vec![0];
req.cdn_prefixes.clear();
assert!(build_plan(&req).unwrap_err().contains("no CDN"));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn fully_cached_plan_completes_without_fetching() {
let dir = super::super::chunk::test_support::temp_dir("cached");
let req = request(dir.to_str().unwrap(), vec![0]);
let plan = build_plan(&req).unwrap();
for &ci in &plan.needed {
std::fs::write(
cached_chunk_path(&plan.cache_dir, &plan.manifest.unique_chunks[ci]),
b"cached",
)
.unwrap();
}
let cancel = AtomicBool::new(false);
let calls = Mutex::new(Vec::new());
let progress = |b: u64, c: u64| calls.lock().unwrap().push((b, c));
let log = |_: &str| {};
let out = run_plan(&plan, &req, &cancel, &progress, &log);
assert!(out.success);
assert_eq!(out.chunks_done, 2);
assert_eq!(out.bytes_credited, 700_001);
assert_eq!(*calls.lock().unwrap(), vec![(700_000, 1), (700_001, 2)]);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn cancel_before_fetch_reports_cancelled() {
let dir = super::super::chunk::test_support::temp_dir("cancel");
let req = request(dir.to_str().unwrap(), vec![0]);
let plan = build_plan(&req).unwrap();
let cancel = AtomicBool::new(true);
let progress = |_: u64, _: u64| {};
let log = |_: &str| {};
let out = run_plan(&plan, &req, &cancel, &progress, &log);
assert!(out.cancelled);
assert!(!out.success);
let _ = std::fs::remove_dir_all(&dir);
}
}
+388
View File
@@ -0,0 +1,388 @@
//! JNI exports for `com.droiddeck.launcher.stores.epic.EpicNative`.
//!
//! ```text
//! nativeStart(manifest: byte[], installDir: String, cdnPrefixes: String[], pendingFileIdx: int[],
//! expectedChunks: int, expectedBytes: long, caBundlePath: String, maxWorkers: int,
//! processWorkers: int, listener: EpicNative.Listener) -> long handle (0 = not started)
//! nativeCancel(handle) — flips the run's AtomicBool; the fetch core stops promptly
//! nativeRelease(handle) — frees the handle (after onComplete)
//! ```
//! Listener (all methods run on native threads):
//! `onPlan(int chunksTotal, long bytesTotal, String chunkDir)` once before any fetch,
//! `onProgress(long bytesDone, long bytesTotal, int chunksDone, int chunksTotal)` per chunk,
//! `onLog(String line)`, `onComplete(boolean success, String error, long bytesCredited)`.
//!
//! Planning (manifest parse + cross-check) runs synchronously on the calling thread so a plan
//! failure returns 0 BEFORE any fetch and Java can fall back to its own pool; the fetch runs on
//! a new thread. JavaVM attach / GlobalRef handling is the same shape as the GOG facade.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
use jni::objects::{GlobalRef, JByteArray, JClass, JIntArray, JObject, JObjectArray, JString, JValue};
use jni::sys::{jint, jlong, JNI_FALSE, JNI_TRUE};
use jni::{JNIEnv, JavaVM};
use super::driver::{build_plan, run_plan, EpicRequest};
#[cfg(target_os = "android")]
#[link(name = "log")]
unsafe extern "C" {
fn __android_log_write(prio: i32, tag: *const i8, text: *const i8) -> i32;
}
fn android_log(message: &str) {
#[cfg(target_os = "android")]
{
use std::ffi::CString;
let Ok(tag) = CString::new(super::LOG_TAG) else {
return;
};
let sanitized = message.replace('\0', " ");
let Ok(text) = CString::new(sanitized) else {
return;
};
unsafe {
let _ = __android_log_write(4, tag.as_ptr().cast(), text.as_ptr().cast());
}
}
#[cfg(not(target_os = "android"))]
{
let _ = message;
}
}
/// Per-run native state behind the `jlong` handle.
pub struct EpicDownloadHandle {
cancel: Arc<AtomicBool>,
}
fn clear_pending_exception(env: &mut JNIEnv) {
if env.exception_check().unwrap_or(false) {
let _ = env.exception_clear();
}
}
fn jstring_to_string(env: &mut JNIEnv, value: &JString) -> Option<String> {
env.get_string(value)
.ok()
.map(|s| s.to_string_lossy().into_owned())
}
fn string_array_to_vec(env: &mut JNIEnv, array: &JObjectArray) -> Vec<String> {
let len = env.get_array_length(array).unwrap_or(0);
let mut out = Vec::with_capacity(len.max(0) as usize);
let mut i = 0;
while i < len {
match env.get_object_array_element(array, i) {
Ok(obj) => {
if obj.is_null() {
out.push(String::new());
} else {
let js = JString::from(obj);
out.push(jstring_to_string(env, &js).unwrap_or_default());
}
}
Err(_) => {
clear_pending_exception(env);
out.push(String::new());
}
}
i += 1;
}
out
}
fn int_array_to_vec(env: &JNIEnv, array: &JIntArray) -> Vec<i32> {
let len = env.get_array_length(array).unwrap_or(0);
if len <= 0 {
return Vec::new();
}
let mut values = vec![0i32; len as usize];
if env.get_int_array_region(array, 0, &mut values).is_err() {
return Vec::new();
}
values
}
fn call_on_log(env: &mut JNIEnv, listener: &JObject, line: &str) {
if listener.is_null() {
return;
}
let Ok(s) = env.new_string(line) else {
clear_pending_exception(env);
return;
};
let obj = JObject::from(s);
let _ = env.call_method(
listener,
"onLog",
"(Ljava/lang/String;)V",
&[JValue::Object(&obj)],
);
clear_pending_exception(env);
}
fn call_on_plan(env: &mut JNIEnv, listener: &JObject, chunks_total: u64, bytes_total: u64, chunk_dir: &str) {
if listener.is_null() {
return;
}
let Ok(s) = env.new_string(chunk_dir) else {
clear_pending_exception(env);
return;
};
let obj = JObject::from(s);
let _ = env.call_method(
listener,
"onPlan",
"(IJLjava/lang/String;)V",
&[
JValue::Int(chunks_total.min(i32::MAX as u64) as jint),
JValue::Long(bytes_total as jlong),
JValue::Object(&obj),
],
);
clear_pending_exception(env);
}
fn call_on_progress(
env: &mut JNIEnv,
listener: &JObject,
bytes_done: u64,
bytes_total: u64,
chunks_done: u64,
chunks_total: u64,
) {
if listener.is_null() {
return;
}
let _ = env.call_method(
listener,
"onProgress",
"(JJII)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
JValue::Int(chunks_done.min(i32::MAX as u64) as jint),
JValue::Int(chunks_total.min(i32::MAX as u64) as jint),
],
);
clear_pending_exception(env);
}
fn call_on_complete(env: &mut JNIEnv, listener: &JObject, success: bool, error: &str, bytes: u64) {
if listener.is_null() {
return;
}
let Ok(s) = env.new_string(error) else {
clear_pending_exception(env);
return;
};
let obj = JObject::from(s);
let _ = env.call_method(
listener,
"onComplete",
"(ZLjava/lang/String;J)V",
&[
JValue::Bool(if success { JNI_TRUE } else { JNI_FALSE }),
JValue::Object(&obj),
JValue::Long(bytes as jlong),
],
);
clear_pending_exception(env);
}
/// Log to logcat AND to the listener (the Kotlin manager mirrors it into its own debug log).
fn log_both(env: &mut JNIEnv, listener: &JObject, line: &str) {
android_log(line);
call_on_log(env, listener, line);
}
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeStart(
mut env: JNIEnv,
_class: JClass,
manifest: JByteArray,
install_dir: JString,
cdn_prefixes: JObjectArray,
pending_file_idx: JIntArray,
expected_chunks: jint,
expected_bytes: jlong,
ca_bundle_path: JString,
max_workers: jint,
process_workers: jint,
listener: JObject,
) -> jlong {
if listener.is_null() {
android_log("nativeStart: null listener");
return 0;
}
let Ok(manifest_bytes) = env.convert_byte_array(manifest) else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "manifest bytes unreadable", 0);
return 0;
};
let install_dir = jstring_to_string(&mut env, &install_dir).unwrap_or_default();
let cdn_prefixes = string_array_to_vec(&mut env, &cdn_prefixes);
let pending_file_indices: Vec<usize> = int_array_to_vec(&env, &pending_file_idx)
.into_iter()
.map(|i| i.max(0) as usize)
.collect();
let ca_bundle_path = jstring_to_string(&mut env, &ca_bundle_path).unwrap_or_default();
let max_workers = max_workers.max(1) as usize;
let process_workers = process_workers.max(1) as usize;
let label = {
let base = std::path::Path::new(&install_dir)
.file_name()
.map(|s| s.to_string_lossy().into_owned())
.unwrap_or_default();
format!("epic app={base}")
};
let host_count = super::plan::distinct_prefixes(&cdn_prefixes).len();
let host_cap = super::plan::per_host_cap(max_workers, host_count);
log_both(
&mut env,
&listener,
&format!(
"engine=rust label=\"{label}\" install_dir={install_dir} cdns={} hosts={host_count} pending_files={} workers={max_workers} per_host_cap={host_cap} process_workers={process_workers} manifest_bytes={}",
cdn_prefixes.len(),
pending_file_indices.len(),
manifest_bytes.len()
),
);
let req = EpicRequest {
manifest_bytes,
install_dir,
cdn_prefixes,
pending_file_indices,
expected_chunks: if expected_chunks >= 0 {
Some(expected_chunks as u64)
} else {
None
},
expected_bytes: if expected_bytes >= 0 {
Some(expected_bytes as u64)
} else {
None
},
ca_bundle_path,
max_workers,
process_workers,
label,
};
let plan = match build_plan(&req) {
Ok(plan) => plan,
Err(err) => {
log_both(&mut env, &listener, &format!("not started: {err}"));
call_on_complete(&mut env, &listener, false, &err, 0);
return 0;
}
};
call_on_plan(
&mut env,
&listener,
plan.needed.len() as u64,
plan.total_bytes,
&plan.manifest.chunk_dir,
);
let Ok(vm) = env.get_java_vm() else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "JavaVM unavailable", 0);
return 0;
};
let Ok(listener) = env.new_global_ref(&listener) else {
clear_pending_exception(&mut env);
call_on_complete(&mut env, &listener, false, "listener ref failed", 0);
return 0;
};
let cancel = Arc::new(AtomicBool::new(false));
let thread_cancel = Arc::clone(&cancel);
let handle = Box::new(EpicDownloadHandle { cancel });
thread::spawn(move || {
run_on_thread(vm, listener, plan, req, thread_cancel);
});
Box::into_raw(handle) as jlong
}
fn run_on_thread(
vm: JavaVM,
listener: GlobalRef,
plan: super::driver::EpicPlan,
req: EpicRequest,
cancel: Arc<AtomicBool>,
) {
let bytes_total = plan.total_bytes;
let chunks_total = plan.needed.len() as u64;
// Progress fires from the core's process-pool threads; attach each as a daemon like the
// Steam engine's `dispatch_download_progress` does.
let progress = |bytes_done: u64, chunks_done: u64| {
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_progress(
&mut env,
listener.as_obj(),
bytes_done,
bytes_total,
chunks_done,
chunks_total,
);
};
let log = |line: &str| {
android_log(line);
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_log(&mut env, listener.as_obj(), line);
};
let outcome = run_plan(&plan, &req, cancel.as_ref(), &progress, &log);
let Ok(mut env) = vm.attach_current_thread_as_daemon() else {
return;
};
call_on_complete(
&mut env,
listener.as_obj(),
outcome.success,
&outcome.error,
outcome.bytes_credited,
);
}
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeCancel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle == 0 {
return;
}
let handle = unsafe { &*(handle as *const EpicDownloadHandle) };
handle.cancel.store(true, Ordering::Relaxed);
android_log("cancel requested");
}
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_epic_EpicNative_nativeRelease(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle == 0 {
return;
}
// The worker thread owns its own Arc clone of the cancel flag and its own GlobalRef, so
// freeing the handle is safe even if a (cancelled) run is still winding down.
drop(unsafe { Box::from_raw(handle as *mut EpicDownloadHandle) });
}
@@ -0,0 +1,987 @@
//! Epic build manifest parsers — a 1:1 port of `EpicDownloadManager.parseManifest` (binary,
//! magic `0x44BEC00C`) and `parseJsonManifest` (legacy JSON manifests).
//!
//! Every quirk of the Java reader is kept on purpose (the Java manager stays the reference and
//! the two engines must derive the same plan from the same bytes): section skips by declared
//! size, `readFString` UTF-16LE/ASCII decoding, duplicate-GUID handling (last value, first
//! position), the JSON `Long.parseLong(.., 16)` / `parseInt` fallbacks, and "any exception →
//! parse failed".
use std::collections::HashMap;
use std::io::Read;
/// Binary manifest magic (`EpicDownloadManager.parseManifest`).
pub const MANIFEST_MAGIC: u32 = 0x44BE_C00C;
/// One entry of the ChunkDataList section (`EpicDownloadManager.ChunkInfo`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ChunkInfo {
/// Four uint32 in binary read order.
pub guid: [u32; 4],
/// uint64 "hash" that forms the first half of the chunk file name.
pub hash: u64,
/// SHA-1 of the DECOMPRESSED chunk; `None` for JSON manifests (Java: `sha1 == null`).
pub sha1: Option<[u8; 20]>,
/// Subfolder number (DECIMAL `%02d`). `i32` so the JSON parser's `Integer.parseInt` range is
/// representable exactly (binary manifests always yield 0..=255).
pub group_num: i32,
/// Uncompressed size (0 for JSON manifests).
pub window_size: i32,
/// Compressed (wire) size. Signed like the Java `long`; JSON manifests may yield anything.
pub file_size: i64,
}
impl ChunkInfo {
/// `String.format("%08X%08X%08X%08X", guid[0..4])`.
pub fn guid_str(&self) -> String {
guid_to_string(&self.guid)
}
/// `ChunkInfo.getPath`: `"<chunkDir>/<%02d groupNum>/<%016X hash>_<GUID>.chunk"`.
pub fn path(&self, chunk_dir: &str) -> String {
format!(
"{}/{:02}/{:016X}_{}.chunk",
chunk_dir,
self.group_num,
self.hash,
self.guid_str()
)
}
/// Java `chunk.sha1 != null && length == 20 && !allZero` — the condition under which the
/// streaming downloader verifies the decompressed chunk.
pub fn verifiable_sha1(&self) -> Option<&[u8; 20]> {
match &self.sha1 {
Some(h) if h.iter().any(|b| *b != 0) => Some(h),
_ => None,
}
}
/// `Math.max(chunk.fileSize, 1)` — the byte credit the Java pool uses for totals/progress.
pub fn credit_bytes(&self) -> u64 {
if self.file_size < 1 {
1
} else {
self.file_size as u64
}
}
}
/// One file part referencing a chunk (`EpicDownloadManager.ChunkPart`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ChunkPart {
pub guid: [u32; 4],
/// Offset into the DECOMPRESSED chunk (Java `int`).
pub offset: i32,
/// Part length (Java `int`; summed as `size & 0xFFFFFFFFL`).
pub size: i32,
}
impl ChunkPart {
pub fn guid_str(&self) -> String {
guid_to_string(&self.guid)
}
}
/// One entry of the FileManifestList section (`EpicDownloadManager.FileInfo`).
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct FileInfo {
pub filename: String,
pub parts: Vec<ChunkPart>,
/// Install-tag names; empty = required/base file.
pub install_tags: Vec<String>,
/// Full-file SHA-1; `None` for JSON manifests.
pub sha1: Option<[u8; 20]>,
}
impl FileInfo {
/// `FileInfo.fileSize()` = Σ `(part.size & 0xFFFFFFFFL)`.
pub fn file_size(&self) -> u64 {
self.parts.iter().map(|p| p.size as u32 as u64).sum()
}
}
/// Parsed manifest (`EpicDownloadManager.EpicManifest.ParsedManifest`, minus the CDN list which
/// the Java side still owns).
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct Manifest {
pub chunk_dir: String,
pub unique_chunks: Vec<ChunkInfo>,
pub files: Vec<FileInfo>,
}
impl Manifest {
/// `byGuid` map as `uniqueChunksForFiles` builds it: `LinkedHashMap.put` per chunk, so a
/// duplicate GUID resolves to the LAST `ChunkInfo`.
pub fn chunk_index_by_guid(&self) -> HashMap<String, usize> {
let mut map = HashMap::with_capacity(self.unique_chunks.len());
for (i, c) in self.unique_chunks.iter().enumerate() {
map.insert(c.guid_str(), i);
}
map
}
}
pub fn guid_to_string(guid: &[u32; 4]) -> String {
format!("{:08X}{:08X}{:08X}{:08X}", guid[0], guid[1], guid[2], guid[3])
}
/// `chunkDir` from the binary header version (`parseManifest`).
pub fn chunk_dir_for_binary_version(version: i32) -> &'static str {
if version >= 15 {
"ChunksV4"
} else if version >= 6 {
"ChunksV3"
} else if version >= 3 {
"ChunksV2"
} else {
"Chunks"
}
}
/// `chunkDir` from `ManifestFileVersion` (`parseJsonManifest`; note the different `< 3` default).
pub fn chunk_dir_for_json_version(version: i32) -> &'static str {
if version >= 15 {
"ChunksV4"
} else if version >= 6 {
"ChunksV3"
} else if version >= 3 {
"ChunksV2"
} else {
"ChunksV4"
}
}
// ── little-endian cursor with ByteBuffer-like failure semantics ─────────────────────────────
/// Mirrors the `ByteBuffer` calls the Java parser makes: every under-run or out-of-range
/// `position()` is an exception there, which the Java code turns into "parse failed" (null).
struct Cursor<'a> {
buf: &'a [u8],
pos: usize,
}
impl<'a> Cursor<'a> {
fn new(buf: &'a [u8]) -> Self {
Self { buf, pos: 0 }
}
fn position(&self) -> usize {
self.pos
}
/// `ByteBuffer.position(p)` — `p` must be within `0..=limit`.
fn set_position(&mut self, p: i64) -> Result<(), String> {
if p < 0 || p as u64 > self.buf.len() as u64 {
return Err(format!("position {p} out of range (limit {})", self.buf.len()));
}
self.pos = p as usize;
Ok(())
}
fn take(&mut self, n: usize) -> Result<&'a [u8], String> {
let end = self
.pos
.checked_add(n)
.ok_or_else(|| "buffer overflow".to_string())?;
if end > self.buf.len() {
return Err(format!(
"buffer underflow: need {n} at {} (limit {})",
self.pos,
self.buf.len()
));
}
let out = &self.buf[self.pos..end];
self.pos = end;
Ok(out)
}
fn u8(&mut self) -> Result<u8, String> {
Ok(self.take(1)?[0])
}
fn i32(&mut self) -> Result<i32, String> {
let b = self.take(4)?;
Ok(i32::from_le_bytes([b[0], b[1], b[2], b[3]]))
}
fn u32(&mut self) -> Result<u32, String> {
Ok(self.i32()? as u32)
}
fn u64(&mut self) -> Result<u64, String> {
let b = self.take(8)?;
Ok(u64::from_le_bytes([
b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7],
]))
}
fn sha1(&mut self) -> Result<[u8; 20], String> {
let b = self.take(20)?;
let mut out = [0u8; 20];
out.copy_from_slice(b);
Ok(out)
}
/// `ByteBuffer.position(position + n)` with the same range check.
fn skip(&mut self, n: i64) -> Result<(), String> {
let target = self.pos as i64 + n;
self.set_position(target)
}
fn remaining(&self) -> &'a [u8] {
&self.buf[self.pos..]
}
/// `EpicDownloadManager.readFString`: int32 length; `0` → ""; negative → `(-len)-1` UTF-16LE
/// code units + 2-byte terminator; positive → `len-1` US-ASCII bytes + 1-byte terminator.
fn fstring(&mut self) -> Result<String, String> {
let length = self.i32()?;
if length == 0 {
return Ok(String::new());
}
if length < 0 {
// Java: `int chars = (-length) - 1;` — `-Integer.MIN_VALUE` wraps to itself, then -1
// gives Integer.MAX_VALUE; `new byte[chars * 2]` overflows negative → exception.
let chars = length.wrapping_neg().wrapping_sub(1);
let byte_len = chars.checked_mul(2).ok_or("fstring length overflow")?;
if byte_len < 0 {
return Err("negative fstring length".to_string());
}
let bytes = self.take(byte_len as usize)?;
self.take(2)?; // null terminator
let units: Vec<u16> = bytes
.chunks_exact(2)
.map(|p| u16::from_le_bytes([p[0], p[1]]))
.collect();
Ok(String::from_utf16_lossy(&units))
} else {
let bytes = self.take((length - 1) as usize)?;
self.take(1)?; // null terminator
// `new String(bytes, US_ASCII)`: every byte > 0x7F decodes to U+FFFD.
Ok(bytes
.iter()
.map(|&b| if b < 0x80 { b as char } else { '\u{FFFD}' })
.collect())
}
}
}
// ── binary manifest ──────────────────────────────────────────────────────────────────────
/// `EpicDownloadManager.parseManifest(bytes)`: binary when the magic matches, otherwise the JSON
/// parser. `Err` wherever Java returns `null`.
pub fn parse_manifest(bytes: &[u8]) -> Result<Manifest, String> {
let mut cur = Cursor::new(bytes);
let magic = cur.u32()?;
if magic != MANIFEST_MAGIC {
return parse_json_manifest(bytes);
}
parse_binary_after_magic(&mut cur)
}
fn parse_binary_after_magic(cur: &mut Cursor<'_>) -> Result<Manifest, String> {
let header_size = cur.i32()?;
let size_uncompressed = cur.i32()?;
let _size_compressed = cur.i32()?;
cur.skip(20)?; // SHA-1 of the body, not checked by the Java parser
let stored_as = cur.u8()?;
let version = cur.i32()?;
let chunk_dir = chunk_dir_for_binary_version(version);
cur.set_position(header_size as i64)?;
let body_slice = cur.remaining();
let inflated;
let body: &[u8] = if (stored_as & 1) != 0 {
if size_uncompressed < 0 {
return Err("negative sizeUncompressed".to_string());
}
// Java: `inflater.inflate(new byte[sizeUncompressed])` — a single call that fills at most
// `sizeUncompressed` bytes; anything else (short, corrupt) is a size mismatch → null.
let mut out = Vec::with_capacity(size_uncompressed as usize);
let mut dec = flate2::read::ZlibDecoder::new(body_slice).take(size_uncompressed as u64);
dec.read_to_end(&mut out)
.map_err(|e| format!("manifest inflate failed: {e}"))?;
if out.len() != size_uncompressed as usize {
return Err(format!(
"decomp size mismatch: expected {size_uncompressed} got {}",
out.len()
));
}
inflated = out;
&inflated
} else {
body_slice
};
let mut b = Cursor::new(body);
// ManifestMeta: skipped by its declared size (the size field counts itself).
let meta_size = b.i32()?;
let meta_end = b.position() as i64 - 4 + meta_size as i64;
b.set_position(meta_end)?;
// ChunkDataList
let cdl_start = b.position();
let cdl_size = b.i32()?;
let _cdl_version = b.u8()?;
let chunk_count = b.i32()?;
if chunk_count < 0 {
return Err("negative chunkCount".to_string()); // `new ArrayList<>(negative)` throws
}
let chunk_count = chunk_count as usize;
let mut chunks: Vec<ChunkInfo> = (0..chunk_count)
.map(|_| ChunkInfo {
guid: [0; 4],
hash: 0,
sha1: None,
group_num: 0,
window_size: 0,
file_size: 0,
})
.collect();
for c in chunks.iter_mut() {
c.guid = [b.u32()?, b.u32()?, b.u32()?, b.u32()?];
}
for c in chunks.iter_mut() {
c.hash = b.u64()?;
}
for c in chunks.iter_mut() {
c.sha1 = Some(b.sha1()?);
}
for c in chunks.iter_mut() {
c.group_num = b.u8()? as i32;
}
for c in chunks.iter_mut() {
c.window_size = b.i32()?;
}
for c in chunks.iter_mut() {
c.file_size = b.u64()? as i64;
}
b.set_position(cdl_start as i64 + cdl_size as i64)?;
// FileManifestList
let fml_start = b.position();
let fml_size = b.i32()?;
let _fml_version = b.u8()?;
let file_count = b.i32()?;
if file_count < 0 {
return Err("negative fileCount".to_string());
}
let file_count = file_count as usize;
let mut files: Vec<FileInfo> = (0..file_count).map(|_| FileInfo::default()).collect();
for f in files.iter_mut() {
f.filename = b.fstring()?;
}
for _ in 0..file_count {
b.fstring()?; // symlink targets
}
for f in files.iter_mut() {
f.sha1 = Some(b.sha1()?);
}
b.skip(file_count as i64)?; // per-file flags byte
for f in files.iter_mut() {
let tag_count = b.i32()?;
let mut j = 0;
while j < tag_count {
let tag = b.fstring()?;
if !tag.is_empty() {
f.install_tags.push(tag);
}
j += 1;
}
}
for f in files.iter_mut() {
let part_count = b.i32()?;
let mut j = 0;
while j < part_count {
let part_start = b.position();
let part_struct_size = b.i32()?;
let guid = [b.u32()?, b.u32()?, b.u32()?, b.u32()?];
let offset = b.i32()?;
let size = b.i32()?;
f.parts.push(ChunkPart { guid, offset, size });
b.set_position(part_start as i64 + part_struct_size as i64)?;
j += 1;
}
}
b.set_position(fml_start as i64 + fml_size as i64)?;
Ok(Manifest {
chunk_dir: chunk_dir.to_string(),
unique_chunks: dedup_last_wins_first_position(chunks),
files,
})
}
/// `LinkedHashMap<String, ChunkInfo>` keyed by GUID string: a repeated key keeps its FIRST
/// position but takes the LAST value.
fn dedup_last_wins_first_position(chunks: Vec<ChunkInfo>) -> Vec<ChunkInfo> {
let mut order: Vec<ChunkInfo> = Vec::with_capacity(chunks.len());
let mut index: HashMap<String, usize> = HashMap::with_capacity(chunks.len());
for c in chunks {
let key = c.guid_str();
match index.get(&key).copied() {
Some(i) => order[i] = c,
None => {
index.insert(key, order.len());
order.push(c);
}
}
}
order
}
// ── JSON manifest (older games) ──────────────────────────────────────────────────────────
/// `org.json` value → string coercion as `optString` / `getString` do on Android (numbers and
/// booleans print as-is, JSON null prints "null", containers print their JSON).
fn json_to_string(v: &serde_json::Value) -> String {
match v {
serde_json::Value::String(s) => s.clone(),
serde_json::Value::Null => "null".to_string(),
other => other.to_string(),
}
}
/// `obj.optString(key, fallback)`.
fn opt_string(obj: &serde_json::Map<String, serde_json::Value>, key: &str, fallback: &str) -> String {
match obj.get(key) {
Some(v) => json_to_string(v),
None => fallback.to_string(),
}
}
/// `(int) Long.parseLong(hex8, 16)` — the Java GUID quarter parse; invalid hex throws out of the
/// whole parse.
fn parse_guid_quarter(s: &str) -> Result<u32, String> {
let v = i64::from_str_radix(s, 16).map_err(|e| format!("bad guid hex '{s}': {e}"))?;
Ok(v as i32 as u32)
}
fn parse_guid_from_hex(s: &str) -> Result<[u32; 4], String> {
// Java substrings by UTF-16 unit and `parseLong` throws on anything non-hex; a non-ASCII
// prefix can never parse, so refuse it before slicing (keeps the slice on char boundaries).
if s.len() < 32 || !s.as_bytes()[..32].is_ascii() {
return Err(format!("bad guid hex '{s}'"));
}
Ok([
parse_guid_quarter(&s[0..8])?,
parse_guid_quarter(&s[8..16])?,
parse_guid_quarter(&s[16..24])?,
parse_guid_quarter(&s[24..32])?,
])
}
/// `EpicDownloadManager.parseJsonManifest`.
pub fn parse_json_manifest(bytes: &[u8]) -> Result<Manifest, String> {
let text = String::from_utf8_lossy(bytes);
let root: serde_json::Value =
serde_json::from_str(&text).map_err(|e| format!("JSON manifest: {e}"))?;
let root = root
.as_object()
.ok_or_else(|| "JSON manifest: root is not an object".to_string())?;
let manifest_version: i32 = opt_string(root, "ManifestFileVersion", "0")
.parse::<i32>()
.unwrap_or(0);
let chunk_dir = chunk_dir_for_json_version(manifest_version);
let chunk_hash_list = root
.get("ChunkHashList")
.and_then(|v| v.as_object())
.ok_or_else(|| "JSON manifest: no ChunkHashList".to_string())?;
let data_group_list = root.get("DataGroupList").and_then(|v| v.as_object());
let chunk_filesize_list = root.get("ChunkFilesizeList").and_then(|v| v.as_object());
// `LinkedHashMap<String, ChunkInfo>` keyed by the RAW key string (not the normalized GUID).
let mut chunk_order: Vec<ChunkInfo> = Vec::new();
let mut chunk_index: HashMap<String, usize> = HashMap::new();
for (guid_hex, hash_value) in chunk_hash_list.iter() {
if guid_hex.len() < 32 || !guid_hex.is_char_boundary(32) {
continue;
}
let hash_hex = json_to_string(hash_value);
let guid = parse_guid_from_hex(guid_hex)?;
let hash = if hash_hex.len() >= 16 && hash_hex.is_char_boundary(16) {
u64::from_str_radix(&hash_hex[0..16], 16).unwrap_or(0)
} else {
0
};
let group_num = match data_group_list {
Some(dgl) => opt_string(dgl, guid_hex, "0").parse::<i32>().unwrap_or(0),
None => 0,
};
let file_size = match chunk_filesize_list {
Some(cfl) => i64::from_str_radix(&opt_string(cfl, guid_hex, "0"), 16).unwrap_or(0),
None => 0,
};
let c = ChunkInfo {
guid,
hash,
sha1: None,
group_num,
window_size: 0,
file_size,
};
match chunk_index.get(guid_hex).copied() {
Some(i) => chunk_order[i] = c,
None => {
chunk_index.insert(guid_hex.clone(), chunk_order.len());
chunk_order.push(c);
}
}
}
let file_list = root
.get("FileManifestList")
.and_then(|v| v.as_array())
.ok_or_else(|| "JSON manifest: no FileManifestList".to_string())?;
let mut files = Vec::with_capacity(file_list.len());
for entry in file_list {
let file_obj = entry
.as_object()
.ok_or_else(|| "JSON manifest: FileManifestList entry is not an object".to_string())?;
let mut fi = FileInfo {
filename: opt_string(file_obj, "Filename", ""),
..FileInfo::default()
};
if let Some(tags) = file_obj.get("InstallTags").and_then(|v| v.as_array()) {
for t in tags {
let tag = json_to_string(t);
if !tag.is_empty() {
fi.install_tags.push(tag);
}
}
}
if let Some(parts) = file_obj.get("FileChunkParts").and_then(|v| v.as_array()) {
for p in parts {
let part_obj = p.as_object().ok_or_else(|| {
"JSON manifest: FileChunkParts entry is not an object".to_string()
})?;
let mut part = ChunkPart {
guid: [0; 4],
offset: 0,
size: 0,
};
let part_guid = opt_string(part_obj, "Guid", "");
if part_guid.len() >= 32 && part_guid.is_char_boundary(32) {
part.guid = parse_guid_from_hex(&part_guid)?;
}
part.offset = opt_string(part_obj, "Offset", "0").parse::<i32>().unwrap_or(0);
part.size = opt_string(part_obj, "Size", "0").parse::<i32>().unwrap_or(0);
fi.parts.push(part);
}
}
files.push(fi);
}
Ok(Manifest {
chunk_dir: chunk_dir.to_string(),
unique_chunks: chunk_order,
files,
})
}
// ── tests ────────────────────────────────────────────────────────────────────────────────
#[cfg(test)]
pub(crate) mod test_support {
//! Synthetic binary-manifest builder used by this module's tests and by `plan`/`driver`
//! tests. Emits exactly the layout `parseManifest` walks (meta / CDL / FML with sizes).
use super::*;
use std::io::Write;
pub struct TestChunk {
pub guid: [u32; 4],
pub hash: u64,
pub sha1: [u8; 20],
pub group: u8,
pub window: i32,
pub file_size: u64,
}
pub struct TestFile {
pub name: String,
pub sha1: [u8; 20],
pub tags: Vec<String>,
/// (guid, offset, size)
pub parts: Vec<([u32; 4], i32, i32)>,
}
pub fn fstring_ascii(s: &str) -> Vec<u8> {
let mut v = Vec::new();
if s.is_empty() {
v.extend_from_slice(&0i32.to_le_bytes());
return v;
}
v.extend_from_slice(&((s.len() as i32) + 1).to_le_bytes());
v.extend_from_slice(s.as_bytes());
v.push(0);
v
}
pub fn fstring_utf16(s: &str) -> Vec<u8> {
let units: Vec<u16> = s.encode_utf16().collect();
let mut v = Vec::new();
v.extend_from_slice(&(-((units.len() as i32) + 1)).to_le_bytes());
for u in units {
v.extend_from_slice(&u.to_le_bytes());
}
v.extend_from_slice(&[0, 0]);
v
}
/// Body (meta + CDL + FML) without the outer header.
pub fn build_body(chunks: &[TestChunk], files: &[TestFile], meta_padding: usize) -> Vec<u8> {
let mut body = Vec::new();
// ManifestMeta: size field counts itself.
let meta_size = 4 + meta_padding;
body.extend_from_slice(&(meta_size as i32).to_le_bytes());
body.extend(std::iter::repeat(0xEEu8).take(meta_padding));
// ChunkDataList
let mut cdl = Vec::new();
cdl.push(0u8); // version
cdl.extend_from_slice(&(chunks.len() as i32).to_le_bytes());
for c in chunks {
for g in c.guid {
cdl.extend_from_slice(&g.to_le_bytes());
}
}
for c in chunks {
cdl.extend_from_slice(&c.hash.to_le_bytes());
}
for c in chunks {
cdl.extend_from_slice(&c.sha1);
}
for c in chunks {
cdl.push(c.group);
}
for c in chunks {
cdl.extend_from_slice(&c.window.to_le_bytes());
}
for c in chunks {
cdl.extend_from_slice(&c.file_size.to_le_bytes());
}
// Trailing padding the parser must skip via cdlSize.
cdl.extend_from_slice(&[0xAA, 0xBB, 0xCC]);
body.extend_from_slice(&((cdl.len() + 4) as i32).to_le_bytes());
body.extend_from_slice(&cdl);
// FileManifestList
let mut fml = Vec::new();
fml.push(0u8); // version
fml.extend_from_slice(&(files.len() as i32).to_le_bytes());
for f in files {
if f.name.is_ascii() {
fml.extend_from_slice(&fstring_ascii(&f.name));
} else {
fml.extend_from_slice(&fstring_utf16(&f.name));
}
}
for _ in files {
fml.extend_from_slice(&fstring_ascii("")); // symlink target
}
for f in files {
fml.extend_from_slice(&f.sha1);
}
for _ in files {
fml.push(0); // flags
}
for f in files {
fml.extend_from_slice(&(f.tags.len() as i32).to_le_bytes());
for t in &f.tags {
fml.extend_from_slice(&fstring_ascii(t));
}
}
for f in files {
fml.extend_from_slice(&(f.parts.len() as i32).to_le_bytes());
for (guid, offset, size) in &f.parts {
// struct size 4 + 16 + 4 + 4 = 28, plus 2 bytes of padding the parser skips.
fml.extend_from_slice(&30i32.to_le_bytes());
for g in guid {
fml.extend_from_slice(&g.to_le_bytes());
}
fml.extend_from_slice(&offset.to_le_bytes());
fml.extend_from_slice(&size.to_le_bytes());
fml.extend_from_slice(&[0x11, 0x22]);
}
}
body.extend_from_slice(&((fml.len() + 4) as i32).to_le_bytes());
body.extend_from_slice(&fml);
body
}
/// Full manifest bytes: header + (optionally zlib-compressed) body.
pub fn build_manifest(
chunks: &[TestChunk],
files: &[TestFile],
version: i32,
compress: bool,
) -> Vec<u8> {
let body = build_body(chunks, files, 7);
let payload = if compress {
let mut enc =
flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::default());
enc.write_all(&body).unwrap();
enc.finish().unwrap()
} else {
body.clone()
};
let header_size: i32 = 41 + 3; // 3 bytes of extra header the parser must jump over
let mut out = Vec::new();
out.extend_from_slice(&MANIFEST_MAGIC.to_le_bytes());
out.extend_from_slice(&header_size.to_le_bytes());
out.extend_from_slice(&(body.len() as i32).to_le_bytes());
out.extend_from_slice(&(payload.len() as i32).to_le_bytes());
out.extend_from_slice(&[0u8; 20]);
out.push(if compress { 1 } else { 0 });
out.extend_from_slice(&version.to_le_bytes());
out.extend_from_slice(&[0xF0, 0xF1, 0xF2]);
assert_eq!(out.len(), header_size as usize);
out.extend_from_slice(&payload);
out
}
pub fn sample_chunks() -> Vec<TestChunk> {
vec![
TestChunk {
guid: [0x0000_0001, 0x0000_0002, 0x0000_0003, 0x0000_0004],
hash: 0x0123_4567_89AB_CDEF,
sha1: [0x11; 20],
group: 5,
window: 1_048_576,
file_size: 700_000,
},
TestChunk {
guid: [0xDEAD_BEEF, 0xFFFF_FFFF, 0x0000_0000, 0x8000_0000],
hash: 0xFFFF_FFFF_FFFF_FFFE,
sha1: [0x22; 20],
group: 123,
window: 4096,
file_size: 0,
},
TestChunk {
guid: [0x0000_00AA, 0x0000_00BB, 0x0000_00CC, 0x0000_00DD],
hash: 0x10,
sha1: [0u8; 20],
group: 0,
window: 10,
file_size: 9,
},
]
}
pub fn sample_files() -> Vec<TestFile> {
let c = sample_chunks();
vec![
TestFile {
name: "Game/Binaries/Win64/Game.exe".to_string(),
sha1: [0x51; 20],
tags: vec![],
parts: vec![(c[0].guid, 0, 1_048_576), (c[1].guid, 100, 4000)],
},
TestFile {
name: "Game\\Content\\Localization\\de.pak".to_string(),
sha1: [0x52; 20],
tags: vec!["German".to_string(), "de-DE".to_string()],
parts: vec![(c[1].guid, 0, 4096)],
},
TestFile {
name: "Game/Content/Ünïcode.pak".to_string(),
sha1: [0x53; 20],
tags: vec!["French".to_string()],
parts: vec![(c[2].guid, 0, 10), (c[0].guid, 5, 7)],
},
]
}
}
#[cfg(test)]
mod tests {
use super::test_support::*;
use super::*;
#[test]
fn binary_manifest_round_trips_uncompressed() {
let bytes = build_manifest(&sample_chunks(), &sample_files(), 21, false);
let m = parse_manifest(&bytes).expect("parse");
assert_eq!(m.chunk_dir, "ChunksV4");
assert_eq!(m.unique_chunks.len(), 3);
assert_eq!(m.files.len(), 3);
let c0 = &m.unique_chunks[0];
assert_eq!(c0.guid_str(), "00000001000000020000000300000004");
assert_eq!(c0.hash, 0x0123_4567_89AB_CDEF);
assert_eq!(c0.sha1, Some([0x11; 20]));
assert_eq!(c0.group_num, 5);
assert_eq!(c0.window_size, 1_048_576);
assert_eq!(c0.file_size, 700_000);
assert_eq!(
c0.path("ChunksV4"),
"ChunksV4/05/0123456789ABCDEF_00000001000000020000000300000004.chunk"
);
let c1 = &m.unique_chunks[1];
assert_eq!(c1.guid_str(), "DEADBEEFFFFFFFFF0000000080000000");
assert_eq!(
c1.path("ChunksV4"),
"ChunksV4/123/FFFFFFFFFFFFFFFE_DEADBEEFFFFFFFFF0000000080000000.chunk"
);
assert_eq!(c1.credit_bytes(), 1, "Math.max(fileSize, 1)");
assert!(c1.verifiable_sha1().is_some());
assert!(m.unique_chunks[2].verifiable_sha1().is_none(), "all-zero SHA-1 → unverifiable");
let f0 = &m.files[0];
assert_eq!(f0.filename, "Game/Binaries/Win64/Game.exe");
assert!(f0.install_tags.is_empty());
assert_eq!(f0.sha1, Some([0x51; 20]));
assert_eq!(f0.parts.len(), 2);
assert_eq!(f0.parts[1].offset, 100);
assert_eq!(f0.parts[1].size, 4000);
assert_eq!(f0.file_size(), 1_048_576 + 4000);
let f1 = &m.files[1];
assert_eq!(f1.filename, "Game\\Content\\Localization\\de.pak");
assert_eq!(f1.install_tags, vec!["German".to_string(), "de-DE".to_string()]);
let f2 = &m.files[2];
assert_eq!(f2.filename, "Game/Content/Ünïcode.pak", "UTF-16LE fstring");
assert_eq!(f2.parts[1].guid_str(), c0.guid_str());
}
#[test]
fn binary_manifest_round_trips_compressed() {
let bytes = build_manifest(&sample_chunks(), &sample_files(), 17, true);
let m = parse_manifest(&bytes).expect("parse");
assert_eq!(m.unique_chunks.len(), 3);
assert_eq!(m.files.len(), 3);
assert_eq!(m.files[2].filename, "Game/Content/Ünïcode.pak");
}
#[test]
fn chunk_dir_follows_version_thresholds() {
for (v, dir) in [(2, "Chunks"), (3, "ChunksV2"), (6, "ChunksV3"), (15, "ChunksV4")] {
let bytes = build_manifest(&sample_chunks(), &sample_files(), v, false);
assert_eq!(parse_manifest(&bytes).unwrap().chunk_dir, dir, "version {v}");
}
assert_eq!(chunk_dir_for_json_version(1), "ChunksV4");
assert_eq!(chunk_dir_for_json_version(4), "ChunksV2");
}
#[test]
fn duplicate_guid_keeps_first_position_last_value() {
let mut chunks = sample_chunks();
let dup = TestChunk {
guid: chunks[0].guid,
hash: 0x99,
sha1: [0x77; 20],
group: 9,
window: 1,
file_size: 1,
};
chunks.push(dup);
let bytes = build_manifest(&chunks, &sample_files(), 21, false);
let m = parse_manifest(&bytes).unwrap();
assert_eq!(m.unique_chunks.len(), 3);
assert_eq!(m.unique_chunks[0].hash, 0x99);
assert_eq!(m.unique_chunks[0].group_num, 9);
}
#[test]
fn truncated_manifest_is_a_parse_failure() {
let bytes = build_manifest(&sample_chunks(), &sample_files(), 21, false);
assert!(parse_manifest(&bytes[..bytes.len() - 40]).is_err());
assert!(parse_manifest(&bytes[..20]).is_err());
}
#[test]
fn bad_uncompressed_size_is_a_parse_failure() {
let good = build_manifest(&sample_chunks(), &sample_files(), 21, true);
let real = i32::from_le_bytes([good[8], good[9], good[10], good[11]]);
// Claim MORE than the body inflates to: Java's single `inflate()` returns fewer bytes
// than `sizeUncompressed` → "Decomp size mismatch" → null.
let mut bigger = good.clone();
bigger[8..12].copy_from_slice(&(real + 1).to_le_bytes());
assert!(parse_manifest(&bigger).is_err());
// Claim LESS: Java fills exactly `sizeUncompressed` bytes (silently truncating), then the
// FileManifestList end position lands past the limit → exception → null.
let mut smaller = good;
smaller[8..12].copy_from_slice(&(real - 1).to_le_bytes());
assert!(parse_manifest(&smaller).is_err());
}
#[test]
fn fstring_decoding_matches_java() {
let mut v = fstring_ascii("abc");
v.extend_from_slice(&fstring_utf16("dé"));
v.extend_from_slice(&fstring_ascii(""));
v.extend_from_slice(&[4, 0, 0, 0, b'a', 0xFF, b'b', 0]); // non-ASCII byte → U+FFFD
let mut c = Cursor::new(&v);
assert_eq!(c.fstring().unwrap(), "abc");
assert_eq!(c.fstring().unwrap(), "dé");
assert_eq!(c.fstring().unwrap(), "");
assert_eq!(c.fstring().unwrap(), "a\u{FFFD}b");
}
#[test]
fn json_manifest_parses_like_java() {
let json = r#"{
"ManifestFileVersion": "013",
"ChunkHashList": {
"0000000100000002000000030000000ff": "0123456789ABCDEF00",
"short": "00",
"DEADBEEFFFFFFFFF0000000080000000": "zz"
},
"DataGroupList": { "0000000100000002000000030000000ff": "7", "DEADBEEFFFFFFFFF0000000080000000": "x" },
"ChunkFilesizeList": { "0000000100000002000000030000000ff": "FF" },
"FileManifestList": [
{ "Filename": "a.bin", "InstallTags": ["German", ""],
"FileChunkParts": [ { "Guid": "0000000100000002000000030000000ff", "Offset": "12", "Size": "34" },
{ "Guid": "tooshort", "Offset": "x", "Size": "-5" } ] },
{ "Filename": "b.bin" }
]
}"#;
let m = parse_manifest(json.as_bytes()).expect("json parse");
assert_eq!(m.chunk_dir, "ChunksV3");
assert_eq!(m.unique_chunks.len(), 2, "keys shorter than 32 chars are skipped");
let c0 = &m.unique_chunks[0];
assert_eq!(c0.guid, [1, 2, 3, 0x0000_000F]);
assert_eq!(c0.hash, 0x0123_4567_89AB_CDEF);
assert_eq!(c0.group_num, 7);
assert_eq!(c0.file_size, 0xFF);
assert_eq!(c0.window_size, 0);
assert!(c0.sha1.is_none());
let c1 = &m.unique_chunks[1];
assert_eq!(c1.hash, 0, "unparseable hash → 0");
assert_eq!(c1.group_num, 0, "unparseable group → 0");
assert_eq!(c1.file_size, 0, "missing size → 0");
assert_eq!(m.files.len(), 2);
assert_eq!(m.files[0].install_tags, vec!["German".to_string()]);
assert_eq!(m.files[0].parts.len(), 2);
assert_eq!(m.files[0].parts[0].offset, 12);
assert_eq!(m.files[0].parts[0].size, 34);
assert_eq!(m.files[0].parts[1].guid, [0; 4]);
assert_eq!(m.files[0].parts[1].offset, 0);
assert_eq!(m.files[0].parts[1].size, -5);
assert!(m.files[0].sha1.is_none());
assert!(m.files[1].parts.is_empty());
}
#[test]
fn json_manifest_failures_match_java_nulls() {
assert!(parse_manifest(br#"{"FileManifestList": []}"#).is_err(), "no ChunkHashList");
assert!(parse_manifest(br#"{"ChunkHashList": {}}"#).is_err(), "no FileManifestList");
assert!(
parse_manifest(br#"{"ChunkHashList": {"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz": "0"}, "FileManifestList": []}"#)
.is_err(),
"invalid GUID hex throws out of the parse"
);
assert!(parse_manifest(b"not json at all").is_err());
}
}
+272
View File
@@ -0,0 +1,272 @@
//! Chunk plan — the part of `EpicDownloadManager.install` between "manifest parsed" and "submit
//! to the pool", ported 1:1: install-tag file selection (`resolveInstallFiles` /
//! `getRequiredInstallFiles`), the unique-chunk list for a file set (`uniqueChunksForFiles`),
//! the byte total (`Σ max(fileSize, 1)`), chunk cache paths and CDN URLs.
//!
//! At runtime the Java manager passes the INDICES of its `pendingFiles` (post delta/verify), so
//! the file selection functions here are exercised by the unit tests (and available to a future
//! caller) while the driver only runs `unique_chunks_for_files` on Java's pending set — the same
//! function Java runs, on the same manifest, in the same order.
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf};
use super::manifest::{ChunkInfo, Manifest};
/// `getRequiredInstallFiles`: files with no install tags; if NONE carries a tag, every file.
pub fn required_install_files(m: &Manifest) -> Vec<usize> {
let required: Vec<usize> = m
.files
.iter()
.enumerate()
.filter(|(_, f)| f.install_tags.is_empty())
.map(|(i, _)| i)
.collect();
if required.is_empty() {
(0..m.files.len()).collect()
} else {
required
}
}
/// `resolveInstallFiles(manifest, selectedTags)`:
/// - `None` → every file (legacy / DLC / overlay path);
/// - `Some([])` → required(base) files only;
/// - otherwise → base files + any file carrying at least one selected tag, falling back to
/// required-only when nothing matched.
pub fn resolve_install_files(m: &Manifest, selected_tags: Option<&[String]>) -> Vec<usize> {
let Some(tags) = selected_tags else {
return (0..m.files.len()).collect();
};
if tags.is_empty() {
return required_install_files(m);
}
let want: HashSet<&str> = tags.iter().map(String::as_str).collect();
let mut out = Vec::new();
for (i, f) in m.files.iter().enumerate() {
if f.install_tags.is_empty() {
out.push(i);
continue;
}
if f.install_tags.iter().any(|t| want.contains(t.as_str())) {
out.push(i);
}
}
if out.is_empty() {
required_install_files(m)
} else {
out
}
}
/// `uniqueChunksForFiles(manifest, files)`: chunk indices (into `unique_chunks`) in first-seen
/// order over `files[*].parts`, each GUID once, GUIDs absent from the chunk list dropped.
pub fn unique_chunks_for_files(m: &Manifest, file_indices: &[usize]) -> Vec<usize> {
let by_guid: HashMap<String, usize> = m.chunk_index_by_guid();
let mut seen: HashSet<String> = HashSet::new();
let mut chunks = Vec::new();
for &fi in file_indices {
let Some(f) = m.files.get(fi) else {
continue;
};
for p in &f.parts {
let g = p.guid_str();
if seen.insert(g.clone()) {
if let Some(&ci) = by_guid.get(&g) {
chunks.push(ci);
}
}
}
}
chunks
}
/// `totalBytes += Math.max(chunk.fileSize, 1)` over the needed chunks.
pub fn total_credit_bytes(m: &Manifest, chunk_indices: &[usize]) -> u64 {
chunk_indices
.iter()
.filter_map(|&i| m.unique_chunks.get(i))
.map(ChunkInfo::credit_bytes)
.sum()
}
/// `<installDir>/.chunks`.
pub fn chunk_cache_dir(install_dir: &str) -> PathBuf {
Path::new(install_dir).join(super::CHUNK_CACHE_DIR)
}
/// Final cache file for a chunk: `new File(chunkCacheDir, chunk.guidStr())`.
pub fn cached_chunk_path(cache_dir: &Path, chunk: &ChunkInfo) -> PathBuf {
cache_dir.join(chunk.guid_str())
}
/// The Java pool's skip rule: a chunk whose cache file EXISTS (any size) is not fetched.
pub fn is_chunk_cached(cache_dir: &Path, chunk: &ChunkInfo) -> bool {
cached_chunk_path(cache_dir, chunk).exists()
}
/// One CDN entry as the Java manager passes it: `cdn.baseUrl + cdn.cloudDir` (auth params are
/// never used on chunk URLs). Chunk URL = `prefix + "/" + chunk.getPath(chunkDir)`.
pub fn chunk_url(cdn_prefix: &str, chunk_dir: &str, chunk: &ChunkInfo) -> String {
format!("{}/{}", cdn_prefix, chunk.path(chunk_dir))
}
/// Undo the JSON string escaping the Java CDN scanner passes through. `EpicApiClient` re-serializes
/// the manifest API response with Android's `org.json`, which escapes every `/` as `\/`, and
/// `parseCdnUrls` takes the raw substrings — so a base URL arrives as `https:\/\/host\` and a cloud
/// dir as `/Builds\/Org\/…\/default\`. Java's `HttpURLConnection` (OkHttp) treats `\` as `/` and
/// only ever reached the first CDN, so it never noticed; reqwest's URL parser does the same
/// mapping, which leaves empty `//` path segments that Fastly/Akamai tolerate but CloudFront
/// rejects outright (immediate 4xx). Produce the URL Java intended: `\/` → `/`, any other `\`
/// dropped, and no trailing slash (the chunk path is joined with one).
pub fn normalize_prefix(raw: &str) -> String {
let unescaped = raw.replace("\\/", "/").replace('\\', "");
unescaped.trim_end_matches('/').to_string()
}
/// Distinct CDN prefixes in first-seen order, normalized (the fetch core wants one host key per
/// distinct endpoint; Java would simply try an identical URL twice, which changes nothing).
pub fn distinct_prefixes(cdn_prefixes: &[String]) -> Vec<String> {
let mut seen = HashSet::new();
let mut out = Vec::new();
for p in cdn_prefixes {
let n = normalize_prefix(p);
if n.is_empty() {
continue;
}
if seen.insert(n.clone()) {
out.push(n);
}
}
out
}
/// Improvements round 1: split the window across the distinct CDNs so the whole ceiling is
/// reachable whatever the host count — `max(6, ceil(max_workers / hosts))` (3 Epic CDNs at a
/// ceiling of 32 → 11 each; a single host gets all 32).
pub fn per_host_cap(max_workers: usize, host_count: usize) -> usize {
let hosts = host_count.max(1);
let workers = max_workers.max(1);
let split = (workers + hosts - 1) / hosts;
split.max(6)
}
#[cfg(test)]
mod tests {
use super::super::manifest::test_support::*;
use super::super::manifest::parse_manifest;
use super::*;
fn manifest() -> Manifest {
parse_manifest(&build_manifest(&sample_chunks(), &sample_files(), 21, false)).unwrap()
}
#[test]
fn install_tag_rules_match_java() {
let m = manifest();
// null → all
assert_eq!(resolve_install_files(&m, None), vec![0, 1, 2]);
// empty → required only (file 0 has no tags)
assert_eq!(resolve_install_files(&m, Some(&[])), vec![0]);
// German → base + the German file
let de = vec!["German".to_string(), "de-DE".to_string()];
assert_eq!(resolve_install_files(&m, Some(&de)), vec![0, 1]);
// unknown tag → nothing extra matched, but base files matched, so base only
let xx = vec!["Klingon".to_string()];
assert_eq!(resolve_install_files(&m, Some(&xx)), vec![0]);
}
#[test]
fn manifest_without_tags_treats_every_file_as_required() {
let mut files = sample_files();
for f in files.iter_mut() {
f.tags.clear();
}
let m = parse_manifest(&build_manifest(&sample_chunks(), &files, 21, false)).unwrap();
assert_eq!(required_install_files(&m), vec![0, 1, 2]);
assert_eq!(resolve_install_files(&m, Some(&[])), vec![0, 1, 2]);
}
#[test]
fn manifest_where_every_file_is_tagged_falls_back_to_all() {
let mut files = sample_files();
files[0].tags = vec!["English".to_string()];
let m = parse_manifest(&build_manifest(&sample_chunks(), &files, 21, false)).unwrap();
// No untagged file, unknown tag selected → matched nothing → required → (none) → all.
let xx = vec!["Klingon".to_string()];
assert_eq!(resolve_install_files(&m, Some(&xx)), vec![0, 1, 2]);
}
#[test]
fn unique_chunks_follow_first_seen_part_order() {
let m = manifest();
// file 2 first: chunk c[2] then c[0]; then file 0: c[0] (dup), c[1].
assert_eq!(unique_chunks_for_files(&m, &[2, 0]), vec![2, 0, 1]);
assert_eq!(unique_chunks_for_files(&m, &[1]), vec![1]);
assert!(unique_chunks_for_files(&m, &[]).is_empty());
assert!(unique_chunks_for_files(&m, &[99]).is_empty(), "out-of-range index ignored");
// total = 700000 + max(0,1) + 9
assert_eq!(total_credit_bytes(&m, &[0, 1, 2]), 700_000 + 1 + 9);
}
#[test]
fn parts_referencing_unknown_guids_are_dropped() {
let mut files = sample_files();
files[0]
.parts
.push(([0x1234, 0x5678, 0x9ABC, 0xDEF0], 0, 1));
let m = parse_manifest(&build_manifest(&sample_chunks(), &files, 21, false)).unwrap();
assert_eq!(unique_chunks_for_files(&m, &[0]), vec![0, 1]);
}
#[test]
fn urls_and_cache_paths_match_java() {
let m = manifest();
let c = &m.unique_chunks[0];
assert_eq!(
chunk_url("https://fastly-download.epicgames.com/Builds/Org/o-x/y/default", "ChunksV4", c),
"https://fastly-download.epicgames.com/Builds/Org/o-x/y/default/ChunksV4/05/0123456789ABCDEF_00000001000000020000000300000004.chunk"
);
let cache = chunk_cache_dir("/data/x/imagefs/epic_games/Game");
assert_eq!(cache, PathBuf::from("/data/x/imagefs/epic_games/Game/.chunks"));
assert_eq!(
cached_chunk_path(&cache, c),
PathBuf::from("/data/x/imagefs/epic_games/Game/.chunks/00000001000000020000000300000004")
);
let prefixes = vec!["a".to_string(), "b".to_string(), "a".to_string()];
assert_eq!(distinct_prefixes(&prefixes), vec!["a".to_string(), "b".to_string()]);
}
#[test]
fn prefixes_are_unescaped_from_the_java_json_scan() {
// Exactly what Bannerlator's Epic debug log shows its Java manager extracting for a real title.
let raw = "https:\\/\\/egdownload.fastly-edge.com\\/Builds\\/Org\\/o-83e8\\/04b7\\/default\\";
assert_eq!(
normalize_prefix(raw),
"https://egdownload.fastly-edge.com/Builds/Org/o-83e8/04b7/default"
);
// Already-clean input is untouched (minus a trailing slash).
assert_eq!(
normalize_prefix("https://download.epicgames.com/Builds/x/default/"),
"https://download.epicgames.com/Builds/x/default"
);
let m = manifest();
let c = &m.unique_chunks[0];
assert_eq!(
chunk_url(&normalize_prefix(raw), "ChunksV4", c),
"https://egdownload.fastly-edge.com/Builds/Org/o-83e8/04b7/default/ChunksV4/05/0123456789ABCDEF_00000001000000020000000300000004.chunk"
);
let two = vec![raw.to_string(), "https://egdownload.fastly-edge.com/Builds/Org/o-83e8/04b7/default".to_string(), String::new()];
assert_eq!(distinct_prefixes(&two).len(), 1, "escaped and clean forms collapse; empty dropped");
}
#[test]
fn per_host_cap_splits_the_window_with_a_floor_of_six() {
assert_eq!(per_host_cap(32, 3), 11);
assert_eq!(per_host_cap(32, 1), 32);
assert_eq!(per_host_cap(32, 2), 16);
assert_eq!(per_host_cap(8, 3), 6, "floor");
assert_eq!(per_host_cap(8, 1), 8);
assert_eq!(per_host_cap(0, 0), 6, "degenerate inputs");
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,211 @@
//! The numbers and the two small types [`fetch_core`](crate::fetch_core) shares with the Steam
//! depot writer it was carved out of.
//!
//! In Bannerlator these live in `depot_writer.rs` and `cdn_client.rs`, next to the Steam CM and
//! depot code, so that the Steam engine and the store engines run on ONE set of tunables and their
//! `fetch-window` log lines stay A/B-comparable. DroidDeck ships no Steam depot engine (the Steam
//! client runs natively in the Linux runtime), so only this slice came across: the values are
//! copied verbatim and keep their names, which is what lets a GOG/Epic/Amazon log from either app
//! be read against the same tuning notes.
//!
//! The important behaviours are structural (slow-start ramp -> hold at plateau -> back off on REAL
//! errors, per-host cap, hard byte budget), not the exact numbers. The r3/r4 notes below record
//! why each number is what it is, because every one of them was arrived at by a device run that
//! went wrong the other way.
use std::time::Duration;
/// Default `User-Agent` of the pooled client. Every store engine overrides it per request with
/// the agent its own launcher sends (`FetchOptions::headers`), so this value is only what a
/// request without an explicit header would carry.
pub const USER_AGENT: &str = "Valve/Steam HTTP Client 1.0";
/// TCP connect deadline for every request the pooled client makes.
pub fn connect_timeout() -> Duration {
Duration::from_secs(15)
}
/// Classified fetch failure so the fetch driver can back off proportionally (429/5xx/timeout/reset
/// shrink the window + cool the host harder than a one-off protocol error).
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum FetchFailKind {
Timeout,
RateLimited,
ServerFault,
Connect,
Other,
}
#[derive(Clone, Debug)]
pub struct AsyncFetchError {
pub message: String,
pub kind: FetchFailKind,
}
/// Attempts per item before the run gives up on it (fetch failures and sink retries both count).
pub const MAX_CHUNK_ATTEMPTS: u32 = 5;
/// Floor for the in-flight byte budget — the pre-r4 fixed value, so no run ever gets *less*
/// buffered memory than it had before the budget started scaling. See [`inflight_budget_bytes`].
pub const FETCH_INFLIGHT_BUDGET_FLOOR_BYTES: u64 = 24 * 1024 * 1024;
/// Hard ceiling for the in-flight byte budget. This is THE OOM guard: whatever the tier, the host
/// count or the chunk size, the RAW bytes in memory for one run can never exceed this. 96 MiB = 4×
/// the old fixed budget and still a small fraction of the heap an Android game installer runs in,
/// and it is only ever reached by a fast-tier run whose CDN pool is large enough to support a 64+
/// window. Raising this is the one change in the fetch layer that can reintroduce OOM, so it stays
/// fixed and small.
pub const FETCH_INFLIGHT_BUDGET_HARD_CAP_BYTES: u64 = 96 * 1024 * 1024;
/// Assumed RAW (on-the-wire, still-compressed) size of one chunk when sizing the budget. Store
/// chunks are ~1 MiB; this is the same nominal [`NOMINAL_CHUNK_RESERVE_BYTES`] uses at dispatch.
pub const TYPICAL_CHUNK_BYTES: u64 = NOMINAL_CHUNK_RESERVE_BYTES;
/// Headroom the budget carries over "one chunk per window slot" (3/2 = 1.5×), as a rational so the
/// arithmetic stays integer. The window is a count of concurrent *requests*; a slot's bytes are held
/// from DISPATCH until its WRITE completes, so at any instant some slots hold a fetched-but-not-yet-
/// written chunk while others hold a reservation for a fetch still on the wire. 1.5× covers that
/// overlap (and the occasional above-nominal chunk) without doubling the memory bound.
pub const BUDGET_HEADROOM_NUM: u64 = 3;
pub const BUDGET_HEADROOM_DEN: u64 = 2;
/// Byte budget for the *compressed* bytes buffered in the fetch→process channel, scaled to the
/// window ceiling this run can actually reach.
///
/// The decouple channel is bounded by BYTES, not chunk count: a count cap makes in-flight memory
/// swing wildly with chunk size (a 4-chunk cap is ~4 MB for 1 MB chunks but ~120 MB for 30 MB
/// chunks). Budgeting bytes keeps the heap bounded regardless of game size while giving the fetch
/// pool bandwidth-delay-product headroom so the socket stays busy while process workers
/// decompress+write.
///
/// r4: a device run proved a fixed 24 MiB budget becomes the binding constraint the moment the
/// adaptive window works — on a 1 Gbps line the window grew to 42 but the budget admitted only ~24
/// chunks, so `in_flight` stalled at 23–28 with `budget_stalls` in the hundreds and
/// `host_stalls=0`. A budget that refuses slots the window has already decided are safe is not a
/// memory guard, it is a throttle. So the budget scales WITH the effective ceiling (tier ∧
/// hosts×per-host-cap ∧ hard cap) and is clamped between the old fixed value and a hard cap. Low
/// tiers are unaffected; a fast link gets a budget that stops binding before the window does.
///
/// Budget only the RAW bytes actually queued (reserved at DISPATCH, released on WRITE-COMPLETE); the
/// decoded expansion is transient inside a process worker and never accumulates. See
/// [`budget_admits`] for the single-chunk deadlock guard.
pub fn inflight_budget_bytes(effective_ceiling: usize) -> u64 {
(effective_ceiling as u64)
.saturating_mul(TYPICAL_CHUNK_BYTES)
.saturating_mul(BUDGET_HEADROOM_NUM)
.saturating_div(BUDGET_HEADROOM_DEN)
.clamp(
FETCH_INFLIGHT_BUDGET_FLOOR_BYTES,
FETCH_INFLIGHT_BUDGET_HARD_CAP_BYTES,
)
}
// ─── Adaptive-window tuning ────────────────────────────────────────────────────────────────────
// r3 retune: the first device run pinned the window at the floor on a 1 Gbps link. The controller
// shrank on throughput dips and latency jitter (normal CDN noise) as well as on errors, shrank a flat
// −6 on every single error, and only grew when throughput rose >10% in a 700 ms sample. The shrinks
// dominated, so the window ratcheted to the floor and stayed there. r3 makes growth eager and makes
// ERRORS the only shrink signal.
/// Max concurrent requests to any single CDN host. CDNs throttle/reset per host, so a big window
/// is reached by spreading across MORE hosts, never by piling onto one.
pub const PER_HOST_CAP: usize = 6;
/// Initial in-flight window. Slow-start doubles from here, so this only sets how fast the first
/// couple of probes get useful — the ceiling still comes from the tier × hosts clamp.
pub const BOOTSTRAP_WINDOW: usize = 8;
/// Hard safety ceiling on the window regardless of tier (also clamped to distinct-hosts × per-host).
pub const WINDOW_HARD_CAP: usize = 256;
/// The window never shrinks below this (keeps a little pipelining even on a rough link).
pub const WINDOW_MIN_FLOOR: usize = 2;
/// Additive growth step once slow-start has ended (after the first real back-off).
pub const WINDOW_STEP_UP: usize = 4;
/// Slow-start multiplier: until the first back-off the window DOUBLES each healthy probe, so a fast
/// link reaches a useful window in seconds instead of never.
pub const WINDOW_SLOW_START_FACTOR: usize = 2;
/// Proportional shrink on a real error signal (never a flat step: a flat −6 slammed a small window
/// straight to the floor). Always at least −1 while above the floor.
pub const WINDOW_SHRINK_FACTOR: f64 = 0.75;
/// How often the adaptive window re-evaluates. Long enough that one jittery sample cannot drive a
/// decision; decisions use the EWMA, not the raw sample.
pub const WINDOW_PROBE_INTERVAL_MS: u64 = 2_000;
/// After a back-off, hold the window (no growth) for this long — hysteresis against oscillation.
pub const WINDOW_COOLDOWN_MS: u64 = 3_000;
/// EWMA smoothing for the measured throughput samples (higher = more responsive).
pub const WINDOW_BPS_EWMA_ALPHA: f64 = 0.4;
/// Throughput above `best × (1 + this)` counts as a genuine improvement (keeps slow-start climbing).
pub const WINDOW_IMPROVE_EPS: f64 = 0.05;
/// Throughput below `best × (1 − this)` counts as falling — growth STOPS (hold). It never shrinks:
/// only real errors shrink.
pub const WINDOW_DECLINE_EPS: f64 = 0.15;
/// Error-rate thresholds over one probe window: above HIGH shrinks; growth needs at/below LOW
/// (in between = hold).
pub const WINDOW_ERR_RATE_HIGH: f64 = 0.10;
pub const WINDOW_ERR_RATE_LOW: f64 = 0.05;
/// Errors inside ONE probe interval that trigger an immediate (don't-wait-for-the-probe) shrink. A
/// single stray timeout no longer moves the window; a burst does. A 429 always shrinks immediately.
pub const WINDOW_ERR_BURST_IMMEDIATE: u32 = 3;
/// Consecutive non-improving probes still allowed to grow before declaring a plateau — throughput
/// lags a window change by a probe or two, so one flat sample must not stop the ramp.
pub const WINDOW_PLATEAU_PATIENCE: u32 = 2;
/// Once plateaued, re-arm the ramp this long later (doubling, capped) to re-probe for headroom if the
/// link improved. Bounded re-probing, not a creep: a weak link answers with errors and shrinks back.
pub const WINDOW_PLATEAU_REARM_MS: u64 = 30_000;
pub const WINDOW_PLATEAU_REARM_MAX_MS: u64 = 300_000;
/// Even when the window does not change, emit one `fetch-window` line every N probes so a stuck
/// window explains itself in the log (`reason=` says why it is not moving).
pub const WINDOW_LOG_EVERY_PROBES: u32 = 5;
/// Every Nth exploit dispatch is instead an exploration pick (epsilon-greedy ≈ 1/N) so a
/// demoted-but-recovered host is retried rather than starved forever.
pub const SERVER_EXPLORE_EVERY: u64 = 8;
/// Reservation for an item whose compressed size the manifest didn't carry (keeps the byte budget
/// honest at dispatch time).
pub const NOMINAL_CHUNK_RESERVE_BYTES: u64 = 1024 * 1024;
/// Extra host cooldown floor when a host answers 429 (rate-limited) — back off harder than a
/// one-off timeout.
pub const RATE_LIMIT_COOLDOWN_MS: u64 = 5_000;
/// Exponential back-off between attempts of one item: 0, 300, 600, 1200, 2400 ms … capped at 4 s.
pub fn retry_backoff_millis(attempt: u32) -> u64 {
if attempt == 0 {
return 0;
}
(300u64 << (attempt - 1)).min(4000)
}
/// Byte-budget admission test for the fetch→process channel. Always admits when nothing is in
/// flight (`in_flight == 0`) so a single item LARGER than the whole budget can never deadlock — it
/// simply waits until the channel drains, then goes through alone.
#[inline]
pub fn budget_admits(in_flight: u64, raw_len: u64, budget: u64) -> bool {
in_flight == 0 || in_flight.saturating_add(raw_len) <= budget
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn backoff_is_exponential_and_capped() {
assert_eq!(retry_backoff_millis(0), 0);
assert_eq!(retry_backoff_millis(1), 300);
assert_eq!(retry_backoff_millis(2), 600);
assert_eq!(retry_backoff_millis(4), 2400);
assert_eq!(retry_backoff_millis(5), 4000);
assert_eq!(retry_backoff_millis(12), 4000);
}
#[test]
fn budget_scales_between_floor_and_cap() {
assert_eq!(inflight_budget_bytes(1), FETCH_INFLIGHT_BUDGET_FLOOR_BYTES);
assert_eq!(inflight_budget_bytes(8), FETCH_INFLIGHT_BUDGET_FLOOR_BYTES);
assert_eq!(inflight_budget_bytes(32), 48 * 1024 * 1024);
assert_eq!(inflight_budget_bytes(1024), FETCH_INFLIGHT_BUDGET_HARD_CAP_BYTES);
}
#[test]
fn single_oversized_item_is_always_admitted() {
assert!(budget_admits(0, u64::MAX, 1));
assert!(budget_admits(10, 5, 15));
assert!(!budget_admits(10, 6, 15));
}
}
+121
View File
@@ -0,0 +1,121 @@
//! GOG gen2 chunk download engine — the Rust replacement for the byte-fetching inner loop of
//! `GogDownloadManager.java` (`assembleDepotFile` / `fetchChunkVerified` driven from the
//! per-file thread pool). Everything around that loop — token/builds/manifest head, base-product
//! + language depot selection, secure-link resolution, disk guard, `_gog_manifest.json` marker,
//! prefs, exe resolution, DLC markers, redist orchestration — stays in Java and is untouched.
//!
//! Parity contract: Bannerlator's `docs/RUST_GOG_PARITY.md`. The adapter is fed the SAME inflated depot-manifest
//! JSON strings Java already parsed, the SAME resolved CDN base (secure link) and the SAME
//! install dir, and produces the SAME on-disk layout (`<file>.bhtmp` staging, atomic rename,
//! size+MD5-verified finals) so a download can be resumed by either engine.
//!
//! Layout:
//! - [`plan`] — depot-manifest JSON → file/chunk plan (mirrors `parseDepotManifest`), CDN path
//! and chunk-URL builders, largest-first ordering.
//! - [`engine`] — resume/skip pass (mirrors `fileVerified`), `FetchItem` list, the `FetchSink`
//! (compressed size/MD5 → inflate → decompressed size/MD5 → positioned write →
//! whole-file size/MD5 → rename), summary logging.
//! - [`jni`] — `GogNative` JNI exports (`nativeStart` / `nativeCancel` / `nativeRelease`).
pub mod engine;
pub mod jni;
pub mod plan;
use std::fs::File;
use std::io::Read;
use std::path::Path;
/// Lowercase hex of `bytes` (matches `GogDownloadManager.toHex`).
pub fn to_hex(bytes: &[u8]) -> String {
const DIGITS: &[u8; 16] = b"0123456789abcdef";
let mut out = String::with_capacity(bytes.len() * 2);
for b in bytes {
out.push(DIGITS[(b >> 4) as usize] as char);
out.push(DIGITS[(b & 0x0f) as usize] as char);
}
out
}
/// Lowercase hex MD5 of an in-memory buffer (`GogDownloadManager.md5Hex`).
pub fn md5_hex(data: &[u8]) -> String {
let mut hasher = crate::md5_small::Md5::new();
hasher.update(data);
to_hex(&hasher.finalize())
}
/// Streaming lowercase hex MD5 of a file, never buffering the whole file
/// (`GogDownloadManager.md5HexFile`). `None` when the file cannot be read.
pub fn md5_hex_file(path: &Path) -> Option<String> {
let mut file = File::open(path).ok()?;
let mut hasher = crate::md5_small::Md5::new();
let mut buf = vec![0u8; 128 * 1024];
loop {
let n = file.read(&mut buf).ok()?;
if n == 0 {
break;
}
hasher.update(&buf[..n]);
}
Some(to_hex(&hasher.finalize()))
}
/// zlib-inflate a chunk body (`GogDownloadManager.inflateZlib`): `None` when the body is not a
/// zlib stream (first byte != 0x78) or fails to inflate — the caller then treats the body as a
/// stored (non-compressed) chunk, exactly like Java's `if (inflated == null) inflated = raw`.
/// A truncated stream yields the partial output (Java's loop breaks on `n == 0` the same way);
/// the decompressed-size check that follows rejects it.
pub fn inflate_zlib(data: &[u8]) -> Option<Vec<u8>> {
if data.len() < 2 || data[0] != 0x78 {
return None;
}
let mut out = Vec::with_capacity(data.len().saturating_mul(3));
let mut decoder = flate2::read::ZlibDecoder::new(data);
match decoder.read_to_end(&mut out) {
Ok(_) => Some(out),
Err(_) => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn hex_is_lowercase_and_padded() {
assert_eq!(to_hex(&[0x00, 0x0a, 0xff, 0x10]), "000aff10");
}
#[test]
fn inflate_roundtrip_and_stored_fallback() {
use flate2::write::ZlibEncoder;
use flate2::Compression;
use std::io::Write;
let payload = b"hello gog chunk payload".repeat(64);
let mut enc = ZlibEncoder::new(Vec::new(), Compression::default());
enc.write_all(&payload).unwrap();
let compressed = enc.finish().unwrap();
assert_eq!(compressed[0], 0x78);
assert_eq!(inflate_zlib(&compressed).unwrap(), payload);
// Not zlib → None (stored chunk path).
assert!(inflate_zlib(b"\x1f\x8bnot zlib").is_none());
assert!(inflate_zlib(b"\x78").is_none());
// Truncated zlib → partial output, like Java's `while (!inf.finished()) { if (n == 0) break; }`
// loop; the decompressed-size check downstream rejects it either way.
let mut truncated = compressed.clone();
let mid = truncated.len() / 2;
truncated.truncate(mid);
assert!(inflate_zlib(&truncated).map(|out| out != payload).unwrap_or(true));
// Corrupt zlib body → None (Java: DataFormatException → null → raw fallback).
let mut corrupt = compressed.clone();
for b in corrupt.iter_mut().skip(2) {
*b = !*b;
}
assert!(inflate_zlib(&corrupt).is_none() || inflate_zlib(&corrupt).unwrap() != payload);
}
#[test]
fn md5_hex_matches_rfc1321_vector() {
assert_eq!(md5_hex(b""), "d41d8cd98f00b204e9800998ecf8427e");
assert_eq!(md5_hex(b"abc"), "900150983cd24fb0d6963f7d28e17f72");
}
}
File diff suppressed because it is too large. Load diff
+303
View File
@@ -0,0 +1,303 @@
//! JNI facade for the GOG Rust download engine — the native half of
//! `com.droiddeck.launcher.stores.gog.GogNative` (Kotlin). Symbol names are bound to that
//! package/class; the `@JvmStatic external` declarations there are the other side of this file.
//!
//! Shape mirrors the Steam download JNI (`jni.rs` `nativeDownloadApp`): `nativeStart` validates
//! its inputs, takes a `GlobalRef` on the listener, spawns the worker thread and returns a handle
//! immediately; callbacks are delivered from the worker / process-pool threads through a
//! `JavaVM` attach; `nativeCancel` flips an `AtomicBool`; `nativeRelease` drops the Java side's
//! reference to the handle (the worker keeps its own until it exits).
//!
//! Log tag `GogNative`: every engine log line goes to logcat here AND to the listener's `onLog`
//! (which the Kotlin manager folds into its own debug log).
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
use jni::objects::{GlobalRef, JClass, JObject, JObjectArray, JString, JValue};
use jni::sys::{jboolean, jint, jlong, JNI_FALSE, JNI_TRUE};
use jni::{JNIEnv, JavaVM};
use super::engine::{self, GogEvents, GogRequest, GogRunResult, PlanKind};
pub const LOG_TAG: &str = "GogNative";
#[cfg(target_os = "android")]
#[link(name = "log")]
unsafe extern "C" {
fn __android_log_write(prio: i32, tag: *const i8, text: *const i8) -> i32;
}
/// `android.util.Log.i(LOG_TAG, message)`; a no-op on non-Android hosts (cargo test).
pub fn android_log(message: &str) {
#[cfg(target_os = "android")]
{
use std::ffi::CString;
let Ok(tag) = CString::new(LOG_TAG) else {
return;
};
let sanitized = message.replace('\0', " ");
let Ok(text) = CString::new(sanitized) else {
return;
};
unsafe {
let _ = __android_log_write(4, tag.as_ptr().cast(), text.as_ptr().cast());
}
}
#[cfg(not(target_os = "android"))]
{
let _ = message;
}
}
/// One in-flight run. The Java side owns a `Box<Arc<..>>` (the `jlong`); the worker thread owns a
/// second `Arc` so `nativeRelease` is safe at any time after `nativeStart` returned.
pub struct GogRunHandle {
pub cancel: AtomicBool,
}
struct JniEvents {
vm: JavaVM,
listener: GlobalRef,
}
impl JniEvents {
fn with_env(&self, f: impl FnOnce(&mut JNIEnv)) {
let Ok(mut env) = self.vm.attach_current_thread_as_daemon() else {
return;
};
f(&mut env);
}
fn clear_exception(env: &mut JNIEnv) {
if env.exception_check().unwrap_or(false) {
let _ = env.exception_clear();
}
}
fn complete(&self, result: &GogRunResult) {
self.with_env(|env| {
let Ok(error) = env.new_string(&result.error) else {
return;
};
let error = JObject::from(error);
let _ = env.call_method(
self.listener.as_obj(),
"onComplete",
"(ZZZLjava/lang/String;JI)V",
&[
JValue::Bool(if result.success { JNI_TRUE } else { JNI_FALSE }),
JValue::Bool(if result.cancelled { JNI_TRUE } else { JNI_FALSE }),
JValue::Bool(if result.link_expiry { JNI_TRUE } else { JNI_FALSE }),
JValue::Object(&error),
JValue::Long(result.bytes_written as jlong),
JValue::Int(result.files_done as jint),
],
);
Self::clear_exception(env);
let _ = env.delete_local_ref(error);
});
}
}
impl GogEvents for JniEvents {
fn on_file_done(
&self,
rel_path: &str,
file_bytes: u64,
verified: bool,
files_done: u32,
files_total: u32,
bytes_done: u64,
bytes_total: u64,
) {
self.with_env(|env| {
let Ok(path) = env.new_string(rel_path) else {
return;
};
let path = JObject::from(path);
let _ = env.call_method(
self.listener.as_obj(),
"onProgress",
"(JJIILjava/lang/String;JZ)V",
&[
JValue::Long(bytes_done as jlong),
JValue::Long(bytes_total as jlong),
JValue::Int(files_done as jint),
JValue::Int(files_total as jint),
JValue::Object(&path),
JValue::Long(file_bytes as jlong),
JValue::Bool(if verified { JNI_TRUE } else { JNI_FALSE }),
],
);
Self::clear_exception(env);
let _ = env.delete_local_ref(path);
});
}
fn on_log(&self, line: &str) {
android_log(line);
self.with_env(|env| {
let Ok(text) = env.new_string(line) else {
return;
};
let text = JObject::from(text);
let _ = env.call_method(
self.listener.as_obj(),
"onLog",
"(Ljava/lang/String;)V",
&[JValue::Object(&text)],
);
Self::clear_exception(env);
let _ = env.delete_local_ref(text);
});
}
}
fn jstring_to_string(env: &mut JNIEnv, value: &JString) -> Option<String> {
env.get_string(value)
.ok()
.map(|s| s.to_string_lossy().into_owned())
}
fn string_array_to_vec(env: &mut JNIEnv, array: &JObjectArray) -> Vec<String> {
if array.is_null() {
return Vec::new();
}
let len = env.get_array_length(array).unwrap_or(0);
let mut out = Vec::with_capacity(len.max(0) as usize);
for i in 0..len {
let Ok(element) = env.get_object_array_element(array, i) else {
continue;
};
if element.is_null() {
continue;
}
let text = JString::from(element);
if let Some(s) = jstring_to_string(env, &text) {
out.push(s);
}
let _ = env.delete_local_ref(text);
}
out
}
unsafe fn handle_ref(handle: jlong) -> Option<&'static Arc<GogRunHandle>> {
let ptr = handle as *const Arc<GogRunHandle>;
if ptr.is_null() {
None
} else {
Some(unsafe { &*ptr })
}
}
/// Symbol-binding probe (see `GogNative.isAvailable`). Returns 1.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeProbe(
_env: JNIEnv,
_class: JClass,
) -> jint {
1
}
/// Starts one download loop on a worker thread and returns a handle (0 = could not start; the
/// listener then receives NO callbacks). Inputs are exactly what `GogDownloadManager` holds after
/// manifest + secure-link resolution — see `GogRequest`.
#[no_mangle]
#[allow(clippy::too_many_arguments)]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeStart(
mut env: JNIEnv,
_class: JClass,
kind: jint,
depot_manifests: JObjectArray,
cdn_base: JString,
install_dir: JString,
skip_paths: JObjectArray,
ca_bundle_path: JString,
max_workers: jint,
process_workers: jint,
sort_largest_first: jboolean,
label: JString,
listener: JObject,
) -> jlong {
if listener.is_null() {
android_log("nativeStart: null listener");
return 0;
}
let Ok(vm) = env.get_java_vm() else {
android_log("nativeStart: JavaVM unavailable");
return 0;
};
let Ok(listener) = env.new_global_ref(&listener) else {
android_log("nativeStart: listener ref failed");
return 0;
};
let request = GogRequest {
kind: PlanKind::from_i32(kind),
depot_manifests: string_array_to_vec(&mut env, &depot_manifests),
cdn_base: jstring_to_string(&mut env, &cdn_base).unwrap_or_default(),
install_dir: jstring_to_string(&mut env, &install_dir).unwrap_or_default(),
skip_paths: string_array_to_vec(&mut env, &skip_paths),
ca_bundle_path: jstring_to_string(&mut env, &ca_bundle_path).unwrap_or_default(),
max_workers: max_workers.max(1) as usize,
process_workers: process_workers.max(1) as usize,
sort_largest_first: sort_largest_first != JNI_FALSE,
label: jstring_to_string(&mut env, &label).unwrap_or_else(|| "gog".to_string()),
};
let needs_base = request.kind == PlanKind::Gen2Chunks;
if (needs_base && request.cdn_base.is_empty())
|| request.install_dir.is_empty()
|| request.depot_manifests.is_empty()
{
android_log("nativeStart: invalid request (empty cdn base / install dir / manifests)");
return 0;
}
let handle = Arc::new(GogRunHandle {
cancel: AtomicBool::new(false),
});
let worker = Arc::clone(&handle);
let spawned = thread::Builder::new()
.name("gog-dl".to_string())
.spawn(move || {
let events = JniEvents { vm, listener };
let result = engine::run(&request, &worker.cancel, &events);
events.complete(&result);
});
if spawned.is_err() {
android_log("nativeStart: worker thread spawn failed");
return 0;
}
Box::into_raw(Box::new(handle)) as jlong
}
/// Requests cancellation; the run stops at its next scheduling point and `onComplete` fires with
/// `cancelled = true`. Idempotent.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeCancel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if let Some(run) = unsafe { handle_ref(handle) } {
run.cancel.store(true, Ordering::Relaxed);
}
}
/// Drops the Java side's reference. Call once, after `onComplete` (or after cancelling — the worker
/// keeps the run alive until it exits either way).
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_gog_GogNative_nativeRelease(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
let ptr = handle as *mut Arc<GogRunHandle>;
if ptr.is_null() {
return;
}
unsafe {
drop(Box::from_raw(ptr));
}
}
+370
View File
@@ -0,0 +1,370 @@
//! Depot-manifest JSON → download plan. Mirrors `GogDownloadManager.parseDepotManifest` rule for
//! rule (see Bannerlator's `docs/RUST_GOG_PARITY.md` §3):
//! - root `depot.items[]`; `path` with `\` → `/` and one leading `/` stripped;
//! - a file is kept only when `path` is non-empty AND `chunks[]` is non-empty;
//! - per chunk: CDN hash = `compressedMd5` when non-empty, else `md5`; a chunk with neither is
//! dropped; `compressedSize` / `size` default 0;
//! - file `md5` kept when non-empty; `total_size` = Σ chunk `size` (decompressed);
//! - a file whose chunks all dropped is not added.
//! `org.json` coercion (`optString` turns numbers/bools into their text, `optLong` parses numeric
//! strings) is reproduced by [`opt_string`] / [`opt_u64`].
//!
//! Additionally each chunk carries `offset` = prefix-sum of the preceding chunks' decompressed
//! sizes — Java appends sequentially so it never needs one; the Rust sink writes chunks at that
//! offset as they arrive so the final bytes are identical.
use serde_json::Value;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ChunkRef {
/// CDN-path hash (`compressedMd5` preferred, else `md5`).
pub hash: String,
/// Verify compressed bytes before inflate (may be empty).
pub compressed_md5: String,
/// Verify decompressed bytes after inflate (may be empty).
pub md5: String,
/// Expected compressed size, 0 if absent.
pub compressed_size: u64,
/// Expected decompressed size, 0 if absent.
pub size: u64,
/// Decompressed offset inside the file (prefix-sum of preceding `size`s).
pub offset: u64,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PlannedFile {
/// Install-dir-relative path, forward slashes, no leading slash.
pub relative_path: String,
/// Whole-file MD5 from the manifest ("" when absent).
pub md5: String,
/// Σ chunk decompressed sizes (0 if unknown).
pub total_size: u64,
pub chunks: Vec<ChunkRef>,
}
/// `JSONObject.optString(key, "")` semantics: strings as-is, numbers/bools rendered, null/absent
/// → "". (`org.json` renders `JSONObject.NULL` as "null"; GOG manifests never carry JSON nulls in
/// these fields, and treating one as absent is the only safe reading.)
pub fn opt_string(obj: &Value, key: &str) -> String {
match obj.get(key) {
Some(Value::String(s)) => s.clone(),
Some(Value::Number(n)) => n.to_string(),
Some(Value::Bool(b)) => b.to_string(),
_ => String::new(),
}
}
/// `JSONObject.optLong(key, 0)` semantics for a size field (numbers, or numeric strings; a
/// negative/float value that Java would truncate is clamped to 0 — sizes are never negative).
pub fn opt_u64(obj: &Value, key: &str) -> u64 {
match obj.get(key) {
Some(Value::Number(n)) => n
.as_u64()
.or_else(|| n.as_i64().map(|v| v.max(0) as u64))
.or_else(|| n.as_f64().map(|v| if v > 0.0 { v as u64 } else { 0 }))
.unwrap_or(0),
Some(Value::String(s)) => s.trim().parse::<i64>().map(|v| v.max(0) as u64).unwrap_or(0),
_ => 0,
}
}
/// Parses one inflated gen2 depot manifest and appends its files to `out`
/// (`GogDownloadManager.parseDepotManifest`). Malformed JSON adds nothing, like Java's catch-all.
pub fn parse_depot_manifest(json: &str, out: &mut Vec<PlannedFile>) {
let Ok(root) = serde_json::from_str::<Value>(json) else {
return;
};
let Some(items) = root
.get("depot")
.and_then(|d| d.get("items"))
.and_then(Value::as_array)
else {
return;
};
for entry in items {
if !entry.is_object() {
continue;
}
let mut path = opt_string(entry, "path").replace('\\', "/");
if let Some(stripped) = path.strip_prefix('/') {
path = stripped.to_string();
}
let chunks = entry.get("chunks").and_then(Value::as_array);
let Some(chunks) = chunks else {
continue;
};
if path.is_empty() || chunks.is_empty() {
continue;
}
let file_md5 = opt_string(entry, "md5");
let mut planned = PlannedFile {
relative_path: path,
md5: file_md5,
total_size: 0,
chunks: Vec::with_capacity(chunks.len()),
};
let mut offset = 0u64;
for chunk in chunks {
if !chunk.is_object() {
continue;
}
let compressed_md5 = opt_string(chunk, "compressedMd5");
let dec_md5 = opt_string(chunk, "md5");
let compressed_size = opt_u64(chunk, "compressedSize");
let dec_size = opt_u64(chunk, "size");
let hash = if !compressed_md5.is_empty() {
compressed_md5.clone()
} else {
dec_md5.clone()
};
if hash.is_empty() {
continue;
}
planned.chunks.push(ChunkRef {
hash,
compressed_md5,
md5: dec_md5,
compressed_size,
size: dec_size,
offset,
});
offset = offset.saturating_add(dec_size);
}
planned.total_size = offset;
if !planned.chunks.is_empty() {
out.push(planned);
}
}
}
/// One gen1 file: a byte range of a (pre-signed) depot blob (`GogDownloadManager.Gen1File`).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Gen1File {
/// Install-dir-relative path, used verbatim (Java's gen1 path does not normalise slashes).
pub path: String,
pub url: String,
pub offset: u64,
pub size: u64,
}
/// Parses a gen1 build manifest (`runGen1`): every `depot[]` entry that is not `support: true`,
/// its `files[]` → (`path`, `url`, `offset` default 0, `size` default 0); an entry with
/// `size == 0` is dropped (Java's `path == null || url == null` checks never fire — `optString`
/// returns "" — so `size == 0` is the only effective filter, mirrored here).
pub fn parse_gen1_manifest(json: &str, out: &mut Vec<Gen1File>) {
let Ok(root) = serde_json::from_str::<Value>(json) else {
return;
};
let Some(depots) = root.get("depot").and_then(Value::as_array) else {
return;
};
for depot in depots {
if !depot.is_object() {
continue;
}
if depot.get("support").and_then(Value::as_bool).unwrap_or(false) {
continue;
}
let Some(files) = depot.get("files").and_then(Value::as_array) else {
continue;
};
for f in files {
if !f.is_object() {
continue;
}
let path = opt_string(f, "path");
let offset = opt_u64(f, "offset");
let size = opt_u64(f, "size");
let url = opt_string(f, "url");
if size == 0 {
continue;
}
out.push(Gen1File { path, url, offset, size });
}
}
}
/// Parses every manifest in order (the order Java fetched the depots) and optionally applies the
/// base-install largest-first (LPT) ordering — a STABLE sort by descending `total_size`, exactly
/// `Collections.sort(files, (a, b) -> Long.compare(b.totalSize, a.totalSize))`. The DLC and
/// dependency paths in Java do not sort, so they pass `sort_largest_first = false`.
pub fn build_plan(manifests: &[String], sort_largest_first: bool) -> Vec<PlannedFile> {
let mut files = Vec::new();
for manifest in manifests {
parse_depot_manifest(manifest, &mut files);
}
if sort_largest_first {
files.sort_by(|a, b| b.total_size.cmp(&a.total_size));
}
files
}
/// `GogDownloadManager.buildCdnPath`: `"ab/cd/abcdef..."`. A hash shorter than 4 chars would
/// throw in Java (and fail that file); here it is returned unchanged and fails at fetch time.
pub fn build_cdn_path(hash: &str) -> String {
if !hash.is_char_boundary(2) || !hash.is_char_boundary(4) || hash.len() < 4 {
return hash.to_string();
}
format!("{}/{}/{}", &hash[0..2], &hash[2..4], hash)
}
/// `GogDownloadManager.buildChunkUrl`: appends the chunk path BEFORE any query string so the
/// secure-link token survives.
pub fn build_chunk_url(base: &str, chunk_path: &str) -> String {
match base.find('?') {
Some(q) => format!("{}/{}{}", &base[..q], chunk_path, &base[q..]),
None => format!("{base}/{chunk_path}"),
}
}
/// Per-host key for the fetch core's host semaphores: `scheme://host[:port]` of `url` (lowercased
/// host). Java has one resolved CDN base per run, so the host list has exactly one entry.
pub fn host_key(url: &str) -> String {
let (scheme, rest) = match url.find("://") {
Some(i) => (&url[..i], &url[i + 3..]),
None => ("https", url),
};
let end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
format!("{}://{}", scheme.to_ascii_lowercase(), rest[..end].to_ascii_lowercase())
}
#[cfg(test)]
mod tests {
use super::*;
const MANIFEST: &str = r#"{
"depot": { "items": [
{ "path": "\\bin\\game.exe", "md5": "AABBCCDDEEFF00112233445566778899",
"chunks": [
{ "compressedMd5": "c1c1c1c1", "md5": "d1d1d1d1", "compressedSize": 100, "size": 250 },
{ "compressedMd5": "c2c2c2c2", "md5": "d2d2d2d2", "compressedSize": "50", "size": 75 }
] },
{ "path": "/data/pak0.pak",
"chunks": [
{ "md5": "onlydec1", "size": 1000 },
{ "compressedSize": 5, "size": 5 }
] },
{ "path": "empty.txt", "chunks": [] },
{ "path": "", "chunks": [ { "compressedMd5": "zz", "size": 1 } ] },
{ "path": "nohash.bin", "chunks": [ { "size": 9 } ] },
{ "path": "dir", "type": "DepotDirectory" },
"not-an-object"
] } }"#;
#[test]
fn parse_mirrors_java_rules() {
let mut files = Vec::new();
parse_depot_manifest(MANIFEST, &mut files);
assert_eq!(files.len(), 2, "empty-chunk, empty-path, no-hash and directory entries drop");
let exe = &files[0];
assert_eq!(exe.relative_path, "bin/game.exe");
assert_eq!(exe.md5, "AABBCCDDEEFF00112233445566778899");
assert_eq!(exe.total_size, 325);
assert_eq!(exe.chunks.len(), 2);
assert_eq!(exe.chunks[0].hash, "c1c1c1c1");
assert_eq!(exe.chunks[0].offset, 0);
assert_eq!(exe.chunks[1].hash, "c2c2c2c2");
assert_eq!(exe.chunks[1].compressed_size, 50, "numeric string coerces like optLong");
assert_eq!(exe.chunks[1].offset, 250);
let pak = &files[1];
assert_eq!(pak.relative_path, "data/pak0.pak");
assert_eq!(pak.md5, "", "absent file md5 → empty (no whole-file verify)");
assert_eq!(pak.chunks.len(), 1, "chunk without any hash is dropped");
assert_eq!(pak.chunks[0].hash, "onlydec1", "hash falls back to md5");
assert_eq!(pak.chunks[0].compressed_md5, "");
assert_eq!(pak.total_size, 1000);
}
#[test]
fn malformed_manifest_adds_nothing() {
let mut files = Vec::new();
parse_depot_manifest("{not json", &mut files);
parse_depot_manifest(r#"{"depot": {}}"#, &mut files);
parse_depot_manifest(r#"{"items": []}"#, &mut files);
assert!(files.is_empty());
}
#[test]
fn plan_orders_largest_first_stably_only_when_asked() {
let m1 = r#"{"depot":{"items":[
{"path":"a","chunks":[{"md5":"a1","size":10}]},
{"path":"b","chunks":[{"md5":"b1","size":30}]}]}}"#
.to_string();
let m2 = r#"{"depot":{"items":[
{"path":"c","chunks":[{"md5":"c1","size":30}]},
{"path":"d","chunks":[{"md5":"d1","size":20}]}]}}"#
.to_string();
let unsorted = build_plan(&[m1.clone(), m2.clone()], false);
let order: Vec<&str> = unsorted.iter().map(|f| f.relative_path.as_str()).collect();
assert_eq!(order, ["a", "b", "c", "d"], "depot order, file order within depot");
let sorted = build_plan(&[m1, m2], true);
let order: Vec<&str> = sorted.iter().map(|f| f.relative_path.as_str()).collect();
assert_eq!(order, ["b", "c", "d", "a"], "stable: b (30) before c (30) keeps source order");
}
#[test]
fn gen1_parse_mirrors_run_gen1() {
let json = r#"{"installDirectory":"Game","depot":[
{"support":true,"files":[{"path":"support.bin","url":"u","offset":0,"size":5}]},
{"files":[
{"path":"a.bin","url":"https://cdn.gog.com/blob?t=1","offset":10,"size":20},
{"path":"zero.bin","url":"u","offset":0,"size":0},
{"path":"b.bin","url":"https://cdn.gog.com/blob?t=1","size":"7"},
"junk"
]},
{"nofiles":true}
]}"#;
let mut files = Vec::new();
parse_gen1_manifest(json, &mut files);
assert_eq!(
files,
vec![
Gen1File { path: "a.bin".into(), url: "https://cdn.gog.com/blob?t=1".into(), offset: 10, size: 20 },
Gen1File { path: "b.bin".into(), url: "https://cdn.gog.com/blob?t=1".into(), offset: 0, size: 7 },
]
);
let mut none = Vec::new();
parse_gen1_manifest(r#"{"depots":[]}"#, &mut none);
assert!(none.is_empty());
}
#[test]
fn cdn_path_and_chunk_url() {
assert_eq!(build_cdn_path("abcdef0123"), "ab/cd/abcdef0123");
assert_eq!(build_cdn_path("abc"), "abc");
assert_eq!(
build_chunk_url("https://cdn.gog.com/content-system/v2/store/1?token=x&y=1", "ab/cd/abcdef"),
"https://cdn.gog.com/content-system/v2/store/1/ab/cd/abcdef?token=x&y=1"
);
assert_eq!(
build_chunk_url("https://gog-cdn-fastly.gog.com/content-system/v2/dependencies/store", "ab/cd/abcdef"),
"https://gog-cdn-fastly.gog.com/content-system/v2/dependencies/store/ab/cd/abcdef"
);
}
#[test]
fn host_key_strips_path_and_query() {
assert_eq!(host_key("https://CDN.gog.com/a/b?c=1"), "https://cdn.gog.com");
assert_eq!(host_key("http://host:8080/x"), "http://host:8080");
assert_eq!(host_key("nohost"), "https://nohost");
}
#[test]
fn opt_helpers_coerce_like_org_json() {
let v: Value = serde_json::from_str(r#"{"n": 12, "s": "34", "b": true, "z": null, "f": 2.5, "neg": -4}"#).unwrap();
assert_eq!(opt_string(&v, "n"), "12");
assert_eq!(opt_string(&v, "s"), "34");
assert_eq!(opt_string(&v, "b"), "true");
assert_eq!(opt_string(&v, "z"), "");
assert_eq!(opt_string(&v, "missing"), "");
assert_eq!(opt_u64(&v, "n"), 12);
assert_eq!(opt_u64(&v, "s"), 34);
assert_eq!(opt_u64(&v, "f"), 2);
assert_eq!(opt_u64(&v, "neg"), 0);
assert_eq!(opt_u64(&v, "b"), 0);
assert_eq!(opt_u64(&v, "missing"), 0);
}
}
+28
View File
@@ -0,0 +1,28 @@
//! DroidDeck's native store download engines (`libdroiddeckstores.so`).
//!
//! The GOG, Epic Games Store and Amazon Games download loops, ported from Bannerlator's
//! `bl-steam-client` crate (GPL-3.0-or-later, itself derived from WinNative's `wn-steam-client`).
//! Only the store engines and the fetch core they run on came across: DroidDeck's Steam client is
//! the real one, running in the Linux runtime, so none of the Steam CM / depot code is here.
//!
//! Each store module owns its manifest → plan → `FetchItem` mapping, its `FetchSink` (inflate +
//! hash + write in the exact output layout of the Kotlin manager that drives it) and its JNI facade.
//! The JNI contract — classes, method names, signatures, threading — is written down in `JNI.md`
//! next to `Cargo.toml`; the Kotlin side mirrors it exactly.
//!
//! What a run looks like: Kotlin hands over a manifest it already parsed (or, for Amazon, the
//! plan it built from one), the resolved CDN base(s) and the install directory, calls `nativeStart`
//! and gets a handle back at once. Everything after that happens on native threads: the fetch core
//! keeps an adaptive window of HTTP requests in flight on one tokio runtime, hands bodies to a sync
//! process pool that inflates, hashes and writes them, and reports progress through the listener
//! object Kotlin passed in. `nativeCancel` flips a flag; `onComplete` fires exactly once.
#![allow(clippy::missing_safety_doc, clippy::result_large_err)]
pub mod amazon;
pub mod epic;
pub mod fetch_core;
pub mod fetch_tuning;
pub mod gog;
pub mod md5_small;
pub mod stores_jni;
+197
View File
@@ -0,0 +1,197 @@
//! Minimal MD5 (RFC 1321) for the store download engines (GOG chunk / file verification).
//!
//! In-crate on purpose: the crate is built `--locked` in CI with no way to add a dependency from
//! this environment, and the algorithm is 100 lines. Not for anything security-sensitive — it only
//! mirrors the checksum the Java managers already compare against.
/// One-shot digest.
pub fn md5(data: &[u8]) -> [u8; 16] {
let mut h = Md5::new();
h.update(data);
h.finalize()
}
/// Lower-case hex of a 16-byte digest.
pub fn md5_hex(digest: &[u8; 16]) -> String {
const HEX: &[u8; 16] = b"0123456789abcdef";
let mut out = String::with_capacity(32);
for b in digest {
out.push(HEX[(b >> 4) as usize] as char);
out.push(HEX[(b & 0x0f) as usize] as char);
}
out
}
/// Streaming MD5: `update` any number of times, then `finalize`.
#[derive(Clone, Debug)]
pub struct Md5 {
state: [u32; 4],
buffer: [u8; 64],
buffer_len: usize,
total_len: u64,
}
impl Default for Md5 {
fn default() -> Self {
Self::new()
}
}
const S: [u32; 64] = [
7, 12, 17, 22, 7, 12, 17, 22, 7, 12, 17, 22, 7, 12, 17, 22, 5, 9, 14, 20, 5, 9, 14, 20, 5, 9,
14, 20, 5, 9, 14, 20, 4, 11, 16, 23, 4, 11, 16, 23, 4, 11, 16, 23, 4, 11, 16, 23, 6, 10, 15,
21, 6, 10, 15, 21, 6, 10, 15, 21, 6, 10, 15, 21,
];
const K: [u32; 64] = [
0xd76aa478, 0xe8c7b756, 0x242070db, 0xc1bdceee, 0xf57c0faf, 0x4787c62a, 0xa8304613, 0xfd469501,
0x698098d8, 0x8b44f7af, 0xffff5bb1, 0x895cd7be, 0x6b901122, 0xfd987193, 0xa679438e, 0x49b40821,
0xf61e2562, 0xc040b340, 0x265e5a51, 0xe9b6c7aa, 0xd62f105d, 0x02441453, 0xd8a1e681, 0xe7d3fbc8,
0x21e1cde6, 0xc33707d6, 0xf4d50d87, 0x455a14ed, 0xa9e3e905, 0xfcefa3f8, 0x676f02d9, 0x8d2a4c8a,
0xfffa3942, 0x8771f681, 0x6d9d6122, 0xfde5380c, 0xa4beea44, 0x4bdecfa9, 0xf6bb4b60, 0xbebfbc70,
0x289b7ec6, 0xeaa127fa, 0xd4ef3085, 0x04881d05, 0xd9d4d039, 0xe6db99e5, 0x1fa27cf8, 0xc4ac5665,
0xf4292244, 0x432aff97, 0xab9423a7, 0xfc93a039, 0x655b59c3, 0x8f0ccc92, 0xffeff47d, 0x85845dd1,
0x6fa87e4f, 0xfe2ce6e0, 0xa3014314, 0x4e0811a1, 0xf7537e82, 0xbd3af235, 0x2ad7d2bb, 0xeb86d391,
];
impl Md5 {
pub fn new() -> Self {
Self {
state: [0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476],
buffer: [0u8; 64],
buffer_len: 0,
total_len: 0,
}
}
pub fn update(&mut self, mut data: &[u8]) {
self.total_len = self.total_len.wrapping_add(data.len() as u64);
if self.buffer_len > 0 {
let take = (64 - self.buffer_len).min(data.len());
self.buffer[self.buffer_len..self.buffer_len + take].copy_from_slice(&data[..take]);
self.buffer_len += take;
data = &data[take..];
if self.buffer_len == 64 {
let block = self.buffer;
self.compress(&block);
self.buffer_len = 0;
}
}
while data.len() >= 64 {
let mut block = [0u8; 64];
block.copy_from_slice(&data[..64]);
self.compress(&block);
data = &data[64..];
}
if !data.is_empty() {
self.buffer[..data.len()].copy_from_slice(data);
self.buffer_len = data.len();
}
}
pub fn finalize(mut self) -> [u8; 16] {
let bit_len = self.total_len.wrapping_mul(8);
let mut pad = [0u8; 72];
pad[0] = 0x80;
// Pad to 56 mod 64, then append the 64-bit little-endian bit length.
let pad_len = if self.buffer_len < 56 {
56 - self.buffer_len
} else {
120 - self.buffer_len
};
let total_len_before = self.total_len;
self.update(&pad[..pad_len]);
self.update(&bit_len.to_le_bytes());
debug_assert_eq!(self.buffer_len, 0);
self.total_len = total_len_before;
let mut out = [0u8; 16];
for (i, word) in self.state.iter().enumerate() {
out[i * 4..i * 4 + 4].copy_from_slice(&word.to_le_bytes());
}
out
}
fn compress(&mut self, block: &[u8; 64]) {
let mut m = [0u32; 16];
for (i, word) in m.iter_mut().enumerate() {
*word = u32::from_le_bytes([
block[i * 4],
block[i * 4 + 1],
block[i * 4 + 2],
block[i * 4 + 3],
]);
}
let [mut a, mut b, mut c, mut d] = self.state;
for i in 0..64 {
let (f, g) = match i / 16 {
0 => ((b & c) | (!b & d), i),
1 => ((d & b) | (!d & c), (5 * i + 1) % 16),
2 => (b ^ c ^ d, (3 * i + 5) % 16),
_ => (c ^ (b | !d), (7 * i) % 16),
};
let f = f.wrapping_add(a).wrapping_add(K[i]).wrapping_add(m[g]);
a = d;
d = c;
c = b;
b = b.wrapping_add(f.rotate_left(S[i]));
}
self.state[0] = self.state[0].wrapping_add(a);
self.state[1] = self.state[1].wrapping_add(b);
self.state[2] = self.state[2].wrapping_add(c);
self.state[3] = self.state[3].wrapping_add(d);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn hex(data: &[u8]) -> String {
md5_hex(&md5(data))
}
#[test]
fn rfc_1321_vectors() {
assert_eq!(hex(b""), "d41d8cd98f00b204e9800998ecf8427e");
assert_eq!(hex(b"a"), "0cc175b9c0f1b6a831c399e269772661");
assert_eq!(hex(b"abc"), "900150983cd24fb0d6963f7d28e17f72");
assert_eq!(hex(b"message digest"), "f96b697d7cb7938d525a2f31aaf161d0");
assert_eq!(
hex(b"abcdefghijklmnopqrstuvwxyz"),
"c3fcd3d76192e4007dfb496cca67e13b"
);
assert_eq!(
hex(b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"),
"d174ab98d277d9f5a5611c2c9f419d9f"
);
assert_eq!(
hex(b"12345678901234567890123456789012345678901234567890123456789012345678901234567890"),
"57edf4a22be3c955ac49da2e2107b67a"
);
}
#[test]
fn streaming_matches_one_shot_across_block_boundaries() {
let data: Vec<u8> = (0..1000u32).map(|i| (i * 7 % 251) as u8).collect();
let whole = md5(&data);
for split in [0usize, 1, 55, 56, 63, 64, 65, 127, 128, 500, 999, 1000] {
let mut h = Md5::new();
h.update(&data[..split]);
h.update(&data[split..]);
assert_eq!(h.finalize(), whole, "split={split}");
}
let mut h = Md5::new();
for b in &data {
h.update(std::slice::from_ref(b));
}
assert_eq!(h.finalize(), whole);
}
#[test]
fn padding_edge_lengths() {
// 55 / 56 / 64 bytes exercise the "pad fits in one block" vs "needs a second block" paths.
assert_eq!(hex(&[b'a'; 55]), "ef1772b6dff9a122358552954ad0df65");
assert_eq!(hex(&[b'a'; 56]), "3b0c8ac703f828b04c6c197006d17218");
assert_eq!(hex(&[b'a'; 64]), "014842d480b571495a4a0363793f7367");
}
}
@@ -0,0 +1,26 @@
//! The one export that is not a store engine: `com.droiddeck.launcher.stores.StoresNative`.
//!
//! `nativeVersion()` exists so the app can prove, at Setup time, that `libdroiddeckstores.so`
//! loaded and its JNI exports bind — before a user is mid-download and an
//! `UnsatisfiedLinkError` would surface as a failed install. It returns the crate version from
//! `Cargo.toml`, which the app can show next to the other component versions.
use jni::objects::JClass;
use jni::sys::jstring;
use jni::JNIEnv;
/// The crate version, as `Cargo.toml` declares it.
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
/// `StoresNative.nativeVersion(): String` — the crate version (`"0.1.0"`). A null return means the
/// JVM could not allocate the string, which the caller treats like a failed load.
#[no_mangle]
pub extern "system" fn Java_com_droiddeck_launcher_stores_StoresNative_nativeVersion(
env: JNIEnv,
_class: JClass,
) -> jstring {
match env.new_string(VERSION) {
Ok(s) => s.into_raw(),
Err(_) => std::ptr::null_mut(),
}
}