perf(ci): reuse verified build inputs

## Summary Follow-up to #1412, focused on repeat work observed in completed CI logs. - Give APK compilation and Android tests the same build-label and API-key inputs. Previously `generateReleaseBuildConfig` changed between invocations and caused Kotlin/Java recompilation before tests. - Continue executing the Android test suite on every CI run: disable reuse of the test task's results in CI while retaining compilation caching. Keep the existing heap and process isolation settings. - Cache the four remote checksum-pinned model/AAR downloads used by add-on packs. Hash-check every cache hit, atomically replace corrupt entries only after a verified download, and verify the destination bytes. Packs are still regenerated from current prepared components and their checksums remain part of the APK. - Bound download time and retry connection resets as well as HTTP errors; keep checksum mismatches fatal. - Include the pack recipes in runtime source freshness checks. - Restrict PR report updates to the workflow bot's own marker comment, addressing the minor extraction review finding from #1412. ## Evidence and validation The previous successful run explicitly recorded changed `BuildConfig.java` as the reason for repeated Kotlin/Java compilation between assembly and tests. The final #1412 normal run spent 38 seconds building/downloading packs, 5m59s building the APK, and 3m03s in the Android test step. A forced run also hit a connection reset while fetching pack inputs and passed on retry. Focused tests cover cold/warm input reuse, corrupt-cache repair, changed-pin invalidation, rejected download bytes, actual local HTTP connection failure/retry, identical build metadata and bot-only comment selection. Actionlint passed. The clean Linux full suite passed 1,149 tests (18 skipped) before the final Gradle test-execution policy block. Host, APK/ABI and Android checks are green on the final head, including the ordinary PR run against the updated main branch and both controlled branch attempts below. Validation will compare cold and warm runs on the same commit, confirm cache-hit hash verification, confirm app compilation remains up-to-date in the test invocation, and confirm the Android suite actually executes on both runs. Source-tree, preload, APK/ABI, all-test, signing and publishing gates remain. Existing Gradle, CMake and Cargo caching remains. Native compilation was tens of seconds in the inspected baseline; no speculative native cache changes or unmeasured speed claim are included. Android test execution itself was about 2m30s; this PR removes avoidable setup/recompilation rather than weakening coverage or changing concurrency. ## Follow-up correction The first CI run failed two artwork-key tests that assumed the compiled-in API key was empty. Keeping build inputs consistent exposed that assumption. Tests now check the configured build key and its trimmed fallback after clearing a user key; existing pure-function empty-key and precedence coverage remains. No key values are included in assertion messages. The failed run confirmed BuildConfig generation and Kotlin/Java compilation were UP-TO-DATE in the test invocation; subsequent full runs passed. The merged #1412 main run40007656633 passed signing, pack publishing, preview announcement and catalog updates. ## Gradle cache audit fixes Completed-run logs showed setup-gradle was read-only on PRs. Same-repository PRs and branch runs can now persist their own cache entries; fork PRs remain read-only. GitHub cache ref isolation is retained. See [Gradle cache guidance](https://github.com/gradle/actions/blob/main/docs/setup-gradle.md) and [GitHub cache scopes](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching). PR caches consume additional shared quota; no claim of reuse between different PR refs is made. COMMIT_TIME now comes from SOURCE_COMMIT, matching the provenance field and avoiding timestamp changes from temporary PR merge commits. The source tree and pack validation remain unchanged. This enables a meaningful same-commit warm check. Per-commit generated Java metadata may still cause cross-commit recompilation; moving metadata into packaged data is a separate candidate, not part of this patch. Head 9835c4e3 passed full CI and the same-commit warm rerun below. ## Confirmed final results Both controlled runs use exactly 9835c4e3e1 and workflow_dispatch, avoiding PR merge-tree changes between attempts. | Measurement | First attempt | Warm attempt | Reduction | | --- | ---: | ---: | ---: | | Request to required gate green | 13m09s | 7m11s | 5m58s (45%) | | Gradle APK build step | 6m11s | 2m08s | 4m03s (66%) | | Android test step | 2m57s | 2m08s | 49s | | Add-on pack build step | 40s | 31s | 9s | [First attempt](https://github.com/Droid-Deck/DroidDeck/actions/runs/40015059221/attempts/1) and [warm attempt](https://github.com/Droid-Deck/DroidDeck/actions/runs/40015059221/attempts/2) both passed. Initial queue delay was 3 seconds in each. This is one observed pair, not a guaranteed latency or isolated attribution of every timing difference to cache changes. The first attempt already had some dependency/build cache reuse. Warm logs confirm the exact head's Gradle-home cache restored; release Kotlin and Java compilation were FROM-CACHE during assembly, then UP-TO-DATE during tests. Assembly reported 22 tasks FROM-CACHE. The Android test executor actually ran; test results were neither cached nor skipped. All four remote pack input hits were hash-verified; packs were regenerated. Runtime inputs and APK native exports/dependencies verified. The ordinary [PR workflow](https://github.com/Droid-Deck/DroidDeck/actions/runs/40013127993/attempts/2) also passed against the updated main merge tree. A fresh Gradle daemon and "no cached configuration" remain expected on a new runner: configuration/task-graph persistence is distinct from build-output caching and currently has no configured encryption key. The measured Kotlin/Java FROM-CACHE results demonstrate output reuse despite that message. This proves same-commit warm reuse. New pushes can still invalidate compilation through changed source, toolchain, generated provenance or pack metadata, and cache eviction/ref scope can cause misses. Cross-commit compilation reuse is not established by this pair. Branch validation intentionally skips main-only publishing; merged #1412's main run40007656633 separately passed signing and publication. The user authorized merging after the final cache-recovery correction passed. ## Final cache-recovery correction Commit c95e1147 addresses the remaining minor review finding: after a download passes its SHA check, an unexpected directory at the exact hash-named cache entry is removed before the verified file is promoted. Invalid downloads leave that directory untouched. A regression test covers failed verification, successful repair and subsequent warm reuse. All five focused download tests passed in a clean Linux export. All final-head PR checks passed in [run40018498031](https://github.com/Droid-Deck/DroidDeck/actions/runs/40018498031): host suite ran 1,164 tests (49 platform skips), APK build and native validation passed, Android tests executed successfully, and the source/build validation status is green. Benchmark results above remain measurements of 9835c4e3.
2026-11-11 02:51:10 +00:00 · 2026-10-23 16:34:50 -04:00
commit 61482ffee7
@@ -403,6 +403,7 @@ jobs:

      - uses: gradle/actions/setup-gradle@v4
        with:
+          cache-read-only: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}
          cache-cleanup: on-success

      - name: Resolve the pinned Android NDK
@@ -484,6 +485,12 @@ jobs:
          name: xbox-arm64
          path: app/src/main/assets/linuxfs/usr/local/lib/droiddeck-xbox

+      - name: Cache checksum-pinned pack inputs
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/droiddeck-pack-inputs
+          key: pack-inputs-v1-${{ hashFiles('tools/packs/build.sh', 'tools/packs/fetch-verified.sh', 'tools/packs/tracking/release.env') }}
+          restore-keys: pack-inputs-v1-
      - name: Build the add-on packs
        run: bash tools/packs/build.sh app/build/packs

@@ -526,6 +533,9 @@ jobs:

      - name: Unit tests
        id: unit-tests
+        env:
+          DROIDDECK_BUILD_LABEL: ${{ steps.build-label.outputs.label }}
+          SGDB_API_KEY: ${{ secrets.SGDB_API_KEY }}
        run: |
          set -o pipefail
          mkdir -p app/build
@@ -80,7 +80,7 @@ def sourceCommit = providers.environmentVariable('DROIDDECK_SOURCE_COMMIT').getO
 def prNumber = (providers.environmentVariable('DROIDDECK_PR_NUMBER').getOrNull() ?: '0').with { it.isInteger() ? it : '0' }
 def ciBuild = providers.environmentVariable('GITHUB_ACTIONS').getOrNull() == 'true'
 def releaseSigner = rootProject.file('keystore/release-signer.sha256').text.trim()
-def commitTime = (gitValue(['show', '-s', '--format=%ct', 'HEAD']) ?: '0').with { it.isLong() ? it : '0' }
+def commitTime = (gitValue(['show', '-s', '--format=%ct', '--end-of-options', sourceCommit]) ?: '0').with { it.isLong() ? it : '0' }
 def desktopKde = rootProject.file('tools/desktop-kde/release.env').readLines()
    .findAll { it.contains('=') && !it.startsWith('#') }
    .collectEntries { def kv = it.split('=', 2); [(kv[0].trim()): kv[1].trim()] }
@@ -220,6 +220,10 @@ android {
        unitTests.returnDefaultValues = true
        unitTests.includeAndroidResources = true
        unitTests.all {
+            if (ciBuild) {
+                outputs.upToDateWhen { false }
+                outputs.doNotCacheIf('CI executes the test suite on every run') { true }
+            }
            // Robolectric keeps a sandbox per SDK level the tests ask for; seven no longer fit in Gradle's default 512 MB.
            maxHeapSize = '2g'
            def logTarget = providers.systemProperty('robolectric.logging').orNull
@@ -1,5 +1,6 @@
 package com.droiddeck.launcher.frontend

+import com.droiddeck.launcher.BuildConfig
 import com.droiddeck.launcher.stores.CredentialCipher
 import org.json.JSONArray
 import org.json.JSONObject
@@ -92,8 +93,9 @@ class AddedGameArtLookupTest {
        assertEquals("build", SteamGridDb.pick("", "build"))
        assertEquals("build", SteamGridDb.pick("  ", "build"))
        assertEquals("", SteamGridDb.pick("", ""))
-        assertEquals("", SteamGridDb.builtInKey())
-        assertEquals("", SteamGridDb.key(app))
+        val builtIn = SteamGridDb.builtInKey()
+        assertTrue(builtIn == BuildConfig.SGDB_API_KEY)
+        assertTrue(SteamGridDb.key(app) == builtIn.trim())
    }

    @Test fun theUsersKeyIsSealedOnDiskAndClears() {
@@ -106,7 +108,7 @@ class AddedGameArtLookupTest {
        assertEquals("abcdef0123456789secretkey", SteamGridDb.key(app))
        SteamGridDb.setUserKey(app, "")
        assertFalse(file.exists())
-        assertEquals("", SteamGridDb.key(app))
+        assertTrue(SteamGridDb.key(app) == SteamGridDb.builtInKey().trim())
    }

    @Test fun aFailedLookupIsNotRememberedAsNoMatch() {
@@ -25,14 +25,7 @@ trap 'rm -rf "$work"' EXIT
 sha256() { if command -v sha256sum >/dev/null; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }

 fetch() {
-    local url=$1 expected=$2 dest=$3
-    curl -fsSL --retry 4 -o "$dest" "$url"
-    local actual
-    actual=$(sha256 "$dest")
-    if [ "$actual" != "$expected" ]; then
-        echo "checksum mismatch for $url: $actual" >&2
-        exit 1
-    fi
+    bash "$here/fetch-verified.sh" "$@"
 }

 pack() {
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+set -euo pipefail
+url=${1:?URL required}
+expected=${2:?SHA-256 required}
+destination=${3:?destination required}
+[[ "$expected" =~ ^[a-f0-9]{64}$ ]] || { echo 'Invalid download SHA-256' >&2; exit 1; }
+cache=${DROIDDECK_PACK_CACHE:-"$HOME/.cache/droiddeck-pack-inputs"}
+mkdir -p "$cache" "$(dirname "$destination")"
+kept="$cache/$expected"
+verify() {
+    if command -v sha256sum >/dev/null; then
+        echo "$expected  $1" | sha256sum -c - >/dev/null 2>&1
+    else
+        echo "$expected  $1" | shasum -a 256 -c - >/dev/null 2>&1
+    fi
+}
+if [ -f "$kept" ] && verify "$kept"; then
+    echo "Verified pack input cache hit: $expected"
+else
+    part=$(mktemp "$cache/$expected.part.XXXXXX")
+    trap 'rm -f "$part"' EXIT
+    curl -fsSL --retry 4 --retry-all-errors --connect-timeout 15 --max-time 180 --retry-max-time 240 \
+        -o "$part" "$url"
+    verify "$part" || { echo "Checksum mismatch for $url" >&2; exit 1; }
+    if [ -d "$kept" ]; then
+        rm -rf -- "$kept"
+    fi
+    mv -f "$part" "$kept"
+    echo "Downloaded and verified pack input: $expected"
+fi
+cp "$kept" "$destination"
+verify "$destination" || { echo 'Copied pack input checksum mismatch' >&2; exit 1; }
@@ -69,7 +69,7 @@ MARK="<!-- droiddeck-pr-build -->"
  echo
 } > /tmp/pr-report.md
 cat /tmp/pr-report.md >> "$GITHUB_STEP_SUMMARY"
-ID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR}/comments" --paginate --jq ".[] | select(.body | startswith(\"$MARK\")) | .id" | head -1 || true)
+ID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR}/comments" --paginate --jq ".[] | select(.user.login == \"github-actions[bot]\" and (.body | startswith(\"$MARK\"))) | .id" | head -1 || true)
 if [ -n "$ID" ]; then
  gh api -X PATCH "repos/${GITHUB_REPOSITORY}/issues/comments/${ID}" -F body=@/tmp/pr-report.md >/dev/null
 else
@@ -21,7 +21,7 @@ PACKED_FILES = {'lib/arm64-v8a/libproot.so': ('proot', 'libproot.so'),
 PACKED_LIBRARIES = ['lib/arm64-v8a/libopencv_java4.so', 'lib/arm64-v8a/libmediapipe_tasks_jni.so']
 PACK_NAMES = ['proot', 'directaudio', 'tracking', 'xbox', 'comet']
 SOURCE_PATHS = ['app/build.gradle', 'app/src/main/cpp/fakeinput_steam.cpp', 'app/src/main/assets/pulseaudio.tzst',
-                'tools/linuxfs', 'tools/proot', 'tools/aaudio-sink', 'tools/directaudio',
+                'tools/linuxfs', 'tools/proot', 'tools/aaudio-sink', 'tools/directaudio', 'tools/packs',
                'tools/gamescope/release.env', 'tools/droiddeck-esync/release.env',
                'tools/mangoapp', 'tools/msitools', 'tools/pipewire', 'tools/comet', 'tools/xbox', 'tools/components', 'tools/ci', 'tools/build_local.sh',
                'tools/release/runtime_inputs.py', 'app/src/main/assets/desktop-removal',
@@ -1,6 +1,9 @@
 import itertools
+import json
 import os
 from pathlib import Path
+import shlex
+import shutil
 import subprocess
 import tempfile
 import unittest
@@ -12,6 +15,37 @@ ROOT = Path(__file__).resolve().parents[2]


 class CiChecksTest(unittest.TestCase):
+    def test_pr_cache_writes_and_source_timestamp_policy(self):
+        jobs = yaml.safe_load((ROOT / '.github/workflows/build.yml').read_text())['jobs']
+        cache = next(step for step in jobs['build']['steps']
+                     if step.get('uses') == 'gradle/actions/setup-gradle@v4')['with']
+        self.assertEqual(cache['cache-read-only'],
+                         "${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}")
+        gradle = (ROOT / 'app/build.gradle').read_text()
+        self.assertIn("gitValue(['show', '-s', '--format=%ct', '--end-of-options', sourceCommit])", gradle)
+
+    def test_report_comment_lookup_ignores_human_marker(self):
+        if not shutil.which('jq'):
+            self.skipTest('jq required to execute comment selection')
+        script = (ROOT / 'tools/release/report-build.sh').read_text()
+        line = next(line for line in script.splitlines() if line.startswith('ID='))
+        arguments = shlex.split(line[len('ID=$('):].split(' | head', 1)[0])
+        query = arguments[arguments.index('--jq') + 1].replace('$MARK', '<!-- build -->')
+        comments = [dict(id=1, user=dict(login='someone'), body='<!-- build --> forged'),
+                    dict(id=2, user=dict(login='github-actions[bot]'), body='<!-- build --> result')]
+        result = subprocess.check_output(['jq', '-r', query], input=json.dumps(comments), text=True)
+        self.assertEqual(result.strip(), '2')
+
+    def test_build_and_tests_use_identical_generated_metadata(self):
+        jobs = yaml.safe_load((ROOT / '.github/workflows/build.yml').read_text())['jobs']
+        steps = {step.get('id'): step for step in jobs['build']['steps']}
+        self.assertEqual(steps['build']['env'], steps['unit-tests']['env'])
+        self.assertIn('DROIDDECK_BUILD_LABEL', steps['unit-tests']['env'])
+        self.assertIn('SGDB_API_KEY', steps['unit-tests']['env'])
+        gradle = (ROOT / 'app/build.gradle').read_text()
+        self.assertIn("outputs.doNotCacheIf('CI executes the test suite on every run') { true }", gradle)
+        self.assertIn('outputs.upToDateWhen { false }', gradle)
+
    def test_independent_jobs_wait_only_for_the_pinned_source(self):
        jobs = yaml.safe_load((ROOT / '.github/workflows/build.yml').read_text())['jobs']
        for name in ('xbox', 'components', 'python-tests'):
@@ -0,0 +1,125 @@
+import hashlib
+import http.server
+import json
+import os
+from pathlib import Path
+import shutil
+import socket
+import subprocess
+import tempfile
+import threading
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+FETCH = ROOT / 'tools/packs/fetch-verified.sh'
+
+
+class PackDownloadsTest(unittest.TestCase):
+    def setUp(self):
+        directory = tempfile.TemporaryDirectory()
+        self.addCleanup(directory.cleanup)
+        self.root = Path(directory.name)
+        self.cache = self.root / 'cache'
+        self.cache.mkdir()
+        self.destination = self.root / 'download'
+        self.remote = self.root / 'remote'
+        self.remote.write_bytes(b'verified input')
+        self.checksum = hashlib.sha256(self.remote.read_bytes()).hexdigest()
+        self.log = self.root / 'curl.log'
+        tool = self.root / 'bin/curl'
+        tool.parent.mkdir()
+        tool.write_text('#!/usr/bin/env python3\nimport json,os,shutil,sys\n'
+                        'args=sys.argv[1:]\n'
+                        'with open(os.environ["CURL_LOG"],"a") as f: f.write(json.dumps(args)+"\\n")\n'
+                        'shutil.copyfile(os.environ["REMOTE"],args[args.index("-o")+1])\n')
+        tool.chmod(0o755)
+        self.env = dict(os.environ, DROIDDECK_PACK_CACHE=str(self.cache), CURL_LOG=str(self.log),
+                        REMOTE=str(self.remote), PATH=str(tool.parent) + os.pathsep + os.environ['PATH'])
+
+    def fetch(self, checksum=None, url='https://example.test/pinned', env=None):
+        return subprocess.run(['bash', str(FETCH), url, checksum or self.checksum, str(self.destination)],
+                              env=env or self.env, capture_output=True, text=True)
+
+    def test_cold_download_then_warm_reuse_and_corruption_repair(self):
+        self.assertEqual(self.fetch().returncode, 0)
+        self.assertEqual(self.fetch().returncode, 0)
+        self.assertEqual(len(self.log.read_text().splitlines()), 1)
+        self.assertEqual(self.destination.read_bytes(), b'verified input')
+        (self.cache / self.checksum).write_bytes(b'corrupt')
+        self.assertEqual(self.fetch().returncode, 0)
+        self.assertEqual(len(self.log.read_text().splitlines()), 2)
+        self.assertEqual((self.cache / self.checksum).read_bytes(), b'verified input')
+
+    def test_directory_cache_entry_is_replaced_only_after_verification(self):
+        entry = self.cache / self.checksum
+        entry.mkdir()
+        sentinel = entry / 'old-entry'
+        sentinel.write_bytes(b'preserve until verified')
+        self.remote.write_bytes(b'tampered download')
+        self.assertNotEqual(self.fetch().returncode, 0)
+        self.assertEqual(sentinel.read_bytes(), b'preserve until verified')
+        self.assertFalse(self.destination.exists())
+        self.remote.write_bytes(b'verified input')
+        result = self.fetch()
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertTrue(entry.is_file())
+        self.assertEqual(entry.read_bytes(), b'verified input')
+        self.assertEqual(self.destination.read_bytes(), b'verified input')
+        self.assertEqual(list(self.cache.iterdir()), [entry])
+        self.assertEqual(self.fetch().returncode, 0)
+        self.assertEqual(len(self.log.read_text().splitlines()), 2)
+
+    def test_changed_pin_cannot_reuse_old_bytes(self):
+        self.assertEqual(self.fetch().returncode, 0)
+        self.remote.write_bytes(b'new approved input')
+        updated = hashlib.sha256(self.remote.read_bytes()).hexdigest()
+        self.assertEqual(self.fetch(updated).returncode, 0)
+        self.assertEqual(len(self.log.read_text().splitlines()), 2)
+        self.assertEqual(self.destination.read_bytes(), b'new approved input')
+
+    def test_bad_download_is_not_cached_or_written(self):
+        self.remote.write_bytes(b'tampered download')
+        self.assertNotEqual(self.fetch().returncode, 0)
+        self.assertFalse((self.cache / self.checksum).exists())
+        self.assertFalse(self.destination.exists())
+        self.assertEqual(list(self.cache.iterdir()), [])
+        arguments = json.loads(self.log.read_text())
+        self.assertIn('--retry-all-errors', arguments)
+        self.assertIn('--max-time', arguments)
+        self.assertIn('--retry-max-time', arguments)
+
+    def test_connection_reset_retries_then_verifies_real_download(self):
+        if not shutil.which('curl'):
+            self.skipTest('curl required for transport test')
+        content = self.remote.read_bytes()
+        attempts = []
+
+        class Handler(http.server.BaseHTTPRequestHandler):
+            def do_GET(self):
+                attempts.append(True)
+                if len(attempts) == 1:
+                    self.connection.shutdown(socket.SHUT_RDWR)
+                    self.connection.close()
+                    return
+                self.send_response(200)
+                self.send_header('Content-Length', str(len(content)))
+                self.end_headers()
+                self.wfile.write(content)
+
+            def log_message(self, *args):
+                pass
+
+        server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler)
+        thread = threading.Thread(target=server.serve_forever, daemon=True)
+        thread.start()
+        try:
+            env = dict(self.env, PATH=os.environ['PATH'])
+            result = self.fetch(url=f'http://127.0.0.1:{server.server_port}/input', env=env)
+            self.assertEqual(result.returncode, 0, result.stderr)
+            self.assertGreaterEqual(len(attempts), 2)
+            self.assertEqual(self.destination.read_bytes(), content)
+        finally:
+            server.shutdown()
+            server.server_close()
+            thread.join()