Add the droiddeck-esync pack pipeline (#189)

Builds esync packs for new Valve, GE and CachyOS Proton 11 releases on arm64
runners, smoke-tests them against stock with esstress, and on main publishes
each pack and a signed index as non-latest releases. A watcher checks Steam
and the GE/CachyOS releases every 20 minutes and starts a build for new ones.
The app side lands separately with feat/userspace-esync-ntsync.
This commit is contained in:
MaxsTechReview authored and GitHub committed 2026-10-04 02:58:55 +00:00
1 parent c535f6ef6e
commit e7b6876877
16 files changed
+9000

No files matched your search

+170
View File
@@ -0,0 +1,170 @@
# droiddeck-esync packs
DroidDeck gives Proton three in-process sync backends, picked with the Wine sync tabs in the
Steam settings' Games group: ntsync, fsync (the default), esync and wineserver. The wineserver tab
sets `PROTON_NO_ESYNC=1`, `PROTON_NO_FSYNC=1` and `PROTON_NO_NTSYNC=1` (unless a game's own
environment sets them), so every sync object goes through the server.
- droiddeck-esync: eventfd and shared memory sync. Stock Wine 11 / Proton 11 builds
have no eventfd-based sync, so the patch series under `patches/<series>` adds it to wineserver
and the unix side of ntdll. A Proton build without a pack (a release newer than the published
packs, or one whose pack failed to install) runs stock with droiddeck-fsync standing in, or
droiddeck-ntsync where fsync is off for the game (`PROTON_NO_FSYNC=1`, or one of the games
Proton itself keeps off fsync and esync), so it still gets in-process sync; `PROTON_NO_NTSYNC=1`
as well leaves it on wineserver-only sync. A Proton with esync built in (GE, cachyos) runs its
own esync, and a game that turns esync off moves to droiddeck-fsync or droiddeck-ntsync there
too.
- droiddeck-fsync: Proton's own fsync on the stock binaries. fsync waits with `futex_waitv(2)`,
which the app sandbox refuses, so `libblsession.so` (`BL_FSYNC=1`) answers that system call in
userspace: every waiter registers the futexes it waits on in a shared table next to the fsync
shared memory and sleeps on a futex of its own, and every `FUTEX_WAKE` on the fsync shared memory
wakes the waiters registered on that word. No pack is involved, so it keeps working the moment
Steam updates Proton. It is the default (the fsync tab, `BL_FSYNC_FIRST=1`, also per game),
ahead of droiddeck-esync; on the esync tab it stands in where no pack fits, and it also takes
over for a game whose environment turns esync off (`PROTON_NO_ESYNC=1`, `WINEESYNC=0`, or too
few file descriptors); where fsync is off as well, such a game gets droiddeck-ntsync, with the
pack's wineserver when a pack fits. A launch into a prefix whose wineserver is still running
keeps the sync choice that server was started with (`prefixes.json` in the store), because a
Wine client that disagrees with its server about fsync exits. It inherits stock fsync's behaviour, the same as on a Linux kernel:
a wait-all on a mutex abandoned by a dead thread is not satisfied, a pending user APC is
delivered ahead of an object that is already signaled, a mutex owned by a process killed with
`TerminateProcess` is not abandoned, and `PulseEvent` can miss a waiter that has not run yet.
- droiddeck-ntsync: a userspace implementation of the `/dev/ntsync` interface in
`libblsession.so` (`BL_SYNC=1`), which stock Proton 11 builds use as they would the kernel
driver. It is the ntsync tab. A pack still supplies the binaries when one fits, and its
wineserver then picks ntsync over esync; a Proton whose wineserver has no ntsync client (no
`/dev/ntsync` in it) gets droiddeck-fsync instead.
wineserver picks the backend at startup: `PROTON_NO_NTSYNC` unset -> `/dev/ntsync`
(droiddeck-ntsync, or a kernel driver the device provides) -> fsync (droiddeck-fsync, or a kernel
and sandbox that allow `futex_waitv`) -> droiddeck-esync (needs `WINEESYNC` nonzero) ->
server-side sync. When a pack is in use and droiddeck-esync is on, the guest's `droiddeck-esync`
script also sets `PROTON_NO_FSYNC=1` unless the game's environment sets it, so droiddeck-esync
comes before fsync unless droiddeck-fsync was chosen. A wineserver from a pack reports the
backend on stderr as `droiddeck-ntsync: up and running.` or `droiddeck-esync: up and running.`;
with droiddeck-fsync the stock wineserver prints `fsync: up and running.` after
`droiddeck-fsync: up and running.`. A pack's wineserver also abandons, when a process dies, every
mutex still owned by one of its threads, and, once a thread killed by `TerminateThread` has really
exited, every mutex it still owns. A killed thread keeps running until the kill signal reaches it
and can take a mutex after the server abandoned its mutexes; without these passes that mutex would
stay locked for good (they cover droiddeck-ntsync and kernel ntsync as well). The ntdll side
reads, writes and polls its eventfds with raw system calls, so no `LD_PRELOAD` wrapper of
`read`/`write`/`poll` (the guest preloads several) sits in the sync path or in the unwind of a
thread Wine kills.
The patch changes no wineserver request: `make_pack.py` refuses a build whose
`server_protocol.h`, `request_handlers.h` or `request_trace.h` differ from the unpatched
control build, or whose ntdll.so exports other symbols. Only `files/lib/wine/aarch64-unix/ntdll.so`
and `files/bin-arm64/wineserver` are replaced, so every Proton build needs its own pair, built
from that build's exact wine source. That pair plus `pack.json` is a pack (`<id>.tzst`). On the
device the guest runs Proton from a shadow tree that links the stock tool and holds the two
patched files (`droiddeck-esync`, store in `~/.local/share/droiddeck-esync`).
## Credits
droiddeck-esync follows the esync design and droiddeck-ntsync implements the ntsync interface;
both are the work of Elizabeth Figura for Wine, Proton and the Linux kernel. droiddeck-fsync runs
fsync, by Elizabeth Figura and Paul Gofman for Proton, over `futex_waitv(2)` by André Almeida.
DroidDeck builds on that work for Android, where neither the ntsync driver, `futex_waitv` nor a
Proton build with esync is available.
## Matching
- `ge`, `cachyos`: built against the release tarball's own binaries. `pack.json` `stock` holds
their sha256; the guest uses the pack only when both files hash the same. `make_pack.py` also
requires the control build to reproduce both files byte for byte, so the patched pair differs
from the release only by the patch (`--allow-unreproduced` lifts this for local experiments).
- `valve`: the depots cannot be downloaded without a Steam login, so `stock` holds the control
build's hashes and `source_match` is true: the guest also accepts a tool whose version token
(second field of `<tool>/version`) and ntdll.so exports hash equal the pack's. With the optional
`STEAM_DEPOT_USER` / `STEAM_DEPOT_TOKEN` secrets (and the `DEPOT_DOWNLOADER_SHA256` variable)
the job downloads the depot instead and, when the depot is at the tag being built, makes an
exact pack. The guest's launch line says which kind matched (`exact match` or `source match`).
- A stock wineserver that already has esync built in (it prints `esync: up and running`) never
gets a pack.
## Building and publishing
`.github/workflows/watch-proton-releases.yml` runs every 20 minutes and decides when a build is
due. It asks Steam anonymously (`steamcmd +app_info_print`) which public build of each Valve ARM64
depot app in `flavors.json` is live, and `discover.py watch` compares that, the newest `keep` tags
and releases of every flavor, each flavor's `rev` and its patch series digest with the state of the
last run (`watch-state.json` in the `droiddeck-esync-index` release). A flavor whose builds, rev or
patch series changed, or whose Steam build moved, starts
`.github/workflows/build-droiddeck-esync-packs.yml` for that flavor; nothing else starts a build.
A run that finds no change takes about a minute and builds nothing. If steamcmd fails, the run
watches the tags only and keeps the last Steam state.
`.github/workflows/build-droiddeck-esync-packs.yml` also runs once a day as a safety net, on pushes
to `tools/droiddeck-esync/**` and by hand (flavor, one tag, publish yes/no). Only runs on `main` of
`Droid-Deck/DroidDeck` publish packs; the schedule and pushes do nothing in other repositories,
and manual runs elsewhere only build.
1. `discover.py matrix` lists the newest `keep` builds of each family in `flavors.json` (Valve
tags, GE and CachyOS releases with an arm64 tarball) and drops the ones the published index,
or the `droiddeck-esync-<flavor>` release, already has at the flavor's `rev`, the ones
`revoked.txt` names, and the ones that failed in the last day at the same `rev` and patch
series (a `<key>-r<rev>-<series digest>.failed` asset in the flavor's release). Naming the tag
in a manual run builds it anyway.
2. Each build runs on an arm64 runner with a read-only token, since it runs upstream code: the
wine source is checked out at the tag (GE also runs its `protonprep-valve-staging.sh`),
`build-pack.sh` builds a control and a patched pair inside the SDK image named by the
source's own `Makefile.in` (arm64-llvm), at the flavor's `build_dir` in `flavors.json` (the
path the release itself was built at, which ends up in the binaries), and `make_pack.py` makes
the pack. Patches apply with `--fuzz=0`: a hunk whose context moved fails the build instead of
landing somewhere else. For GE and CachyOS, whose release tarball holds the whole Proton,
`smoke-test.sh` then boots a prefix with that stock Proton and runs `tests/esstress.c` (built
with the SDK's llvm-mingw) under server-side sync as a baseline, and again with the pack under
droiddeck-esync. A pack that fails while stock passes is not published; when stock itself cannot
run on the runner the step only warns.
3. The publish job checks each build's artifact against the matrix (`discover.py packs`) and
uploads it to the `droiddeck-esync-valve`, `droiddeck-esync-ge` or `droiddeck-esync-cachyos`
release as `<id>.tzst` with its index entry `<id>.json`. A build that made no pack gets its
`.failed` marker there (`discover.py failures`), so a patch that does not apply or a stock
wineserver with esync built in is not rebuilt on every run.
4. The index job (environment `droiddeck-esync-signing`) runs only when the run built a pack or
a published pack is not in the index yet. It collects every `<id>.json`, keeps the
highest `rev` per stock pair and version, marks revoked ids, signs `index.json` (ECDSA P-256,
`index-key.pub`) and publishes `bundle-<YYYYMMDD>-<HHMMSS>.tzst` (named after `generated`, so
every signed index has its own), `index.json` and `index.json.sig` to `droiddeck-esync-index`.
`generated` only grows. Packs stay in the index until they are revoked, also when their
entry asset disappears; the previous index counts only when its signature verifies.
A pack is rebuilt only for a new `rev`: bump `rev` of the flavor in `flavors.json` whenever the
patch series or the build changes. Local builds: `BUILD_DIR=<the flavor's build_dir> SYNC_PROFILE=<the
flavor's profile> build-local.sh <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]`,
then `make_pack.py`.
## Setup
- Signing key: create the repository environment `droiddeck-esync-signing` (Settings ->
Environments, limit it to `main`) and store the PEM private key as its secret
`SYNC_INDEX_KEY`: `gh secret set SYNC_INDEX_KEY --env droiddeck-esync-signing < index-key.pem`.
The key never enters the repository; the app trusts only `index-key.pub` (also compiled into
`EsyncPacks.kt`), so a new key needs an app release.
- Optional: `STEAM_DEPOT_USER` and `STEAM_DEPOT_TOKEN` repository secrets for exact Valve packs,
plus the repository variable `DEPOT_DOWNLOADER_SHA256`: the sha256 of
`DepotDownloader-linux-arm64.zip` from the DepotDownloader release the workflow pins
(`DEPOT_DOWNLOADER_TAG`). The depot step is skipped while it is unset.
- APK bundle: after an index run, the job summary prints three lines. Put them in
`tools/droiddeck-esync/release.env`:
```
SYNC_BUNDLE_TAG=droiddeck-esync-index
SYNC_BUNDLE_ASSET=bundle-20261002-061733.tzst
SYNC_BUNDLE_SHA256=<sha256>
```
`build.yml` then unpacks that bundle into `app/src/main/assets/droiddeck-esync/` (gitignored). A
published bundle is never overwritten, so the pinned sha256 stays valid; bump `release.env`
to ship newer packs inside the apk. The app fetches newer packs on its own. A pack the bundle
carries that `revoked.txt` names is left out of the apk, with a warning to bump `release.env`.
## Revocation
Add the pack id as the first word of a line in `revoked.txt` and push to `main`. The workflow
re-signs the index with `"revoked": true` for that id; the app deletes installed copies and
never installs it again, and `discover.py` does not rebuild it at the same `rev`. The apk build
also leaves it out of the bundled packs. To ship a fixed pack for the same Proton build, bump the
flavor's `rev`. Runs that only build (other refs, or manual runs without publish) never queue in
front of a publishing run, so a revocation is signed by the next run on `main`.
+27
View File
@@ -0,0 +1,27 @@
#!/bin/bash
set -euo pipefail
if [ $# -lt 4 ]; then
echo "usage: $0 <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]" >&2
echo "env: BUILD_DIR (path the build runs at inside the SDK), SYNC_PROFILE (valve or cachyos), JOBS, SOURCE_DATE_EPOCH" >&2
exit 2
fi
here=$(cd "$(dirname "$0")" && pwd)
sdk=$(realpath "$1")
src=$(realpath "$2")
work=$(realpath -m "$3")
out=$(realpath -m "$4")
patches=${5:+$(realpath "$5")}
build_dir=${BUILD_DIR:-/builds/proton/proton/build-dir}
rm -rf "$work"
mkdir -p "$work" "$out"
"$here/build-pack.sh" prepare "$src" "$work" ${patches:+"$patches"}
bwrap --unshare-user --uid 0 --gid 0 --unshare-ipc --unshare-pid \
--bind "$sdk" / --proc /proc --dev /dev --tmpfs /tmp \
--bind "$work" "$build_dir" --bind "$out" /out --ro-bind "$here/build-pack.sh" /build-pack.sh \
--setenv PATH /opt/llvm-mingw/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
--setenv HOME /tmp --setenv LANG C.UTF-8 --setenv JOBS "${JOBS:-$(nproc)}" \
--setenv SOURCE_DATE_EPOCH "${SOURCE_DATE_EPOCH:-1700000000}" --setenv SYNC_PROFILE "${SYNC_PROFILE:-valve}" \
--chdir "$build_dir" \
/bin/bash /build-pack.sh build "$build_dir" /out
sha256sum "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
+125
View File
@@ -0,0 +1,125 @@
#!/bin/bash
set -euo pipefail
usage() {
echo "usage: $0 prepare <wine-source> <build-dir> [patch-dir]" >&2
echo " $0 build <build-dir> <out-dir>" >&2
echo " $0 all <wine-source> <out-dir> [patch-dir]" >&2
exit 2
}
prepare() {
local origin=$1 b=$2 patches=${3:-}
local s=$b/src-wine
mkdir -p "$s"
rsync --filter=:C --exclude '*~' --exclude .git --exclude compile_commands.json \
--exclude configure --exclude autom4te.cache --exclude include/config.h.in --exclude include/config.h.ink \
--exclude include/wine/vulkan.h --exclude dlls/vulkan-1/vulkan-1.spec \
--exclude dlls/winevulkan/loader_thunks.c --exclude dlls/winevulkan/loader_thunks.h \
--exclude dlls/winevulkan/vulkan_thunks.c --exclude dlls/winevulkan/vulkan_thunks.h \
--exclude dlls/winevulkan/winevulkan.json --exclude dlls/winevulkan/winevulkan.spec \
--exclude dlls/ntdll/ntsyscalls.h --exclude dlls/win32u/win32syscalls.h \
--exclude include/wine/server_protocol.h --exclude server/request_handlers.h --exclude server/request_trace.h \
-Oarx --delete "$origin/" "$s/"
for f in include/wine/server_protocol.h server/request_handlers.h server/request_trace.h; do
cp -a "$origin/$f" "$s/$f"
done
if [ -n "$patches" ]; then
local p found=0
for p in "$patches"/*.patch; do
[ -f "$p" ] || continue
patch -d "$s" -p1 --forward --fuzz=0 --no-backup-if-mismatch -s < "$p"
found=1
done
[ "$found" = 1 ] || { echo "no patches in $patches" >&2; exit 1; }
fi
(
cd "$s"
XDG_CACHE_HOME=$b/cache dlls/winevulkan/make_vulkan -x vk.xml -X video.xml
tools/make_specfiles || true
tools/make_requests
) > "$b/generate.log" 2>&1
touch "$b/.prepared"
}
build() {
local b=$1 out=$2
local s=$b/src-wine o=$b/obj-wine-aarch64 d=$b/dst-wine-aarch64
[ -f "$b/.prepared" ] || { echo "$b is not prepared" >&2; exit 1; }
local jobs=${JOBS:-$(nproc)}
local epoch=$(( ${SOURCE_DATE_EPOCH:-1700000000} - 20 ))
local flags args pe_common i386_flags x86_64_flags conf
case ${SYNC_PROFILE:-valve} in
valve)
flags="-Wno-discarded-qualifiers -march=armv8.2-a -mtune=cortex-x3 -O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
args=(--enable-werror --with-mingw=clang --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
pe_common="-O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
i386_flags="-Wno-discarded-qualifiers -mstackrealign -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
x86_64_flags="-Wno-discarded-qualifiers -mcmodel=small -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
conf="../$(basename "$s")/configure"
;;
cachyos)
flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -O2 -march=armv8.2-a -mtune=cortex-x3 -fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
args=(--with-mingw=clang --enable-build-id --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
pe_common="-fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
i386_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mstackrealign -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
x86_64_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mcmodel=small -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
conf="$s/configure"
;;
*)
echo "unknown SYNC_PROFILE ${SYNC_PROFILE}" >&2
exit 2
;;
esac
local env=(STRIP=llvm-strip AR=aarch64-linux-gnu-ar RANLIB=aarch64-linux-gnu-ranlib
CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ LD=aarch64-linux-gnu-ld
RC=aarch64-w64-mingw32-windres WIDL=aarch64-w64-mingw32-widl PKG_CONFIG=aarch64-linux-gnu-pkg-config
"PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig"
"CFLAGS=$flags" "CPPFLAGS=$flags" "CXXFLAGS=-std=c++17 $flags" "LDFLAGS=" "SOURCE_DATE_EPOCH=$epoch"
"CROSSCFLAGS=$flags" "CROSSLDFLAGS="
"aarch64_CFLAGS=$flags" "aarch64_CPPFLAGS=$flags" "aarch64_CXXFLAGS=-std=c++17 $flags" "aarch64_LDFLAGS="
i386_AR=i686-w64-mingw32-ar i386_RANLIB=i686-w64-mingw32-ranlib i386_CC=i686-w64-mingw32-gcc
i386_CXX=i686-w64-mingw32-g++ i386_LD=i686-w64-mingw32-ld
"i386_CFLAGS=$i386_flags" "i386_CPPFLAGS=$i386_flags" "i386_CXXFLAGS=-std=c++17 $i386_flags" "i386_LDFLAGS="
"i386_PKG_CONFIG_LIBDIR=/usr/lib/i386-w64-mingw32/pkgconfig:/usr/share/pkgconfig"
x86_64_AR=x86_64-w64-mingw32-ar x86_64_RANLIB=x86_64-w64-mingw32-ranlib x86_64_CC=x86_64-w64-mingw32-gcc
x86_64_CXX=x86_64-w64-mingw32-g++ x86_64_LD=x86_64-w64-mingw32-ld
"x86_64_CFLAGS=$x86_64_flags" "x86_64_CPPFLAGS=$x86_64_flags" "x86_64_CXXFLAGS=-std=c++17 $x86_64_flags" "x86_64_LDFLAGS="
"x86_64_PKG_CONFIG_LIBDIR=/usr/lib/x86_64-w64-mingw32/pkgconfig:/usr/share/pkgconfig")
export PATH=/opt/llvm-mingw/bin:$PATH
( cd "$s" && autoreconf -fi && rm -rf autom4te.cache ) > "$b/autoreconf.log" 2>&1
rm -rf "$o" "$d"
mkdir -p "$o" "$d"
( cd "$o" && env "${env[@]}" "$conf" -C --prefix="$d" --libdir="$d/lib" --host=aarch64-linux-gnu "${env[@]}" \
"${args[@]}" ) > "$b/configure.log" 2>&1
( cd "$o" && env "${env[@]}" make -j"$jobs" __tooldeps__ ) > "$b/tools.log" 2>&1
( cd "$o" && env "${env[@]}" make -j"$jobs" dlls/ntdll/ntdll.so server/wineserver ) > "$b/make.log" 2>&1
mkdir -p "$out/files/lib/wine/aarch64-unix" "$out/files/bin-arm64"
llvm-objcopy --strip-debug --set-section-flags .text=contents,alloc,load,readonly,code \
"$o/dlls/ntdll/ntdll.so" "$out/files/lib/wine/aarch64-unix/ntdll.so"
llvm-objcopy --strip-debug "$o/server/wineserver" "$out/files/bin-arm64/wineserver"
chmod 0755 "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
cp "$s/include/wine/server_protocol.h" "$s/server/request_handlers.h" "$s/server/request_trace.h" "$out/"
sed -n 's/^#define SERVER_PROTOCOL_VERSION \([0-9]*\)$/\1/p' "$out/server_protocol.h" > "$out/protocol"
}
[ $# -ge 1 ] || usage
case $1 in
prepare)
[ $# -ge 3 ] || usage
prepare "$(realpath "$2")" "$(realpath -m "$3")" "${4:+$(realpath "$4")}"
;;
build)
[ $# -eq 3 ] || usage
build "$(realpath "$2")" "$(realpath -m "$3")"
;;
all)
[ $# -ge 3 ] || usage
b=${BUILD_DIR:-/builds/proton/proton/build-dir}
prepare "$(realpath "$2")" "$b" "${4:+$(realpath "$4")}"
build "$b" "$(realpath -m "$3")"
;;
*)
usage
;;
esac
+654
View File
@@ -0,0 +1,654 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import subprocess
import sys
import urllib.parse
from datetime import datetime, timedelta, timezone
from pathlib import Path
from make_index import check_entry
from make_pack import FLAVORS, HERE, PATCHES, REVOKED, SERIES, PackError, patch_digest, read_revoked, sha256_file, slug
FLAVORS_FILE = HERE / "flavors.json"
ARCHIVE = re.compile(r"\.tar\.(gz|xz)$")
ASSIGNMENT = re.compile(r"(?:export\s+|override\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*(?:\?|\+|!|:{1,3})?=\s*(.*)")
REFERENCE = re.compile(r"\$[({]([A-Za-z_][A-Za-z0-9_]*)[)}]")
IMAGE = re.compile(r"[A-Za-z0-9][A-Za-z0-9.-]*(?::[0-9]+)?(?:/[A-Za-z0-9._-]+)+(?::[A-Za-z0-9._-]+)?(?:@sha256:[0-9a-f]{64})?")
REPO = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+")
CLONES = ("shallow", "treeless")
PACK_REV = re.compile(r"-r([0-9]+)$")
ARTIFACT_PREFIX = "droiddeck-esync-pack-"
FAILED = ".failed"
RETRY_AFTER = timedelta(days=1)
VDF_TOKEN = re.compile(r'"((?:[^"\\]|\\.)*)"|([{}])')
APP_HEADER = re.compile(r"^AppID : ([0-9]+),", re.M)
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
def gh(*args: str) -> subprocess.CompletedProcess:
try:
return subprocess.run(["gh", *args], capture_output=True, text=True, check=False)
except FileNotFoundError as e:
raise PackError("the GitHub CLI (gh) is not installed") from e
def json_values(text: str) -> list:
decoder = json.JSONDecoder()
values = []
position = 0
while True:
while position < len(text) and text[position].isspace():
position += 1
if position == len(text):
return values
value, position = decoder.raw_decode(text, position)
values.append(value)
def gh_api(path: str, paginate: bool = False, missing_ok: bool = False):
result = gh("api", *(["--paginate"] if paginate else []), path)
if result.returncode != 0:
if missing_ok and "HTTP 404" in result.stderr:
return None
raise PackError(f"gh api {path}: {result.stderr.strip() or f'exit {result.returncode}'}")
try:
values = json_values(result.stdout)
except ValueError as e:
raise PackError(f"gh api {path}: {e}") from e
if len(values) == 1:
return values[0]
if values and all(isinstance(value, list) for value in values):
return [item for value in values for item in value]
raise PackError(f"gh api {path}: unexpected output")
def quote(value: str) -> str:
return urllib.parse.quote(value, safe="")
def load_flavors(path: Path) -> dict:
try:
data = json.loads(path.read_text())
except (OSError, ValueError) as e:
raise PackError(f"cannot read {path}: {e}") from e
def fail(why: str):
raise PackError(f"{path}: {why}")
def compiles(pattern: str, where: str):
try:
re.compile(pattern)
except re.error as e:
fail(f"{where}: {pattern!r} is not a regular expression: {e}")
if not isinstance(data, dict) or data.get("format") != 1 or not isinstance(data.get("flavors"), dict):
fail("is not a format 1 flavor list")
image = data.get("sdk_image")
if not isinstance(image, dict) or not all(isinstance(image.get(key), str) and image[key] for key in ("file", "variable", "match")):
fail("sdk_image needs file, variable and match")
for name, cfg in data["flavors"].items():
if name not in FLAVORS:
fail(f"unknown flavor {name!r}")
if not isinstance(cfg, dict) or not isinstance(cfg.get("repo"), str) or not REPO.fullmatch(cfg["repo"]):
fail(f"{name}: repo must be owner/name")
for key in ("keep", "rev"):
if type(cfg.get(key)) is not int or cfg[key] < 1:
fail(f"{name}: {key} must be a positive number")
if not isinstance(cfg.get("series"), str) or not SERIES.fullmatch(cfg["series"]):
fail(f"{name}: series is not a patch series name")
if cfg.get("profile") not in ("valve", "cachyos"):
fail(f"{name}: profile must be valve or cachyos")
if type(cfg.get("source_match")) is not bool:
fail(f"{name}: source_match must be a boolean")
build_dir = cfg.get("build_dir")
if not isinstance(build_dir, str) or not re.fullmatch(r"(?:/[A-Za-z0-9._-]+)+", build_dir) or ".." in build_dir.split("/"):
fail(f"{name}: build_dir must be the absolute path the release was built at")
wine = cfg.get("wine")
submodules = wine.get("submodules") if isinstance(wine, dict) else None
if not isinstance(submodules, list) or "wine" not in submodules \
or not all(isinstance(item, str) and re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", item) for item in submodules):
fail(f"{name}: wine.submodules must list the submodules to check out, wine among them")
if wine.get("clone") not in CLONES:
fail(f"{name}: wine.clone must be one of {', '.join(CLONES)}")
prep = wine.get("prep")
if not isinstance(prep, str) or (prep and not re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", prep)) or ".." in prep.split("/"):
fail(f"{name}: wine.prep must be a path inside the checkout or empty")
if cfg.get("list") == "tags":
families = cfg.get("families")
if not isinstance(families, dict) or not families:
fail(f"{name}: families must name at least one tag family")
for family, settings in families.items():
if not isinstance(settings, dict) or not isinstance(settings.get("tag"), str) \
or not isinstance(settings.get("prefix"), str) or not re.fullmatch(r"[A-Za-z0-9._-]+", settings["prefix"]) \
or not isinstance(settings.get("depot_app", ""), str):
fail(f"{name}: family {family} needs a tag prefix and a tag pattern")
compiles(settings["tag"], f"{name}: family {family}")
if not isinstance(cfg.get("version_token"), str) or "{tag}" not in cfg["version_token"]:
fail(f"{name}: version_token must contain {{tag}}")
elif cfg.get("list") == "releases":
if not isinstance(cfg.get("tag"), str) or not isinstance(cfg.get("assets"), list) or not cfg["assets"] \
or not all(isinstance(pattern, str) for pattern in cfg["assets"]):
fail(f"{name}: releases need a tag pattern and asset patterns")
for pattern in (cfg["tag"], *cfg["assets"]):
compiles(pattern, name)
else:
fail(f"{name}: list must be tags or releases")
return data
def natural_key(text: str) -> list[tuple]:
return [(0, int(part), "") if part[0] in "0123456789" else (1, 0, part) for part in re.findall(r"[0-9]+|[^0-9]+", text)]
def candidate(flavor: str, cfg: dict, tag: str, asset: str = "", token: str = "", depot_app: str = "") -> dict:
return {
"key": slug(f"{flavor}-{ARCHIVE.sub('', asset) if asset else tag}"),
"flavor": flavor,
"repo": cfg["repo"],
"tag": tag,
"asset": asset,
"token": token,
"depot_app": depot_app,
"rev": cfg["rev"],
"series": cfg["series"],
"profile": cfg["profile"],
"source_match": cfg["source_match"],
"submodules": " ".join(cfg["wine"]["submodules"]),
"clone": cfg["wine"]["clone"],
"prep": cfg["wine"]["prep"],
"build_dir": cfg["build_dir"],
}
def tag_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
found = []
for family, settings in sorted(cfg["families"].items()):
if ref and not ref.startswith(settings["prefix"]):
continue
refs = gh_api(f"repos/{cfg['repo']}/git/matching-refs/tags/{quote(settings['prefix'])}", paginate=True)
if not isinstance(refs, list):
raise PackError(f"{cfg['repo']}: the tag list is not a list")
pattern = re.compile(settings["tag"])
names = set()
for item in refs:
name = item.get("ref", "") if isinstance(item, dict) else ""
if isinstance(name, str) and name.startswith("refs/tags/") and pattern.search(name[len("refs/tags/"):]):
names.add(name[len("refs/tags/"):])
ordered = sorted(names, key=natural_key, reverse=True)
chosen = [name for name in ordered if name == ref] if ref else ordered[:cfg["keep"]]
for name in chosen:
found.append(candidate(flavor, cfg, name, token=cfg["version_token"].format(tag=name), depot_app=settings.get("depot_app", "")))
return found
def pick_assets(assets: list, patterns: list[re.Pattern]) -> list[str]:
chosen = {}
for asset in assets:
name = asset.get("name") if isinstance(asset, dict) else None
if not isinstance(name, str) or not any(pattern.search(name) for pattern in patterns):
continue
stem = ARCHIVE.sub("", name)
if stem not in chosen or name.endswith(".tar.xz"):
chosen[stem] = name
return sorted(chosen.values())
def release_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
repo = cfg["repo"]
if ref:
release = gh_api(f"repos/{repo}/releases/tags/{quote(ref)}", missing_ok=True)
releases = [release] if release else []
else:
releases = gh_api(f"repos/{repo}/releases?per_page=100")
if not isinstance(releases, list):
raise PackError(f"{repo}: the release list is not a list")
pattern = re.compile(cfg["tag"])
patterns = [re.compile(item) for item in cfg["assets"]]
usable = []
for release in releases:
tag = release.get("tag_name") if isinstance(release, dict) else None
if not isinstance(tag, str) or release.get("draft") or not pattern.search(tag):
continue
assets = pick_assets(release.get("assets") or [], patterns)
if assets:
usable.append((release.get("published_at") or release.get("created_at") or "", natural_key(tag), tag, assets))
usable.sort(reverse=True)
if not ref:
usable = usable[:cfg["keep"]]
return [candidate(flavor, cfg, tag, asset=asset) for _, _, tag, assets in usable for asset in assets]
def built(index: dict) -> dict[tuple[str, str, str], int]:
revs = {}
packs = index.get("packs") if isinstance(index, dict) else None
for pack in packs if isinstance(packs, list) else []:
if not isinstance(pack, dict) or not isinstance(pack.get("source"), dict):
continue
rev = pack.get("rev") if type(pack.get("rev")) is int else 0
key = (pack.get("flavor"), pack["source"].get("ref"), pack["source"].get("asset", ""))
revs[key] = max(revs.get(key, 0), rev)
return revs
def revoked_build(entry: dict, revoked: set[str]) -> bool:
prefixes = {f"{entry['flavor']}-{slug(name)}-" for name in (entry["token"], entry["tag"]) if name}
suffix = f"-r{entry['rev']}"
return any(ident.endswith(suffix) and ident.startswith(prefix) for ident in revoked for prefix in prefixes)
def chosen_flavors(flavors: dict, selected: str) -> list[str]:
return sorted(flavors["flavors"]) if selected == "all" else [selected]
def failure_marker(entry: dict, digests: dict[str, str] | None = None) -> str:
series = entry["series"]
if not SERIES.fullmatch(series):
raise PackError(f"{series!r} is not a patch series name")
digest = digests.get(series) if digests is not None else None
if digest is None:
digest = patch_digest(PATCHES / series)
if digests is not None:
digests[series] = digest
return f"{entry['key']}-r{entry['rev']}-{digest[:12]}{FAILED}"
def discover(flavors: dict, selected: str, ref: str, index: dict, revoked: set[str],
failed: frozenset[str] = frozenset()) -> list[dict]:
names = chosen_flavors(flavors, selected)
done = built(index)
digests = {}
entries = []
seen = set()
matched = False
for flavor in names:
cfg = flavors["flavors"].get(flavor)
if cfg is None:
raise PackError(f"flavors.json has no {flavor}")
found = tag_candidates(flavor, cfg, ref) if cfg["list"] == "tags" else release_candidates(flavor, cfg, ref)
matched = matched or bool(found)
for entry in found:
if entry["key"] in seen:
continue
seen.add(entry["key"])
label = f"{flavor} {entry['tag']}{' ' + entry['asset'] if entry['asset'] else ''}"
if revoked_build(entry, revoked):
print(f"discover: {label}: revoked at rev {entry['rev']}", file=sys.stderr)
elif done.get((flavor, entry["tag"], entry["asset"]), 0) >= entry["rev"]:
print(f"discover: {label}: in the index at rev {entry['rev']}", file=sys.stderr)
elif not ref and failure_marker(entry, digests) in failed:
print(f"discover: {label}: failed at rev {entry['rev']} with this patch series in the last "
f"{RETRY_AFTER.days} days; skipped (bump rev, change the series or build the tag by name)", file=sys.stderr)
else:
print(f"discover: {label}: to build", file=sys.stderr)
entries.append(entry)
if ref and not matched:
raise PackError(f"no {'' if selected == 'all' else selected + ' '}build is tagged {ref}")
return entries
def vdf(text: str) -> dict:
stack = [{}]
key = None
for match in VDF_TOKEN.finditer(text):
word, brace = match.group(1), match.group(2)
if brace == "{":
if key is None:
raise PackError("app info: a block has no name")
child = {}
stack[-1][key] = child
stack.append(child)
key = None
elif brace == "}":
if len(stack) == 1:
raise PackError("app info: unbalanced braces")
stack.pop()
key = None
elif key is None:
key = word
else:
stack[-1][key] = word
key = None
if len(stack) != 1:
raise PackError("app info: unbalanced braces")
return stack[0]
def steam_builds(text: str, apps: list[str]) -> dict[str, dict]:
found = {}
text = ANSI.sub("", text)
starts = [(m.start(), m.group(1)) for m in APP_HEADER.finditer(text)]
for number, (start, app) in enumerate(starts):
end = starts[number + 1][0] if number + 1 < len(starts) else len(text)
if app not in apps:
continue
body = text[text.find("\n", start) + 1:end]
info = vdf(body).get(app)
depots = info.get("depots") if isinstance(info, dict) else None
public = depots.get("branches", {}).get("public") if isinstance(depots, dict) else None
if not isinstance(public, dict) or not str(public.get("buildid", "")).isdigit():
continue
manifests = sorted(depot["manifests"]["public"]["gid"] for name, depot in depots.items()
if name.isdigit() and isinstance(depot, dict) and isinstance(depot.get("manifests"), dict)
and isinstance(depot["manifests"].get("public"), dict) and depot["manifests"]["public"].get("gid"))
found[app] = {"buildid": public["buildid"], "built": public.get("timebuildupdated", ""), "manifests": manifests}
return found
def steam_apps(flavors: dict) -> list[str]:
return sorted({family["depot_app"] for cfg in flavors["flavors"].values() if cfg["list"] == "tags"
for family in cfg["families"].values() if family.get("depot_app")})
def watch(flavors: dict, previous: dict, steam: dict[str, dict] | None) -> tuple[dict, list[str], list[str]]:
state = {}
changed = []
notes = []
for flavor in sorted(flavors["flavors"]):
cfg = flavors["flavors"][flavor]
found = tag_candidates(flavor, cfg, "") if cfg["list"] == "tags" else release_candidates(flavor, cfg, "")
builds = sorted(f"{entry['tag']} {entry['asset']}".strip() for entry in found)
ships = {}
if cfg["list"] == "tags" and steam is not None:
for family in cfg["families"].values():
app = family.get("depot_app")
if app and app in steam:
ships[app] = steam[app]
old = previous.get(flavor) if isinstance(previous.get(flavor), dict) else {}
if steam is None and isinstance(old.get("steam"), dict):
ships = old["steam"]
mark = {"builds": builds, "rev": cfg["rev"], "series": cfg["series"], "patch": patch_digest(PATCHES / cfg["series"])}
fingerprint = hashlib.sha256(json.dumps(mark, sort_keys=True).encode()).hexdigest()
state[flavor] = {**mark, "fingerprint": fingerprint, "steam": ships}
if old.get("fingerprint") != fingerprint:
changed.append(flavor)
fresh = sorted(set(builds) - set(old.get("builds") or []))
notes.append(f"{flavor}: {', '.join(fresh) if fresh else 'the patch series or rev changed'}")
for app, ship in sorted(ships.items()):
before = (old.get("steam") or {}).get(app) if isinstance(old.get("steam"), dict) else None
if isinstance(before, dict) and before.get("buildid") != ship.get("buildid"):
notes.append(f"{flavor}: Steam app {app} moved from build {before.get('buildid')} to {ship.get('buildid')}")
if flavor not in changed:
changed.append(flavor)
return state, changed, notes
def asset_text(repo: str, asset_id, what: str) -> str:
result = gh("api", "-H", "Accept: application/octet-stream", f"repos/{repo}/releases/assets/{asset_id}")
if result.returncode != 0:
raise PackError(f"cannot download {what}: {result.stderr.strip()}")
return result.stdout
def release_index(repo: str, tag: str) -> dict:
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True)
if release is None:
return {}
for asset in release.get("assets") or []:
if isinstance(asset, dict) and asset.get("name") == "index.json":
text = asset_text(repo, asset["id"], f"the index of {repo}")
try:
return json.loads(text)
except ValueError as e:
raise PackError(f"the index of {repo} is not JSON: {e}") from e
return {}
def published(repo: str, flavors: dict, selected: str, index: dict) -> list[dict]:
packs = index.get("packs") if isinstance(index, dict) else None
listed = {pack.get("id") for pack in packs if isinstance(pack, dict)} if isinstance(packs, list) else set()
found = []
for flavor in chosen_flavors(flavors, selected):
cfg = flavors["flavors"].get(flavor)
tag = f"droiddeck-esync-{flavor}"
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if cfg else None
if not isinstance(release, dict) or type(release.get("id")) is not int:
continue
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
if not isinstance(assets, list):
raise PackError(f"{repo} {tag}: the asset list is not a list")
for asset in assets:
name = asset.get("name") if isinstance(asset, dict) else None
if not isinstance(name, str) or not name.endswith(".json"):
continue
ident = name[:-len(".json")]
rev = PACK_REV.search(ident)
if ident in listed or not rev or int(rev.group(1)) < cfg["rev"]:
continue
try:
entry = json.loads(asset_text(repo, asset.get("id"), f"{tag} {name}"))
except ValueError:
entry = None
if not isinstance(entry, dict) or entry.get("id") != ident or entry.get("flavor") != flavor:
print(f"discover: {tag} {name} is not a pack entry", file=sys.stderr)
continue
print(f"discover: {tag} {name}: published, not in the index yet", file=sys.stderr)
found.append(entry)
return found
def parse_time(value) -> datetime | None:
if not isinstance(value, str):
return None
try:
moment = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)
def recent_failures(repo: str, flavors: dict, selected: str, now: datetime) -> frozenset[str]:
found = set()
for flavor in chosen_flavors(flavors, selected):
tag = f"droiddeck-esync-{flavor}"
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if flavor in flavors["flavors"] else None
if not isinstance(release, dict) or type(release.get("id")) is not int:
continue
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
if not isinstance(assets, list):
raise PackError(f"{repo} {tag}: the asset list is not a list")
for asset in assets:
name = asset.get("name") if isinstance(asset, dict) else None
if not isinstance(name, str) or not name.endswith(FAILED):
continue
moment = parse_time(asset.get("updated_at")) or parse_time(asset.get("created_at"))
if moment is not None and now - moment < RETRY_AFTER:
found.add(name)
return frozenset(found)
def failed_builds(matrix: dict, directory: Path) -> list[tuple[str, str]]:
rows = matrix.get("include") if isinstance(matrix, dict) else None
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
raise PackError("the build matrix has no include list")
made = {row["key"] for row in rows if (directory / f"{ARTIFACT_PREFIX}{row['key']}").is_dir()}
digests = {}
return [(row["flavor"], failure_marker(row, digests)) for row in rows if row["key"] not in made]
def artifact_packs(matrix: dict, directory: Path) -> list[tuple[str, Path, str]]:
rows = matrix.get("include") if isinstance(matrix, dict) else None
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
raise PackError("the build matrix has no include list")
builds = {row["key"]: row for row in rows}
found = []
for folder in sorted(directory.iterdir()) if directory.is_dir() else []:
row = builds.get(folder.name[len(ARTIFACT_PREFIX):]) if folder.name.startswith(ARTIFACT_PREFIX) else None
if row is None or folder.is_symlink() or not folder.is_dir():
raise PackError(f"{folder.name} is not the artifact of a build in this run")
names = sorted(path.name for path in folder.iterdir())
entries = [name for name in names if name.endswith(".json")]
if len(entries) != 1 or names != sorted([entries[0], entries[0][:-len(".json")] + ".tzst"]):
raise PackError(f"{folder.name} must hold exactly <id>.json and <id>.tzst")
ident = entries[0][:-len(".json")]
entry_path, archive = folder / entries[0], folder / f"{ident}.tzst"
if entry_path.is_symlink() or archive.is_symlink() or not entry_path.is_file() or not archive.is_file():
raise PackError(f"{folder.name} holds links or special files")
try:
data = json.loads(entry_path.read_text())
except ValueError as e:
raise PackError(f"{entry_path}: {e}") from e
entry = check_entry(data, str(entry_path))
source = data.get("source") if isinstance(data.get("source"), dict) else {}
if entry["id"] != ident or entry["flavor"] != row["flavor"] or entry["rev"] != row["rev"] \
or source.get("ref") != row["tag"] or source.get("asset", "") != row["asset"]:
raise PackError(f"{entry_path} is not the pack of {row['flavor']} {row['tag']} at rev {row['rev']}")
if archive.stat().st_size != entry["asset"]["size"] or sha256_file(archive) != entry["asset"]["sha256"]:
raise PackError(f"{archive} is not the asset its entry describes")
found.append((row["flavor"], folder, ident))
return found
def emit(entries: list[dict], unindexed: int = 0) -> None:
matrix = json.dumps({"include": entries}, sort_keys=True, separators=(",", ":"))
path = os.environ.get("GITHUB_OUTPUT")
if path:
with open(path, "a") as stream:
stream.write(f"matrix={matrix}\ncount={len(entries)}\nunindexed={unindexed}\n")
print(matrix)
def make_variables(text: str) -> tuple[dict[str, str], list[tuple[str, str]]]:
variables = {}
assignments = []
for raw in text.splitlines():
match = ASSIGNMENT.fullmatch(raw.split("#", 1)[0].strip())
if match:
name, value = match.group(1), match.group(2).strip()
variables.setdefault(name, value)
assignments.append((name, value))
return variables, assignments
def expand(value: str, variables: dict[str, str], depth: int = 0) -> str:
if depth > 16:
raise PackError("Makefile variables nest too deeply")
def lookup(match: re.Match) -> str:
if match.group(1) not in variables:
raise PackError(f"$({match.group(1)}) is not set in the Makefile")
return expand(variables[match.group(1)], variables, depth + 1)
return REFERENCE.sub(lookup, value)
def sdk_image(text: str, variable: str = "STEAMRT_IMAGE", match: str = "arm64-llvm") -> str:
variables, assignments = make_variables(text)
problem = None
for name, value in assignments:
if not name.startswith(variable):
continue
try:
image = expand(value, variables)
except PackError as e:
problem = problem or e
continue
if match not in image:
continue
if not IMAGE.fullmatch(image):
raise PackError(f"{name} = {image!r} is not an image reference")
return image
raise problem or PackError(f"no {variable} assignment names an {match} image")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Find the Proton builds that have no droiddeck-esync pack yet, as a GitHub Actions matrix.")
parser.add_argument("--flavors", default=str(FLAVORS_FILE))
commands = parser.add_subparsers(dest="command", required=True)
matrix = commands.add_parser("matrix")
matrix.add_argument("--flavor", default="all", choices=("all", *FLAVORS))
matrix.add_argument("--ref", default="")
matrix.add_argument("--index", default="")
matrix.add_argument("--index-release", default="")
matrix.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY", "Droid-Deck/DroidDeck"))
matrix.add_argument("--revoked", default=str(REVOKED))
image = commands.add_parser("sdk-image")
image.add_argument("makefile")
artifacts = commands.add_parser("packs")
artifacts.add_argument("--matrix", required=True)
artifacts.add_argument("directory")
failures = commands.add_parser("failures")
failures.add_argument("--matrix", required=True)
failures.add_argument("directory")
commands.add_parser("steam-apps")
watching = commands.add_parser("watch")
watching.add_argument("--state", default="")
watching.add_argument("--steam-info", default="")
watching.add_argument("--out", required=True)
args = parser.parse_args(argv)
try:
flavors = load_flavors(Path(args.flavors))
if args.command == "sdk-image":
settings = flavors["sdk_image"]
print(sdk_image(Path(args.makefile).read_text(), settings["variable"], settings["match"]))
return 0
if args.command == "packs":
for flavor, folder, ident in artifact_packs(json.loads(args.matrix), Path(args.directory)):
print(f"{flavor}\t{folder}\t{ident}")
return 0
if args.command == "failures":
for flavor, marker in failed_builds(json.loads(args.matrix), Path(args.directory)):
print(f"{flavor}\t{marker}")
return 0
if args.command == "steam-apps":
print(" ".join(steam_apps(flavors)))
return 0
if args.command == "watch":
previous = {}
if args.state and Path(args.state).is_file():
try:
previous = json.loads(Path(args.state).read_text())
except ValueError:
print("discover: the previous watch state is not JSON; every flavor counts as changed", file=sys.stderr)
previous = previous.get("flavors", {}) if isinstance(previous, dict) and previous.get("format") == 1 else {}
steam = None
if args.steam_info:
try:
steam = steam_builds(Path(args.steam_info).read_text(errors="replace"), steam_apps(flavors))
except (OSError, PackError) as e:
print(f"discover: Steam app info unusable ({e}); watching the source tags only", file=sys.stderr)
if steam is not None and len(steam) != len(steam_apps(flavors)):
print(f"discover: Steam app info covers {sorted(steam)} of {steam_apps(flavors)}", file=sys.stderr)
state, changed, notes = watch(flavors, previous, steam)
Path(args.out).write_text(json.dumps({"format": 1, "flavors": state}, indent=1, sort_keys=True) + "\n")
for note in notes:
print(f"discover: {note}", file=sys.stderr)
path = os.environ.get("GITHUB_OUTPUT")
if path:
with open(path, "a") as stream:
stream.write(f"changed={' '.join(changed)}\n")
print(" ".join(changed))
return 0
if args.index and args.index_release:
raise PackError("pass --index or --index-release, not both")
failed = frozenset()
unindexed = 0
if args.index:
path = Path(args.index)
index = json.loads(path.read_text()) if path.exists() else {}
elif args.index_release:
if not REPO.fullmatch(args.repo):
raise PackError(f"{args.repo!r} is not owner/name")
index = release_index(args.repo, args.index_release)
listed = index.get("packs") if isinstance(index, dict) and isinstance(index.get("packs"), list) else []
waiting = published(args.repo, flavors, args.flavor, index)
unindexed = len(waiting)
index = {"packs": [*listed, *waiting]}
failed = recent_failures(args.repo, flavors, args.flavor, datetime.now(timezone.utc))
else:
index = {}
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
emit(discover(flavors, args.flavor, args.ref.strip(), index, revoked, failed), unindexed)
except (PackError, OSError, ValueError) as e:
print(f"discover: {e}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+84
View File
@@ -0,0 +1,84 @@
{
"format": 1,
"sdk_image": {
"file": "Makefile.in",
"variable": "STEAMRT_IMAGE",
"match": "arm64-llvm"
},
"flavors": {
"valve": {
"repo": "ValveSoftware/Proton",
"list": "tags",
"families": {
"experimental": {
"prefix": "experimental-11.0-",
"tag": "^experimental-11\\.0-\\d{8}[a-z]?$",
"depot_app": "4427310"
},
"proton": {
"prefix": "proton-11.0-",
"tag": "^proton-11\\.0-\\d+[a-z]?$",
"depot_app": "4628740"
}
},
"version_token": "{tag}-arm64",
"keep": 2,
"rev": 1,
"series": "valve-11",
"profile": "valve",
"source_match": true,
"build_dir": "/builds/proton/proton/build-dir",
"wine": {
"submodules": [
"wine"
],
"clone": "shallow",
"prep": ""
}
},
"ge": {
"repo": "GloriousEggroll/proton-ge-custom",
"list": "releases",
"tag": "^GE-Proton11-\\d+$",
"assets": [
"aarch64\\.tar\\.(gz|xz)$",
"arm64\\.tar\\.(gz|xz)$"
],
"keep": 3,
"rev": 1,
"series": "valve-11",
"profile": "valve",
"source_match": false,
"build_dir": "/home/tcrider/build",
"wine": {
"submodules": [
"wine",
"wine-staging"
],
"clone": "treeless",
"prep": "patches/protonprep-valve-staging.sh"
}
},
"cachyos": {
"repo": "CachyOS/proton-cachyos",
"list": "releases",
"tag": "^cachyos-11\\.0-",
"assets": [
"arm64\\.tar\\.(gz|xz)$"
],
"keep": 3,
"rev": 1,
"series": "cachyos-11",
"profile": "cachyos",
"source_match": false,
"build_dir": "/home/runner/work/proton-cachyos/proton-cachyos/build",
"wine": {
"submodules": [
"wine"
],
"clone": "shallow",
"prep": ""
}
}
}
}
+4
View File
@@ -0,0 +1,4 @@
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFyqZQAMh3g3nsEGLU+kBGVPBEkXr
Dv5F3kPtuQ+FV1DkTCGmYUQ1LS9wJtHGt0Ht4wToHGGEp9yzDibYZ4/zhg==
-----END PUBLIC KEY-----
+285
View File
@@ -0,0 +1,285 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
import re
import subprocess
import sys
import tempfile
import time
from pathlib import Path
from make_pack import (
BINARIES,
EPOCH,
FLAVORS,
HERE,
NTDLL,
PACK_ID,
REVOKED,
SHA256,
WINESERVER,
PackError,
compress,
dump_json,
read_revoked,
sha256_bytes,
sha256_file,
tar_bytes,
write_atomic,
)
PUBKEY = HERE / "index-key.pub"
SCHEMA = 1
INDEX_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
def check_entry(entry, origin: str) -> dict:
def fail(why: str):
raise PackError(f"{origin}: {why}")
if not isinstance(entry, dict):
fail("is not an object")
ident = entry.get("id")
if not isinstance(ident, str) or not PACK_ID.fullmatch(ident):
fail(f"{ident!r} is not a pack id")
if entry.get("flavor") not in FLAVORS:
fail(f"unknown flavor {entry.get('flavor')!r}")
if type(entry.get("rev")) is not int or entry["rev"] < 1:
fail("rev is not a positive number")
version, line = entry.get("version"), entry.get("version_line")
fields = line.split() if isinstance(line, str) else []
if not isinstance(version, str) or len(fields) < 2 or fields[1] != version or not EPOCH.fullmatch(fields[0]):
fail("version and version_line disagree")
for field in ("stock", "files"):
value = entry.get(field)
if not isinstance(value, dict) or set(value) != set(BINARIES) or not all(isinstance(v, str) and SHA256.fullmatch(v) for v in value.values()):
fail(f"{field} must give the sha256 of {' and '.join(BINARIES)}")
if not isinstance(entry.get("exports"), str) or not SHA256.fullmatch(entry["exports"]):
fail("exports is not a sha256")
if type(entry.get("source_match")) is not bool:
fail("source_match is not a boolean")
asset = entry.get("asset")
if not isinstance(asset, dict) or not isinstance(asset.get("sha256"), str) or not SHA256.fullmatch(asset["sha256"]) \
or type(asset.get("size")) is not int or asset["size"] <= 0:
fail("asset needs its sha256 and size")
clean = {key: entry[key] for key in INDEX_KEYS if key in entry}
clean["asset"] = {"sha256": asset["sha256"], "size": asset["size"]}
return clean
def load_entries(directory: Path) -> list[dict]:
if not directory.is_dir():
raise PackError(f"{directory} is not a directory")
entries = []
for path in sorted(directory.glob("*.json")):
try:
data = json.loads(path.read_text())
except ValueError as e:
raise PackError(f"{path}: {e}") from e
entry = check_entry(data, str(path))
if path.stem != entry["id"]:
raise PackError(f"{path} holds the entry of {entry['id']}")
entries.append(entry)
return entries
def load_previous(path: Path) -> tuple[int, list[dict], list]:
try:
data = json.loads(path.read_text())
except (OSError, ValueError) as e:
raise PackError(f"cannot read the previous index {path}: {e}") from e
if not isinstance(data, dict) or data.get("schema") != SCHEMA or type(data.get("generated")) is not int \
or not isinstance(data.get("packs"), list):
raise PackError(f"{path} is not a schema {SCHEMA} index")
packs = []
for number, pack in enumerate(data["packs"]):
entry = check_entry(pack, f"{path} pack {number}")
entry["revoked"] = pack.get("revoked") is True
packs.append(entry)
return data["generated"], packs, data["packs"]
def epoch(entry: dict) -> int:
return int(entry["version_line"].split()[0])
def rank(entry: dict) -> tuple:
return entry["rev"], epoch(entry), entry["id"]
def choose(entries: list[dict], previous: list[dict], revoked: set[str]) -> list[dict]:
known = {entry["id"]: entry for entry in entries}
for entry in previous:
known.setdefault(entry["id"], entry)
active = {}
withdrawn = []
for entry in known.values():
if entry["id"] in revoked:
withdrawn.append(dict(entry, revoked=True))
continue
key = (entry["stock"][NTDLL], entry["stock"][WINESERVER], entry["version"])
if key not in active or rank(entry) > rank(active[key]):
active[key] = dict(entry, revoked=False)
return sorted([*active.values(), *withdrawn], key=lambda entry: entry["id"])
def build_index(packs: list[dict], base_url: str, generated: int) -> dict:
listed = []
for pack in packs:
entry = dict(pack)
entry["asset"] = {
"url": f"{base_url}/droiddeck-esync-{pack['flavor']}/{pack['id']}.tzst",
"sha256": pack["asset"]["sha256"],
"size": pack["asset"]["size"],
}
listed.append(entry)
return {"schema": SCHEMA, "generated": generated, "packs": listed}
def family(entry: dict) -> tuple[str, str, str]:
token = entry["version"]
major = re.search(r"[0-9]+", token)
return entry["flavor"], re.match(r"[A-Za-z]*", token).group(0).lower(), major.group(0) if major else ""
def bundle_choice(packs: list[dict], available: set[str] | None = None) -> list[dict]:
newest = {}
for pack in packs:
if pack["revoked"] or (available is not None and pack["id"] not in available):
continue
key = family(pack)
if key not in newest or (epoch(pack), pack["rev"], pack["id"]) > (epoch(newest[key]), newest[key]["rev"], newest[key]["id"]):
newest[key] = pack
return sorted(newest.values(), key=lambda pack: pack["id"])
def openssl(*args: str) -> subprocess.CompletedProcess:
try:
return subprocess.run(["openssl", *args], capture_output=True, text=True, check=False)
except FileNotFoundError as e:
raise PackError("the openssl command is not installed") from e
def verified(index: Path, signature: Path, pubkey: Path) -> bool:
result = openssl("dgst", "-sha256", "-verify", str(pubkey), "-signature", str(signature), str(index))
return result.returncode == 0 and "Verified OK" in result.stdout
def trusted_previous(path: Path, signature: Path | None, pubkey: Path) -> tuple[int, list[dict], list | None]:
if signature is not None and not (path.is_file() and signature.is_file() and verified(path, signature, pubkey)):
print(f"make_index: the previous index {path} does not verify with {signature} and {pubkey}; it is ignored", file=sys.stderr)
return 0, [], None
return load_previous(path)
def sign(index: Path, signature: Path, key: Path | None, pubkey: Path) -> None:
with tempfile.TemporaryDirectory() as tmp:
if key is None:
pem = os.environ.get("SYNC_INDEX_KEY", "")
if not pem.strip():
raise PackError("no signing key: pass --key or set SYNC_INDEX_KEY")
key = Path(tmp) / "key.pem"
with os.fdopen(os.open(key, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as stream:
stream.write(pem if pem.endswith("\n") else pem + "\n")
result = openssl("dgst", "-sha256", "-sign", str(key), "-out", str(signature), str(index))
if result.returncode != 0:
raise PackError(f"openssl could not sign the index: {result.stderr.strip()}")
if not verified(index, signature, pubkey):
raise PackError(f"the signature does not verify with {pubkey}: the signing key is not the one the app trusts")
def bundle_name(generated: int) -> str:
return f"bundle-{dt.datetime.fromtimestamp(generated, dt.timezone.utc):%Y%m%d-%H%M%S}.tzst"
def bundle(out: Path, packs: list[dict], index: bytes, signature: bytes, packs_dir: Path, generated: int) -> tuple[str, str]:
if not packs_dir.is_dir():
raise PackError(f"{packs_dir} is not a directory")
available = {path.name[:-len(".tzst")] for path in packs_dir.glob("*.tzst") if path.is_file()}
members = {"index.json": (index, 0o644), "index.json.sig": (signature, 0o644)}
for pack in bundle_choice(packs, available):
path = packs_dir / f"{pack['id']}.tzst"
if path.stat().st_size != pack["asset"]["size"] or sha256_file(path) != pack["asset"]["sha256"]:
raise PackError(f"{path} is not the asset the index lists for {pack['id']}")
members[f"packs/{pack['id']}.tzst"] = (path.read_bytes(), 0o644)
name = bundle_name(generated)
archive = compress(tar_bytes(members))
write_atomic(out / name, archive)
return name, sha256_bytes(archive)
def report(values: dict[str, str]) -> None:
path = os.environ.get("GITHUB_OUTPUT")
if path:
with open(path, "a") as stream:
for key, value in values.items():
stream.write(f"{key}={value}\n")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Make, sign and bundle the droiddeck-esync pack index from the published pack entries.")
parser.add_argument("--entries", required=True)
parser.add_argument("--base-url", required=True)
parser.add_argument("--key", default="")
parser.add_argument("--pubkey", default=str(PUBKEY))
parser.add_argument("--out", default="")
parser.add_argument("--bundle-packs-dir", default="")
parser.add_argument("--revoked", default=str(REVOKED))
parser.add_argument("--previous", default="")
parser.add_argument("--previous-sig", default="")
parser.add_argument("--list-bundle", action="store_true")
args = parser.parse_args(argv)
try:
base_url = args.base_url.rstrip("/")
if not base_url.startswith("https://"):
raise PackError("--base-url must be an https URL")
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
if args.previous_sig and not args.previous:
raise PackError("--previous-sig needs --previous")
previous_signature = Path(args.previous_sig) if args.previous_sig else None
previous_generated, previous, previous_raw = trusted_previous(Path(args.previous), previous_signature, Path(args.pubkey)) \
if args.previous else (0, [], None)
packs = choose(load_entries(Path(args.entries)), previous, revoked)
if args.list_bundle:
for pack in bundle_choice(packs):
print(f"{pack['flavor']}\t{pack['id']}")
return 0
if not args.out:
raise PackError("--out is required")
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
generated = max(int(time.time()), previous_generated + 1)
index = build_index(packs, base_url, generated)
data = dump_json(index)
changed = bool(packs) if previous_raw is None else previous_raw != index["packs"]
temp_index = out / f".index.json.tmp-{os.getpid()}"
temp_signature = out / f".index.json.sig.tmp-{os.getpid()}"
try:
temp_index.write_bytes(data)
sign(temp_index, temp_signature, Path(args.key) if args.key else None, Path(args.pubkey))
signature = temp_signature.read_bytes()
os.replace(temp_index, out / "index.json")
os.replace(temp_signature, out / "index.json.sig")
finally:
temp_index.unlink(missing_ok=True)
temp_signature.unlink(missing_ok=True)
outputs = {"changed": "true" if changed else "false", "generated": str(generated), "index_sha256": sha256_bytes(data)}
live = sum(1 for pack in packs if not pack["revoked"])
print(f"index.json {outputs['index_sha256']} {live} pack(s), {len(packs) - live} revoked, generated {generated}, {'changed' if changed else 'unchanged'}")
if args.bundle_packs_dir:
name, digest = bundle(out, packs, data, signature, Path(args.bundle_packs_dir), generated)
outputs.update(bundle=name, bundle_sha256=digest)
print(f"{name} {digest}")
report(outputs)
except (PackError, OSError) as e:
print(f"make_index: {e}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+367
View File
@@ -0,0 +1,367 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import functools
import hashlib
import importlib.machinery
import importlib.util
import io
import json
import os
import re
import subprocess
import sys
import tarfile
from pathlib import Path
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[1]
GUEST = ROOT / "tools/linuxfs/overlay/usr/local/bin/droiddeck-esync"
PATCHES = HERE / "patches"
REVOKED = HERE / "revoked.txt"
NTDLL = "files/lib/wine/aarch64-unix/ntdll.so"
WINESERVER = "files/bin-arm64/wineserver"
BINARIES = (NTDLL, WINESERVER)
COPY = ("files/bin-arm64/wine", "files/lib/wine/aarch64-unix/wine", "files/lib/wine/aarch64-unix/wine-preloader")
HEADERS = ("server_protocol.h", "request_handlers.h", "request_trace.h")
FLAVORS = ("valve", "ge", "cachyos")
SOURCE_KEYS = ("repo", "ref", "commit", "wine_commit", "sdk_image")
ENTRY_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
PACK_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}")
SERIES = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")
SHA256 = re.compile(r"[0-9a-f]{64}")
EPOCH = re.compile(r"[0-9]{1,12}")
PROTOCOL = re.compile(rb"^#define SERVER_PROTOCOL_VERSION ([0-9]+)[ \t]*$", re.M)
BUILTIN_ESYNC = b"esync: up and running"
ELF_MAGIC = b"\x7fELF"
GUEST_MODULE = "droiddeck_esync_tools"
class PackError(RuntimeError):
pass
@functools.lru_cache(maxsize=None)
def guest():
loader = importlib.machinery.SourceFileLoader(GUEST_MODULE, str(GUEST))
module = importlib.util.module_from_spec(importlib.util.spec_from_loader(GUEST_MODULE, loader))
sys.modules[GUEST_MODULE] = module
write_bytecode = sys.dont_write_bytecode
sys.dont_write_bytecode = True
try:
loader.exec_module(module)
except Exception as e:
sys.modules.pop(GUEST_MODULE, None)
raise PackError(f"cannot load {GUEST}: {e}") from e
finally:
sys.dont_write_bytecode = write_bytecode
return module
def sha256_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
digest.update(block)
return digest.hexdigest()
def slug(text: str) -> str:
return re.sub(r"[^A-Za-z0-9._-]", "-", text)
def dump_json(value) -> bytes:
return (json.dumps(value, indent=1, sort_keys=True, ensure_ascii=True) + "\n").encode()
def write_atomic(path: Path, data: bytes) -> None:
temp = path.with_name(f".{path.name}.tmp-{os.getpid()}")
try:
with temp.open("wb") as stream:
stream.write(data)
stream.flush()
os.fsync(stream.fileno())
os.replace(temp, path)
except BaseException:
temp.unlink(missing_ok=True)
raise
def read_revoked(path: Path | None) -> set[str]:
if path is None:
return set()
try:
text = path.read_text()
except OSError as e:
raise PackError(f"cannot read the revocation list {path}: {e}") from e
revoked = set()
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
ident = line.split()[0]
if not PACK_ID.fullmatch(ident):
raise PackError(f"{path}: {ident!r} is not a pack id")
revoked.add(ident)
return revoked
def tar_bytes(members: dict[str, tuple[bytes, int]]) -> bytes:
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w", format=tarfile.USTAR_FORMAT) as tar:
for name in sorted(members):
data, mode = members[name]
info = tarfile.TarInfo(name)
info.size = len(data)
info.mode = mode
info.mtime = 0
info.uid = 0
info.gid = 0
info.uname = ""
info.gname = ""
tar.addfile(info, io.BytesIO(data))
return buffer.getvalue()
def zstd(data: bytes, *args: str) -> bytes:
try:
result = subprocess.run(["zstd", "-q", *args], input=data, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
except FileNotFoundError as e:
raise PackError("the zstd command is not installed") from e
if result.returncode != 0:
raise PackError(f"zstd {' '.join(args)} failed: {result.stderr.decode(errors='replace').strip()}")
return result.stdout
def compress(data: bytes) -> bytes:
return zstd(data, "-19", "-T1", "-c")
def unpack(archive: bytes) -> dict[str, tuple[bytes, int]]:
members = {}
try:
with tarfile.open(fileobj=io.BytesIO(zstd(archive, "-d", "-c")), mode="r:") as tar:
for info in tar:
if not info.isreg() or info.name in members:
raise PackError(f"unexpected entry {info.name!r} in the archive")
members[info.name] = (tar.extractfile(info).read(), info.mode)
except tarfile.TarError as e:
raise PackError(f"the archive does not read back: {e}") from e
return members
def binary(base: Path, rel: str) -> Path:
path = base / rel
if path.is_symlink() or not path.is_file():
raise PackError(f"{path} is not a regular file")
with path.open("rb") as stream:
if stream.read(len(ELF_MAGIC)) != ELF_MAGIC:
raise PackError(f"{path} is not an ELF file")
return path
def exports_hash(path: Path) -> str:
try:
value = guest().exports_hash(path)
except PackError:
raise
except Exception as e:
raise PackError(f"{path}: cannot read its exported symbols: {e}") from e
if not isinstance(value, str) or not SHA256.fullmatch(value):
raise PackError(f"{path}: has no exported symbols")
return value
def parse_version_line(line: str) -> tuple[int, str]:
fields = line.split()
if "\n" in line or len(fields) < 2 or not EPOCH.fullmatch(fields[0]):
raise PackError(f"version line {line!r} is not '<build epoch> <version>'")
token = guest().version_token(line)
if not isinstance(token, str) or not token:
raise PackError(f"version line {line!r} has no version token")
return int(fields[0]), token
def parse_source(text: str) -> dict[str, str]:
try:
source = json.loads(text)
except ValueError as e:
raise PackError(f"--source-json is not JSON: {e}") from e
if not isinstance(source, dict) or not all(isinstance(key, str) and isinstance(value, str) for key, value in source.items()):
raise PackError("--source-json must be an object of strings")
missing = [key for key in SOURCE_KEYS if not source.get(key)]
if missing:
raise PackError(f"--source-json lacks {', '.join(missing)}")
return source
def patch_digest(directory: Path) -> str:
patches = sorted(path for path in directory.glob("*.patch") if path.is_file())
if not patches:
raise PackError(f"{directory} holds no *.patch files")
digest = hashlib.sha256()
for path in patches:
digest.update(f"{path.name}\0{sha256_file(path)}\n".encode())
return digest.hexdigest()
def protocol_version(control: Path, patched: Path) -> int:
for name in HEADERS:
before, after = control / name, patched / name
if not before.is_file() or not after.is_file():
raise PackError(f"{name} is missing from the control or the patched build")
if before.read_bytes() != after.read_bytes():
raise PackError(f"the patch changes {name}: the droiddeck-esync patch must not change the wineserver protocol")
found = PROTOCOL.findall((control / HEADERS[0]).read_bytes())
if len(found) != 1:
raise PackError("server_protocol.h does not define SERVER_PROTOCOL_VERSION once")
version = int(found[0])
stamp = control / "protocol"
if stamp.is_file() and stamp.read_text().strip() != str(version):
raise PackError(f"{stamp} disagrees with server_protocol.h ({version})")
return version
def make_pack(
flavor: str,
rev: int,
stock_dir: Path | None,
version_line: str,
control: Path,
patched: Path,
source: dict[str, str],
series: str,
out: Path,
source_match: bool,
patch_dir: Path | None = None,
allow_unreproduced: bool = False,
) -> dict:
if flavor not in FLAVORS:
raise PackError(f"unknown flavor {flavor!r}")
if type(rev) is not int or rev < 1:
raise PackError("--rev must be a positive number")
if not SERIES.fullmatch(series):
raise PackError(f"{series!r} is not a patch series name")
if stock_dir is None and not source_match:
raise PackError("without --stock-dir a pack can only match by source: pass --source-match")
if stock_dir is not None and source_match:
raise PackError("a pack made against stock binaries matches them exactly: drop --source-match")
patch_dir = patch_dir or PATCHES / series
if not patch_dir.is_dir():
raise PackError(f"the patch series {patch_dir} does not exist")
version_line = version_line.strip()
if stock_dir is not None:
stock_line = guest().version_line(stock_dir)
if stock_line and version_line and stock_line != version_line:
raise PackError(f"{stock_dir}/version says {stock_line!r}, not {version_line!r}")
version_line = version_line or stock_line
epoch, token = parse_version_line(version_line)
stock_paths = {rel: binary(stock_dir or control, rel) for rel in BINARIES}
control_paths = {rel: binary(control, rel) for rel in BINARIES}
patched_paths = {rel: binary(patched, rel) for rel in BINARIES}
if BUILTIN_ESYNC in stock_paths[WINESERVER].read_bytes():
raise PackError(f"{stock_paths[WINESERVER]} has esync built in and needs no pack")
stock = {rel: sha256_file(path) for rel, path in stock_paths.items()}
built = {rel: sha256_file(path) for rel, path in control_paths.items()}
files = {rel: sha256_file(path) for rel, path in patched_paths.items()}
for rel in BINARIES:
if files[rel] == built[rel]:
raise PackError(f"the patch left {rel} unchanged")
exports = exports_hash(stock_paths[NTDLL])
control_exports = exports_hash(control_paths[NTDLL])
if exports_hash(patched_paths[NTDLL]) != control_exports:
raise PackError("the patch changes the symbols ntdll.so exports")
if exports != control_exports:
raise PackError("the control build of ntdll.so exports other symbols than the stock one: the wine source does not match this release")
protocol = protocol_version(control, patched)
if stock_dir is not None:
for rel in BINARIES:
state = "reproduces" if stock[rel] == built[rel] else "differs from"
print(f"make_pack: the control build {state} the stock {rel}", file=sys.stderr)
unreproduced = [rel for rel in BINARIES if stock[rel] != built[rel]]
if unreproduced and not allow_unreproduced:
raise PackError(f"the control build does not reproduce the stock {', '.join(unreproduced)}: the patched files would differ "
"from the release in more than the patch (check the build directory and the SDK image, or pass --allow-unreproduced)")
pack_id = f"{flavor}-{slug(token)}-{epoch}-{stock[NTDLL][:12]}-r{rev}"
if not PACK_ID.fullmatch(pack_id):
raise PackError(f"{pack_id!r} is not a valid pack id")
pack = {
"format": 1,
"id": pack_id,
"flavor": flavor,
"rev": rev,
"version": token,
"version_line": version_line,
"stock": stock,
"exports": exports,
"source_match": source_match,
"files": files,
"copy": list(COPY),
"protocol": protocol,
"source": dict(source),
"patch": {"series": series, "sha256": patch_digest(patch_dir)},
}
members = {
"pack.json": (dump_json(pack), 0o644),
NTDLL: (patched_paths[NTDLL].read_bytes(), 0o755),
WINESERVER: (patched_paths[WINESERVER].read_bytes(), 0o755),
}
archive = compress(tar_bytes(members))
if unpack(archive) != members:
raise PackError("the archive does not read back as written")
entry = {key: pack[key] for key in ENTRY_KEYS}
entry["asset"] = {"sha256": sha256_bytes(archive), "size": len(archive)}
entry["revoked"] = False
out.mkdir(parents=True, exist_ok=True)
write_atomic(out / f"{pack_id}.tzst", archive)
write_atomic(out / f"{pack_id}.json", dump_json(entry))
return entry
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Make a droiddeck-esync pack from a control and a patched wine build of one Proton release.")
parser.add_argument("--flavor", required=True, choices=FLAVORS)
parser.add_argument("--rev", required=True, type=int)
parser.add_argument("--stock-dir", default="")
parser.add_argument("--version-line", default="")
parser.add_argument("--control", required=True)
parser.add_argument("--patched", required=True)
parser.add_argument("--source-json", required=True)
parser.add_argument("--patch-series", required=True)
parser.add_argument("--patch-dir", default="")
parser.add_argument("--out", required=True)
parser.add_argument("--source-match", action="store_true")
parser.add_argument("--allow-unreproduced", action="store_true")
args = parser.parse_args(argv)
try:
entry = make_pack(
flavor=args.flavor,
rev=args.rev,
stock_dir=Path(args.stock_dir) if args.stock_dir else None,
version_line=args.version_line,
control=Path(args.control),
patched=Path(args.patched),
source=parse_source(args.source_json),
series=args.patch_series,
out=Path(args.out),
source_match=args.source_match,
patch_dir=Path(args.patch_dir) if args.patch_dir else None,
allow_unreproduced=args.allow_unreproduced,
)
except (PackError, OSError) as e:
print(f"make_pack: {e}", file=sys.stderr)
return 1
print(entry["id"])
return 0
if __name__ == "__main__":
raise SystemExit(main())
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
View File
Whitespace-only changes.
+62
View File
@@ -0,0 +1,62 @@
#!/bin/bash
set -euo pipefail
if [ $# -ne 4 ]; then
echo "usage: $0 <proton-dir> <pack.tzst> <esstress.exe> <work-dir>" >&2
exit 2
fi
tool=$(realpath "$1")
archive=$(realpath "$2")
exe=$(realpath "$3")
work=$(realpath -m "$4")
here=$(cd "$(dirname "$0")" && pwd)
guest=$here/../linuxfs/overlay/usr/local/bin/droiddeck-esync
rm -rf "$work"
mkdir -p "$work/pack"
tar --zstd -xf "$archive" -C "$work/pack"
python3 "$guest" build-shadow "$tool" "$work/pack" "$work/dist" >/dev/null
[ -e "$work/dist/files/bin" ] || ln -s bin-arm64 "$work/dist/files/bin"
export WINEDEBUG=-all WINEDLLOVERRIDES="mscoree,mshtml="
run() {
local mode=$1 bin=$2 want=$3
shift 3
local prefix=$work/prefix-$mode
mkdir -p "$prefix"
(
export WINEPREFIX=$prefix WINESERVER=$bin/wineserver "$@"
timeout "${BOOT_TIMEOUT:-900}" "$bin/wine" wineboot -i > "$work/$mode-boot.log" 2>&1 || true
"$bin/wineserver" -w || true
if ! grep -q "$want" "$work/$mode-boot.log"; then
echo "$mode: wine never said \"$want\""
tail -n 30 "$work/$mode-boot.log"
exit 1
fi
cp "$exe" "$prefix/drive_c/esstress.exe"
status=0
timeout "${STRESS_TIMEOUT:-1800}" "$bin/wine" 'C:\esstress.exe' > "$work/$mode-stress.log" 2>&1 || status=$?
"$bin/wineserver" -k 2>/dev/null || true
if ! grep -q '^esstress:' "$work/$mode-stress.log"; then
echo "$mode: esstress did not finish (status $status)"
tail -n 30 "$work/$mode-stress.log"
exit 1
fi
grep -o '^FAIL [^ ]*:[0-9]*:' "$work/$mode-stress.log" | sort -u > "$work/$mode-failures" || true
echo "$mode: $(grep -E '^esstress:' "$work/$mode-stress.log" | tail -n 1)"
)
}
if ! run stock "$tool/files/bin-arm64" "server-side synchronization" PROTON_NO_NTSYNC=1 WINEESYNC=0 WINEFSYNC=0; then
echo "stock Proton does not run here either; the pack cannot be judged on this machine"
exit 3
fi
run esync "$work/dist/files/bin-arm64" "esync: up and running" PROTON_NO_NTSYNC=1 WINEESYNC=1 WINEFSYNC=0
if [ -s "$work/stock-failures" ]; then
echo "stock fails these as well, so they do not count against the pack: $(tr '\n' ' ' < "$work/stock-failures")"
fi
new=$(comm -13 "$work/stock-failures" "$work/esync-failures")
if [ -n "$new" ]; then
echo "esync: fails checks stock passes:"
grep -F "$new" "$work/esync-stress.log" || echo "$new"
exit 1
fi
+618
View File
@@ -0,0 +1,618 @@
#include <windows.h>
#include <winternl.h>
#include <stdio.h>
#include <stdlib.h>
static int failures;
#define CHECK(c, ...) do { if (!(c)) { failures++; printf("FAIL %s:%d: ", __FILE__, __LINE__); printf(__VA_ARGS__); printf("\n"); fflush(stdout); } } while (0)
static volatile LONG counter;
static HANDLE mutex, sem, ev_auto, ev_manual, start_ev;
static volatile LONG stop;
static DWORD WINAPI mutex_worker(void *arg)
{
int i;
WaitForSingleObject(start_ev, INFINITE);
for (i = 0; i < 20000; i++)
{
DWORD r = WaitForSingleObject(mutex, INFINITE);
LONG v;
if (r != WAIT_OBJECT_0) { CHECK(0, "mutex wait %lu", r); break; }
v = counter;
if (i % 7 == 0) WaitForSingleObject(mutex, 0), ReleaseMutex(mutex);
counter = v + 1;
ReleaseMutex(mutex);
}
return 0;
}
static void test_mutex_contention(void)
{
HANDLE th[8];
int i;
counter = 0;
mutex = CreateMutexA(NULL, FALSE, NULL);
start_ev = CreateEventA(NULL, TRUE, FALSE, NULL);
for (i = 0; i < 8; i++) th[i] = CreateThread(NULL, 0, mutex_worker, NULL, 0, NULL);
SetEvent(start_ev);
WaitForMultipleObjects(8, th, TRUE, INFINITE);
CHECK(counter == 8 * 20000, "mutex counter %ld", counter);
for (i = 0; i < 8; i++) CloseHandle(th[i]);
CloseHandle(mutex);
CloseHandle(start_ev);
printf("mutex_contention ok counter=%ld\n", counter);
}
static volatile LONG produced, consumed;
static DWORD WINAPI sem_producer(void *arg)
{
int i;
for (i = 0; i < 50000; i++)
{
LONG prev;
while (!ReleaseSemaphore(sem, 1, &prev))
{
CHECK(GetLastError() == ERROR_TOO_MANY_POSTS, "release err %lu", GetLastError());
Sleep(0);
}
InterlockedIncrement(&produced);
}
return 0;
}
static DWORD WINAPI sem_consumer(void *arg)
{
for (;;)
{
DWORD r = WaitForSingleObject(sem, 2000);
if (r == WAIT_TIMEOUT) break;
CHECK(r == WAIT_OBJECT_0, "sem wait %lu", r);
InterlockedIncrement(&consumed);
}
return 0;
}
static void test_semaphore(void)
{
HANDLE th[8];
LONG prev;
int i;
produced = consumed = 0;
sem = CreateSemaphoreA(NULL, 0, 16, NULL);
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, sem_producer, NULL, 0, NULL);
for (i = 4; i < 8; i++) th[i] = CreateThread(NULL, 0, sem_consumer, NULL, 0, NULL);
WaitForMultipleObjects(8, th, TRUE, INFINITE);
CHECK(produced == 200000 && consumed == 200000, "sem produced %ld consumed %ld", produced, consumed);
CHECK(!ReleaseSemaphore(sem, 17, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "limit");
CHECK(ReleaseSemaphore(sem, 16, &prev) && prev == 0, "prev %ld", prev);
CHECK(!ReleaseSemaphore(sem, 1, &prev), "over max");
for (i = 0; i < 8; i++) CloseHandle(th[i]);
CloseHandle(sem);
printf("semaphore ok %ld/%ld\n", produced, consumed);
}
static volatile LONG auto_hits;
static DWORD WINAPI auto_waiter(void *arg)
{
while (!stop)
{
DWORD r = WaitForSingleObject(ev_auto, 50);
if (r == WAIT_OBJECT_0) InterlockedIncrement(&auto_hits);
}
return 0;
}
static void test_auto_event(void)
{
HANDLE th[6];
int i;
LONG expected = 0;
stop = 0; auto_hits = 0;
ev_auto = CreateEventA(NULL, FALSE, FALSE, NULL);
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, auto_waiter, NULL, 0, NULL);
for (i = 0; i < 30000; i++)
{
LONG before = auto_hits;
SetEvent(ev_auto);
expected++;
while (auto_hits == before) SwitchToThread();
}
stop = 1;
WaitForMultipleObjects(6, th, TRUE, INFINITE);
CHECK(auto_hits == expected, "auto hits %ld expected %ld", auto_hits, expected);
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto event left signaled");
SetEvent(ev_auto); SetEvent(ev_auto);
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_OBJECT_0, "auto set");
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto coalesce");
for (i = 0; i < 6; i++) CloseHandle(th[i]);
CloseHandle(ev_auto);
printf("auto_event ok hits=%ld\n", auto_hits);
}
static volatile LONG manual_seen;
static DWORD WINAPI manual_waiter(void *arg)
{
DWORD r = WaitForSingleObject(ev_manual, 10000);
if (r == WAIT_OBJECT_0) InterlockedIncrement(&manual_seen);
return 0;
}
static void test_manual_event(void)
{
HANDLE th[16];
int i, round;
ev_manual = CreateEventA(NULL, TRUE, FALSE, NULL);
for (round = 0; round < 50; round++)
{
manual_seen = 0;
ResetEvent(ev_manual);
for (i = 0; i < 16; i++) th[i] = CreateThread(NULL, 0, manual_waiter, NULL, 0, NULL);
Sleep(2);
SetEvent(ev_manual);
WaitForMultipleObjects(16, th, TRUE, INFINITE);
CHECK(manual_seen == 16, "manual round %d seen %ld", round, manual_seen);
for (i = 0; i < 16; i++) CloseHandle(th[i]);
}
for (i = 0; i < 20000; i++)
{
SetEvent(ev_manual);
if (i & 1) ResetEvent(ev_manual);
}
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_TIMEOUT, "manual final reset");
SetEvent(ev_manual);
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays");
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays 2");
CloseHandle(ev_manual);
printf("manual_event ok\n");
}
static HANDLE wa_objs[3];
static volatile LONG wa_count;
static DWORD WINAPI wait_all_worker(void *arg)
{
int i;
for (i = 0; i < 3000; i++)
{
DWORD r = WaitForMultipleObjects(3, wa_objs, TRUE, 5000);
if (r != WAIT_OBJECT_0) { CHECK(0, "wait all %lu", r); break; }
InterlockedIncrement(&wa_count);
ReleaseMutex(wa_objs[0]);
ReleaseSemaphore(wa_objs[1], 1, NULL);
SetEvent(wa_objs[2]);
}
return 0;
}
static void test_wait_all(void)
{
HANDLE th[6], dup[2];
int i;
DWORD r;
wa_count = 0;
wa_objs[0] = CreateMutexA(NULL, FALSE, NULL);
wa_objs[1] = CreateSemaphoreA(NULL, 1, 1, NULL);
wa_objs[2] = CreateEventA(NULL, FALSE, TRUE, NULL);
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, wait_all_worker, NULL, 0, NULL);
WaitForMultipleObjects(6, th, TRUE, INFINITE);
CHECK(wa_count == 18000, "wait all count %ld", wa_count);
dup[0] = wa_objs[2];
DuplicateHandle(GetCurrentProcess(), wa_objs[2], GetCurrentProcess(), &dup[1], 0, FALSE, DUPLICATE_SAME_ACCESS);
r = WaitForMultipleObjects(2, dup, TRUE, 0);
CHECK(r == WAIT_FAILED && GetLastError() == ERROR_INVALID_PARAMETER, "dup wait all %lu %lu", r, GetLastError());
CloseHandle(dup[1]);
for (i = 0; i < 6; i++) CloseHandle(th[i]);
for (i = 0; i < 3; i++) CloseHandle(wa_objs[i]);
printf("wait_all ok %ld\n", wa_count);
}
static DWORD WINAPI hold_and_die(void *arg)
{
WaitForSingleObject((HANDLE)arg, INFINITE);
return 0;
}
static void test_abandoned(void)
{
HANDLE m = CreateMutexA(NULL, FALSE, NULL);
HANDLE th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
HANDLE objs[2];
DWORD r;
WaitForSingleObject(th, INFINITE);
r = WaitForSingleObject(m, 1000);
CHECK(r == WAIT_ABANDONED, "abandoned %lu", r);
CHECK(ReleaseMutex(m), "release after abandon");
th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
WaitForSingleObject(th, INFINITE);
objs[0] = CreateEventA(NULL, TRUE, TRUE, NULL);
objs[1] = m;
r = WaitForMultipleObjects(2, objs, TRUE, 1000);
CHECK(r == WAIT_ABANDONED_0, "abandoned all %lu", r);
ReleaseMutex(m);
r = WaitForMultipleObjects(2, objs, FALSE, 1000);
CHECK(r == WAIT_OBJECT_0, "any after %lu", r);
CloseHandle(th); CloseHandle(objs[0]); CloseHandle(m);
printf("abandoned ok\n");
}
static volatile LONG apc_ran;
static void CALLBACK apc_proc(ULONG_PTR p) { InterlockedIncrement(&apc_ran); }
static HANDLE apc_ready;
static DWORD WINAPI alert_waiter(void *arg)
{
SetEvent(apc_ready);
return WaitForSingleObjectEx((HANDLE)arg, 5000, TRUE);
}
static void test_apc(void)
{
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL);
HANDLE th;
DWORD r, code;
int i;
apc_ready = CreateEventA(NULL, FALSE, FALSE, NULL);
for (i = 0; i < 200; i++)
{
apc_ran = 0;
th = CreateThread(NULL, 0, alert_waiter, ev, 0, NULL);
WaitForSingleObject(apc_ready, INFINITE);
if (i & 1) Sleep(1);
QueueUserAPC(apc_proc, th, 0);
WaitForSingleObject(th, INFINITE);
GetExitCodeThread(th, &code);
CHECK(code == WAIT_IO_COMPLETION && apc_ran == 1, "apc round %d code %lu ran %ld", i, code, apc_ran);
CloseHandle(th);
}
apc_ran = 0;
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
r = SleepEx(1000, TRUE);
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "sleepex %lu %ld", r, apc_ran);
r = SleepEx(10, TRUE);
CHECK(r == 0, "sleepex timeout %lu", r);
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
r = WaitForSingleObjectEx(ev, 0, FALSE);
CHECK(r == WAIT_TIMEOUT, "non alertable %lu", r);
r = WaitForSingleObjectEx(ev, 0, TRUE);
CHECK(r == WAIT_IO_COMPLETION, "alertable pending %lu", r);
CloseHandle(ev);
printf("apc ok\n");
}
static DWORD WINAPI close_waiter(void *arg)
{
return WaitForSingleObject((HANDLE)arg, 300);
}
static void test_close_while_waiting(void)
{
int i;
for (i = 0; i < 300; i++)
{
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), ev2, th;
DWORD code;
th = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
Sleep(0);
CloseHandle(ev);
ev2 = CreateEventA(NULL, TRUE, TRUE, NULL);
WaitForSingleObject(th, INFINITE);
GetExitCodeThread(th, &code);
CHECK(code == WAIT_TIMEOUT || code == WAIT_FAILED || code == WAIT_OBJECT_0, "close round %d code %lu", i, code);
CHECK(WaitForSingleObject(ev2, 0) == WAIT_OBJECT_0, "reused handle state");
CloseHandle(ev2);
CloseHandle(th);
}
printf("close_while_waiting ok\n");
}
static void test_pulse(void)
{
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), th[4];
DWORD code;
int i, woke = 0;
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
Sleep(50);
PulseEvent(ev);
WaitForMultipleObjects(4, th, TRUE, INFINITE);
for (i = 0; i < 4; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left signaled");
CHECK(woke == 4, "pulse woke %d of 4", woke);
CloseHandle(ev);
ev = CreateEventA(NULL, FALSE, FALSE, NULL);
PulseEvent(ev);
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse without waiter left signaled");
th[0] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
th[1] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
Sleep(50);
PulseEvent(ev);
WaitForMultipleObjects(2, th, TRUE, INFINITE);
woke = 0;
for (i = 0; i < 2; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
CHECK(woke == 1, "auto pulse woke %d of 2", woke);
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse left signaled");
CloseHandle(ev);
printf("pulse ok\n");
}
static void test_semantics(void)
{
HANDLE ev = CreateEventA(NULL, TRUE, TRUE, NULL), sem = CreateSemaphoreA(NULL, 2, 3, NULL), h[2];
NTSTATUS (WINAPI *pNtWaitForSingleObject)(HANDLE, BOOLEAN, const LARGE_INTEGER *);
LARGE_INTEGER when;
LONG prev = -1;
DWORD r, start;
int i;
pNtWaitForSingleObject = (void *)GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtWaitForSingleObject");
apc_ran = 0;
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
r = WaitForSingleObjectEx(ev, INFINITE, TRUE);
CHECK(r == WAIT_OBJECT_0 && apc_ran == 0, "object before apc %lu ran %ld", r, apc_ran);
r = SleepEx(0, TRUE);
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "apc after object %lu ran %ld", r, apc_ran);
r = SleepEx(0, TRUE);
CHECK(r == 0, "sleepex zero %lu", r);
ResetEvent(ev);
GetSystemTimeAsFileTime((FILETIME *)&when);
when.QuadPart += 2000000;
start = GetTickCount();
r = pNtWaitForSingleObject(ev, FALSE, &when);
CHECK(r == STATUS_TIMEOUT, "absolute timeout status %lx", r);
CHECK(GetTickCount() - start >= 150 && GetTickCount() - start < 1500, "absolute timeout took %lu", GetTickCount() - start);
CHECK(!ReleaseSemaphore(sem, 2, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "semaphore limit");
CHECK(ReleaseSemaphore(sem, 1, &prev) && prev == 2, "semaphore prev %ld", prev);
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(sem, 0) == WAIT_OBJECT_0, "semaphore take %d", i);
CHECK(WaitForSingleObject(sem, 0) == WAIT_TIMEOUT, "semaphore empty");
h[0] = sem; h[1] = ev;
ReleaseSemaphore(sem, 1, NULL);
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_TIMEOUT, "wait all partial");
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 1, "wait all rollback lost token prev %ld", prev);
SetEvent(ev);
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_OBJECT_0, "wait all ready");
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 0, "wait all consumed prev %ld", prev);
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "manual stays set");
CloseHandle(sem);
CloseHandle(ev);
printf("semantics ok\n");
}
static HANDLE token_sem;
static volatile LONG token_count;
static DWORD WINAPI token_worker(void *arg)
{
HANDLE h[2] = { token_sem, (HANDLE)arg };
int i;
for (i = 0; i < 20000; i++)
{
DWORD r = (i & 1) ? WaitForSingleObject(token_sem, INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
if (r != WAIT_OBJECT_0) { CHECK(0, "token wait %lu", r); break; }
if (InterlockedIncrement(&token_count) > 3) CHECK(0, "token overcommit");
InterlockedDecrement(&token_count);
if (!(i & 1)) SetEvent((HANDLE)arg);
ReleaseSemaphore(token_sem, 1, NULL);
}
return 0;
}
static void test_semaphore_tokens(void)
{
HANDLE th[6], ev = CreateEventA(NULL, FALSE, TRUE, NULL);
LONG prev = -1;
int i;
token_sem = CreateSemaphoreA(NULL, 3, 3, NULL);
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, token_worker, ev, 0, NULL);
WaitForMultipleObjects(6, th, TRUE, INFINITE);
for (i = 0; i < 6; i++) CloseHandle(th[i]);
CHECK(ReleaseSemaphore(token_sem, 0, &prev) && prev == 3, "token final count %ld", prev);
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(token_sem, 0) == WAIT_OBJECT_0, "token drain %d", i);
CHECK(WaitForSingleObject(token_sem, 0) == WAIT_TIMEOUT, "token extra");
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "token event lost");
CloseHandle(token_sem);
CloseHandle(ev);
printf("semaphore_tokens ok\n");
}
static volatile LONG churn_ready;
static DWORD WINAPI churn_mutex(void *arg)
{
HANDLE h[2] = { (HANDLE)arg, ev_manual };
unsigned int i = 0;
if (WaitForMultipleObjects(2, h, TRUE, INFINITE) <= WAIT_ABANDONED_0 + 1) ReleaseMutex(h[0]);
InterlockedIncrement(&churn_ready);
for (;;)
{
DWORD r = (i++ & 1) ? WaitForSingleObject(h[0], INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
if (r == WAIT_OBJECT_0 || r == WAIT_ABANDONED || r == WAIT_ABANDONED_0) ReleaseMutex(h[0]);
}
return 0;
}
static DWORD WINAPI blocked_waiter(void *arg)
{
WaitForSingleObject((HANDLE)arg, 0);
InterlockedIncrement(&churn_ready);
return WaitForSingleObject((HANDLE)arg, INFINITE);
}
static void test_terminate(void)
{
HANDLE th[4], m, ev;
DWORD r, start;
int round, i, stuck = 0;
ev_manual = CreateEventA(NULL, TRUE, TRUE, NULL);
for (round = 0; round < 150; round++)
{
m = CreateMutexA(NULL, FALSE, NULL);
churn_ready = 0;
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, churn_mutex, m, 0, NULL);
while (churn_ready < 4) Sleep(0);
Sleep(round % 4);
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
WaitForMultipleObjects(4, th, TRUE, INFINITE);
r = WaitForSingleObject(m, 3000);
if (r == WAIT_TIMEOUT) stuck++;
else ReleaseMutex(m);
for (i = 0; i < 4; i++) CloseHandle(th[i]);
CloseHandle(m);
}
CHECK(!stuck, "mutex stuck after terminate in %d rounds", stuck);
ev = CreateEventA(NULL, TRUE, FALSE, NULL);
churn_ready = 0;
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, blocked_waiter, ev, 0, NULL);
while (churn_ready < 4) Sleep(0);
Sleep(50);
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
WaitForMultipleObjects(4, th, TRUE, INFINITE);
for (i = 0; i < 4; i++) CloseHandle(th[i]);
start = GetTickCount();
for (i = 0; i < 100; i++) PulseEvent(ev);
CHECK(GetTickCount() - start < 400, "pulse after killed waiters took %lu ms", GetTickCount() - start);
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left event set");
CloseHandle(ev);
CloseHandle(ev_manual);
printf("terminate ok\n");
}
static void test_cross_process(const char *self)
{
HANDLE ev = CreateEventA(NULL, FALSE, FALSE, "Local\\esstress_ev");
HANDLE done = CreateEventA(NULL, TRUE, FALSE, "Local\\esstress_done");
HANDLE sm = CreateSemaphoreA(NULL, 0, 100000, "Local\\esstress_sem");
PROCESS_INFORMATION pi;
STARTUPINFOA si = { sizeof(si) };
char cmd[MAX_PATH + 16];
DWORD r, code;
int i;
sprintf(cmd, "\"%s\" child", self);
CHECK(CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi), "createprocess %lu", GetLastError());
for (i = 0; i < 5000; i++)
{
SetEvent(ev);
r = WaitForSingleObject(sm, 5000);
if (r != WAIT_OBJECT_0) { CHECK(0, "xproc wait %lu at %d", r, i); break; }
}
SetEvent(done);
r = WaitForSingleObject(pi.hProcess, 10000);
CHECK(r == WAIT_OBJECT_0, "child exit wait %lu", r);
GetExitCodeProcess(pi.hProcess, &code);
CHECK(code == 0, "child code %lu", code);
CloseHandle(pi.hProcess); CloseHandle(pi.hThread);
CloseHandle(ev); CloseHandle(done); CloseHandle(sm);
printf("cross_process ok\n");
}
static int child(void)
{
HANDLE ev = OpenEventA(SYNCHRONIZE | EVENT_MODIFY_STATE, FALSE, "Local\\esstress_ev");
HANDLE done = OpenEventA(SYNCHRONIZE, FALSE, "Local\\esstress_done");
HANDLE sm = OpenSemaphoreA(SEMAPHORE_MODIFY_STATE | SYNCHRONIZE, FALSE, "Local\\esstress_sem");
HANDLE objs[2];
objs[0] = done; objs[1] = ev;
if (!ev || !done || !sm) return 1;
for (;;)
{
DWORD r = WaitForMultipleObjects(2, objs, FALSE, 10000);
if (r == WAIT_OBJECT_0) return 0;
if (r != WAIT_OBJECT_0 + 1) return 2;
ReleaseSemaphore(sm, 1, NULL);
}
}
static void test_kill_holder(const char *self)
{
HANDLE m = CreateMutexA(NULL, FALSE, "Local\\esstress_mutex");
PROCESS_INFORMATION pi;
STARTUPINFOA si = { sizeof(si) };
char cmd[MAX_PATH + 16];
DWORD r;
sprintf(cmd, "\"%s\" holder", self);
CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);
Sleep(500);
TerminateProcess(pi.hProcess, 7);
WaitForSingleObject(pi.hProcess, INFINITE);
r = WaitForSingleObject(m, 3000);
CHECK(r == WAIT_ABANDONED, "killed holder %lu", r);
ReleaseMutex(m);
CloseHandle(pi.hProcess); CloseHandle(pi.hThread); CloseHandle(m);
printf("kill_holder ok\n");
}
static int holder(void)
{
HANDLE m = OpenMutexA(SYNCHRONIZE, FALSE, "Local\\esstress_mutex");
WaitForSingleObject(m, INFINITE);
Sleep(INFINITE);
return 0;
}
static void test_many_objects(void)
{
static HANDLE h[20000];
int i, n = 0;
for (i = 0; i < 20000; i++)
{
if (!(h[i] = CreateEventA(NULL, i & 1, i & 2, NULL))) break;
n++;
}
CHECK(n == 20000, "created %d", n);
for (i = 0; i < n; i++) CHECK(WaitForSingleObject(h[i], 0) == ((i & 2) ? WAIT_OBJECT_0 : WAIT_TIMEOUT), "obj %d", i);
for (i = 0; i < n; i++) CloseHandle(h[i]);
for (i = 0; i < 100000; i++)
{
HANDLE e = CreateEventA(NULL, FALSE, TRUE, NULL);
if (WaitForSingleObject(e, 0) != WAIT_OBJECT_0) { CHECK(0, "churn %d", i); CloseHandle(e); break; }
CloseHandle(e);
}
printf("many_objects ok\n");
}
static void test_timers_and_threads(void)
{
HANDLE t = CreateWaitableTimerA(NULL, FALSE, NULL);
LARGE_INTEGER due;
DWORD start = GetTickCount(), r;
int i;
due.QuadPart = -10 * 1000 * 20;
SetWaitableTimer(t, &due, 10, NULL, NULL, FALSE);
for (i = 0; i < 20; i++)
{
r = WaitForSingleObject(t, 1000);
CHECK(r == WAIT_OBJECT_0, "timer %d %lu", i, r);
}
CHECK(GetTickCount() - start < 2000, "timer slow %lu", GetTickCount() - start);
CancelWaitableTimer(t);
CloseHandle(t);
r = WaitForSingleObject(GetCurrentProcess(), 10);
CHECK(r == WAIT_TIMEOUT, "process self %lu", r);
printf("timers ok\n");
}
int main(int argc, char **argv)
{
if (argc > 1 && !strcmp(argv[1], "child")) return child();
if (argc > 1 && !strcmp(argv[1], "holder")) return holder();
test_mutex_contention();
test_semaphore();
test_auto_event();
test_manual_event();
test_wait_all();
test_abandoned();
test_apc();
test_close_while_waiting();
test_pulse();
test_semantics();
test_semaphore_tokens();
if (argc > 1 && !strcmp(argv[1], "terminate")) test_terminate();
test_cross_process(argv[0]);
test_kill_holder(argv[0]);
test_many_objects();
test_timers_and_threads();
printf("esstress: %d failures\n", failures);
return failures ? 1 : 0;
}
File diff suppressed because it is too large. Load diff