mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-21 13:55:58 +00:00
Add the droiddeck-esync pack pipeline (#189)
Builds esync packs for new Valve, GE and CachyOS Proton 11 releases on arm64 runners, smoke-tests them against stock with esstress, and on main publishes each pack and a signed index as non-latest releases. A watcher checks Steam and the GE/CachyOS releases every 20 minutes and starts a build for new ones. The app side lands separately with feat/userspace-esync-ntsync.
This commit is contained in:
1 parent
c535f6ef6e
commit
e7b6876877
16 files changed
+9000
No files matched your search
@@ -0,0 +1,170 @@
|
||||
# droiddeck-esync packs
|
||||
|
||||
DroidDeck gives Proton three in-process sync backends, picked with the Wine sync tabs in the
|
||||
Steam settings' Games group: ntsync, fsync (the default), esync and wineserver. The wineserver tab
|
||||
sets `PROTON_NO_ESYNC=1`, `PROTON_NO_FSYNC=1` and `PROTON_NO_NTSYNC=1` (unless a game's own
|
||||
environment sets them), so every sync object goes through the server.
|
||||
|
||||
- droiddeck-esync: eventfd and shared memory sync. Stock Wine 11 / Proton 11 builds
|
||||
have no eventfd-based sync, so the patch series under `patches/<series>` adds it to wineserver
|
||||
and the unix side of ntdll. A Proton build without a pack (a release newer than the published
|
||||
packs, or one whose pack failed to install) runs stock with droiddeck-fsync standing in, or
|
||||
droiddeck-ntsync where fsync is off for the game (`PROTON_NO_FSYNC=1`, or one of the games
|
||||
Proton itself keeps off fsync and esync), so it still gets in-process sync; `PROTON_NO_NTSYNC=1`
|
||||
as well leaves it on wineserver-only sync. A Proton with esync built in (GE, cachyos) runs its
|
||||
own esync, and a game that turns esync off moves to droiddeck-fsync or droiddeck-ntsync there
|
||||
too.
|
||||
- droiddeck-fsync: Proton's own fsync on the stock binaries. fsync waits with `futex_waitv(2)`,
|
||||
which the app sandbox refuses, so `libblsession.so` (`BL_FSYNC=1`) answers that system call in
|
||||
userspace: every waiter registers the futexes it waits on in a shared table next to the fsync
|
||||
shared memory and sleeps on a futex of its own, and every `FUTEX_WAKE` on the fsync shared memory
|
||||
wakes the waiters registered on that word. No pack is involved, so it keeps working the moment
|
||||
Steam updates Proton. It is the default (the fsync tab, `BL_FSYNC_FIRST=1`, also per game),
|
||||
ahead of droiddeck-esync; on the esync tab it stands in where no pack fits, and it also takes
|
||||
over for a game whose environment turns esync off (`PROTON_NO_ESYNC=1`, `WINEESYNC=0`, or too
|
||||
few file descriptors); where fsync is off as well, such a game gets droiddeck-ntsync, with the
|
||||
pack's wineserver when a pack fits. A launch into a prefix whose wineserver is still running
|
||||
keeps the sync choice that server was started with (`prefixes.json` in the store), because a
|
||||
Wine client that disagrees with its server about fsync exits. It inherits stock fsync's behaviour, the same as on a Linux kernel:
|
||||
a wait-all on a mutex abandoned by a dead thread is not satisfied, a pending user APC is
|
||||
delivered ahead of an object that is already signaled, a mutex owned by a process killed with
|
||||
`TerminateProcess` is not abandoned, and `PulseEvent` can miss a waiter that has not run yet.
|
||||
- droiddeck-ntsync: a userspace implementation of the `/dev/ntsync` interface in
|
||||
`libblsession.so` (`BL_SYNC=1`), which stock Proton 11 builds use as they would the kernel
|
||||
driver. It is the ntsync tab. A pack still supplies the binaries when one fits, and its
|
||||
wineserver then picks ntsync over esync; a Proton whose wineserver has no ntsync client (no
|
||||
`/dev/ntsync` in it) gets droiddeck-fsync instead.
|
||||
|
||||
wineserver picks the backend at startup: `PROTON_NO_NTSYNC` unset -> `/dev/ntsync`
|
||||
(droiddeck-ntsync, or a kernel driver the device provides) -> fsync (droiddeck-fsync, or a kernel
|
||||
and sandbox that allow `futex_waitv`) -> droiddeck-esync (needs `WINEESYNC` nonzero) ->
|
||||
server-side sync. When a pack is in use and droiddeck-esync is on, the guest's `droiddeck-esync`
|
||||
script also sets `PROTON_NO_FSYNC=1` unless the game's environment sets it, so droiddeck-esync
|
||||
comes before fsync unless droiddeck-fsync was chosen. A wineserver from a pack reports the
|
||||
backend on stderr as `droiddeck-ntsync: up and running.` or `droiddeck-esync: up and running.`;
|
||||
with droiddeck-fsync the stock wineserver prints `fsync: up and running.` after
|
||||
`droiddeck-fsync: up and running.`. A pack's wineserver also abandons, when a process dies, every
|
||||
mutex still owned by one of its threads, and, once a thread killed by `TerminateThread` has really
|
||||
exited, every mutex it still owns. A killed thread keeps running until the kill signal reaches it
|
||||
and can take a mutex after the server abandoned its mutexes; without these passes that mutex would
|
||||
stay locked for good (they cover droiddeck-ntsync and kernel ntsync as well). The ntdll side
|
||||
reads, writes and polls its eventfds with raw system calls, so no `LD_PRELOAD` wrapper of
|
||||
`read`/`write`/`poll` (the guest preloads several) sits in the sync path or in the unwind of a
|
||||
thread Wine kills.
|
||||
|
||||
The patch changes no wineserver request: `make_pack.py` refuses a build whose
|
||||
`server_protocol.h`, `request_handlers.h` or `request_trace.h` differ from the unpatched
|
||||
control build, or whose ntdll.so exports other symbols. Only `files/lib/wine/aarch64-unix/ntdll.so`
|
||||
and `files/bin-arm64/wineserver` are replaced, so every Proton build needs its own pair, built
|
||||
from that build's exact wine source. That pair plus `pack.json` is a pack (`<id>.tzst`). On the
|
||||
device the guest runs Proton from a shadow tree that links the stock tool and holds the two
|
||||
patched files (`droiddeck-esync`, store in `~/.local/share/droiddeck-esync`).
|
||||
|
||||
## Credits
|
||||
|
||||
droiddeck-esync follows the esync design and droiddeck-ntsync implements the ntsync interface;
|
||||
both are the work of Elizabeth Figura for Wine, Proton and the Linux kernel. droiddeck-fsync runs
|
||||
fsync, by Elizabeth Figura and Paul Gofman for Proton, over `futex_waitv(2)` by André Almeida.
|
||||
DroidDeck builds on that work for Android, where neither the ntsync driver, `futex_waitv` nor a
|
||||
Proton build with esync is available.
|
||||
|
||||
## Matching
|
||||
|
||||
- `ge`, `cachyos`: built against the release tarball's own binaries. `pack.json` `stock` holds
|
||||
their sha256; the guest uses the pack only when both files hash the same. `make_pack.py` also
|
||||
requires the control build to reproduce both files byte for byte, so the patched pair differs
|
||||
from the release only by the patch (`--allow-unreproduced` lifts this for local experiments).
|
||||
- `valve`: the depots cannot be downloaded without a Steam login, so `stock` holds the control
|
||||
build's hashes and `source_match` is true: the guest also accepts a tool whose version token
|
||||
(second field of `<tool>/version`) and ntdll.so exports hash equal the pack's. With the optional
|
||||
`STEAM_DEPOT_USER` / `STEAM_DEPOT_TOKEN` secrets (and the `DEPOT_DOWNLOADER_SHA256` variable)
|
||||
the job downloads the depot instead and, when the depot is at the tag being built, makes an
|
||||
exact pack. The guest's launch line says which kind matched (`exact match` or `source match`).
|
||||
- A stock wineserver that already has esync built in (it prints `esync: up and running`) never
|
||||
gets a pack.
|
||||
|
||||
## Building and publishing
|
||||
|
||||
`.github/workflows/watch-proton-releases.yml` runs every 20 minutes and decides when a build is
|
||||
due. It asks Steam anonymously (`steamcmd +app_info_print`) which public build of each Valve ARM64
|
||||
depot app in `flavors.json` is live, and `discover.py watch` compares that, the newest `keep` tags
|
||||
and releases of every flavor, each flavor's `rev` and its patch series digest with the state of the
|
||||
last run (`watch-state.json` in the `droiddeck-esync-index` release). A flavor whose builds, rev or
|
||||
patch series changed, or whose Steam build moved, starts
|
||||
`.github/workflows/build-droiddeck-esync-packs.yml` for that flavor; nothing else starts a build.
|
||||
A run that finds no change takes about a minute and builds nothing. If steamcmd fails, the run
|
||||
watches the tags only and keeps the last Steam state.
|
||||
|
||||
`.github/workflows/build-droiddeck-esync-packs.yml` also runs once a day as a safety net, on pushes
|
||||
to `tools/droiddeck-esync/**` and by hand (flavor, one tag, publish yes/no). Only runs on `main` of
|
||||
`Droid-Deck/DroidDeck` publish packs; the schedule and pushes do nothing in other repositories,
|
||||
and manual runs elsewhere only build.
|
||||
|
||||
1. `discover.py matrix` lists the newest `keep` builds of each family in `flavors.json` (Valve
|
||||
tags, GE and CachyOS releases with an arm64 tarball) and drops the ones the published index,
|
||||
or the `droiddeck-esync-<flavor>` release, already has at the flavor's `rev`, the ones
|
||||
`revoked.txt` names, and the ones that failed in the last day at the same `rev` and patch
|
||||
series (a `<key>-r<rev>-<series digest>.failed` asset in the flavor's release). Naming the tag
|
||||
in a manual run builds it anyway.
|
||||
2. Each build runs on an arm64 runner with a read-only token, since it runs upstream code: the
|
||||
wine source is checked out at the tag (GE also runs its `protonprep-valve-staging.sh`),
|
||||
`build-pack.sh` builds a control and a patched pair inside the SDK image named by the
|
||||
source's own `Makefile.in` (arm64-llvm), at the flavor's `build_dir` in `flavors.json` (the
|
||||
path the release itself was built at, which ends up in the binaries), and `make_pack.py` makes
|
||||
the pack. Patches apply with `--fuzz=0`: a hunk whose context moved fails the build instead of
|
||||
landing somewhere else. For GE and CachyOS, whose release tarball holds the whole Proton,
|
||||
`smoke-test.sh` then boots a prefix with that stock Proton and runs `tests/esstress.c` (built
|
||||
with the SDK's llvm-mingw) under server-side sync as a baseline, and again with the pack under
|
||||
droiddeck-esync. A pack that fails while stock passes is not published; when stock itself cannot
|
||||
run on the runner the step only warns.
|
||||
3. The publish job checks each build's artifact against the matrix (`discover.py packs`) and
|
||||
uploads it to the `droiddeck-esync-valve`, `droiddeck-esync-ge` or `droiddeck-esync-cachyos`
|
||||
release as `<id>.tzst` with its index entry `<id>.json`. A build that made no pack gets its
|
||||
`.failed` marker there (`discover.py failures`), so a patch that does not apply or a stock
|
||||
wineserver with esync built in is not rebuilt on every run.
|
||||
4. The index job (environment `droiddeck-esync-signing`) runs only when the run built a pack or
|
||||
a published pack is not in the index yet. It collects every `<id>.json`, keeps the
|
||||
highest `rev` per stock pair and version, marks revoked ids, signs `index.json` (ECDSA P-256,
|
||||
`index-key.pub`) and publishes `bundle-<YYYYMMDD>-<HHMMSS>.tzst` (named after `generated`, so
|
||||
every signed index has its own), `index.json` and `index.json.sig` to `droiddeck-esync-index`.
|
||||
`generated` only grows. Packs stay in the index until they are revoked, also when their
|
||||
entry asset disappears; the previous index counts only when its signature verifies.
|
||||
|
||||
A pack is rebuilt only for a new `rev`: bump `rev` of the flavor in `flavors.json` whenever the
|
||||
patch series or the build changes. Local builds: `BUILD_DIR=<the flavor's build_dir> SYNC_PROFILE=<the
|
||||
flavor's profile> build-local.sh <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]`,
|
||||
then `make_pack.py`.
|
||||
|
||||
## Setup
|
||||
|
||||
- Signing key: create the repository environment `droiddeck-esync-signing` (Settings ->
|
||||
Environments, limit it to `main`) and store the PEM private key as its secret
|
||||
`SYNC_INDEX_KEY`: `gh secret set SYNC_INDEX_KEY --env droiddeck-esync-signing < index-key.pem`.
|
||||
The key never enters the repository; the app trusts only `index-key.pub` (also compiled into
|
||||
`EsyncPacks.kt`), so a new key needs an app release.
|
||||
- Optional: `STEAM_DEPOT_USER` and `STEAM_DEPOT_TOKEN` repository secrets for exact Valve packs,
|
||||
plus the repository variable `DEPOT_DOWNLOADER_SHA256`: the sha256 of
|
||||
`DepotDownloader-linux-arm64.zip` from the DepotDownloader release the workflow pins
|
||||
(`DEPOT_DOWNLOADER_TAG`). The depot step is skipped while it is unset.
|
||||
- APK bundle: after an index run, the job summary prints three lines. Put them in
|
||||
`tools/droiddeck-esync/release.env`:
|
||||
|
||||
```
|
||||
SYNC_BUNDLE_TAG=droiddeck-esync-index
|
||||
SYNC_BUNDLE_ASSET=bundle-20261002-061733.tzst
|
||||
SYNC_BUNDLE_SHA256=<sha256>
|
||||
```
|
||||
|
||||
`build.yml` then unpacks that bundle into `app/src/main/assets/droiddeck-esync/` (gitignored). A
|
||||
published bundle is never overwritten, so the pinned sha256 stays valid; bump `release.env`
|
||||
to ship newer packs inside the apk. The app fetches newer packs on its own. A pack the bundle
|
||||
carries that `revoked.txt` names is left out of the apk, with a warning to bump `release.env`.
|
||||
|
||||
## Revocation
|
||||
|
||||
Add the pack id as the first word of a line in `revoked.txt` and push to `main`. The workflow
|
||||
re-signs the index with `"revoked": true` for that id; the app deletes installed copies and
|
||||
never installs it again, and `discover.py` does not rebuild it at the same `rev`. The apk build
|
||||
also leaves it out of the bundled packs. To ship a fixed pack for the same Proton build, bump the
|
||||
flavor's `rev`. Runs that only build (other refs, or manual runs without publish) never queue in
|
||||
front of a publishing run, so a revocation is signed by the next run on `main`.
|
||||
Executable
+27
@@ -0,0 +1,27 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
if [ $# -lt 4 ]; then
|
||||
echo "usage: $0 <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]" >&2
|
||||
echo "env: BUILD_DIR (path the build runs at inside the SDK), SYNC_PROFILE (valve or cachyos), JOBS, SOURCE_DATE_EPOCH" >&2
|
||||
exit 2
|
||||
fi
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
sdk=$(realpath "$1")
|
||||
src=$(realpath "$2")
|
||||
work=$(realpath -m "$3")
|
||||
out=$(realpath -m "$4")
|
||||
patches=${5:+$(realpath "$5")}
|
||||
build_dir=${BUILD_DIR:-/builds/proton/proton/build-dir}
|
||||
rm -rf "$work"
|
||||
mkdir -p "$work" "$out"
|
||||
"$here/build-pack.sh" prepare "$src" "$work" ${patches:+"$patches"}
|
||||
bwrap --unshare-user --uid 0 --gid 0 --unshare-ipc --unshare-pid \
|
||||
--bind "$sdk" / --proc /proc --dev /dev --tmpfs /tmp \
|
||||
--bind "$work" "$build_dir" --bind "$out" /out --ro-bind "$here/build-pack.sh" /build-pack.sh \
|
||||
--setenv PATH /opt/llvm-mingw/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
||||
--setenv HOME /tmp --setenv LANG C.UTF-8 --setenv JOBS "${JOBS:-$(nproc)}" \
|
||||
--setenv SOURCE_DATE_EPOCH "${SOURCE_DATE_EPOCH:-1700000000}" --setenv SYNC_PROFILE "${SYNC_PROFILE:-valve}" \
|
||||
--chdir "$build_dir" \
|
||||
/bin/bash /build-pack.sh build "$build_dir" /out
|
||||
sha256sum "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
|
||||
Executable
+125
@@ -0,0 +1,125 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
echo "usage: $0 prepare <wine-source> <build-dir> [patch-dir]" >&2
|
||||
echo " $0 build <build-dir> <out-dir>" >&2
|
||||
echo " $0 all <wine-source> <out-dir> [patch-dir]" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
prepare() {
|
||||
local origin=$1 b=$2 patches=${3:-}
|
||||
local s=$b/src-wine
|
||||
mkdir -p "$s"
|
||||
rsync --filter=:C --exclude '*~' --exclude .git --exclude compile_commands.json \
|
||||
--exclude configure --exclude autom4te.cache --exclude include/config.h.in --exclude include/config.h.ink \
|
||||
--exclude include/wine/vulkan.h --exclude dlls/vulkan-1/vulkan-1.spec \
|
||||
--exclude dlls/winevulkan/loader_thunks.c --exclude dlls/winevulkan/loader_thunks.h \
|
||||
--exclude dlls/winevulkan/vulkan_thunks.c --exclude dlls/winevulkan/vulkan_thunks.h \
|
||||
--exclude dlls/winevulkan/winevulkan.json --exclude dlls/winevulkan/winevulkan.spec \
|
||||
--exclude dlls/ntdll/ntsyscalls.h --exclude dlls/win32u/win32syscalls.h \
|
||||
--exclude include/wine/server_protocol.h --exclude server/request_handlers.h --exclude server/request_trace.h \
|
||||
-Oarx --delete "$origin/" "$s/"
|
||||
for f in include/wine/server_protocol.h server/request_handlers.h server/request_trace.h; do
|
||||
cp -a "$origin/$f" "$s/$f"
|
||||
done
|
||||
if [ -n "$patches" ]; then
|
||||
local p found=0
|
||||
for p in "$patches"/*.patch; do
|
||||
[ -f "$p" ] || continue
|
||||
patch -d "$s" -p1 --forward --fuzz=0 --no-backup-if-mismatch -s < "$p"
|
||||
found=1
|
||||
done
|
||||
[ "$found" = 1 ] || { echo "no patches in $patches" >&2; exit 1; }
|
||||
fi
|
||||
(
|
||||
cd "$s"
|
||||
XDG_CACHE_HOME=$b/cache dlls/winevulkan/make_vulkan -x vk.xml -X video.xml
|
||||
tools/make_specfiles || true
|
||||
tools/make_requests
|
||||
) > "$b/generate.log" 2>&1
|
||||
touch "$b/.prepared"
|
||||
}
|
||||
|
||||
build() {
|
||||
local b=$1 out=$2
|
||||
local s=$b/src-wine o=$b/obj-wine-aarch64 d=$b/dst-wine-aarch64
|
||||
[ -f "$b/.prepared" ] || { echo "$b is not prepared" >&2; exit 1; }
|
||||
local jobs=${JOBS:-$(nproc)}
|
||||
local epoch=$(( ${SOURCE_DATE_EPOCH:-1700000000} - 20 ))
|
||||
local flags args pe_common i386_flags x86_64_flags conf
|
||||
case ${SYNC_PROFILE:-valve} in
|
||||
valve)
|
||||
flags="-Wno-discarded-qualifiers -march=armv8.2-a -mtune=cortex-x3 -O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
args=(--enable-werror --with-mingw=clang --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
|
||||
pe_common="-O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
i386_flags="-Wno-discarded-qualifiers -mstackrealign -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
|
||||
x86_64_flags="-Wno-discarded-qualifiers -mcmodel=small -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
|
||||
conf="../$(basename "$s")/configure"
|
||||
;;
|
||||
cachyos)
|
||||
flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -O2 -march=armv8.2-a -mtune=cortex-x3 -fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
args=(--with-mingw=clang --enable-build-id --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
|
||||
pe_common="-fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
i386_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mstackrealign -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
|
||||
x86_64_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mcmodel=small -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
|
||||
conf="$s/configure"
|
||||
;;
|
||||
*)
|
||||
echo "unknown SYNC_PROFILE ${SYNC_PROFILE}" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
local env=(STRIP=llvm-strip AR=aarch64-linux-gnu-ar RANLIB=aarch64-linux-gnu-ranlib
|
||||
CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ LD=aarch64-linux-gnu-ld
|
||||
RC=aarch64-w64-mingw32-windres WIDL=aarch64-w64-mingw32-widl PKG_CONFIG=aarch64-linux-gnu-pkg-config
|
||||
"PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig"
|
||||
"CFLAGS=$flags" "CPPFLAGS=$flags" "CXXFLAGS=-std=c++17 $flags" "LDFLAGS=" "SOURCE_DATE_EPOCH=$epoch"
|
||||
"CROSSCFLAGS=$flags" "CROSSLDFLAGS="
|
||||
"aarch64_CFLAGS=$flags" "aarch64_CPPFLAGS=$flags" "aarch64_CXXFLAGS=-std=c++17 $flags" "aarch64_LDFLAGS="
|
||||
i386_AR=i686-w64-mingw32-ar i386_RANLIB=i686-w64-mingw32-ranlib i386_CC=i686-w64-mingw32-gcc
|
||||
i386_CXX=i686-w64-mingw32-g++ i386_LD=i686-w64-mingw32-ld
|
||||
"i386_CFLAGS=$i386_flags" "i386_CPPFLAGS=$i386_flags" "i386_CXXFLAGS=-std=c++17 $i386_flags" "i386_LDFLAGS="
|
||||
"i386_PKG_CONFIG_LIBDIR=/usr/lib/i386-w64-mingw32/pkgconfig:/usr/share/pkgconfig"
|
||||
x86_64_AR=x86_64-w64-mingw32-ar x86_64_RANLIB=x86_64-w64-mingw32-ranlib x86_64_CC=x86_64-w64-mingw32-gcc
|
||||
x86_64_CXX=x86_64-w64-mingw32-g++ x86_64_LD=x86_64-w64-mingw32-ld
|
||||
"x86_64_CFLAGS=$x86_64_flags" "x86_64_CPPFLAGS=$x86_64_flags" "x86_64_CXXFLAGS=-std=c++17 $x86_64_flags" "x86_64_LDFLAGS="
|
||||
"x86_64_PKG_CONFIG_LIBDIR=/usr/lib/x86_64-w64-mingw32/pkgconfig:/usr/share/pkgconfig")
|
||||
export PATH=/opt/llvm-mingw/bin:$PATH
|
||||
( cd "$s" && autoreconf -fi && rm -rf autom4te.cache ) > "$b/autoreconf.log" 2>&1
|
||||
rm -rf "$o" "$d"
|
||||
mkdir -p "$o" "$d"
|
||||
( cd "$o" && env "${env[@]}" "$conf" -C --prefix="$d" --libdir="$d/lib" --host=aarch64-linux-gnu "${env[@]}" \
|
||||
"${args[@]}" ) > "$b/configure.log" 2>&1
|
||||
( cd "$o" && env "${env[@]}" make -j"$jobs" __tooldeps__ ) > "$b/tools.log" 2>&1
|
||||
( cd "$o" && env "${env[@]}" make -j"$jobs" dlls/ntdll/ntdll.so server/wineserver ) > "$b/make.log" 2>&1
|
||||
mkdir -p "$out/files/lib/wine/aarch64-unix" "$out/files/bin-arm64"
|
||||
llvm-objcopy --strip-debug --set-section-flags .text=contents,alloc,load,readonly,code \
|
||||
"$o/dlls/ntdll/ntdll.so" "$out/files/lib/wine/aarch64-unix/ntdll.so"
|
||||
llvm-objcopy --strip-debug "$o/server/wineserver" "$out/files/bin-arm64/wineserver"
|
||||
chmod 0755 "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
|
||||
cp "$s/include/wine/server_protocol.h" "$s/server/request_handlers.h" "$s/server/request_trace.h" "$out/"
|
||||
sed -n 's/^#define SERVER_PROTOCOL_VERSION \([0-9]*\)$/\1/p' "$out/server_protocol.h" > "$out/protocol"
|
||||
}
|
||||
|
||||
[ $# -ge 1 ] || usage
|
||||
case $1 in
|
||||
prepare)
|
||||
[ $# -ge 3 ] || usage
|
||||
prepare "$(realpath "$2")" "$(realpath -m "$3")" "${4:+$(realpath "$4")}"
|
||||
;;
|
||||
build)
|
||||
[ $# -eq 3 ] || usage
|
||||
build "$(realpath "$2")" "$(realpath -m "$3")"
|
||||
;;
|
||||
all)
|
||||
[ $# -ge 3 ] || usage
|
||||
b=${BUILD_DIR:-/builds/proton/proton/build-dir}
|
||||
prepare "$(realpath "$2")" "$b" "${4:+$(realpath "$4")}"
|
||||
build "$b" "$(realpath -m "$3")"
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
Executable
+654
@@ -0,0 +1,654 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from make_index import check_entry
|
||||
from make_pack import FLAVORS, HERE, PATCHES, REVOKED, SERIES, PackError, patch_digest, read_revoked, sha256_file, slug
|
||||
|
||||
FLAVORS_FILE = HERE / "flavors.json"
|
||||
ARCHIVE = re.compile(r"\.tar\.(gz|xz)$")
|
||||
ASSIGNMENT = re.compile(r"(?:export\s+|override\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*(?:\?|\+|!|:{1,3})?=\s*(.*)")
|
||||
REFERENCE = re.compile(r"\$[({]([A-Za-z_][A-Za-z0-9_]*)[)}]")
|
||||
IMAGE = re.compile(r"[A-Za-z0-9][A-Za-z0-9.-]*(?::[0-9]+)?(?:/[A-Za-z0-9._-]+)+(?::[A-Za-z0-9._-]+)?(?:@sha256:[0-9a-f]{64})?")
|
||||
REPO = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+")
|
||||
CLONES = ("shallow", "treeless")
|
||||
PACK_REV = re.compile(r"-r([0-9]+)$")
|
||||
ARTIFACT_PREFIX = "droiddeck-esync-pack-"
|
||||
FAILED = ".failed"
|
||||
RETRY_AFTER = timedelta(days=1)
|
||||
VDF_TOKEN = re.compile(r'"((?:[^"\\]|\\.)*)"|([{}])')
|
||||
APP_HEADER = re.compile(r"^AppID : ([0-9]+),", re.M)
|
||||
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
|
||||
|
||||
def gh(*args: str) -> subprocess.CompletedProcess:
|
||||
try:
|
||||
return subprocess.run(["gh", *args], capture_output=True, text=True, check=False)
|
||||
except FileNotFoundError as e:
|
||||
raise PackError("the GitHub CLI (gh) is not installed") from e
|
||||
|
||||
|
||||
def json_values(text: str) -> list:
|
||||
decoder = json.JSONDecoder()
|
||||
values = []
|
||||
position = 0
|
||||
while True:
|
||||
while position < len(text) and text[position].isspace():
|
||||
position += 1
|
||||
if position == len(text):
|
||||
return values
|
||||
value, position = decoder.raw_decode(text, position)
|
||||
values.append(value)
|
||||
|
||||
|
||||
def gh_api(path: str, paginate: bool = False, missing_ok: bool = False):
|
||||
result = gh("api", *(["--paginate"] if paginate else []), path)
|
||||
if result.returncode != 0:
|
||||
if missing_ok and "HTTP 404" in result.stderr:
|
||||
return None
|
||||
raise PackError(f"gh api {path}: {result.stderr.strip() or f'exit {result.returncode}'}")
|
||||
try:
|
||||
values = json_values(result.stdout)
|
||||
except ValueError as e:
|
||||
raise PackError(f"gh api {path}: {e}") from e
|
||||
if len(values) == 1:
|
||||
return values[0]
|
||||
if values and all(isinstance(value, list) for value in values):
|
||||
return [item for value in values for item in value]
|
||||
raise PackError(f"gh api {path}: unexpected output")
|
||||
|
||||
|
||||
def quote(value: str) -> str:
|
||||
return urllib.parse.quote(value, safe="")
|
||||
|
||||
|
||||
def load_flavors(path: Path) -> dict:
|
||||
try:
|
||||
data = json.loads(path.read_text())
|
||||
except (OSError, ValueError) as e:
|
||||
raise PackError(f"cannot read {path}: {e}") from e
|
||||
|
||||
def fail(why: str):
|
||||
raise PackError(f"{path}: {why}")
|
||||
|
||||
def compiles(pattern: str, where: str):
|
||||
try:
|
||||
re.compile(pattern)
|
||||
except re.error as e:
|
||||
fail(f"{where}: {pattern!r} is not a regular expression: {e}")
|
||||
|
||||
if not isinstance(data, dict) or data.get("format") != 1 or not isinstance(data.get("flavors"), dict):
|
||||
fail("is not a format 1 flavor list")
|
||||
image = data.get("sdk_image")
|
||||
if not isinstance(image, dict) or not all(isinstance(image.get(key), str) and image[key] for key in ("file", "variable", "match")):
|
||||
fail("sdk_image needs file, variable and match")
|
||||
for name, cfg in data["flavors"].items():
|
||||
if name not in FLAVORS:
|
||||
fail(f"unknown flavor {name!r}")
|
||||
if not isinstance(cfg, dict) or not isinstance(cfg.get("repo"), str) or not REPO.fullmatch(cfg["repo"]):
|
||||
fail(f"{name}: repo must be owner/name")
|
||||
for key in ("keep", "rev"):
|
||||
if type(cfg.get(key)) is not int or cfg[key] < 1:
|
||||
fail(f"{name}: {key} must be a positive number")
|
||||
if not isinstance(cfg.get("series"), str) or not SERIES.fullmatch(cfg["series"]):
|
||||
fail(f"{name}: series is not a patch series name")
|
||||
if cfg.get("profile") not in ("valve", "cachyos"):
|
||||
fail(f"{name}: profile must be valve or cachyos")
|
||||
if type(cfg.get("source_match")) is not bool:
|
||||
fail(f"{name}: source_match must be a boolean")
|
||||
build_dir = cfg.get("build_dir")
|
||||
if not isinstance(build_dir, str) or not re.fullmatch(r"(?:/[A-Za-z0-9._-]+)+", build_dir) or ".." in build_dir.split("/"):
|
||||
fail(f"{name}: build_dir must be the absolute path the release was built at")
|
||||
wine = cfg.get("wine")
|
||||
submodules = wine.get("submodules") if isinstance(wine, dict) else None
|
||||
if not isinstance(submodules, list) or "wine" not in submodules \
|
||||
or not all(isinstance(item, str) and re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", item) for item in submodules):
|
||||
fail(f"{name}: wine.submodules must list the submodules to check out, wine among them")
|
||||
if wine.get("clone") not in CLONES:
|
||||
fail(f"{name}: wine.clone must be one of {', '.join(CLONES)}")
|
||||
prep = wine.get("prep")
|
||||
if not isinstance(prep, str) or (prep and not re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", prep)) or ".." in prep.split("/"):
|
||||
fail(f"{name}: wine.prep must be a path inside the checkout or empty")
|
||||
if cfg.get("list") == "tags":
|
||||
families = cfg.get("families")
|
||||
if not isinstance(families, dict) or not families:
|
||||
fail(f"{name}: families must name at least one tag family")
|
||||
for family, settings in families.items():
|
||||
if not isinstance(settings, dict) or not isinstance(settings.get("tag"), str) \
|
||||
or not isinstance(settings.get("prefix"), str) or not re.fullmatch(r"[A-Za-z0-9._-]+", settings["prefix"]) \
|
||||
or not isinstance(settings.get("depot_app", ""), str):
|
||||
fail(f"{name}: family {family} needs a tag prefix and a tag pattern")
|
||||
compiles(settings["tag"], f"{name}: family {family}")
|
||||
if not isinstance(cfg.get("version_token"), str) or "{tag}" not in cfg["version_token"]:
|
||||
fail(f"{name}: version_token must contain {{tag}}")
|
||||
elif cfg.get("list") == "releases":
|
||||
if not isinstance(cfg.get("tag"), str) or not isinstance(cfg.get("assets"), list) or not cfg["assets"] \
|
||||
or not all(isinstance(pattern, str) for pattern in cfg["assets"]):
|
||||
fail(f"{name}: releases need a tag pattern and asset patterns")
|
||||
for pattern in (cfg["tag"], *cfg["assets"]):
|
||||
compiles(pattern, name)
|
||||
else:
|
||||
fail(f"{name}: list must be tags or releases")
|
||||
return data
|
||||
|
||||
|
||||
def natural_key(text: str) -> list[tuple]:
|
||||
return [(0, int(part), "") if part[0] in "0123456789" else (1, 0, part) for part in re.findall(r"[0-9]+|[^0-9]+", text)]
|
||||
|
||||
|
||||
def candidate(flavor: str, cfg: dict, tag: str, asset: str = "", token: str = "", depot_app: str = "") -> dict:
|
||||
return {
|
||||
"key": slug(f"{flavor}-{ARCHIVE.sub('', asset) if asset else tag}"),
|
||||
"flavor": flavor,
|
||||
"repo": cfg["repo"],
|
||||
"tag": tag,
|
||||
"asset": asset,
|
||||
"token": token,
|
||||
"depot_app": depot_app,
|
||||
"rev": cfg["rev"],
|
||||
"series": cfg["series"],
|
||||
"profile": cfg["profile"],
|
||||
"source_match": cfg["source_match"],
|
||||
"submodules": " ".join(cfg["wine"]["submodules"]),
|
||||
"clone": cfg["wine"]["clone"],
|
||||
"prep": cfg["wine"]["prep"],
|
||||
"build_dir": cfg["build_dir"],
|
||||
}
|
||||
|
||||
|
||||
def tag_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
|
||||
found = []
|
||||
for family, settings in sorted(cfg["families"].items()):
|
||||
if ref and not ref.startswith(settings["prefix"]):
|
||||
continue
|
||||
refs = gh_api(f"repos/{cfg['repo']}/git/matching-refs/tags/{quote(settings['prefix'])}", paginate=True)
|
||||
if not isinstance(refs, list):
|
||||
raise PackError(f"{cfg['repo']}: the tag list is not a list")
|
||||
pattern = re.compile(settings["tag"])
|
||||
names = set()
|
||||
for item in refs:
|
||||
name = item.get("ref", "") if isinstance(item, dict) else ""
|
||||
if isinstance(name, str) and name.startswith("refs/tags/") and pattern.search(name[len("refs/tags/"):]):
|
||||
names.add(name[len("refs/tags/"):])
|
||||
ordered = sorted(names, key=natural_key, reverse=True)
|
||||
chosen = [name for name in ordered if name == ref] if ref else ordered[:cfg["keep"]]
|
||||
for name in chosen:
|
||||
found.append(candidate(flavor, cfg, name, token=cfg["version_token"].format(tag=name), depot_app=settings.get("depot_app", "")))
|
||||
return found
|
||||
|
||||
|
||||
def pick_assets(assets: list, patterns: list[re.Pattern]) -> list[str]:
|
||||
chosen = {}
|
||||
for asset in assets:
|
||||
name = asset.get("name") if isinstance(asset, dict) else None
|
||||
if not isinstance(name, str) or not any(pattern.search(name) for pattern in patterns):
|
||||
continue
|
||||
stem = ARCHIVE.sub("", name)
|
||||
if stem not in chosen or name.endswith(".tar.xz"):
|
||||
chosen[stem] = name
|
||||
return sorted(chosen.values())
|
||||
|
||||
|
||||
def release_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
|
||||
repo = cfg["repo"]
|
||||
if ref:
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(ref)}", missing_ok=True)
|
||||
releases = [release] if release else []
|
||||
else:
|
||||
releases = gh_api(f"repos/{repo}/releases?per_page=100")
|
||||
if not isinstance(releases, list):
|
||||
raise PackError(f"{repo}: the release list is not a list")
|
||||
pattern = re.compile(cfg["tag"])
|
||||
patterns = [re.compile(item) for item in cfg["assets"]]
|
||||
usable = []
|
||||
for release in releases:
|
||||
tag = release.get("tag_name") if isinstance(release, dict) else None
|
||||
if not isinstance(tag, str) or release.get("draft") or not pattern.search(tag):
|
||||
continue
|
||||
assets = pick_assets(release.get("assets") or [], patterns)
|
||||
if assets:
|
||||
usable.append((release.get("published_at") or release.get("created_at") or "", natural_key(tag), tag, assets))
|
||||
usable.sort(reverse=True)
|
||||
if not ref:
|
||||
usable = usable[:cfg["keep"]]
|
||||
return [candidate(flavor, cfg, tag, asset=asset) for _, _, tag, assets in usable for asset in assets]
|
||||
|
||||
|
||||
def built(index: dict) -> dict[tuple[str, str, str], int]:
|
||||
revs = {}
|
||||
packs = index.get("packs") if isinstance(index, dict) else None
|
||||
for pack in packs if isinstance(packs, list) else []:
|
||||
if not isinstance(pack, dict) or not isinstance(pack.get("source"), dict):
|
||||
continue
|
||||
rev = pack.get("rev") if type(pack.get("rev")) is int else 0
|
||||
key = (pack.get("flavor"), pack["source"].get("ref"), pack["source"].get("asset", ""))
|
||||
revs[key] = max(revs.get(key, 0), rev)
|
||||
return revs
|
||||
|
||||
|
||||
def revoked_build(entry: dict, revoked: set[str]) -> bool:
|
||||
prefixes = {f"{entry['flavor']}-{slug(name)}-" for name in (entry["token"], entry["tag"]) if name}
|
||||
suffix = f"-r{entry['rev']}"
|
||||
return any(ident.endswith(suffix) and ident.startswith(prefix) for ident in revoked for prefix in prefixes)
|
||||
|
||||
|
||||
def chosen_flavors(flavors: dict, selected: str) -> list[str]:
|
||||
return sorted(flavors["flavors"]) if selected == "all" else [selected]
|
||||
|
||||
|
||||
def failure_marker(entry: dict, digests: dict[str, str] | None = None) -> str:
|
||||
series = entry["series"]
|
||||
if not SERIES.fullmatch(series):
|
||||
raise PackError(f"{series!r} is not a patch series name")
|
||||
digest = digests.get(series) if digests is not None else None
|
||||
if digest is None:
|
||||
digest = patch_digest(PATCHES / series)
|
||||
if digests is not None:
|
||||
digests[series] = digest
|
||||
return f"{entry['key']}-r{entry['rev']}-{digest[:12]}{FAILED}"
|
||||
|
||||
|
||||
def discover(flavors: dict, selected: str, ref: str, index: dict, revoked: set[str],
|
||||
failed: frozenset[str] = frozenset()) -> list[dict]:
|
||||
names = chosen_flavors(flavors, selected)
|
||||
done = built(index)
|
||||
digests = {}
|
||||
entries = []
|
||||
seen = set()
|
||||
matched = False
|
||||
for flavor in names:
|
||||
cfg = flavors["flavors"].get(flavor)
|
||||
if cfg is None:
|
||||
raise PackError(f"flavors.json has no {flavor}")
|
||||
found = tag_candidates(flavor, cfg, ref) if cfg["list"] == "tags" else release_candidates(flavor, cfg, ref)
|
||||
matched = matched or bool(found)
|
||||
for entry in found:
|
||||
if entry["key"] in seen:
|
||||
continue
|
||||
seen.add(entry["key"])
|
||||
label = f"{flavor} {entry['tag']}{' ' + entry['asset'] if entry['asset'] else ''}"
|
||||
if revoked_build(entry, revoked):
|
||||
print(f"discover: {label}: revoked at rev {entry['rev']}", file=sys.stderr)
|
||||
elif done.get((flavor, entry["tag"], entry["asset"]), 0) >= entry["rev"]:
|
||||
print(f"discover: {label}: in the index at rev {entry['rev']}", file=sys.stderr)
|
||||
elif not ref and failure_marker(entry, digests) in failed:
|
||||
print(f"discover: {label}: failed at rev {entry['rev']} with this patch series in the last "
|
||||
f"{RETRY_AFTER.days} days; skipped (bump rev, change the series or build the tag by name)", file=sys.stderr)
|
||||
else:
|
||||
print(f"discover: {label}: to build", file=sys.stderr)
|
||||
entries.append(entry)
|
||||
if ref and not matched:
|
||||
raise PackError(f"no {'' if selected == 'all' else selected + ' '}build is tagged {ref}")
|
||||
return entries
|
||||
|
||||
|
||||
def vdf(text: str) -> dict:
|
||||
stack = [{}]
|
||||
key = None
|
||||
for match in VDF_TOKEN.finditer(text):
|
||||
word, brace = match.group(1), match.group(2)
|
||||
if brace == "{":
|
||||
if key is None:
|
||||
raise PackError("app info: a block has no name")
|
||||
child = {}
|
||||
stack[-1][key] = child
|
||||
stack.append(child)
|
||||
key = None
|
||||
elif brace == "}":
|
||||
if len(stack) == 1:
|
||||
raise PackError("app info: unbalanced braces")
|
||||
stack.pop()
|
||||
key = None
|
||||
elif key is None:
|
||||
key = word
|
||||
else:
|
||||
stack[-1][key] = word
|
||||
key = None
|
||||
if len(stack) != 1:
|
||||
raise PackError("app info: unbalanced braces")
|
||||
return stack[0]
|
||||
|
||||
|
||||
def steam_builds(text: str, apps: list[str]) -> dict[str, dict]:
|
||||
found = {}
|
||||
text = ANSI.sub("", text)
|
||||
starts = [(m.start(), m.group(1)) for m in APP_HEADER.finditer(text)]
|
||||
for number, (start, app) in enumerate(starts):
|
||||
end = starts[number + 1][0] if number + 1 < len(starts) else len(text)
|
||||
if app not in apps:
|
||||
continue
|
||||
body = text[text.find("\n", start) + 1:end]
|
||||
info = vdf(body).get(app)
|
||||
depots = info.get("depots") if isinstance(info, dict) else None
|
||||
public = depots.get("branches", {}).get("public") if isinstance(depots, dict) else None
|
||||
if not isinstance(public, dict) or not str(public.get("buildid", "")).isdigit():
|
||||
continue
|
||||
manifests = sorted(depot["manifests"]["public"]["gid"] for name, depot in depots.items()
|
||||
if name.isdigit() and isinstance(depot, dict) and isinstance(depot.get("manifests"), dict)
|
||||
and isinstance(depot["manifests"].get("public"), dict) and depot["manifests"]["public"].get("gid"))
|
||||
found[app] = {"buildid": public["buildid"], "built": public.get("timebuildupdated", ""), "manifests": manifests}
|
||||
return found
|
||||
|
||||
|
||||
def steam_apps(flavors: dict) -> list[str]:
|
||||
return sorted({family["depot_app"] for cfg in flavors["flavors"].values() if cfg["list"] == "tags"
|
||||
for family in cfg["families"].values() if family.get("depot_app")})
|
||||
|
||||
|
||||
def watch(flavors: dict, previous: dict, steam: dict[str, dict] | None) -> tuple[dict, list[str], list[str]]:
|
||||
state = {}
|
||||
changed = []
|
||||
notes = []
|
||||
for flavor in sorted(flavors["flavors"]):
|
||||
cfg = flavors["flavors"][flavor]
|
||||
found = tag_candidates(flavor, cfg, "") if cfg["list"] == "tags" else release_candidates(flavor, cfg, "")
|
||||
builds = sorted(f"{entry['tag']} {entry['asset']}".strip() for entry in found)
|
||||
ships = {}
|
||||
if cfg["list"] == "tags" and steam is not None:
|
||||
for family in cfg["families"].values():
|
||||
app = family.get("depot_app")
|
||||
if app and app in steam:
|
||||
ships[app] = steam[app]
|
||||
old = previous.get(flavor) if isinstance(previous.get(flavor), dict) else {}
|
||||
if steam is None and isinstance(old.get("steam"), dict):
|
||||
ships = old["steam"]
|
||||
mark = {"builds": builds, "rev": cfg["rev"], "series": cfg["series"], "patch": patch_digest(PATCHES / cfg["series"])}
|
||||
fingerprint = hashlib.sha256(json.dumps(mark, sort_keys=True).encode()).hexdigest()
|
||||
state[flavor] = {**mark, "fingerprint": fingerprint, "steam": ships}
|
||||
if old.get("fingerprint") != fingerprint:
|
||||
changed.append(flavor)
|
||||
fresh = sorted(set(builds) - set(old.get("builds") or []))
|
||||
notes.append(f"{flavor}: {', '.join(fresh) if fresh else 'the patch series or rev changed'}")
|
||||
for app, ship in sorted(ships.items()):
|
||||
before = (old.get("steam") or {}).get(app) if isinstance(old.get("steam"), dict) else None
|
||||
if isinstance(before, dict) and before.get("buildid") != ship.get("buildid"):
|
||||
notes.append(f"{flavor}: Steam app {app} moved from build {before.get('buildid')} to {ship.get('buildid')}")
|
||||
if flavor not in changed:
|
||||
changed.append(flavor)
|
||||
return state, changed, notes
|
||||
|
||||
|
||||
def asset_text(repo: str, asset_id, what: str) -> str:
|
||||
result = gh("api", "-H", "Accept: application/octet-stream", f"repos/{repo}/releases/assets/{asset_id}")
|
||||
if result.returncode != 0:
|
||||
raise PackError(f"cannot download {what}: {result.stderr.strip()}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def release_index(repo: str, tag: str) -> dict:
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True)
|
||||
if release is None:
|
||||
return {}
|
||||
for asset in release.get("assets") or []:
|
||||
if isinstance(asset, dict) and asset.get("name") == "index.json":
|
||||
text = asset_text(repo, asset["id"], f"the index of {repo}")
|
||||
try:
|
||||
return json.loads(text)
|
||||
except ValueError as e:
|
||||
raise PackError(f"the index of {repo} is not JSON: {e}") from e
|
||||
return {}
|
||||
|
||||
|
||||
def published(repo: str, flavors: dict, selected: str, index: dict) -> list[dict]:
|
||||
packs = index.get("packs") if isinstance(index, dict) else None
|
||||
listed = {pack.get("id") for pack in packs if isinstance(pack, dict)} if isinstance(packs, list) else set()
|
||||
found = []
|
||||
for flavor in chosen_flavors(flavors, selected):
|
||||
cfg = flavors["flavors"].get(flavor)
|
||||
tag = f"droiddeck-esync-{flavor}"
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if cfg else None
|
||||
if not isinstance(release, dict) or type(release.get("id")) is not int:
|
||||
continue
|
||||
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
|
||||
if not isinstance(assets, list):
|
||||
raise PackError(f"{repo} {tag}: the asset list is not a list")
|
||||
for asset in assets:
|
||||
name = asset.get("name") if isinstance(asset, dict) else None
|
||||
if not isinstance(name, str) or not name.endswith(".json"):
|
||||
continue
|
||||
ident = name[:-len(".json")]
|
||||
rev = PACK_REV.search(ident)
|
||||
if ident in listed or not rev or int(rev.group(1)) < cfg["rev"]:
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(asset_text(repo, asset.get("id"), f"{tag} {name}"))
|
||||
except ValueError:
|
||||
entry = None
|
||||
if not isinstance(entry, dict) or entry.get("id") != ident or entry.get("flavor") != flavor:
|
||||
print(f"discover: {tag} {name} is not a pack entry", file=sys.stderr)
|
||||
continue
|
||||
print(f"discover: {tag} {name}: published, not in the index yet", file=sys.stderr)
|
||||
found.append(entry)
|
||||
return found
|
||||
|
||||
|
||||
def parse_time(value) -> datetime | None:
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
try:
|
||||
moment = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def recent_failures(repo: str, flavors: dict, selected: str, now: datetime) -> frozenset[str]:
|
||||
found = set()
|
||||
for flavor in chosen_flavors(flavors, selected):
|
||||
tag = f"droiddeck-esync-{flavor}"
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if flavor in flavors["flavors"] else None
|
||||
if not isinstance(release, dict) or type(release.get("id")) is not int:
|
||||
continue
|
||||
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
|
||||
if not isinstance(assets, list):
|
||||
raise PackError(f"{repo} {tag}: the asset list is not a list")
|
||||
for asset in assets:
|
||||
name = asset.get("name") if isinstance(asset, dict) else None
|
||||
if not isinstance(name, str) or not name.endswith(FAILED):
|
||||
continue
|
||||
moment = parse_time(asset.get("updated_at")) or parse_time(asset.get("created_at"))
|
||||
if moment is not None and now - moment < RETRY_AFTER:
|
||||
found.add(name)
|
||||
return frozenset(found)
|
||||
|
||||
|
||||
def failed_builds(matrix: dict, directory: Path) -> list[tuple[str, str]]:
|
||||
rows = matrix.get("include") if isinstance(matrix, dict) else None
|
||||
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
|
||||
raise PackError("the build matrix has no include list")
|
||||
made = {row["key"] for row in rows if (directory / f"{ARTIFACT_PREFIX}{row['key']}").is_dir()}
|
||||
digests = {}
|
||||
return [(row["flavor"], failure_marker(row, digests)) for row in rows if row["key"] not in made]
|
||||
|
||||
|
||||
def artifact_packs(matrix: dict, directory: Path) -> list[tuple[str, Path, str]]:
|
||||
rows = matrix.get("include") if isinstance(matrix, dict) else None
|
||||
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
|
||||
raise PackError("the build matrix has no include list")
|
||||
builds = {row["key"]: row for row in rows}
|
||||
found = []
|
||||
for folder in sorted(directory.iterdir()) if directory.is_dir() else []:
|
||||
row = builds.get(folder.name[len(ARTIFACT_PREFIX):]) if folder.name.startswith(ARTIFACT_PREFIX) else None
|
||||
if row is None or folder.is_symlink() or not folder.is_dir():
|
||||
raise PackError(f"{folder.name} is not the artifact of a build in this run")
|
||||
names = sorted(path.name for path in folder.iterdir())
|
||||
entries = [name for name in names if name.endswith(".json")]
|
||||
if len(entries) != 1 or names != sorted([entries[0], entries[0][:-len(".json")] + ".tzst"]):
|
||||
raise PackError(f"{folder.name} must hold exactly <id>.json and <id>.tzst")
|
||||
ident = entries[0][:-len(".json")]
|
||||
entry_path, archive = folder / entries[0], folder / f"{ident}.tzst"
|
||||
if entry_path.is_symlink() or archive.is_symlink() or not entry_path.is_file() or not archive.is_file():
|
||||
raise PackError(f"{folder.name} holds links or special files")
|
||||
try:
|
||||
data = json.loads(entry_path.read_text())
|
||||
except ValueError as e:
|
||||
raise PackError(f"{entry_path}: {e}") from e
|
||||
entry = check_entry(data, str(entry_path))
|
||||
source = data.get("source") if isinstance(data.get("source"), dict) else {}
|
||||
if entry["id"] != ident or entry["flavor"] != row["flavor"] or entry["rev"] != row["rev"] \
|
||||
or source.get("ref") != row["tag"] or source.get("asset", "") != row["asset"]:
|
||||
raise PackError(f"{entry_path} is not the pack of {row['flavor']} {row['tag']} at rev {row['rev']}")
|
||||
if archive.stat().st_size != entry["asset"]["size"] or sha256_file(archive) != entry["asset"]["sha256"]:
|
||||
raise PackError(f"{archive} is not the asset its entry describes")
|
||||
found.append((row["flavor"], folder, ident))
|
||||
return found
|
||||
|
||||
|
||||
def emit(entries: list[dict], unindexed: int = 0) -> None:
|
||||
matrix = json.dumps({"include": entries}, sort_keys=True, separators=(",", ":"))
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as stream:
|
||||
stream.write(f"matrix={matrix}\ncount={len(entries)}\nunindexed={unindexed}\n")
|
||||
print(matrix)
|
||||
|
||||
|
||||
def make_variables(text: str) -> tuple[dict[str, str], list[tuple[str, str]]]:
|
||||
variables = {}
|
||||
assignments = []
|
||||
for raw in text.splitlines():
|
||||
match = ASSIGNMENT.fullmatch(raw.split("#", 1)[0].strip())
|
||||
if match:
|
||||
name, value = match.group(1), match.group(2).strip()
|
||||
variables.setdefault(name, value)
|
||||
assignments.append((name, value))
|
||||
return variables, assignments
|
||||
|
||||
|
||||
def expand(value: str, variables: dict[str, str], depth: int = 0) -> str:
|
||||
if depth > 16:
|
||||
raise PackError("Makefile variables nest too deeply")
|
||||
|
||||
def lookup(match: re.Match) -> str:
|
||||
if match.group(1) not in variables:
|
||||
raise PackError(f"$({match.group(1)}) is not set in the Makefile")
|
||||
return expand(variables[match.group(1)], variables, depth + 1)
|
||||
|
||||
return REFERENCE.sub(lookup, value)
|
||||
|
||||
|
||||
def sdk_image(text: str, variable: str = "STEAMRT_IMAGE", match: str = "arm64-llvm") -> str:
|
||||
variables, assignments = make_variables(text)
|
||||
problem = None
|
||||
for name, value in assignments:
|
||||
if not name.startswith(variable):
|
||||
continue
|
||||
try:
|
||||
image = expand(value, variables)
|
||||
except PackError as e:
|
||||
problem = problem or e
|
||||
continue
|
||||
if match not in image:
|
||||
continue
|
||||
if not IMAGE.fullmatch(image):
|
||||
raise PackError(f"{name} = {image!r} is not an image reference")
|
||||
return image
|
||||
raise problem or PackError(f"no {variable} assignment names an {match} image")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Find the Proton builds that have no droiddeck-esync pack yet, as a GitHub Actions matrix.")
|
||||
parser.add_argument("--flavors", default=str(FLAVORS_FILE))
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
matrix = commands.add_parser("matrix")
|
||||
matrix.add_argument("--flavor", default="all", choices=("all", *FLAVORS))
|
||||
matrix.add_argument("--ref", default="")
|
||||
matrix.add_argument("--index", default="")
|
||||
matrix.add_argument("--index-release", default="")
|
||||
matrix.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY", "Droid-Deck/DroidDeck"))
|
||||
matrix.add_argument("--revoked", default=str(REVOKED))
|
||||
image = commands.add_parser("sdk-image")
|
||||
image.add_argument("makefile")
|
||||
artifacts = commands.add_parser("packs")
|
||||
artifacts.add_argument("--matrix", required=True)
|
||||
artifacts.add_argument("directory")
|
||||
failures = commands.add_parser("failures")
|
||||
failures.add_argument("--matrix", required=True)
|
||||
failures.add_argument("directory")
|
||||
commands.add_parser("steam-apps")
|
||||
watching = commands.add_parser("watch")
|
||||
watching.add_argument("--state", default="")
|
||||
watching.add_argument("--steam-info", default="")
|
||||
watching.add_argument("--out", required=True)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
flavors = load_flavors(Path(args.flavors))
|
||||
if args.command == "sdk-image":
|
||||
settings = flavors["sdk_image"]
|
||||
print(sdk_image(Path(args.makefile).read_text(), settings["variable"], settings["match"]))
|
||||
return 0
|
||||
if args.command == "packs":
|
||||
for flavor, folder, ident in artifact_packs(json.loads(args.matrix), Path(args.directory)):
|
||||
print(f"{flavor}\t{folder}\t{ident}")
|
||||
return 0
|
||||
if args.command == "failures":
|
||||
for flavor, marker in failed_builds(json.loads(args.matrix), Path(args.directory)):
|
||||
print(f"{flavor}\t{marker}")
|
||||
return 0
|
||||
if args.command == "steam-apps":
|
||||
print(" ".join(steam_apps(flavors)))
|
||||
return 0
|
||||
if args.command == "watch":
|
||||
previous = {}
|
||||
if args.state and Path(args.state).is_file():
|
||||
try:
|
||||
previous = json.loads(Path(args.state).read_text())
|
||||
except ValueError:
|
||||
print("discover: the previous watch state is not JSON; every flavor counts as changed", file=sys.stderr)
|
||||
previous = previous.get("flavors", {}) if isinstance(previous, dict) and previous.get("format") == 1 else {}
|
||||
steam = None
|
||||
if args.steam_info:
|
||||
try:
|
||||
steam = steam_builds(Path(args.steam_info).read_text(errors="replace"), steam_apps(flavors))
|
||||
except (OSError, PackError) as e:
|
||||
print(f"discover: Steam app info unusable ({e}); watching the source tags only", file=sys.stderr)
|
||||
if steam is not None and len(steam) != len(steam_apps(flavors)):
|
||||
print(f"discover: Steam app info covers {sorted(steam)} of {steam_apps(flavors)}", file=sys.stderr)
|
||||
state, changed, notes = watch(flavors, previous, steam)
|
||||
Path(args.out).write_text(json.dumps({"format": 1, "flavors": state}, indent=1, sort_keys=True) + "\n")
|
||||
for note in notes:
|
||||
print(f"discover: {note}", file=sys.stderr)
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as stream:
|
||||
stream.write(f"changed={' '.join(changed)}\n")
|
||||
print(" ".join(changed))
|
||||
return 0
|
||||
if args.index and args.index_release:
|
||||
raise PackError("pass --index or --index-release, not both")
|
||||
failed = frozenset()
|
||||
unindexed = 0
|
||||
if args.index:
|
||||
path = Path(args.index)
|
||||
index = json.loads(path.read_text()) if path.exists() else {}
|
||||
elif args.index_release:
|
||||
if not REPO.fullmatch(args.repo):
|
||||
raise PackError(f"{args.repo!r} is not owner/name")
|
||||
index = release_index(args.repo, args.index_release)
|
||||
listed = index.get("packs") if isinstance(index, dict) and isinstance(index.get("packs"), list) else []
|
||||
waiting = published(args.repo, flavors, args.flavor, index)
|
||||
unindexed = len(waiting)
|
||||
index = {"packs": [*listed, *waiting]}
|
||||
failed = recent_failures(args.repo, flavors, args.flavor, datetime.now(timezone.utc))
|
||||
else:
|
||||
index = {}
|
||||
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
|
||||
emit(discover(flavors, args.flavor, args.ref.strip(), index, revoked, failed), unindexed)
|
||||
except (PackError, OSError, ValueError) as e:
|
||||
print(f"discover: {e}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"format": 1,
|
||||
"sdk_image": {
|
||||
"file": "Makefile.in",
|
||||
"variable": "STEAMRT_IMAGE",
|
||||
"match": "arm64-llvm"
|
||||
},
|
||||
"flavors": {
|
||||
"valve": {
|
||||
"repo": "ValveSoftware/Proton",
|
||||
"list": "tags",
|
||||
"families": {
|
||||
"experimental": {
|
||||
"prefix": "experimental-11.0-",
|
||||
"tag": "^experimental-11\\.0-\\d{8}[a-z]?$",
|
||||
"depot_app": "4427310"
|
||||
},
|
||||
"proton": {
|
||||
"prefix": "proton-11.0-",
|
||||
"tag": "^proton-11\\.0-\\d+[a-z]?$",
|
||||
"depot_app": "4628740"
|
||||
}
|
||||
},
|
||||
"version_token": "{tag}-arm64",
|
||||
"keep": 2,
|
||||
"rev": 1,
|
||||
"series": "valve-11",
|
||||
"profile": "valve",
|
||||
"source_match": true,
|
||||
"build_dir": "/builds/proton/proton/build-dir",
|
||||
"wine": {
|
||||
"submodules": [
|
||||
"wine"
|
||||
],
|
||||
"clone": "shallow",
|
||||
"prep": ""
|
||||
}
|
||||
},
|
||||
"ge": {
|
||||
"repo": "GloriousEggroll/proton-ge-custom",
|
||||
"list": "releases",
|
||||
"tag": "^GE-Proton11-\\d+$",
|
||||
"assets": [
|
||||
"aarch64\\.tar\\.(gz|xz)$",
|
||||
"arm64\\.tar\\.(gz|xz)$"
|
||||
],
|
||||
"keep": 3,
|
||||
"rev": 1,
|
||||
"series": "valve-11",
|
||||
"profile": "valve",
|
||||
"source_match": false,
|
||||
"build_dir": "/home/tcrider/build",
|
||||
"wine": {
|
||||
"submodules": [
|
||||
"wine",
|
||||
"wine-staging"
|
||||
],
|
||||
"clone": "treeless",
|
||||
"prep": "patches/protonprep-valve-staging.sh"
|
||||
}
|
||||
},
|
||||
"cachyos": {
|
||||
"repo": "CachyOS/proton-cachyos",
|
||||
"list": "releases",
|
||||
"tag": "^cachyos-11\\.0-",
|
||||
"assets": [
|
||||
"arm64\\.tar\\.(gz|xz)$"
|
||||
],
|
||||
"keep": 3,
|
||||
"rev": 1,
|
||||
"series": "cachyos-11",
|
||||
"profile": "cachyos",
|
||||
"source_match": false,
|
||||
"build_dir": "/home/runner/work/proton-cachyos/proton-cachyos/build",
|
||||
"wine": {
|
||||
"submodules": [
|
||||
"wine"
|
||||
],
|
||||
"clone": "shallow",
|
||||
"prep": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
-----BEGIN PUBLIC KEY-----
|
||||
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFyqZQAMh3g3nsEGLU+kBGVPBEkXr
|
||||
Dv5F3kPtuQ+FV1DkTCGmYUQ1LS9wJtHGt0Ht4wToHGGEp9yzDibYZ4/zhg==
|
||||
-----END PUBLIC KEY-----
|
||||
Executable
+285
@@ -0,0 +1,285 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from make_pack import (
|
||||
BINARIES,
|
||||
EPOCH,
|
||||
FLAVORS,
|
||||
HERE,
|
||||
NTDLL,
|
||||
PACK_ID,
|
||||
REVOKED,
|
||||
SHA256,
|
||||
WINESERVER,
|
||||
PackError,
|
||||
compress,
|
||||
dump_json,
|
||||
read_revoked,
|
||||
sha256_bytes,
|
||||
sha256_file,
|
||||
tar_bytes,
|
||||
write_atomic,
|
||||
)
|
||||
|
||||
PUBKEY = HERE / "index-key.pub"
|
||||
SCHEMA = 1
|
||||
INDEX_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
|
||||
|
||||
|
||||
def check_entry(entry, origin: str) -> dict:
|
||||
def fail(why: str):
|
||||
raise PackError(f"{origin}: {why}")
|
||||
|
||||
if not isinstance(entry, dict):
|
||||
fail("is not an object")
|
||||
ident = entry.get("id")
|
||||
if not isinstance(ident, str) or not PACK_ID.fullmatch(ident):
|
||||
fail(f"{ident!r} is not a pack id")
|
||||
if entry.get("flavor") not in FLAVORS:
|
||||
fail(f"unknown flavor {entry.get('flavor')!r}")
|
||||
if type(entry.get("rev")) is not int or entry["rev"] < 1:
|
||||
fail("rev is not a positive number")
|
||||
version, line = entry.get("version"), entry.get("version_line")
|
||||
fields = line.split() if isinstance(line, str) else []
|
||||
if not isinstance(version, str) or len(fields) < 2 or fields[1] != version or not EPOCH.fullmatch(fields[0]):
|
||||
fail("version and version_line disagree")
|
||||
for field in ("stock", "files"):
|
||||
value = entry.get(field)
|
||||
if not isinstance(value, dict) or set(value) != set(BINARIES) or not all(isinstance(v, str) and SHA256.fullmatch(v) for v in value.values()):
|
||||
fail(f"{field} must give the sha256 of {' and '.join(BINARIES)}")
|
||||
if not isinstance(entry.get("exports"), str) or not SHA256.fullmatch(entry["exports"]):
|
||||
fail("exports is not a sha256")
|
||||
if type(entry.get("source_match")) is not bool:
|
||||
fail("source_match is not a boolean")
|
||||
asset = entry.get("asset")
|
||||
if not isinstance(asset, dict) or not isinstance(asset.get("sha256"), str) or not SHA256.fullmatch(asset["sha256"]) \
|
||||
or type(asset.get("size")) is not int or asset["size"] <= 0:
|
||||
fail("asset needs its sha256 and size")
|
||||
clean = {key: entry[key] for key in INDEX_KEYS if key in entry}
|
||||
clean["asset"] = {"sha256": asset["sha256"], "size": asset["size"]}
|
||||
return clean
|
||||
|
||||
|
||||
def load_entries(directory: Path) -> list[dict]:
|
||||
if not directory.is_dir():
|
||||
raise PackError(f"{directory} is not a directory")
|
||||
entries = []
|
||||
for path in sorted(directory.glob("*.json")):
|
||||
try:
|
||||
data = json.loads(path.read_text())
|
||||
except ValueError as e:
|
||||
raise PackError(f"{path}: {e}") from e
|
||||
entry = check_entry(data, str(path))
|
||||
if path.stem != entry["id"]:
|
||||
raise PackError(f"{path} holds the entry of {entry['id']}")
|
||||
entries.append(entry)
|
||||
return entries
|
||||
|
||||
|
||||
def load_previous(path: Path) -> tuple[int, list[dict], list]:
|
||||
try:
|
||||
data = json.loads(path.read_text())
|
||||
except (OSError, ValueError) as e:
|
||||
raise PackError(f"cannot read the previous index {path}: {e}") from e
|
||||
if not isinstance(data, dict) or data.get("schema") != SCHEMA or type(data.get("generated")) is not int \
|
||||
or not isinstance(data.get("packs"), list):
|
||||
raise PackError(f"{path} is not a schema {SCHEMA} index")
|
||||
packs = []
|
||||
for number, pack in enumerate(data["packs"]):
|
||||
entry = check_entry(pack, f"{path} pack {number}")
|
||||
entry["revoked"] = pack.get("revoked") is True
|
||||
packs.append(entry)
|
||||
return data["generated"], packs, data["packs"]
|
||||
|
||||
|
||||
def epoch(entry: dict) -> int:
|
||||
return int(entry["version_line"].split()[0])
|
||||
|
||||
|
||||
def rank(entry: dict) -> tuple:
|
||||
return entry["rev"], epoch(entry), entry["id"]
|
||||
|
||||
|
||||
def choose(entries: list[dict], previous: list[dict], revoked: set[str]) -> list[dict]:
|
||||
known = {entry["id"]: entry for entry in entries}
|
||||
for entry in previous:
|
||||
known.setdefault(entry["id"], entry)
|
||||
active = {}
|
||||
withdrawn = []
|
||||
for entry in known.values():
|
||||
if entry["id"] in revoked:
|
||||
withdrawn.append(dict(entry, revoked=True))
|
||||
continue
|
||||
key = (entry["stock"][NTDLL], entry["stock"][WINESERVER], entry["version"])
|
||||
if key not in active or rank(entry) > rank(active[key]):
|
||||
active[key] = dict(entry, revoked=False)
|
||||
return sorted([*active.values(), *withdrawn], key=lambda entry: entry["id"])
|
||||
|
||||
|
||||
def build_index(packs: list[dict], base_url: str, generated: int) -> dict:
|
||||
listed = []
|
||||
for pack in packs:
|
||||
entry = dict(pack)
|
||||
entry["asset"] = {
|
||||
"url": f"{base_url}/droiddeck-esync-{pack['flavor']}/{pack['id']}.tzst",
|
||||
"sha256": pack["asset"]["sha256"],
|
||||
"size": pack["asset"]["size"],
|
||||
}
|
||||
listed.append(entry)
|
||||
return {"schema": SCHEMA, "generated": generated, "packs": listed}
|
||||
|
||||
|
||||
def family(entry: dict) -> tuple[str, str, str]:
|
||||
token = entry["version"]
|
||||
major = re.search(r"[0-9]+", token)
|
||||
return entry["flavor"], re.match(r"[A-Za-z]*", token).group(0).lower(), major.group(0) if major else ""
|
||||
|
||||
|
||||
def bundle_choice(packs: list[dict], available: set[str] | None = None) -> list[dict]:
|
||||
newest = {}
|
||||
for pack in packs:
|
||||
if pack["revoked"] or (available is not None and pack["id"] not in available):
|
||||
continue
|
||||
key = family(pack)
|
||||
if key not in newest or (epoch(pack), pack["rev"], pack["id"]) > (epoch(newest[key]), newest[key]["rev"], newest[key]["id"]):
|
||||
newest[key] = pack
|
||||
return sorted(newest.values(), key=lambda pack: pack["id"])
|
||||
|
||||
|
||||
def openssl(*args: str) -> subprocess.CompletedProcess:
|
||||
try:
|
||||
return subprocess.run(["openssl", *args], capture_output=True, text=True, check=False)
|
||||
except FileNotFoundError as e:
|
||||
raise PackError("the openssl command is not installed") from e
|
||||
|
||||
|
||||
def verified(index: Path, signature: Path, pubkey: Path) -> bool:
|
||||
result = openssl("dgst", "-sha256", "-verify", str(pubkey), "-signature", str(signature), str(index))
|
||||
return result.returncode == 0 and "Verified OK" in result.stdout
|
||||
|
||||
|
||||
def trusted_previous(path: Path, signature: Path | None, pubkey: Path) -> tuple[int, list[dict], list | None]:
|
||||
if signature is not None and not (path.is_file() and signature.is_file() and verified(path, signature, pubkey)):
|
||||
print(f"make_index: the previous index {path} does not verify with {signature} and {pubkey}; it is ignored", file=sys.stderr)
|
||||
return 0, [], None
|
||||
return load_previous(path)
|
||||
|
||||
|
||||
def sign(index: Path, signature: Path, key: Path | None, pubkey: Path) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
if key is None:
|
||||
pem = os.environ.get("SYNC_INDEX_KEY", "")
|
||||
if not pem.strip():
|
||||
raise PackError("no signing key: pass --key or set SYNC_INDEX_KEY")
|
||||
key = Path(tmp) / "key.pem"
|
||||
with os.fdopen(os.open(key, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as stream:
|
||||
stream.write(pem if pem.endswith("\n") else pem + "\n")
|
||||
result = openssl("dgst", "-sha256", "-sign", str(key), "-out", str(signature), str(index))
|
||||
if result.returncode != 0:
|
||||
raise PackError(f"openssl could not sign the index: {result.stderr.strip()}")
|
||||
if not verified(index, signature, pubkey):
|
||||
raise PackError(f"the signature does not verify with {pubkey}: the signing key is not the one the app trusts")
|
||||
|
||||
|
||||
def bundle_name(generated: int) -> str:
|
||||
return f"bundle-{dt.datetime.fromtimestamp(generated, dt.timezone.utc):%Y%m%d-%H%M%S}.tzst"
|
||||
|
||||
|
||||
def bundle(out: Path, packs: list[dict], index: bytes, signature: bytes, packs_dir: Path, generated: int) -> tuple[str, str]:
|
||||
if not packs_dir.is_dir():
|
||||
raise PackError(f"{packs_dir} is not a directory")
|
||||
available = {path.name[:-len(".tzst")] for path in packs_dir.glob("*.tzst") if path.is_file()}
|
||||
members = {"index.json": (index, 0o644), "index.json.sig": (signature, 0o644)}
|
||||
for pack in bundle_choice(packs, available):
|
||||
path = packs_dir / f"{pack['id']}.tzst"
|
||||
if path.stat().st_size != pack["asset"]["size"] or sha256_file(path) != pack["asset"]["sha256"]:
|
||||
raise PackError(f"{path} is not the asset the index lists for {pack['id']}")
|
||||
members[f"packs/{pack['id']}.tzst"] = (path.read_bytes(), 0o644)
|
||||
name = bundle_name(generated)
|
||||
archive = compress(tar_bytes(members))
|
||||
write_atomic(out / name, archive)
|
||||
return name, sha256_bytes(archive)
|
||||
|
||||
|
||||
def report(values: dict[str, str]) -> None:
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as stream:
|
||||
for key, value in values.items():
|
||||
stream.write(f"{key}={value}\n")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Make, sign and bundle the droiddeck-esync pack index from the published pack entries.")
|
||||
parser.add_argument("--entries", required=True)
|
||||
parser.add_argument("--base-url", required=True)
|
||||
parser.add_argument("--key", default="")
|
||||
parser.add_argument("--pubkey", default=str(PUBKEY))
|
||||
parser.add_argument("--out", default="")
|
||||
parser.add_argument("--bundle-packs-dir", default="")
|
||||
parser.add_argument("--revoked", default=str(REVOKED))
|
||||
parser.add_argument("--previous", default="")
|
||||
parser.add_argument("--previous-sig", default="")
|
||||
parser.add_argument("--list-bundle", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
base_url = args.base_url.rstrip("/")
|
||||
if not base_url.startswith("https://"):
|
||||
raise PackError("--base-url must be an https URL")
|
||||
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
|
||||
if args.previous_sig and not args.previous:
|
||||
raise PackError("--previous-sig needs --previous")
|
||||
previous_signature = Path(args.previous_sig) if args.previous_sig else None
|
||||
previous_generated, previous, previous_raw = trusted_previous(Path(args.previous), previous_signature, Path(args.pubkey)) \
|
||||
if args.previous else (0, [], None)
|
||||
packs = choose(load_entries(Path(args.entries)), previous, revoked)
|
||||
if args.list_bundle:
|
||||
for pack in bundle_choice(packs):
|
||||
print(f"{pack['flavor']}\t{pack['id']}")
|
||||
return 0
|
||||
if not args.out:
|
||||
raise PackError("--out is required")
|
||||
out = Path(args.out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
generated = max(int(time.time()), previous_generated + 1)
|
||||
index = build_index(packs, base_url, generated)
|
||||
data = dump_json(index)
|
||||
changed = bool(packs) if previous_raw is None else previous_raw != index["packs"]
|
||||
temp_index = out / f".index.json.tmp-{os.getpid()}"
|
||||
temp_signature = out / f".index.json.sig.tmp-{os.getpid()}"
|
||||
try:
|
||||
temp_index.write_bytes(data)
|
||||
sign(temp_index, temp_signature, Path(args.key) if args.key else None, Path(args.pubkey))
|
||||
signature = temp_signature.read_bytes()
|
||||
os.replace(temp_index, out / "index.json")
|
||||
os.replace(temp_signature, out / "index.json.sig")
|
||||
finally:
|
||||
temp_index.unlink(missing_ok=True)
|
||||
temp_signature.unlink(missing_ok=True)
|
||||
outputs = {"changed": "true" if changed else "false", "generated": str(generated), "index_sha256": sha256_bytes(data)}
|
||||
live = sum(1 for pack in packs if not pack["revoked"])
|
||||
print(f"index.json {outputs['index_sha256']} {live} pack(s), {len(packs) - live} revoked, generated {generated}, {'changed' if changed else 'unchanged'}")
|
||||
if args.bundle_packs_dir:
|
||||
name, digest = bundle(out, packs, data, signature, Path(args.bundle_packs_dir), generated)
|
||||
outputs.update(bundle=name, bundle_sha256=digest)
|
||||
print(f"{name} {digest}")
|
||||
report(outputs)
|
||||
except (PackError, OSError) as e:
|
||||
print(f"make_index: {e}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+367
@@ -0,0 +1,367 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import functools
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
ROOT = HERE.parents[1]
|
||||
GUEST = ROOT / "tools/linuxfs/overlay/usr/local/bin/droiddeck-esync"
|
||||
PATCHES = HERE / "patches"
|
||||
REVOKED = HERE / "revoked.txt"
|
||||
NTDLL = "files/lib/wine/aarch64-unix/ntdll.so"
|
||||
WINESERVER = "files/bin-arm64/wineserver"
|
||||
BINARIES = (NTDLL, WINESERVER)
|
||||
COPY = ("files/bin-arm64/wine", "files/lib/wine/aarch64-unix/wine", "files/lib/wine/aarch64-unix/wine-preloader")
|
||||
HEADERS = ("server_protocol.h", "request_handlers.h", "request_trace.h")
|
||||
FLAVORS = ("valve", "ge", "cachyos")
|
||||
SOURCE_KEYS = ("repo", "ref", "commit", "wine_commit", "sdk_image")
|
||||
ENTRY_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
|
||||
PACK_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}")
|
||||
SERIES = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
EPOCH = re.compile(r"[0-9]{1,12}")
|
||||
PROTOCOL = re.compile(rb"^#define SERVER_PROTOCOL_VERSION ([0-9]+)[ \t]*$", re.M)
|
||||
BUILTIN_ESYNC = b"esync: up and running"
|
||||
ELF_MAGIC = b"\x7fELF"
|
||||
GUEST_MODULE = "droiddeck_esync_tools"
|
||||
|
||||
|
||||
class PackError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=None)
|
||||
def guest():
|
||||
loader = importlib.machinery.SourceFileLoader(GUEST_MODULE, str(GUEST))
|
||||
module = importlib.util.module_from_spec(importlib.util.spec_from_loader(GUEST_MODULE, loader))
|
||||
sys.modules[GUEST_MODULE] = module
|
||||
write_bytecode = sys.dont_write_bytecode
|
||||
sys.dont_write_bytecode = True
|
||||
try:
|
||||
loader.exec_module(module)
|
||||
except Exception as e:
|
||||
sys.modules.pop(GUEST_MODULE, None)
|
||||
raise PackError(f"cannot load {GUEST}: {e}") from e
|
||||
finally:
|
||||
sys.dont_write_bytecode = write_bytecode
|
||||
return module
|
||||
|
||||
|
||||
def sha256_bytes(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
return re.sub(r"[^A-Za-z0-9._-]", "-", text)
|
||||
|
||||
|
||||
def dump_json(value) -> bytes:
|
||||
return (json.dumps(value, indent=1, sort_keys=True, ensure_ascii=True) + "\n").encode()
|
||||
|
||||
|
||||
def write_atomic(path: Path, data: bytes) -> None:
|
||||
temp = path.with_name(f".{path.name}.tmp-{os.getpid()}")
|
||||
try:
|
||||
with temp.open("wb") as stream:
|
||||
stream.write(data)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temp, path)
|
||||
except BaseException:
|
||||
temp.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
|
||||
def read_revoked(path: Path | None) -> set[str]:
|
||||
if path is None:
|
||||
return set()
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as e:
|
||||
raise PackError(f"cannot read the revocation list {path}: {e}") from e
|
||||
revoked = set()
|
||||
for raw in text.splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
ident = line.split()[0]
|
||||
if not PACK_ID.fullmatch(ident):
|
||||
raise PackError(f"{path}: {ident!r} is not a pack id")
|
||||
revoked.add(ident)
|
||||
return revoked
|
||||
|
||||
|
||||
def tar_bytes(members: dict[str, tuple[bytes, int]]) -> bytes:
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w", format=tarfile.USTAR_FORMAT) as tar:
|
||||
for name in sorted(members):
|
||||
data, mode = members[name]
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(data)
|
||||
info.mode = mode
|
||||
info.mtime = 0
|
||||
info.uid = 0
|
||||
info.gid = 0
|
||||
info.uname = ""
|
||||
info.gname = ""
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def zstd(data: bytes, *args: str) -> bytes:
|
||||
try:
|
||||
result = subprocess.run(["zstd", "-q", *args], input=data, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
|
||||
except FileNotFoundError as e:
|
||||
raise PackError("the zstd command is not installed") from e
|
||||
if result.returncode != 0:
|
||||
raise PackError(f"zstd {' '.join(args)} failed: {result.stderr.decode(errors='replace').strip()}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def compress(data: bytes) -> bytes:
|
||||
return zstd(data, "-19", "-T1", "-c")
|
||||
|
||||
|
||||
def unpack(archive: bytes) -> dict[str, tuple[bytes, int]]:
|
||||
members = {}
|
||||
try:
|
||||
with tarfile.open(fileobj=io.BytesIO(zstd(archive, "-d", "-c")), mode="r:") as tar:
|
||||
for info in tar:
|
||||
if not info.isreg() or info.name in members:
|
||||
raise PackError(f"unexpected entry {info.name!r} in the archive")
|
||||
members[info.name] = (tar.extractfile(info).read(), info.mode)
|
||||
except tarfile.TarError as e:
|
||||
raise PackError(f"the archive does not read back: {e}") from e
|
||||
return members
|
||||
|
||||
|
||||
def binary(base: Path, rel: str) -> Path:
|
||||
path = base / rel
|
||||
if path.is_symlink() or not path.is_file():
|
||||
raise PackError(f"{path} is not a regular file")
|
||||
with path.open("rb") as stream:
|
||||
if stream.read(len(ELF_MAGIC)) != ELF_MAGIC:
|
||||
raise PackError(f"{path} is not an ELF file")
|
||||
return path
|
||||
|
||||
|
||||
def exports_hash(path: Path) -> str:
|
||||
try:
|
||||
value = guest().exports_hash(path)
|
||||
except PackError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise PackError(f"{path}: cannot read its exported symbols: {e}") from e
|
||||
if not isinstance(value, str) or not SHA256.fullmatch(value):
|
||||
raise PackError(f"{path}: has no exported symbols")
|
||||
return value
|
||||
|
||||
|
||||
def parse_version_line(line: str) -> tuple[int, str]:
|
||||
fields = line.split()
|
||||
if "\n" in line or len(fields) < 2 or not EPOCH.fullmatch(fields[0]):
|
||||
raise PackError(f"version line {line!r} is not '<build epoch> <version>'")
|
||||
token = guest().version_token(line)
|
||||
if not isinstance(token, str) or not token:
|
||||
raise PackError(f"version line {line!r} has no version token")
|
||||
return int(fields[0]), token
|
||||
|
||||
|
||||
def parse_source(text: str) -> dict[str, str]:
|
||||
try:
|
||||
source = json.loads(text)
|
||||
except ValueError as e:
|
||||
raise PackError(f"--source-json is not JSON: {e}") from e
|
||||
if not isinstance(source, dict) or not all(isinstance(key, str) and isinstance(value, str) for key, value in source.items()):
|
||||
raise PackError("--source-json must be an object of strings")
|
||||
missing = [key for key in SOURCE_KEYS if not source.get(key)]
|
||||
if missing:
|
||||
raise PackError(f"--source-json lacks {', '.join(missing)}")
|
||||
return source
|
||||
|
||||
|
||||
def patch_digest(directory: Path) -> str:
|
||||
patches = sorted(path for path in directory.glob("*.patch") if path.is_file())
|
||||
if not patches:
|
||||
raise PackError(f"{directory} holds no *.patch files")
|
||||
digest = hashlib.sha256()
|
||||
for path in patches:
|
||||
digest.update(f"{path.name}\0{sha256_file(path)}\n".encode())
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def protocol_version(control: Path, patched: Path) -> int:
|
||||
for name in HEADERS:
|
||||
before, after = control / name, patched / name
|
||||
if not before.is_file() or not after.is_file():
|
||||
raise PackError(f"{name} is missing from the control or the patched build")
|
||||
if before.read_bytes() != after.read_bytes():
|
||||
raise PackError(f"the patch changes {name}: the droiddeck-esync patch must not change the wineserver protocol")
|
||||
found = PROTOCOL.findall((control / HEADERS[0]).read_bytes())
|
||||
if len(found) != 1:
|
||||
raise PackError("server_protocol.h does not define SERVER_PROTOCOL_VERSION once")
|
||||
version = int(found[0])
|
||||
stamp = control / "protocol"
|
||||
if stamp.is_file() and stamp.read_text().strip() != str(version):
|
||||
raise PackError(f"{stamp} disagrees with server_protocol.h ({version})")
|
||||
return version
|
||||
|
||||
|
||||
def make_pack(
|
||||
flavor: str,
|
||||
rev: int,
|
||||
stock_dir: Path | None,
|
||||
version_line: str,
|
||||
control: Path,
|
||||
patched: Path,
|
||||
source: dict[str, str],
|
||||
series: str,
|
||||
out: Path,
|
||||
source_match: bool,
|
||||
patch_dir: Path | None = None,
|
||||
allow_unreproduced: bool = False,
|
||||
) -> dict:
|
||||
if flavor not in FLAVORS:
|
||||
raise PackError(f"unknown flavor {flavor!r}")
|
||||
if type(rev) is not int or rev < 1:
|
||||
raise PackError("--rev must be a positive number")
|
||||
if not SERIES.fullmatch(series):
|
||||
raise PackError(f"{series!r} is not a patch series name")
|
||||
if stock_dir is None and not source_match:
|
||||
raise PackError("without --stock-dir a pack can only match by source: pass --source-match")
|
||||
if stock_dir is not None and source_match:
|
||||
raise PackError("a pack made against stock binaries matches them exactly: drop --source-match")
|
||||
patch_dir = patch_dir or PATCHES / series
|
||||
if not patch_dir.is_dir():
|
||||
raise PackError(f"the patch series {patch_dir} does not exist")
|
||||
version_line = version_line.strip()
|
||||
if stock_dir is not None:
|
||||
stock_line = guest().version_line(stock_dir)
|
||||
if stock_line and version_line and stock_line != version_line:
|
||||
raise PackError(f"{stock_dir}/version says {stock_line!r}, not {version_line!r}")
|
||||
version_line = version_line or stock_line
|
||||
epoch, token = parse_version_line(version_line)
|
||||
|
||||
stock_paths = {rel: binary(stock_dir or control, rel) for rel in BINARIES}
|
||||
control_paths = {rel: binary(control, rel) for rel in BINARIES}
|
||||
patched_paths = {rel: binary(patched, rel) for rel in BINARIES}
|
||||
if BUILTIN_ESYNC in stock_paths[WINESERVER].read_bytes():
|
||||
raise PackError(f"{stock_paths[WINESERVER]} has esync built in and needs no pack")
|
||||
stock = {rel: sha256_file(path) for rel, path in stock_paths.items()}
|
||||
built = {rel: sha256_file(path) for rel, path in control_paths.items()}
|
||||
files = {rel: sha256_file(path) for rel, path in patched_paths.items()}
|
||||
for rel in BINARIES:
|
||||
if files[rel] == built[rel]:
|
||||
raise PackError(f"the patch left {rel} unchanged")
|
||||
|
||||
exports = exports_hash(stock_paths[NTDLL])
|
||||
control_exports = exports_hash(control_paths[NTDLL])
|
||||
if exports_hash(patched_paths[NTDLL]) != control_exports:
|
||||
raise PackError("the patch changes the symbols ntdll.so exports")
|
||||
if exports != control_exports:
|
||||
raise PackError("the control build of ntdll.so exports other symbols than the stock one: the wine source does not match this release")
|
||||
protocol = protocol_version(control, patched)
|
||||
if stock_dir is not None:
|
||||
for rel in BINARIES:
|
||||
state = "reproduces" if stock[rel] == built[rel] else "differs from"
|
||||
print(f"make_pack: the control build {state} the stock {rel}", file=sys.stderr)
|
||||
unreproduced = [rel for rel in BINARIES if stock[rel] != built[rel]]
|
||||
if unreproduced and not allow_unreproduced:
|
||||
raise PackError(f"the control build does not reproduce the stock {', '.join(unreproduced)}: the patched files would differ "
|
||||
"from the release in more than the patch (check the build directory and the SDK image, or pass --allow-unreproduced)")
|
||||
|
||||
pack_id = f"{flavor}-{slug(token)}-{epoch}-{stock[NTDLL][:12]}-r{rev}"
|
||||
if not PACK_ID.fullmatch(pack_id):
|
||||
raise PackError(f"{pack_id!r} is not a valid pack id")
|
||||
pack = {
|
||||
"format": 1,
|
||||
"id": pack_id,
|
||||
"flavor": flavor,
|
||||
"rev": rev,
|
||||
"version": token,
|
||||
"version_line": version_line,
|
||||
"stock": stock,
|
||||
"exports": exports,
|
||||
"source_match": source_match,
|
||||
"files": files,
|
||||
"copy": list(COPY),
|
||||
"protocol": protocol,
|
||||
"source": dict(source),
|
||||
"patch": {"series": series, "sha256": patch_digest(patch_dir)},
|
||||
}
|
||||
members = {
|
||||
"pack.json": (dump_json(pack), 0o644),
|
||||
NTDLL: (patched_paths[NTDLL].read_bytes(), 0o755),
|
||||
WINESERVER: (patched_paths[WINESERVER].read_bytes(), 0o755),
|
||||
}
|
||||
archive = compress(tar_bytes(members))
|
||||
if unpack(archive) != members:
|
||||
raise PackError("the archive does not read back as written")
|
||||
entry = {key: pack[key] for key in ENTRY_KEYS}
|
||||
entry["asset"] = {"sha256": sha256_bytes(archive), "size": len(archive)}
|
||||
entry["revoked"] = False
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
write_atomic(out / f"{pack_id}.tzst", archive)
|
||||
write_atomic(out / f"{pack_id}.json", dump_json(entry))
|
||||
return entry
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Make a droiddeck-esync pack from a control and a patched wine build of one Proton release.")
|
||||
parser.add_argument("--flavor", required=True, choices=FLAVORS)
|
||||
parser.add_argument("--rev", required=True, type=int)
|
||||
parser.add_argument("--stock-dir", default="")
|
||||
parser.add_argument("--version-line", default="")
|
||||
parser.add_argument("--control", required=True)
|
||||
parser.add_argument("--patched", required=True)
|
||||
parser.add_argument("--source-json", required=True)
|
||||
parser.add_argument("--patch-series", required=True)
|
||||
parser.add_argument("--patch-dir", default="")
|
||||
parser.add_argument("--out", required=True)
|
||||
parser.add_argument("--source-match", action="store_true")
|
||||
parser.add_argument("--allow-unreproduced", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
entry = make_pack(
|
||||
flavor=args.flavor,
|
||||
rev=args.rev,
|
||||
stock_dir=Path(args.stock_dir) if args.stock_dir else None,
|
||||
version_line=args.version_line,
|
||||
control=Path(args.control),
|
||||
patched=Path(args.patched),
|
||||
source=parse_source(args.source_json),
|
||||
series=args.patch_series,
|
||||
out=Path(args.out),
|
||||
source_match=args.source_match,
|
||||
patch_dir=Path(args.patch_dir) if args.patch_dir else None,
|
||||
allow_unreproduced=args.allow_unreproduced,
|
||||
)
|
||||
except (PackError, OSError) as e:
|
||||
print(f"make_pack: {e}", file=sys.stderr)
|
||||
return 1
|
||||
print(entry["id"])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
Whitespace-only changes.
Executable
+62
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
if [ $# -ne 4 ]; then
|
||||
echo "usage: $0 <proton-dir> <pack.tzst> <esstress.exe> <work-dir>" >&2
|
||||
exit 2
|
||||
fi
|
||||
tool=$(realpath "$1")
|
||||
archive=$(realpath "$2")
|
||||
exe=$(realpath "$3")
|
||||
work=$(realpath -m "$4")
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
guest=$here/../linuxfs/overlay/usr/local/bin/droiddeck-esync
|
||||
rm -rf "$work"
|
||||
mkdir -p "$work/pack"
|
||||
tar --zstd -xf "$archive" -C "$work/pack"
|
||||
python3 "$guest" build-shadow "$tool" "$work/pack" "$work/dist" >/dev/null
|
||||
[ -e "$work/dist/files/bin" ] || ln -s bin-arm64 "$work/dist/files/bin"
|
||||
export WINEDEBUG=-all WINEDLLOVERRIDES="mscoree,mshtml="
|
||||
|
||||
run() {
|
||||
local mode=$1 bin=$2 want=$3
|
||||
shift 3
|
||||
local prefix=$work/prefix-$mode
|
||||
mkdir -p "$prefix"
|
||||
(
|
||||
export WINEPREFIX=$prefix WINESERVER=$bin/wineserver "$@"
|
||||
timeout "${BOOT_TIMEOUT:-900}" "$bin/wine" wineboot -i > "$work/$mode-boot.log" 2>&1 || true
|
||||
"$bin/wineserver" -w || true
|
||||
if ! grep -q "$want" "$work/$mode-boot.log"; then
|
||||
echo "$mode: wine never said \"$want\""
|
||||
tail -n 30 "$work/$mode-boot.log"
|
||||
exit 1
|
||||
fi
|
||||
cp "$exe" "$prefix/drive_c/esstress.exe"
|
||||
status=0
|
||||
timeout "${STRESS_TIMEOUT:-1800}" "$bin/wine" 'C:\esstress.exe' > "$work/$mode-stress.log" 2>&1 || status=$?
|
||||
"$bin/wineserver" -k 2>/dev/null || true
|
||||
if ! grep -q '^esstress:' "$work/$mode-stress.log"; then
|
||||
echo "$mode: esstress did not finish (status $status)"
|
||||
tail -n 30 "$work/$mode-stress.log"
|
||||
exit 1
|
||||
fi
|
||||
grep -o '^FAIL [^ ]*:[0-9]*:' "$work/$mode-stress.log" | sort -u > "$work/$mode-failures" || true
|
||||
echo "$mode: $(grep -E '^esstress:' "$work/$mode-stress.log" | tail -n 1)"
|
||||
)
|
||||
}
|
||||
|
||||
if ! run stock "$tool/files/bin-arm64" "server-side synchronization" PROTON_NO_NTSYNC=1 WINEESYNC=0 WINEFSYNC=0; then
|
||||
echo "stock Proton does not run here either; the pack cannot be judged on this machine"
|
||||
exit 3
|
||||
fi
|
||||
run esync "$work/dist/files/bin-arm64" "esync: up and running" PROTON_NO_NTSYNC=1 WINEESYNC=1 WINEFSYNC=0
|
||||
if [ -s "$work/stock-failures" ]; then
|
||||
echo "stock fails these as well, so they do not count against the pack: $(tr '\n' ' ' < "$work/stock-failures")"
|
||||
fi
|
||||
new=$(comm -13 "$work/stock-failures" "$work/esync-failures")
|
||||
if [ -n "$new" ]; then
|
||||
echo "esync: fails checks stock passes:"
|
||||
grep -F "$new" "$work/esync-stress.log" || echo "$new"
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,618 @@
|
||||
#include <windows.h>
|
||||
#include <winternl.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
static int failures;
|
||||
#define CHECK(c, ...) do { if (!(c)) { failures++; printf("FAIL %s:%d: ", __FILE__, __LINE__); printf(__VA_ARGS__); printf("\n"); fflush(stdout); } } while (0)
|
||||
|
||||
static volatile LONG counter;
|
||||
static HANDLE mutex, sem, ev_auto, ev_manual, start_ev;
|
||||
static volatile LONG stop;
|
||||
|
||||
static DWORD WINAPI mutex_worker(void *arg)
|
||||
{
|
||||
int i;
|
||||
WaitForSingleObject(start_ev, INFINITE);
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(mutex, INFINITE);
|
||||
LONG v;
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "mutex wait %lu", r); break; }
|
||||
v = counter;
|
||||
if (i % 7 == 0) WaitForSingleObject(mutex, 0), ReleaseMutex(mutex);
|
||||
counter = v + 1;
|
||||
ReleaseMutex(mutex);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_mutex_contention(void)
|
||||
{
|
||||
HANDLE th[8];
|
||||
int i;
|
||||
counter = 0;
|
||||
mutex = CreateMutexA(NULL, FALSE, NULL);
|
||||
start_ev = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
for (i = 0; i < 8; i++) th[i] = CreateThread(NULL, 0, mutex_worker, NULL, 0, NULL);
|
||||
SetEvent(start_ev);
|
||||
WaitForMultipleObjects(8, th, TRUE, INFINITE);
|
||||
CHECK(counter == 8 * 20000, "mutex counter %ld", counter);
|
||||
for (i = 0; i < 8; i++) CloseHandle(th[i]);
|
||||
CloseHandle(mutex);
|
||||
CloseHandle(start_ev);
|
||||
printf("mutex_contention ok counter=%ld\n", counter);
|
||||
}
|
||||
|
||||
static volatile LONG produced, consumed;
|
||||
|
||||
static DWORD WINAPI sem_producer(void *arg)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < 50000; i++)
|
||||
{
|
||||
LONG prev;
|
||||
while (!ReleaseSemaphore(sem, 1, &prev))
|
||||
{
|
||||
CHECK(GetLastError() == ERROR_TOO_MANY_POSTS, "release err %lu", GetLastError());
|
||||
Sleep(0);
|
||||
}
|
||||
InterlockedIncrement(&produced);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static DWORD WINAPI sem_consumer(void *arg)
|
||||
{
|
||||
for (;;)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(sem, 2000);
|
||||
if (r == WAIT_TIMEOUT) break;
|
||||
CHECK(r == WAIT_OBJECT_0, "sem wait %lu", r);
|
||||
InterlockedIncrement(&consumed);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_semaphore(void)
|
||||
{
|
||||
HANDLE th[8];
|
||||
LONG prev;
|
||||
int i;
|
||||
produced = consumed = 0;
|
||||
sem = CreateSemaphoreA(NULL, 0, 16, NULL);
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, sem_producer, NULL, 0, NULL);
|
||||
for (i = 4; i < 8; i++) th[i] = CreateThread(NULL, 0, sem_consumer, NULL, 0, NULL);
|
||||
WaitForMultipleObjects(8, th, TRUE, INFINITE);
|
||||
CHECK(produced == 200000 && consumed == 200000, "sem produced %ld consumed %ld", produced, consumed);
|
||||
CHECK(!ReleaseSemaphore(sem, 17, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "limit");
|
||||
CHECK(ReleaseSemaphore(sem, 16, &prev) && prev == 0, "prev %ld", prev);
|
||||
CHECK(!ReleaseSemaphore(sem, 1, &prev), "over max");
|
||||
for (i = 0; i < 8; i++) CloseHandle(th[i]);
|
||||
CloseHandle(sem);
|
||||
printf("semaphore ok %ld/%ld\n", produced, consumed);
|
||||
}
|
||||
|
||||
static volatile LONG auto_hits;
|
||||
|
||||
static DWORD WINAPI auto_waiter(void *arg)
|
||||
{
|
||||
while (!stop)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(ev_auto, 50);
|
||||
if (r == WAIT_OBJECT_0) InterlockedIncrement(&auto_hits);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_auto_event(void)
|
||||
{
|
||||
HANDLE th[6];
|
||||
int i;
|
||||
LONG expected = 0;
|
||||
stop = 0; auto_hits = 0;
|
||||
ev_auto = CreateEventA(NULL, FALSE, FALSE, NULL);
|
||||
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, auto_waiter, NULL, 0, NULL);
|
||||
for (i = 0; i < 30000; i++)
|
||||
{
|
||||
LONG before = auto_hits;
|
||||
SetEvent(ev_auto);
|
||||
expected++;
|
||||
while (auto_hits == before) SwitchToThread();
|
||||
}
|
||||
stop = 1;
|
||||
WaitForMultipleObjects(6, th, TRUE, INFINITE);
|
||||
CHECK(auto_hits == expected, "auto hits %ld expected %ld", auto_hits, expected);
|
||||
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto event left signaled");
|
||||
SetEvent(ev_auto); SetEvent(ev_auto);
|
||||
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_OBJECT_0, "auto set");
|
||||
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto coalesce");
|
||||
for (i = 0; i < 6; i++) CloseHandle(th[i]);
|
||||
CloseHandle(ev_auto);
|
||||
printf("auto_event ok hits=%ld\n", auto_hits);
|
||||
}
|
||||
|
||||
static volatile LONG manual_seen;
|
||||
|
||||
static DWORD WINAPI manual_waiter(void *arg)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(ev_manual, 10000);
|
||||
if (r == WAIT_OBJECT_0) InterlockedIncrement(&manual_seen);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_manual_event(void)
|
||||
{
|
||||
HANDLE th[16];
|
||||
int i, round;
|
||||
ev_manual = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
for (round = 0; round < 50; round++)
|
||||
{
|
||||
manual_seen = 0;
|
||||
ResetEvent(ev_manual);
|
||||
for (i = 0; i < 16; i++) th[i] = CreateThread(NULL, 0, manual_waiter, NULL, 0, NULL);
|
||||
Sleep(2);
|
||||
SetEvent(ev_manual);
|
||||
WaitForMultipleObjects(16, th, TRUE, INFINITE);
|
||||
CHECK(manual_seen == 16, "manual round %d seen %ld", round, manual_seen);
|
||||
for (i = 0; i < 16; i++) CloseHandle(th[i]);
|
||||
}
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
SetEvent(ev_manual);
|
||||
if (i & 1) ResetEvent(ev_manual);
|
||||
}
|
||||
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_TIMEOUT, "manual final reset");
|
||||
SetEvent(ev_manual);
|
||||
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays");
|
||||
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays 2");
|
||||
CloseHandle(ev_manual);
|
||||
printf("manual_event ok\n");
|
||||
}
|
||||
|
||||
static HANDLE wa_objs[3];
|
||||
static volatile LONG wa_count;
|
||||
|
||||
static DWORD WINAPI wait_all_worker(void *arg)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < 3000; i++)
|
||||
{
|
||||
DWORD r = WaitForMultipleObjects(3, wa_objs, TRUE, 5000);
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "wait all %lu", r); break; }
|
||||
InterlockedIncrement(&wa_count);
|
||||
ReleaseMutex(wa_objs[0]);
|
||||
ReleaseSemaphore(wa_objs[1], 1, NULL);
|
||||
SetEvent(wa_objs[2]);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_wait_all(void)
|
||||
{
|
||||
HANDLE th[6], dup[2];
|
||||
int i;
|
||||
DWORD r;
|
||||
wa_count = 0;
|
||||
wa_objs[0] = CreateMutexA(NULL, FALSE, NULL);
|
||||
wa_objs[1] = CreateSemaphoreA(NULL, 1, 1, NULL);
|
||||
wa_objs[2] = CreateEventA(NULL, FALSE, TRUE, NULL);
|
||||
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, wait_all_worker, NULL, 0, NULL);
|
||||
WaitForMultipleObjects(6, th, TRUE, INFINITE);
|
||||
CHECK(wa_count == 18000, "wait all count %ld", wa_count);
|
||||
dup[0] = wa_objs[2];
|
||||
DuplicateHandle(GetCurrentProcess(), wa_objs[2], GetCurrentProcess(), &dup[1], 0, FALSE, DUPLICATE_SAME_ACCESS);
|
||||
r = WaitForMultipleObjects(2, dup, TRUE, 0);
|
||||
CHECK(r == WAIT_FAILED && GetLastError() == ERROR_INVALID_PARAMETER, "dup wait all %lu %lu", r, GetLastError());
|
||||
CloseHandle(dup[1]);
|
||||
for (i = 0; i < 6; i++) CloseHandle(th[i]);
|
||||
for (i = 0; i < 3; i++) CloseHandle(wa_objs[i]);
|
||||
printf("wait_all ok %ld\n", wa_count);
|
||||
}
|
||||
|
||||
static DWORD WINAPI hold_and_die(void *arg)
|
||||
{
|
||||
WaitForSingleObject((HANDLE)arg, INFINITE);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_abandoned(void)
|
||||
{
|
||||
HANDLE m = CreateMutexA(NULL, FALSE, NULL);
|
||||
HANDLE th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
|
||||
HANDLE objs[2];
|
||||
DWORD r;
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
r = WaitForSingleObject(m, 1000);
|
||||
CHECK(r == WAIT_ABANDONED, "abandoned %lu", r);
|
||||
CHECK(ReleaseMutex(m), "release after abandon");
|
||||
th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
objs[0] = CreateEventA(NULL, TRUE, TRUE, NULL);
|
||||
objs[1] = m;
|
||||
r = WaitForMultipleObjects(2, objs, TRUE, 1000);
|
||||
CHECK(r == WAIT_ABANDONED_0, "abandoned all %lu", r);
|
||||
ReleaseMutex(m);
|
||||
r = WaitForMultipleObjects(2, objs, FALSE, 1000);
|
||||
CHECK(r == WAIT_OBJECT_0, "any after %lu", r);
|
||||
CloseHandle(th); CloseHandle(objs[0]); CloseHandle(m);
|
||||
printf("abandoned ok\n");
|
||||
}
|
||||
|
||||
static volatile LONG apc_ran;
|
||||
static void CALLBACK apc_proc(ULONG_PTR p) { InterlockedIncrement(&apc_ran); }
|
||||
|
||||
static HANDLE apc_ready;
|
||||
|
||||
static DWORD WINAPI alert_waiter(void *arg)
|
||||
{
|
||||
SetEvent(apc_ready);
|
||||
return WaitForSingleObjectEx((HANDLE)arg, 5000, TRUE);
|
||||
}
|
||||
|
||||
static void test_apc(void)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
HANDLE th;
|
||||
DWORD r, code;
|
||||
int i;
|
||||
apc_ready = CreateEventA(NULL, FALSE, FALSE, NULL);
|
||||
for (i = 0; i < 200; i++)
|
||||
{
|
||||
apc_ran = 0;
|
||||
th = CreateThread(NULL, 0, alert_waiter, ev, 0, NULL);
|
||||
WaitForSingleObject(apc_ready, INFINITE);
|
||||
if (i & 1) Sleep(1);
|
||||
QueueUserAPC(apc_proc, th, 0);
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
GetExitCodeThread(th, &code);
|
||||
CHECK(code == WAIT_IO_COMPLETION && apc_ran == 1, "apc round %d code %lu ran %ld", i, code, apc_ran);
|
||||
CloseHandle(th);
|
||||
}
|
||||
apc_ran = 0;
|
||||
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
|
||||
r = SleepEx(1000, TRUE);
|
||||
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "sleepex %lu %ld", r, apc_ran);
|
||||
r = SleepEx(10, TRUE);
|
||||
CHECK(r == 0, "sleepex timeout %lu", r);
|
||||
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
|
||||
r = WaitForSingleObjectEx(ev, 0, FALSE);
|
||||
CHECK(r == WAIT_TIMEOUT, "non alertable %lu", r);
|
||||
r = WaitForSingleObjectEx(ev, 0, TRUE);
|
||||
CHECK(r == WAIT_IO_COMPLETION, "alertable pending %lu", r);
|
||||
CloseHandle(ev);
|
||||
printf("apc ok\n");
|
||||
}
|
||||
|
||||
static DWORD WINAPI close_waiter(void *arg)
|
||||
{
|
||||
return WaitForSingleObject((HANDLE)arg, 300);
|
||||
}
|
||||
|
||||
static void test_close_while_waiting(void)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < 300; i++)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), ev2, th;
|
||||
DWORD code;
|
||||
th = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
Sleep(0);
|
||||
CloseHandle(ev);
|
||||
ev2 = CreateEventA(NULL, TRUE, TRUE, NULL);
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
GetExitCodeThread(th, &code);
|
||||
CHECK(code == WAIT_TIMEOUT || code == WAIT_FAILED || code == WAIT_OBJECT_0, "close round %d code %lu", i, code);
|
||||
CHECK(WaitForSingleObject(ev2, 0) == WAIT_OBJECT_0, "reused handle state");
|
||||
CloseHandle(ev2);
|
||||
CloseHandle(th);
|
||||
}
|
||||
printf("close_while_waiting ok\n");
|
||||
}
|
||||
|
||||
static void test_pulse(void)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), th[4];
|
||||
DWORD code;
|
||||
int i, woke = 0;
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
Sleep(50);
|
||||
PulseEvent(ev);
|
||||
WaitForMultipleObjects(4, th, TRUE, INFINITE);
|
||||
for (i = 0; i < 4; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left signaled");
|
||||
CHECK(woke == 4, "pulse woke %d of 4", woke);
|
||||
CloseHandle(ev);
|
||||
ev = CreateEventA(NULL, FALSE, FALSE, NULL);
|
||||
PulseEvent(ev);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse without waiter left signaled");
|
||||
th[0] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
th[1] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
Sleep(50);
|
||||
PulseEvent(ev);
|
||||
WaitForMultipleObjects(2, th, TRUE, INFINITE);
|
||||
woke = 0;
|
||||
for (i = 0; i < 2; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
|
||||
CHECK(woke == 1, "auto pulse woke %d of 2", woke);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse left signaled");
|
||||
CloseHandle(ev);
|
||||
printf("pulse ok\n");
|
||||
}
|
||||
|
||||
static void test_semantics(void)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, TRUE, NULL), sem = CreateSemaphoreA(NULL, 2, 3, NULL), h[2];
|
||||
NTSTATUS (WINAPI *pNtWaitForSingleObject)(HANDLE, BOOLEAN, const LARGE_INTEGER *);
|
||||
LARGE_INTEGER when;
|
||||
LONG prev = -1;
|
||||
DWORD r, start;
|
||||
int i;
|
||||
pNtWaitForSingleObject = (void *)GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtWaitForSingleObject");
|
||||
apc_ran = 0;
|
||||
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
|
||||
r = WaitForSingleObjectEx(ev, INFINITE, TRUE);
|
||||
CHECK(r == WAIT_OBJECT_0 && apc_ran == 0, "object before apc %lu ran %ld", r, apc_ran);
|
||||
r = SleepEx(0, TRUE);
|
||||
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "apc after object %lu ran %ld", r, apc_ran);
|
||||
r = SleepEx(0, TRUE);
|
||||
CHECK(r == 0, "sleepex zero %lu", r);
|
||||
ResetEvent(ev);
|
||||
GetSystemTimeAsFileTime((FILETIME *)&when);
|
||||
when.QuadPart += 2000000;
|
||||
start = GetTickCount();
|
||||
r = pNtWaitForSingleObject(ev, FALSE, &when);
|
||||
CHECK(r == STATUS_TIMEOUT, "absolute timeout status %lx", r);
|
||||
CHECK(GetTickCount() - start >= 150 && GetTickCount() - start < 1500, "absolute timeout took %lu", GetTickCount() - start);
|
||||
CHECK(!ReleaseSemaphore(sem, 2, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "semaphore limit");
|
||||
CHECK(ReleaseSemaphore(sem, 1, &prev) && prev == 2, "semaphore prev %ld", prev);
|
||||
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(sem, 0) == WAIT_OBJECT_0, "semaphore take %d", i);
|
||||
CHECK(WaitForSingleObject(sem, 0) == WAIT_TIMEOUT, "semaphore empty");
|
||||
h[0] = sem; h[1] = ev;
|
||||
ReleaseSemaphore(sem, 1, NULL);
|
||||
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_TIMEOUT, "wait all partial");
|
||||
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 1, "wait all rollback lost token prev %ld", prev);
|
||||
SetEvent(ev);
|
||||
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_OBJECT_0, "wait all ready");
|
||||
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 0, "wait all consumed prev %ld", prev);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "manual stays set");
|
||||
CloseHandle(sem);
|
||||
CloseHandle(ev);
|
||||
printf("semantics ok\n");
|
||||
}
|
||||
|
||||
static HANDLE token_sem;
|
||||
static volatile LONG token_count;
|
||||
|
||||
static DWORD WINAPI token_worker(void *arg)
|
||||
{
|
||||
HANDLE h[2] = { token_sem, (HANDLE)arg };
|
||||
int i;
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
DWORD r = (i & 1) ? WaitForSingleObject(token_sem, INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "token wait %lu", r); break; }
|
||||
if (InterlockedIncrement(&token_count) > 3) CHECK(0, "token overcommit");
|
||||
InterlockedDecrement(&token_count);
|
||||
if (!(i & 1)) SetEvent((HANDLE)arg);
|
||||
ReleaseSemaphore(token_sem, 1, NULL);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_semaphore_tokens(void)
|
||||
{
|
||||
HANDLE th[6], ev = CreateEventA(NULL, FALSE, TRUE, NULL);
|
||||
LONG prev = -1;
|
||||
int i;
|
||||
token_sem = CreateSemaphoreA(NULL, 3, 3, NULL);
|
||||
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, token_worker, ev, 0, NULL);
|
||||
WaitForMultipleObjects(6, th, TRUE, INFINITE);
|
||||
for (i = 0; i < 6; i++) CloseHandle(th[i]);
|
||||
CHECK(ReleaseSemaphore(token_sem, 0, &prev) && prev == 3, "token final count %ld", prev);
|
||||
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(token_sem, 0) == WAIT_OBJECT_0, "token drain %d", i);
|
||||
CHECK(WaitForSingleObject(token_sem, 0) == WAIT_TIMEOUT, "token extra");
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "token event lost");
|
||||
CloseHandle(token_sem);
|
||||
CloseHandle(ev);
|
||||
printf("semaphore_tokens ok\n");
|
||||
}
|
||||
|
||||
static volatile LONG churn_ready;
|
||||
|
||||
static DWORD WINAPI churn_mutex(void *arg)
|
||||
{
|
||||
HANDLE h[2] = { (HANDLE)arg, ev_manual };
|
||||
unsigned int i = 0;
|
||||
if (WaitForMultipleObjects(2, h, TRUE, INFINITE) <= WAIT_ABANDONED_0 + 1) ReleaseMutex(h[0]);
|
||||
InterlockedIncrement(&churn_ready);
|
||||
for (;;)
|
||||
{
|
||||
DWORD r = (i++ & 1) ? WaitForSingleObject(h[0], INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
|
||||
if (r == WAIT_OBJECT_0 || r == WAIT_ABANDONED || r == WAIT_ABANDONED_0) ReleaseMutex(h[0]);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static DWORD WINAPI blocked_waiter(void *arg)
|
||||
{
|
||||
WaitForSingleObject((HANDLE)arg, 0);
|
||||
InterlockedIncrement(&churn_ready);
|
||||
return WaitForSingleObject((HANDLE)arg, INFINITE);
|
||||
}
|
||||
|
||||
static void test_terminate(void)
|
||||
{
|
||||
HANDLE th[4], m, ev;
|
||||
DWORD r, start;
|
||||
int round, i, stuck = 0;
|
||||
ev_manual = CreateEventA(NULL, TRUE, TRUE, NULL);
|
||||
for (round = 0; round < 150; round++)
|
||||
{
|
||||
m = CreateMutexA(NULL, FALSE, NULL);
|
||||
churn_ready = 0;
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, churn_mutex, m, 0, NULL);
|
||||
while (churn_ready < 4) Sleep(0);
|
||||
Sleep(round % 4);
|
||||
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
|
||||
WaitForMultipleObjects(4, th, TRUE, INFINITE);
|
||||
r = WaitForSingleObject(m, 3000);
|
||||
if (r == WAIT_TIMEOUT) stuck++;
|
||||
else ReleaseMutex(m);
|
||||
for (i = 0; i < 4; i++) CloseHandle(th[i]);
|
||||
CloseHandle(m);
|
||||
}
|
||||
CHECK(!stuck, "mutex stuck after terminate in %d rounds", stuck);
|
||||
ev = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
churn_ready = 0;
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, blocked_waiter, ev, 0, NULL);
|
||||
while (churn_ready < 4) Sleep(0);
|
||||
Sleep(50);
|
||||
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
|
||||
WaitForMultipleObjects(4, th, TRUE, INFINITE);
|
||||
for (i = 0; i < 4; i++) CloseHandle(th[i]);
|
||||
start = GetTickCount();
|
||||
for (i = 0; i < 100; i++) PulseEvent(ev);
|
||||
CHECK(GetTickCount() - start < 400, "pulse after killed waiters took %lu ms", GetTickCount() - start);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left event set");
|
||||
CloseHandle(ev);
|
||||
CloseHandle(ev_manual);
|
||||
printf("terminate ok\n");
|
||||
}
|
||||
|
||||
static void test_cross_process(const char *self)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, FALSE, FALSE, "Local\\esstress_ev");
|
||||
HANDLE done = CreateEventA(NULL, TRUE, FALSE, "Local\\esstress_done");
|
||||
HANDLE sm = CreateSemaphoreA(NULL, 0, 100000, "Local\\esstress_sem");
|
||||
PROCESS_INFORMATION pi;
|
||||
STARTUPINFOA si = { sizeof(si) };
|
||||
char cmd[MAX_PATH + 16];
|
||||
DWORD r, code;
|
||||
int i;
|
||||
sprintf(cmd, "\"%s\" child", self);
|
||||
CHECK(CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi), "createprocess %lu", GetLastError());
|
||||
for (i = 0; i < 5000; i++)
|
||||
{
|
||||
SetEvent(ev);
|
||||
r = WaitForSingleObject(sm, 5000);
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "xproc wait %lu at %d", r, i); break; }
|
||||
}
|
||||
SetEvent(done);
|
||||
r = WaitForSingleObject(pi.hProcess, 10000);
|
||||
CHECK(r == WAIT_OBJECT_0, "child exit wait %lu", r);
|
||||
GetExitCodeProcess(pi.hProcess, &code);
|
||||
CHECK(code == 0, "child code %lu", code);
|
||||
CloseHandle(pi.hProcess); CloseHandle(pi.hThread);
|
||||
CloseHandle(ev); CloseHandle(done); CloseHandle(sm);
|
||||
printf("cross_process ok\n");
|
||||
}
|
||||
|
||||
static int child(void)
|
||||
{
|
||||
HANDLE ev = OpenEventA(SYNCHRONIZE | EVENT_MODIFY_STATE, FALSE, "Local\\esstress_ev");
|
||||
HANDLE done = OpenEventA(SYNCHRONIZE, FALSE, "Local\\esstress_done");
|
||||
HANDLE sm = OpenSemaphoreA(SEMAPHORE_MODIFY_STATE | SYNCHRONIZE, FALSE, "Local\\esstress_sem");
|
||||
HANDLE objs[2];
|
||||
objs[0] = done; objs[1] = ev;
|
||||
if (!ev || !done || !sm) return 1;
|
||||
for (;;)
|
||||
{
|
||||
DWORD r = WaitForMultipleObjects(2, objs, FALSE, 10000);
|
||||
if (r == WAIT_OBJECT_0) return 0;
|
||||
if (r != WAIT_OBJECT_0 + 1) return 2;
|
||||
ReleaseSemaphore(sm, 1, NULL);
|
||||
}
|
||||
}
|
||||
|
||||
static void test_kill_holder(const char *self)
|
||||
{
|
||||
HANDLE m = CreateMutexA(NULL, FALSE, "Local\\esstress_mutex");
|
||||
PROCESS_INFORMATION pi;
|
||||
STARTUPINFOA si = { sizeof(si) };
|
||||
char cmd[MAX_PATH + 16];
|
||||
DWORD r;
|
||||
sprintf(cmd, "\"%s\" holder", self);
|
||||
CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);
|
||||
Sleep(500);
|
||||
TerminateProcess(pi.hProcess, 7);
|
||||
WaitForSingleObject(pi.hProcess, INFINITE);
|
||||
r = WaitForSingleObject(m, 3000);
|
||||
CHECK(r == WAIT_ABANDONED, "killed holder %lu", r);
|
||||
ReleaseMutex(m);
|
||||
CloseHandle(pi.hProcess); CloseHandle(pi.hThread); CloseHandle(m);
|
||||
printf("kill_holder ok\n");
|
||||
}
|
||||
|
||||
static int holder(void)
|
||||
{
|
||||
HANDLE m = OpenMutexA(SYNCHRONIZE, FALSE, "Local\\esstress_mutex");
|
||||
WaitForSingleObject(m, INFINITE);
|
||||
Sleep(INFINITE);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_many_objects(void)
|
||||
{
|
||||
static HANDLE h[20000];
|
||||
int i, n = 0;
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
if (!(h[i] = CreateEventA(NULL, i & 1, i & 2, NULL))) break;
|
||||
n++;
|
||||
}
|
||||
CHECK(n == 20000, "created %d", n);
|
||||
for (i = 0; i < n; i++) CHECK(WaitForSingleObject(h[i], 0) == ((i & 2) ? WAIT_OBJECT_0 : WAIT_TIMEOUT), "obj %d", i);
|
||||
for (i = 0; i < n; i++) CloseHandle(h[i]);
|
||||
for (i = 0; i < 100000; i++)
|
||||
{
|
||||
HANDLE e = CreateEventA(NULL, FALSE, TRUE, NULL);
|
||||
if (WaitForSingleObject(e, 0) != WAIT_OBJECT_0) { CHECK(0, "churn %d", i); CloseHandle(e); break; }
|
||||
CloseHandle(e);
|
||||
}
|
||||
printf("many_objects ok\n");
|
||||
}
|
||||
|
||||
static void test_timers_and_threads(void)
|
||||
{
|
||||
HANDLE t = CreateWaitableTimerA(NULL, FALSE, NULL);
|
||||
LARGE_INTEGER due;
|
||||
DWORD start = GetTickCount(), r;
|
||||
int i;
|
||||
due.QuadPart = -10 * 1000 * 20;
|
||||
SetWaitableTimer(t, &due, 10, NULL, NULL, FALSE);
|
||||
for (i = 0; i < 20; i++)
|
||||
{
|
||||
r = WaitForSingleObject(t, 1000);
|
||||
CHECK(r == WAIT_OBJECT_0, "timer %d %lu", i, r);
|
||||
}
|
||||
CHECK(GetTickCount() - start < 2000, "timer slow %lu", GetTickCount() - start);
|
||||
CancelWaitableTimer(t);
|
||||
CloseHandle(t);
|
||||
r = WaitForSingleObject(GetCurrentProcess(), 10);
|
||||
CHECK(r == WAIT_TIMEOUT, "process self %lu", r);
|
||||
printf("timers ok\n");
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
if (argc > 1 && !strcmp(argv[1], "child")) return child();
|
||||
if (argc > 1 && !strcmp(argv[1], "holder")) return holder();
|
||||
test_mutex_contention();
|
||||
test_semaphore();
|
||||
test_auto_event();
|
||||
test_manual_event();
|
||||
test_wait_all();
|
||||
test_abandoned();
|
||||
test_apc();
|
||||
test_close_while_waiting();
|
||||
test_pulse();
|
||||
test_semantics();
|
||||
test_semaphore_tokens();
|
||||
if (argc > 1 && !strcmp(argv[1], "terminate")) test_terminate();
|
||||
test_cross_process(argv[0]);
|
||||
test_kill_holder(argv[0]);
|
||||
test_many_objects();
|
||||
test_timers_and_threads();
|
||||
printf("esstress: %d failures\n", failures);
|
||||
return failures ? 1 : 0;
|
||||
}
|
||||
+1171
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user