mirror of
https://github.com/Droid-Deck/DroidDeck.git
synced 2026-11-21 22:05:59 +00:00
ESync, FSync, NTSync + OpenNow + FEX for Desktop Apps (#197)
* Make the Compatible Proton ARM64 the default and label compatible builds
The CompatToolMapping "0" default always points at DroidDeck's Proton
ARM64 tool, and per-game picks of adopted GE/CachyOS builds are kept,
including when the shortcuts writer remaps.
The tool is named after the Proton build it runs, Proton Experimental
ARM64 (Compatible), or Proton 11.0 ARM64 (Compatible) once that newer build
is installed. Adopted builds carry the same Compatible mark, read from a
string resource the app stages into the runtime, and the Proton page
explains which builds have the prefix fixes.
The registrar is renamed steam-compatibility and the tool
droiddeck-proton-arm64. Existing installs migrate on the next session: the
old tool directory is removed, every mapping that named it moves to the new
tool, and the retired bannerlator-steam-compat script is removed from the
runtime. CI and local packaging include the renamed script.
* Drop the duplicate R import left by merging main
* Add userspace ntsync for Proton on Android
Android kernels have no /dev/ntsync and its app policy refuses futex_waitv,
so every Proton 11 build here fell back to wineserver for every wait.
libblsession now serves /dev/ntsync itself: wineserver opens a memfd region,
each sync object is a timerfd that carries its slot, and the kernel ntsync
ioctls are answered in shared memory with futexes, following the kernel
driver's wake order. Valve's Proton Experimental ARM64, GE-Proton and
proton-cachyos use it unchanged.
Locks record their owner so a killed process cannot wedge the others,
signals are held off inside each critical section, dead processes and
exiting threads are swept, and object lifetimes follow close, dup2, dup3,
close_range and received descriptors.
Fast sync is on by default under Performance > Session fixes, and
PROTON_NO_NTSYNC=1 turns it off per game.
* Add droiddeck-esync as the default sync for Proton on Android
Without a kernel ntsync driver, Proton 11 builds here fell back to
wineserver for every wait. droiddeck-esync brings eventfd and shared-memory
sync to the exact Proton build a game runs on: a pack holds that build's
ntdll.so and wineserver with the patch series in tools/droiddeck-esync
applied, and the guest runs Proton from a shadow tree that links the stock
tool and swaps in those two files. A wineserver from a pack still prefers
/dev/ntsync when a kernel driver or droiddeck-ntsync is there.
Packs are built in Valve's SDK image from each release's own wine source
(Proton Experimental ARM64 tags, GE-Proton and proton-cachyos releases); the
control build must reproduce the release binaries byte for byte, and a
signed index lists the packs. The APK bundles the packs pinned in
release.env, the app fetches newer ones, and Steam's compatibility tool
picks the pack for each Proton build, so new Proton releases get packs
without an app update.
The ntdll side reads, writes and polls its eventfds with raw system calls,
so the guest's LD_PRELOAD wrappers stay out of the sync path. The server
abandons a mutex once a thread killed by TerminateThread has really exited
and when its process dies, and a mutex's owner and recursion count change
in one atomic step, so a killed owner cannot touch the next owner's count.
droiddeck-esync is on by default and PROTON_NO_ESYNC=1 turns it off per
game; droiddeck-ntsync, the userspace ntsync, is now an opt-in toggle.
* Watch Valve's Proton releases and build droiddeck-esync packs for them
A scheduled workflow reads the Proton ARM64 builds Steam ships (anonymous
steamcmd app info) and the upstream source tags, compares them with the
previous watch state and starts the pack build for any build that has no
pack yet. The pack build applies the patch series with --fuzz=0 and runs a
smoke test of each new pack against the esstress suite before it is
published.
* Add droiddeck-fsync, keep Steam and the app on one Proton, fix DirectAudio
droiddeck-fsync: libblsession answers futex_waitv(2), which the app
sandbox refuses, in userspace, so Proton's own fsync runs on the stock
Proton files with no pack. Waiters register the words they wait on in a
table beside the fsync shared memory and sleep on a futex of their own;
FUTEX_WAKE on the fsync memory wakes the registered waiters. Slots carry
one owner word that the sweeper and the owner claim with a CAS, and the
mapping table is rebuilt under a lock held across the real munmap, so a
reused address or a recycled slot cannot lose a waiter. It stands in for
a Proton build without a pack (droiddeck-ntsync where fsync is off), takes
over for games with esync off, and the "droiddeck-fsync first" toggle
(BL_FSYNC_FIRST, also per game) puts it ahead of the packs. The games
Proton keeps off fsync and esync never get it. A launch into a prefix
whose wineserver is still running keeps that server's sync choice.
One Proton for Steam and the app: the Components page's Proton is the
default Steam uses, applied live through the client while Steam runs
(droiddeck-steam-compat) and at the next start otherwise; a choice made in
Steam's menu comes back to the app. Requests are ordered by a sequence that
never goes backwards, and the page no longer jumps back to the default
after a swap or restore.
DirectAudio: the wrapper wrote the driver key with its backslashes
stripped, so no game ever used DirectAudio. Wine 11 also loads a builtin
only from the prefix, so the driver is now linked into system32 and
syswow64 before "directaudio,pulse" is selected; mmdevapi falls back to
winepulse whenever DirectAudio cannot start.
* Pick esync, ntsync or fsync from the Steam settings' Games group
A three-way tab row above the FEX preset sets the sync Proton games use. It
writes the existing switches together (droiddeck-ntsync, droiddeck-fsync
first, droiddeck-esync), so the Performance toggles and the tabs always agree,
starts on esync and keeps the choice until it is changed again.
* Make droiddeck-fsync the default sync for Proton games
A fresh install, or one that never picked a sync, starts on fsync; a choice made
in the Steam settings' Games group still sticks. Games Proton keeps off fsync
still fall back to droiddeck-esync.
* Add wineserver to the Wine sync tabs and order them ntsync, fsync, esync, wineserver
wineserver turns every in-process sync off so games wait through wineserver;
picking any other tab turns sync back on. The tabs shrink to fit a narrow pane.
* Shorten the Wine sync description
* Answer openat2 and execveat for FEX in the session preload
FEX opens every file of its x86 rootfs with openat2(RESOLVE_IN_ROOT) and
only falls back to the host path on EXDEV. Under proot on Android openat2
answers ENOSYS, so each lookup of an x86 library silently went to the host
and missed. The exported syscall() now turns openat2 into openat once the
real call reports ENOSYS: RESOLVE_IN_ROOT paths are taken relative to the
root fd (FEX has already resolved the links inside the rootfs), and an
absolute path under RESOLVE_BENEATH fails with EXDEV as the kernel would.
execveat with an absolute path, or relative to AT_FDCWD, and without
AT_EMPTY_PATH or AT_SYMLINK_NOFOLLOW, is plain execve; it is sent as such
because proot refuses execveat with a directory fd, which FEX uses when it
re-executes itself for a child x86 program.
* Run x86 Linux games and programs through FEX
Steam: Linux-only titles that have no ARM64 build (from the client's
appinfo cache: type game, demo, application or beta) are mapped to a new
compatibility tool, droiddeck-fex ("Linux x86 (FEX)", linux to linux,
no required runtime). Valve's own "fex" tool needs pressure-vessel, which
cannot run under proot, so a mapping to it or to a Steam Linux Runtime is
replaced; any other pick the user made stays. Titles that also ship for
Windows stay on Proton. When such a title is installed and FEX or the
sniper runtime is missing, the client is asked to install them.
droiddeck-fex runs the program under the FEX that Steam installs (app
3127680), with Steam Linux Runtime 3.0 (sniper, app 1628350) as the x86
rootfs:
- the sniper platform is linked in place from its usr-mtree (relative
links only, files/usr -> .), once per runtime version;
- the x86_64 and i386 builds of libblsession and libfakeinput go into the
runtime's lib/<triplet>/droiddeck and its etc/ld.so.preload, mirroring
the session's own list, so x86 programs get the pads and the session
fixes (FEX passes their system calls straight to the kernel, so the
ARM64 preloads cannot reach them);
- Config.json sets RootFS and the GL and Vulkan thunks, so x86 games draw
with the device's Turnip and Mesa;
- the FEX preset and the game's own environment from the Steam settings
apply, and ARM64 entries are dropped from LD_PRELOAD/LD_LIBRARY_PATH.
It also links ~/.steam/sdk64 and sdk32 to the client's x86 steamclient
when the client has one and nothing is there yet.
Desktop: x86_64 and i386 AppImages can be imported now; they unpack
through FEX (or unsquashfs) and record their architecture. Scripts,
AppImages and programs started in a run session use FEX automatically
when they are x86, and each added app has a "Use FEX" Auto/On/Off choice
in its edit dialog. The launchers' "still running" checks also look at
FEX processes, whose exe is FEX rather than the program. A GitHub release
without an ARM64 AppImage falls back to its x86_64 one. Decky's x86
loader starts through droiddeck-fex instead of the fex-mesa rootfs path,
which no runtime ships.
The x86 preloads are built against glibc 2.31 (Debian 11) so they load
in sniper: tools/linuxfs/build-x86-preloads.sh, run in CI and by
build_local.sh. fakeinput's stat hooks fall back to __xstat on a glibc
without a stat export, and mmap64 takes off64_t for the i386 build.
* Keep a FEX unixlib from outliving its DLL
A FEX package or original bundle without the aarch64-unix helpers left
the previous version's libarm64ecfex.so / libwow64fex.so in the Proton,
so the new DLLs loaded a unixlib built for another FEX. A swap now removes
the helpers the package does not carry; FEX then runs DLL-only, which it
already falls back to when Wine cannot load the unixlib.
* Fix the Wine sync choices an audit found
- wineserver tab: it passed Proton's environment through, so a build with
esync built in (GE, cachyos) still ran esync and a kernel ntsync would
still be used. It now sets PROTON_NO_ESYNC, PROTON_NO_FSYNC and
PROTON_NO_NTSYNC (unless the game's own environment sets them).
- ntsync tab: a Proton whose wineserver has no /dev/ntsync client got
wineserver-only sync although fsync would work. The hash cache records
whether the wineserver has one; without it the launch goes to
droiddeck-fsync, and no stand-in or pack turns droiddeck-ntsync on.
- A build with esync built in returned before the per-game fallbacks, so
a game that turns esync off ran with no in-process sync. It now gets
droiddeck-fsync, or droiddeck-ntsync where fsync is off too.
- When a pack's proton cannot start and the stock one is used, or the
selection fails, the prefix's record now says stock, so a second
launch into the same running prefix no longer picks the pack's ntdll.
- droiddeck-fsync: its mutex is held across fork and re-made in the
child, a closed or replaced shm fd is forgotten (close, dup2, dup3,
close_range, closefrom), and a waiter woken while it was leaving on a
timeout or signal reports the wake instead of losing it.
- BL_SYNC_MIN_NOFILE applies per game to every verb, debug logs carry
prefixes.json, the device report names the chosen sync instead of the
three raw switches, and PATCHES.md and the proot notes describe the
selector, the fsync default and droiddeck-fsync's futex_waitv.
* Name the runtime's scripts and files droiddeck, migrating old installs
The guest scripts are droiddeck-* now (droiddeck-session and the other 19),
as are the files the app and scripts generate (etc/droiddeck, the network
state files, the components dir, the Steam install stamp, the redist marker,
the extra-Proton wrapper and backups) and the compositor's own names
(libdroiddeckwayland, droiddeck_* sources, DROIDDECK_* switches, the
wl_output and seat names).
Installed runtimes carry on: LegacyNames moves etc/bannerlator and
share/bannerlator over and leaves relative links at the old paths before
every proot command, turns any old script into a link to its new name,
moves the generated files and points menu entries at the new scripts.
The session scripts and steam-compatibility move their markers on first
use, the components dir's desired.tsv is rewritten with it, and the
runtime counts as installed under either session script.
Kept: the banner_ahb_v1/banner_desktop_v1 wire names and BANNER_AUDIO_DIRECT
(read by prebuilt Turnip and Wine builds), Banners-Turnip and other
third-party names, BL_* switches, libblsession, and credits to Bannerlator.
* Unpack x86 AppImages without FEX, then ask Steam for FEX
Importing an x86 AppImage failed when FEX was not installed yet ("needs FEX
to unpack and no unsquashfs is installed"). AppImages now unpack with
uruntime (VHSgunzo, MIT, v0.8.1 aarch64, pinned by sha256 and shipped
unmodified with its licence), which reads SquashFS and DwarFS images of any
architecture natively. FEX's own --appimage-extract and unsquashfs stay as
fallbacks, and an ARM64 image whose own runtime cannot unpack it goes the
same way.
An x86 import with FEX missing records that FEX is wanted, so the next
Steam session requests FEX and the Steam Linux Runtime, and the program's
page says FEX is not set up yet and how to get it.
* Run Vulkan Qt Quick AppImages on Wayland
OpenNOW (Qt 6.8, Qt Quick on Vulkan) closed within two seconds in a run
mode. The session sends toolkits to gamescope's Xwayland, and Qt's xcb
backend only presents with Vulkan when Mesa's X11 WSI does, which needs
DRI3; Xwayland on KGSL has none, so Qt found no queue that can present and
aborted ("Failed to create RHI").
An AppImage that bundles Qt Quick, the Vulkan loader and Qt's Wayland
plugin now runs on gamescope's Wayland socket (the desktop's, outside
gamescope), where Turnip presents as it does for games, with Qt's xcb
backend as the fallback. SDL_VIDEODRIVER and XDG_SESSION_TYPE follow, so
helpers that pick their window system from them (OpenNOW's streamer) agree
with Qt. Everything else stays on X11; a qt file in the app's directory
(wayland or xcb) overrides the choice.
* Keep gamescope's WSI layer away from Qt apps on its Wayland socket
OpenNOW still closed at once: its FFmpeg (the Raspberry Pi fork, libavutil
61) creates its Vulkan queues internally synchronized whenever the driver
offers VK_KHR_internally_synchronized_queues, which DD-Turnip does. The
gamescope WSI layer registers every queue of a new device through
vkGetDeviceQueue, which only returns queues created without flags, so it
got VK_NULL_HANDLE and vkroots aborted on assert(obj) before Qt opened a
window.
Programs that draw on gamescope's Wayland socket do not need the layer,
which only carries X11 windows past Xwayland, so it is turned off for
them. OpenNOW bundles no Vulkan loader (it reaches Vulkan through Qt and
dlopen), so a binary that uses QVulkanInstance counts as a Vulkan app too,
and a Qt Quick app left on X11 says why in the session log.
* Keep the end of the session log, and say when FEX is missing
The session sends its output through head -c into session.log. head
writes through stdio, so it held up to 4 KB that proot's --kill-on-exit
threw away when the guest ended: every log stopped mid-line at a 4096-byte
boundary, losing exactly the lines that say why a program quit (an x86
AppImage's "x86 Linux programs need FEX" among them). dd writes each read
as it comes and keeps the same cap.
A run mode always ends with status 0 (gamescope does not pass its child's
on), so a program that could not start just closed. When the log says FEX
is missing, the ended screen now says so and how to get it.
The app reads FEX's readiness from status.json, which only a launch or an
import wrote, so it still said "not set up" after Steam installed FEX.
droiddeck-fex status rewrites it, and a Steam session ends with it while
FEX is wanted. FEX is also looked for at the tool's root and under files/,
and a tool folder without it lists what it holds.
* Let FEX and its runtime install runnable on an SD card library
A Steam library on shared storage loses the executable bits, so native
tools are staged in private directories bound over it; only Proton and
SteamLinuxRuntime folders were. FEX (3127680) installed there could never
run, droiddeck-fex saw no FEX and kept asking Steam for it. FEX and the
sniper runtime (1628350) now get private directories from their first
download, whatever their install folder is called.
* Keep fsync off in the pack smoke test
A runner whose kernel allows futex_waitv started stock Proton with fsync,
so the server-side baseline never said what the test waits for and every
pack was waved through as untestable; the esync run would have measured
fsync too.
* Judge a pack against stock's own esstress failures
Stock Proton fails esstress's dup wait all under server-side sync as well
(a Wine bug, seen on the phone too), and any stock failure made the test
call the runner unusable. A pack now fails only when it crashes, hangs or
fails a check stock passes.
* Ask Steam for FEX and its runtime one at a time
Both steam://install requests were started together, and the running client
took only the first: FEX installed, the sniper runtime FEX needs never did.
Each request now waits for the client to take it before the next.
* Bundle the first published droiddeck-esync packs in the apk
* Log how an x86 program under FEX ends
FEX keeps its own errors quiet unless asked, so a program it could not run
left nothing in the session log. Its log now goes to stderr, and the AppImage
launcher records the program's exit status or the signal that ended it.
* Show the log a failed AppImage program wrote
Programs like Eden stop themselves on a failed check and say why only in
their own log under the home folder. When the program ends badly, the end of
any log it wrote during the run is copied into the session log.
* Report where an x86 program under FEX hit an illegal instruction
A tiny x86_64 preload, built without libc, catches SIGILL and writes the
address, the instruction bytes and the mapping they belong to before the
program dies as it would have. droiddeck-fex preloads it into x86_64 programs.
* Load the fault reporter from the x86 runtime
Programs under FEX look libraries up in the x86 runtime, so the reporter is
copied there with the other preloads and named by its path inside it.
* Hand FEX its x86 runtime and thunks through the environment
Valve's FEX runs portable and reads its config beside itself, so the
Config.json droiddeck-fex writes was not seen: programs ran without the
runtime. The runtime, thunk libraries and thunk list now also go in FEX's
own environment variables, and each launch says what it was given.
* Use the sync pack built for the installed Proton release
Some Proton releases ship byte-identical ntdll.so and wineserver (Valve
11.0-2b/2c, CachyOS 0702/0703), so a pack built for a newer release
matched an older install exactly and the older install never fetched its
own pack. A pack from a newer release no longer counts as an exact match;
among matches the one built for the installed version wins, then rev.
The app ranks the pack a tool already uses the same way and only fetches
a pack that ranks higher, reads the release epoch from tools.tsv and
refetches when tools.tsv changes. A prefix whose server is still running
re-checks its recorded pack on waitforexitandrun and starts fresh when a
better one now applies.
* Keep published sync packs and revocations consistent
The publish step never replaces a pack that has an entry beside it; a
pack left without its entry by an interrupted run is replaced so the
build stops repeating. Discovery counts revocations the signed index does
not show yet, including superseded packs only the flavor releases list,
so the index is signed again. The release watch always builds every
flavor.
* Fetch the droiddeck-esync packs from DroidDeck-Components
The packs, the signed index and the apk bundle now live in the releases
of Droid-Deck/DroidDeck-Components, together with the pipeline that
builds them, so this repository keeps only the app and the guest code
that consumes them. The app's index and asset URLs point there,
release.env names the repository its bundle comes from, and the apk
builds drop bundled packs that the published index revokes.
* Stop the apk build when the published sync index cannot be read
* Ask Steam for one FEX dependency at a time
The client shows a confirmation dialog for each steam://install, and a
second request replaces the first dialog, so FEX was never installed
when the runtime was requested right after it. The next dependency is
now requested once the previous one has an app manifest.
* Use generic names in the AppImage test fixtures
* Give x86-64 apps under FEX the Vulkan thunk
Apps that bring their own glibc and Vulkan loader skipped FEX's thunk
overlay and loaded their bundled desktop Mesa drivers, which crash on the
phone's GPU during Vulkan setup. droiddeck-fex now adds an x86-64 audit
module when a program's bundled glibc and libstdc++ meet the thunk's
version needs. The module sends libvulkan.so.1 lookups to a nodelete stub
that needs FEX's thunk path. That keeps the thunk mapped when an app
unloads Vulkan and loads it again, since FEX keeps its first trampoline
links and the old addresses get reused.
The fault reporter now restores the default action before re-raising,
because FEX ignores SA_RESETHAND and the handler looped.
This commit is contained in:
1 parent
4cc9f2a3f3
commit
49a5f4aa94
165 files changed
+11709
-9667
No files matched your search
@@ -1,170 +0,0 @@
|
||||
# droiddeck-esync packs
|
||||
|
||||
DroidDeck gives Proton three in-process sync backends, picked with the Wine sync tabs in the
|
||||
Steam settings' Games group: ntsync, fsync (the default), esync and wineserver. The wineserver tab
|
||||
sets `PROTON_NO_ESYNC=1`, `PROTON_NO_FSYNC=1` and `PROTON_NO_NTSYNC=1` (unless a game's own
|
||||
environment sets them), so every sync object goes through the server.
|
||||
|
||||
- droiddeck-esync: eventfd and shared memory sync. Stock Wine 11 / Proton 11 builds
|
||||
have no eventfd-based sync, so the patch series under `patches/<series>` adds it to wineserver
|
||||
and the unix side of ntdll. A Proton build without a pack (a release newer than the published
|
||||
packs, or one whose pack failed to install) runs stock with droiddeck-fsync standing in, or
|
||||
droiddeck-ntsync where fsync is off for the game (`PROTON_NO_FSYNC=1`, or one of the games
|
||||
Proton itself keeps off fsync and esync), so it still gets in-process sync; `PROTON_NO_NTSYNC=1`
|
||||
as well leaves it on wineserver-only sync. A Proton with esync built in (GE, cachyos) runs its
|
||||
own esync, and a game that turns esync off moves to droiddeck-fsync or droiddeck-ntsync there
|
||||
too.
|
||||
- droiddeck-fsync: Proton's own fsync on the stock binaries. fsync waits with `futex_waitv(2)`,
|
||||
which the app sandbox refuses, so `libblsession.so` (`BL_FSYNC=1`) answers that system call in
|
||||
userspace: every waiter registers the futexes it waits on in a shared table next to the fsync
|
||||
shared memory and sleeps on a futex of its own, and every `FUTEX_WAKE` on the fsync shared memory
|
||||
wakes the waiters registered on that word. No pack is involved, so it keeps working the moment
|
||||
Steam updates Proton. It is the default (the fsync tab, `BL_FSYNC_FIRST=1`, also per game),
|
||||
ahead of droiddeck-esync; on the esync tab it stands in where no pack fits, and it also takes
|
||||
over for a game whose environment turns esync off (`PROTON_NO_ESYNC=1`, `WINEESYNC=0`, or too
|
||||
few file descriptors); where fsync is off as well, such a game gets droiddeck-ntsync, with the
|
||||
pack's wineserver when a pack fits. A launch into a prefix whose wineserver is still running
|
||||
keeps the sync choice that server was started with (`prefixes.json` in the store), because a
|
||||
Wine client that disagrees with its server about fsync exits. It inherits stock fsync's behaviour, the same as on a Linux kernel:
|
||||
a wait-all on a mutex abandoned by a dead thread is not satisfied, a pending user APC is
|
||||
delivered ahead of an object that is already signaled, a mutex owned by a process killed with
|
||||
`TerminateProcess` is not abandoned, and `PulseEvent` can miss a waiter that has not run yet.
|
||||
- droiddeck-ntsync: a userspace implementation of the `/dev/ntsync` interface in
|
||||
`libblsession.so` (`BL_SYNC=1`), which stock Proton 11 builds use as they would the kernel
|
||||
driver. It is the ntsync tab. A pack still supplies the binaries when one fits, and its
|
||||
wineserver then picks ntsync over esync; a Proton whose wineserver has no ntsync client (no
|
||||
`/dev/ntsync` in it) gets droiddeck-fsync instead.
|
||||
|
||||
wineserver picks the backend at startup: `PROTON_NO_NTSYNC` unset -> `/dev/ntsync`
|
||||
(droiddeck-ntsync, or a kernel driver the device provides) -> fsync (droiddeck-fsync, or a kernel
|
||||
and sandbox that allow `futex_waitv`) -> droiddeck-esync (needs `WINEESYNC` nonzero) ->
|
||||
server-side sync. When a pack is in use and droiddeck-esync is on, the guest's `droiddeck-esync`
|
||||
script also sets `PROTON_NO_FSYNC=1` unless the game's environment sets it, so droiddeck-esync
|
||||
comes before fsync unless droiddeck-fsync was chosen. A wineserver from a pack reports the
|
||||
backend on stderr as `droiddeck-ntsync: up and running.` or `droiddeck-esync: up and running.`;
|
||||
with droiddeck-fsync the stock wineserver prints `fsync: up and running.` after
|
||||
`droiddeck-fsync: up and running.`. A pack's wineserver also abandons, when a process dies, every
|
||||
mutex still owned by one of its threads, and, once a thread killed by `TerminateThread` has really
|
||||
exited, every mutex it still owns. A killed thread keeps running until the kill signal reaches it
|
||||
and can take a mutex after the server abandoned its mutexes; without these passes that mutex would
|
||||
stay locked for good (they cover droiddeck-ntsync and kernel ntsync as well). The ntdll side
|
||||
reads, writes and polls its eventfds with raw system calls, so no `LD_PRELOAD` wrapper of
|
||||
`read`/`write`/`poll` (the guest preloads several) sits in the sync path or in the unwind of a
|
||||
thread Wine kills.
|
||||
|
||||
The patch changes no wineserver request: `make_pack.py` refuses a build whose
|
||||
`server_protocol.h`, `request_handlers.h` or `request_trace.h` differ from the unpatched
|
||||
control build, or whose ntdll.so exports other symbols. Only `files/lib/wine/aarch64-unix/ntdll.so`
|
||||
and `files/bin-arm64/wineserver` are replaced, so every Proton build needs its own pair, built
|
||||
from that build's exact wine source. That pair plus `pack.json` is a pack (`<id>.tzst`). On the
|
||||
device the guest runs Proton from a shadow tree that links the stock tool and holds the two
|
||||
patched files (`droiddeck-esync`, store in `~/.local/share/droiddeck-esync`).
|
||||
|
||||
## Credits
|
||||
|
||||
droiddeck-esync follows the esync design and droiddeck-ntsync implements the ntsync interface;
|
||||
both are the work of Elizabeth Figura for Wine, Proton and the Linux kernel. droiddeck-fsync runs
|
||||
fsync, by Elizabeth Figura and Paul Gofman for Proton, over `futex_waitv(2)` by André Almeida.
|
||||
DroidDeck builds on that work for Android, where neither the ntsync driver, `futex_waitv` nor a
|
||||
Proton build with esync is available.
|
||||
|
||||
## Matching
|
||||
|
||||
- `ge`, `cachyos`: built against the release tarball's own binaries. `pack.json` `stock` holds
|
||||
their sha256; the guest uses the pack only when both files hash the same. `make_pack.py` also
|
||||
requires the control build to reproduce both files byte for byte, so the patched pair differs
|
||||
from the release only by the patch (`--allow-unreproduced` lifts this for local experiments).
|
||||
- `valve`: the depots cannot be downloaded without a Steam login, so `stock` holds the control
|
||||
build's hashes and `source_match` is true: the guest also accepts a tool whose version token
|
||||
(second field of `<tool>/version`) and ntdll.so exports hash equal the pack's. With the optional
|
||||
`STEAM_DEPOT_USER` / `STEAM_DEPOT_TOKEN` secrets (and the `DEPOT_DOWNLOADER_SHA256` variable)
|
||||
the job downloads the depot instead and, when the depot is at the tag being built, makes an
|
||||
exact pack. The guest's launch line says which kind matched (`exact match` or `source match`).
|
||||
- A stock wineserver that already has esync built in (it prints `esync: up and running`) never
|
||||
gets a pack.
|
||||
|
||||
## Building and publishing
|
||||
|
||||
`.github/workflows/watch-proton-releases.yml` runs every 20 minutes and decides when a build is
|
||||
due. It asks Steam anonymously (`steamcmd +app_info_print`) which public build of each Valve ARM64
|
||||
depot app in `flavors.json` is live, and `discover.py watch` compares that, the newest `keep` tags
|
||||
and releases of every flavor, each flavor's `rev` and its patch series digest with the state of the
|
||||
last run (`watch-state.json` in the `droiddeck-esync-index` release). A flavor whose builds, rev or
|
||||
patch series changed, or whose Steam build moved, starts
|
||||
`.github/workflows/build-droiddeck-esync-packs.yml` for that flavor; nothing else starts a build.
|
||||
A run that finds no change takes about a minute and builds nothing. If steamcmd fails, the run
|
||||
watches the tags only and keeps the last Steam state.
|
||||
|
||||
`.github/workflows/build-droiddeck-esync-packs.yml` also runs once a day as a safety net, on pushes
|
||||
to `tools/droiddeck-esync/**` and by hand (flavor, one tag, publish yes/no). Only runs on `main` of
|
||||
`Droid-Deck/DroidDeck` publish packs; the schedule and pushes do nothing in other repositories,
|
||||
and manual runs elsewhere only build.
|
||||
|
||||
1. `discover.py matrix` lists the newest `keep` builds of each family in `flavors.json` (Valve
|
||||
tags, GE and CachyOS releases with an arm64 tarball) and drops the ones the published index,
|
||||
or the `droiddeck-esync-<flavor>` release, already has at the flavor's `rev`, the ones
|
||||
`revoked.txt` names, and the ones that failed in the last day at the same `rev` and patch
|
||||
series (a `<key>-r<rev>-<series digest>.failed` asset in the flavor's release). Naming the tag
|
||||
in a manual run builds it anyway.
|
||||
2. Each build runs on an arm64 runner with a read-only token, since it runs upstream code: the
|
||||
wine source is checked out at the tag (GE also runs its `protonprep-valve-staging.sh`),
|
||||
`build-pack.sh` builds a control and a patched pair inside the SDK image named by the
|
||||
source's own `Makefile.in` (arm64-llvm), at the flavor's `build_dir` in `flavors.json` (the
|
||||
path the release itself was built at, which ends up in the binaries), and `make_pack.py` makes
|
||||
the pack. Patches apply with `--fuzz=0`: a hunk whose context moved fails the build instead of
|
||||
landing somewhere else. For GE and CachyOS, whose release tarball holds the whole Proton,
|
||||
`smoke-test.sh` then boots a prefix with that stock Proton and runs `tests/esstress.c` (built
|
||||
with the SDK's llvm-mingw) under server-side sync as a baseline, and again with the pack under
|
||||
droiddeck-esync. A pack that fails while stock passes is not published; when stock itself cannot
|
||||
run on the runner the step only warns.
|
||||
3. The publish job checks each build's artifact against the matrix (`discover.py packs`) and
|
||||
uploads it to the `droiddeck-esync-valve`, `droiddeck-esync-ge` or `droiddeck-esync-cachyos`
|
||||
release as `<id>.tzst` with its index entry `<id>.json`. A build that made no pack gets its
|
||||
`.failed` marker there (`discover.py failures`), so a patch that does not apply or a stock
|
||||
wineserver with esync built in is not rebuilt on every run.
|
||||
4. The index job (environment `droiddeck-esync-signing`) runs only when the run built a pack or
|
||||
a published pack is not in the index yet. It collects every `<id>.json`, keeps the
|
||||
highest `rev` per stock pair and version, marks revoked ids, signs `index.json` (ECDSA P-256,
|
||||
`index-key.pub`) and publishes `bundle-<YYYYMMDD>-<HHMMSS>.tzst` (named after `generated`, so
|
||||
every signed index has its own), `index.json` and `index.json.sig` to `droiddeck-esync-index`.
|
||||
`generated` only grows. Packs stay in the index until they are revoked, also when their
|
||||
entry asset disappears; the previous index counts only when its signature verifies.
|
||||
|
||||
A pack is rebuilt only for a new `rev`: bump `rev` of the flavor in `flavors.json` whenever the
|
||||
patch series or the build changes. Local builds: `BUILD_DIR=<the flavor's build_dir> SYNC_PROFILE=<the
|
||||
flavor's profile> build-local.sh <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]`,
|
||||
then `make_pack.py`.
|
||||
|
||||
## Setup
|
||||
|
||||
- Signing key: create the repository environment `droiddeck-esync-signing` (Settings ->
|
||||
Environments, limit it to `main`) and store the PEM private key as its secret
|
||||
`SYNC_INDEX_KEY`: `gh secret set SYNC_INDEX_KEY --env droiddeck-esync-signing < index-key.pem`.
|
||||
The key never enters the repository; the app trusts only `index-key.pub` (also compiled into
|
||||
`EsyncPacks.kt`), so a new key needs an app release.
|
||||
- Optional: `STEAM_DEPOT_USER` and `STEAM_DEPOT_TOKEN` repository secrets for exact Valve packs,
|
||||
plus the repository variable `DEPOT_DOWNLOADER_SHA256`: the sha256 of
|
||||
`DepotDownloader-linux-arm64.zip` from the DepotDownloader release the workflow pins
|
||||
(`DEPOT_DOWNLOADER_TAG`). The depot step is skipped while it is unset.
|
||||
- APK bundle: after an index run, the job summary prints three lines. Put them in
|
||||
`tools/droiddeck-esync/release.env`:
|
||||
|
||||
```
|
||||
SYNC_BUNDLE_TAG=droiddeck-esync-index
|
||||
SYNC_BUNDLE_ASSET=bundle-20261002-061733.tzst
|
||||
SYNC_BUNDLE_SHA256=<sha256>
|
||||
```
|
||||
|
||||
`build.yml` then unpacks that bundle into `app/src/main/assets/droiddeck-esync/` (gitignored). A
|
||||
published bundle is never overwritten, so the pinned sha256 stays valid; bump `release.env`
|
||||
to ship newer packs inside the apk. The app fetches newer packs on its own. A pack the bundle
|
||||
carries that `revoked.txt` names is left out of the apk, with a warning to bump `release.env`.
|
||||
|
||||
## Revocation
|
||||
|
||||
Add the pack id as the first word of a line in `revoked.txt` and push to `main`. The workflow
|
||||
re-signs the index with `"revoked": true` for that id; the app deletes installed copies and
|
||||
never installs it again, and `discover.py` does not rebuild it at the same `rev`. The apk build
|
||||
also leaves it out of the bundled packs. To ship a fixed pack for the same Proton build, bump the
|
||||
flavor's `rev`. Runs that only build (other refs, or manual runs without publish) never queue in
|
||||
front of a publishing run, so a revocation is signed by the next run on `main`.
|
||||
@@ -1,27 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
if [ $# -lt 4 ]; then
|
||||
echo "usage: $0 <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]" >&2
|
||||
echo "env: BUILD_DIR (path the build runs at inside the SDK), SYNC_PROFILE (valve or cachyos), JOBS, SOURCE_DATE_EPOCH" >&2
|
||||
exit 2
|
||||
fi
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
sdk=$(realpath "$1")
|
||||
src=$(realpath "$2")
|
||||
work=$(realpath -m "$3")
|
||||
out=$(realpath -m "$4")
|
||||
patches=${5:+$(realpath "$5")}
|
||||
build_dir=${BUILD_DIR:-/builds/proton/proton/build-dir}
|
||||
rm -rf "$work"
|
||||
mkdir -p "$work" "$out"
|
||||
"$here/build-pack.sh" prepare "$src" "$work" ${patches:+"$patches"}
|
||||
bwrap --unshare-user --uid 0 --gid 0 --unshare-ipc --unshare-pid \
|
||||
--bind "$sdk" / --proc /proc --dev /dev --tmpfs /tmp \
|
||||
--bind "$work" "$build_dir" --bind "$out" /out --ro-bind "$here/build-pack.sh" /build-pack.sh \
|
||||
--setenv PATH /opt/llvm-mingw/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
||||
--setenv HOME /tmp --setenv LANG C.UTF-8 --setenv JOBS "${JOBS:-$(nproc)}" \
|
||||
--setenv SOURCE_DATE_EPOCH "${SOURCE_DATE_EPOCH:-1700000000}" --setenv SYNC_PROFILE "${SYNC_PROFILE:-valve}" \
|
||||
--chdir "$build_dir" \
|
||||
/bin/bash /build-pack.sh build "$build_dir" /out
|
||||
sha256sum "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
|
||||
@@ -1,125 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
echo "usage: $0 prepare <wine-source> <build-dir> [patch-dir]" >&2
|
||||
echo " $0 build <build-dir> <out-dir>" >&2
|
||||
echo " $0 all <wine-source> <out-dir> [patch-dir]" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
prepare() {
|
||||
local origin=$1 b=$2 patches=${3:-}
|
||||
local s=$b/src-wine
|
||||
mkdir -p "$s"
|
||||
rsync --filter=:C --exclude '*~' --exclude .git --exclude compile_commands.json \
|
||||
--exclude configure --exclude autom4te.cache --exclude include/config.h.in --exclude include/config.h.ink \
|
||||
--exclude include/wine/vulkan.h --exclude dlls/vulkan-1/vulkan-1.spec \
|
||||
--exclude dlls/winevulkan/loader_thunks.c --exclude dlls/winevulkan/loader_thunks.h \
|
||||
--exclude dlls/winevulkan/vulkan_thunks.c --exclude dlls/winevulkan/vulkan_thunks.h \
|
||||
--exclude dlls/winevulkan/winevulkan.json --exclude dlls/winevulkan/winevulkan.spec \
|
||||
--exclude dlls/ntdll/ntsyscalls.h --exclude dlls/win32u/win32syscalls.h \
|
||||
--exclude include/wine/server_protocol.h --exclude server/request_handlers.h --exclude server/request_trace.h \
|
||||
-Oarx --delete "$origin/" "$s/"
|
||||
for f in include/wine/server_protocol.h server/request_handlers.h server/request_trace.h; do
|
||||
cp -a "$origin/$f" "$s/$f"
|
||||
done
|
||||
if [ -n "$patches" ]; then
|
||||
local p found=0
|
||||
for p in "$patches"/*.patch; do
|
||||
[ -f "$p" ] || continue
|
||||
patch -d "$s" -p1 --forward --fuzz=0 --no-backup-if-mismatch -s < "$p"
|
||||
found=1
|
||||
done
|
||||
[ "$found" = 1 ] || { echo "no patches in $patches" >&2; exit 1; }
|
||||
fi
|
||||
(
|
||||
cd "$s"
|
||||
XDG_CACHE_HOME=$b/cache dlls/winevulkan/make_vulkan -x vk.xml -X video.xml
|
||||
tools/make_specfiles || true
|
||||
tools/make_requests
|
||||
) > "$b/generate.log" 2>&1
|
||||
touch "$b/.prepared"
|
||||
}
|
||||
|
||||
build() {
|
||||
local b=$1 out=$2
|
||||
local s=$b/src-wine o=$b/obj-wine-aarch64 d=$b/dst-wine-aarch64
|
||||
[ -f "$b/.prepared" ] || { echo "$b is not prepared" >&2; exit 1; }
|
||||
local jobs=${JOBS:-$(nproc)}
|
||||
local epoch=$(( ${SOURCE_DATE_EPOCH:-1700000000} - 20 ))
|
||||
local flags args pe_common i386_flags x86_64_flags conf
|
||||
case ${SYNC_PROFILE:-valve} in
|
||||
valve)
|
||||
flags="-Wno-discarded-qualifiers -march=armv8.2-a -mtune=cortex-x3 -O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
args=(--enable-werror --with-mingw=clang --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
|
||||
pe_common="-O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
i386_flags="-Wno-discarded-qualifiers -mstackrealign -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
|
||||
x86_64_flags="-Wno-discarded-qualifiers -mcmodel=small -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
|
||||
conf="../$(basename "$s")/configure"
|
||||
;;
|
||||
cachyos)
|
||||
flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -O2 -march=armv8.2-a -mtune=cortex-x3 -fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
args=(--with-mingw=clang --enable-build-id --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
|
||||
pe_common="-fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
|
||||
i386_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mstackrealign -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
|
||||
x86_64_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mcmodel=small -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
|
||||
conf="$s/configure"
|
||||
;;
|
||||
*)
|
||||
echo "unknown SYNC_PROFILE ${SYNC_PROFILE}" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
local env=(STRIP=llvm-strip AR=aarch64-linux-gnu-ar RANLIB=aarch64-linux-gnu-ranlib
|
||||
CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ LD=aarch64-linux-gnu-ld
|
||||
RC=aarch64-w64-mingw32-windres WIDL=aarch64-w64-mingw32-widl PKG_CONFIG=aarch64-linux-gnu-pkg-config
|
||||
"PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig"
|
||||
"CFLAGS=$flags" "CPPFLAGS=$flags" "CXXFLAGS=-std=c++17 $flags" "LDFLAGS=" "SOURCE_DATE_EPOCH=$epoch"
|
||||
"CROSSCFLAGS=$flags" "CROSSLDFLAGS="
|
||||
"aarch64_CFLAGS=$flags" "aarch64_CPPFLAGS=$flags" "aarch64_CXXFLAGS=-std=c++17 $flags" "aarch64_LDFLAGS="
|
||||
i386_AR=i686-w64-mingw32-ar i386_RANLIB=i686-w64-mingw32-ranlib i386_CC=i686-w64-mingw32-gcc
|
||||
i386_CXX=i686-w64-mingw32-g++ i386_LD=i686-w64-mingw32-ld
|
||||
"i386_CFLAGS=$i386_flags" "i386_CPPFLAGS=$i386_flags" "i386_CXXFLAGS=-std=c++17 $i386_flags" "i386_LDFLAGS="
|
||||
"i386_PKG_CONFIG_LIBDIR=/usr/lib/i386-w64-mingw32/pkgconfig:/usr/share/pkgconfig"
|
||||
x86_64_AR=x86_64-w64-mingw32-ar x86_64_RANLIB=x86_64-w64-mingw32-ranlib x86_64_CC=x86_64-w64-mingw32-gcc
|
||||
x86_64_CXX=x86_64-w64-mingw32-g++ x86_64_LD=x86_64-w64-mingw32-ld
|
||||
"x86_64_CFLAGS=$x86_64_flags" "x86_64_CPPFLAGS=$x86_64_flags" "x86_64_CXXFLAGS=-std=c++17 $x86_64_flags" "x86_64_LDFLAGS="
|
||||
"x86_64_PKG_CONFIG_LIBDIR=/usr/lib/x86_64-w64-mingw32/pkgconfig:/usr/share/pkgconfig")
|
||||
export PATH=/opt/llvm-mingw/bin:$PATH
|
||||
( cd "$s" && autoreconf -fi && rm -rf autom4te.cache ) > "$b/autoreconf.log" 2>&1
|
||||
rm -rf "$o" "$d"
|
||||
mkdir -p "$o" "$d"
|
||||
( cd "$o" && env "${env[@]}" "$conf" -C --prefix="$d" --libdir="$d/lib" --host=aarch64-linux-gnu "${env[@]}" \
|
||||
"${args[@]}" ) > "$b/configure.log" 2>&1
|
||||
( cd "$o" && env "${env[@]}" make -j"$jobs" __tooldeps__ ) > "$b/tools.log" 2>&1
|
||||
( cd "$o" && env "${env[@]}" make -j"$jobs" dlls/ntdll/ntdll.so server/wineserver ) > "$b/make.log" 2>&1
|
||||
mkdir -p "$out/files/lib/wine/aarch64-unix" "$out/files/bin-arm64"
|
||||
llvm-objcopy --strip-debug --set-section-flags .text=contents,alloc,load,readonly,code \
|
||||
"$o/dlls/ntdll/ntdll.so" "$out/files/lib/wine/aarch64-unix/ntdll.so"
|
||||
llvm-objcopy --strip-debug "$o/server/wineserver" "$out/files/bin-arm64/wineserver"
|
||||
chmod 0755 "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
|
||||
cp "$s/include/wine/server_protocol.h" "$s/server/request_handlers.h" "$s/server/request_trace.h" "$out/"
|
||||
sed -n 's/^#define SERVER_PROTOCOL_VERSION \([0-9]*\)$/\1/p' "$out/server_protocol.h" > "$out/protocol"
|
||||
}
|
||||
|
||||
[ $# -ge 1 ] || usage
|
||||
case $1 in
|
||||
prepare)
|
||||
[ $# -ge 3 ] || usage
|
||||
prepare "$(realpath "$2")" "$(realpath -m "$3")" "${4:+$(realpath "$4")}"
|
||||
;;
|
||||
build)
|
||||
[ $# -eq 3 ] || usage
|
||||
build "$(realpath "$2")" "$(realpath -m "$3")"
|
||||
;;
|
||||
all)
|
||||
[ $# -ge 3 ] || usage
|
||||
b=${BUILD_DIR:-/builds/proton/proton/build-dir}
|
||||
prepare "$(realpath "$2")" "$b" "${4:+$(realpath "$4")}"
|
||||
build "$b" "$(realpath -m "$3")"
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
@@ -1,654 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from make_index import check_entry
|
||||
from make_pack import FLAVORS, HERE, PATCHES, REVOKED, SERIES, PackError, patch_digest, read_revoked, sha256_file, slug
|
||||
|
||||
FLAVORS_FILE = HERE / "flavors.json"
|
||||
ARCHIVE = re.compile(r"\.tar\.(gz|xz)$")
|
||||
ASSIGNMENT = re.compile(r"(?:export\s+|override\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*(?:\?|\+|!|:{1,3})?=\s*(.*)")
|
||||
REFERENCE = re.compile(r"\$[({]([A-Za-z_][A-Za-z0-9_]*)[)}]")
|
||||
IMAGE = re.compile(r"[A-Za-z0-9][A-Za-z0-9.-]*(?::[0-9]+)?(?:/[A-Za-z0-9._-]+)+(?::[A-Za-z0-9._-]+)?(?:@sha256:[0-9a-f]{64})?")
|
||||
REPO = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+")
|
||||
CLONES = ("shallow", "treeless")
|
||||
PACK_REV = re.compile(r"-r([0-9]+)$")
|
||||
ARTIFACT_PREFIX = "droiddeck-esync-pack-"
|
||||
FAILED = ".failed"
|
||||
RETRY_AFTER = timedelta(days=1)
|
||||
VDF_TOKEN = re.compile(r'"((?:[^"\\]|\\.)*)"|([{}])')
|
||||
APP_HEADER = re.compile(r"^AppID : ([0-9]+),", re.M)
|
||||
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
|
||||
|
||||
def gh(*args: str) -> subprocess.CompletedProcess:
|
||||
try:
|
||||
return subprocess.run(["gh", *args], capture_output=True, text=True, check=False)
|
||||
except FileNotFoundError as e:
|
||||
raise PackError("the GitHub CLI (gh) is not installed") from e
|
||||
|
||||
|
||||
def json_values(text: str) -> list:
|
||||
decoder = json.JSONDecoder()
|
||||
values = []
|
||||
position = 0
|
||||
while True:
|
||||
while position < len(text) and text[position].isspace():
|
||||
position += 1
|
||||
if position == len(text):
|
||||
return values
|
||||
value, position = decoder.raw_decode(text, position)
|
||||
values.append(value)
|
||||
|
||||
|
||||
def gh_api(path: str, paginate: bool = False, missing_ok: bool = False):
|
||||
result = gh("api", *(["--paginate"] if paginate else []), path)
|
||||
if result.returncode != 0:
|
||||
if missing_ok and "HTTP 404" in result.stderr:
|
||||
return None
|
||||
raise PackError(f"gh api {path}: {result.stderr.strip() or f'exit {result.returncode}'}")
|
||||
try:
|
||||
values = json_values(result.stdout)
|
||||
except ValueError as e:
|
||||
raise PackError(f"gh api {path}: {e}") from e
|
||||
if len(values) == 1:
|
||||
return values[0]
|
||||
if values and all(isinstance(value, list) for value in values):
|
||||
return [item for value in values for item in value]
|
||||
raise PackError(f"gh api {path}: unexpected output")
|
||||
|
||||
|
||||
def quote(value: str) -> str:
|
||||
return urllib.parse.quote(value, safe="")
|
||||
|
||||
|
||||
def load_flavors(path: Path) -> dict:
|
||||
try:
|
||||
data = json.loads(path.read_text())
|
||||
except (OSError, ValueError) as e:
|
||||
raise PackError(f"cannot read {path}: {e}") from e
|
||||
|
||||
def fail(why: str):
|
||||
raise PackError(f"{path}: {why}")
|
||||
|
||||
def compiles(pattern: str, where: str):
|
||||
try:
|
||||
re.compile(pattern)
|
||||
except re.error as e:
|
||||
fail(f"{where}: {pattern!r} is not a regular expression: {e}")
|
||||
|
||||
if not isinstance(data, dict) or data.get("format") != 1 or not isinstance(data.get("flavors"), dict):
|
||||
fail("is not a format 1 flavor list")
|
||||
image = data.get("sdk_image")
|
||||
if not isinstance(image, dict) or not all(isinstance(image.get(key), str) and image[key] for key in ("file", "variable", "match")):
|
||||
fail("sdk_image needs file, variable and match")
|
||||
for name, cfg in data["flavors"].items():
|
||||
if name not in FLAVORS:
|
||||
fail(f"unknown flavor {name!r}")
|
||||
if not isinstance(cfg, dict) or not isinstance(cfg.get("repo"), str) or not REPO.fullmatch(cfg["repo"]):
|
||||
fail(f"{name}: repo must be owner/name")
|
||||
for key in ("keep", "rev"):
|
||||
if type(cfg.get(key)) is not int or cfg[key] < 1:
|
||||
fail(f"{name}: {key} must be a positive number")
|
||||
if not isinstance(cfg.get("series"), str) or not SERIES.fullmatch(cfg["series"]):
|
||||
fail(f"{name}: series is not a patch series name")
|
||||
if cfg.get("profile") not in ("valve", "cachyos"):
|
||||
fail(f"{name}: profile must be valve or cachyos")
|
||||
if type(cfg.get("source_match")) is not bool:
|
||||
fail(f"{name}: source_match must be a boolean")
|
||||
build_dir = cfg.get("build_dir")
|
||||
if not isinstance(build_dir, str) or not re.fullmatch(r"(?:/[A-Za-z0-9._-]+)+", build_dir) or ".." in build_dir.split("/"):
|
||||
fail(f"{name}: build_dir must be the absolute path the release was built at")
|
||||
wine = cfg.get("wine")
|
||||
submodules = wine.get("submodules") if isinstance(wine, dict) else None
|
||||
if not isinstance(submodules, list) or "wine" not in submodules \
|
||||
or not all(isinstance(item, str) and re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", item) for item in submodules):
|
||||
fail(f"{name}: wine.submodules must list the submodules to check out, wine among them")
|
||||
if wine.get("clone") not in CLONES:
|
||||
fail(f"{name}: wine.clone must be one of {', '.join(CLONES)}")
|
||||
prep = wine.get("prep")
|
||||
if not isinstance(prep, str) or (prep and not re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", prep)) or ".." in prep.split("/"):
|
||||
fail(f"{name}: wine.prep must be a path inside the checkout or empty")
|
||||
if cfg.get("list") == "tags":
|
||||
families = cfg.get("families")
|
||||
if not isinstance(families, dict) or not families:
|
||||
fail(f"{name}: families must name at least one tag family")
|
||||
for family, settings in families.items():
|
||||
if not isinstance(settings, dict) or not isinstance(settings.get("tag"), str) \
|
||||
or not isinstance(settings.get("prefix"), str) or not re.fullmatch(r"[A-Za-z0-9._-]+", settings["prefix"]) \
|
||||
or not isinstance(settings.get("depot_app", ""), str):
|
||||
fail(f"{name}: family {family} needs a tag prefix and a tag pattern")
|
||||
compiles(settings["tag"], f"{name}: family {family}")
|
||||
if not isinstance(cfg.get("version_token"), str) or "{tag}" not in cfg["version_token"]:
|
||||
fail(f"{name}: version_token must contain {{tag}}")
|
||||
elif cfg.get("list") == "releases":
|
||||
if not isinstance(cfg.get("tag"), str) or not isinstance(cfg.get("assets"), list) or not cfg["assets"] \
|
||||
or not all(isinstance(pattern, str) for pattern in cfg["assets"]):
|
||||
fail(f"{name}: releases need a tag pattern and asset patterns")
|
||||
for pattern in (cfg["tag"], *cfg["assets"]):
|
||||
compiles(pattern, name)
|
||||
else:
|
||||
fail(f"{name}: list must be tags or releases")
|
||||
return data
|
||||
|
||||
|
||||
def natural_key(text: str) -> list[tuple]:
|
||||
return [(0, int(part), "") if part[0] in "0123456789" else (1, 0, part) for part in re.findall(r"[0-9]+|[^0-9]+", text)]
|
||||
|
||||
|
||||
def candidate(flavor: str, cfg: dict, tag: str, asset: str = "", token: str = "", depot_app: str = "") -> dict:
|
||||
return {
|
||||
"key": slug(f"{flavor}-{ARCHIVE.sub('', asset) if asset else tag}"),
|
||||
"flavor": flavor,
|
||||
"repo": cfg["repo"],
|
||||
"tag": tag,
|
||||
"asset": asset,
|
||||
"token": token,
|
||||
"depot_app": depot_app,
|
||||
"rev": cfg["rev"],
|
||||
"series": cfg["series"],
|
||||
"profile": cfg["profile"],
|
||||
"source_match": cfg["source_match"],
|
||||
"submodules": " ".join(cfg["wine"]["submodules"]),
|
||||
"clone": cfg["wine"]["clone"],
|
||||
"prep": cfg["wine"]["prep"],
|
||||
"build_dir": cfg["build_dir"],
|
||||
}
|
||||
|
||||
|
||||
def tag_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
|
||||
found = []
|
||||
for family, settings in sorted(cfg["families"].items()):
|
||||
if ref and not ref.startswith(settings["prefix"]):
|
||||
continue
|
||||
refs = gh_api(f"repos/{cfg['repo']}/git/matching-refs/tags/{quote(settings['prefix'])}", paginate=True)
|
||||
if not isinstance(refs, list):
|
||||
raise PackError(f"{cfg['repo']}: the tag list is not a list")
|
||||
pattern = re.compile(settings["tag"])
|
||||
names = set()
|
||||
for item in refs:
|
||||
name = item.get("ref", "") if isinstance(item, dict) else ""
|
||||
if isinstance(name, str) and name.startswith("refs/tags/") and pattern.search(name[len("refs/tags/"):]):
|
||||
names.add(name[len("refs/tags/"):])
|
||||
ordered = sorted(names, key=natural_key, reverse=True)
|
||||
chosen = [name for name in ordered if name == ref] if ref else ordered[:cfg["keep"]]
|
||||
for name in chosen:
|
||||
found.append(candidate(flavor, cfg, name, token=cfg["version_token"].format(tag=name), depot_app=settings.get("depot_app", "")))
|
||||
return found
|
||||
|
||||
|
||||
def pick_assets(assets: list, patterns: list[re.Pattern]) -> list[str]:
|
||||
chosen = {}
|
||||
for asset in assets:
|
||||
name = asset.get("name") if isinstance(asset, dict) else None
|
||||
if not isinstance(name, str) or not any(pattern.search(name) for pattern in patterns):
|
||||
continue
|
||||
stem = ARCHIVE.sub("", name)
|
||||
if stem not in chosen or name.endswith(".tar.xz"):
|
||||
chosen[stem] = name
|
||||
return sorted(chosen.values())
|
||||
|
||||
|
||||
def release_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
|
||||
repo = cfg["repo"]
|
||||
if ref:
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(ref)}", missing_ok=True)
|
||||
releases = [release] if release else []
|
||||
else:
|
||||
releases = gh_api(f"repos/{repo}/releases?per_page=100")
|
||||
if not isinstance(releases, list):
|
||||
raise PackError(f"{repo}: the release list is not a list")
|
||||
pattern = re.compile(cfg["tag"])
|
||||
patterns = [re.compile(item) for item in cfg["assets"]]
|
||||
usable = []
|
||||
for release in releases:
|
||||
tag = release.get("tag_name") if isinstance(release, dict) else None
|
||||
if not isinstance(tag, str) or release.get("draft") or not pattern.search(tag):
|
||||
continue
|
||||
assets = pick_assets(release.get("assets") or [], patterns)
|
||||
if assets:
|
||||
usable.append((release.get("published_at") or release.get("created_at") or "", natural_key(tag), tag, assets))
|
||||
usable.sort(reverse=True)
|
||||
if not ref:
|
||||
usable = usable[:cfg["keep"]]
|
||||
return [candidate(flavor, cfg, tag, asset=asset) for _, _, tag, assets in usable for asset in assets]
|
||||
|
||||
|
||||
def built(index: dict) -> dict[tuple[str, str, str], int]:
|
||||
revs = {}
|
||||
packs = index.get("packs") if isinstance(index, dict) else None
|
||||
for pack in packs if isinstance(packs, list) else []:
|
||||
if not isinstance(pack, dict) or not isinstance(pack.get("source"), dict):
|
||||
continue
|
||||
rev = pack.get("rev") if type(pack.get("rev")) is int else 0
|
||||
key = (pack.get("flavor"), pack["source"].get("ref"), pack["source"].get("asset", ""))
|
||||
revs[key] = max(revs.get(key, 0), rev)
|
||||
return revs
|
||||
|
||||
|
||||
def revoked_build(entry: dict, revoked: set[str]) -> bool:
|
||||
prefixes = {f"{entry['flavor']}-{slug(name)}-" for name in (entry["token"], entry["tag"]) if name}
|
||||
suffix = f"-r{entry['rev']}"
|
||||
return any(ident.endswith(suffix) and ident.startswith(prefix) for ident in revoked for prefix in prefixes)
|
||||
|
||||
|
||||
def chosen_flavors(flavors: dict, selected: str) -> list[str]:
|
||||
return sorted(flavors["flavors"]) if selected == "all" else [selected]
|
||||
|
||||
|
||||
def failure_marker(entry: dict, digests: dict[str, str] | None = None) -> str:
|
||||
series = entry["series"]
|
||||
if not SERIES.fullmatch(series):
|
||||
raise PackError(f"{series!r} is not a patch series name")
|
||||
digest = digests.get(series) if digests is not None else None
|
||||
if digest is None:
|
||||
digest = patch_digest(PATCHES / series)
|
||||
if digests is not None:
|
||||
digests[series] = digest
|
||||
return f"{entry['key']}-r{entry['rev']}-{digest[:12]}{FAILED}"
|
||||
|
||||
|
||||
def discover(flavors: dict, selected: str, ref: str, index: dict, revoked: set[str],
|
||||
failed: frozenset[str] = frozenset()) -> list[dict]:
|
||||
names = chosen_flavors(flavors, selected)
|
||||
done = built(index)
|
||||
digests = {}
|
||||
entries = []
|
||||
seen = set()
|
||||
matched = False
|
||||
for flavor in names:
|
||||
cfg = flavors["flavors"].get(flavor)
|
||||
if cfg is None:
|
||||
raise PackError(f"flavors.json has no {flavor}")
|
||||
found = tag_candidates(flavor, cfg, ref) if cfg["list"] == "tags" else release_candidates(flavor, cfg, ref)
|
||||
matched = matched or bool(found)
|
||||
for entry in found:
|
||||
if entry["key"] in seen:
|
||||
continue
|
||||
seen.add(entry["key"])
|
||||
label = f"{flavor} {entry['tag']}{' ' + entry['asset'] if entry['asset'] else ''}"
|
||||
if revoked_build(entry, revoked):
|
||||
print(f"discover: {label}: revoked at rev {entry['rev']}", file=sys.stderr)
|
||||
elif done.get((flavor, entry["tag"], entry["asset"]), 0) >= entry["rev"]:
|
||||
print(f"discover: {label}: in the index at rev {entry['rev']}", file=sys.stderr)
|
||||
elif not ref and failure_marker(entry, digests) in failed:
|
||||
print(f"discover: {label}: failed at rev {entry['rev']} with this patch series in the last "
|
||||
f"{RETRY_AFTER.days} days; skipped (bump rev, change the series or build the tag by name)", file=sys.stderr)
|
||||
else:
|
||||
print(f"discover: {label}: to build", file=sys.stderr)
|
||||
entries.append(entry)
|
||||
if ref and not matched:
|
||||
raise PackError(f"no {'' if selected == 'all' else selected + ' '}build is tagged {ref}")
|
||||
return entries
|
||||
|
||||
|
||||
def vdf(text: str) -> dict:
|
||||
stack = [{}]
|
||||
key = None
|
||||
for match in VDF_TOKEN.finditer(text):
|
||||
word, brace = match.group(1), match.group(2)
|
||||
if brace == "{":
|
||||
if key is None:
|
||||
raise PackError("app info: a block has no name")
|
||||
child = {}
|
||||
stack[-1][key] = child
|
||||
stack.append(child)
|
||||
key = None
|
||||
elif brace == "}":
|
||||
if len(stack) == 1:
|
||||
raise PackError("app info: unbalanced braces")
|
||||
stack.pop()
|
||||
key = None
|
||||
elif key is None:
|
||||
key = word
|
||||
else:
|
||||
stack[-1][key] = word
|
||||
key = None
|
||||
if len(stack) != 1:
|
||||
raise PackError("app info: unbalanced braces")
|
||||
return stack[0]
|
||||
|
||||
|
||||
def steam_builds(text: str, apps: list[str]) -> dict[str, dict]:
|
||||
found = {}
|
||||
text = ANSI.sub("", text)
|
||||
starts = [(m.start(), m.group(1)) for m in APP_HEADER.finditer(text)]
|
||||
for number, (start, app) in enumerate(starts):
|
||||
end = starts[number + 1][0] if number + 1 < len(starts) else len(text)
|
||||
if app not in apps:
|
||||
continue
|
||||
body = text[text.find("\n", start) + 1:end]
|
||||
info = vdf(body).get(app)
|
||||
depots = info.get("depots") if isinstance(info, dict) else None
|
||||
public = depots.get("branches", {}).get("public") if isinstance(depots, dict) else None
|
||||
if not isinstance(public, dict) or not str(public.get("buildid", "")).isdigit():
|
||||
continue
|
||||
manifests = sorted(depot["manifests"]["public"]["gid"] for name, depot in depots.items()
|
||||
if name.isdigit() and isinstance(depot, dict) and isinstance(depot.get("manifests"), dict)
|
||||
and isinstance(depot["manifests"].get("public"), dict) and depot["manifests"]["public"].get("gid"))
|
||||
found[app] = {"buildid": public["buildid"], "built": public.get("timebuildupdated", ""), "manifests": manifests}
|
||||
return found
|
||||
|
||||
|
||||
def steam_apps(flavors: dict) -> list[str]:
|
||||
return sorted({family["depot_app"] for cfg in flavors["flavors"].values() if cfg["list"] == "tags"
|
||||
for family in cfg["families"].values() if family.get("depot_app")})
|
||||
|
||||
|
||||
def watch(flavors: dict, previous: dict, steam: dict[str, dict] | None) -> tuple[dict, list[str], list[str]]:
|
||||
state = {}
|
||||
changed = []
|
||||
notes = []
|
||||
for flavor in sorted(flavors["flavors"]):
|
||||
cfg = flavors["flavors"][flavor]
|
||||
found = tag_candidates(flavor, cfg, "") if cfg["list"] == "tags" else release_candidates(flavor, cfg, "")
|
||||
builds = sorted(f"{entry['tag']} {entry['asset']}".strip() for entry in found)
|
||||
ships = {}
|
||||
if cfg["list"] == "tags" and steam is not None:
|
||||
for family in cfg["families"].values():
|
||||
app = family.get("depot_app")
|
||||
if app and app in steam:
|
||||
ships[app] = steam[app]
|
||||
old = previous.get(flavor) if isinstance(previous.get(flavor), dict) else {}
|
||||
if steam is None and isinstance(old.get("steam"), dict):
|
||||
ships = old["steam"]
|
||||
mark = {"builds": builds, "rev": cfg["rev"], "series": cfg["series"], "patch": patch_digest(PATCHES / cfg["series"])}
|
||||
fingerprint = hashlib.sha256(json.dumps(mark, sort_keys=True).encode()).hexdigest()
|
||||
state[flavor] = {**mark, "fingerprint": fingerprint, "steam": ships}
|
||||
if old.get("fingerprint") != fingerprint:
|
||||
changed.append(flavor)
|
||||
fresh = sorted(set(builds) - set(old.get("builds") or []))
|
||||
notes.append(f"{flavor}: {', '.join(fresh) if fresh else 'the patch series or rev changed'}")
|
||||
for app, ship in sorted(ships.items()):
|
||||
before = (old.get("steam") or {}).get(app) if isinstance(old.get("steam"), dict) else None
|
||||
if isinstance(before, dict) and before.get("buildid") != ship.get("buildid"):
|
||||
notes.append(f"{flavor}: Steam app {app} moved from build {before.get('buildid')} to {ship.get('buildid')}")
|
||||
if flavor not in changed:
|
||||
changed.append(flavor)
|
||||
return state, changed, notes
|
||||
|
||||
|
||||
def asset_text(repo: str, asset_id, what: str) -> str:
|
||||
result = gh("api", "-H", "Accept: application/octet-stream", f"repos/{repo}/releases/assets/{asset_id}")
|
||||
if result.returncode != 0:
|
||||
raise PackError(f"cannot download {what}: {result.stderr.strip()}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def release_index(repo: str, tag: str) -> dict:
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True)
|
||||
if release is None:
|
||||
return {}
|
||||
for asset in release.get("assets") or []:
|
||||
if isinstance(asset, dict) and asset.get("name") == "index.json":
|
||||
text = asset_text(repo, asset["id"], f"the index of {repo}")
|
||||
try:
|
||||
return json.loads(text)
|
||||
except ValueError as e:
|
||||
raise PackError(f"the index of {repo} is not JSON: {e}") from e
|
||||
return {}
|
||||
|
||||
|
||||
def published(repo: str, flavors: dict, selected: str, index: dict) -> list[dict]:
|
||||
packs = index.get("packs") if isinstance(index, dict) else None
|
||||
listed = {pack.get("id") for pack in packs if isinstance(pack, dict)} if isinstance(packs, list) else set()
|
||||
found = []
|
||||
for flavor in chosen_flavors(flavors, selected):
|
||||
cfg = flavors["flavors"].get(flavor)
|
||||
tag = f"droiddeck-esync-{flavor}"
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if cfg else None
|
||||
if not isinstance(release, dict) or type(release.get("id")) is not int:
|
||||
continue
|
||||
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
|
||||
if not isinstance(assets, list):
|
||||
raise PackError(f"{repo} {tag}: the asset list is not a list")
|
||||
for asset in assets:
|
||||
name = asset.get("name") if isinstance(asset, dict) else None
|
||||
if not isinstance(name, str) or not name.endswith(".json"):
|
||||
continue
|
||||
ident = name[:-len(".json")]
|
||||
rev = PACK_REV.search(ident)
|
||||
if ident in listed or not rev or int(rev.group(1)) < cfg["rev"]:
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(asset_text(repo, asset.get("id"), f"{tag} {name}"))
|
||||
except ValueError:
|
||||
entry = None
|
||||
if not isinstance(entry, dict) or entry.get("id") != ident or entry.get("flavor") != flavor:
|
||||
print(f"discover: {tag} {name} is not a pack entry", file=sys.stderr)
|
||||
continue
|
||||
print(f"discover: {tag} {name}: published, not in the index yet", file=sys.stderr)
|
||||
found.append(entry)
|
||||
return found
|
||||
|
||||
|
||||
def parse_time(value) -> datetime | None:
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
try:
|
||||
moment = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def recent_failures(repo: str, flavors: dict, selected: str, now: datetime) -> frozenset[str]:
|
||||
found = set()
|
||||
for flavor in chosen_flavors(flavors, selected):
|
||||
tag = f"droiddeck-esync-{flavor}"
|
||||
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if flavor in flavors["flavors"] else None
|
||||
if not isinstance(release, dict) or type(release.get("id")) is not int:
|
||||
continue
|
||||
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
|
||||
if not isinstance(assets, list):
|
||||
raise PackError(f"{repo} {tag}: the asset list is not a list")
|
||||
for asset in assets:
|
||||
name = asset.get("name") if isinstance(asset, dict) else None
|
||||
if not isinstance(name, str) or not name.endswith(FAILED):
|
||||
continue
|
||||
moment = parse_time(asset.get("updated_at")) or parse_time(asset.get("created_at"))
|
||||
if moment is not None and now - moment < RETRY_AFTER:
|
||||
found.add(name)
|
||||
return frozenset(found)
|
||||
|
||||
|
||||
def failed_builds(matrix: dict, directory: Path) -> list[tuple[str, str]]:
|
||||
rows = matrix.get("include") if isinstance(matrix, dict) else None
|
||||
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
|
||||
raise PackError("the build matrix has no include list")
|
||||
made = {row["key"] for row in rows if (directory / f"{ARTIFACT_PREFIX}{row['key']}").is_dir()}
|
||||
digests = {}
|
||||
return [(row["flavor"], failure_marker(row, digests)) for row in rows if row["key"] not in made]
|
||||
|
||||
|
||||
def artifact_packs(matrix: dict, directory: Path) -> list[tuple[str, Path, str]]:
|
||||
rows = matrix.get("include") if isinstance(matrix, dict) else None
|
||||
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
|
||||
raise PackError("the build matrix has no include list")
|
||||
builds = {row["key"]: row for row in rows}
|
||||
found = []
|
||||
for folder in sorted(directory.iterdir()) if directory.is_dir() else []:
|
||||
row = builds.get(folder.name[len(ARTIFACT_PREFIX):]) if folder.name.startswith(ARTIFACT_PREFIX) else None
|
||||
if row is None or folder.is_symlink() or not folder.is_dir():
|
||||
raise PackError(f"{folder.name} is not the artifact of a build in this run")
|
||||
names = sorted(path.name for path in folder.iterdir())
|
||||
entries = [name for name in names if name.endswith(".json")]
|
||||
if len(entries) != 1 or names != sorted([entries[0], entries[0][:-len(".json")] + ".tzst"]):
|
||||
raise PackError(f"{folder.name} must hold exactly <id>.json and <id>.tzst")
|
||||
ident = entries[0][:-len(".json")]
|
||||
entry_path, archive = folder / entries[0], folder / f"{ident}.tzst"
|
||||
if entry_path.is_symlink() or archive.is_symlink() or not entry_path.is_file() or not archive.is_file():
|
||||
raise PackError(f"{folder.name} holds links or special files")
|
||||
try:
|
||||
data = json.loads(entry_path.read_text())
|
||||
except ValueError as e:
|
||||
raise PackError(f"{entry_path}: {e}") from e
|
||||
entry = check_entry(data, str(entry_path))
|
||||
source = data.get("source") if isinstance(data.get("source"), dict) else {}
|
||||
if entry["id"] != ident or entry["flavor"] != row["flavor"] or entry["rev"] != row["rev"] \
|
||||
or source.get("ref") != row["tag"] or source.get("asset", "") != row["asset"]:
|
||||
raise PackError(f"{entry_path} is not the pack of {row['flavor']} {row['tag']} at rev {row['rev']}")
|
||||
if archive.stat().st_size != entry["asset"]["size"] or sha256_file(archive) != entry["asset"]["sha256"]:
|
||||
raise PackError(f"{archive} is not the asset its entry describes")
|
||||
found.append((row["flavor"], folder, ident))
|
||||
return found
|
||||
|
||||
|
||||
def emit(entries: list[dict], unindexed: int = 0) -> None:
|
||||
matrix = json.dumps({"include": entries}, sort_keys=True, separators=(",", ":"))
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as stream:
|
||||
stream.write(f"matrix={matrix}\ncount={len(entries)}\nunindexed={unindexed}\n")
|
||||
print(matrix)
|
||||
|
||||
|
||||
def make_variables(text: str) -> tuple[dict[str, str], list[tuple[str, str]]]:
|
||||
variables = {}
|
||||
assignments = []
|
||||
for raw in text.splitlines():
|
||||
match = ASSIGNMENT.fullmatch(raw.split("#", 1)[0].strip())
|
||||
if match:
|
||||
name, value = match.group(1), match.group(2).strip()
|
||||
variables.setdefault(name, value)
|
||||
assignments.append((name, value))
|
||||
return variables, assignments
|
||||
|
||||
|
||||
def expand(value: str, variables: dict[str, str], depth: int = 0) -> str:
|
||||
if depth > 16:
|
||||
raise PackError("Makefile variables nest too deeply")
|
||||
|
||||
def lookup(match: re.Match) -> str:
|
||||
if match.group(1) not in variables:
|
||||
raise PackError(f"$({match.group(1)}) is not set in the Makefile")
|
||||
return expand(variables[match.group(1)], variables, depth + 1)
|
||||
|
||||
return REFERENCE.sub(lookup, value)
|
||||
|
||||
|
||||
def sdk_image(text: str, variable: str = "STEAMRT_IMAGE", match: str = "arm64-llvm") -> str:
|
||||
variables, assignments = make_variables(text)
|
||||
problem = None
|
||||
for name, value in assignments:
|
||||
if not name.startswith(variable):
|
||||
continue
|
||||
try:
|
||||
image = expand(value, variables)
|
||||
except PackError as e:
|
||||
problem = problem or e
|
||||
continue
|
||||
if match not in image:
|
||||
continue
|
||||
if not IMAGE.fullmatch(image):
|
||||
raise PackError(f"{name} = {image!r} is not an image reference")
|
||||
return image
|
||||
raise problem or PackError(f"no {variable} assignment names an {match} image")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Find the Proton builds that have no droiddeck-esync pack yet, as a GitHub Actions matrix.")
|
||||
parser.add_argument("--flavors", default=str(FLAVORS_FILE))
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
matrix = commands.add_parser("matrix")
|
||||
matrix.add_argument("--flavor", default="all", choices=("all", *FLAVORS))
|
||||
matrix.add_argument("--ref", default="")
|
||||
matrix.add_argument("--index", default="")
|
||||
matrix.add_argument("--index-release", default="")
|
||||
matrix.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY", "Droid-Deck/DroidDeck"))
|
||||
matrix.add_argument("--revoked", default=str(REVOKED))
|
||||
image = commands.add_parser("sdk-image")
|
||||
image.add_argument("makefile")
|
||||
artifacts = commands.add_parser("packs")
|
||||
artifacts.add_argument("--matrix", required=True)
|
||||
artifacts.add_argument("directory")
|
||||
failures = commands.add_parser("failures")
|
||||
failures.add_argument("--matrix", required=True)
|
||||
failures.add_argument("directory")
|
||||
commands.add_parser("steam-apps")
|
||||
watching = commands.add_parser("watch")
|
||||
watching.add_argument("--state", default="")
|
||||
watching.add_argument("--steam-info", default="")
|
||||
watching.add_argument("--out", required=True)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
flavors = load_flavors(Path(args.flavors))
|
||||
if args.command == "sdk-image":
|
||||
settings = flavors["sdk_image"]
|
||||
print(sdk_image(Path(args.makefile).read_text(), settings["variable"], settings["match"]))
|
||||
return 0
|
||||
if args.command == "packs":
|
||||
for flavor, folder, ident in artifact_packs(json.loads(args.matrix), Path(args.directory)):
|
||||
print(f"{flavor}\t{folder}\t{ident}")
|
||||
return 0
|
||||
if args.command == "failures":
|
||||
for flavor, marker in failed_builds(json.loads(args.matrix), Path(args.directory)):
|
||||
print(f"{flavor}\t{marker}")
|
||||
return 0
|
||||
if args.command == "steam-apps":
|
||||
print(" ".join(steam_apps(flavors)))
|
||||
return 0
|
||||
if args.command == "watch":
|
||||
previous = {}
|
||||
if args.state and Path(args.state).is_file():
|
||||
try:
|
||||
previous = json.loads(Path(args.state).read_text())
|
||||
except ValueError:
|
||||
print("discover: the previous watch state is not JSON; every flavor counts as changed", file=sys.stderr)
|
||||
previous = previous.get("flavors", {}) if isinstance(previous, dict) and previous.get("format") == 1 else {}
|
||||
steam = None
|
||||
if args.steam_info:
|
||||
try:
|
||||
steam = steam_builds(Path(args.steam_info).read_text(errors="replace"), steam_apps(flavors))
|
||||
except (OSError, PackError) as e:
|
||||
print(f"discover: Steam app info unusable ({e}); watching the source tags only", file=sys.stderr)
|
||||
if steam is not None and len(steam) != len(steam_apps(flavors)):
|
||||
print(f"discover: Steam app info covers {sorted(steam)} of {steam_apps(flavors)}", file=sys.stderr)
|
||||
state, changed, notes = watch(flavors, previous, steam)
|
||||
Path(args.out).write_text(json.dumps({"format": 1, "flavors": state}, indent=1, sort_keys=True) + "\n")
|
||||
for note in notes:
|
||||
print(f"discover: {note}", file=sys.stderr)
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as stream:
|
||||
stream.write(f"changed={' '.join(changed)}\n")
|
||||
print(" ".join(changed))
|
||||
return 0
|
||||
if args.index and args.index_release:
|
||||
raise PackError("pass --index or --index-release, not both")
|
||||
failed = frozenset()
|
||||
unindexed = 0
|
||||
if args.index:
|
||||
path = Path(args.index)
|
||||
index = json.loads(path.read_text()) if path.exists() else {}
|
||||
elif args.index_release:
|
||||
if not REPO.fullmatch(args.repo):
|
||||
raise PackError(f"{args.repo!r} is not owner/name")
|
||||
index = release_index(args.repo, args.index_release)
|
||||
listed = index.get("packs") if isinstance(index, dict) and isinstance(index.get("packs"), list) else []
|
||||
waiting = published(args.repo, flavors, args.flavor, index)
|
||||
unindexed = len(waiting)
|
||||
index = {"packs": [*listed, *waiting]}
|
||||
failed = recent_failures(args.repo, flavors, args.flavor, datetime.now(timezone.utc))
|
||||
else:
|
||||
index = {}
|
||||
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
|
||||
emit(discover(flavors, args.flavor, args.ref.strip(), index, revoked, failed), unindexed)
|
||||
except (PackError, OSError, ValueError) as e:
|
||||
print(f"discover: {e}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,84 +0,0 @@
|
||||
{
|
||||
"format": 1,
|
||||
"sdk_image": {
|
||||
"file": "Makefile.in",
|
||||
"variable": "STEAMRT_IMAGE",
|
||||
"match": "arm64-llvm"
|
||||
},
|
||||
"flavors": {
|
||||
"valve": {
|
||||
"repo": "ValveSoftware/Proton",
|
||||
"list": "tags",
|
||||
"families": {
|
||||
"experimental": {
|
||||
"prefix": "experimental-11.0-",
|
||||
"tag": "^experimental-11\\.0-\\d{8}[a-z]?$",
|
||||
"depot_app": "4427310"
|
||||
},
|
||||
"proton": {
|
||||
"prefix": "proton-11.0-",
|
||||
"tag": "^proton-11\\.0-\\d+[a-z]?$",
|
||||
"depot_app": "4628740"
|
||||
}
|
||||
},
|
||||
"version_token": "{tag}-arm64",
|
||||
"keep": 2,
|
||||
"rev": 1,
|
||||
"series": "valve-11",
|
||||
"profile": "valve",
|
||||
"source_match": true,
|
||||
"build_dir": "/builds/proton/proton/build-dir",
|
||||
"wine": {
|
||||
"submodules": [
|
||||
"wine"
|
||||
],
|
||||
"clone": "shallow",
|
||||
"prep": ""
|
||||
}
|
||||
},
|
||||
"ge": {
|
||||
"repo": "GloriousEggroll/proton-ge-custom",
|
||||
"list": "releases",
|
||||
"tag": "^GE-Proton11-\\d+$",
|
||||
"assets": [
|
||||
"aarch64\\.tar\\.(gz|xz)$",
|
||||
"arm64\\.tar\\.(gz|xz)$"
|
||||
],
|
||||
"keep": 3,
|
||||
"rev": 1,
|
||||
"series": "valve-11",
|
||||
"profile": "valve",
|
||||
"source_match": false,
|
||||
"build_dir": "/home/tcrider/build",
|
||||
"wine": {
|
||||
"submodules": [
|
||||
"wine",
|
||||
"wine-staging"
|
||||
],
|
||||
"clone": "treeless",
|
||||
"prep": "patches/protonprep-valve-staging.sh"
|
||||
}
|
||||
},
|
||||
"cachyos": {
|
||||
"repo": "CachyOS/proton-cachyos",
|
||||
"list": "releases",
|
||||
"tag": "^cachyos-11\\.0-",
|
||||
"assets": [
|
||||
"arm64\\.tar\\.(gz|xz)$"
|
||||
],
|
||||
"keep": 3,
|
||||
"rev": 1,
|
||||
"series": "cachyos-11",
|
||||
"profile": "cachyos",
|
||||
"source_match": false,
|
||||
"build_dir": "/home/runner/work/proton-cachyos/proton-cachyos/build",
|
||||
"wine": {
|
||||
"submodules": [
|
||||
"wine"
|
||||
],
|
||||
"clone": "shallow",
|
||||
"prep": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
-----BEGIN PUBLIC KEY-----
|
||||
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFyqZQAMh3g3nsEGLU+kBGVPBEkXr
|
||||
Dv5F3kPtuQ+FV1DkTCGmYUQ1LS9wJtHGt0Ht4wToHGGEp9yzDibYZ4/zhg==
|
||||
-----END PUBLIC KEY-----
|
||||
@@ -1,285 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from make_pack import (
|
||||
BINARIES,
|
||||
EPOCH,
|
||||
FLAVORS,
|
||||
HERE,
|
||||
NTDLL,
|
||||
PACK_ID,
|
||||
REVOKED,
|
||||
SHA256,
|
||||
WINESERVER,
|
||||
PackError,
|
||||
compress,
|
||||
dump_json,
|
||||
read_revoked,
|
||||
sha256_bytes,
|
||||
sha256_file,
|
||||
tar_bytes,
|
||||
write_atomic,
|
||||
)
|
||||
|
||||
PUBKEY = HERE / "index-key.pub"
|
||||
SCHEMA = 1
|
||||
INDEX_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
|
||||
|
||||
|
||||
def check_entry(entry, origin: str) -> dict:
|
||||
def fail(why: str):
|
||||
raise PackError(f"{origin}: {why}")
|
||||
|
||||
if not isinstance(entry, dict):
|
||||
fail("is not an object")
|
||||
ident = entry.get("id")
|
||||
if not isinstance(ident, str) or not PACK_ID.fullmatch(ident):
|
||||
fail(f"{ident!r} is not a pack id")
|
||||
if entry.get("flavor") not in FLAVORS:
|
||||
fail(f"unknown flavor {entry.get('flavor')!r}")
|
||||
if type(entry.get("rev")) is not int or entry["rev"] < 1:
|
||||
fail("rev is not a positive number")
|
||||
version, line = entry.get("version"), entry.get("version_line")
|
||||
fields = line.split() if isinstance(line, str) else []
|
||||
if not isinstance(version, str) or len(fields) < 2 or fields[1] != version or not EPOCH.fullmatch(fields[0]):
|
||||
fail("version and version_line disagree")
|
||||
for field in ("stock", "files"):
|
||||
value = entry.get(field)
|
||||
if not isinstance(value, dict) or set(value) != set(BINARIES) or not all(isinstance(v, str) and SHA256.fullmatch(v) for v in value.values()):
|
||||
fail(f"{field} must give the sha256 of {' and '.join(BINARIES)}")
|
||||
if not isinstance(entry.get("exports"), str) or not SHA256.fullmatch(entry["exports"]):
|
||||
fail("exports is not a sha256")
|
||||
if type(entry.get("source_match")) is not bool:
|
||||
fail("source_match is not a boolean")
|
||||
asset = entry.get("asset")
|
||||
if not isinstance(asset, dict) or not isinstance(asset.get("sha256"), str) or not SHA256.fullmatch(asset["sha256"]) \
|
||||
or type(asset.get("size")) is not int or asset["size"] <= 0:
|
||||
fail("asset needs its sha256 and size")
|
||||
clean = {key: entry[key] for key in INDEX_KEYS if key in entry}
|
||||
clean["asset"] = {"sha256": asset["sha256"], "size": asset["size"]}
|
||||
return clean
|
||||
|
||||
|
||||
def load_entries(directory: Path) -> list[dict]:
|
||||
if not directory.is_dir():
|
||||
raise PackError(f"{directory} is not a directory")
|
||||
entries = []
|
||||
for path in sorted(directory.glob("*.json")):
|
||||
try:
|
||||
data = json.loads(path.read_text())
|
||||
except ValueError as e:
|
||||
raise PackError(f"{path}: {e}") from e
|
||||
entry = check_entry(data, str(path))
|
||||
if path.stem != entry["id"]:
|
||||
raise PackError(f"{path} holds the entry of {entry['id']}")
|
||||
entries.append(entry)
|
||||
return entries
|
||||
|
||||
|
||||
def load_previous(path: Path) -> tuple[int, list[dict], list]:
|
||||
try:
|
||||
data = json.loads(path.read_text())
|
||||
except (OSError, ValueError) as e:
|
||||
raise PackError(f"cannot read the previous index {path}: {e}") from e
|
||||
if not isinstance(data, dict) or data.get("schema") != SCHEMA or type(data.get("generated")) is not int \
|
||||
or not isinstance(data.get("packs"), list):
|
||||
raise PackError(f"{path} is not a schema {SCHEMA} index")
|
||||
packs = []
|
||||
for number, pack in enumerate(data["packs"]):
|
||||
entry = check_entry(pack, f"{path} pack {number}")
|
||||
entry["revoked"] = pack.get("revoked") is True
|
||||
packs.append(entry)
|
||||
return data["generated"], packs, data["packs"]
|
||||
|
||||
|
||||
def epoch(entry: dict) -> int:
|
||||
return int(entry["version_line"].split()[0])
|
||||
|
||||
|
||||
def rank(entry: dict) -> tuple:
|
||||
return entry["rev"], epoch(entry), entry["id"]
|
||||
|
||||
|
||||
def choose(entries: list[dict], previous: list[dict], revoked: set[str]) -> list[dict]:
|
||||
known = {entry["id"]: entry for entry in entries}
|
||||
for entry in previous:
|
||||
known.setdefault(entry["id"], entry)
|
||||
active = {}
|
||||
withdrawn = []
|
||||
for entry in known.values():
|
||||
if entry["id"] in revoked:
|
||||
withdrawn.append(dict(entry, revoked=True))
|
||||
continue
|
||||
key = (entry["stock"][NTDLL], entry["stock"][WINESERVER], entry["version"])
|
||||
if key not in active or rank(entry) > rank(active[key]):
|
||||
active[key] = dict(entry, revoked=False)
|
||||
return sorted([*active.values(), *withdrawn], key=lambda entry: entry["id"])
|
||||
|
||||
|
||||
def build_index(packs: list[dict], base_url: str, generated: int) -> dict:
|
||||
listed = []
|
||||
for pack in packs:
|
||||
entry = dict(pack)
|
||||
entry["asset"] = {
|
||||
"url": f"{base_url}/droiddeck-esync-{pack['flavor']}/{pack['id']}.tzst",
|
||||
"sha256": pack["asset"]["sha256"],
|
||||
"size": pack["asset"]["size"],
|
||||
}
|
||||
listed.append(entry)
|
||||
return {"schema": SCHEMA, "generated": generated, "packs": listed}
|
||||
|
||||
|
||||
def family(entry: dict) -> tuple[str, str, str]:
|
||||
token = entry["version"]
|
||||
major = re.search(r"[0-9]+", token)
|
||||
return entry["flavor"], re.match(r"[A-Za-z]*", token).group(0).lower(), major.group(0) if major else ""
|
||||
|
||||
|
||||
def bundle_choice(packs: list[dict], available: set[str] | None = None) -> list[dict]:
|
||||
newest = {}
|
||||
for pack in packs:
|
||||
if pack["revoked"] or (available is not None and pack["id"] not in available):
|
||||
continue
|
||||
key = family(pack)
|
||||
if key not in newest or (epoch(pack), pack["rev"], pack["id"]) > (epoch(newest[key]), newest[key]["rev"], newest[key]["id"]):
|
||||
newest[key] = pack
|
||||
return sorted(newest.values(), key=lambda pack: pack["id"])
|
||||
|
||||
|
||||
def openssl(*args: str) -> subprocess.CompletedProcess:
|
||||
try:
|
||||
return subprocess.run(["openssl", *args], capture_output=True, text=True, check=False)
|
||||
except FileNotFoundError as e:
|
||||
raise PackError("the openssl command is not installed") from e
|
||||
|
||||
|
||||
def verified(index: Path, signature: Path, pubkey: Path) -> bool:
|
||||
result = openssl("dgst", "-sha256", "-verify", str(pubkey), "-signature", str(signature), str(index))
|
||||
return result.returncode == 0 and "Verified OK" in result.stdout
|
||||
|
||||
|
||||
def trusted_previous(path: Path, signature: Path | None, pubkey: Path) -> tuple[int, list[dict], list | None]:
|
||||
if signature is not None and not (path.is_file() and signature.is_file() and verified(path, signature, pubkey)):
|
||||
print(f"make_index: the previous index {path} does not verify with {signature} and {pubkey}; it is ignored", file=sys.stderr)
|
||||
return 0, [], None
|
||||
return load_previous(path)
|
||||
|
||||
|
||||
def sign(index: Path, signature: Path, key: Path | None, pubkey: Path) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
if key is None:
|
||||
pem = os.environ.get("SYNC_INDEX_KEY", "")
|
||||
if not pem.strip():
|
||||
raise PackError("no signing key: pass --key or set SYNC_INDEX_KEY")
|
||||
key = Path(tmp) / "key.pem"
|
||||
with os.fdopen(os.open(key, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as stream:
|
||||
stream.write(pem if pem.endswith("\n") else pem + "\n")
|
||||
result = openssl("dgst", "-sha256", "-sign", str(key), "-out", str(signature), str(index))
|
||||
if result.returncode != 0:
|
||||
raise PackError(f"openssl could not sign the index: {result.stderr.strip()}")
|
||||
if not verified(index, signature, pubkey):
|
||||
raise PackError(f"the signature does not verify with {pubkey}: the signing key is not the one the app trusts")
|
||||
|
||||
|
||||
def bundle_name(generated: int) -> str:
|
||||
return f"bundle-{dt.datetime.fromtimestamp(generated, dt.timezone.utc):%Y%m%d-%H%M%S}.tzst"
|
||||
|
||||
|
||||
def bundle(out: Path, packs: list[dict], index: bytes, signature: bytes, packs_dir: Path, generated: int) -> tuple[str, str]:
|
||||
if not packs_dir.is_dir():
|
||||
raise PackError(f"{packs_dir} is not a directory")
|
||||
available = {path.name[:-len(".tzst")] for path in packs_dir.glob("*.tzst") if path.is_file()}
|
||||
members = {"index.json": (index, 0o644), "index.json.sig": (signature, 0o644)}
|
||||
for pack in bundle_choice(packs, available):
|
||||
path = packs_dir / f"{pack['id']}.tzst"
|
||||
if path.stat().st_size != pack["asset"]["size"] or sha256_file(path) != pack["asset"]["sha256"]:
|
||||
raise PackError(f"{path} is not the asset the index lists for {pack['id']}")
|
||||
members[f"packs/{pack['id']}.tzst"] = (path.read_bytes(), 0o644)
|
||||
name = bundle_name(generated)
|
||||
archive = compress(tar_bytes(members))
|
||||
write_atomic(out / name, archive)
|
||||
return name, sha256_bytes(archive)
|
||||
|
||||
|
||||
def report(values: dict[str, str]) -> None:
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as stream:
|
||||
for key, value in values.items():
|
||||
stream.write(f"{key}={value}\n")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Make, sign and bundle the droiddeck-esync pack index from the published pack entries.")
|
||||
parser.add_argument("--entries", required=True)
|
||||
parser.add_argument("--base-url", required=True)
|
||||
parser.add_argument("--key", default="")
|
||||
parser.add_argument("--pubkey", default=str(PUBKEY))
|
||||
parser.add_argument("--out", default="")
|
||||
parser.add_argument("--bundle-packs-dir", default="")
|
||||
parser.add_argument("--revoked", default=str(REVOKED))
|
||||
parser.add_argument("--previous", default="")
|
||||
parser.add_argument("--previous-sig", default="")
|
||||
parser.add_argument("--list-bundle", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
base_url = args.base_url.rstrip("/")
|
||||
if not base_url.startswith("https://"):
|
||||
raise PackError("--base-url must be an https URL")
|
||||
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
|
||||
if args.previous_sig and not args.previous:
|
||||
raise PackError("--previous-sig needs --previous")
|
||||
previous_signature = Path(args.previous_sig) if args.previous_sig else None
|
||||
previous_generated, previous, previous_raw = trusted_previous(Path(args.previous), previous_signature, Path(args.pubkey)) \
|
||||
if args.previous else (0, [], None)
|
||||
packs = choose(load_entries(Path(args.entries)), previous, revoked)
|
||||
if args.list_bundle:
|
||||
for pack in bundle_choice(packs):
|
||||
print(f"{pack['flavor']}\t{pack['id']}")
|
||||
return 0
|
||||
if not args.out:
|
||||
raise PackError("--out is required")
|
||||
out = Path(args.out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
generated = max(int(time.time()), previous_generated + 1)
|
||||
index = build_index(packs, base_url, generated)
|
||||
data = dump_json(index)
|
||||
changed = bool(packs) if previous_raw is None else previous_raw != index["packs"]
|
||||
temp_index = out / f".index.json.tmp-{os.getpid()}"
|
||||
temp_signature = out / f".index.json.sig.tmp-{os.getpid()}"
|
||||
try:
|
||||
temp_index.write_bytes(data)
|
||||
sign(temp_index, temp_signature, Path(args.key) if args.key else None, Path(args.pubkey))
|
||||
signature = temp_signature.read_bytes()
|
||||
os.replace(temp_index, out / "index.json")
|
||||
os.replace(temp_signature, out / "index.json.sig")
|
||||
finally:
|
||||
temp_index.unlink(missing_ok=True)
|
||||
temp_signature.unlink(missing_ok=True)
|
||||
outputs = {"changed": "true" if changed else "false", "generated": str(generated), "index_sha256": sha256_bytes(data)}
|
||||
live = sum(1 for pack in packs if not pack["revoked"])
|
||||
print(f"index.json {outputs['index_sha256']} {live} pack(s), {len(packs) - live} revoked, generated {generated}, {'changed' if changed else 'unchanged'}")
|
||||
if args.bundle_packs_dir:
|
||||
name, digest = bundle(out, packs, data, signature, Path(args.bundle_packs_dir), generated)
|
||||
outputs.update(bundle=name, bundle_sha256=digest)
|
||||
print(f"{name} {digest}")
|
||||
report(outputs)
|
||||
except (PackError, OSError) as e:
|
||||
print(f"make_index: {e}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,367 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import functools
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
ROOT = HERE.parents[1]
|
||||
GUEST = ROOT / "tools/linuxfs/overlay/usr/local/bin/droiddeck-esync"
|
||||
PATCHES = HERE / "patches"
|
||||
REVOKED = HERE / "revoked.txt"
|
||||
NTDLL = "files/lib/wine/aarch64-unix/ntdll.so"
|
||||
WINESERVER = "files/bin-arm64/wineserver"
|
||||
BINARIES = (NTDLL, WINESERVER)
|
||||
COPY = ("files/bin-arm64/wine", "files/lib/wine/aarch64-unix/wine", "files/lib/wine/aarch64-unix/wine-preloader")
|
||||
HEADERS = ("server_protocol.h", "request_handlers.h", "request_trace.h")
|
||||
FLAVORS = ("valve", "ge", "cachyos")
|
||||
SOURCE_KEYS = ("repo", "ref", "commit", "wine_commit", "sdk_image")
|
||||
ENTRY_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
|
||||
PACK_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}")
|
||||
SERIES = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
EPOCH = re.compile(r"[0-9]{1,12}")
|
||||
PROTOCOL = re.compile(rb"^#define SERVER_PROTOCOL_VERSION ([0-9]+)[ \t]*$", re.M)
|
||||
BUILTIN_ESYNC = b"esync: up and running"
|
||||
ELF_MAGIC = b"\x7fELF"
|
||||
GUEST_MODULE = "droiddeck_esync_tools"
|
||||
|
||||
|
||||
class PackError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=None)
|
||||
def guest():
|
||||
loader = importlib.machinery.SourceFileLoader(GUEST_MODULE, str(GUEST))
|
||||
module = importlib.util.module_from_spec(importlib.util.spec_from_loader(GUEST_MODULE, loader))
|
||||
sys.modules[GUEST_MODULE] = module
|
||||
write_bytecode = sys.dont_write_bytecode
|
||||
sys.dont_write_bytecode = True
|
||||
try:
|
||||
loader.exec_module(module)
|
||||
except Exception as e:
|
||||
sys.modules.pop(GUEST_MODULE, None)
|
||||
raise PackError(f"cannot load {GUEST}: {e}") from e
|
||||
finally:
|
||||
sys.dont_write_bytecode = write_bytecode
|
||||
return module
|
||||
|
||||
|
||||
def sha256_bytes(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
return re.sub(r"[^A-Za-z0-9._-]", "-", text)
|
||||
|
||||
|
||||
def dump_json(value) -> bytes:
|
||||
return (json.dumps(value, indent=1, sort_keys=True, ensure_ascii=True) + "\n").encode()
|
||||
|
||||
|
||||
def write_atomic(path: Path, data: bytes) -> None:
|
||||
temp = path.with_name(f".{path.name}.tmp-{os.getpid()}")
|
||||
try:
|
||||
with temp.open("wb") as stream:
|
||||
stream.write(data)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temp, path)
|
||||
except BaseException:
|
||||
temp.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
|
||||
def read_revoked(path: Path | None) -> set[str]:
|
||||
if path is None:
|
||||
return set()
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as e:
|
||||
raise PackError(f"cannot read the revocation list {path}: {e}") from e
|
||||
revoked = set()
|
||||
for raw in text.splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
ident = line.split()[0]
|
||||
if not PACK_ID.fullmatch(ident):
|
||||
raise PackError(f"{path}: {ident!r} is not a pack id")
|
||||
revoked.add(ident)
|
||||
return revoked
|
||||
|
||||
|
||||
def tar_bytes(members: dict[str, tuple[bytes, int]]) -> bytes:
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w", format=tarfile.USTAR_FORMAT) as tar:
|
||||
for name in sorted(members):
|
||||
data, mode = members[name]
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(data)
|
||||
info.mode = mode
|
||||
info.mtime = 0
|
||||
info.uid = 0
|
||||
info.gid = 0
|
||||
info.uname = ""
|
||||
info.gname = ""
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def zstd(data: bytes, *args: str) -> bytes:
|
||||
try:
|
||||
result = subprocess.run(["zstd", "-q", *args], input=data, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
|
||||
except FileNotFoundError as e:
|
||||
raise PackError("the zstd command is not installed") from e
|
||||
if result.returncode != 0:
|
||||
raise PackError(f"zstd {' '.join(args)} failed: {result.stderr.decode(errors='replace').strip()}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def compress(data: bytes) -> bytes:
|
||||
return zstd(data, "-19", "-T1", "-c")
|
||||
|
||||
|
||||
def unpack(archive: bytes) -> dict[str, tuple[bytes, int]]:
|
||||
members = {}
|
||||
try:
|
||||
with tarfile.open(fileobj=io.BytesIO(zstd(archive, "-d", "-c")), mode="r:") as tar:
|
||||
for info in tar:
|
||||
if not info.isreg() or info.name in members:
|
||||
raise PackError(f"unexpected entry {info.name!r} in the archive")
|
||||
members[info.name] = (tar.extractfile(info).read(), info.mode)
|
||||
except tarfile.TarError as e:
|
||||
raise PackError(f"the archive does not read back: {e}") from e
|
||||
return members
|
||||
|
||||
|
||||
def binary(base: Path, rel: str) -> Path:
|
||||
path = base / rel
|
||||
if path.is_symlink() or not path.is_file():
|
||||
raise PackError(f"{path} is not a regular file")
|
||||
with path.open("rb") as stream:
|
||||
if stream.read(len(ELF_MAGIC)) != ELF_MAGIC:
|
||||
raise PackError(f"{path} is not an ELF file")
|
||||
return path
|
||||
|
||||
|
||||
def exports_hash(path: Path) -> str:
|
||||
try:
|
||||
value = guest().exports_hash(path)
|
||||
except PackError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise PackError(f"{path}: cannot read its exported symbols: {e}") from e
|
||||
if not isinstance(value, str) or not SHA256.fullmatch(value):
|
||||
raise PackError(f"{path}: has no exported symbols")
|
||||
return value
|
||||
|
||||
|
||||
def parse_version_line(line: str) -> tuple[int, str]:
|
||||
fields = line.split()
|
||||
if "\n" in line or len(fields) < 2 or not EPOCH.fullmatch(fields[0]):
|
||||
raise PackError(f"version line {line!r} is not '<build epoch> <version>'")
|
||||
token = guest().version_token(line)
|
||||
if not isinstance(token, str) or not token:
|
||||
raise PackError(f"version line {line!r} has no version token")
|
||||
return int(fields[0]), token
|
||||
|
||||
|
||||
def parse_source(text: str) -> dict[str, str]:
|
||||
try:
|
||||
source = json.loads(text)
|
||||
except ValueError as e:
|
||||
raise PackError(f"--source-json is not JSON: {e}") from e
|
||||
if not isinstance(source, dict) or not all(isinstance(key, str) and isinstance(value, str) for key, value in source.items()):
|
||||
raise PackError("--source-json must be an object of strings")
|
||||
missing = [key for key in SOURCE_KEYS if not source.get(key)]
|
||||
if missing:
|
||||
raise PackError(f"--source-json lacks {', '.join(missing)}")
|
||||
return source
|
||||
|
||||
|
||||
def patch_digest(directory: Path) -> str:
|
||||
patches = sorted(path for path in directory.glob("*.patch") if path.is_file())
|
||||
if not patches:
|
||||
raise PackError(f"{directory} holds no *.patch files")
|
||||
digest = hashlib.sha256()
|
||||
for path in patches:
|
||||
digest.update(f"{path.name}\0{sha256_file(path)}\n".encode())
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def protocol_version(control: Path, patched: Path) -> int:
|
||||
for name in HEADERS:
|
||||
before, after = control / name, patched / name
|
||||
if not before.is_file() or not after.is_file():
|
||||
raise PackError(f"{name} is missing from the control or the patched build")
|
||||
if before.read_bytes() != after.read_bytes():
|
||||
raise PackError(f"the patch changes {name}: the droiddeck-esync patch must not change the wineserver protocol")
|
||||
found = PROTOCOL.findall((control / HEADERS[0]).read_bytes())
|
||||
if len(found) != 1:
|
||||
raise PackError("server_protocol.h does not define SERVER_PROTOCOL_VERSION once")
|
||||
version = int(found[0])
|
||||
stamp = control / "protocol"
|
||||
if stamp.is_file() and stamp.read_text().strip() != str(version):
|
||||
raise PackError(f"{stamp} disagrees with server_protocol.h ({version})")
|
||||
return version
|
||||
|
||||
|
||||
def make_pack(
|
||||
flavor: str,
|
||||
rev: int,
|
||||
stock_dir: Path | None,
|
||||
version_line: str,
|
||||
control: Path,
|
||||
patched: Path,
|
||||
source: dict[str, str],
|
||||
series: str,
|
||||
out: Path,
|
||||
source_match: bool,
|
||||
patch_dir: Path | None = None,
|
||||
allow_unreproduced: bool = False,
|
||||
) -> dict:
|
||||
if flavor not in FLAVORS:
|
||||
raise PackError(f"unknown flavor {flavor!r}")
|
||||
if type(rev) is not int or rev < 1:
|
||||
raise PackError("--rev must be a positive number")
|
||||
if not SERIES.fullmatch(series):
|
||||
raise PackError(f"{series!r} is not a patch series name")
|
||||
if stock_dir is None and not source_match:
|
||||
raise PackError("without --stock-dir a pack can only match by source: pass --source-match")
|
||||
if stock_dir is not None and source_match:
|
||||
raise PackError("a pack made against stock binaries matches them exactly: drop --source-match")
|
||||
patch_dir = patch_dir or PATCHES / series
|
||||
if not patch_dir.is_dir():
|
||||
raise PackError(f"the patch series {patch_dir} does not exist")
|
||||
version_line = version_line.strip()
|
||||
if stock_dir is not None:
|
||||
stock_line = guest().version_line(stock_dir)
|
||||
if stock_line and version_line and stock_line != version_line:
|
||||
raise PackError(f"{stock_dir}/version says {stock_line!r}, not {version_line!r}")
|
||||
version_line = version_line or stock_line
|
||||
epoch, token = parse_version_line(version_line)
|
||||
|
||||
stock_paths = {rel: binary(stock_dir or control, rel) for rel in BINARIES}
|
||||
control_paths = {rel: binary(control, rel) for rel in BINARIES}
|
||||
patched_paths = {rel: binary(patched, rel) for rel in BINARIES}
|
||||
if BUILTIN_ESYNC in stock_paths[WINESERVER].read_bytes():
|
||||
raise PackError(f"{stock_paths[WINESERVER]} has esync built in and needs no pack")
|
||||
stock = {rel: sha256_file(path) for rel, path in stock_paths.items()}
|
||||
built = {rel: sha256_file(path) for rel, path in control_paths.items()}
|
||||
files = {rel: sha256_file(path) for rel, path in patched_paths.items()}
|
||||
for rel in BINARIES:
|
||||
if files[rel] == built[rel]:
|
||||
raise PackError(f"the patch left {rel} unchanged")
|
||||
|
||||
exports = exports_hash(stock_paths[NTDLL])
|
||||
control_exports = exports_hash(control_paths[NTDLL])
|
||||
if exports_hash(patched_paths[NTDLL]) != control_exports:
|
||||
raise PackError("the patch changes the symbols ntdll.so exports")
|
||||
if exports != control_exports:
|
||||
raise PackError("the control build of ntdll.so exports other symbols than the stock one: the wine source does not match this release")
|
||||
protocol = protocol_version(control, patched)
|
||||
if stock_dir is not None:
|
||||
for rel in BINARIES:
|
||||
state = "reproduces" if stock[rel] == built[rel] else "differs from"
|
||||
print(f"make_pack: the control build {state} the stock {rel}", file=sys.stderr)
|
||||
unreproduced = [rel for rel in BINARIES if stock[rel] != built[rel]]
|
||||
if unreproduced and not allow_unreproduced:
|
||||
raise PackError(f"the control build does not reproduce the stock {', '.join(unreproduced)}: the patched files would differ "
|
||||
"from the release in more than the patch (check the build directory and the SDK image, or pass --allow-unreproduced)")
|
||||
|
||||
pack_id = f"{flavor}-{slug(token)}-{epoch}-{stock[NTDLL][:12]}-r{rev}"
|
||||
if not PACK_ID.fullmatch(pack_id):
|
||||
raise PackError(f"{pack_id!r} is not a valid pack id")
|
||||
pack = {
|
||||
"format": 1,
|
||||
"id": pack_id,
|
||||
"flavor": flavor,
|
||||
"rev": rev,
|
||||
"version": token,
|
||||
"version_line": version_line,
|
||||
"stock": stock,
|
||||
"exports": exports,
|
||||
"source_match": source_match,
|
||||
"files": files,
|
||||
"copy": list(COPY),
|
||||
"protocol": protocol,
|
||||
"source": dict(source),
|
||||
"patch": {"series": series, "sha256": patch_digest(patch_dir)},
|
||||
}
|
||||
members = {
|
||||
"pack.json": (dump_json(pack), 0o644),
|
||||
NTDLL: (patched_paths[NTDLL].read_bytes(), 0o755),
|
||||
WINESERVER: (patched_paths[WINESERVER].read_bytes(), 0o755),
|
||||
}
|
||||
archive = compress(tar_bytes(members))
|
||||
if unpack(archive) != members:
|
||||
raise PackError("the archive does not read back as written")
|
||||
entry = {key: pack[key] for key in ENTRY_KEYS}
|
||||
entry["asset"] = {"sha256": sha256_bytes(archive), "size": len(archive)}
|
||||
entry["revoked"] = False
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
write_atomic(out / f"{pack_id}.tzst", archive)
|
||||
write_atomic(out / f"{pack_id}.json", dump_json(entry))
|
||||
return entry
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Make a droiddeck-esync pack from a control and a patched wine build of one Proton release.")
|
||||
parser.add_argument("--flavor", required=True, choices=FLAVORS)
|
||||
parser.add_argument("--rev", required=True, type=int)
|
||||
parser.add_argument("--stock-dir", default="")
|
||||
parser.add_argument("--version-line", default="")
|
||||
parser.add_argument("--control", required=True)
|
||||
parser.add_argument("--patched", required=True)
|
||||
parser.add_argument("--source-json", required=True)
|
||||
parser.add_argument("--patch-series", required=True)
|
||||
parser.add_argument("--patch-dir", default="")
|
||||
parser.add_argument("--out", required=True)
|
||||
parser.add_argument("--source-match", action="store_true")
|
||||
parser.add_argument("--allow-unreproduced", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
entry = make_pack(
|
||||
flavor=args.flavor,
|
||||
rev=args.rev,
|
||||
stock_dir=Path(args.stock_dir) if args.stock_dir else None,
|
||||
version_line=args.version_line,
|
||||
control=Path(args.control),
|
||||
patched=Path(args.patched),
|
||||
source=parse_source(args.source_json),
|
||||
series=args.patch_series,
|
||||
out=Path(args.out),
|
||||
source_match=args.source_match,
|
||||
patch_dir=Path(args.patch_dir) if args.patch_dir else None,
|
||||
allow_unreproduced=args.allow_unreproduced,
|
||||
)
|
||||
except (PackError, OSError) as e:
|
||||
print(f"make_pack: {e}", file=sys.stderr)
|
||||
return 1
|
||||
print(entry["id"])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,4 @@
|
||||
SYNC_BUNDLE_REPO=Droid-Deck/DroidDeck-Components
|
||||
SYNC_BUNDLE_TAG=droiddeck-esync-index
|
||||
SYNC_BUNDLE_ASSET=bundle-20261004-142603.tzst
|
||||
SYNC_BUNDLE_SHA256=7e311e92444f0709de7779ef5334708511345d60556aa429e6761b86d711a9b2
|
||||
Whitespace-only changes.
@@ -1,62 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
if [ $# -ne 4 ]; then
|
||||
echo "usage: $0 <proton-dir> <pack.tzst> <esstress.exe> <work-dir>" >&2
|
||||
exit 2
|
||||
fi
|
||||
tool=$(realpath "$1")
|
||||
archive=$(realpath "$2")
|
||||
exe=$(realpath "$3")
|
||||
work=$(realpath -m "$4")
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
guest=$here/../linuxfs/overlay/usr/local/bin/droiddeck-esync
|
||||
rm -rf "$work"
|
||||
mkdir -p "$work/pack"
|
||||
tar --zstd -xf "$archive" -C "$work/pack"
|
||||
python3 "$guest" build-shadow "$tool" "$work/pack" "$work/dist" >/dev/null
|
||||
[ -e "$work/dist/files/bin" ] || ln -s bin-arm64 "$work/dist/files/bin"
|
||||
export WINEDEBUG=-all WINEDLLOVERRIDES="mscoree,mshtml="
|
||||
|
||||
run() {
|
||||
local mode=$1 bin=$2 want=$3
|
||||
shift 3
|
||||
local prefix=$work/prefix-$mode
|
||||
mkdir -p "$prefix"
|
||||
(
|
||||
export WINEPREFIX=$prefix WINESERVER=$bin/wineserver "$@"
|
||||
timeout "${BOOT_TIMEOUT:-900}" "$bin/wine" wineboot -i > "$work/$mode-boot.log" 2>&1 || true
|
||||
"$bin/wineserver" -w || true
|
||||
if ! grep -q "$want" "$work/$mode-boot.log"; then
|
||||
echo "$mode: wine never said \"$want\""
|
||||
tail -n 30 "$work/$mode-boot.log"
|
||||
exit 1
|
||||
fi
|
||||
cp "$exe" "$prefix/drive_c/esstress.exe"
|
||||
status=0
|
||||
timeout "${STRESS_TIMEOUT:-1800}" "$bin/wine" 'C:\esstress.exe' > "$work/$mode-stress.log" 2>&1 || status=$?
|
||||
"$bin/wineserver" -k 2>/dev/null || true
|
||||
if ! grep -q '^esstress:' "$work/$mode-stress.log"; then
|
||||
echo "$mode: esstress did not finish (status $status)"
|
||||
tail -n 30 "$work/$mode-stress.log"
|
||||
exit 1
|
||||
fi
|
||||
grep -o '^FAIL [^ ]*:[0-9]*:' "$work/$mode-stress.log" | sort -u > "$work/$mode-failures" || true
|
||||
echo "$mode: $(grep -E '^esstress:' "$work/$mode-stress.log" | tail -n 1)"
|
||||
)
|
||||
}
|
||||
|
||||
if ! run stock "$tool/files/bin-arm64" "server-side synchronization" PROTON_NO_NTSYNC=1 WINEESYNC=0 WINEFSYNC=0; then
|
||||
echo "stock Proton does not run here either; the pack cannot be judged on this machine"
|
||||
exit 3
|
||||
fi
|
||||
run esync "$work/dist/files/bin-arm64" "esync: up and running" PROTON_NO_NTSYNC=1 WINEESYNC=1 WINEFSYNC=0
|
||||
if [ -s "$work/stock-failures" ]; then
|
||||
echo "stock fails these as well, so they do not count against the pack: $(tr '\n' ' ' < "$work/stock-failures")"
|
||||
fi
|
||||
new=$(comm -13 "$work/stock-failures" "$work/esync-failures")
|
||||
if [ -n "$new" ]; then
|
||||
echo "esync: fails checks stock passes:"
|
||||
grep -F "$new" "$work/esync-stress.log" || echo "$new"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,618 +0,0 @@
|
||||
#include <windows.h>
|
||||
#include <winternl.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
static int failures;
|
||||
#define CHECK(c, ...) do { if (!(c)) { failures++; printf("FAIL %s:%d: ", __FILE__, __LINE__); printf(__VA_ARGS__); printf("\n"); fflush(stdout); } } while (0)
|
||||
|
||||
static volatile LONG counter;
|
||||
static HANDLE mutex, sem, ev_auto, ev_manual, start_ev;
|
||||
static volatile LONG stop;
|
||||
|
||||
static DWORD WINAPI mutex_worker(void *arg)
|
||||
{
|
||||
int i;
|
||||
WaitForSingleObject(start_ev, INFINITE);
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(mutex, INFINITE);
|
||||
LONG v;
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "mutex wait %lu", r); break; }
|
||||
v = counter;
|
||||
if (i % 7 == 0) WaitForSingleObject(mutex, 0), ReleaseMutex(mutex);
|
||||
counter = v + 1;
|
||||
ReleaseMutex(mutex);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_mutex_contention(void)
|
||||
{
|
||||
HANDLE th[8];
|
||||
int i;
|
||||
counter = 0;
|
||||
mutex = CreateMutexA(NULL, FALSE, NULL);
|
||||
start_ev = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
for (i = 0; i < 8; i++) th[i] = CreateThread(NULL, 0, mutex_worker, NULL, 0, NULL);
|
||||
SetEvent(start_ev);
|
||||
WaitForMultipleObjects(8, th, TRUE, INFINITE);
|
||||
CHECK(counter == 8 * 20000, "mutex counter %ld", counter);
|
||||
for (i = 0; i < 8; i++) CloseHandle(th[i]);
|
||||
CloseHandle(mutex);
|
||||
CloseHandle(start_ev);
|
||||
printf("mutex_contention ok counter=%ld\n", counter);
|
||||
}
|
||||
|
||||
static volatile LONG produced, consumed;
|
||||
|
||||
static DWORD WINAPI sem_producer(void *arg)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < 50000; i++)
|
||||
{
|
||||
LONG prev;
|
||||
while (!ReleaseSemaphore(sem, 1, &prev))
|
||||
{
|
||||
CHECK(GetLastError() == ERROR_TOO_MANY_POSTS, "release err %lu", GetLastError());
|
||||
Sleep(0);
|
||||
}
|
||||
InterlockedIncrement(&produced);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static DWORD WINAPI sem_consumer(void *arg)
|
||||
{
|
||||
for (;;)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(sem, 2000);
|
||||
if (r == WAIT_TIMEOUT) break;
|
||||
CHECK(r == WAIT_OBJECT_0, "sem wait %lu", r);
|
||||
InterlockedIncrement(&consumed);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_semaphore(void)
|
||||
{
|
||||
HANDLE th[8];
|
||||
LONG prev;
|
||||
int i;
|
||||
produced = consumed = 0;
|
||||
sem = CreateSemaphoreA(NULL, 0, 16, NULL);
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, sem_producer, NULL, 0, NULL);
|
||||
for (i = 4; i < 8; i++) th[i] = CreateThread(NULL, 0, sem_consumer, NULL, 0, NULL);
|
||||
WaitForMultipleObjects(8, th, TRUE, INFINITE);
|
||||
CHECK(produced == 200000 && consumed == 200000, "sem produced %ld consumed %ld", produced, consumed);
|
||||
CHECK(!ReleaseSemaphore(sem, 17, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "limit");
|
||||
CHECK(ReleaseSemaphore(sem, 16, &prev) && prev == 0, "prev %ld", prev);
|
||||
CHECK(!ReleaseSemaphore(sem, 1, &prev), "over max");
|
||||
for (i = 0; i < 8; i++) CloseHandle(th[i]);
|
||||
CloseHandle(sem);
|
||||
printf("semaphore ok %ld/%ld\n", produced, consumed);
|
||||
}
|
||||
|
||||
static volatile LONG auto_hits;
|
||||
|
||||
static DWORD WINAPI auto_waiter(void *arg)
|
||||
{
|
||||
while (!stop)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(ev_auto, 50);
|
||||
if (r == WAIT_OBJECT_0) InterlockedIncrement(&auto_hits);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_auto_event(void)
|
||||
{
|
||||
HANDLE th[6];
|
||||
int i;
|
||||
LONG expected = 0;
|
||||
stop = 0; auto_hits = 0;
|
||||
ev_auto = CreateEventA(NULL, FALSE, FALSE, NULL);
|
||||
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, auto_waiter, NULL, 0, NULL);
|
||||
for (i = 0; i < 30000; i++)
|
||||
{
|
||||
LONG before = auto_hits;
|
||||
SetEvent(ev_auto);
|
||||
expected++;
|
||||
while (auto_hits == before) SwitchToThread();
|
||||
}
|
||||
stop = 1;
|
||||
WaitForMultipleObjects(6, th, TRUE, INFINITE);
|
||||
CHECK(auto_hits == expected, "auto hits %ld expected %ld", auto_hits, expected);
|
||||
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto event left signaled");
|
||||
SetEvent(ev_auto); SetEvent(ev_auto);
|
||||
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_OBJECT_0, "auto set");
|
||||
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto coalesce");
|
||||
for (i = 0; i < 6; i++) CloseHandle(th[i]);
|
||||
CloseHandle(ev_auto);
|
||||
printf("auto_event ok hits=%ld\n", auto_hits);
|
||||
}
|
||||
|
||||
static volatile LONG manual_seen;
|
||||
|
||||
static DWORD WINAPI manual_waiter(void *arg)
|
||||
{
|
||||
DWORD r = WaitForSingleObject(ev_manual, 10000);
|
||||
if (r == WAIT_OBJECT_0) InterlockedIncrement(&manual_seen);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_manual_event(void)
|
||||
{
|
||||
HANDLE th[16];
|
||||
int i, round;
|
||||
ev_manual = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
for (round = 0; round < 50; round++)
|
||||
{
|
||||
manual_seen = 0;
|
||||
ResetEvent(ev_manual);
|
||||
for (i = 0; i < 16; i++) th[i] = CreateThread(NULL, 0, manual_waiter, NULL, 0, NULL);
|
||||
Sleep(2);
|
||||
SetEvent(ev_manual);
|
||||
WaitForMultipleObjects(16, th, TRUE, INFINITE);
|
||||
CHECK(manual_seen == 16, "manual round %d seen %ld", round, manual_seen);
|
||||
for (i = 0; i < 16; i++) CloseHandle(th[i]);
|
||||
}
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
SetEvent(ev_manual);
|
||||
if (i & 1) ResetEvent(ev_manual);
|
||||
}
|
||||
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_TIMEOUT, "manual final reset");
|
||||
SetEvent(ev_manual);
|
||||
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays");
|
||||
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays 2");
|
||||
CloseHandle(ev_manual);
|
||||
printf("manual_event ok\n");
|
||||
}
|
||||
|
||||
static HANDLE wa_objs[3];
|
||||
static volatile LONG wa_count;
|
||||
|
||||
static DWORD WINAPI wait_all_worker(void *arg)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < 3000; i++)
|
||||
{
|
||||
DWORD r = WaitForMultipleObjects(3, wa_objs, TRUE, 5000);
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "wait all %lu", r); break; }
|
||||
InterlockedIncrement(&wa_count);
|
||||
ReleaseMutex(wa_objs[0]);
|
||||
ReleaseSemaphore(wa_objs[1], 1, NULL);
|
||||
SetEvent(wa_objs[2]);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_wait_all(void)
|
||||
{
|
||||
HANDLE th[6], dup[2];
|
||||
int i;
|
||||
DWORD r;
|
||||
wa_count = 0;
|
||||
wa_objs[0] = CreateMutexA(NULL, FALSE, NULL);
|
||||
wa_objs[1] = CreateSemaphoreA(NULL, 1, 1, NULL);
|
||||
wa_objs[2] = CreateEventA(NULL, FALSE, TRUE, NULL);
|
||||
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, wait_all_worker, NULL, 0, NULL);
|
||||
WaitForMultipleObjects(6, th, TRUE, INFINITE);
|
||||
CHECK(wa_count == 18000, "wait all count %ld", wa_count);
|
||||
dup[0] = wa_objs[2];
|
||||
DuplicateHandle(GetCurrentProcess(), wa_objs[2], GetCurrentProcess(), &dup[1], 0, FALSE, DUPLICATE_SAME_ACCESS);
|
||||
r = WaitForMultipleObjects(2, dup, TRUE, 0);
|
||||
CHECK(r == WAIT_FAILED && GetLastError() == ERROR_INVALID_PARAMETER, "dup wait all %lu %lu", r, GetLastError());
|
||||
CloseHandle(dup[1]);
|
||||
for (i = 0; i < 6; i++) CloseHandle(th[i]);
|
||||
for (i = 0; i < 3; i++) CloseHandle(wa_objs[i]);
|
||||
printf("wait_all ok %ld\n", wa_count);
|
||||
}
|
||||
|
||||
static DWORD WINAPI hold_and_die(void *arg)
|
||||
{
|
||||
WaitForSingleObject((HANDLE)arg, INFINITE);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_abandoned(void)
|
||||
{
|
||||
HANDLE m = CreateMutexA(NULL, FALSE, NULL);
|
||||
HANDLE th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
|
||||
HANDLE objs[2];
|
||||
DWORD r;
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
r = WaitForSingleObject(m, 1000);
|
||||
CHECK(r == WAIT_ABANDONED, "abandoned %lu", r);
|
||||
CHECK(ReleaseMutex(m), "release after abandon");
|
||||
th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
objs[0] = CreateEventA(NULL, TRUE, TRUE, NULL);
|
||||
objs[1] = m;
|
||||
r = WaitForMultipleObjects(2, objs, TRUE, 1000);
|
||||
CHECK(r == WAIT_ABANDONED_0, "abandoned all %lu", r);
|
||||
ReleaseMutex(m);
|
||||
r = WaitForMultipleObjects(2, objs, FALSE, 1000);
|
||||
CHECK(r == WAIT_OBJECT_0, "any after %lu", r);
|
||||
CloseHandle(th); CloseHandle(objs[0]); CloseHandle(m);
|
||||
printf("abandoned ok\n");
|
||||
}
|
||||
|
||||
static volatile LONG apc_ran;
|
||||
static void CALLBACK apc_proc(ULONG_PTR p) { InterlockedIncrement(&apc_ran); }
|
||||
|
||||
static HANDLE apc_ready;
|
||||
|
||||
static DWORD WINAPI alert_waiter(void *arg)
|
||||
{
|
||||
SetEvent(apc_ready);
|
||||
return WaitForSingleObjectEx((HANDLE)arg, 5000, TRUE);
|
||||
}
|
||||
|
||||
static void test_apc(void)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
HANDLE th;
|
||||
DWORD r, code;
|
||||
int i;
|
||||
apc_ready = CreateEventA(NULL, FALSE, FALSE, NULL);
|
||||
for (i = 0; i < 200; i++)
|
||||
{
|
||||
apc_ran = 0;
|
||||
th = CreateThread(NULL, 0, alert_waiter, ev, 0, NULL);
|
||||
WaitForSingleObject(apc_ready, INFINITE);
|
||||
if (i & 1) Sleep(1);
|
||||
QueueUserAPC(apc_proc, th, 0);
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
GetExitCodeThread(th, &code);
|
||||
CHECK(code == WAIT_IO_COMPLETION && apc_ran == 1, "apc round %d code %lu ran %ld", i, code, apc_ran);
|
||||
CloseHandle(th);
|
||||
}
|
||||
apc_ran = 0;
|
||||
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
|
||||
r = SleepEx(1000, TRUE);
|
||||
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "sleepex %lu %ld", r, apc_ran);
|
||||
r = SleepEx(10, TRUE);
|
||||
CHECK(r == 0, "sleepex timeout %lu", r);
|
||||
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
|
||||
r = WaitForSingleObjectEx(ev, 0, FALSE);
|
||||
CHECK(r == WAIT_TIMEOUT, "non alertable %lu", r);
|
||||
r = WaitForSingleObjectEx(ev, 0, TRUE);
|
||||
CHECK(r == WAIT_IO_COMPLETION, "alertable pending %lu", r);
|
||||
CloseHandle(ev);
|
||||
printf("apc ok\n");
|
||||
}
|
||||
|
||||
static DWORD WINAPI close_waiter(void *arg)
|
||||
{
|
||||
return WaitForSingleObject((HANDLE)arg, 300);
|
||||
}
|
||||
|
||||
static void test_close_while_waiting(void)
|
||||
{
|
||||
int i;
|
||||
for (i = 0; i < 300; i++)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), ev2, th;
|
||||
DWORD code;
|
||||
th = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
Sleep(0);
|
||||
CloseHandle(ev);
|
||||
ev2 = CreateEventA(NULL, TRUE, TRUE, NULL);
|
||||
WaitForSingleObject(th, INFINITE);
|
||||
GetExitCodeThread(th, &code);
|
||||
CHECK(code == WAIT_TIMEOUT || code == WAIT_FAILED || code == WAIT_OBJECT_0, "close round %d code %lu", i, code);
|
||||
CHECK(WaitForSingleObject(ev2, 0) == WAIT_OBJECT_0, "reused handle state");
|
||||
CloseHandle(ev2);
|
||||
CloseHandle(th);
|
||||
}
|
||||
printf("close_while_waiting ok\n");
|
||||
}
|
||||
|
||||
static void test_pulse(void)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), th[4];
|
||||
DWORD code;
|
||||
int i, woke = 0;
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
Sleep(50);
|
||||
PulseEvent(ev);
|
||||
WaitForMultipleObjects(4, th, TRUE, INFINITE);
|
||||
for (i = 0; i < 4; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left signaled");
|
||||
CHECK(woke == 4, "pulse woke %d of 4", woke);
|
||||
CloseHandle(ev);
|
||||
ev = CreateEventA(NULL, FALSE, FALSE, NULL);
|
||||
PulseEvent(ev);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse without waiter left signaled");
|
||||
th[0] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
th[1] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
|
||||
Sleep(50);
|
||||
PulseEvent(ev);
|
||||
WaitForMultipleObjects(2, th, TRUE, INFINITE);
|
||||
woke = 0;
|
||||
for (i = 0; i < 2; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
|
||||
CHECK(woke == 1, "auto pulse woke %d of 2", woke);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse left signaled");
|
||||
CloseHandle(ev);
|
||||
printf("pulse ok\n");
|
||||
}
|
||||
|
||||
static void test_semantics(void)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, TRUE, TRUE, NULL), sem = CreateSemaphoreA(NULL, 2, 3, NULL), h[2];
|
||||
NTSTATUS (WINAPI *pNtWaitForSingleObject)(HANDLE, BOOLEAN, const LARGE_INTEGER *);
|
||||
LARGE_INTEGER when;
|
||||
LONG prev = -1;
|
||||
DWORD r, start;
|
||||
int i;
|
||||
pNtWaitForSingleObject = (void *)GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtWaitForSingleObject");
|
||||
apc_ran = 0;
|
||||
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
|
||||
r = WaitForSingleObjectEx(ev, INFINITE, TRUE);
|
||||
CHECK(r == WAIT_OBJECT_0 && apc_ran == 0, "object before apc %lu ran %ld", r, apc_ran);
|
||||
r = SleepEx(0, TRUE);
|
||||
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "apc after object %lu ran %ld", r, apc_ran);
|
||||
r = SleepEx(0, TRUE);
|
||||
CHECK(r == 0, "sleepex zero %lu", r);
|
||||
ResetEvent(ev);
|
||||
GetSystemTimeAsFileTime((FILETIME *)&when);
|
||||
when.QuadPart += 2000000;
|
||||
start = GetTickCount();
|
||||
r = pNtWaitForSingleObject(ev, FALSE, &when);
|
||||
CHECK(r == STATUS_TIMEOUT, "absolute timeout status %lx", r);
|
||||
CHECK(GetTickCount() - start >= 150 && GetTickCount() - start < 1500, "absolute timeout took %lu", GetTickCount() - start);
|
||||
CHECK(!ReleaseSemaphore(sem, 2, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "semaphore limit");
|
||||
CHECK(ReleaseSemaphore(sem, 1, &prev) && prev == 2, "semaphore prev %ld", prev);
|
||||
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(sem, 0) == WAIT_OBJECT_0, "semaphore take %d", i);
|
||||
CHECK(WaitForSingleObject(sem, 0) == WAIT_TIMEOUT, "semaphore empty");
|
||||
h[0] = sem; h[1] = ev;
|
||||
ReleaseSemaphore(sem, 1, NULL);
|
||||
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_TIMEOUT, "wait all partial");
|
||||
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 1, "wait all rollback lost token prev %ld", prev);
|
||||
SetEvent(ev);
|
||||
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_OBJECT_0, "wait all ready");
|
||||
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 0, "wait all consumed prev %ld", prev);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "manual stays set");
|
||||
CloseHandle(sem);
|
||||
CloseHandle(ev);
|
||||
printf("semantics ok\n");
|
||||
}
|
||||
|
||||
static HANDLE token_sem;
|
||||
static volatile LONG token_count;
|
||||
|
||||
static DWORD WINAPI token_worker(void *arg)
|
||||
{
|
||||
HANDLE h[2] = { token_sem, (HANDLE)arg };
|
||||
int i;
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
DWORD r = (i & 1) ? WaitForSingleObject(token_sem, INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "token wait %lu", r); break; }
|
||||
if (InterlockedIncrement(&token_count) > 3) CHECK(0, "token overcommit");
|
||||
InterlockedDecrement(&token_count);
|
||||
if (!(i & 1)) SetEvent((HANDLE)arg);
|
||||
ReleaseSemaphore(token_sem, 1, NULL);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_semaphore_tokens(void)
|
||||
{
|
||||
HANDLE th[6], ev = CreateEventA(NULL, FALSE, TRUE, NULL);
|
||||
LONG prev = -1;
|
||||
int i;
|
||||
token_sem = CreateSemaphoreA(NULL, 3, 3, NULL);
|
||||
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, token_worker, ev, 0, NULL);
|
||||
WaitForMultipleObjects(6, th, TRUE, INFINITE);
|
||||
for (i = 0; i < 6; i++) CloseHandle(th[i]);
|
||||
CHECK(ReleaseSemaphore(token_sem, 0, &prev) && prev == 3, "token final count %ld", prev);
|
||||
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(token_sem, 0) == WAIT_OBJECT_0, "token drain %d", i);
|
||||
CHECK(WaitForSingleObject(token_sem, 0) == WAIT_TIMEOUT, "token extra");
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "token event lost");
|
||||
CloseHandle(token_sem);
|
||||
CloseHandle(ev);
|
||||
printf("semaphore_tokens ok\n");
|
||||
}
|
||||
|
||||
static volatile LONG churn_ready;
|
||||
|
||||
static DWORD WINAPI churn_mutex(void *arg)
|
||||
{
|
||||
HANDLE h[2] = { (HANDLE)arg, ev_manual };
|
||||
unsigned int i = 0;
|
||||
if (WaitForMultipleObjects(2, h, TRUE, INFINITE) <= WAIT_ABANDONED_0 + 1) ReleaseMutex(h[0]);
|
||||
InterlockedIncrement(&churn_ready);
|
||||
for (;;)
|
||||
{
|
||||
DWORD r = (i++ & 1) ? WaitForSingleObject(h[0], INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
|
||||
if (r == WAIT_OBJECT_0 || r == WAIT_ABANDONED || r == WAIT_ABANDONED_0) ReleaseMutex(h[0]);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static DWORD WINAPI blocked_waiter(void *arg)
|
||||
{
|
||||
WaitForSingleObject((HANDLE)arg, 0);
|
||||
InterlockedIncrement(&churn_ready);
|
||||
return WaitForSingleObject((HANDLE)arg, INFINITE);
|
||||
}
|
||||
|
||||
static void test_terminate(void)
|
||||
{
|
||||
HANDLE th[4], m, ev;
|
||||
DWORD r, start;
|
||||
int round, i, stuck = 0;
|
||||
ev_manual = CreateEventA(NULL, TRUE, TRUE, NULL);
|
||||
for (round = 0; round < 150; round++)
|
||||
{
|
||||
m = CreateMutexA(NULL, FALSE, NULL);
|
||||
churn_ready = 0;
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, churn_mutex, m, 0, NULL);
|
||||
while (churn_ready < 4) Sleep(0);
|
||||
Sleep(round % 4);
|
||||
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
|
||||
WaitForMultipleObjects(4, th, TRUE, INFINITE);
|
||||
r = WaitForSingleObject(m, 3000);
|
||||
if (r == WAIT_TIMEOUT) stuck++;
|
||||
else ReleaseMutex(m);
|
||||
for (i = 0; i < 4; i++) CloseHandle(th[i]);
|
||||
CloseHandle(m);
|
||||
}
|
||||
CHECK(!stuck, "mutex stuck after terminate in %d rounds", stuck);
|
||||
ev = CreateEventA(NULL, TRUE, FALSE, NULL);
|
||||
churn_ready = 0;
|
||||
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, blocked_waiter, ev, 0, NULL);
|
||||
while (churn_ready < 4) Sleep(0);
|
||||
Sleep(50);
|
||||
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
|
||||
WaitForMultipleObjects(4, th, TRUE, INFINITE);
|
||||
for (i = 0; i < 4; i++) CloseHandle(th[i]);
|
||||
start = GetTickCount();
|
||||
for (i = 0; i < 100; i++) PulseEvent(ev);
|
||||
CHECK(GetTickCount() - start < 400, "pulse after killed waiters took %lu ms", GetTickCount() - start);
|
||||
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left event set");
|
||||
CloseHandle(ev);
|
||||
CloseHandle(ev_manual);
|
||||
printf("terminate ok\n");
|
||||
}
|
||||
|
||||
static void test_cross_process(const char *self)
|
||||
{
|
||||
HANDLE ev = CreateEventA(NULL, FALSE, FALSE, "Local\\esstress_ev");
|
||||
HANDLE done = CreateEventA(NULL, TRUE, FALSE, "Local\\esstress_done");
|
||||
HANDLE sm = CreateSemaphoreA(NULL, 0, 100000, "Local\\esstress_sem");
|
||||
PROCESS_INFORMATION pi;
|
||||
STARTUPINFOA si = { sizeof(si) };
|
||||
char cmd[MAX_PATH + 16];
|
||||
DWORD r, code;
|
||||
int i;
|
||||
sprintf(cmd, "\"%s\" child", self);
|
||||
CHECK(CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi), "createprocess %lu", GetLastError());
|
||||
for (i = 0; i < 5000; i++)
|
||||
{
|
||||
SetEvent(ev);
|
||||
r = WaitForSingleObject(sm, 5000);
|
||||
if (r != WAIT_OBJECT_0) { CHECK(0, "xproc wait %lu at %d", r, i); break; }
|
||||
}
|
||||
SetEvent(done);
|
||||
r = WaitForSingleObject(pi.hProcess, 10000);
|
||||
CHECK(r == WAIT_OBJECT_0, "child exit wait %lu", r);
|
||||
GetExitCodeProcess(pi.hProcess, &code);
|
||||
CHECK(code == 0, "child code %lu", code);
|
||||
CloseHandle(pi.hProcess); CloseHandle(pi.hThread);
|
||||
CloseHandle(ev); CloseHandle(done); CloseHandle(sm);
|
||||
printf("cross_process ok\n");
|
||||
}
|
||||
|
||||
static int child(void)
|
||||
{
|
||||
HANDLE ev = OpenEventA(SYNCHRONIZE | EVENT_MODIFY_STATE, FALSE, "Local\\esstress_ev");
|
||||
HANDLE done = OpenEventA(SYNCHRONIZE, FALSE, "Local\\esstress_done");
|
||||
HANDLE sm = OpenSemaphoreA(SEMAPHORE_MODIFY_STATE | SYNCHRONIZE, FALSE, "Local\\esstress_sem");
|
||||
HANDLE objs[2];
|
||||
objs[0] = done; objs[1] = ev;
|
||||
if (!ev || !done || !sm) return 1;
|
||||
for (;;)
|
||||
{
|
||||
DWORD r = WaitForMultipleObjects(2, objs, FALSE, 10000);
|
||||
if (r == WAIT_OBJECT_0) return 0;
|
||||
if (r != WAIT_OBJECT_0 + 1) return 2;
|
||||
ReleaseSemaphore(sm, 1, NULL);
|
||||
}
|
||||
}
|
||||
|
||||
static void test_kill_holder(const char *self)
|
||||
{
|
||||
HANDLE m = CreateMutexA(NULL, FALSE, "Local\\esstress_mutex");
|
||||
PROCESS_INFORMATION pi;
|
||||
STARTUPINFOA si = { sizeof(si) };
|
||||
char cmd[MAX_PATH + 16];
|
||||
DWORD r;
|
||||
sprintf(cmd, "\"%s\" holder", self);
|
||||
CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);
|
||||
Sleep(500);
|
||||
TerminateProcess(pi.hProcess, 7);
|
||||
WaitForSingleObject(pi.hProcess, INFINITE);
|
||||
r = WaitForSingleObject(m, 3000);
|
||||
CHECK(r == WAIT_ABANDONED, "killed holder %lu", r);
|
||||
ReleaseMutex(m);
|
||||
CloseHandle(pi.hProcess); CloseHandle(pi.hThread); CloseHandle(m);
|
||||
printf("kill_holder ok\n");
|
||||
}
|
||||
|
||||
static int holder(void)
|
||||
{
|
||||
HANDLE m = OpenMutexA(SYNCHRONIZE, FALSE, "Local\\esstress_mutex");
|
||||
WaitForSingleObject(m, INFINITE);
|
||||
Sleep(INFINITE);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void test_many_objects(void)
|
||||
{
|
||||
static HANDLE h[20000];
|
||||
int i, n = 0;
|
||||
for (i = 0; i < 20000; i++)
|
||||
{
|
||||
if (!(h[i] = CreateEventA(NULL, i & 1, i & 2, NULL))) break;
|
||||
n++;
|
||||
}
|
||||
CHECK(n == 20000, "created %d", n);
|
||||
for (i = 0; i < n; i++) CHECK(WaitForSingleObject(h[i], 0) == ((i & 2) ? WAIT_OBJECT_0 : WAIT_TIMEOUT), "obj %d", i);
|
||||
for (i = 0; i < n; i++) CloseHandle(h[i]);
|
||||
for (i = 0; i < 100000; i++)
|
||||
{
|
||||
HANDLE e = CreateEventA(NULL, FALSE, TRUE, NULL);
|
||||
if (WaitForSingleObject(e, 0) != WAIT_OBJECT_0) { CHECK(0, "churn %d", i); CloseHandle(e); break; }
|
||||
CloseHandle(e);
|
||||
}
|
||||
printf("many_objects ok\n");
|
||||
}
|
||||
|
||||
static void test_timers_and_threads(void)
|
||||
{
|
||||
HANDLE t = CreateWaitableTimerA(NULL, FALSE, NULL);
|
||||
LARGE_INTEGER due;
|
||||
DWORD start = GetTickCount(), r;
|
||||
int i;
|
||||
due.QuadPart = -10 * 1000 * 20;
|
||||
SetWaitableTimer(t, &due, 10, NULL, NULL, FALSE);
|
||||
for (i = 0; i < 20; i++)
|
||||
{
|
||||
r = WaitForSingleObject(t, 1000);
|
||||
CHECK(r == WAIT_OBJECT_0, "timer %d %lu", i, r);
|
||||
}
|
||||
CHECK(GetTickCount() - start < 2000, "timer slow %lu", GetTickCount() - start);
|
||||
CancelWaitableTimer(t);
|
||||
CloseHandle(t);
|
||||
r = WaitForSingleObject(GetCurrentProcess(), 10);
|
||||
CHECK(r == WAIT_TIMEOUT, "process self %lu", r);
|
||||
printf("timers ok\n");
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
if (argc > 1 && !strcmp(argv[1], "child")) return child();
|
||||
if (argc > 1 && !strcmp(argv[1], "holder")) return holder();
|
||||
test_mutex_contention();
|
||||
test_semaphore();
|
||||
test_auto_event();
|
||||
test_manual_event();
|
||||
test_wait_all();
|
||||
test_abandoned();
|
||||
test_apc();
|
||||
test_close_while_waiting();
|
||||
test_pulse();
|
||||
test_semantics();
|
||||
test_semaphore_tokens();
|
||||
if (argc > 1 && !strcmp(argv[1], "terminate")) test_terminate();
|
||||
test_cross_process(argv[0]);
|
||||
test_kill_holder(argv[0]);
|
||||
test_many_objects();
|
||||
test_timers_and_threads();
|
||||
printf("esstress: %d failures\n", failures);
|
||||
return failures ? 1 : 0;
|
||||
}
|
||||
Reference in new issue
Block a user