ESync, FSync, NTSync + OpenNow + FEX for Desktop Apps (#197)

* Make the Compatible Proton ARM64 the default and label compatible builds

The CompatToolMapping "0" default always points at DroidDeck's Proton
ARM64 tool, and per-game picks of adopted GE/CachyOS builds are kept,
including when the shortcuts writer remaps.

The tool is named after the Proton build it runs, Proton Experimental
ARM64 (Compatible), or Proton 11.0 ARM64 (Compatible) once that newer build
is installed. Adopted builds carry the same Compatible mark, read from a
string resource the app stages into the runtime, and the Proton page
explains which builds have the prefix fixes.

The registrar is renamed steam-compatibility and the tool
droiddeck-proton-arm64. Existing installs migrate on the next session: the
old tool directory is removed, every mapping that named it moves to the new
tool, and the retired bannerlator-steam-compat script is removed from the
runtime. CI and local packaging include the renamed script.

* Drop the duplicate R import left by merging main

* Add userspace ntsync for Proton on Android

Android kernels have no /dev/ntsync and its app policy refuses futex_waitv,
so every Proton 11 build here fell back to wineserver for every wait.
libblsession now serves /dev/ntsync itself: wineserver opens a memfd region,
each sync object is a timerfd that carries its slot, and the kernel ntsync
ioctls are answered in shared memory with futexes, following the kernel
driver's wake order. Valve's Proton Experimental ARM64, GE-Proton and
proton-cachyos use it unchanged.

Locks record their owner so a killed process cannot wedge the others,
signals are held off inside each critical section, dead processes and
exiting threads are swept, and object lifetimes follow close, dup2, dup3,
close_range and received descriptors.

Fast sync is on by default under Performance > Session fixes, and
PROTON_NO_NTSYNC=1 turns it off per game.

* Add droiddeck-esync as the default sync for Proton on Android

Without a kernel ntsync driver, Proton 11 builds here fell back to
wineserver for every wait. droiddeck-esync brings eventfd and shared-memory
sync to the exact Proton build a game runs on: a pack holds that build's
ntdll.so and wineserver with the patch series in tools/droiddeck-esync
applied, and the guest runs Proton from a shadow tree that links the stock
tool and swaps in those two files. A wineserver from a pack still prefers
/dev/ntsync when a kernel driver or droiddeck-ntsync is there.

Packs are built in Valve's SDK image from each release's own wine source
(Proton Experimental ARM64 tags, GE-Proton and proton-cachyos releases); the
control build must reproduce the release binaries byte for byte, and a
signed index lists the packs. The APK bundles the packs pinned in
release.env, the app fetches newer ones, and Steam's compatibility tool
picks the pack for each Proton build, so new Proton releases get packs
without an app update.

The ntdll side reads, writes and polls its eventfds with raw system calls,
so the guest's LD_PRELOAD wrappers stay out of the sync path. The server
abandons a mutex once a thread killed by TerminateThread has really exited
and when its process dies, and a mutex's owner and recursion count change
in one atomic step, so a killed owner cannot touch the next owner's count.

droiddeck-esync is on by default and PROTON_NO_ESYNC=1 turns it off per
game; droiddeck-ntsync, the userspace ntsync, is now an opt-in toggle.

* Watch Valve's Proton releases and build droiddeck-esync packs for them

A scheduled workflow reads the Proton ARM64 builds Steam ships (anonymous
steamcmd app info) and the upstream source tags, compares them with the
previous watch state and starts the pack build for any build that has no
pack yet. The pack build applies the patch series with --fuzz=0 and runs a
smoke test of each new pack against the esstress suite before it is
published.

* Add droiddeck-fsync, keep Steam and the app on one Proton, fix DirectAudio

droiddeck-fsync: libblsession answers futex_waitv(2), which the app
sandbox refuses, in userspace, so Proton's own fsync runs on the stock
Proton files with no pack. Waiters register the words they wait on in a
table beside the fsync shared memory and sleep on a futex of their own;
FUTEX_WAKE on the fsync memory wakes the registered waiters. Slots carry
one owner word that the sweeper and the owner claim with a CAS, and the
mapping table is rebuilt under a lock held across the real munmap, so a
reused address or a recycled slot cannot lose a waiter. It stands in for
a Proton build without a pack (droiddeck-ntsync where fsync is off), takes
over for games with esync off, and the "droiddeck-fsync first" toggle
(BL_FSYNC_FIRST, also per game) puts it ahead of the packs. The games
Proton keeps off fsync and esync never get it. A launch into a prefix
whose wineserver is still running keeps that server's sync choice.

One Proton for Steam and the app: the Components page's Proton is the
default Steam uses, applied live through the client while Steam runs
(droiddeck-steam-compat) and at the next start otherwise; a choice made in
Steam's menu comes back to the app. Requests are ordered by a sequence that
never goes backwards, and the page no longer jumps back to the default
after a swap or restore.

DirectAudio: the wrapper wrote the driver key with its backslashes
stripped, so no game ever used DirectAudio. Wine 11 also loads a builtin
only from the prefix, so the driver is now linked into system32 and
syswow64 before "directaudio,pulse" is selected; mmdevapi falls back to
winepulse whenever DirectAudio cannot start.

* Pick esync, ntsync or fsync from the Steam settings' Games group

A three-way tab row above the FEX preset sets the sync Proton games use. It
writes the existing switches together (droiddeck-ntsync, droiddeck-fsync
first, droiddeck-esync), so the Performance toggles and the tabs always agree,
starts on esync and keeps the choice until it is changed again.

* Make droiddeck-fsync the default sync for Proton games

A fresh install, or one that never picked a sync, starts on fsync; a choice made
in the Steam settings' Games group still sticks. Games Proton keeps off fsync
still fall back to droiddeck-esync.

* Add wineserver to the Wine sync tabs and order them ntsync, fsync, esync, wineserver

wineserver turns every in-process sync off so games wait through wineserver;
picking any other tab turns sync back on. The tabs shrink to fit a narrow pane.

* Shorten the Wine sync description

* Answer openat2 and execveat for FEX in the session preload

FEX opens every file of its x86 rootfs with openat2(RESOLVE_IN_ROOT) and
only falls back to the host path on EXDEV. Under proot on Android openat2
answers ENOSYS, so each lookup of an x86 library silently went to the host
and missed. The exported syscall() now turns openat2 into openat once the
real call reports ENOSYS: RESOLVE_IN_ROOT paths are taken relative to the
root fd (FEX has already resolved the links inside the rootfs), and an
absolute path under RESOLVE_BENEATH fails with EXDEV as the kernel would.

execveat with an absolute path, or relative to AT_FDCWD, and without
AT_EMPTY_PATH or AT_SYMLINK_NOFOLLOW, is plain execve; it is sent as such
because proot refuses execveat with a directory fd, which FEX uses when it
re-executes itself for a child x86 program.

* Run x86 Linux games and programs through FEX

Steam: Linux-only titles that have no ARM64 build (from the client's
appinfo cache: type game, demo, application or beta) are mapped to a new
compatibility tool, droiddeck-fex ("Linux x86 (FEX)", linux to linux,
no required runtime). Valve's own "fex" tool needs pressure-vessel, which
cannot run under proot, so a mapping to it or to a Steam Linux Runtime is
replaced; any other pick the user made stays. Titles that also ship for
Windows stay on Proton. When such a title is installed and FEX or the
sniper runtime is missing, the client is asked to install them.

droiddeck-fex runs the program under the FEX that Steam installs (app
3127680), with Steam Linux Runtime 3.0 (sniper, app 1628350) as the x86
rootfs:
- the sniper platform is linked in place from its usr-mtree (relative
  links only, files/usr -> .), once per runtime version;
- the x86_64 and i386 builds of libblsession and libfakeinput go into the
  runtime's lib/<triplet>/droiddeck and its etc/ld.so.preload, mirroring
  the session's own list, so x86 programs get the pads and the session
  fixes (FEX passes their system calls straight to the kernel, so the
  ARM64 preloads cannot reach them);
- Config.json sets RootFS and the GL and Vulkan thunks, so x86 games draw
  with the device's Turnip and Mesa;
- the FEX preset and the game's own environment from the Steam settings
  apply, and ARM64 entries are dropped from LD_PRELOAD/LD_LIBRARY_PATH.
It also links ~/.steam/sdk64 and sdk32 to the client's x86 steamclient
when the client has one and nothing is there yet.

Desktop: x86_64 and i386 AppImages can be imported now; they unpack
through FEX (or unsquashfs) and record their architecture. Scripts,
AppImages and programs started in a run session use FEX automatically
when they are x86, and each added app has a "Use FEX" Auto/On/Off choice
in its edit dialog. The launchers' "still running" checks also look at
FEX processes, whose exe is FEX rather than the program. A GitHub release
without an ARM64 AppImage falls back to its x86_64 one. Decky's x86
loader starts through droiddeck-fex instead of the fex-mesa rootfs path,
which no runtime ships.

The x86 preloads are built against glibc 2.31 (Debian 11) so they load
in sniper: tools/linuxfs/build-x86-preloads.sh, run in CI and by
build_local.sh. fakeinput's stat hooks fall back to __xstat on a glibc
without a stat export, and mmap64 takes off64_t for the i386 build.

* Keep a FEX unixlib from outliving its DLL

A FEX package or original bundle without the aarch64-unix helpers left
the previous version's libarm64ecfex.so / libwow64fex.so in the Proton,
so the new DLLs loaded a unixlib built for another FEX. A swap now removes
the helpers the package does not carry; FEX then runs DLL-only, which it
already falls back to when Wine cannot load the unixlib.

* Fix the Wine sync choices an audit found

- wineserver tab: it passed Proton's environment through, so a build with
  esync built in (GE, cachyos) still ran esync and a kernel ntsync would
  still be used. It now sets PROTON_NO_ESYNC, PROTON_NO_FSYNC and
  PROTON_NO_NTSYNC (unless the game's own environment sets them).
- ntsync tab: a Proton whose wineserver has no /dev/ntsync client got
  wineserver-only sync although fsync would work. The hash cache records
  whether the wineserver has one; without it the launch goes to
  droiddeck-fsync, and no stand-in or pack turns droiddeck-ntsync on.
- A build with esync built in returned before the per-game fallbacks, so
  a game that turns esync off ran with no in-process sync. It now gets
  droiddeck-fsync, or droiddeck-ntsync where fsync is off too.
- When a pack's proton cannot start and the stock one is used, or the
  selection fails, the prefix's record now says stock, so a second
  launch into the same running prefix no longer picks the pack's ntdll.
- droiddeck-fsync: its mutex is held across fork and re-made in the
  child, a closed or replaced shm fd is forgotten (close, dup2, dup3,
  close_range, closefrom), and a waiter woken while it was leaving on a
  timeout or signal reports the wake instead of losing it.
- BL_SYNC_MIN_NOFILE applies per game to every verb, debug logs carry
  prefixes.json, the device report names the chosen sync instead of the
  three raw switches, and PATCHES.md and the proot notes describe the
  selector, the fsync default and droiddeck-fsync's futex_waitv.

* Name the runtime's scripts and files droiddeck, migrating old installs

The guest scripts are droiddeck-* now (droiddeck-session and the other 19),
as are the files the app and scripts generate (etc/droiddeck, the network
state files, the components dir, the Steam install stamp, the redist marker,
the extra-Proton wrapper and backups) and the compositor's own names
(libdroiddeckwayland, droiddeck_* sources, DROIDDECK_* switches, the
wl_output and seat names).

Installed runtimes carry on: LegacyNames moves etc/bannerlator and
share/bannerlator over and leaves relative links at the old paths before
every proot command, turns any old script into a link to its new name,
moves the generated files and points menu entries at the new scripts.
The session scripts and steam-compatibility move their markers on first
use, the components dir's desired.tsv is rewritten with it, and the
runtime counts as installed under either session script.

Kept: the banner_ahb_v1/banner_desktop_v1 wire names and BANNER_AUDIO_DIRECT
(read by prebuilt Turnip and Wine builds), Banners-Turnip and other
third-party names, BL_* switches, libblsession, and credits to Bannerlator.

* Unpack x86 AppImages without FEX, then ask Steam for FEX

Importing an x86 AppImage failed when FEX was not installed yet ("needs FEX
to unpack and no unsquashfs is installed"). AppImages now unpack with
uruntime (VHSgunzo, MIT, v0.8.1 aarch64, pinned by sha256 and shipped
unmodified with its licence), which reads SquashFS and DwarFS images of any
architecture natively. FEX's own --appimage-extract and unsquashfs stay as
fallbacks, and an ARM64 image whose own runtime cannot unpack it goes the
same way.

An x86 import with FEX missing records that FEX is wanted, so the next
Steam session requests FEX and the Steam Linux Runtime, and the program's
page says FEX is not set up yet and how to get it.

* Run Vulkan Qt Quick AppImages on Wayland

OpenNOW (Qt 6.8, Qt Quick on Vulkan) closed within two seconds in a run
mode. The session sends toolkits to gamescope's Xwayland, and Qt's xcb
backend only presents with Vulkan when Mesa's X11 WSI does, which needs
DRI3; Xwayland on KGSL has none, so Qt found no queue that can present and
aborted ("Failed to create RHI").

An AppImage that bundles Qt Quick, the Vulkan loader and Qt's Wayland
plugin now runs on gamescope's Wayland socket (the desktop's, outside
gamescope), where Turnip presents as it does for games, with Qt's xcb
backend as the fallback. SDL_VIDEODRIVER and XDG_SESSION_TYPE follow, so
helpers that pick their window system from them (OpenNOW's streamer) agree
with Qt. Everything else stays on X11; a qt file in the app's directory
(wayland or xcb) overrides the choice.

* Keep gamescope's WSI layer away from Qt apps on its Wayland socket

OpenNOW still closed at once: its FFmpeg (the Raspberry Pi fork, libavutil
61) creates its Vulkan queues internally synchronized whenever the driver
offers VK_KHR_internally_synchronized_queues, which DD-Turnip does. The
gamescope WSI layer registers every queue of a new device through
vkGetDeviceQueue, which only returns queues created without flags, so it
got VK_NULL_HANDLE and vkroots aborted on assert(obj) before Qt opened a
window.

Programs that draw on gamescope's Wayland socket do not need the layer,
which only carries X11 windows past Xwayland, so it is turned off for
them. OpenNOW bundles no Vulkan loader (it reaches Vulkan through Qt and
dlopen), so a binary that uses QVulkanInstance counts as a Vulkan app too,
and a Qt Quick app left on X11 says why in the session log.

* Keep the end of the session log, and say when FEX is missing

The session sends its output through head -c into session.log. head
writes through stdio, so it held up to 4 KB that proot's --kill-on-exit
threw away when the guest ended: every log stopped mid-line at a 4096-byte
boundary, losing exactly the lines that say why a program quit (an x86
AppImage's "x86 Linux programs need FEX" among them). dd writes each read
as it comes and keeps the same cap.

A run mode always ends with status 0 (gamescope does not pass its child's
on), so a program that could not start just closed. When the log says FEX
is missing, the ended screen now says so and how to get it.

The app reads FEX's readiness from status.json, which only a launch or an
import wrote, so it still said "not set up" after Steam installed FEX.
droiddeck-fex status rewrites it, and a Steam session ends with it while
FEX is wanted. FEX is also looked for at the tool's root and under files/,
and a tool folder without it lists what it holds.

* Let FEX and its runtime install runnable on an SD card library

A Steam library on shared storage loses the executable bits, so native
tools are staged in private directories bound over it; only Proton and
SteamLinuxRuntime folders were. FEX (3127680) installed there could never
run, droiddeck-fex saw no FEX and kept asking Steam for it. FEX and the
sniper runtime (1628350) now get private directories from their first
download, whatever their install folder is called.

* Keep fsync off in the pack smoke test

A runner whose kernel allows futex_waitv started stock Proton with fsync,
so the server-side baseline never said what the test waits for and every
pack was waved through as untestable; the esync run would have measured
fsync too.

* Judge a pack against stock's own esstress failures

Stock Proton fails esstress's dup wait all under server-side sync as well
(a Wine bug, seen on the phone too), and any stock failure made the test
call the runner unusable. A pack now fails only when it crashes, hangs or
fails a check stock passes.

* Ask Steam for FEX and its runtime one at a time

Both steam://install requests were started together, and the running client
took only the first: FEX installed, the sniper runtime FEX needs never did.
Each request now waits for the client to take it before the next.

* Bundle the first published droiddeck-esync packs in the apk

* Log how an x86 program under FEX ends

FEX keeps its own errors quiet unless asked, so a program it could not run
left nothing in the session log. Its log now goes to stderr, and the AppImage
launcher records the program's exit status or the signal that ended it.

* Show the log a failed AppImage program wrote

Programs like Eden stop themselves on a failed check and say why only in
their own log under the home folder. When the program ends badly, the end of
any log it wrote during the run is copied into the session log.

* Report where an x86 program under FEX hit an illegal instruction

A tiny x86_64 preload, built without libc, catches SIGILL and writes the
address, the instruction bytes and the mapping they belong to before the
program dies as it would have. droiddeck-fex preloads it into x86_64 programs.

* Load the fault reporter from the x86 runtime

Programs under FEX look libraries up in the x86 runtime, so the reporter is
copied there with the other preloads and named by its path inside it.

* Hand FEX its x86 runtime and thunks through the environment

Valve's FEX runs portable and reads its config beside itself, so the
Config.json droiddeck-fex writes was not seen: programs ran without the
runtime. The runtime, thunk libraries and thunk list now also go in FEX's
own environment variables, and each launch says what it was given.

* Use the sync pack built for the installed Proton release

Some Proton releases ship byte-identical ntdll.so and wineserver (Valve
11.0-2b/2c, CachyOS 0702/0703), so a pack built for a newer release
matched an older install exactly and the older install never fetched its
own pack. A pack from a newer release no longer counts as an exact match;
among matches the one built for the installed version wins, then rev.

The app ranks the pack a tool already uses the same way and only fetches
a pack that ranks higher, reads the release epoch from tools.tsv and
refetches when tools.tsv changes. A prefix whose server is still running
re-checks its recorded pack on waitforexitandrun and starts fresh when a
better one now applies.

* Keep published sync packs and revocations consistent

The publish step never replaces a pack that has an entry beside it; a
pack left without its entry by an interrupted run is replaced so the
build stops repeating. Discovery counts revocations the signed index does
not show yet, including superseded packs only the flavor releases list,
so the index is signed again. The release watch always builds every
flavor.

* Fetch the droiddeck-esync packs from DroidDeck-Components

The packs, the signed index and the apk bundle now live in the releases
of Droid-Deck/DroidDeck-Components, together with the pipeline that
builds them, so this repository keeps only the app and the guest code
that consumes them. The app's index and asset URLs point there,
release.env names the repository its bundle comes from, and the apk
builds drop bundled packs that the published index revokes.

* Stop the apk build when the published sync index cannot be read

* Ask Steam for one FEX dependency at a time

The client shows a confirmation dialog for each steam://install, and a
second request replaces the first dialog, so FEX was never installed
when the runtime was requested right after it. The next dependency is
now requested once the previous one has an app manifest.

* Use generic names in the AppImage test fixtures

* Give x86-64 apps under FEX the Vulkan thunk

Apps that bring their own glibc and Vulkan loader skipped FEX's thunk
overlay and loaded their bundled desktop Mesa drivers, which crash on the
phone's GPU during Vulkan setup. droiddeck-fex now adds an x86-64 audit
module when a program's bundled glibc and libstdc++ meet the thunk's
version needs. The module sends libvulkan.so.1 lookups to a nodelete stub
that needs FEX's thunk path. That keeps the thunk mapped when an app
unloads Vulkan and loads it again, since FEX keeps its first trampoline
links and the old addresses get reused.

The fault reporter now restores the default action before re-raising,
because FEX ignores SA_RESETHAND and the handler looped.
This commit is contained in:
MaxsTechReview authored and GitHub committed 2026-10-04 12:43:57 -04:00
1 parent 4cc9f2a3f3
commit 49a5f4aa94
165 files changed
+11709 -9667

No files matched your search

-170
View File
@@ -1,170 +0,0 @@
# droiddeck-esync packs
DroidDeck gives Proton three in-process sync backends, picked with the Wine sync tabs in the
Steam settings' Games group: ntsync, fsync (the default), esync and wineserver. The wineserver tab
sets `PROTON_NO_ESYNC=1`, `PROTON_NO_FSYNC=1` and `PROTON_NO_NTSYNC=1` (unless a game's own
environment sets them), so every sync object goes through the server.
- droiddeck-esync: eventfd and shared memory sync. Stock Wine 11 / Proton 11 builds
have no eventfd-based sync, so the patch series under `patches/<series>` adds it to wineserver
and the unix side of ntdll. A Proton build without a pack (a release newer than the published
packs, or one whose pack failed to install) runs stock with droiddeck-fsync standing in, or
droiddeck-ntsync where fsync is off for the game (`PROTON_NO_FSYNC=1`, or one of the games
Proton itself keeps off fsync and esync), so it still gets in-process sync; `PROTON_NO_NTSYNC=1`
as well leaves it on wineserver-only sync. A Proton with esync built in (GE, cachyos) runs its
own esync, and a game that turns esync off moves to droiddeck-fsync or droiddeck-ntsync there
too.
- droiddeck-fsync: Proton's own fsync on the stock binaries. fsync waits with `futex_waitv(2)`,
which the app sandbox refuses, so `libblsession.so` (`BL_FSYNC=1`) answers that system call in
userspace: every waiter registers the futexes it waits on in a shared table next to the fsync
shared memory and sleeps on a futex of its own, and every `FUTEX_WAKE` on the fsync shared memory
wakes the waiters registered on that word. No pack is involved, so it keeps working the moment
Steam updates Proton. It is the default (the fsync tab, `BL_FSYNC_FIRST=1`, also per game),
ahead of droiddeck-esync; on the esync tab it stands in where no pack fits, and it also takes
over for a game whose environment turns esync off (`PROTON_NO_ESYNC=1`, `WINEESYNC=0`, or too
few file descriptors); where fsync is off as well, such a game gets droiddeck-ntsync, with the
pack's wineserver when a pack fits. A launch into a prefix whose wineserver is still running
keeps the sync choice that server was started with (`prefixes.json` in the store), because a
Wine client that disagrees with its server about fsync exits. It inherits stock fsync's behaviour, the same as on a Linux kernel:
a wait-all on a mutex abandoned by a dead thread is not satisfied, a pending user APC is
delivered ahead of an object that is already signaled, a mutex owned by a process killed with
`TerminateProcess` is not abandoned, and `PulseEvent` can miss a waiter that has not run yet.
- droiddeck-ntsync: a userspace implementation of the `/dev/ntsync` interface in
`libblsession.so` (`BL_SYNC=1`), which stock Proton 11 builds use as they would the kernel
driver. It is the ntsync tab. A pack still supplies the binaries when one fits, and its
wineserver then picks ntsync over esync; a Proton whose wineserver has no ntsync client (no
`/dev/ntsync` in it) gets droiddeck-fsync instead.
wineserver picks the backend at startup: `PROTON_NO_NTSYNC` unset -> `/dev/ntsync`
(droiddeck-ntsync, or a kernel driver the device provides) -> fsync (droiddeck-fsync, or a kernel
and sandbox that allow `futex_waitv`) -> droiddeck-esync (needs `WINEESYNC` nonzero) ->
server-side sync. When a pack is in use and droiddeck-esync is on, the guest's `droiddeck-esync`
script also sets `PROTON_NO_FSYNC=1` unless the game's environment sets it, so droiddeck-esync
comes before fsync unless droiddeck-fsync was chosen. A wineserver from a pack reports the
backend on stderr as `droiddeck-ntsync: up and running.` or `droiddeck-esync: up and running.`;
with droiddeck-fsync the stock wineserver prints `fsync: up and running.` after
`droiddeck-fsync: up and running.`. A pack's wineserver also abandons, when a process dies, every
mutex still owned by one of its threads, and, once a thread killed by `TerminateThread` has really
exited, every mutex it still owns. A killed thread keeps running until the kill signal reaches it
and can take a mutex after the server abandoned its mutexes; without these passes that mutex would
stay locked for good (they cover droiddeck-ntsync and kernel ntsync as well). The ntdll side
reads, writes and polls its eventfds with raw system calls, so no `LD_PRELOAD` wrapper of
`read`/`write`/`poll` (the guest preloads several) sits in the sync path or in the unwind of a
thread Wine kills.
The patch changes no wineserver request: `make_pack.py` refuses a build whose
`server_protocol.h`, `request_handlers.h` or `request_trace.h` differ from the unpatched
control build, or whose ntdll.so exports other symbols. Only `files/lib/wine/aarch64-unix/ntdll.so`
and `files/bin-arm64/wineserver` are replaced, so every Proton build needs its own pair, built
from that build's exact wine source. That pair plus `pack.json` is a pack (`<id>.tzst`). On the
device the guest runs Proton from a shadow tree that links the stock tool and holds the two
patched files (`droiddeck-esync`, store in `~/.local/share/droiddeck-esync`).
## Credits
droiddeck-esync follows the esync design and droiddeck-ntsync implements the ntsync interface;
both are the work of Elizabeth Figura for Wine, Proton and the Linux kernel. droiddeck-fsync runs
fsync, by Elizabeth Figura and Paul Gofman for Proton, over `futex_waitv(2)` by André Almeida.
DroidDeck builds on that work for Android, where neither the ntsync driver, `futex_waitv` nor a
Proton build with esync is available.
## Matching
- `ge`, `cachyos`: built against the release tarball's own binaries. `pack.json` `stock` holds
their sha256; the guest uses the pack only when both files hash the same. `make_pack.py` also
requires the control build to reproduce both files byte for byte, so the patched pair differs
from the release only by the patch (`--allow-unreproduced` lifts this for local experiments).
- `valve`: the depots cannot be downloaded without a Steam login, so `stock` holds the control
build's hashes and `source_match` is true: the guest also accepts a tool whose version token
(second field of `<tool>/version`) and ntdll.so exports hash equal the pack's. With the optional
`STEAM_DEPOT_USER` / `STEAM_DEPOT_TOKEN` secrets (and the `DEPOT_DOWNLOADER_SHA256` variable)
the job downloads the depot instead and, when the depot is at the tag being built, makes an
exact pack. The guest's launch line says which kind matched (`exact match` or `source match`).
- A stock wineserver that already has esync built in (it prints `esync: up and running`) never
gets a pack.
## Building and publishing
`.github/workflows/watch-proton-releases.yml` runs every 20 minutes and decides when a build is
due. It asks Steam anonymously (`steamcmd +app_info_print`) which public build of each Valve ARM64
depot app in `flavors.json` is live, and `discover.py watch` compares that, the newest `keep` tags
and releases of every flavor, each flavor's `rev` and its patch series digest with the state of the
last run (`watch-state.json` in the `droiddeck-esync-index` release). A flavor whose builds, rev or
patch series changed, or whose Steam build moved, starts
`.github/workflows/build-droiddeck-esync-packs.yml` for that flavor; nothing else starts a build.
A run that finds no change takes about a minute and builds nothing. If steamcmd fails, the run
watches the tags only and keeps the last Steam state.
`.github/workflows/build-droiddeck-esync-packs.yml` also runs once a day as a safety net, on pushes
to `tools/droiddeck-esync/**` and by hand (flavor, one tag, publish yes/no). Only runs on `main` of
`Droid-Deck/DroidDeck` publish packs; the schedule and pushes do nothing in other repositories,
and manual runs elsewhere only build.
1. `discover.py matrix` lists the newest `keep` builds of each family in `flavors.json` (Valve
tags, GE and CachyOS releases with an arm64 tarball) and drops the ones the published index,
or the `droiddeck-esync-<flavor>` release, already has at the flavor's `rev`, the ones
`revoked.txt` names, and the ones that failed in the last day at the same `rev` and patch
series (a `<key>-r<rev>-<series digest>.failed` asset in the flavor's release). Naming the tag
in a manual run builds it anyway.
2. Each build runs on an arm64 runner with a read-only token, since it runs upstream code: the
wine source is checked out at the tag (GE also runs its `protonprep-valve-staging.sh`),
`build-pack.sh` builds a control and a patched pair inside the SDK image named by the
source's own `Makefile.in` (arm64-llvm), at the flavor's `build_dir` in `flavors.json` (the
path the release itself was built at, which ends up in the binaries), and `make_pack.py` makes
the pack. Patches apply with `--fuzz=0`: a hunk whose context moved fails the build instead of
landing somewhere else. For GE and CachyOS, whose release tarball holds the whole Proton,
`smoke-test.sh` then boots a prefix with that stock Proton and runs `tests/esstress.c` (built
with the SDK's llvm-mingw) under server-side sync as a baseline, and again with the pack under
droiddeck-esync. A pack that fails while stock passes is not published; when stock itself cannot
run on the runner the step only warns.
3. The publish job checks each build's artifact against the matrix (`discover.py packs`) and
uploads it to the `droiddeck-esync-valve`, `droiddeck-esync-ge` or `droiddeck-esync-cachyos`
release as `<id>.tzst` with its index entry `<id>.json`. A build that made no pack gets its
`.failed` marker there (`discover.py failures`), so a patch that does not apply or a stock
wineserver with esync built in is not rebuilt on every run.
4. The index job (environment `droiddeck-esync-signing`) runs only when the run built a pack or
a published pack is not in the index yet. It collects every `<id>.json`, keeps the
highest `rev` per stock pair and version, marks revoked ids, signs `index.json` (ECDSA P-256,
`index-key.pub`) and publishes `bundle-<YYYYMMDD>-<HHMMSS>.tzst` (named after `generated`, so
every signed index has its own), `index.json` and `index.json.sig` to `droiddeck-esync-index`.
`generated` only grows. Packs stay in the index until they are revoked, also when their
entry asset disappears; the previous index counts only when its signature verifies.
A pack is rebuilt only for a new `rev`: bump `rev` of the flavor in `flavors.json` whenever the
patch series or the build changes. Local builds: `BUILD_DIR=<the flavor's build_dir> SYNC_PROFILE=<the
flavor's profile> build-local.sh <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]`,
then `make_pack.py`.
## Setup
- Signing key: create the repository environment `droiddeck-esync-signing` (Settings ->
Environments, limit it to `main`) and store the PEM private key as its secret
`SYNC_INDEX_KEY`: `gh secret set SYNC_INDEX_KEY --env droiddeck-esync-signing < index-key.pem`.
The key never enters the repository; the app trusts only `index-key.pub` (also compiled into
`EsyncPacks.kt`), so a new key needs an app release.
- Optional: `STEAM_DEPOT_USER` and `STEAM_DEPOT_TOKEN` repository secrets for exact Valve packs,
plus the repository variable `DEPOT_DOWNLOADER_SHA256`: the sha256 of
`DepotDownloader-linux-arm64.zip` from the DepotDownloader release the workflow pins
(`DEPOT_DOWNLOADER_TAG`). The depot step is skipped while it is unset.
- APK bundle: after an index run, the job summary prints three lines. Put them in
`tools/droiddeck-esync/release.env`:
```
SYNC_BUNDLE_TAG=droiddeck-esync-index
SYNC_BUNDLE_ASSET=bundle-20261002-061733.tzst
SYNC_BUNDLE_SHA256=<sha256>
```
`build.yml` then unpacks that bundle into `app/src/main/assets/droiddeck-esync/` (gitignored). A
published bundle is never overwritten, so the pinned sha256 stays valid; bump `release.env`
to ship newer packs inside the apk. The app fetches newer packs on its own. A pack the bundle
carries that `revoked.txt` names is left out of the apk, with a warning to bump `release.env`.
## Revocation
Add the pack id as the first word of a line in `revoked.txt` and push to `main`. The workflow
re-signs the index with `"revoked": true` for that id; the app deletes installed copies and
never installs it again, and `discover.py` does not rebuild it at the same `rev`. The apk build
also leaves it out of the bundled packs. To ship a fixed pack for the same Proton build, bump the
flavor's `rev`. Runs that only build (other refs, or manual runs without publish) never queue in
front of a publishing run, so a revocation is signed by the next run on `main`.
-27
View File
@@ -1,27 +0,0 @@
#!/bin/bash
set -euo pipefail
if [ $# -lt 4 ]; then
echo "usage: $0 <sdk-rootfs> <wine-source> <work-dir> <out-dir> [patch-dir]" >&2
echo "env: BUILD_DIR (path the build runs at inside the SDK), SYNC_PROFILE (valve or cachyos), JOBS, SOURCE_DATE_EPOCH" >&2
exit 2
fi
here=$(cd "$(dirname "$0")" && pwd)
sdk=$(realpath "$1")
src=$(realpath "$2")
work=$(realpath -m "$3")
out=$(realpath -m "$4")
patches=${5:+$(realpath "$5")}
build_dir=${BUILD_DIR:-/builds/proton/proton/build-dir}
rm -rf "$work"
mkdir -p "$work" "$out"
"$here/build-pack.sh" prepare "$src" "$work" ${patches:+"$patches"}
bwrap --unshare-user --uid 0 --gid 0 --unshare-ipc --unshare-pid \
--bind "$sdk" / --proc /proc --dev /dev --tmpfs /tmp \
--bind "$work" "$build_dir" --bind "$out" /out --ro-bind "$here/build-pack.sh" /build-pack.sh \
--setenv PATH /opt/llvm-mingw/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
--setenv HOME /tmp --setenv LANG C.UTF-8 --setenv JOBS "${JOBS:-$(nproc)}" \
--setenv SOURCE_DATE_EPOCH "${SOURCE_DATE_EPOCH:-1700000000}" --setenv SYNC_PROFILE "${SYNC_PROFILE:-valve}" \
--chdir "$build_dir" \
/bin/bash /build-pack.sh build "$build_dir" /out
sha256sum "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
-125
View File
@@ -1,125 +0,0 @@
#!/bin/bash
set -euo pipefail
usage() {
echo "usage: $0 prepare <wine-source> <build-dir> [patch-dir]" >&2
echo " $0 build <build-dir> <out-dir>" >&2
echo " $0 all <wine-source> <out-dir> [patch-dir]" >&2
exit 2
}
prepare() {
local origin=$1 b=$2 patches=${3:-}
local s=$b/src-wine
mkdir -p "$s"
rsync --filter=:C --exclude '*~' --exclude .git --exclude compile_commands.json \
--exclude configure --exclude autom4te.cache --exclude include/config.h.in --exclude include/config.h.ink \
--exclude include/wine/vulkan.h --exclude dlls/vulkan-1/vulkan-1.spec \
--exclude dlls/winevulkan/loader_thunks.c --exclude dlls/winevulkan/loader_thunks.h \
--exclude dlls/winevulkan/vulkan_thunks.c --exclude dlls/winevulkan/vulkan_thunks.h \
--exclude dlls/winevulkan/winevulkan.json --exclude dlls/winevulkan/winevulkan.spec \
--exclude dlls/ntdll/ntsyscalls.h --exclude dlls/win32u/win32syscalls.h \
--exclude include/wine/server_protocol.h --exclude server/request_handlers.h --exclude server/request_trace.h \
-Oarx --delete "$origin/" "$s/"
for f in include/wine/server_protocol.h server/request_handlers.h server/request_trace.h; do
cp -a "$origin/$f" "$s/$f"
done
if [ -n "$patches" ]; then
local p found=0
for p in "$patches"/*.patch; do
[ -f "$p" ] || continue
patch -d "$s" -p1 --forward --fuzz=0 --no-backup-if-mismatch -s < "$p"
found=1
done
[ "$found" = 1 ] || { echo "no patches in $patches" >&2; exit 1; }
fi
(
cd "$s"
XDG_CACHE_HOME=$b/cache dlls/winevulkan/make_vulkan -x vk.xml -X video.xml
tools/make_specfiles || true
tools/make_requests
) > "$b/generate.log" 2>&1
touch "$b/.prepared"
}
build() {
local b=$1 out=$2
local s=$b/src-wine o=$b/obj-wine-aarch64 d=$b/dst-wine-aarch64
[ -f "$b/.prepared" ] || { echo "$b is not prepared" >&2; exit 1; }
local jobs=${JOBS:-$(nproc)}
local epoch=$(( ${SOURCE_DATE_EPOCH:-1700000000} - 20 ))
local flags args pe_common i386_flags x86_64_flags conf
case ${SYNC_PROFILE:-valve} in
valve)
flags="-Wno-discarded-qualifiers -march=armv8.2-a -mtune=cortex-x3 -O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
args=(--enable-werror --with-mingw=clang --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
pe_common="-O2 -fwrapv -fno-strict-aliasing -ggdb -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
i386_flags="-Wno-discarded-qualifiers -mstackrealign -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
x86_64_flags="-Wno-discarded-qualifiers -mcmodel=small -march=nocona -mtune=core-avx2 -mfpmath=sse $pe_common"
conf="../$(basename "$s")/configure"
;;
cachyos)
flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -O2 -march=armv8.2-a -mtune=cortex-x3 -fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
args=(--with-mingw=clang --enable-build-id --disable-tests --enable-archs=arm64ec,aarch64,i386,x86_64)
pe_common="-fwrapv -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-omit-frame-pointer -ffile-prefix-map=$s=."
i386_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mstackrealign -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
x86_64_flags="-Wno-discarded-qualifiers -Wno-stringop-overflow -Wno-incompatible-pointer-types -mcmodel=small -O2 -march=nocona -mtune=core-avx2 -pipe -mfpmath=sse $pe_common"
conf="$s/configure"
;;
*)
echo "unknown SYNC_PROFILE ${SYNC_PROFILE}" >&2
exit 2
;;
esac
local env=(STRIP=llvm-strip AR=aarch64-linux-gnu-ar RANLIB=aarch64-linux-gnu-ranlib
CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ LD=aarch64-linux-gnu-ld
RC=aarch64-w64-mingw32-windres WIDL=aarch64-w64-mingw32-widl PKG_CONFIG=aarch64-linux-gnu-pkg-config
"PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig"
"CFLAGS=$flags" "CPPFLAGS=$flags" "CXXFLAGS=-std=c++17 $flags" "LDFLAGS=" "SOURCE_DATE_EPOCH=$epoch"
"CROSSCFLAGS=$flags" "CROSSLDFLAGS="
"aarch64_CFLAGS=$flags" "aarch64_CPPFLAGS=$flags" "aarch64_CXXFLAGS=-std=c++17 $flags" "aarch64_LDFLAGS="
i386_AR=i686-w64-mingw32-ar i386_RANLIB=i686-w64-mingw32-ranlib i386_CC=i686-w64-mingw32-gcc
i386_CXX=i686-w64-mingw32-g++ i386_LD=i686-w64-mingw32-ld
"i386_CFLAGS=$i386_flags" "i386_CPPFLAGS=$i386_flags" "i386_CXXFLAGS=-std=c++17 $i386_flags" "i386_LDFLAGS="
"i386_PKG_CONFIG_LIBDIR=/usr/lib/i386-w64-mingw32/pkgconfig:/usr/share/pkgconfig"
x86_64_AR=x86_64-w64-mingw32-ar x86_64_RANLIB=x86_64-w64-mingw32-ranlib x86_64_CC=x86_64-w64-mingw32-gcc
x86_64_CXX=x86_64-w64-mingw32-g++ x86_64_LD=x86_64-w64-mingw32-ld
"x86_64_CFLAGS=$x86_64_flags" "x86_64_CPPFLAGS=$x86_64_flags" "x86_64_CXXFLAGS=-std=c++17 $x86_64_flags" "x86_64_LDFLAGS="
"x86_64_PKG_CONFIG_LIBDIR=/usr/lib/x86_64-w64-mingw32/pkgconfig:/usr/share/pkgconfig")
export PATH=/opt/llvm-mingw/bin:$PATH
( cd "$s" && autoreconf -fi && rm -rf autom4te.cache ) > "$b/autoreconf.log" 2>&1
rm -rf "$o" "$d"
mkdir -p "$o" "$d"
( cd "$o" && env "${env[@]}" "$conf" -C --prefix="$d" --libdir="$d/lib" --host=aarch64-linux-gnu "${env[@]}" \
"${args[@]}" ) > "$b/configure.log" 2>&1
( cd "$o" && env "${env[@]}" make -j"$jobs" __tooldeps__ ) > "$b/tools.log" 2>&1
( cd "$o" && env "${env[@]}" make -j"$jobs" dlls/ntdll/ntdll.so server/wineserver ) > "$b/make.log" 2>&1
mkdir -p "$out/files/lib/wine/aarch64-unix" "$out/files/bin-arm64"
llvm-objcopy --strip-debug --set-section-flags .text=contents,alloc,load,readonly,code \
"$o/dlls/ntdll/ntdll.so" "$out/files/lib/wine/aarch64-unix/ntdll.so"
llvm-objcopy --strip-debug "$o/server/wineserver" "$out/files/bin-arm64/wineserver"
chmod 0755 "$out/files/lib/wine/aarch64-unix/ntdll.so" "$out/files/bin-arm64/wineserver"
cp "$s/include/wine/server_protocol.h" "$s/server/request_handlers.h" "$s/server/request_trace.h" "$out/"
sed -n 's/^#define SERVER_PROTOCOL_VERSION \([0-9]*\)$/\1/p' "$out/server_protocol.h" > "$out/protocol"
}
[ $# -ge 1 ] || usage
case $1 in
prepare)
[ $# -ge 3 ] || usage
prepare "$(realpath "$2")" "$(realpath -m "$3")" "${4:+$(realpath "$4")}"
;;
build)
[ $# -eq 3 ] || usage
build "$(realpath "$2")" "$(realpath -m "$3")"
;;
all)
[ $# -ge 3 ] || usage
b=${BUILD_DIR:-/builds/proton/proton/build-dir}
prepare "$(realpath "$2")" "$b" "${4:+$(realpath "$4")}"
build "$b" "$(realpath -m "$3")"
;;
*)
usage
;;
esac
-654
View File
@@ -1,654 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import subprocess
import sys
import urllib.parse
from datetime import datetime, timedelta, timezone
from pathlib import Path
from make_index import check_entry
from make_pack import FLAVORS, HERE, PATCHES, REVOKED, SERIES, PackError, patch_digest, read_revoked, sha256_file, slug
FLAVORS_FILE = HERE / "flavors.json"
ARCHIVE = re.compile(r"\.tar\.(gz|xz)$")
ASSIGNMENT = re.compile(r"(?:export\s+|override\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*(?:\?|\+|!|:{1,3})?=\s*(.*)")
REFERENCE = re.compile(r"\$[({]([A-Za-z_][A-Za-z0-9_]*)[)}]")
IMAGE = re.compile(r"[A-Za-z0-9][A-Za-z0-9.-]*(?::[0-9]+)?(?:/[A-Za-z0-9._-]+)+(?::[A-Za-z0-9._-]+)?(?:@sha256:[0-9a-f]{64})?")
REPO = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+")
CLONES = ("shallow", "treeless")
PACK_REV = re.compile(r"-r([0-9]+)$")
ARTIFACT_PREFIX = "droiddeck-esync-pack-"
FAILED = ".failed"
RETRY_AFTER = timedelta(days=1)
VDF_TOKEN = re.compile(r'"((?:[^"\\]|\\.)*)"|([{}])')
APP_HEADER = re.compile(r"^AppID : ([0-9]+),", re.M)
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
def gh(*args: str) -> subprocess.CompletedProcess:
try:
return subprocess.run(["gh", *args], capture_output=True, text=True, check=False)
except FileNotFoundError as e:
raise PackError("the GitHub CLI (gh) is not installed") from e
def json_values(text: str) -> list:
decoder = json.JSONDecoder()
values = []
position = 0
while True:
while position < len(text) and text[position].isspace():
position += 1
if position == len(text):
return values
value, position = decoder.raw_decode(text, position)
values.append(value)
def gh_api(path: str, paginate: bool = False, missing_ok: bool = False):
result = gh("api", *(["--paginate"] if paginate else []), path)
if result.returncode != 0:
if missing_ok and "HTTP 404" in result.stderr:
return None
raise PackError(f"gh api {path}: {result.stderr.strip() or f'exit {result.returncode}'}")
try:
values = json_values(result.stdout)
except ValueError as e:
raise PackError(f"gh api {path}: {e}") from e
if len(values) == 1:
return values[0]
if values and all(isinstance(value, list) for value in values):
return [item for value in values for item in value]
raise PackError(f"gh api {path}: unexpected output")
def quote(value: str) -> str:
return urllib.parse.quote(value, safe="")
def load_flavors(path: Path) -> dict:
try:
data = json.loads(path.read_text())
except (OSError, ValueError) as e:
raise PackError(f"cannot read {path}: {e}") from e
def fail(why: str):
raise PackError(f"{path}: {why}")
def compiles(pattern: str, where: str):
try:
re.compile(pattern)
except re.error as e:
fail(f"{where}: {pattern!r} is not a regular expression: {e}")
if not isinstance(data, dict) or data.get("format") != 1 or not isinstance(data.get("flavors"), dict):
fail("is not a format 1 flavor list")
image = data.get("sdk_image")
if not isinstance(image, dict) or not all(isinstance(image.get(key), str) and image[key] for key in ("file", "variable", "match")):
fail("sdk_image needs file, variable and match")
for name, cfg in data["flavors"].items():
if name not in FLAVORS:
fail(f"unknown flavor {name!r}")
if not isinstance(cfg, dict) or not isinstance(cfg.get("repo"), str) or not REPO.fullmatch(cfg["repo"]):
fail(f"{name}: repo must be owner/name")
for key in ("keep", "rev"):
if type(cfg.get(key)) is not int or cfg[key] < 1:
fail(f"{name}: {key} must be a positive number")
if not isinstance(cfg.get("series"), str) or not SERIES.fullmatch(cfg["series"]):
fail(f"{name}: series is not a patch series name")
if cfg.get("profile") not in ("valve", "cachyos"):
fail(f"{name}: profile must be valve or cachyos")
if type(cfg.get("source_match")) is not bool:
fail(f"{name}: source_match must be a boolean")
build_dir = cfg.get("build_dir")
if not isinstance(build_dir, str) or not re.fullmatch(r"(?:/[A-Za-z0-9._-]+)+", build_dir) or ".." in build_dir.split("/"):
fail(f"{name}: build_dir must be the absolute path the release was built at")
wine = cfg.get("wine")
submodules = wine.get("submodules") if isinstance(wine, dict) else None
if not isinstance(submodules, list) or "wine" not in submodules \
or not all(isinstance(item, str) and re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", item) for item in submodules):
fail(f"{name}: wine.submodules must list the submodules to check out, wine among them")
if wine.get("clone") not in CLONES:
fail(f"{name}: wine.clone must be one of {', '.join(CLONES)}")
prep = wine.get("prep")
if not isinstance(prep, str) or (prep and not re.fullmatch(r"[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*", prep)) or ".." in prep.split("/"):
fail(f"{name}: wine.prep must be a path inside the checkout or empty")
if cfg.get("list") == "tags":
families = cfg.get("families")
if not isinstance(families, dict) or not families:
fail(f"{name}: families must name at least one tag family")
for family, settings in families.items():
if not isinstance(settings, dict) or not isinstance(settings.get("tag"), str) \
or not isinstance(settings.get("prefix"), str) or not re.fullmatch(r"[A-Za-z0-9._-]+", settings["prefix"]) \
or not isinstance(settings.get("depot_app", ""), str):
fail(f"{name}: family {family} needs a tag prefix and a tag pattern")
compiles(settings["tag"], f"{name}: family {family}")
if not isinstance(cfg.get("version_token"), str) or "{tag}" not in cfg["version_token"]:
fail(f"{name}: version_token must contain {{tag}}")
elif cfg.get("list") == "releases":
if not isinstance(cfg.get("tag"), str) or not isinstance(cfg.get("assets"), list) or not cfg["assets"] \
or not all(isinstance(pattern, str) for pattern in cfg["assets"]):
fail(f"{name}: releases need a tag pattern and asset patterns")
for pattern in (cfg["tag"], *cfg["assets"]):
compiles(pattern, name)
else:
fail(f"{name}: list must be tags or releases")
return data
def natural_key(text: str) -> list[tuple]:
return [(0, int(part), "") if part[0] in "0123456789" else (1, 0, part) for part in re.findall(r"[0-9]+|[^0-9]+", text)]
def candidate(flavor: str, cfg: dict, tag: str, asset: str = "", token: str = "", depot_app: str = "") -> dict:
return {
"key": slug(f"{flavor}-{ARCHIVE.sub('', asset) if asset else tag}"),
"flavor": flavor,
"repo": cfg["repo"],
"tag": tag,
"asset": asset,
"token": token,
"depot_app": depot_app,
"rev": cfg["rev"],
"series": cfg["series"],
"profile": cfg["profile"],
"source_match": cfg["source_match"],
"submodules": " ".join(cfg["wine"]["submodules"]),
"clone": cfg["wine"]["clone"],
"prep": cfg["wine"]["prep"],
"build_dir": cfg["build_dir"],
}
def tag_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
found = []
for family, settings in sorted(cfg["families"].items()):
if ref and not ref.startswith(settings["prefix"]):
continue
refs = gh_api(f"repos/{cfg['repo']}/git/matching-refs/tags/{quote(settings['prefix'])}", paginate=True)
if not isinstance(refs, list):
raise PackError(f"{cfg['repo']}: the tag list is not a list")
pattern = re.compile(settings["tag"])
names = set()
for item in refs:
name = item.get("ref", "") if isinstance(item, dict) else ""
if isinstance(name, str) and name.startswith("refs/tags/") and pattern.search(name[len("refs/tags/"):]):
names.add(name[len("refs/tags/"):])
ordered = sorted(names, key=natural_key, reverse=True)
chosen = [name for name in ordered if name == ref] if ref else ordered[:cfg["keep"]]
for name in chosen:
found.append(candidate(flavor, cfg, name, token=cfg["version_token"].format(tag=name), depot_app=settings.get("depot_app", "")))
return found
def pick_assets(assets: list, patterns: list[re.Pattern]) -> list[str]:
chosen = {}
for asset in assets:
name = asset.get("name") if isinstance(asset, dict) else None
if not isinstance(name, str) or not any(pattern.search(name) for pattern in patterns):
continue
stem = ARCHIVE.sub("", name)
if stem not in chosen or name.endswith(".tar.xz"):
chosen[stem] = name
return sorted(chosen.values())
def release_candidates(flavor: str, cfg: dict, ref: str) -> list[dict]:
repo = cfg["repo"]
if ref:
release = gh_api(f"repos/{repo}/releases/tags/{quote(ref)}", missing_ok=True)
releases = [release] if release else []
else:
releases = gh_api(f"repos/{repo}/releases?per_page=100")
if not isinstance(releases, list):
raise PackError(f"{repo}: the release list is not a list")
pattern = re.compile(cfg["tag"])
patterns = [re.compile(item) for item in cfg["assets"]]
usable = []
for release in releases:
tag = release.get("tag_name") if isinstance(release, dict) else None
if not isinstance(tag, str) or release.get("draft") or not pattern.search(tag):
continue
assets = pick_assets(release.get("assets") or [], patterns)
if assets:
usable.append((release.get("published_at") or release.get("created_at") or "", natural_key(tag), tag, assets))
usable.sort(reverse=True)
if not ref:
usable = usable[:cfg["keep"]]
return [candidate(flavor, cfg, tag, asset=asset) for _, _, tag, assets in usable for asset in assets]
def built(index: dict) -> dict[tuple[str, str, str], int]:
revs = {}
packs = index.get("packs") if isinstance(index, dict) else None
for pack in packs if isinstance(packs, list) else []:
if not isinstance(pack, dict) or not isinstance(pack.get("source"), dict):
continue
rev = pack.get("rev") if type(pack.get("rev")) is int else 0
key = (pack.get("flavor"), pack["source"].get("ref"), pack["source"].get("asset", ""))
revs[key] = max(revs.get(key, 0), rev)
return revs
def revoked_build(entry: dict, revoked: set[str]) -> bool:
prefixes = {f"{entry['flavor']}-{slug(name)}-" for name in (entry["token"], entry["tag"]) if name}
suffix = f"-r{entry['rev']}"
return any(ident.endswith(suffix) and ident.startswith(prefix) for ident in revoked for prefix in prefixes)
def chosen_flavors(flavors: dict, selected: str) -> list[str]:
return sorted(flavors["flavors"]) if selected == "all" else [selected]
def failure_marker(entry: dict, digests: dict[str, str] | None = None) -> str:
series = entry["series"]
if not SERIES.fullmatch(series):
raise PackError(f"{series!r} is not a patch series name")
digest = digests.get(series) if digests is not None else None
if digest is None:
digest = patch_digest(PATCHES / series)
if digests is not None:
digests[series] = digest
return f"{entry['key']}-r{entry['rev']}-{digest[:12]}{FAILED}"
def discover(flavors: dict, selected: str, ref: str, index: dict, revoked: set[str],
failed: frozenset[str] = frozenset()) -> list[dict]:
names = chosen_flavors(flavors, selected)
done = built(index)
digests = {}
entries = []
seen = set()
matched = False
for flavor in names:
cfg = flavors["flavors"].get(flavor)
if cfg is None:
raise PackError(f"flavors.json has no {flavor}")
found = tag_candidates(flavor, cfg, ref) if cfg["list"] == "tags" else release_candidates(flavor, cfg, ref)
matched = matched or bool(found)
for entry in found:
if entry["key"] in seen:
continue
seen.add(entry["key"])
label = f"{flavor} {entry['tag']}{' ' + entry['asset'] if entry['asset'] else ''}"
if revoked_build(entry, revoked):
print(f"discover: {label}: revoked at rev {entry['rev']}", file=sys.stderr)
elif done.get((flavor, entry["tag"], entry["asset"]), 0) >= entry["rev"]:
print(f"discover: {label}: in the index at rev {entry['rev']}", file=sys.stderr)
elif not ref and failure_marker(entry, digests) in failed:
print(f"discover: {label}: failed at rev {entry['rev']} with this patch series in the last "
f"{RETRY_AFTER.days} days; skipped (bump rev, change the series or build the tag by name)", file=sys.stderr)
else:
print(f"discover: {label}: to build", file=sys.stderr)
entries.append(entry)
if ref and not matched:
raise PackError(f"no {'' if selected == 'all' else selected + ' '}build is tagged {ref}")
return entries
def vdf(text: str) -> dict:
stack = [{}]
key = None
for match in VDF_TOKEN.finditer(text):
word, brace = match.group(1), match.group(2)
if brace == "{":
if key is None:
raise PackError("app info: a block has no name")
child = {}
stack[-1][key] = child
stack.append(child)
key = None
elif brace == "}":
if len(stack) == 1:
raise PackError("app info: unbalanced braces")
stack.pop()
key = None
elif key is None:
key = word
else:
stack[-1][key] = word
key = None
if len(stack) != 1:
raise PackError("app info: unbalanced braces")
return stack[0]
def steam_builds(text: str, apps: list[str]) -> dict[str, dict]:
found = {}
text = ANSI.sub("", text)
starts = [(m.start(), m.group(1)) for m in APP_HEADER.finditer(text)]
for number, (start, app) in enumerate(starts):
end = starts[number + 1][0] if number + 1 < len(starts) else len(text)
if app not in apps:
continue
body = text[text.find("\n", start) + 1:end]
info = vdf(body).get(app)
depots = info.get("depots") if isinstance(info, dict) else None
public = depots.get("branches", {}).get("public") if isinstance(depots, dict) else None
if not isinstance(public, dict) or not str(public.get("buildid", "")).isdigit():
continue
manifests = sorted(depot["manifests"]["public"]["gid"] for name, depot in depots.items()
if name.isdigit() and isinstance(depot, dict) and isinstance(depot.get("manifests"), dict)
and isinstance(depot["manifests"].get("public"), dict) and depot["manifests"]["public"].get("gid"))
found[app] = {"buildid": public["buildid"], "built": public.get("timebuildupdated", ""), "manifests": manifests}
return found
def steam_apps(flavors: dict) -> list[str]:
return sorted({family["depot_app"] for cfg in flavors["flavors"].values() if cfg["list"] == "tags"
for family in cfg["families"].values() if family.get("depot_app")})
def watch(flavors: dict, previous: dict, steam: dict[str, dict] | None) -> tuple[dict, list[str], list[str]]:
state = {}
changed = []
notes = []
for flavor in sorted(flavors["flavors"]):
cfg = flavors["flavors"][flavor]
found = tag_candidates(flavor, cfg, "") if cfg["list"] == "tags" else release_candidates(flavor, cfg, "")
builds = sorted(f"{entry['tag']} {entry['asset']}".strip() for entry in found)
ships = {}
if cfg["list"] == "tags" and steam is not None:
for family in cfg["families"].values():
app = family.get("depot_app")
if app and app in steam:
ships[app] = steam[app]
old = previous.get(flavor) if isinstance(previous.get(flavor), dict) else {}
if steam is None and isinstance(old.get("steam"), dict):
ships = old["steam"]
mark = {"builds": builds, "rev": cfg["rev"], "series": cfg["series"], "patch": patch_digest(PATCHES / cfg["series"])}
fingerprint = hashlib.sha256(json.dumps(mark, sort_keys=True).encode()).hexdigest()
state[flavor] = {**mark, "fingerprint": fingerprint, "steam": ships}
if old.get("fingerprint") != fingerprint:
changed.append(flavor)
fresh = sorted(set(builds) - set(old.get("builds") or []))
notes.append(f"{flavor}: {', '.join(fresh) if fresh else 'the patch series or rev changed'}")
for app, ship in sorted(ships.items()):
before = (old.get("steam") or {}).get(app) if isinstance(old.get("steam"), dict) else None
if isinstance(before, dict) and before.get("buildid") != ship.get("buildid"):
notes.append(f"{flavor}: Steam app {app} moved from build {before.get('buildid')} to {ship.get('buildid')}")
if flavor not in changed:
changed.append(flavor)
return state, changed, notes
def asset_text(repo: str, asset_id, what: str) -> str:
result = gh("api", "-H", "Accept: application/octet-stream", f"repos/{repo}/releases/assets/{asset_id}")
if result.returncode != 0:
raise PackError(f"cannot download {what}: {result.stderr.strip()}")
return result.stdout
def release_index(repo: str, tag: str) -> dict:
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True)
if release is None:
return {}
for asset in release.get("assets") or []:
if isinstance(asset, dict) and asset.get("name") == "index.json":
text = asset_text(repo, asset["id"], f"the index of {repo}")
try:
return json.loads(text)
except ValueError as e:
raise PackError(f"the index of {repo} is not JSON: {e}") from e
return {}
def published(repo: str, flavors: dict, selected: str, index: dict) -> list[dict]:
packs = index.get("packs") if isinstance(index, dict) else None
listed = {pack.get("id") for pack in packs if isinstance(pack, dict)} if isinstance(packs, list) else set()
found = []
for flavor in chosen_flavors(flavors, selected):
cfg = flavors["flavors"].get(flavor)
tag = f"droiddeck-esync-{flavor}"
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if cfg else None
if not isinstance(release, dict) or type(release.get("id")) is not int:
continue
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
if not isinstance(assets, list):
raise PackError(f"{repo} {tag}: the asset list is not a list")
for asset in assets:
name = asset.get("name") if isinstance(asset, dict) else None
if not isinstance(name, str) or not name.endswith(".json"):
continue
ident = name[:-len(".json")]
rev = PACK_REV.search(ident)
if ident in listed or not rev or int(rev.group(1)) < cfg["rev"]:
continue
try:
entry = json.loads(asset_text(repo, asset.get("id"), f"{tag} {name}"))
except ValueError:
entry = None
if not isinstance(entry, dict) or entry.get("id") != ident or entry.get("flavor") != flavor:
print(f"discover: {tag} {name} is not a pack entry", file=sys.stderr)
continue
print(f"discover: {tag} {name}: published, not in the index yet", file=sys.stderr)
found.append(entry)
return found
def parse_time(value) -> datetime | None:
if not isinstance(value, str):
return None
try:
moment = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)
def recent_failures(repo: str, flavors: dict, selected: str, now: datetime) -> frozenset[str]:
found = set()
for flavor in chosen_flavors(flavors, selected):
tag = f"droiddeck-esync-{flavor}"
release = gh_api(f"repos/{repo}/releases/tags/{quote(tag)}", missing_ok=True) if flavor in flavors["flavors"] else None
if not isinstance(release, dict) or type(release.get("id")) is not int:
continue
assets = gh_api(f"repos/{repo}/releases/{release['id']}/assets?per_page=100", paginate=True)
if not isinstance(assets, list):
raise PackError(f"{repo} {tag}: the asset list is not a list")
for asset in assets:
name = asset.get("name") if isinstance(asset, dict) else None
if not isinstance(name, str) or not name.endswith(FAILED):
continue
moment = parse_time(asset.get("updated_at")) or parse_time(asset.get("created_at"))
if moment is not None and now - moment < RETRY_AFTER:
found.add(name)
return frozenset(found)
def failed_builds(matrix: dict, directory: Path) -> list[tuple[str, str]]:
rows = matrix.get("include") if isinstance(matrix, dict) else None
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
raise PackError("the build matrix has no include list")
made = {row["key"] for row in rows if (directory / f"{ARTIFACT_PREFIX}{row['key']}").is_dir()}
digests = {}
return [(row["flavor"], failure_marker(row, digests)) for row in rows if row["key"] not in made]
def artifact_packs(matrix: dict, directory: Path) -> list[tuple[str, Path, str]]:
rows = matrix.get("include") if isinstance(matrix, dict) else None
if not isinstance(rows, list) or not all(isinstance(row, dict) and isinstance(row.get("key"), str) for row in rows):
raise PackError("the build matrix has no include list")
builds = {row["key"]: row for row in rows}
found = []
for folder in sorted(directory.iterdir()) if directory.is_dir() else []:
row = builds.get(folder.name[len(ARTIFACT_PREFIX):]) if folder.name.startswith(ARTIFACT_PREFIX) else None
if row is None or folder.is_symlink() or not folder.is_dir():
raise PackError(f"{folder.name} is not the artifact of a build in this run")
names = sorted(path.name for path in folder.iterdir())
entries = [name for name in names if name.endswith(".json")]
if len(entries) != 1 or names != sorted([entries[0], entries[0][:-len(".json")] + ".tzst"]):
raise PackError(f"{folder.name} must hold exactly <id>.json and <id>.tzst")
ident = entries[0][:-len(".json")]
entry_path, archive = folder / entries[0], folder / f"{ident}.tzst"
if entry_path.is_symlink() or archive.is_symlink() or not entry_path.is_file() or not archive.is_file():
raise PackError(f"{folder.name} holds links or special files")
try:
data = json.loads(entry_path.read_text())
except ValueError as e:
raise PackError(f"{entry_path}: {e}") from e
entry = check_entry(data, str(entry_path))
source = data.get("source") if isinstance(data.get("source"), dict) else {}
if entry["id"] != ident or entry["flavor"] != row["flavor"] or entry["rev"] != row["rev"] \
or source.get("ref") != row["tag"] or source.get("asset", "") != row["asset"]:
raise PackError(f"{entry_path} is not the pack of {row['flavor']} {row['tag']} at rev {row['rev']}")
if archive.stat().st_size != entry["asset"]["size"] or sha256_file(archive) != entry["asset"]["sha256"]:
raise PackError(f"{archive} is not the asset its entry describes")
found.append((row["flavor"], folder, ident))
return found
def emit(entries: list[dict], unindexed: int = 0) -> None:
matrix = json.dumps({"include": entries}, sort_keys=True, separators=(",", ":"))
path = os.environ.get("GITHUB_OUTPUT")
if path:
with open(path, "a") as stream:
stream.write(f"matrix={matrix}\ncount={len(entries)}\nunindexed={unindexed}\n")
print(matrix)
def make_variables(text: str) -> tuple[dict[str, str], list[tuple[str, str]]]:
variables = {}
assignments = []
for raw in text.splitlines():
match = ASSIGNMENT.fullmatch(raw.split("#", 1)[0].strip())
if match:
name, value = match.group(1), match.group(2).strip()
variables.setdefault(name, value)
assignments.append((name, value))
return variables, assignments
def expand(value: str, variables: dict[str, str], depth: int = 0) -> str:
if depth > 16:
raise PackError("Makefile variables nest too deeply")
def lookup(match: re.Match) -> str:
if match.group(1) not in variables:
raise PackError(f"$({match.group(1)}) is not set in the Makefile")
return expand(variables[match.group(1)], variables, depth + 1)
return REFERENCE.sub(lookup, value)
def sdk_image(text: str, variable: str = "STEAMRT_IMAGE", match: str = "arm64-llvm") -> str:
variables, assignments = make_variables(text)
problem = None
for name, value in assignments:
if not name.startswith(variable):
continue
try:
image = expand(value, variables)
except PackError as e:
problem = problem or e
continue
if match not in image:
continue
if not IMAGE.fullmatch(image):
raise PackError(f"{name} = {image!r} is not an image reference")
return image
raise problem or PackError(f"no {variable} assignment names an {match} image")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Find the Proton builds that have no droiddeck-esync pack yet, as a GitHub Actions matrix.")
parser.add_argument("--flavors", default=str(FLAVORS_FILE))
commands = parser.add_subparsers(dest="command", required=True)
matrix = commands.add_parser("matrix")
matrix.add_argument("--flavor", default="all", choices=("all", *FLAVORS))
matrix.add_argument("--ref", default="")
matrix.add_argument("--index", default="")
matrix.add_argument("--index-release", default="")
matrix.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY", "Droid-Deck/DroidDeck"))
matrix.add_argument("--revoked", default=str(REVOKED))
image = commands.add_parser("sdk-image")
image.add_argument("makefile")
artifacts = commands.add_parser("packs")
artifacts.add_argument("--matrix", required=True)
artifacts.add_argument("directory")
failures = commands.add_parser("failures")
failures.add_argument("--matrix", required=True)
failures.add_argument("directory")
commands.add_parser("steam-apps")
watching = commands.add_parser("watch")
watching.add_argument("--state", default="")
watching.add_argument("--steam-info", default="")
watching.add_argument("--out", required=True)
args = parser.parse_args(argv)
try:
flavors = load_flavors(Path(args.flavors))
if args.command == "sdk-image":
settings = flavors["sdk_image"]
print(sdk_image(Path(args.makefile).read_text(), settings["variable"], settings["match"]))
return 0
if args.command == "packs":
for flavor, folder, ident in artifact_packs(json.loads(args.matrix), Path(args.directory)):
print(f"{flavor}\t{folder}\t{ident}")
return 0
if args.command == "failures":
for flavor, marker in failed_builds(json.loads(args.matrix), Path(args.directory)):
print(f"{flavor}\t{marker}")
return 0
if args.command == "steam-apps":
print(" ".join(steam_apps(flavors)))
return 0
if args.command == "watch":
previous = {}
if args.state and Path(args.state).is_file():
try:
previous = json.loads(Path(args.state).read_text())
except ValueError:
print("discover: the previous watch state is not JSON; every flavor counts as changed", file=sys.stderr)
previous = previous.get("flavors", {}) if isinstance(previous, dict) and previous.get("format") == 1 else {}
steam = None
if args.steam_info:
try:
steam = steam_builds(Path(args.steam_info).read_text(errors="replace"), steam_apps(flavors))
except (OSError, PackError) as e:
print(f"discover: Steam app info unusable ({e}); watching the source tags only", file=sys.stderr)
if steam is not None and len(steam) != len(steam_apps(flavors)):
print(f"discover: Steam app info covers {sorted(steam)} of {steam_apps(flavors)}", file=sys.stderr)
state, changed, notes = watch(flavors, previous, steam)
Path(args.out).write_text(json.dumps({"format": 1, "flavors": state}, indent=1, sort_keys=True) + "\n")
for note in notes:
print(f"discover: {note}", file=sys.stderr)
path = os.environ.get("GITHUB_OUTPUT")
if path:
with open(path, "a") as stream:
stream.write(f"changed={' '.join(changed)}\n")
print(" ".join(changed))
return 0
if args.index and args.index_release:
raise PackError("pass --index or --index-release, not both")
failed = frozenset()
unindexed = 0
if args.index:
path = Path(args.index)
index = json.loads(path.read_text()) if path.exists() else {}
elif args.index_release:
if not REPO.fullmatch(args.repo):
raise PackError(f"{args.repo!r} is not owner/name")
index = release_index(args.repo, args.index_release)
listed = index.get("packs") if isinstance(index, dict) and isinstance(index.get("packs"), list) else []
waiting = published(args.repo, flavors, args.flavor, index)
unindexed = len(waiting)
index = {"packs": [*listed, *waiting]}
failed = recent_failures(args.repo, flavors, args.flavor, datetime.now(timezone.utc))
else:
index = {}
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
emit(discover(flavors, args.flavor, args.ref.strip(), index, revoked, failed), unindexed)
except (PackError, OSError, ValueError) as e:
print(f"discover: {e}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
-84
View File
@@ -1,84 +0,0 @@
{
"format": 1,
"sdk_image": {
"file": "Makefile.in",
"variable": "STEAMRT_IMAGE",
"match": "arm64-llvm"
},
"flavors": {
"valve": {
"repo": "ValveSoftware/Proton",
"list": "tags",
"families": {
"experimental": {
"prefix": "experimental-11.0-",
"tag": "^experimental-11\\.0-\\d{8}[a-z]?$",
"depot_app": "4427310"
},
"proton": {
"prefix": "proton-11.0-",
"tag": "^proton-11\\.0-\\d+[a-z]?$",
"depot_app": "4628740"
}
},
"version_token": "{tag}-arm64",
"keep": 2,
"rev": 1,
"series": "valve-11",
"profile": "valve",
"source_match": true,
"build_dir": "/builds/proton/proton/build-dir",
"wine": {
"submodules": [
"wine"
],
"clone": "shallow",
"prep": ""
}
},
"ge": {
"repo": "GloriousEggroll/proton-ge-custom",
"list": "releases",
"tag": "^GE-Proton11-\\d+$",
"assets": [
"aarch64\\.tar\\.(gz|xz)$",
"arm64\\.tar\\.(gz|xz)$"
],
"keep": 3,
"rev": 1,
"series": "valve-11",
"profile": "valve",
"source_match": false,
"build_dir": "/home/tcrider/build",
"wine": {
"submodules": [
"wine",
"wine-staging"
],
"clone": "treeless",
"prep": "patches/protonprep-valve-staging.sh"
}
},
"cachyos": {
"repo": "CachyOS/proton-cachyos",
"list": "releases",
"tag": "^cachyos-11\\.0-",
"assets": [
"arm64\\.tar\\.(gz|xz)$"
],
"keep": 3,
"rev": 1,
"series": "cachyos-11",
"profile": "cachyos",
"source_match": false,
"build_dir": "/home/runner/work/proton-cachyos/proton-cachyos/build",
"wine": {
"submodules": [
"wine"
],
"clone": "shallow",
"prep": ""
}
}
}
}
-4
View File
@@ -1,4 +0,0 @@
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFyqZQAMh3g3nsEGLU+kBGVPBEkXr
Dv5F3kPtuQ+FV1DkTCGmYUQ1LS9wJtHGt0Ht4wToHGGEp9yzDibYZ4/zhg==
-----END PUBLIC KEY-----
-285
View File
@@ -1,285 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
import re
import subprocess
import sys
import tempfile
import time
from pathlib import Path
from make_pack import (
BINARIES,
EPOCH,
FLAVORS,
HERE,
NTDLL,
PACK_ID,
REVOKED,
SHA256,
WINESERVER,
PackError,
compress,
dump_json,
read_revoked,
sha256_bytes,
sha256_file,
tar_bytes,
write_atomic,
)
PUBKEY = HERE / "index-key.pub"
SCHEMA = 1
INDEX_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
def check_entry(entry, origin: str) -> dict:
def fail(why: str):
raise PackError(f"{origin}: {why}")
if not isinstance(entry, dict):
fail("is not an object")
ident = entry.get("id")
if not isinstance(ident, str) or not PACK_ID.fullmatch(ident):
fail(f"{ident!r} is not a pack id")
if entry.get("flavor") not in FLAVORS:
fail(f"unknown flavor {entry.get('flavor')!r}")
if type(entry.get("rev")) is not int or entry["rev"] < 1:
fail("rev is not a positive number")
version, line = entry.get("version"), entry.get("version_line")
fields = line.split() if isinstance(line, str) else []
if not isinstance(version, str) or len(fields) < 2 or fields[1] != version or not EPOCH.fullmatch(fields[0]):
fail("version and version_line disagree")
for field in ("stock", "files"):
value = entry.get(field)
if not isinstance(value, dict) or set(value) != set(BINARIES) or not all(isinstance(v, str) and SHA256.fullmatch(v) for v in value.values()):
fail(f"{field} must give the sha256 of {' and '.join(BINARIES)}")
if not isinstance(entry.get("exports"), str) or not SHA256.fullmatch(entry["exports"]):
fail("exports is not a sha256")
if type(entry.get("source_match")) is not bool:
fail("source_match is not a boolean")
asset = entry.get("asset")
if not isinstance(asset, dict) or not isinstance(asset.get("sha256"), str) or not SHA256.fullmatch(asset["sha256"]) \
or type(asset.get("size")) is not int or asset["size"] <= 0:
fail("asset needs its sha256 and size")
clean = {key: entry[key] for key in INDEX_KEYS if key in entry}
clean["asset"] = {"sha256": asset["sha256"], "size": asset["size"]}
return clean
def load_entries(directory: Path) -> list[dict]:
if not directory.is_dir():
raise PackError(f"{directory} is not a directory")
entries = []
for path in sorted(directory.glob("*.json")):
try:
data = json.loads(path.read_text())
except ValueError as e:
raise PackError(f"{path}: {e}") from e
entry = check_entry(data, str(path))
if path.stem != entry["id"]:
raise PackError(f"{path} holds the entry of {entry['id']}")
entries.append(entry)
return entries
def load_previous(path: Path) -> tuple[int, list[dict], list]:
try:
data = json.loads(path.read_text())
except (OSError, ValueError) as e:
raise PackError(f"cannot read the previous index {path}: {e}") from e
if not isinstance(data, dict) or data.get("schema") != SCHEMA or type(data.get("generated")) is not int \
or not isinstance(data.get("packs"), list):
raise PackError(f"{path} is not a schema {SCHEMA} index")
packs = []
for number, pack in enumerate(data["packs"]):
entry = check_entry(pack, f"{path} pack {number}")
entry["revoked"] = pack.get("revoked") is True
packs.append(entry)
return data["generated"], packs, data["packs"]
def epoch(entry: dict) -> int:
return int(entry["version_line"].split()[0])
def rank(entry: dict) -> tuple:
return entry["rev"], epoch(entry), entry["id"]
def choose(entries: list[dict], previous: list[dict], revoked: set[str]) -> list[dict]:
known = {entry["id"]: entry for entry in entries}
for entry in previous:
known.setdefault(entry["id"], entry)
active = {}
withdrawn = []
for entry in known.values():
if entry["id"] in revoked:
withdrawn.append(dict(entry, revoked=True))
continue
key = (entry["stock"][NTDLL], entry["stock"][WINESERVER], entry["version"])
if key not in active or rank(entry) > rank(active[key]):
active[key] = dict(entry, revoked=False)
return sorted([*active.values(), *withdrawn], key=lambda entry: entry["id"])
def build_index(packs: list[dict], base_url: str, generated: int) -> dict:
listed = []
for pack in packs:
entry = dict(pack)
entry["asset"] = {
"url": f"{base_url}/droiddeck-esync-{pack['flavor']}/{pack['id']}.tzst",
"sha256": pack["asset"]["sha256"],
"size": pack["asset"]["size"],
}
listed.append(entry)
return {"schema": SCHEMA, "generated": generated, "packs": listed}
def family(entry: dict) -> tuple[str, str, str]:
token = entry["version"]
major = re.search(r"[0-9]+", token)
return entry["flavor"], re.match(r"[A-Za-z]*", token).group(0).lower(), major.group(0) if major else ""
def bundle_choice(packs: list[dict], available: set[str] | None = None) -> list[dict]:
newest = {}
for pack in packs:
if pack["revoked"] or (available is not None and pack["id"] not in available):
continue
key = family(pack)
if key not in newest or (epoch(pack), pack["rev"], pack["id"]) > (epoch(newest[key]), newest[key]["rev"], newest[key]["id"]):
newest[key] = pack
return sorted(newest.values(), key=lambda pack: pack["id"])
def openssl(*args: str) -> subprocess.CompletedProcess:
try:
return subprocess.run(["openssl", *args], capture_output=True, text=True, check=False)
except FileNotFoundError as e:
raise PackError("the openssl command is not installed") from e
def verified(index: Path, signature: Path, pubkey: Path) -> bool:
result = openssl("dgst", "-sha256", "-verify", str(pubkey), "-signature", str(signature), str(index))
return result.returncode == 0 and "Verified OK" in result.stdout
def trusted_previous(path: Path, signature: Path | None, pubkey: Path) -> tuple[int, list[dict], list | None]:
if signature is not None and not (path.is_file() and signature.is_file() and verified(path, signature, pubkey)):
print(f"make_index: the previous index {path} does not verify with {signature} and {pubkey}; it is ignored", file=sys.stderr)
return 0, [], None
return load_previous(path)
def sign(index: Path, signature: Path, key: Path | None, pubkey: Path) -> None:
with tempfile.TemporaryDirectory() as tmp:
if key is None:
pem = os.environ.get("SYNC_INDEX_KEY", "")
if not pem.strip():
raise PackError("no signing key: pass --key or set SYNC_INDEX_KEY")
key = Path(tmp) / "key.pem"
with os.fdopen(os.open(key, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as stream:
stream.write(pem if pem.endswith("\n") else pem + "\n")
result = openssl("dgst", "-sha256", "-sign", str(key), "-out", str(signature), str(index))
if result.returncode != 0:
raise PackError(f"openssl could not sign the index: {result.stderr.strip()}")
if not verified(index, signature, pubkey):
raise PackError(f"the signature does not verify with {pubkey}: the signing key is not the one the app trusts")
def bundle_name(generated: int) -> str:
return f"bundle-{dt.datetime.fromtimestamp(generated, dt.timezone.utc):%Y%m%d-%H%M%S}.tzst"
def bundle(out: Path, packs: list[dict], index: bytes, signature: bytes, packs_dir: Path, generated: int) -> tuple[str, str]:
if not packs_dir.is_dir():
raise PackError(f"{packs_dir} is not a directory")
available = {path.name[:-len(".tzst")] for path in packs_dir.glob("*.tzst") if path.is_file()}
members = {"index.json": (index, 0o644), "index.json.sig": (signature, 0o644)}
for pack in bundle_choice(packs, available):
path = packs_dir / f"{pack['id']}.tzst"
if path.stat().st_size != pack["asset"]["size"] or sha256_file(path) != pack["asset"]["sha256"]:
raise PackError(f"{path} is not the asset the index lists for {pack['id']}")
members[f"packs/{pack['id']}.tzst"] = (path.read_bytes(), 0o644)
name = bundle_name(generated)
archive = compress(tar_bytes(members))
write_atomic(out / name, archive)
return name, sha256_bytes(archive)
def report(values: dict[str, str]) -> None:
path = os.environ.get("GITHUB_OUTPUT")
if path:
with open(path, "a") as stream:
for key, value in values.items():
stream.write(f"{key}={value}\n")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Make, sign and bundle the droiddeck-esync pack index from the published pack entries.")
parser.add_argument("--entries", required=True)
parser.add_argument("--base-url", required=True)
parser.add_argument("--key", default="")
parser.add_argument("--pubkey", default=str(PUBKEY))
parser.add_argument("--out", default="")
parser.add_argument("--bundle-packs-dir", default="")
parser.add_argument("--revoked", default=str(REVOKED))
parser.add_argument("--previous", default="")
parser.add_argument("--previous-sig", default="")
parser.add_argument("--list-bundle", action="store_true")
args = parser.parse_args(argv)
try:
base_url = args.base_url.rstrip("/")
if not base_url.startswith("https://"):
raise PackError("--base-url must be an https URL")
revoked = read_revoked(Path(args.revoked)) if args.revoked else set()
if args.previous_sig and not args.previous:
raise PackError("--previous-sig needs --previous")
previous_signature = Path(args.previous_sig) if args.previous_sig else None
previous_generated, previous, previous_raw = trusted_previous(Path(args.previous), previous_signature, Path(args.pubkey)) \
if args.previous else (0, [], None)
packs = choose(load_entries(Path(args.entries)), previous, revoked)
if args.list_bundle:
for pack in bundle_choice(packs):
print(f"{pack['flavor']}\t{pack['id']}")
return 0
if not args.out:
raise PackError("--out is required")
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
generated = max(int(time.time()), previous_generated + 1)
index = build_index(packs, base_url, generated)
data = dump_json(index)
changed = bool(packs) if previous_raw is None else previous_raw != index["packs"]
temp_index = out / f".index.json.tmp-{os.getpid()}"
temp_signature = out / f".index.json.sig.tmp-{os.getpid()}"
try:
temp_index.write_bytes(data)
sign(temp_index, temp_signature, Path(args.key) if args.key else None, Path(args.pubkey))
signature = temp_signature.read_bytes()
os.replace(temp_index, out / "index.json")
os.replace(temp_signature, out / "index.json.sig")
finally:
temp_index.unlink(missing_ok=True)
temp_signature.unlink(missing_ok=True)
outputs = {"changed": "true" if changed else "false", "generated": str(generated), "index_sha256": sha256_bytes(data)}
live = sum(1 for pack in packs if not pack["revoked"])
print(f"index.json {outputs['index_sha256']} {live} pack(s), {len(packs) - live} revoked, generated {generated}, {'changed' if changed else 'unchanged'}")
if args.bundle_packs_dir:
name, digest = bundle(out, packs, data, signature, Path(args.bundle_packs_dir), generated)
outputs.update(bundle=name, bundle_sha256=digest)
print(f"{name} {digest}")
report(outputs)
except (PackError, OSError) as e:
print(f"make_index: {e}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
-367
View File
@@ -1,367 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import functools
import hashlib
import importlib.machinery
import importlib.util
import io
import json
import os
import re
import subprocess
import sys
import tarfile
from pathlib import Path
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[1]
GUEST = ROOT / "tools/linuxfs/overlay/usr/local/bin/droiddeck-esync"
PATCHES = HERE / "patches"
REVOKED = HERE / "revoked.txt"
NTDLL = "files/lib/wine/aarch64-unix/ntdll.so"
WINESERVER = "files/bin-arm64/wineserver"
BINARIES = (NTDLL, WINESERVER)
COPY = ("files/bin-arm64/wine", "files/lib/wine/aarch64-unix/wine", "files/lib/wine/aarch64-unix/wine-preloader")
HEADERS = ("server_protocol.h", "request_handlers.h", "request_trace.h")
FLAVORS = ("valve", "ge", "cachyos")
SOURCE_KEYS = ("repo", "ref", "commit", "wine_commit", "sdk_image")
ENTRY_KEYS = ("id", "flavor", "rev", "version", "version_line", "stock", "exports", "source_match", "files", "protocol", "source", "patch")
PACK_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}")
SERIES = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")
SHA256 = re.compile(r"[0-9a-f]{64}")
EPOCH = re.compile(r"[0-9]{1,12}")
PROTOCOL = re.compile(rb"^#define SERVER_PROTOCOL_VERSION ([0-9]+)[ \t]*$", re.M)
BUILTIN_ESYNC = b"esync: up and running"
ELF_MAGIC = b"\x7fELF"
GUEST_MODULE = "droiddeck_esync_tools"
class PackError(RuntimeError):
pass
@functools.lru_cache(maxsize=None)
def guest():
loader = importlib.machinery.SourceFileLoader(GUEST_MODULE, str(GUEST))
module = importlib.util.module_from_spec(importlib.util.spec_from_loader(GUEST_MODULE, loader))
sys.modules[GUEST_MODULE] = module
write_bytecode = sys.dont_write_bytecode
sys.dont_write_bytecode = True
try:
loader.exec_module(module)
except Exception as e:
sys.modules.pop(GUEST_MODULE, None)
raise PackError(f"cannot load {GUEST}: {e}") from e
finally:
sys.dont_write_bytecode = write_bytecode
return module
def sha256_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
digest.update(block)
return digest.hexdigest()
def slug(text: str) -> str:
return re.sub(r"[^A-Za-z0-9._-]", "-", text)
def dump_json(value) -> bytes:
return (json.dumps(value, indent=1, sort_keys=True, ensure_ascii=True) + "\n").encode()
def write_atomic(path: Path, data: bytes) -> None:
temp = path.with_name(f".{path.name}.tmp-{os.getpid()}")
try:
with temp.open("wb") as stream:
stream.write(data)
stream.flush()
os.fsync(stream.fileno())
os.replace(temp, path)
except BaseException:
temp.unlink(missing_ok=True)
raise
def read_revoked(path: Path | None) -> set[str]:
if path is None:
return set()
try:
text = path.read_text()
except OSError as e:
raise PackError(f"cannot read the revocation list {path}: {e}") from e
revoked = set()
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
ident = line.split()[0]
if not PACK_ID.fullmatch(ident):
raise PackError(f"{path}: {ident!r} is not a pack id")
revoked.add(ident)
return revoked
def tar_bytes(members: dict[str, tuple[bytes, int]]) -> bytes:
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w", format=tarfile.USTAR_FORMAT) as tar:
for name in sorted(members):
data, mode = members[name]
info = tarfile.TarInfo(name)
info.size = len(data)
info.mode = mode
info.mtime = 0
info.uid = 0
info.gid = 0
info.uname = ""
info.gname = ""
tar.addfile(info, io.BytesIO(data))
return buffer.getvalue()
def zstd(data: bytes, *args: str) -> bytes:
try:
result = subprocess.run(["zstd", "-q", *args], input=data, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
except FileNotFoundError as e:
raise PackError("the zstd command is not installed") from e
if result.returncode != 0:
raise PackError(f"zstd {' '.join(args)} failed: {result.stderr.decode(errors='replace').strip()}")
return result.stdout
def compress(data: bytes) -> bytes:
return zstd(data, "-19", "-T1", "-c")
def unpack(archive: bytes) -> dict[str, tuple[bytes, int]]:
members = {}
try:
with tarfile.open(fileobj=io.BytesIO(zstd(archive, "-d", "-c")), mode="r:") as tar:
for info in tar:
if not info.isreg() or info.name in members:
raise PackError(f"unexpected entry {info.name!r} in the archive")
members[info.name] = (tar.extractfile(info).read(), info.mode)
except tarfile.TarError as e:
raise PackError(f"the archive does not read back: {e}") from e
return members
def binary(base: Path, rel: str) -> Path:
path = base / rel
if path.is_symlink() or not path.is_file():
raise PackError(f"{path} is not a regular file")
with path.open("rb") as stream:
if stream.read(len(ELF_MAGIC)) != ELF_MAGIC:
raise PackError(f"{path} is not an ELF file")
return path
def exports_hash(path: Path) -> str:
try:
value = guest().exports_hash(path)
except PackError:
raise
except Exception as e:
raise PackError(f"{path}: cannot read its exported symbols: {e}") from e
if not isinstance(value, str) or not SHA256.fullmatch(value):
raise PackError(f"{path}: has no exported symbols")
return value
def parse_version_line(line: str) -> tuple[int, str]:
fields = line.split()
if "\n" in line or len(fields) < 2 or not EPOCH.fullmatch(fields[0]):
raise PackError(f"version line {line!r} is not '<build epoch> <version>'")
token = guest().version_token(line)
if not isinstance(token, str) or not token:
raise PackError(f"version line {line!r} has no version token")
return int(fields[0]), token
def parse_source(text: str) -> dict[str, str]:
try:
source = json.loads(text)
except ValueError as e:
raise PackError(f"--source-json is not JSON: {e}") from e
if not isinstance(source, dict) or not all(isinstance(key, str) and isinstance(value, str) for key, value in source.items()):
raise PackError("--source-json must be an object of strings")
missing = [key for key in SOURCE_KEYS if not source.get(key)]
if missing:
raise PackError(f"--source-json lacks {', '.join(missing)}")
return source
def patch_digest(directory: Path) -> str:
patches = sorted(path for path in directory.glob("*.patch") if path.is_file())
if not patches:
raise PackError(f"{directory} holds no *.patch files")
digest = hashlib.sha256()
for path in patches:
digest.update(f"{path.name}\0{sha256_file(path)}\n".encode())
return digest.hexdigest()
def protocol_version(control: Path, patched: Path) -> int:
for name in HEADERS:
before, after = control / name, patched / name
if not before.is_file() or not after.is_file():
raise PackError(f"{name} is missing from the control or the patched build")
if before.read_bytes() != after.read_bytes():
raise PackError(f"the patch changes {name}: the droiddeck-esync patch must not change the wineserver protocol")
found = PROTOCOL.findall((control / HEADERS[0]).read_bytes())
if len(found) != 1:
raise PackError("server_protocol.h does not define SERVER_PROTOCOL_VERSION once")
version = int(found[0])
stamp = control / "protocol"
if stamp.is_file() and stamp.read_text().strip() != str(version):
raise PackError(f"{stamp} disagrees with server_protocol.h ({version})")
return version
def make_pack(
flavor: str,
rev: int,
stock_dir: Path | None,
version_line: str,
control: Path,
patched: Path,
source: dict[str, str],
series: str,
out: Path,
source_match: bool,
patch_dir: Path | None = None,
allow_unreproduced: bool = False,
) -> dict:
if flavor not in FLAVORS:
raise PackError(f"unknown flavor {flavor!r}")
if type(rev) is not int or rev < 1:
raise PackError("--rev must be a positive number")
if not SERIES.fullmatch(series):
raise PackError(f"{series!r} is not a patch series name")
if stock_dir is None and not source_match:
raise PackError("without --stock-dir a pack can only match by source: pass --source-match")
if stock_dir is not None and source_match:
raise PackError("a pack made against stock binaries matches them exactly: drop --source-match")
patch_dir = patch_dir or PATCHES / series
if not patch_dir.is_dir():
raise PackError(f"the patch series {patch_dir} does not exist")
version_line = version_line.strip()
if stock_dir is not None:
stock_line = guest().version_line(stock_dir)
if stock_line and version_line and stock_line != version_line:
raise PackError(f"{stock_dir}/version says {stock_line!r}, not {version_line!r}")
version_line = version_line or stock_line
epoch, token = parse_version_line(version_line)
stock_paths = {rel: binary(stock_dir or control, rel) for rel in BINARIES}
control_paths = {rel: binary(control, rel) for rel in BINARIES}
patched_paths = {rel: binary(patched, rel) for rel in BINARIES}
if BUILTIN_ESYNC in stock_paths[WINESERVER].read_bytes():
raise PackError(f"{stock_paths[WINESERVER]} has esync built in and needs no pack")
stock = {rel: sha256_file(path) for rel, path in stock_paths.items()}
built = {rel: sha256_file(path) for rel, path in control_paths.items()}
files = {rel: sha256_file(path) for rel, path in patched_paths.items()}
for rel in BINARIES:
if files[rel] == built[rel]:
raise PackError(f"the patch left {rel} unchanged")
exports = exports_hash(stock_paths[NTDLL])
control_exports = exports_hash(control_paths[NTDLL])
if exports_hash(patched_paths[NTDLL]) != control_exports:
raise PackError("the patch changes the symbols ntdll.so exports")
if exports != control_exports:
raise PackError("the control build of ntdll.so exports other symbols than the stock one: the wine source does not match this release")
protocol = protocol_version(control, patched)
if stock_dir is not None:
for rel in BINARIES:
state = "reproduces" if stock[rel] == built[rel] else "differs from"
print(f"make_pack: the control build {state} the stock {rel}", file=sys.stderr)
unreproduced = [rel for rel in BINARIES if stock[rel] != built[rel]]
if unreproduced and not allow_unreproduced:
raise PackError(f"the control build does not reproduce the stock {', '.join(unreproduced)}: the patched files would differ "
"from the release in more than the patch (check the build directory and the SDK image, or pass --allow-unreproduced)")
pack_id = f"{flavor}-{slug(token)}-{epoch}-{stock[NTDLL][:12]}-r{rev}"
if not PACK_ID.fullmatch(pack_id):
raise PackError(f"{pack_id!r} is not a valid pack id")
pack = {
"format": 1,
"id": pack_id,
"flavor": flavor,
"rev": rev,
"version": token,
"version_line": version_line,
"stock": stock,
"exports": exports,
"source_match": source_match,
"files": files,
"copy": list(COPY),
"protocol": protocol,
"source": dict(source),
"patch": {"series": series, "sha256": patch_digest(patch_dir)},
}
members = {
"pack.json": (dump_json(pack), 0o644),
NTDLL: (patched_paths[NTDLL].read_bytes(), 0o755),
WINESERVER: (patched_paths[WINESERVER].read_bytes(), 0o755),
}
archive = compress(tar_bytes(members))
if unpack(archive) != members:
raise PackError("the archive does not read back as written")
entry = {key: pack[key] for key in ENTRY_KEYS}
entry["asset"] = {"sha256": sha256_bytes(archive), "size": len(archive)}
entry["revoked"] = False
out.mkdir(parents=True, exist_ok=True)
write_atomic(out / f"{pack_id}.tzst", archive)
write_atomic(out / f"{pack_id}.json", dump_json(entry))
return entry
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Make a droiddeck-esync pack from a control and a patched wine build of one Proton release.")
parser.add_argument("--flavor", required=True, choices=FLAVORS)
parser.add_argument("--rev", required=True, type=int)
parser.add_argument("--stock-dir", default="")
parser.add_argument("--version-line", default="")
parser.add_argument("--control", required=True)
parser.add_argument("--patched", required=True)
parser.add_argument("--source-json", required=True)
parser.add_argument("--patch-series", required=True)
parser.add_argument("--patch-dir", default="")
parser.add_argument("--out", required=True)
parser.add_argument("--source-match", action="store_true")
parser.add_argument("--allow-unreproduced", action="store_true")
args = parser.parse_args(argv)
try:
entry = make_pack(
flavor=args.flavor,
rev=args.rev,
stock_dir=Path(args.stock_dir) if args.stock_dir else None,
version_line=args.version_line,
control=Path(args.control),
patched=Path(args.patched),
source=parse_source(args.source_json),
series=args.patch_series,
out=Path(args.out),
source_match=args.source_match,
patch_dir=Path(args.patch_dir) if args.patch_dir else None,
allow_unreproduced=args.allow_unreproduced,
)
except (PackError, OSError) as e:
print(f"make_pack: {e}", file=sys.stderr)
return 1
print(entry["id"])
return 0
if __name__ == "__main__":
raise SystemExit(main())
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+4
View File
@@ -0,0 +1,4 @@
SYNC_BUNDLE_REPO=Droid-Deck/DroidDeck-Components
SYNC_BUNDLE_TAG=droiddeck-esync-index
SYNC_BUNDLE_ASSET=bundle-20261004-142603.tzst
SYNC_BUNDLE_SHA256=7e311e92444f0709de7779ef5334708511345d60556aa429e6761b86d711a9b2
View File
Whitespace-only changes.
-62
View File
@@ -1,62 +0,0 @@
#!/bin/bash
set -euo pipefail
if [ $# -ne 4 ]; then
echo "usage: $0 <proton-dir> <pack.tzst> <esstress.exe> <work-dir>" >&2
exit 2
fi
tool=$(realpath "$1")
archive=$(realpath "$2")
exe=$(realpath "$3")
work=$(realpath -m "$4")
here=$(cd "$(dirname "$0")" && pwd)
guest=$here/../linuxfs/overlay/usr/local/bin/droiddeck-esync
rm -rf "$work"
mkdir -p "$work/pack"
tar --zstd -xf "$archive" -C "$work/pack"
python3 "$guest" build-shadow "$tool" "$work/pack" "$work/dist" >/dev/null
[ -e "$work/dist/files/bin" ] || ln -s bin-arm64 "$work/dist/files/bin"
export WINEDEBUG=-all WINEDLLOVERRIDES="mscoree,mshtml="
run() {
local mode=$1 bin=$2 want=$3
shift 3
local prefix=$work/prefix-$mode
mkdir -p "$prefix"
(
export WINEPREFIX=$prefix WINESERVER=$bin/wineserver "$@"
timeout "${BOOT_TIMEOUT:-900}" "$bin/wine" wineboot -i > "$work/$mode-boot.log" 2>&1 || true
"$bin/wineserver" -w || true
if ! grep -q "$want" "$work/$mode-boot.log"; then
echo "$mode: wine never said \"$want\""
tail -n 30 "$work/$mode-boot.log"
exit 1
fi
cp "$exe" "$prefix/drive_c/esstress.exe"
status=0
timeout "${STRESS_TIMEOUT:-1800}" "$bin/wine" 'C:\esstress.exe' > "$work/$mode-stress.log" 2>&1 || status=$?
"$bin/wineserver" -k 2>/dev/null || true
if ! grep -q '^esstress:' "$work/$mode-stress.log"; then
echo "$mode: esstress did not finish (status $status)"
tail -n 30 "$work/$mode-stress.log"
exit 1
fi
grep -o '^FAIL [^ ]*:[0-9]*:' "$work/$mode-stress.log" | sort -u > "$work/$mode-failures" || true
echo "$mode: $(grep -E '^esstress:' "$work/$mode-stress.log" | tail -n 1)"
)
}
if ! run stock "$tool/files/bin-arm64" "server-side synchronization" PROTON_NO_NTSYNC=1 WINEESYNC=0 WINEFSYNC=0; then
echo "stock Proton does not run here either; the pack cannot be judged on this machine"
exit 3
fi
run esync "$work/dist/files/bin-arm64" "esync: up and running" PROTON_NO_NTSYNC=1 WINEESYNC=1 WINEFSYNC=0
if [ -s "$work/stock-failures" ]; then
echo "stock fails these as well, so they do not count against the pack: $(tr '\n' ' ' < "$work/stock-failures")"
fi
new=$(comm -13 "$work/stock-failures" "$work/esync-failures")
if [ -n "$new" ]; then
echo "esync: fails checks stock passes:"
grep -F "$new" "$work/esync-stress.log" || echo "$new"
exit 1
fi
-618
View File
@@ -1,618 +0,0 @@
#include <windows.h>
#include <winternl.h>
#include <stdio.h>
#include <stdlib.h>
static int failures;
#define CHECK(c, ...) do { if (!(c)) { failures++; printf("FAIL %s:%d: ", __FILE__, __LINE__); printf(__VA_ARGS__); printf("\n"); fflush(stdout); } } while (0)
static volatile LONG counter;
static HANDLE mutex, sem, ev_auto, ev_manual, start_ev;
static volatile LONG stop;
static DWORD WINAPI mutex_worker(void *arg)
{
int i;
WaitForSingleObject(start_ev, INFINITE);
for (i = 0; i < 20000; i++)
{
DWORD r = WaitForSingleObject(mutex, INFINITE);
LONG v;
if (r != WAIT_OBJECT_0) { CHECK(0, "mutex wait %lu", r); break; }
v = counter;
if (i % 7 == 0) WaitForSingleObject(mutex, 0), ReleaseMutex(mutex);
counter = v + 1;
ReleaseMutex(mutex);
}
return 0;
}
static void test_mutex_contention(void)
{
HANDLE th[8];
int i;
counter = 0;
mutex = CreateMutexA(NULL, FALSE, NULL);
start_ev = CreateEventA(NULL, TRUE, FALSE, NULL);
for (i = 0; i < 8; i++) th[i] = CreateThread(NULL, 0, mutex_worker, NULL, 0, NULL);
SetEvent(start_ev);
WaitForMultipleObjects(8, th, TRUE, INFINITE);
CHECK(counter == 8 * 20000, "mutex counter %ld", counter);
for (i = 0; i < 8; i++) CloseHandle(th[i]);
CloseHandle(mutex);
CloseHandle(start_ev);
printf("mutex_contention ok counter=%ld\n", counter);
}
static volatile LONG produced, consumed;
static DWORD WINAPI sem_producer(void *arg)
{
int i;
for (i = 0; i < 50000; i++)
{
LONG prev;
while (!ReleaseSemaphore(sem, 1, &prev))
{
CHECK(GetLastError() == ERROR_TOO_MANY_POSTS, "release err %lu", GetLastError());
Sleep(0);
}
InterlockedIncrement(&produced);
}
return 0;
}
static DWORD WINAPI sem_consumer(void *arg)
{
for (;;)
{
DWORD r = WaitForSingleObject(sem, 2000);
if (r == WAIT_TIMEOUT) break;
CHECK(r == WAIT_OBJECT_0, "sem wait %lu", r);
InterlockedIncrement(&consumed);
}
return 0;
}
static void test_semaphore(void)
{
HANDLE th[8];
LONG prev;
int i;
produced = consumed = 0;
sem = CreateSemaphoreA(NULL, 0, 16, NULL);
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, sem_producer, NULL, 0, NULL);
for (i = 4; i < 8; i++) th[i] = CreateThread(NULL, 0, sem_consumer, NULL, 0, NULL);
WaitForMultipleObjects(8, th, TRUE, INFINITE);
CHECK(produced == 200000 && consumed == 200000, "sem produced %ld consumed %ld", produced, consumed);
CHECK(!ReleaseSemaphore(sem, 17, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "limit");
CHECK(ReleaseSemaphore(sem, 16, &prev) && prev == 0, "prev %ld", prev);
CHECK(!ReleaseSemaphore(sem, 1, &prev), "over max");
for (i = 0; i < 8; i++) CloseHandle(th[i]);
CloseHandle(sem);
printf("semaphore ok %ld/%ld\n", produced, consumed);
}
static volatile LONG auto_hits;
static DWORD WINAPI auto_waiter(void *arg)
{
while (!stop)
{
DWORD r = WaitForSingleObject(ev_auto, 50);
if (r == WAIT_OBJECT_0) InterlockedIncrement(&auto_hits);
}
return 0;
}
static void test_auto_event(void)
{
HANDLE th[6];
int i;
LONG expected = 0;
stop = 0; auto_hits = 0;
ev_auto = CreateEventA(NULL, FALSE, FALSE, NULL);
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, auto_waiter, NULL, 0, NULL);
for (i = 0; i < 30000; i++)
{
LONG before = auto_hits;
SetEvent(ev_auto);
expected++;
while (auto_hits == before) SwitchToThread();
}
stop = 1;
WaitForMultipleObjects(6, th, TRUE, INFINITE);
CHECK(auto_hits == expected, "auto hits %ld expected %ld", auto_hits, expected);
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto event left signaled");
SetEvent(ev_auto); SetEvent(ev_auto);
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_OBJECT_0, "auto set");
CHECK(WaitForSingleObject(ev_auto, 0) == WAIT_TIMEOUT, "auto coalesce");
for (i = 0; i < 6; i++) CloseHandle(th[i]);
CloseHandle(ev_auto);
printf("auto_event ok hits=%ld\n", auto_hits);
}
static volatile LONG manual_seen;
static DWORD WINAPI manual_waiter(void *arg)
{
DWORD r = WaitForSingleObject(ev_manual, 10000);
if (r == WAIT_OBJECT_0) InterlockedIncrement(&manual_seen);
return 0;
}
static void test_manual_event(void)
{
HANDLE th[16];
int i, round;
ev_manual = CreateEventA(NULL, TRUE, FALSE, NULL);
for (round = 0; round < 50; round++)
{
manual_seen = 0;
ResetEvent(ev_manual);
for (i = 0; i < 16; i++) th[i] = CreateThread(NULL, 0, manual_waiter, NULL, 0, NULL);
Sleep(2);
SetEvent(ev_manual);
WaitForMultipleObjects(16, th, TRUE, INFINITE);
CHECK(manual_seen == 16, "manual round %d seen %ld", round, manual_seen);
for (i = 0; i < 16; i++) CloseHandle(th[i]);
}
for (i = 0; i < 20000; i++)
{
SetEvent(ev_manual);
if (i & 1) ResetEvent(ev_manual);
}
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_TIMEOUT, "manual final reset");
SetEvent(ev_manual);
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays");
CHECK(WaitForSingleObject(ev_manual, 0) == WAIT_OBJECT_0, "manual stays 2");
CloseHandle(ev_manual);
printf("manual_event ok\n");
}
static HANDLE wa_objs[3];
static volatile LONG wa_count;
static DWORD WINAPI wait_all_worker(void *arg)
{
int i;
for (i = 0; i < 3000; i++)
{
DWORD r = WaitForMultipleObjects(3, wa_objs, TRUE, 5000);
if (r != WAIT_OBJECT_0) { CHECK(0, "wait all %lu", r); break; }
InterlockedIncrement(&wa_count);
ReleaseMutex(wa_objs[0]);
ReleaseSemaphore(wa_objs[1], 1, NULL);
SetEvent(wa_objs[2]);
}
return 0;
}
static void test_wait_all(void)
{
HANDLE th[6], dup[2];
int i;
DWORD r;
wa_count = 0;
wa_objs[0] = CreateMutexA(NULL, FALSE, NULL);
wa_objs[1] = CreateSemaphoreA(NULL, 1, 1, NULL);
wa_objs[2] = CreateEventA(NULL, FALSE, TRUE, NULL);
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, wait_all_worker, NULL, 0, NULL);
WaitForMultipleObjects(6, th, TRUE, INFINITE);
CHECK(wa_count == 18000, "wait all count %ld", wa_count);
dup[0] = wa_objs[2];
DuplicateHandle(GetCurrentProcess(), wa_objs[2], GetCurrentProcess(), &dup[1], 0, FALSE, DUPLICATE_SAME_ACCESS);
r = WaitForMultipleObjects(2, dup, TRUE, 0);
CHECK(r == WAIT_FAILED && GetLastError() == ERROR_INVALID_PARAMETER, "dup wait all %lu %lu", r, GetLastError());
CloseHandle(dup[1]);
for (i = 0; i < 6; i++) CloseHandle(th[i]);
for (i = 0; i < 3; i++) CloseHandle(wa_objs[i]);
printf("wait_all ok %ld\n", wa_count);
}
static DWORD WINAPI hold_and_die(void *arg)
{
WaitForSingleObject((HANDLE)arg, INFINITE);
return 0;
}
static void test_abandoned(void)
{
HANDLE m = CreateMutexA(NULL, FALSE, NULL);
HANDLE th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
HANDLE objs[2];
DWORD r;
WaitForSingleObject(th, INFINITE);
r = WaitForSingleObject(m, 1000);
CHECK(r == WAIT_ABANDONED, "abandoned %lu", r);
CHECK(ReleaseMutex(m), "release after abandon");
th = CreateThread(NULL, 0, hold_and_die, m, 0, NULL);
WaitForSingleObject(th, INFINITE);
objs[0] = CreateEventA(NULL, TRUE, TRUE, NULL);
objs[1] = m;
r = WaitForMultipleObjects(2, objs, TRUE, 1000);
CHECK(r == WAIT_ABANDONED_0, "abandoned all %lu", r);
ReleaseMutex(m);
r = WaitForMultipleObjects(2, objs, FALSE, 1000);
CHECK(r == WAIT_OBJECT_0, "any after %lu", r);
CloseHandle(th); CloseHandle(objs[0]); CloseHandle(m);
printf("abandoned ok\n");
}
static volatile LONG apc_ran;
static void CALLBACK apc_proc(ULONG_PTR p) { InterlockedIncrement(&apc_ran); }
static HANDLE apc_ready;
static DWORD WINAPI alert_waiter(void *arg)
{
SetEvent(apc_ready);
return WaitForSingleObjectEx((HANDLE)arg, 5000, TRUE);
}
static void test_apc(void)
{
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL);
HANDLE th;
DWORD r, code;
int i;
apc_ready = CreateEventA(NULL, FALSE, FALSE, NULL);
for (i = 0; i < 200; i++)
{
apc_ran = 0;
th = CreateThread(NULL, 0, alert_waiter, ev, 0, NULL);
WaitForSingleObject(apc_ready, INFINITE);
if (i & 1) Sleep(1);
QueueUserAPC(apc_proc, th, 0);
WaitForSingleObject(th, INFINITE);
GetExitCodeThread(th, &code);
CHECK(code == WAIT_IO_COMPLETION && apc_ran == 1, "apc round %d code %lu ran %ld", i, code, apc_ran);
CloseHandle(th);
}
apc_ran = 0;
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
r = SleepEx(1000, TRUE);
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "sleepex %lu %ld", r, apc_ran);
r = SleepEx(10, TRUE);
CHECK(r == 0, "sleepex timeout %lu", r);
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
r = WaitForSingleObjectEx(ev, 0, FALSE);
CHECK(r == WAIT_TIMEOUT, "non alertable %lu", r);
r = WaitForSingleObjectEx(ev, 0, TRUE);
CHECK(r == WAIT_IO_COMPLETION, "alertable pending %lu", r);
CloseHandle(ev);
printf("apc ok\n");
}
static DWORD WINAPI close_waiter(void *arg)
{
return WaitForSingleObject((HANDLE)arg, 300);
}
static void test_close_while_waiting(void)
{
int i;
for (i = 0; i < 300; i++)
{
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), ev2, th;
DWORD code;
th = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
Sleep(0);
CloseHandle(ev);
ev2 = CreateEventA(NULL, TRUE, TRUE, NULL);
WaitForSingleObject(th, INFINITE);
GetExitCodeThread(th, &code);
CHECK(code == WAIT_TIMEOUT || code == WAIT_FAILED || code == WAIT_OBJECT_0, "close round %d code %lu", i, code);
CHECK(WaitForSingleObject(ev2, 0) == WAIT_OBJECT_0, "reused handle state");
CloseHandle(ev2);
CloseHandle(th);
}
printf("close_while_waiting ok\n");
}
static void test_pulse(void)
{
HANDLE ev = CreateEventA(NULL, TRUE, FALSE, NULL), th[4];
DWORD code;
int i, woke = 0;
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
Sleep(50);
PulseEvent(ev);
WaitForMultipleObjects(4, th, TRUE, INFINITE);
for (i = 0; i < 4; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left signaled");
CHECK(woke == 4, "pulse woke %d of 4", woke);
CloseHandle(ev);
ev = CreateEventA(NULL, FALSE, FALSE, NULL);
PulseEvent(ev);
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse without waiter left signaled");
th[0] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
th[1] = CreateThread(NULL, 0, close_waiter, ev, 0, NULL);
Sleep(50);
PulseEvent(ev);
WaitForMultipleObjects(2, th, TRUE, INFINITE);
woke = 0;
for (i = 0; i < 2; i++) { GetExitCodeThread(th[i], &code); if (code == WAIT_OBJECT_0) woke++; CloseHandle(th[i]); }
CHECK(woke == 1, "auto pulse woke %d of 2", woke);
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "auto pulse left signaled");
CloseHandle(ev);
printf("pulse ok\n");
}
static void test_semantics(void)
{
HANDLE ev = CreateEventA(NULL, TRUE, TRUE, NULL), sem = CreateSemaphoreA(NULL, 2, 3, NULL), h[2];
NTSTATUS (WINAPI *pNtWaitForSingleObject)(HANDLE, BOOLEAN, const LARGE_INTEGER *);
LARGE_INTEGER when;
LONG prev = -1;
DWORD r, start;
int i;
pNtWaitForSingleObject = (void *)GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtWaitForSingleObject");
apc_ran = 0;
QueueUserAPC(apc_proc, GetCurrentThread(), 0);
r = WaitForSingleObjectEx(ev, INFINITE, TRUE);
CHECK(r == WAIT_OBJECT_0 && apc_ran == 0, "object before apc %lu ran %ld", r, apc_ran);
r = SleepEx(0, TRUE);
CHECK(r == WAIT_IO_COMPLETION && apc_ran == 1, "apc after object %lu ran %ld", r, apc_ran);
r = SleepEx(0, TRUE);
CHECK(r == 0, "sleepex zero %lu", r);
ResetEvent(ev);
GetSystemTimeAsFileTime((FILETIME *)&when);
when.QuadPart += 2000000;
start = GetTickCount();
r = pNtWaitForSingleObject(ev, FALSE, &when);
CHECK(r == STATUS_TIMEOUT, "absolute timeout status %lx", r);
CHECK(GetTickCount() - start >= 150 && GetTickCount() - start < 1500, "absolute timeout took %lu", GetTickCount() - start);
CHECK(!ReleaseSemaphore(sem, 2, &prev) && GetLastError() == ERROR_TOO_MANY_POSTS, "semaphore limit");
CHECK(ReleaseSemaphore(sem, 1, &prev) && prev == 2, "semaphore prev %ld", prev);
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(sem, 0) == WAIT_OBJECT_0, "semaphore take %d", i);
CHECK(WaitForSingleObject(sem, 0) == WAIT_TIMEOUT, "semaphore empty");
h[0] = sem; h[1] = ev;
ReleaseSemaphore(sem, 1, NULL);
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_TIMEOUT, "wait all partial");
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 1, "wait all rollback lost token prev %ld", prev);
SetEvent(ev);
CHECK(WaitForMultipleObjects(2, h, TRUE, 50) == WAIT_OBJECT_0, "wait all ready");
CHECK(ReleaseSemaphore(sem, 0, &prev) && prev == 0, "wait all consumed prev %ld", prev);
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "manual stays set");
CloseHandle(sem);
CloseHandle(ev);
printf("semantics ok\n");
}
static HANDLE token_sem;
static volatile LONG token_count;
static DWORD WINAPI token_worker(void *arg)
{
HANDLE h[2] = { token_sem, (HANDLE)arg };
int i;
for (i = 0; i < 20000; i++)
{
DWORD r = (i & 1) ? WaitForSingleObject(token_sem, INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
if (r != WAIT_OBJECT_0) { CHECK(0, "token wait %lu", r); break; }
if (InterlockedIncrement(&token_count) > 3) CHECK(0, "token overcommit");
InterlockedDecrement(&token_count);
if (!(i & 1)) SetEvent((HANDLE)arg);
ReleaseSemaphore(token_sem, 1, NULL);
}
return 0;
}
static void test_semaphore_tokens(void)
{
HANDLE th[6], ev = CreateEventA(NULL, FALSE, TRUE, NULL);
LONG prev = -1;
int i;
token_sem = CreateSemaphoreA(NULL, 3, 3, NULL);
for (i = 0; i < 6; i++) th[i] = CreateThread(NULL, 0, token_worker, ev, 0, NULL);
WaitForMultipleObjects(6, th, TRUE, INFINITE);
for (i = 0; i < 6; i++) CloseHandle(th[i]);
CHECK(ReleaseSemaphore(token_sem, 0, &prev) && prev == 3, "token final count %ld", prev);
for (i = 0; i < 3; i++) CHECK(WaitForSingleObject(token_sem, 0) == WAIT_OBJECT_0, "token drain %d", i);
CHECK(WaitForSingleObject(token_sem, 0) == WAIT_TIMEOUT, "token extra");
CHECK(WaitForSingleObject(ev, 0) == WAIT_OBJECT_0, "token event lost");
CloseHandle(token_sem);
CloseHandle(ev);
printf("semaphore_tokens ok\n");
}
static volatile LONG churn_ready;
static DWORD WINAPI churn_mutex(void *arg)
{
HANDLE h[2] = { (HANDLE)arg, ev_manual };
unsigned int i = 0;
if (WaitForMultipleObjects(2, h, TRUE, INFINITE) <= WAIT_ABANDONED_0 + 1) ReleaseMutex(h[0]);
InterlockedIncrement(&churn_ready);
for (;;)
{
DWORD r = (i++ & 1) ? WaitForSingleObject(h[0], INFINITE) : WaitForMultipleObjects(2, h, TRUE, INFINITE);
if (r == WAIT_OBJECT_0 || r == WAIT_ABANDONED || r == WAIT_ABANDONED_0) ReleaseMutex(h[0]);
}
return 0;
}
static DWORD WINAPI blocked_waiter(void *arg)
{
WaitForSingleObject((HANDLE)arg, 0);
InterlockedIncrement(&churn_ready);
return WaitForSingleObject((HANDLE)arg, INFINITE);
}
static void test_terminate(void)
{
HANDLE th[4], m, ev;
DWORD r, start;
int round, i, stuck = 0;
ev_manual = CreateEventA(NULL, TRUE, TRUE, NULL);
for (round = 0; round < 150; round++)
{
m = CreateMutexA(NULL, FALSE, NULL);
churn_ready = 0;
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, churn_mutex, m, 0, NULL);
while (churn_ready < 4) Sleep(0);
Sleep(round % 4);
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
WaitForMultipleObjects(4, th, TRUE, INFINITE);
r = WaitForSingleObject(m, 3000);
if (r == WAIT_TIMEOUT) stuck++;
else ReleaseMutex(m);
for (i = 0; i < 4; i++) CloseHandle(th[i]);
CloseHandle(m);
}
CHECK(!stuck, "mutex stuck after terminate in %d rounds", stuck);
ev = CreateEventA(NULL, TRUE, FALSE, NULL);
churn_ready = 0;
for (i = 0; i < 4; i++) th[i] = CreateThread(NULL, 0, blocked_waiter, ev, 0, NULL);
while (churn_ready < 4) Sleep(0);
Sleep(50);
for (i = 0; i < 4; i++) TerminateThread(th[i], 7);
WaitForMultipleObjects(4, th, TRUE, INFINITE);
for (i = 0; i < 4; i++) CloseHandle(th[i]);
start = GetTickCount();
for (i = 0; i < 100; i++) PulseEvent(ev);
CHECK(GetTickCount() - start < 400, "pulse after killed waiters took %lu ms", GetTickCount() - start);
CHECK(WaitForSingleObject(ev, 0) == WAIT_TIMEOUT, "pulse left event set");
CloseHandle(ev);
CloseHandle(ev_manual);
printf("terminate ok\n");
}
static void test_cross_process(const char *self)
{
HANDLE ev = CreateEventA(NULL, FALSE, FALSE, "Local\\esstress_ev");
HANDLE done = CreateEventA(NULL, TRUE, FALSE, "Local\\esstress_done");
HANDLE sm = CreateSemaphoreA(NULL, 0, 100000, "Local\\esstress_sem");
PROCESS_INFORMATION pi;
STARTUPINFOA si = { sizeof(si) };
char cmd[MAX_PATH + 16];
DWORD r, code;
int i;
sprintf(cmd, "\"%s\" child", self);
CHECK(CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi), "createprocess %lu", GetLastError());
for (i = 0; i < 5000; i++)
{
SetEvent(ev);
r = WaitForSingleObject(sm, 5000);
if (r != WAIT_OBJECT_0) { CHECK(0, "xproc wait %lu at %d", r, i); break; }
}
SetEvent(done);
r = WaitForSingleObject(pi.hProcess, 10000);
CHECK(r == WAIT_OBJECT_0, "child exit wait %lu", r);
GetExitCodeProcess(pi.hProcess, &code);
CHECK(code == 0, "child code %lu", code);
CloseHandle(pi.hProcess); CloseHandle(pi.hThread);
CloseHandle(ev); CloseHandle(done); CloseHandle(sm);
printf("cross_process ok\n");
}
static int child(void)
{
HANDLE ev = OpenEventA(SYNCHRONIZE | EVENT_MODIFY_STATE, FALSE, "Local\\esstress_ev");
HANDLE done = OpenEventA(SYNCHRONIZE, FALSE, "Local\\esstress_done");
HANDLE sm = OpenSemaphoreA(SEMAPHORE_MODIFY_STATE | SYNCHRONIZE, FALSE, "Local\\esstress_sem");
HANDLE objs[2];
objs[0] = done; objs[1] = ev;
if (!ev || !done || !sm) return 1;
for (;;)
{
DWORD r = WaitForMultipleObjects(2, objs, FALSE, 10000);
if (r == WAIT_OBJECT_0) return 0;
if (r != WAIT_OBJECT_0 + 1) return 2;
ReleaseSemaphore(sm, 1, NULL);
}
}
static void test_kill_holder(const char *self)
{
HANDLE m = CreateMutexA(NULL, FALSE, "Local\\esstress_mutex");
PROCESS_INFORMATION pi;
STARTUPINFOA si = { sizeof(si) };
char cmd[MAX_PATH + 16];
DWORD r;
sprintf(cmd, "\"%s\" holder", self);
CreateProcessA(NULL, cmd, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);
Sleep(500);
TerminateProcess(pi.hProcess, 7);
WaitForSingleObject(pi.hProcess, INFINITE);
r = WaitForSingleObject(m, 3000);
CHECK(r == WAIT_ABANDONED, "killed holder %lu", r);
ReleaseMutex(m);
CloseHandle(pi.hProcess); CloseHandle(pi.hThread); CloseHandle(m);
printf("kill_holder ok\n");
}
static int holder(void)
{
HANDLE m = OpenMutexA(SYNCHRONIZE, FALSE, "Local\\esstress_mutex");
WaitForSingleObject(m, INFINITE);
Sleep(INFINITE);
return 0;
}
static void test_many_objects(void)
{
static HANDLE h[20000];
int i, n = 0;
for (i = 0; i < 20000; i++)
{
if (!(h[i] = CreateEventA(NULL, i & 1, i & 2, NULL))) break;
n++;
}
CHECK(n == 20000, "created %d", n);
for (i = 0; i < n; i++) CHECK(WaitForSingleObject(h[i], 0) == ((i & 2) ? WAIT_OBJECT_0 : WAIT_TIMEOUT), "obj %d", i);
for (i = 0; i < n; i++) CloseHandle(h[i]);
for (i = 0; i < 100000; i++)
{
HANDLE e = CreateEventA(NULL, FALSE, TRUE, NULL);
if (WaitForSingleObject(e, 0) != WAIT_OBJECT_0) { CHECK(0, "churn %d", i); CloseHandle(e); break; }
CloseHandle(e);
}
printf("many_objects ok\n");
}
static void test_timers_and_threads(void)
{
HANDLE t = CreateWaitableTimerA(NULL, FALSE, NULL);
LARGE_INTEGER due;
DWORD start = GetTickCount(), r;
int i;
due.QuadPart = -10 * 1000 * 20;
SetWaitableTimer(t, &due, 10, NULL, NULL, FALSE);
for (i = 0; i < 20; i++)
{
r = WaitForSingleObject(t, 1000);
CHECK(r == WAIT_OBJECT_0, "timer %d %lu", i, r);
}
CHECK(GetTickCount() - start < 2000, "timer slow %lu", GetTickCount() - start);
CancelWaitableTimer(t);
CloseHandle(t);
r = WaitForSingleObject(GetCurrentProcess(), 10);
CHECK(r == WAIT_TIMEOUT, "process self %lu", r);
printf("timers ok\n");
}
int main(int argc, char **argv)
{
if (argc > 1 && !strcmp(argv[1], "child")) return child();
if (argc > 1 && !strcmp(argv[1], "holder")) return holder();
test_mutex_contention();
test_semaphore();
test_auto_event();
test_manual_event();
test_wait_all();
test_abandoned();
test_apc();
test_close_while_waiting();
test_pulse();
test_semantics();
test_semaphore_tokens();
if (argc > 1 && !strcmp(argv[1], "terminate")) test_terminate();
test_cross_process(argv[0]);
test_kill_holder(argv[0]);
test_many_objects();
test_timers_and_threads();
printf("esstress: %d failures\n", failures);
return failures ? 1 : 0;
}