Merge pull request #1401 from Droid-Deck/perf/ntsync-fast-paths

## Summary

droiddeck-ntsync (the userspace stand-in for the Linux ntsync driver,
Wine sync = ntsync) was slower than droiddeck-esync on every cheap
operation. On the tablet a `SetEvent` cost 0.94 µs against esync's 0.21
µs, and a wait on a signaled event 0.45 µs against 0.08 µs. Four things
made it slow:

- **Signal masking:** every operation blocked and restored the signal
mask, so each call made 2 `rt_sigprocmask` syscalls, and a blocking wait
made 4. The masking itself is needed: if Wine's suspend signal stopped a
thread while it held an object lock, a GC thread that suspended it and
then touched the same object would deadlock.
- **Queueing for objects that were already free:** a wait linked itself
onto every object and unlinked again even when the first object could
simply be taken.
- **Wakes nobody needed:** a futex wake syscall went out for every
handoff, even when the waiter was still spinning and would have seen the
change anyway.
- **One shared counter:** every call took and dropped a reference on a
single process-wide device counter, so threads working on unrelated
objects bounced one cache line between cores.

### Changes

- **Signals are deferred instead of masked:** handlers installed by
Wine's `ntdll.so` run behind a small trampoline. If such a signal
arrives inside a critical section (a per-thread depth count), it is
saved with its `siginfo` and sent again with `rt_tgsigqueueinfo`
(falling back to `tgkill`) as soon as the section ends.
  - `sigaction` queries still return Wine's own handler.
- Synchronous signals (SEGV, BUS, ILL, FPE, TRAP, SYS) are never
wrapped, as before.
  - Handlers installed by anything other than `ntdll.so` are left alone.
- **Wait-any fast path:** an object that is signaled and has nobody
queued on it is taken directly, in index order. A wait whose timeout has
already passed returns `ETIMEDOUT` without building a queue.
- **Short spin before sleeping:** a waiter spins for up to 4 µs, bounded
by time rather than by iteration count, before it sleeps on the futex.
- **Wake only sleeping waiters:** the waiter publishes a `sleeping`
flag, and the waker only calls `FUTEX_WAKE` when it is set. Both sides
use seq_cst ordering, so a wake can't be lost. The queue layout changed,
so `NS_VERSION` is now 2.
- **Per-thread device reference:** a thread takes one reference on the
device the first time it uses it and keeps it until it exits or moves to
another device, instead of two atomics on the shared counter per call.
- **More than four signals in one section:** a fifth distinct signal is
held pending with the thread's signal mask, then released once the
section ends. Its handler never runs while an object lock is held.
- **Replay:** deferred signals are replayed with every signal blocked,
so each one is sent exactly once. A held signal is released at the same
signal-nesting level that held it. Handlers installed with
`SA_RESETHAND` are not wrapped, so a one-shot handler keeps its kernel
behaviour.
- **Waits from a handler during another wait's sleep:** they get their
own queue and, for a second device, a reference for that call only. The
outer wait keeps its queue and can't pick up the inner wait's result.

### ntsync becomes the default

- **New installs:** start on ntsync.
- **Existing installs:** move to ntsync once, on the first start of this
version, whatever sync they were on before (a one-time `ntsyncDefaultV1`
marker). Anything picked after that is kept, including esync, fsync and
wineserver.
- **Wording:** the ntsync switch in Performance no longer says
"experimental" or "for testing", in all 33 languages.
- **Unchanged:** per-game sync choices on a game's settings page still
win over the app-wide choice. Proton builds without ntsync still fall
back to droiddeck-esync, then droiddeck-fsync.
- **Tests:** `SyncBackendTest` covers a fresh install, each earlier
choice moving to ntsync once, and a later choice surviving the next
start. All 2,160 unit tests pass.

## Performance (REDMAGIC tablet)

Windows microbenchmark (`syncbench.exe`, x86-64 under FEX), µs per
operation:

| | esync | ntsync before | ntsync after |
|---|---|---|---|
| SetEvent | 0.57 / 0.70 | 0.51 / 0.65 | 0.38 / 0.35 |
| wait on signaled event | 0.13 / 0.13 | 0.44 / 0.43 | 0.27 / 0.20 |
| wait, timeout 0 | 0.27 / 0.25 | 0.52 / 0.51 | 0.19 / 0.18 |
| semaphore release + wait | 0.72 / 0.68 | 0.66 / 0.63 | 0.12 / 0.16 |
| mutex wait + release | 0.77 / 0.73 | 0.66 / 0.64 | 0.12 / 0.12 |
| wait on 8 objects | 0.10 / 0.09 | 0.59 / 0.57 | 0.15 / 0.15 |
| alertable wait | 0.06 / 0.05 | 0.43 / 0.42 | 0.07 / 0.07 |
| event ping-pong, 2 threads | 25.6 / 24.5 | 24.4 / 29.3 | 1.97 / 1.66 |
| 4 threads on one semaphore | 9.4 / 4.6 | 20.3 / 19.1 | 7.3 / 7.6 |

The backend each run actually used was checked from its file
descriptors:
- **esync:** wineserver held eventfds.
- **ntsync:** wineserver held the `droiddeck-ntsync` device and its
timerfd objects.

Native arm64 benchmark against the device directly, µs (before / after):
- set: 0.31–0.36 → 0.06–0.08
- signaled wait: 0.37–0.45 → 0.04–0.08
- semaphore pair: 0.56–0.58 → 0.06–0.07
- mutex pair: 0.56–0.61 → 0.04
- ping-pong: 15–30 → 1.7–2.5

Removing the shared device counter on its own takes 4 threads working on
their own objects from 0.21 to 0.04 µs per operation (x86-64 host).

Games, default Steam launch, The Forest and PEAK at the same scene each
run, 20 s pacing. Columns are esync, ntsync before and ntsync after;
each cell gives two runs.

| | esync | ntsync before | ntsync after |
|---|---|---|---|
| The Forest Low, fps | 64.5 / 61.9 | 70.1 / 74.8 | 72.1 / 72.1 |
| The Forest Low, CPU | 489 / 511% | 434 / 419% | 436 / 426% |
| The Forest High, fps | 58.3 / 57.8 | 72.0 / 69.8 | 72.5 / 69.9 |
| The Forest High, CPU | 506 / 504% | 417 / 429% | 425 / 435% |
| PEAK, fps | 50.4 / 48.6 | — | 47.0 |
| RDR2, launch to signed in / game process | — | — | 29.2 / 42.0 s and
28.3 / 41.1 s |

Two more games, esync against ntsync after, three runs each, default
settings:

| | esync | ntsync after |
|---|---|---|
| Among Us, menu fps (59 cap) | 59.0 / 58.9 / 59.3 | 59.0 / 59.0 / 58.7
|
| Among Us, launch to main menu | 6.65 / 6.84 / 6.85 s | 6.39 / 6.52 /
6.64 s |
| Among Us, CPU | 123 / 132 / 133% | 133 / 133 / 124% |
| Mortal Kombat X attract fight, fps | 56.7 / 54.1 / 56.3 | 61.1 / 56.9
/ 56.2 |
| Mortal Kombat X, 1% low | 20.0 / 30.3 / 30.3 | 30.3 / 30.3 / 30.3 |
| Mortal Kombat X, launch to window | 7.75 / 7.79 / 7.75 s | 7.81 / 7.71
/ 7.63 s |

Mortal Kombat X's CPU moves between 140% and 275% in both modes because
the attract mode changes fighters and stage, so it isn't compared.

The Forest's 1% low was 41.7 fps in every run. PEAK is GPU-bound and
stays in run-to-run noise. Its second ntsync run stopped at the title
menu because the rig's menu input didn't register; the game was still
running at 87 fps, and the same rig miss happened on esync before. RDR2
signs in and reaches the game on the new code.

## Tests

- **`tools/tests/test_ntsync_preload.py` (new):** builds the preload and
checks the emulation against the semantics documented for the kernel
driver:
  - semaphore counts and `EOVERFLOW`;
- mutex owners, recursion, `EPERM`, kill/abandon with `EOWNERDEAD` on
read and on the next wait;
  - auto and manual events, set/reset/pulse previous state;
- wait-any index order, duplicates, the alert object, `EINVAL` limits,
timeouts;
  - wait-all atomicity, duplicates and alert;
- wakes across threads and across processes (inherited fds and fds sent
with `SCM_RIGHTS`);
- 20,000-round ping-pong and 4-thread semaphore stress with infinite
timeouts;
- two signal cases. In one, a handler that re-enters the same object is
fired 2,000 times at a busy thread. In the other, a handler parks the
thread while another thread uses the same object, as Wine's suspend does
under a GC.
- eight real-time signals fired in bursts at a thread busy on one
object, each handler touching that object, so more signals arrive inside
a section than can be deferred;
- a handler that waits on the same device and on a second device while
its thread sleeps in another wait;
- a one-shot (`SA_RESETHAND`) handler installed and fired 300 times at a
busy thread, every one of which must run.
- **The signal tests catch a broken implementation:** with signal
protection removed, the program hangs and the test fails. Before the
last fix, the burst test hung and the outer wait returned the nested
wait's result.
- **The suite passes:** 8 of 8 runs natively on arm64 under proot on the
tablet, and 40 of 40 on the host.
- **On the final build:** The Forest Low ran at 77.7 fps, PEAK at 47.5
fps, and Among Us and two Mortal Kombat X runs played through.
- **The new paths stay out of the way in games:** a build that logged
them saw about 1,400 Wine signals land during ntsync sleeps over two
Mortal Kombat X runs and one The Forest run. None of those handlers
waited on ntsync and no signal was held back, so games run the same code
as before the last fix. The Forest Low ran at 71.6 fps and PEAK at 47.2
fps on the final build.
- **Existing preload tests still pass:** fsync, openat2, usr2 (also with
`BL_SYNC=1`), tracer, dircache, noexec, binfmt, faultreport and esync
packs.

## Credits

- [Linux](https://github.com/torvalds/linux)
`Documentation/userspace-api/ntsync.rst` (GPL-2.0): the object, wait and
error semantics the new test checks. Also added to the README credits.
This commit is contained in:
MaxsTechReview authored and GitHub committed 2026-10-23 15:10:28 -04:00
commit 8f0bd51fd8
39 files changed
+966 -144

No files matched your search

+279 -74
View File
@@ -17,6 +17,7 @@
#include <sys/syscall.h>
#include <sys/timerfd.h>
#include <time.h>
#include <ucontext.h>
#include <unistd.h>
struct ns_sem_args {
@@ -68,12 +69,13 @@ struct ns_wait_args {
#define NS_PROCS 256u
#define NS_FDS 65536
#define NS_DEVS 8
#define NS_VERSION 1u
#define NS_VERSION 2u
#define NS_HEAD_MAGIC 0x31434e5953544e44ull
#define NS_DEV_MAGIC 0x4e54445600000000ull
#define NS_OBJ_MAGIC 0x27ull
#define NS_WAITERS 0x80000000u
#define NS_SWEEP_NS 2000000000ull
#define NS_SPIN_NS 4000ull
#define NS_DEAD 0xffffffffu
#ifndef CLOSE_RANGE_CLOEXEC
@@ -144,6 +146,7 @@ struct ns_q {
uint32_t waiting;
uint32_t inuse;
uint32_t next_free;
uint32_t sleeping;
struct ns_ent ent[NS_ENTS];
};
@@ -168,6 +171,7 @@ struct ns_tls {
struct ns_dev *d;
uint32_t q;
int waiting;
int sleep;
};
static struct ns_dev ns_devs[NS_DEVS];
@@ -255,19 +259,148 @@ static void ns_unlock(uint32_t *w) {
}
}
static void ns_block(sigset_t *old) {
sigset_t s;
sigfillset(&s);
sigdelset(&s, SIGSEGV);
sigdelset(&s, SIGBUS);
sigdelset(&s, SIGILL);
sigdelset(&s, SIGFPE);
sigdelset(&s, SIGTRAP);
pthread_sigmask(SIG_BLOCK, &s, old);
#define NS_DEFER 4
#define NS_LEVELS 8
struct ns_cs {
int depth;
int count;
int level;
unsigned held_levels;
int sigs[NS_DEFER];
siginfo_t info[NS_DEFER];
sigset_t held[NS_LEVELS];
};
static __thread struct ns_cs ns_cs;
static uintptr_t ns_handlers[_NSIG];
static uint8_t ns_siginfo[_NSIG];
int bl_from_ntdll(const void *caller) __attribute__((visibility("hidden")));
static int ns_deferrable(int sig) {
return sig > 0 && sig < _NSIG && sig != SIGKILL && sig != SIGSTOP && sig != SIGSEGV && sig != SIGBUS && sig != SIGILL
&& sig != SIGFPE && sig != SIGTRAP && sig != SIGSYS;
}
static void ns_restore(const sigset_t *old) {
pthread_sigmask(SIG_SETMASK, old, NULL);
static int ns_defer(int sig, const siginfo_t *si) {
struct ns_cs *cs = &ns_cs;
if (sig < SIGRTMIN)
for (int i = 0; i < NS_DEFER; i++)
if (__atomic_load_n(&cs->sigs[i], __ATOMIC_RELAXED) == sig) return 1;
int i = __atomic_fetch_add(&cs->count, 1, __ATOMIC_RELAXED);
if (i >= NS_DEFER) {
__atomic_fetch_sub(&cs->count, 1, __ATOMIC_RELAXED);
return 0;
}
if (si) memcpy(&cs->info[i], si, sizeof(*si));
else memset(&cs->info[i], 0, sizeof(*si));
__atomic_signal_fence(__ATOMIC_SEQ_CST);
__atomic_store_n(&cs->sigs[i], sig, __ATOMIC_RELAXED);
return 1;
}
static int ns_park(int sig, const siginfo_t *si, ucontext_t *uc) {
int level = ns_cs.level;
if (level < 0 || level >= NS_LEVELS) return 0;
sigset_t one;
sigemptyset(&one);
sigaddset(&one, sig);
int saved = errno;
pthread_sigmask(SIG_BLOCK, &one, NULL);
pid_t pid = getpid(), tid = (pid_t)ns_self();
int ok = (si && syscall(SYS_rt_tgsigqueueinfo, pid, tid, sig, si) == 0) || syscall(SYS_tgkill, pid, tid, sig) == 0;
if (!ok) pthread_sigmask(SIG_UNBLOCK, &one, NULL);
errno = saved;
if (!ok) return 0;
sigaddset(&uc->uc_sigmask, sig);
unsigned long *words = (unsigned long *)&ns_cs.held[level];
unsigned bits = 8 * sizeof(unsigned long);
__atomic_fetch_or(&words[(unsigned)(sig - 1) / bits], 1ul << ((unsigned)(sig - 1) % bits), __ATOMIC_RELAXED);
__atomic_fetch_or(&ns_cs.held_levels, 1u << level, __ATOMIC_RELAXED);
return 1;
}
static void ns_trampoline(int sig, siginfo_t *si, void *uc) {
if (__atomic_load_n(&ns_cs.depth, __ATOMIC_RELAXED) > 0 && (ns_defer(sig, si) || (uc && ns_park(sig, si, uc)))) return;
uintptr_t h = __atomic_load_n(&ns_handlers[sig], __ATOMIC_ACQUIRE);
if (!h) return;
ns_cs.level++;
__atomic_signal_fence(__ATOMIC_SEQ_CST);
if (__atomic_load_n(&ns_siginfo[sig], __ATOMIC_RELAXED)) ((void (*)(int, siginfo_t *, void *))h)(sig, si, uc);
else ((void (*)(int))h)(sig);
__atomic_signal_fence(__ATOMIC_SEQ_CST);
ns_cs.level--;
}
static void ns_flush(void) {
struct ns_cs *cs = &ns_cs;
sigset_t all, mask;
sigfillset(&all);
int saved = errno;
pthread_sigmask(SIG_BLOCK, &all, &mask);
int n = cs->count < NS_DEFER ? cs->count : NS_DEFER;
pid_t pid = getpid(), tid = (pid_t)ns_self();
for (int i = 0; i < n; i++)
if (cs->sigs[i] > 0 && syscall(SYS_rt_tgsigqueueinfo, pid, tid, cs->sigs[i], &cs->info[i]) != 0) syscall(SYS_tgkill, pid, tid, cs->sigs[i]);
memset(cs->sigs, 0, sizeof(cs->sigs));
cs->count = 0;
int level = cs->level;
if (level >= 0 && level < NS_LEVELS && (cs->held_levels & (1u << level))) {
for (int sig = 1; sig < _NSIG; sig++)
if (sigismember(&cs->held[level], sig) == 1) sigdelset(&mask, sig);
sigemptyset(&cs->held[level]);
cs->held_levels &= ~(1u << level);
}
pthread_sigmask(SIG_SETMASK, &mask, NULL);
errno = saved;
}
static void ns_block(void) {
ns_cs.depth++;
__atomic_signal_fence(__ATOMIC_SEQ_CST);
}
static void ns_restore(void) {
__atomic_signal_fence(__ATOMIC_SEQ_CST);
if (--ns_cs.depth == 0) {
__atomic_signal_fence(__ATOMIC_SEQ_CST);
if (ns_cs.count || (ns_cs.held_levels && ns_cs.level >= 0 && ns_cs.level < NS_LEVELS && (ns_cs.held_levels & (1u << ns_cs.level))))
ns_flush();
}
}
int sigaction(int sig, const struct sigaction *act, struct sigaction *old) {
static int (*real)(int, const struct sigaction *, struct sigaction *);
if (!real) real = (int (*)(int, const struct sigaction *, struct sigaction *))dlsym(RTLD_NEXT, "sigaction");
if (!real) {
errno = ENOSYS;
return -1;
}
if (!ns_deferrable(sig) || !ns_on()) return real(sig, act, old);
uintptr_t prev = __atomic_load_n(&ns_handlers[sig], __ATOMIC_ACQUIRE);
uint8_t prev_info = __atomic_load_n(&ns_siginfo[sig], __ATOMIC_RELAXED);
struct sigaction mine;
const struct sigaction *use = act;
if (act && act->sa_handler != SIG_DFL && act->sa_handler != SIG_IGN && !(act->sa_flags & SA_RESETHAND) &&
bl_from_ntdll(__builtin_return_address(0))) {
mine = *act;
mine.sa_sigaction = ns_trampoline;
mine.sa_flags |= SA_SIGINFO;
__atomic_store_n(&ns_siginfo[sig], (act->sa_flags & SA_SIGINFO) ? 1 : 0, __ATOMIC_RELAXED);
__atomic_store_n(&ns_handlers[sig], (uintptr_t)act->sa_sigaction, __ATOMIC_RELEASE);
use = &mine;
}
int r = real(sig, use, old);
if (r != 0 && use == &mine) {
__atomic_store_n(&ns_siginfo[sig], prev_info, __ATOMIC_RELAXED);
__atomic_store_n(&ns_handlers[sig], prev, __ATOMIC_RELEASE);
}
if (r == 0 && old && old->sa_sigaction == ns_trampoline) {
old->sa_sigaction = (void (*)(int, siginfo_t *, void *))prev;
if (!prev_info) old->sa_flags &= ~SA_SIGINFO;
}
return r;
}
static inline struct ns_obj *ns_obj(struct ns_head *h, uint32_t s) {
@@ -378,10 +511,11 @@ static int ns_claim(struct ns_q *q, int32_t index) {
if (dead) return 0;
}
int32_t want = -1;
return __atomic_compare_exchange_n(&q->signaled, &want, index, 0, __ATOMIC_ACQ_REL, __ATOMIC_RELAXED);
return __atomic_compare_exchange_n(&q->signaled, &want, index, 0, __ATOMIC_SEQ_CST, __ATOMIC_RELAXED);
}
static void ns_wake(struct ns_q *q) {
if (!__atomic_load_n(&q->sleeping, __ATOMIC_SEQ_CST)) return;
int saved = errno;
ns_futex(&q->signaled, FUTEX_WAKE, 1, NULL, 0);
errno = saved;
@@ -618,14 +752,13 @@ static void ns_dev_put_locked(struct ns_dev *d) {
static void ns_dev_put(struct ns_dev *d) {
if (__atomic_sub_fetch(&d->refs, 1, __ATOMIC_ACQ_REL)) return;
sigset_t old;
int saved = errno;
ns_block(&old);
ns_block();
pthread_mutex_lock(&ns_mx);
uint32_t zero = 0;
if (d->h && __atomic_compare_exchange_n(&d->refs, &zero, NS_DEAD, 0, __ATOMIC_ACQ_REL, __ATOMIC_ACQUIRE)) ns_release(d);
pthread_mutex_unlock(&ns_mx);
ns_restore(&old);
ns_restore();
errno = saved;
}
@@ -682,20 +815,18 @@ static void ns_forget_locked(int fd) {
}
static void ns_forget(int fd) {
sigset_t old;
int saved = errno;
ns_block(&old);
ns_block();
pthread_mutex_lock(&ns_mx);
ns_forget_locked(fd);
pthread_mutex_unlock(&ns_mx);
ns_restore(&old);
ns_restore();
errno = saved;
}
static uint64_t ns_adopt(struct ns_dev *d, uint32_t tag, int fd, uint32_t gen, uint32_t s) {
uint64_t e = 0;
sigset_t old;
ns_block(&old);
ns_block();
pthread_mutex_lock(&ns_mx);
if (!d->h || d->tag != tag || !ns_table() || ns_attach(d) < 0 || s >= NS_SLOTS) goto out;
if ((e = __atomic_load_n(&ns_fdt[fd], __ATOMIC_ACQUIRE))) goto out;
@@ -710,7 +841,7 @@ static uint64_t ns_adopt(struct ns_dev *d, uint32_t tag, int fd, uint32_t gen, u
ns_unlock(&o->lock);
out:
pthread_mutex_unlock(&ns_mx);
ns_restore(&old);
ns_restore();
return e;
}
@@ -719,8 +850,7 @@ static uint64_t ns_note_device(int fd) {
uint64_t e = 0;
if (syscall(SYS_fstat, fd, &st) != 0 || !S_ISREG(st.st_mode)) return 0;
if (((uint64_t)st.st_size & ~0xffffull) != NS_DEV_MAGIC) return 0;
sigset_t old;
ns_block(&old);
ns_block();
pthread_mutex_lock(&ns_mx);
struct ns_dev *d = ns_table() ? ns_map(fd, (uint32_t)st.st_size & 0xffff, (uint64_t)st.st_ino) : NULL;
if (d) {
@@ -732,7 +862,7 @@ static uint64_t ns_note_device(int fd) {
}
}
pthread_mutex_unlock(&ns_mx);
ns_restore(&old);
ns_restore();
return e;
}
@@ -773,15 +903,14 @@ static int ns_ready(struct ns_dev *d) {
static void ns_thread_gone(void *p) {
struct ns_tls *t = p;
if (!t || !t->d || !t->q) return;
sigset_t old;
if (!t || !t->d) return;
struct ns_dev *d = t->d;
ns_block(&old);
ns_free_q(d->h, t->q - 1);
ns_block();
if (t->q) ns_free_q(d->h, t->q - 1);
t->q = 0;
t->waiting = 0;
t->d = NULL;
ns_restore(&old);
ns_restore();
ns_dev_put(d);
}
@@ -789,24 +918,28 @@ static void ns_key_init(void) {
pthread_key_create(&ns_key, ns_thread_gone);
}
static struct ns_q *ns_thread_q(struct ns_dev *d, uint32_t *qi) {
struct ns_head *h = d->h;
if (ns_tq.d && ns_tq.q && ns_tq.waiting) {
ns_drop_q(ns_tq.d->h, ns_tq.q - 1);
ns_tq.waiting = 0;
}
if (ns_tq.d == d && ns_tq.q) {
*qi = ns_tq.q - 1;
return ns_q(h, *qi);
}
if (ns_tq.d && ns_tq.q) {
struct ns_dev *old = ns_tq.d;
ns_free_q(old->h, ns_tq.q - 1);
static int ns_hold(struct ns_dev *d) {
if (ns_tq.d == d) return 1;
if (!ns_dev_get(d)) return 0;
if (ns_tq.sleep) return 2;
struct ns_dev *old = ns_tq.d;
if (old) {
ns_block();
if (ns_tq.q) ns_free_q(old->h, ns_tq.q - 1);
ns_tq.q = 0;
ns_tq.waiting = 0;
ns_tq.d = NULL;
ns_restore();
ns_dev_put(old);
}
pthread_once(&ns_once, ns_key_init);
ns_tq.d = d;
pthread_setspecific(ns_key, &ns_tq);
return 1;
}
static struct ns_q *ns_new_q(struct ns_dev *d, uint32_t *qi) {
struct ns_head *h = d->h;
uint32_t i;
if (!ns_pop(h, &h->q_free, 0, &i) && !ns_grow(&h->q_hwm, NS_QS, &i)) {
ns_sweep_dead(h);
@@ -815,22 +948,35 @@ static struct ns_q *ns_thread_q(struct ns_dev *d, uint32_t *qi) {
struct ns_q *q = ns_q(h, i);
q->proc = (uint32_t)d->proc;
q->waiting = 0;
q->sleeping = 0;
q->inuse = 1;
__atomic_add_fetch(&d->refs, 1, __ATOMIC_ACQ_REL);
ns_tq.d = d;
ns_tq.q = i + 1;
ns_tq.waiting = 0;
pthread_setspecific(ns_key, &ns_tq);
*qi = i;
return q;
}
static struct ns_q *ns_thread_q(struct ns_dev *d, uint32_t *qi) {
struct ns_head *h = d->h;
if (ns_tq.d != d) return NULL;
if (ns_tq.q && ns_tq.waiting) {
ns_drop_q(h, ns_tq.q - 1);
ns_tq.waiting = 0;
}
if (ns_tq.q) {
*qi = ns_tq.q - 1;
return ns_q(h, *qi);
}
struct ns_q *q = ns_new_q(d, qi);
if (!q) return NULL;
ns_tq.q = *qi + 1;
ns_tq.waiting = 0;
return q;
}
static int ns_create(struct ns_dev *d, int type, uint32_t a, uint32_t b) {
struct ns_head *h = d->h;
sigset_t old;
int fd = -1, err = 0;
uint32_t s = 0;
ns_block(&old);
ns_block();
pthread_mutex_lock(&ns_mx);
if (!ns_table() || ns_attach(d) < 0) {
err = ENOMEM;
@@ -879,7 +1025,7 @@ static int ns_create(struct ns_dev *d, int type, uint32_t a, uint32_t b) {
ns_unlock(&o->lock);
out:
pthread_mutex_unlock(&ns_mx);
ns_restore(&old);
ns_restore();
if (fd < 0) errno = err;
return fd;
}
@@ -890,6 +1036,20 @@ static int ns_expired(const struct ns_wait_args *args) {
return (uint64_t)now.tv_sec * 1000000000ull + (uint64_t)now.tv_nsec >= args->timeout;
}
static uint64_t ns_now(void) {
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
return (uint64_t)now.tv_sec * 1000000000ull + (uint64_t)now.tv_nsec;
}
static void ns_spin(struct ns_q *q) {
uint64_t end = ns_now() + NS_SPIN_NS;
for (unsigned i = 1; __atomic_load_n(&q->signaled, __ATOMIC_ACQUIRE) == -1; i++) {
ns_relax();
if (!(i & 63) && ns_now() >= end) return;
}
}
static int ns_sleep(struct ns_q *q, const struct ns_wait_args *args) {
struct timespec ts, *tsp = NULL;
int op = FUTEX_WAIT_BITSET | ((args->flags & NS_WAIT_REALTIME) ? FUTEX_CLOCK_REALTIME : 0);
@@ -898,14 +1058,45 @@ static int ns_sleep(struct ns_q *q, const struct ns_wait_args *args) {
ts.tv_nsec = (long)(args->timeout % 1000000000ull);
tsp = &ts;
}
while (__atomic_load_n(&q->signaled, __ATOMIC_ACQUIRE) == -1) {
if (tsp && ns_expired(args)) return ETIMEDOUT;
if (ns_futex(&q->signaled, op, (uint32_t)-1, tsp, FUTEX_BITSET_MATCH_ANY) == -1) {
if (errno == ETIMEDOUT) return ETIMEDOUT;
if (errno == EINTR) return EINTR;
}
ns_spin(q);
int ret = 0;
__atomic_store_n(&q->sleeping, 1, __ATOMIC_SEQ_CST);
while (!ret && __atomic_load_n(&q->signaled, __ATOMIC_SEQ_CST) == -1) {
if (tsp && ns_expired(args)) ret = ETIMEDOUT;
else if (ns_futex(&q->signaled, op, (uint32_t)-1, tsp, FUTEX_BITSET_MATCH_ANY) == -1 && (errno == ETIMEDOUT || errno == EINTR))
ret = errno;
}
return 0;
__atomic_store_n(&q->sleeping, 0, __ATOMIC_RELAXED);
return ret;
}
static int ns_try_any(struct ns_head *h, const uint32_t *slots, const uint32_t *gens, uint32_t total, uint32_t owner, int *dead) {
for (uint32_t i = 0; i < total; i++) {
struct ns_obj *o = ns_obj(h, slots[i]);
ns_lock(&o->lock);
int busy = o->type == NS_FREE || o->gen != gens[i] || o->any_head || o->all_hint;
if (!busy && ns_signaled(o, owner)) {
switch (o->type) {
case NS_SEM:
o->a--;
break;
case NS_MUTEX:
*dead = (int)o->ownerdead;
o->ownerdead = 0;
o->a++;
o->b = owner;
break;
case NS_EVENT:
if (!o->a) o->b = 0;
break;
}
ns_unlock(&o->lock);
return (int)i;
}
ns_unlock(&o->lock);
if (busy) return -1;
}
return -2;
}
static int ns_wait(struct ns_dev *d, struct ns_wait_args *ua, int all) {
@@ -927,12 +1118,24 @@ static int ns_wait(struct ns_dev *d, struct ns_wait_args *ua, int all) {
for (uint32_t j = 0; j < i; j++)
if (slots[j] == slots[i]) return EINVAL;
}
sigset_t old;
ns_block(&old);
int nested = ns_tq.sleep > 0;
ns_block();
if (!all && (nested || !ns_tq.waiting)) {
int dead = 0, got = ns_try_any(h, slots, gens, total, args.owner, &dead);
if (got >= 0) {
ns_restore();
ua->index = (uint32_t)got;
return dead ? EOWNERDEAD : 0;
}
if (got == -2 && args.timeout != UINT64_MAX && ns_expired(&args)) {
ns_restore();
return ETIMEDOUT;
}
}
uint32_t qi;
struct ns_q *q = ns_ready(d) ? ns_thread_q(d, &qi) : NULL;
struct ns_q *q = !ns_ready(d) ? NULL : nested ? ns_new_q(d, &qi) : ns_thread_q(d, &qi);
if (!q) {
ns_restore(&old);
ns_restore();
return ENOMEM;
}
ns_maybe_sweep(h);
@@ -952,7 +1155,7 @@ static int ns_wait(struct ns_dev *d, struct ns_wait_args *ua, int all) {
}
q->total = total;
q->waiting = 1;
ns_tq.waiting = 1;
if (!nested) ns_tq.waiting = 1;
int bad = 0;
if (!all) {
for (uint32_t i = 0; i < total && !bad; i++) {
@@ -1004,16 +1207,18 @@ static int ns_wait(struct ns_dev *d, struct ns_wait_args *ua, int all) {
}
int ret = 0;
if (!bad && __atomic_load_n(&q->signaled, __ATOMIC_ACQUIRE) == -1) {
sigset_t held;
ns_restore(&old);
ns_tq.sleep++;
ns_restore();
ret = ns_sleep(q, &args);
ns_block(&held);
ns_block();
ns_tq.sleep--;
}
ns_drop_q(h, qi);
ns_tq.waiting = 0;
if (!nested) ns_tq.waiting = 0;
int32_t got = __atomic_load_n(&q->signaled, __ATOMIC_ACQUIRE);
int dead = (int)q->ownerdead;
ns_restore(&old);
if (nested) ns_free_q(h, qi);
ns_restore();
if (bad) return EINVAL;
if (got != -1) {
ua->index = (uint32_t)got;
@@ -1061,12 +1266,11 @@ static int ns_obj_ioctl(struct ns_dev *d, uint64_t e, unsigned long req, void *a
return ENOTTY;
}
if (type != (uint32_t)want) return EINVAL;
sigset_t old;
ns_block(&old);
ns_block();
int all = ns_lock_obj(h, o);
if (o->type != (uint32_t)want || o->gen != gen) {
ns_unlock_obj(h, o, all);
ns_restore(&old);
ns_restore();
return EINVAL;
}
switch (req) {
@@ -1133,7 +1337,7 @@ static int ns_obj_ioctl(struct ns_dev *d, uint64_t e, unsigned long req, void *a
}
}
ns_unlock_obj(h, o, all);
ns_restore(&old);
ns_restore();
if (ret) return ret;
if (req == NS_SEM_RELEASE || req == NS_EVENT_SET || req == NS_EVENT_RESET || req == NS_EVENT_PULSE)
memcpy(arg, &prev, sizeof(prev));
@@ -1176,9 +1380,10 @@ int bl_ntsync_ioctl(int fd, unsigned long req, void *arg, int *rc) {
uint64_t e = ns_lookup(fd);
if (!e) return 0;
struct ns_dev *d = &ns_devs[(e >> 56) & 7];
if (!ns_dev_get(d)) return 0;
int held = ns_hold(d);
if (!held) return 0;
int out = 0, err = (e & NS_E_DEVICE) ? ns_dev_ioctl(d, req, arg, &out) : ns_obj_ioctl(d, e, req, arg);
ns_dev_put(d);
if (held == 2) ns_dev_put(d);
if (err) {
errno = err;
*rc = -1;