Show every VR frame in order at 120 fps: wait for SteamVR's GPU work, present only finished copies, skip the grace on idle shared counters

docs/cardboard-vr.md

@@ -104,7 +104,9 @@ switches the picture to side-by-side and runs SteamVR with three DroidDeck piece
      compositor signals only after the client has submitted and said so, which deadlocked every app
      before its first frame (`helloxr`'s swapchain acquire, DXVK's submit thread in a Proton game).
      A submit's wait on an imported shared value that is neither reached nor queued in that process
-     gets 3 ms for the other process to reach it, then is dropped;
+     gets 3 ms for the other process to reach it, then is dropped. On a counter that has not moved for
+     100 ms it is dropped at once: Angry Birds VR waits on two such values a frame, and the grace cost
+     its submit thread 6 ms a frame and held it at 60 fps;
    - Steam's web helper has the semaphore bridge whenever SteamVR is installed: its GPU process
      outlives SteamVR runs, and one started without the layer crashed when SteamVR showed Steam's
      overlays, until Steam quit;
@@ -148,8 +150,13 @@ a pose written to the shared file turns SteamVR's view. Started from the drawer
   A presses show up in the ring the driver reads.
 - **Layers**: only the first layer of each frame is drawn. SteamVR's compositor composites the
   dashboard into it, so this has not mattered yet.
-- **Timing**: no reprojection, and the presenter blits whatever was submitted last, a guard delay
-  after it arrived. The chain is SteamVR → presenter → DroidDeck compositor → lens pass.
+- **Timing**: no reprojection. The presenter blits the frame submitted last once `vrcompositor`'s GPU
+  work for it is done: the layer fences that process's submits and counts issued and finished ones in
+  `present-sync` beside the `active` marker, and the presenter sleeps on the count (a futex). Without
+  the file it falls back to a guard delay after the frame arrived (`DROIDDECK_VR_PRESENT_GUARD_MS`,
+  4 ms). It hands its own picture to Wayland only once its blit is done, because KGSL does not order
+  the DroidDeck compositor's copy after it. The chain is SteamVR → presenter → DroidDeck compositor →
+  lens pass.
 - **Cross-process GPU sync** is a poll away from exact (see the semaphore note above); binary
   semaphores are not bridged.
 - **Apps and games**: Windows VR games run through Proton ARM64 (FEX for the x86-64 game, DXVK, and

docs/development/cardboard-vr-learnings.md

@@ -44,12 +44,15 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
    - `helloxr` blocked in a wait-idle in `vrclient.so`;
    - DXVK's `dxvk-queue` thread blocked in Proton games.
 
-   The layer now gives a submit's wait on an imported shared value 3 ms, then drops it. It is kept
-   when the value is already reached, or when this process has queued the signal itself. Two details
-   mattered:
+   The layer now gives a submit's wait on an imported shared value 3 ms, then drops it, at once when
+   the shared counter has not moved for 100 ms. It is kept when the value is already reached, or when
+   this process has queued the signal itself. Three details mattered:
    - the waits are judged before the submit's own signals count, because an acquire waits on one
      value and signals the next on the same semaphore;
-   - SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there.
+   - SteamVR's client interop device is Vulkan 1.0, so only the `KHR` entry points resolve there;
+   - a Proton game waits on two values a frame that are never signalled (5,500 dropped, none reached
+     within 200 ms afterwards). With the grace on each, Angry Birds VR's submit thread lost 6 ms a
+     frame and the game ran at 60 fps; without it, 120.
 4. **UBWC layouts must agree.** An imported image has to be created with the same
    `VkImageFormatListCreateInfo` (sRGB + UNORM) as its creator, or Turnip lays out the compression
    differently and the reader sees garbage. The presenter's eye-texture import does this.
@@ -83,6 +86,15 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
 - **Memory.** Steam, SteamVR and one VR game fit in the Thor's 12 GB. A large game such as The Lab, or
   two games at once, push the device into critical pressure, and the low-memory killer takes
   DroidDeck itself (`ended-without-teardown.txt` in the session logs, `lowmemorykiller` in logcat).
+- **Frames shown out of order.** Video from the glasses jumped back a few frames in bursts while the
+  head turned. Two causes, each found by stamping a frame counter into the presenter's output:
+  - the presenter copied SteamVR's eye texture a fixed 4 ms after `Present`, with no fence. Under
+    GPU load `vrcompositor` had not drawn it yet, so the copy got that texture's previous frame. The
+    layer now fences `vrcompositor`'s submits and the presenter waits for the frame's count;
+  - DroidDeck's compositor copies a Wayland client's dma-buf without waiting for its implicit fence,
+    and KGSL does not order the copy after the client's GPU work. The presenter's blit sat behind the
+    game's frame, so the compositor showed the swapchain image's older picture. The presenter now
+    waits for its blit before it presents.
 - **Head-tracking axes.** With the display at `ROTATION_90`, screen x is minus device y and screen y is
   device x. Getting this wrong made a turn at the hips tilt the view. Recentring now keeps the world
   vertical and resets the heading only.
@@ -123,8 +135,9 @@ OpenVR to SteamVR's ARM64 `vrclient.so`, so no x86 SteamVR is involved.
 - **SteamVR Home** needs `IVROverlayView_002` in Proton's `vrclient`, or an ARM64 build of Home.
 - **The Lab and other large games** need memory headroom. Candidates: stop Steam's own web helpers
   while a VR game runs, or lower SteamVR's footprint.
-- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync. A submit thread in the
-  layer, deferring submits until their waits are reachable, would be exact.
+- **Wait drops are a heuristic.** A 3 ms grace keeps the common case in sync, and an idle counter
+  skips it. A submit thread in the layer, deferring submits until their waits are reachable, would be
+  exact.
 - **Layers:** only the first layer of each frame is presented. No reprojection.
 - **Controllers** are a gaze pointer driven by the pad. Pad presses did not close the SteamVR menu in
   testing. Untested: whether that came from gamescope focus being on Steam, or from the system button

tools/vr/src/present_sync.h

@@ -0,0 +1,10 @@
+#pragma once
+#include <stdint.h>
+
+#define DROIDDECK_PRESENT_SYNC_NAME "present-sync"
+#define DROIDDECK_PRESENT_SYNC_SIZE 4096
+
+struct droiddeck_present_sync {
+    uint32_t issued;
+    uint32_t done;
+};

tools/vr/src/presenter.cpp

@@ -10,9 +10,12 @@ typedef unsigned long VisualID;
 #include "xdg-shell-client-protocol.h"
 
 #include "presenter.h"
+#include "present_sync.h"
 
+#include <algorithm>
 #include <atomic>
 #include <chrono>
+#include <climits>
 #include <condition_variable>
 #include <cstdarg>
 #include <cstdio>
@@ -20,7 +23,11 @@ typedef unsigned long VisualID;
 #include <cstring>
 #include <dlfcn.h>
 #include <fcntl.h>
+#include <linux/futex.h>
 #include <mutex>
+#include <string>
+#include <sys/mman.h>
+#include <sys/syscall.h>
 #include <thread>
 #include <unistd.h>
 #include <vector>
@@ -141,6 +148,11 @@ public:
         std::lock_guard<std::mutex> lock(mutex_);
         frame_ = f;
         frameAt_ = std::chrono::steady_clock::now();
+        const droiddeck_present_sync *sync = MapSync(frameAt_);
+        const uint32_t issued = sync ? __atomic_load_n(&sync->issued, __ATOMIC_ACQUIRE) : 0;
+        frameTracked_ = sync && issued != lastIssued_;
+        frameMark_ = issued;
+        lastIssued_ = issued;
         haveFrame_ = true;
         submitted_++;
         cv_.notify_one();
@@ -176,11 +188,15 @@ private:
             Log("presenter: unavailable, frames are not shown");
             return;
         }
-        uint64_t shown = 0;
+        if (const char *g = getenv("DROIDDECK_VR_PRESENT_GUARD_MS")) guard_ = std::chrono::microseconds((long)(atof(g) * 1000));
+        uint64_t shown = 0, waited = 0, late = 0;
+        std::chrono::steady_clock::duration longest{};
         auto lastLog = std::chrono::steady_clock::now();
         while (true) {
             EyeFrame f;
             std::chrono::steady_clock::time_point at;
+            uint32_t mark = 0;
+            bool tracked = false;
             std::vector<PendingImport> imports;
             std::vector<uint64_t> removals;
             {
@@ -191,6 +207,8 @@ private:
                 removals.swap(removals_);
                 f = frame_;
                 at = frameAt_;
+                mark = frameMark_;
+                tracked = frameTracked_;
                 bool draw = haveFrame_;
                 haveFrame_ = false;
                 if (!draw) f.texture[0] = f.texture[1] = 0;
@@ -201,13 +219,26 @@ private:
             // SteamVR hands the textures over at Present with no fence the driver can wait on: give the
             // compositor's GPU work a moment to land before copying, as ALVR does (a torn or half-drawn
             // frame otherwise).
-            if (f.texture[0]) std::this_thread::sleep_until(at + guard_);
+            if (f.texture[0] && tracked) {
+                const auto start = std::chrono::steady_clock::now();
+                const int r = WaitDrawn(mark, start + kDrawnTimeout);
+                if (r > 0) {
+                    waited++;
+                    longest = std::max(longest, std::chrono::steady_clock::now() - start);
+                } else if (r < 0) {
+                    late++;
+                }
+            } else if (f.texture[0]) {
+                std::this_thread::sleep_until(at + guard_);
+            }
             if (f.texture[0] && Draw(f)) shown++;
             auto now = std::chrono::steady_clock::now();
             if (now - lastLog > std::chrono::seconds(10)) {
-                Log("presenter: %llu frames shown, %llu submitted (last 10 s)", (unsigned long long)shown,
-                    (unsigned long long)submitted_.exchange(0));
-                shown = 0;
+                Log("presenter: %llu frames shown, %llu submitted, %llu waited for the GPU (longest %.1f ms), %llu late (last 10 s)",
+                    (unsigned long long)shown, (unsigned long long)submitted_.exchange(0), (unsigned long long)waited,
+                    std::chrono::duration<double, std::milli>(longest).count(), (unsigned long long)late);
+                shown = waited = late = 0;
+                longest = {};
                 lastLog = now;
             }
         }
@@ -215,6 +246,38 @@ private:
     }
 
     std::chrono::microseconds guard_{4000};
+    static constexpr std::chrono::milliseconds kDrawnTimeout{100};
+
+    const droiddeck_present_sync *MapSync(std::chrono::steady_clock::time_point now) {
+        if (sync_) return sync_;
+        if (syncTried_.time_since_epoch().count() && now - syncTried_ < std::chrono::seconds(1)) return nullptr;
+        syncTried_ = now;
+        const char *rt = getenv("XDG_RUNTIME_DIR");
+        std::string path = std::string(rt && *rt ? rt : "/run/droiddeck") + "/droiddeck-steamvr/" DROIDDECK_PRESENT_SYNC_NAME;
+        int fd = open(path.c_str(), O_RDONLY | O_CLOEXEC);
+        if (fd < 0) return nullptr;
+        void *m = mmap(nullptr, DROIDDECK_PRESENT_SYNC_SIZE, PROT_READ, MAP_SHARED, fd, 0);
+        close(fd);
+        if (m == MAP_FAILED) return nullptr;
+        sync_ = (const droiddeck_present_sync *)m;
+        Log("presenter: frames wait for the compositor's GPU work (%s)", path.c_str());
+        return sync_;
+    }
+
+    int WaitDrawn(uint32_t mark, std::chrono::steady_clock::time_point until) {
+        const droiddeck_present_sync *s = sync_;
+        int slept = 0;
+        while (true) {
+            const uint32_t done = __atomic_load_n(&s->done, __ATOMIC_ACQUIRE);
+            if ((int32_t)(done - mark) >= 0) return slept;
+            const auto left = until - std::chrono::steady_clock::now();
+            if (left <= std::chrono::steady_clock::duration::zero()) return -1;
+            const auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(left).count();
+            struct timespec ts = {(time_t)(ns / 1000000000), (long)(ns % 1000000000)};
+            syscall(SYS_futex, const_cast<uint32_t *>(&s->done), FUTEX_WAIT, done, &ts, nullptr, 0);
+            slept = 1;
+        }
+    }
 
     // ------------------------------------------------------------ Wayland: straight to DroidDeck's compositor
     // The compositor the app runs (wayland-0 in the session's runtime dir, beside gamescope's socket)
@@ -289,7 +352,6 @@ private:
     }
 
     bool InitWindow() {
-        if (const char *g = getenv("DROIDDECK_VR_PRESENT_GUARD_MS")) guard_ = std::chrono::microseconds((long)(atof(g) * 1000));
         if (!x_.Load()) { Log("presenter: no libX11"); return false; }
         // gamescope's first Xwayland (:0) is where its overlays (mangoapp, Steam's) live; SteamVR's
         // processes inherit the games' display (:1), so it is not taken from DISPLAY.
@@ -556,6 +618,10 @@ private:
         VkPipelineStageFlags wait = VK_PIPELINE_STAGE_TRANSFER_BIT;
         VkSubmitInfo si = {VK_STRUCTURE_TYPE_SUBMIT_INFO, nullptr, 1, &acquired_, &wait, 1, &cmd_, 1, &rendered_};
         if ((r = vk_.QueueSubmit(queue_, 1, &si, fence_)) != VK_SUCCESS) { Log("presenter: submit %d", r); return false; }
+        if (vk_.WaitForFences(dev_, 1, &fence_, VK_TRUE, (uint64_t)std::chrono::nanoseconds(kDrawnTimeout).count()) == VK_TIMEOUT) {
+            static int said;
+            if (said++ < 3) Log("presenter: the copy took over %lld ms; showing it anyway", (long long)kDrawnTimeout.count());
+        }
         VkPresentInfoKHR pi = {VK_STRUCTURE_TYPE_PRESENT_INFO_KHR, nullptr, 1, &rendered_, 1, &swapchain_, &idx, nullptr};
         r = vk_.QueuePresentKHR(queue_, &pi);
         if (r != VK_SUCCESS && r != VK_SUBOPTIMAL_KHR) {
@@ -571,6 +637,10 @@ private:
     std::vector<uint64_t> removals_;
     EyeFrame frame_{};
     std::chrono::steady_clock::time_point frameAt_{};
+    uint32_t frameMark_ = 0, lastIssued_ = 0;
+    bool frameTracked_ = false;
+    const droiddeck_present_sync *sync_ = nullptr;
+    std::chrono::steady_clock::time_point syncTried_{};
     bool haveFrame_ = false, exit_ = false, started_ = false;
     std::atomic<uint64_t> submitted_{0};
     std::thread thread_;

tools/vr/src/vk_layer_droiddeck_vr.c

@@ -295,11 +295,13 @@ static VkResult VKAPI_CALL CreateDevice(VkPhysicalDevice pd, const VkDeviceCreat
 }
 
 static void forget_device_semaphores(VkDevice dev);
+static void track_forget_device(VkDevice dev);
 
 static void VKAPI_CALL DestroyDevice(VkDevice device, const VkAllocationCallbacks *alloc) {
     struct dev *d = dev_for(device);
     if (!d) return;
     forget_device_semaphores(device);
+    track_forget_device(device);
     PFN_vkDestroyDevice f = d->destroy;
     pthread_mutex_lock(&g_lock);
     memset(d, 0, sizeof(*d));
@@ -588,6 +590,7 @@ struct bridged {
     int timeline; /* 0: binary, bridged only by dropping its waits */
     unsigned long ino; /* the memfd's inode, which names the semaphore across processes in traces */
     uint64_t submitted; /* the highest value this process has queued a signal of */
+    uint64_t seen, seen_us;
 };
 static pthread_mutex_t g_br_lock = PTHREAD_MUTEX_INITIALIZER;
 static struct bridged g_br[MAX_SEMS];
@@ -634,6 +637,8 @@ static int sem_trace_on(void) {
     return on;
 }
 
+static uint64_t now_us(void);
+
 static void *poller(void *arg) {
     unsigned ticks = 0;
     for (;;) {
@@ -651,6 +656,7 @@ static void *poller(void *arg) {
             uint64_t local = 0;
             if (!get || get(b->dev, b->sem, &local) != VK_SUCCESS) continue;
             uint64_t shared = atomic_load(b->counter);
+            if (shared != b->seen) { b->seen = shared; b->seen_us = now_us(); }
             if (trace) {
                 char line[192];
                 snprintf(line, sizeof(line), "%s sem %p %s counter %lu local %llu shared %llu", program_invocation_short_name,
@@ -729,7 +735,7 @@ static VkResult VKAPI_CALL CreateSemaphore(VkDevice d, const VkSemaphoreCreateIn
     atomic_store(counter, timeline ? type_info->initialValue : 0);
     pthread_mutex_lock(&g_br_lock);
     struct bridged *slot = bridged_find(VK_NULL_HANDLE);
-    if (slot) *slot = (struct bridged){*out, d, counter, fd, timeline, fd_ino(fd), 0};
+    if (slot) *slot = (struct bridged){*out, d, counter, fd, timeline, fd_ino(fd), 0, atomic_load(counter), now_us()};
     pthread_mutex_unlock(&g_br_lock);
     if (!slot) { munmap((void *)counter, 4096); close(fd); return r; }
     if (timeline) poller_start();
@@ -775,7 +781,7 @@ static VkResult VKAPI_CALL ImportSemaphoreFdKHR(VkDevice d, const VkImportSemaph
     struct bridged *b = bridged_find(info->semaphore);
     if (b) bridged_drop(b);
     b = bridged_find(VK_NULL_HANDLE);
-    if (b && counter) *b = (struct bridged){info->semaphore, d, counter, -1, timeline, ino, 0};
+    if (b && counter) *b = (struct bridged){info->semaphore, d, counter, -1, timeline, ino, 0, atomic_load(counter), now_us()};
     else if (counter) munmap((void *)counter, 4096);
     pthread_mutex_unlock(&g_br_lock);
     if (timeline) poller_start();
@@ -802,6 +808,7 @@ static uint64_t now_us(void) {
 
 /* How long a submit may wait for another process to reach a shared value before the wait is dropped. */
 #define SUBMIT_GRACE_US 3000
+#define SHARED_IDLE_US 100000
 
 /* Whether a submit should drop its GPU wait on s for value.
  *
@@ -818,6 +825,7 @@ static int submit_drop_wait(VkSemaphore s, uint64_t value) {
     if (!b->timeline) { int r = b->fd < 0; pthread_mutex_unlock(&g_br_lock); return r; }
     const int imported = b->fd < 0;
     const uint64_t queued = b->submitted;
+    const uint64_t seen_us = b->seen_us;
     VkDevice dev = b->dev;
     _Atomic uint64_t *counter = b->counter;
     pthread_mutex_unlock(&g_br_lock);
@@ -830,7 +838,8 @@ static int submit_drop_wait(VkSemaphore s, uint64_t value) {
     PFN_vkSignalSemaphore sig = (PFN_vkSignalSemaphore)d->gdpa(dev, "vkSignalSemaphore");
     if (!sig) sig = (PFN_vkSignalSemaphore)d->gdpa(dev, "vkSignalSemaphoreKHR");
     if (!get) return 0;
-    const uint64_t until = now_us() + SUBMIT_GRACE_US;
+    const uint64_t start = now_us();
+    const uint64_t until = start + (start - seen_us > SHARED_IDLE_US ? 0 : SUBMIT_GRACE_US);
     for (;;) {
         uint64_t local = 0;
         if (get(dev, s, &local) == VK_SUCCESS && local >= value) return 0;
@@ -901,6 +910,167 @@ static void trace_sem(const char *what, VkSemaphore sem, uint64_t value) {
     say("%s", line);
 }
 
+#include <limits.h>
+#include <linux/futex.h>
+#include <sys/syscall.h>
+#include "present_sync.h"
+
+#define TRACK_SLOTS 64
+
+struct tracked {
+    VkDevice dev;
+    VkFence fence;
+    PFN_vkWaitForFences wait;
+    PFN_vkResetFences reset;
+    PFN_vkDestroyFence destroy;
+    int busy;
+};
+static pthread_mutex_t g_trk_lock = PTHREAD_MUTEX_INITIALIZER;
+static pthread_cond_t g_trk_cond = PTHREAD_COND_INITIALIZER;
+static struct tracked g_trk[TRACK_SLOTS];
+static int g_trk_queue[TRACK_SLOTS];
+static unsigned g_trk_head, g_trk_tail;
+static struct droiddeck_present_sync *g_sync;
+static pthread_once_t g_sync_once = PTHREAD_ONCE_INIT;
+
+static void sync_wake(struct droiddeck_present_sync *s) {
+    syscall(SYS_futex, &s->done, FUTEX_WAKE, INT_MAX, NULL, NULL, 0);
+}
+
+static void *completer(void *arg) {
+    struct droiddeck_present_sync *s = arg;
+    for (;;) {
+        pthread_mutex_lock(&g_trk_lock);
+        while (g_trk_head == g_trk_tail) pthread_cond_wait(&g_trk_cond, &g_trk_lock);
+        const int slot = g_trk_queue[g_trk_tail % TRACK_SLOTS];
+        struct tracked t = g_trk[slot];
+        pthread_mutex_unlock(&g_trk_lock);
+        VkResult r;
+        do r = t.wait(t.dev, 1, &t.fence, VK_TRUE, 100000000ull); while (r == VK_TIMEOUT);
+        t.reset(t.dev, 1, &t.fence);
+        pthread_mutex_lock(&g_trk_lock);
+        g_trk[slot].busy = 0;
+        g_trk_tail++;
+        pthread_cond_broadcast(&g_trk_cond);
+        pthread_mutex_unlock(&g_trk_lock);
+        __atomic_fetch_add(&s->done, 1, __ATOMIC_RELEASE);
+        sync_wake(s);
+    }
+    return NULL;
+}
+
+static void sync_init(void) {
+    if (strcmp(program_invocation_short_name, "vrcompositor")) return;
+    const char *rt = getenv("XDG_RUNTIME_DIR");
+    char path[512];
+    snprintf(path, sizeof(path), "%s/droiddeck-steamvr/%s", rt && *rt ? rt : "/run/droiddeck", DROIDDECK_PRESENT_SYNC_NAME);
+    int fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0600);
+    if (fd < 0) return;
+    void *m = ftruncate(fd, DROIDDECK_PRESENT_SYNC_SIZE) == 0
+        ? mmap(NULL, DROIDDECK_PRESENT_SYNC_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0) : MAP_FAILED;
+    close(fd);
+    if (m == MAP_FAILED) return;
+    struct droiddeck_present_sync *s = m;
+    __atomic_store_n(&s->done, __atomic_load_n(&s->issued, __ATOMIC_ACQUIRE), __ATOMIC_RELEASE);
+    sync_wake(s);
+    pthread_t t;
+    if (pthread_create(&t, NULL, completer, s) != 0) { munmap(m, DROIDDECK_PRESENT_SYNC_SIZE); return; }
+    pthread_detach(t);
+    g_sync = s;
+    say("%s", "frames go to the presenter once their GPU work is done");
+}
+
+static int track_take(VkQueue q) {
+    pthread_once(&g_sync_once, sync_init);
+    if (!g_sync) return -1;
+    struct dev *d = dev_for(q);
+    if (!d) return -1;
+    VkDevice dev = d->handle;
+    pthread_mutex_lock(&g_trk_lock);
+    for (;;) {
+        int own = -1, empty = -1, idle = -1;
+        for (int i = 0; i < TRACK_SLOTS && own < 0; i++) {
+            if (g_trk[i].busy) continue;
+            if (g_trk[i].dev == dev) own = i;
+            else if (!g_trk[i].dev) { if (empty < 0) empty = i; }
+            else if (idle < 0) idle = i;
+        }
+        if (own < 0 && empty < 0 && idle >= 0) {
+            g_trk[idle].destroy(g_trk[idle].dev, g_trk[idle].fence, NULL);
+            memset(&g_trk[idle], 0, sizeof(g_trk[idle]));
+            empty = idle;
+        }
+        if (own < 0 && empty >= 0) {
+            struct tracked t = {.dev = dev};
+            t.wait = (PFN_vkWaitForFences)d->gdpa(dev, "vkWaitForFences");
+            t.reset = (PFN_vkResetFences)d->gdpa(dev, "vkResetFences");
+            t.destroy = (PFN_vkDestroyFence)d->gdpa(dev, "vkDestroyFence");
+            PFN_vkCreateFence create = (PFN_vkCreateFence)d->gdpa(dev, "vkCreateFence");
+            VkFenceCreateInfo fci = {VK_STRUCTURE_TYPE_FENCE_CREATE_INFO, NULL, 0};
+            if (!t.wait || !t.reset || !t.destroy || !create || create(dev, &fci, NULL, &t.fence) != VK_SUCCESS) {
+                pthread_mutex_unlock(&g_trk_lock);
+                return -1;
+            }
+            g_trk[empty] = t;
+            own = empty;
+        }
+        if (own >= 0) {
+            g_trk[own].busy = 1;
+            pthread_mutex_unlock(&g_trk_lock);
+            return own;
+        }
+        pthread_cond_wait(&g_trk_cond, &g_trk_lock);
+    }
+}
+
+static void track_submitted(int slot, VkResult r) {
+    pthread_mutex_lock(&g_trk_lock);
+    if (r == VK_SUCCESS) {
+        g_trk_queue[g_trk_head % TRACK_SLOTS] = slot;
+        g_trk_head++;
+        __atomic_fetch_add(&g_sync->issued, 1, __ATOMIC_RELEASE);
+    } else {
+        g_trk[slot].reset(g_trk[slot].dev, 1, &g_trk[slot].fence);
+        g_trk[slot].busy = 0;
+    }
+    pthread_cond_broadcast(&g_trk_cond);
+    pthread_mutex_unlock(&g_trk_lock);
+}
+
+static void track_forget_device(VkDevice dev) {
+    pthread_mutex_lock(&g_trk_lock);
+    for (;;) {
+        int busy = 0;
+        for (int i = 0; i < TRACK_SLOTS; i++) busy |= g_trk[i].dev == dev && g_trk[i].busy;
+        if (!busy) break;
+        pthread_cond_wait(&g_trk_cond, &g_trk_lock);
+    }
+    for (int i = 0; i < TRACK_SLOTS; i++) {
+        if (g_trk[i].dev != dev) continue;
+        g_trk[i].destroy(dev, g_trk[i].fence, NULL);
+        memset(&g_trk[i], 0, sizeof(g_trk[i]));
+    }
+    pthread_mutex_unlock(&g_trk_lock);
+}
+
+static VkResult submit_tracked(PFN_vkQueueSubmit next, VkQueue q, uint32_t count, const VkSubmitInfo *submits, VkFence fence) {
+    const int slot = track_take(q);
+    if (slot < 0) return next(q, count, submits, fence);
+    const VkFence mine = g_trk[slot].fence;
+    VkResult r = next(q, count, submits, fence ? fence : mine);
+    track_submitted(slot, r == VK_SUCCESS && fence ? next(q, 0, NULL, mine) : r);
+    return r;
+}
+
+static VkResult submit2_tracked(PFN_vkQueueSubmit2 next, VkQueue q, uint32_t count, const VkSubmitInfo2 *submits, VkFence fence) {
+    const int slot = track_take(q);
+    if (slot < 0) return next(q, count, submits, fence);
+    const VkFence mine = g_trk[slot].fence;
+    VkResult r = next(q, count, submits, fence ? fence : mine);
+    track_submitted(slot, r == VK_SUCCESS && fence ? next(q, 0, NULL, mine) : r);
+    return r;
+}
+
 static VkResult VKAPI_CALL QueueSubmit(VkQueue q, uint32_t count, const VkSubmitInfo *submits, VkFence fence) {
     PFN_vkQueueSubmit next = (PFN_vkQueueSubmit)dev_for(q)->gdpa(dev_for(q)->handle, "vkQueueSubmit");
     if (sem_trace_on())
@@ -924,7 +1094,7 @@ static VkResult VKAPI_CALL QueueSubmit(VkQueue q, uint32_t count, const VkSubmit
             any = submit_drop_wait(submits[i].pWaitSemaphores[j], t && j < t->waitSemaphoreValueCount ? t->pWaitSemaphoreValues[j] : 0);
     }
     if (!any) {
-        VkResult r = next(q, count, submits, fence);
+        VkResult r = submit_tracked(next, q, count, submits, fence);
         note_submit_signals(count, submits);
         return r;
     }
@@ -960,7 +1130,7 @@ static VkResult VKAPI_CALL QueueSubmit(VkQueue q, uint32_t count, const VkSubmit
             copy[i].pNext = &tl[i]; /* other structs before it are dropped: SteamVR chains only this one */
         }
     }
-    VkResult r = next(q, count, copy, fence);
+    VkResult r = submit_tracked(next, q, count, copy, fence);
     note_submit_signals(count, submits);
     for (uint32_t i = 0; i < count * 3; i++) free(bufs[i]);
     free(bufs); free(tl); free(copy);
@@ -982,7 +1152,7 @@ static VkResult VKAPI_CALL QueueSubmit2(VkQueue q, uint32_t count, const VkSubmi
         for (uint32_t j = 0; j < submits[i].waitSemaphoreInfoCount && !any; j++)
             any = submit_drop_wait(submits[i].pWaitSemaphoreInfos[j].semaphore, submits[i].pWaitSemaphoreInfos[j].value);
     if (!any) {
-        VkResult r = next(q, count, submits, fence);
+        VkResult r = submit2_tracked(next, q, count, submits, fence);
         note_submit2_signals(count, submits);
         return r;
     }
@@ -998,7 +1168,7 @@ static VkResult VKAPI_CALL QueueSubmit2(VkQueue q, uint32_t count, const VkSubmi
         copy[i].waitSemaphoreInfoCount = k;
         copy[i].pWaitSemaphoreInfos = waits[i];
     }
-    VkResult r = next(q, count, copy, fence);
+    VkResult r = submit2_tracked(next, q, count, copy, fence);
     note_submit2_signals(count, submits);
     for (uint32_t i = 0; i < count; i++) free(waits[i]);
     free(waits); free(copy);