Wait out a short out-of-memory spell instead of ending the game

Crash logs ended in fatal errors when the system's commit ran out for a moment: a staging buffer (-2), a guest commit
with the system at 24442 of 24447 MB, image creation (-2). Windows grows a system-managed page file and some drivers
charge device memory to the process commit, so the same request often succeeds a moment later.

oom.c holds the wait nvn_api_cmd.c realloc_wait already used: it requests a release, retries every 10 ms for up to
30 s, logs the first 8 with the commit numbers. rt_oom_request sets a flag that vk_texture_new_submit takes (one atomic
load per submit): it drops up to 256 released textures of the two kinds sweep_pressure drops (overwritten, clean
mirrored), never GPU-written content, and frees the headroom blocks. vk_oom_release does at once what needs no submit:
headroom blocks, finished upload batches and deferred objects.

- memory.c commit: waits and retries without frz_lock between tries; every caller waits, boot ones too.
- vk_staging and the draw-data chunks (vk_exec_draw.c scratch): release, then wait; the open upload batch is not
  submitted, its caller still records into it.
- create_tex: release and one retry; no wait under the texture lock, the texture is asked for again at its next use.
- the texture write-back buffer: release, then wait.
- the headroom thread keeps no spare block for 30 s after a failure.

TOTK_TEST_OOM=<site>:<n> (commit, staging, image, chunk) fails the first n attempts; probe-kernel forces 3 refused
commits, probe-vkexec 2 staging buffers, 1 image and 1 chunk, and both pass through them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valiantsin DzerakhandClaude Opus 5.5 committed 2026-11-09 09:53:30 +03:00
1 parent 5e479f593b
commit 213bc6af6a
13 files changed
+323 -32

No files matched your search

+21 -4
View File
@@ -363,6 +363,16 @@ static CbRes *cb_res(VkExec *ex) {
return D->cur;
}
typedef struct { const VkBufferCreateInfo *bi; const VmaAllocationCreateInfo *ai; Chunk *c; VmaAllocationInfo info;
VkResult r; } ChunkTry;
static int chunk_try(void *arg) {
ChunkTry *t = arg;
vk_mem_collect(); // the fence thread's collect waits while the caller holds cmd_lock
t->r = rt_oom_forced(RT_OOM_CHUNK) ? VK_ERROR_OUT_OF_DEVICE_MEMORY
: vmaCreateBuffer(vk_ctx()->vma, t->bi, t->ai, &t->c->buf, &t->c->alloc, &t->info);
return t->r == VK_SUCCESS;
}
static uint8_t *scratch(VkExec *ex, uint64_t size, VkBuffer *buf, VkDeviceSize *off) {
CbRes *r = cb_res(ex);
size = (size + 255) & ~255ull;
@@ -396,10 +406,17 @@ static uint8_t *scratch(VkExec *ex, uint64_t size, VkBuffer *buf, VkDeviceSize *
VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_HOST,
.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT,
.requiredFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT};
VmaAllocationInfo info;
VkResult res = vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &c->buf, &c->alloc, &info);
if (res != VK_SUCCESS) rt_fatal("vk: transient buffer of %llu bytes failed (%d)", (unsigned long long)n, (int)res);
c->map = info.pMappedData, c->size = n, c->used = 0;
// Out of memory: the recording waits for memory (rt_oom_wait) with what it holds (the texture lock at most,
// which vk_oom_release takes before cmd_lock); new chunks are made only when the spare ones ran out.
ChunkTry ct = {.bi = &bi, .ai = &ai, .c = c};
ct.r = rt_oom_forced(RT_OOM_CHUNK) ? VK_ERROR_OUT_OF_DEVICE_MEMORY
: vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &c->buf, &c->alloc, &ct.info);
if (ct.r == VK_ERROR_OUT_OF_HOST_MEMORY || ct.r == VK_ERROR_OUT_OF_DEVICE_MEMORY) {
vk_oom_release();
rt_oom_wait("vk: transient buffer", n, chunk_try, &ct);
}
if (ct.r != VK_SUCCESS) rt_fatal("vk: transient buffer of %llu bytes failed (%d)", (unsigned long long)n, (int)ct.r);
c->map = ct.info.pMappedData, c->size = n, c->used = 0;
r->nchunks++;
}
*buf = c->buf;