Keep NZCV in a local of each generated function instead of c->nzcv

Every flag-setting instruction packed N, Z, C and V into c->nzcv and every reader unpacked them. recomp/flags.py
now runs a reaching-definitions pass per function: a compare stores its operands in a local Flags F, and a reader
whose definitions are all of one kind computes its condition from them, so cmp and b.cond become one host compare.
CCMP, FCMP, MSR NZCV and ADCS write the packed form, which readers reached by several kinds also use. Calls and
returns publish nothing (AAPCS64 neither passes nor preserves NZCV); tail branches, falls into the next function and
br through rt_dispatch publish c->nzcv, since discovery can start a function inside another one. Functions with an
unresolved computed br keep flags in c->nzcv.

1.4.3 main: 492k direct and 89k packed flag reads, .text 342 -> 299 MiB. Fight bench (bench-fight2-2026-10-27b,
frames 5400-7500, uncapped) 13.15 and 13.63 ms vs 13.9 before. tests/flags_check.py checks the new helpers against
the packed ones; tests/diff/func.py no longer compares NZCV, which generated code leaves untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/building.md

@@ -138,7 +138,7 @@ change passes these, all run with `.venv`:
 | `tests\diff\lift.py` | instructions sampled from the extracted modules; `--coverage` lists every word the lifter rejects or traps | yes |
 | `tests\diff\simd.py` | the whole FP/SIMD encoding space | no |
 | `tests\diff\atomic.py` | exclusive load/store sequences and ARMv8.1 atomics | no |
-| `tests\diff\func.py` | whole leaf functions as the generator emits them, loops and jump tables included | yes |
+| `tests\diff\func.py` | whole functions as the generator emits them, loops, jump tables and calls included (calls go to a stub that checks the registers at the call) | yes |
 | `tests\excl_mt.py` | the exclusive monitor helpers under contention and ABA, on several threads | no |
 | `tests\jumptables_check.py` | `recomp/jumptables.py` against a Unicorn oracle; must report 0 disagreements, on 1.4.3 too | yes |
 

recomp/flags.py

@@ -0,0 +1,144 @@
+"""Function-local NZCV for recomp/gen.py.
+
+The lifter writes flags the way the guest does: every flag-setting instruction packs N, Z, C and V into c->nzcv
+(subs64() and friends in runtime/cpu.h) and every reader unpacks them (COND_xx(c)). Compilers keep NZCV inside a
+function, and AAPCS64 neither passes it to a callee nor preserves it across a call. So a generated function keeps its
+flags in a local `Flags F` (runtime/cpu.h), which clang holds in registers, and writes c->nzcv only where it leaves
+for another function other than by a call or a return (see plan()).
+
+A flag-setting instruction of a known kind (subtract, add or logic result; 32 or 64 bits) stores its operands in F.a
+and F.b. A reader all of whose reaching definitions are of one kind computes its condition from those operands, so
+`cmp x1, x2; b.lt` becomes a signed compare in C. Instructions whose result is only known packed (CCMP, FCMP, MSR
+NZCV, ADCS) write F.nz, the packed form; a reader whose definitions are of several kinds reads F.nz, and those
+definitions then also write it.
+
+A read that can see flags a callee left reads the caller's flags from before the call instead. AAPCS64 code never
+reads a callee's flags, so such a read only sits on a path that does not run: in TotK it follows calls to noreturn
+functions (asserts) that the compiler placed before another branch's target. Plan.after_call counts them.
+"""
+import re
+
+KIND_DEF = re.compile(r'\b(subs|adds|logic)(32|64)\(c, ')
+CARRY_DEF = re.compile(r'\b(adcs|sbcs)(32|64)\(c, ')
+CARRY_USE = re.compile(r'\b(adc|sbc)(32|64)\(c, ')
+COND = re.compile(r'COND_([A-Z]{2})\(c\)')
+NZ_DEF = 'c->nzcv = '
+NZ_USE = '(c->nzcv)'
+KIND_NAME = {'subs': 'sub', 'adds': 'add', 'logic': 'logic'}
+
+ENTRY, CALL = 'entry', 'call'
+
+
+class FlagError(Exception):
+    pass
+
+
+def effect(code):
+    """(reads flags, kind of the flags written or None) of one instruction's C."""
+    reads = bool(COND.search(code) or CARRY_USE.search(code) or CARRY_DEF.search(code) or NZ_USE in code)
+    if NZ_DEF in code or CARRY_DEF.search(code):
+        return reads, 'nz'
+    defs = KIND_DEF.findall(code)
+    if len(defs) > 1:
+        raise FlagError(f'several flag definitions in one instruction: {code}')
+    if defs:
+        op, size = defs[0]
+        return reads, KIND_NAME[op] + size
+    return reads, None
+
+
+class Plan:
+    """Per instruction: the kind its flag reads use, whether its flag definition also writes F.nz, and what it writes
+    to c->nzcv before it leaves for another function."""
+
+    def __init__(self, n):
+        self.reader = [None] * n
+        self.pack = [False] * n
+        self.exit = [None] * n      # kind of the flags to publish in c->nzcv at a tail exit, or None
+        self.entry = False          # F.nz starts as c->nzcv: flags from the entry reach a read or an exit
+        self.entry_read = False     # they reach a read, not only an exit
+        self.after_call = 0         # reads that can see flags a callee left (see the module docstring)
+        self.used = False
+
+    def decl(self):
+        """Declaration of F at the top of the function."""
+        return 'Flags F = {.nz = c->nzcv};' if self.entry else 'Flags F = {0};'
+
+    def publish(self, i):
+        """Statement writing the flags to c->nzcv at instruction i's tail exit ('' when it need not)."""
+        k = self.exit[i]
+        return f'c->nzcv = fnz_{k}(&F); ' if k else ''
+
+    def rewrite(self, i, code):
+        """code of instruction i with its flag reads and writes on F."""
+        rk = self.reader[i]
+        _, dk = effect(code)
+        if rk:
+            code = COND.sub(lambda m: f'fc_{rk}(&F, CC_{m.group(1)})', code)
+            code = CARRY_DEF.sub(lambda m: f'{m.group(1)}{m.group(2)}f(&F, fc_{rk}(&F, CC_CS), ', code)
+            code = CARRY_USE.sub(lambda m: f'{m.group(1)}{m.group(2)}f(fc_{rk}(&F, CC_CS), ', code)
+            code = code.replace(NZ_USE, f'(fnz_{rk}(&F))')
+        if dk == 'nz':
+            code = code.replace(NZ_DEF, 'F.nz = ')
+            code = re.sub(r'\(void\)(subs|adds)(32|64)\(c, ', lambda m: f'F.nz = nz_{KIND_NAME[m.group(1)]}{m.group(2)}(',
+                          code)
+        elif dk:
+            code = KIND_DEF.sub(lambda m: f'f{KIND_NAME[m.group(1)]}{m.group(2)}(&F, ', code)
+            if self.pack[i]:
+                code += f' F.nz = fnz_{dk}(&F);'
+        if 'c->nzcv' in code or COND.search(code) or KIND_DEF.search(code) or CARRY_USE.search(code):
+            raise FlagError(f'flag access left after the rewrite: {code}')
+        return code
+
+
+def plan(effects, succ, calls, exits):
+    """effects[i]: (reads, kind or None); succ[i]: successor indices inside the function; calls: indices of calls,
+    whose callee may leave any flags; exits: indices that can leave for another function other than by a call or a
+    return (a tail branch, falling into the next function, a br through rt_dispatch).
+
+    Discovery can start a function inside another one (a branch target, a materialised address), and the code
+    before it then falls or branches into it with live flags. So a tail exit publishes the flags in c->nzcv, and a
+    function whose reads can see its entry flags starts F.nz from c->nzcv. Calls and returns publish nothing: AAPCS64
+    code neither passes flags to a callee nor reads a callee's, so a call's flags count as no definition."""
+    n = len(effects)
+    p = Plan(n)
+    if not any(r or k for r, k in effects):
+        return p
+    p.used = True
+    reach = [set() for _ in range(n)]
+    reach[0].add(ENTRY)
+    work = [0]
+    while work:
+        i = work.pop()
+        out = {i} if effects[i][1] else ({CALL} if i in calls else reach[i])
+        for s in succ[i]:
+            if not out <= reach[s]:
+                reach[s] |= out
+                work.append(s)
+
+    def kind(d):
+        return 'nz' if d == ENTRY else effects[d][1]
+
+    def use(r):
+        """Kind a use of the definitions r reads; several kinds read the packed form, which they then all write."""
+        kinds = {kind(d) for d in r}
+        if ENTRY in r:
+            p.entry = True
+        if len(kinds) == 1:
+            return kinds.pop()
+        for d in r:
+            if d != ENTRY:
+                p.pack[d] = True
+        return 'nz'   # several kinds, or none (unreachable)
+
+    for i, (reads, _) in enumerate(effects):
+        if reads:
+            p.after_call += CALL in reach[i]
+            p.reader[i] = use(reach[i] - {CALL})
+            p.entry_read |= ENTRY in reach[i]
+    for i in exits:
+        # Entry flags alone are still in c->nzcv.
+        r = reach[i] - {CALL}
+        if r and r != {ENTRY}:
+            p.exit[i] = use(r)
+    return p

recomp/gen.py

@@ -38,6 +38,7 @@ ROOT = Path(__file__).resolve().parents[1]
 sys.path.insert(0, str(ROOT / 'tools'))
 sys.path.insert(0, str(ROOT / 'recomp'))
 
+import flags  # noqa: E402
 import jumptables  # noqa: E402
 import patches  # noqa: E402
 from check_content import GENERATED_MARKER  # noqa: E402
@@ -246,6 +247,66 @@ class Emitter:
                 raise ValueError(f'{entry + 4 * i:#x}: asm instruction from the portable lifter')
         return labels, tables
 
+    def replacements(self, a):
+        """Lifted replacement words of the patched site a."""
+        return [self.mod.lifter.lift(w, a) for w in self.patched[a][1]]
+
+    def flag_plan(self, insns, entry, end, tables):
+        """recomp/flags.py plan for the function: the intra-function control flow and each instruction's flag use.
+        Index len(insns) stands for falling through into the next function."""
+        n = len(insns)
+        succ, calls, effects, exits = [], set(), [], {n}
+
+        def idx(t):
+            return [(t - entry) // 4] if entry <= t < end else []
+
+        for i, ins in enumerate(insns):
+            k, a = ins.kind, entry + 4 * i
+            nxt = [i + 1]
+            if k == 'op':
+                succ.append(nxt)
+            elif k in ('bl', 'blr'):
+                succ.append(nxt)
+                calls.add(i)
+            elif k == 'b':
+                succ.append(idx(ins.target))
+                if not idx(ins.target):
+                    exits.add(i)
+            elif k == 'bcond':
+                succ.append(idx(ins.target) + nxt)
+                if not idx(ins.target):
+                    exits.add(i)
+            elif k == 'br':
+                jt = tables.get(a)
+                succ.append([t for x in jt for t in idx(x)] if jt and jt != 'all' else [])
+                exits.add(i)   # through rt_dispatch, also a table's default
+            else:   # ret, trap
+                succ.append([])
+            if a in self.patched:
+                effects.append(self.patched_effect(ins, a))
+                if any(r.kind == 'bl' for r in self.replacements(a)):
+                    calls.add(i)
+            else:
+                effects.append(flags.effect(ins.cond if k == 'bcond' else ins.code or ''))
+        succ.append([])
+        effects.append((False, None))
+        return flags.plan(effects, succ, calls, exits)
+
+    def patched_effect(self, ins, a):
+        """Flag effect of a patched site: both alternatives must have the same one, in at most one word each."""
+        alts = []
+        for words in ([ins], self.replacements(a)):
+            fx = [flags.effect(w.code or '') for w in words if w.kind == 'op']
+            fx = [e for e in fx if e[0] or e[1]]
+            if len(fx) > 1:
+                raise patches.PatchError(f'{a:#x}: several flag-using words in one alternative')
+            alts.append(fx[0] if fx else (False, None))
+        if alts[0][1] != alts[1][1]:
+            raise patches.PatchError(f'{a:#x}: the alternatives write different flags ({alts[0][1]}, {alts[1][1]})')
+        if (alts[0][1] or alts[0][0] or alts[1][0]) and ins.kind == 'bl':
+            raise patches.PatchError(f'{a:#x}: a patched call that uses flags')
+        return alts[0][0] or alts[1][0], alts[0][1]
+
     def function(self, entry, end):
         """(lines of C, referenced function starts) for the guest function [entry, end)."""
         mod, starts = self.mod, self.starts
@@ -255,15 +316,32 @@ class Emitter:
         all_labels = 'all' in tables.values()
         if all_labels:
             labels = set(range(entry, end, 4))
+        # An unresolved computed br re-enters the function through fb_<entry>, a C function of its own, which local
+        # flags would not survive; such functions keep NZCV in c->nzcv.
+        fp = None if all_labels else self.flag_plan(insns, entry, end, tables)
+        if fp and fp.used:
+            self.stats['local_flag_functions'] += 1
+            self.stats['flag_reads_direct'] += sum(1 for k in fp.reader if k and k != 'nz')
+            self.stats['flag_reads_packed'] += sum(1 for k in fp.reader if k == 'nz')
+            self.stats['flag_entry_functions'] += fp.entry
+            self.stats['flag_entry_reads'] += fp.entry_read
+            self.stats['flag_reads_after_call'] += fp.after_call
+            self.stats['flag_publishing_exits'] += sum(1 for k in fp.exit if k)
 
-        def jump(t):
+        def fx(i, code):
+            return fp.rewrite(i, code) if fp and fp.used else code
+
+        def publish(i):
+            return fp.publish(i) if fp and fp.used else ''
+
+        def jump(t, i):
             if entry <= t < end:
                 return f'goto L_{t:x};'
             if t in starts:
                 refs.add(t)
-                return f'{MUSTTAIL} {fname(t)}(c);'
+                return f'{publish(i)}{MUSTTAIL} {fname(t)}(c);'
             self.stats['external_branches'] += 1
-            return f'c->pc = {hx(t)}; {MUSTTAIL} rt_dispatch(c);'
+            return f'{publish(i)}c->pc = {hx(t)}; {MUSTTAIL} rt_dispatch(c);'
 
         if all_labels:
             # An unresolved computed br re-enters the function through fb_<entry>, which switches on c->pc to any
@@ -280,6 +358,9 @@ class Emitter:
         else:
             redispatch = f'{MUSTTAIL} rt_dispatch(c);'
             out = [f'void {fname(entry)}(Cpu *c) {{', f'    FN_ENTER({hx(entry)});']
+            if fp.used:
+                out.append('    ' + fp.decl())
+
         def call(t, a):
             if t in starts:
                 refs.add(t)
@@ -287,21 +368,20 @@ class Emitter:
             self.stats['external_calls'] += 1
             return f'c->x[30] = {hx(a + 4)}; c->pc = {hx(t)}; rt_dispatch(c);'
 
-        def patch(ins, a):
+        def patch(i, ins, a):
             """if (switch) { replacement } else { original } for a patched site (recomp/patches.py)."""
             k, words = self.patched[a]
             if ins.kind not in ('op', 'bl'):
                 raise patches.PatchError(f'{a:#x}: patched instruction is a {ins.kind}')
             repl = []
-            for j, w in enumerate(words):
-                r = mod.lifter.lift(w, a)
+            for j, (w, r) in enumerate(zip(words, self.replacements(a))):
                 if r.kind == 'op':
-                    repl.append(r.code)
+                    repl.append(fx(i, r.code))
                 elif r.kind == 'bl' and j == len(words) - 1:
                     repl.append(call(r.target, a))
                 else:
                     raise patches.PatchError(f'{a:#x}: replacement word {w:08x} is a {r.kind}')
-            orig = ins.code if ins.kind == 'op' else call(ins.target, a)
+            orig = fx(i, ins.code) if ins.kind == 'op' else call(ins.target, a)
             self.uses_patches = True
             self.stats['patched_sites'] += 1
             return f'    if (rt_patch_on[{k}]) {{ {" ".join(repl)} }} else {{ {orig} }}'
@@ -313,13 +393,13 @@ class Emitter:
                 out.append(f'L_{a:x}:;')
             k = ins.kind
             if a in self.patched:
-                out.append(patch(ins, a))
+                out.append(patch(i, ins, a))
             elif k == 'op' or k == 'trap':
-                out.append('    ' + ins.code)
+                out.append('    ' + fx(i, ins.code))
             elif k == 'b':
-                out.append('    ' + jump(ins.target))
+                out.append('    ' + jump(ins.target, i))
             elif k == 'bcond':
-                out.append(f'    if ({ins.cond}) {{ {jump(ins.target)} }}')
+                out.append(f'    if ({fx(i, ins.cond)}) {{ {jump(ins.target, i)} }}')
             elif k == 'bl':
                 out.append('    ' + call(ins.target, a))
             elif k == 'blr':
@@ -331,16 +411,16 @@ class Emitter:
                 elif jt:
                     cases = ' '.join(f'case {hx(t)}: goto L_{t:x};' for t in jt)
                     out.append(f'    switch ({ins.target}) {{ {cases} default: break; }}')
-                    out.append(f'    c->pc = {ins.target}; {redispatch}')
+                    out.append(f'    {publish(i)}c->pc = {ins.target}; {redispatch}')
                 else:
-                    out.append(f'    c->pc = {ins.target}; {MUSTTAIL} rt_dispatch(c);')
+                    out.append(f'    {publish(i)}c->pc = {ins.target}; {MUSTTAIL} rt_dispatch(c);')
             elif k == 'ret':
                 out.append('    return;')
             else:
                 raise ValueError(f'{a:#x}: unknown instruction kind {k!r}')
         if k not in TERMINATORS:  # falls through into the next function
             if end < mod.hi:
-                out.append('    ' + jump(end))
+                out.append('    ' + jump(end, len(insns)))
             else:
                 out.append(f'    rt_udf(c, {hx(end)});')
         out.append('}')
@@ -562,6 +642,13 @@ def main():
                 'br_dispatch': stats['br_dispatch'],
                 'traps': stats['traps'],
                 'patched_sites': stats['patched_sites'],
+                'local_flag_functions': stats['local_flag_functions'],
+                'flag_reads_direct': stats['flag_reads_direct'],
+                'flag_reads_packed': stats['flag_reads_packed'],
+                'flag_entry_functions': stats['flag_entry_functions'],
+                'flag_entry_reads': stats['flag_entry_reads'],
+                'flag_reads_after_call': stats['flag_reads_after_call'],
+                'flag_publishing_exits': stats['flag_publishing_exits'],
                 'unsupported': stats['unsupported'],
                 'external_branches': stats['external_branches'],
                 'external_calls': stats['external_calls'],

runtime/cpu.h

@@ -215,6 +215,107 @@ static inline uint32_t logic32(Cpu *c, uint32_t r) { c->nzcv = MKFLAGS(r >> 31,
 ADC_IMPL(64, )
 ADC_IMPL(32, )
 
+// ---- function-local flags (recomp/flags.py)
+// Generated functions keep NZCV in a local Flags F, never in c->nzcv. A definition of a known kind stores its
+// operands (a - b, a + b, or a logic result in a); fc_<kind>(&F, CC_xx) computes a condition from them, so a compare
+// and its branch become one host compare. F.nz holds the packed form for definitions only known packed (CCMP, FCMP,
+// MSR, ADCS) and for readers that several kinds of definition reach. Every helper is always inlined with a constant
+// condition, so F stays in registers and only the condition's own arithmetic remains.
+typedef struct { uint64_t a, b, nz; } Flags;
+enum { CC_EQ, CC_NE, CC_CS, CC_CC, CC_MI, CC_PL, CC_VS, CC_VC, CC_HI, CC_LS, CC_GE, CC_LT, CC_GT, CC_LE };
+#define CPU_FL static inline __attribute__((always_inline))
+
+CPU_FL int fl_cond(int cc, int n, int z, int cy, int v) {
+    switch (cc) {
+    case CC_EQ: return z;
+    case CC_NE: return !z;
+    case CC_CS: return cy;
+    case CC_CC: return !cy;
+    case CC_MI: return n;
+    case CC_PL: return !n;
+    case CC_VS: return v;
+    case CC_VC: return !v;
+    case CC_HI: return cy && !z;
+    case CC_LS: return !cy || z;
+    case CC_GE: return n == v;
+    case CC_LT: return n != v;
+    case CC_GT: return !z && n == v;
+    default: return z || n != v;
+    }
+}
+
+// Subtract (SUBS, CMP): conditions as host compares of the operands.
+#define FL_SUB(N, U, S)                                                                    \
+    CPU_FL U fsub##N(Flags *f, U a, U b) { f->a = a; f->b = b; return (U)(a - b); }            \
+    CPU_FL int fc_sub##N(const Flags *f, int cc) {                                             \
+        U a = (U)f->a, b = (U)f->b, r = (U)(a - b);                                        \
+        switch (cc) {                                                                      \
+        case CC_EQ: return a == b;                                                         \
+        case CC_NE: return a != b;                                                         \
+        case CC_CS: return a >= b;                                                         \
+        case CC_CC: return a < b;                                                          \
+        case CC_MI: return (S)r < 0;                                                       \
+        case CC_PL: return (S)r >= 0;                                                      \
+        case CC_VS: return (S)((a ^ b) & (a ^ r)) < 0;                                     \
+        case CC_VC: return (S)((a ^ b) & (a ^ r)) >= 0;                                    \
+        case CC_HI: return a > b;                                                          \
+        case CC_LS: return a <= b;                                                         \
+        case CC_GE: return (S)a >= (S)b;                                                   \
+        case CC_LT: return (S)a < (S)b;                                                    \
+        case CC_GT: return (S)a > (S)b;                                                    \
+        default: return (S)a <= (S)b;                                                      \
+        }                                                                                  \
+    }                                                                                      \
+    CPU_FL uint64_t nz_sub##N(U a, U b) {                                                      \
+        U r = (U)(a - b);                                                                  \
+        return MKFLAGS((S)r < 0, r == 0, a >= b, (S)((a ^ b) & (a ^ r)) < 0);              \
+    }                                                                                      \
+    CPU_FL uint64_t fnz_sub##N(const Flags *f) { return nz_sub##N((U)f->a, (U)f->b); }
+FL_SUB(64, uint64_t, int64_t)
+FL_SUB(32, uint32_t, int32_t)
+
+// Add (ADDS, CMN) and logic results (ANDS, TST, BICS): conditions from N, Z, C, V.
+#define FL_ADD(N, U, S)                                                                    \
+    CPU_FL U fadd##N(Flags *f, U a, U b) { f->a = a; f->b = b; return (U)(a + b); }            \
+    CPU_FL uint64_t nz_add##N(U a, U b) {                                                      \
+        U r = (U)(a + b);                                                                  \
+        return MKFLAGS((S)r < 0, r == 0, r < a, (S)((a ^ r) & (b ^ r)) < 0);               \
+    }                                                                                      \
+    CPU_FL int fc_add##N(const Flags *f, int cc) {                                             \
+        U a = (U)f->a, b = (U)f->b, r = (U)(a + b);                                        \
+        return fl_cond(cc, (S)r < 0, r == 0, r < a, (S)((a ^ r) & (b ^ r)) < 0);           \
+    }                                                                                      \
+    CPU_FL uint64_t fnz_add##N(const Flags *f) { return nz_add##N((U)f->a, (U)f->b); }         \
+    CPU_FL U flogic##N(Flags *f, U r) { f->a = r; return r; }                                  \
+    CPU_FL int fc_logic##N(const Flags *f, int cc) {                                           \
+        U r = (U)f->a;                                                                     \
+        return fl_cond(cc, (S)r < 0, r == 0, 0, 0);                                        \
+    }                                                                                      \
+    CPU_FL uint64_t fnz_logic##N(const Flags *f) { return MKFLAGS((S)f->a < 0, (U)f->a == 0, 0, 0); }
+FL_ADD(64, uint64_t, int64_t)
+FL_ADD(32, uint32_t, int32_t)
+
+// Packed flags.
+CPU_FL int fc_nz(const Flags *f, int cc) {
+    return fl_cond(cc, f->nz >> 31 & 1, f->nz >> 30 & 1, f->nz >> 29 & 1, f->nz >> 28 & 1);
+}
+CPU_FL uint64_t fnz_nz(const Flags *f) { return f->nz; }
+
+// ADC, SBC, ADCS, SBCS with the carry in from F.
+#define FL_ADC(N, U)                                                                       \
+    CPU_FL U adc##N##f(int cin, U a, U b) { return (U)(a + b + (U)cin); }                      \
+    CPU_FL U sbc##N##f(int cin, U a, U b) { return (U)(a - b - 1 + (U)cin); }                  \
+    CPU_FL U adcs##N##f(Flags *f, int cin, U a, U b) {                                         \
+        U r = (U)(a + b + (U)cin);                                                         \
+        int carry = cin ? r <= a : r < a;                                                  \
+        f->nz = MKFLAGS(r >> (N - 1), r == 0, carry, ((a ^ r) & (b ^ r)) >> (N - 1));      \
+        return r;                                                                          \
+    }                                                                                      \
+    CPU_FL U sbcs##N##f(Flags *f, int cin, U a, U b) { return adcs##N##f(f, cin, a, (U)~b); }
+FL_ADC(64, uint64_t)
+FL_ADC(32, uint32_t)
+#undef CPU_FL
+
 // ---- bit ops
 #define ROR64(v, r) ((uint64_t)(v) >> ((r) & 63) | (uint64_t)(v) << ((64 - ((r) & 63)) & 63))
 #define ROR32(v, r) ((uint32_t)(v) >> ((r) & 31) | (uint32_t)(v) << ((32 - ((r) & 31)) & 31))

tests/diff/common.py

@@ -258,14 +258,14 @@ def run_harness(exe, work, buf0, cases, images, stubs=False):
     return recs, err
 
 
-def diff_state(got, want, check_pc, check_mem):
+def diff_state(got, want, check_pc, check_mem, check_nzcv=True):
     """got: harness record; want: (x, sp, nzcv, pc, v, mem). Returns a list of mismatches."""
     hx, hsp, hnz, hpc, hmem, hv = got
     ex, esp, enz, epc, ev, emem = want
     d = [f'x{j}: got {hx[j]:x} want {ex[j]:x}' for j in range(31) if hx[j] != ex[j]]
     if hsp != esp:
         d.append(f'sp: got {hsp:x} want {esp:x}')
-    if hnz != enz:
+    if check_nzcv and hnz != enz:
         d.append(f'nzcv: got {hnz:x} want {enz:x}')
     if check_pc and hpc != epc:
         d.append(f'pc: got {hpc:x} want {epc:x}')

tests/diff/func.py

@@ -7,8 +7,8 @@ Emits them with the generator's own Emitter, compiles them with tests/diff/harne
 states: x0-x7 point into the scratch buffer (or hold small integers), the buffer holds pointers into itself and random
 data, SP sits near the buffer's top and LR is a sentinel address. Unicorn (Cortex-A57) stops when PC reaches the
 sentinel. A case is skipped when Unicorn faults, writes outside the scratch buffer (module images are mapped
-read-only), leaves the function, or runs longer than --steps instructions. Compares X0-X30, SP, NZCV, V0-V31 and the
-scratch buffer.
+read-only), leaves the function, or runs longer than --steps instructions. Compares X0-X30, SP, V0-V31 and the
+scratch buffer; NZCV is local to a generated function (recomp/flags.py).
 
 Every BL and BLR goes to one stub on both sides (rt_stub in harness.c, Runner.stub here) instead of its callee. The
 stub folds the guest state at the call into a digest and sets the caller-saved registers from it, so the test sees
@@ -387,7 +387,9 @@ def main():
 
     fails, nfail = {}, 0
     for case, want_state, got in zip(cases, expect, recs):
-        d = diff_state(got, want_state, check_pc=False, check_mem=True)
+        # Generated functions keep NZCV in a local (recomp/flags.py) and leave c->nzcv as it was: flags only count
+        # where the function reads them, which the registers and memory show.
+        d = diff_state(got, want_state, check_pc=False, check_mem=True, check_nzcv=False)
         if d:
             nfail += 1
             fails.setdefault(case[0], d[:4])

tests/flags_check.c

@@ -0,0 +1,111 @@
+// The function-local flag helpers of runtime/cpu.h (Flags, fc_*, fnz_*, nz_*, adc*f) against the packed helpers
+// (subs64() and friends, COND_xx), which tests/diff/lift.py checks against Unicorn. Built and run by
+// tests/flags_check.py; prints the first mismatches and exits 1 on any.
+#include <stdio.h>
+
+#include "cpu.h"
+
+static const uint64_t EDGE[] = {0, 1, 2, 0x7f, 0x80, 0xff, 0x7fff, 0x8000, 0xffff, 0x7fffffff, 0x80000000,
+                                0x80000001, 0xfffffffe, 0xffffffff, 0x100000000ull, 0x7fffffffffffffffull,
+                                0x8000000000000000ull, 0x8000000000000001ull, 0xfffffffffffffffeull,
+                                0xffffffffffffffffull, 0x123456789abcdef0ull, 0xfedcba9876543210ull};
+#define NEDGE (sizeof EDGE / sizeof EDGE[0])
+
+static uint64_t rng = 0x9E3779B97F4A7C15ull;
+static uint64_t next(void) {
+    rng ^= rng << 13, rng ^= rng >> 7, rng ^= rng << 17;
+    return rng;
+}
+
+static int fails;
+static void bad(const char *what, int cc, uint64_t a, uint64_t b, uint64_t got, uint64_t want) {
+    if (fails++ < 20)
+        printf("%s cc %d a %llx b %llx: got %llx want %llx\n", what, cc, (unsigned long long)a, (unsigned long long)b,
+               (unsigned long long)got, (unsigned long long)want);
+}
+
+static int cond(const Cpu *c, int cc) {
+    switch (cc) {
+    case CC_EQ: return COND_EQ(c);
+    case CC_NE: return COND_NE(c);
+    case CC_CS: return COND_CS(c);
+    case CC_CC: return COND_CC(c);
+    case CC_MI: return COND_MI(c);
+    case CC_PL: return COND_PL(c);
+    case CC_VS: return COND_VS(c);
+    case CC_VC: return COND_VC(c);
+    case CC_HI: return COND_HI(c);
+    case CC_LS: return COND_LS(c);
+    case CC_GE: return COND_GE(c);
+    case CC_LT: return COND_LT(c);
+    case CC_GT: return COND_GT(c);
+    default: return COND_LE(c);
+    }
+}
+
+// The condition argument is not constant here, so this also covers the switch default paths.
+#define CHECK(NAME, OLD, NEW, FC, FNZ)                                                     \
+    do {                                                                                   \
+        Cpu c = {0};                                                                       \
+        Flags f = {0};                                                                     \
+        uint64_t ro = OLD, rn = NEW;                                                       \
+        if (ro != rn) bad(NAME " result", -1, a, b, rn, ro);                               \
+        for (int cc = 0; cc < 14; cc++)                                                    \
+            if (!FC(&f, cc) != !cond(&c, cc)) bad(NAME, cc, a, b, FC(&f, cc), cond(&c, cc)); \
+        f.nz = 0;                                                                          \
+        if (FNZ(&f) != c.nzcv) bad(NAME " packed", -1, a, b, FNZ(&f), c.nzcv);             \
+        Flags g = {.nz = c.nzcv};                                                          \
+        for (int cc = 0; cc < 14; cc++)                                                    \
+            if (!fc_nz(&g, cc) != !cond(&c, cc)) bad(NAME " nz", cc, a, b, fc_nz(&g, cc), cond(&c, cc)); \
+    } while (0)
+
+static void one(uint64_t a, uint64_t b) {
+    uint32_t a32 = (uint32_t)a, b32 = (uint32_t)b;
+    CHECK("sub64", subs64(&c, a, b), fsub64(&f, a, b), fc_sub64, fnz_sub64);
+    CHECK("sub32", subs32(&c, a32, b32), fsub32(&f, a32, b32), fc_sub32, fnz_sub32);
+    CHECK("add64", adds64(&c, a, b), fadd64(&f, a, b), fc_add64, fnz_add64);
+    CHECK("add32", adds32(&c, a32, b32), fadd32(&f, a32, b32), fc_add32, fnz_add32);
+    CHECK("logic64", logic64(&c, a & b), flogic64(&f, a & b), fc_logic64, fnz_logic64);
+    CHECK("logic32", logic32(&c, a32 & b32), flogic32(&f, a32 & b32), fc_logic32, fnz_logic32);
+    {
+        Cpu c = {0};
+        if (nz_sub64(a, b) != (subs64(&c, a, b), c.nzcv)) bad("nz_sub64", -1, a, b, nz_sub64(a, b), c.nzcv);
+        if (nz_sub32(a32, b32) != (subs32(&c, a32, b32), c.nzcv)) bad("nz_sub32", -1, a, b, nz_sub32(a32, b32), c.nzcv);
+        if (nz_add64(a, b) != (adds64(&c, a, b), c.nzcv)) bad("nz_add64", -1, a, b, nz_add64(a, b), c.nzcv);
+        if (nz_add32(a32, b32) != (adds32(&c, a32, b32), c.nzcv)) bad("nz_add32", -1, a, b, nz_add32(a32, b32), c.nzcv);
+    }
+    for (int cin = 0; cin < 2; cin++) {
+        Cpu c = {.nzcv = (uint64_t)cin << 29};
+        Flags f = {0};
+        if (adc64f(cin, a, b) != adc64(&c, a, b)) bad("adc64", cin, a, b, adc64f(cin, a, b), adc64(&c, a, b));
+        if (sbc64f(cin, a, b) != sbc64(&c, a, b)) bad("sbc64", cin, a, b, sbc64f(cin, a, b), sbc64(&c, a, b));
+        if (adc32f(cin, a32, b32) != adc32(&c, a32, b32)) bad("adc32", cin, a, b, adc32f(cin, a32, b32), adc32(&c, a32, b32));
+        if (sbc32f(cin, a32, b32) != sbc32(&c, a32, b32)) bad("sbc32", cin, a, b, sbc32f(cin, a32, b32), sbc32(&c, a32, b32));
+        uint64_t r;
+#define CARRY(NAME, NEW, OLD)                                                              \
+        c.nzcv = (uint64_t)cin << 29;                                                      \
+        r = NEW;                                                                           \
+        if (r != OLD || f.nz != c.nzcv) bad(NAME, cin, a, b, f.nz, c.nzcv);
+        CARRY("adcs64", adcs64f(&f, cin, a, b), adcs64(&c, a, b))
+        CARRY("sbcs64", sbcs64f(&f, cin, a, b), sbcs64(&c, a, b))
+        CARRY("adcs32", adcs32f(&f, cin, a32, b32), adcs32(&c, a32, b32))
+        CARRY("sbcs32", sbcs32f(&f, cin, a32, b32), sbcs32(&c, a32, b32))
+    }
+}
+
+int main(void) {
+    long n = 0;
+    for (size_t i = 0; i < NEDGE; i++)
+        for (size_t j = 0; j < NEDGE; j++, n++) one(EDGE[i], EDGE[j]);
+    for (int k = 0; k < 2000000; k++, n++) {
+        uint64_t a = next(), b = next();
+        switch (k & 3) {
+        case 1: b = a; break;                       // equal operands
+        case 2: b = a + (next() & 3) - 1; break;    // neighbours
+        case 3: a &= 0xff, b &= 0xff; break;        // small values
+        }
+        one(a, b);
+    }
+    printf("%ld operand pairs, %d mismatches\n", n, fails);
+    return fails != 0;
+}

tests/flags_check.py

@@ -0,0 +1,32 @@
+"""Build and run tests/flags_check.c: runtime/cpu.h's function-local flag helpers against its packed flag helpers.
+
+Usage: flags_check.py
+"""
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / 'tests' / 'diff'))
+from common import MSYS_BIN, compiler  # noqa: E402
+
+
+def main():
+    work = ROOT / 'build' / 'test' / 'flags'
+    work.mkdir(parents=True, exist_ok=True)
+    exe = work / ('flags_check.exe' if os.name == 'nt' else 'flags_check')
+    env = dict(os.environ)
+    if os.name == 'nt' and MSYS_BIN.exists():
+        env['PATH'] = str(MSYS_BIN) + os.pathsep + env.get('PATH', '')
+    for opt in ('-O0', '-O1'):
+        subprocess.check_call([compiler(), opt, '-fno-strict-aliasing', '-I', str(ROOT / 'runtime'),
+                               str(ROOT / 'tests' / 'flags_check.c'), '-o', str(exe)], env=env)
+        print(opt, end=': ', flush=True)
+        if subprocess.run([str(exe)], env=env).returncode:
+            return 1
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())