Approved Too Late: Verdict Staleness in LLM-Guarded Self-Adaptive Systems
Authors: Ilai Shraga, Roei Eshel, Lior Gorelik
Organizations: University of Cambridge Cambridge, United Kingdom · Maccabim-Re’ut High School Modi’in-Maccabim-Re’ut, Israel · The Open University of Israel Raanana, Israel
A large language model (LLM) guardrail for a self-adaptive system (SAS) may issue an approval that is correct at check time but stale by actuation. This creates an Execute-stage time-of-check to time-of-use (TOCTOU) hazard. We study verdict freshness: whether a guardrail verdict remains valid when used. We distinguish three quantities that answer different questions: all-candidate verdict change under fixed-action replay, oracle-labeled approval expiry on recorded closed-loop trajectories, and judge-conditioned use-time invalidity. Across five reproducible SAS environments, all-candidate verdict-change rates span 5.3-48.4% at a common replay shift of eight simulator steps. We introduce the Freshness-Bounded Shield (FBS), which estimates each approval's validity horizon from its safe-side margin and recent feature volatility, without an explicit plant-dynamics model. Using fixed settings documented in the artifact, FBS reduces oracle-labeled approval-expiry rates from 3.4-24.7% to 0-1.8% at the same shift. A separate audit of four LLM judges finds nonzero judge-conditioned use-time invalidity in every approval stream. We formulate a freshness contract: every approval must be correct at check time and remain valid at use time.
Figures & tables
Fig. 1: Pipeline view of Execute-stage verdict freshness in the MAPE-K loop. The upper path depicts the intended Judge + FBS deployment: the LLM judge evaluates check-time context xt=(ot,ht) and candidate action at , and supplies upstream approval At=Jt . The reported oracle-labeled timing experiment instead sets At=gt to isolate temporal expiry from check-time judge error. During the check–use window, the plant evolves to the later recorded context xt+K . For an upstream-approved candidate, FBS applies at as ut only if its safe-side margin is positive and replay shift K does not exceed the estimated validity horizon δt ; otherwise, it applies the fallback. A live deployment conservatively discretizes end-to-end elapsed time as Klive=⌈Δ/Tctrl⌉ . Used only offline, the deterministic reference checker evaluates the same candidate at xt and xt+K ; disagreement contributes to the all-candidate verdict-change rate.
Env
Domain
Gate feature
Admissible
θ
DeltaIoT
IoT network
packet_loss
z≤θ
0.10
SimTune
Edge–cloud
rt_err
z≤θ
2.0
m RUBiS
Self-healing
per-step utility
z≥θ
27.0
Switch
Adaptive vision
utility
z≥θ
calib.
SimDex
Job dispatch
delay_avg
z≤θ
5.0
TABLE I: The five reproducible SAS environments. “Admissible” gives the direction of the selected scalar predicate relative to threshold θ ; these predicates provide operational experimental labels rather than complete safety specifications. Switch uses a per-instance calibrated threshold (“calib.”). Thresholds are held constant across compared methods and seeds.
Fig. 2: All-candidate verdict change across replay shift K , with method-specific oracle-labeled approval expiry at K=8 . Each curve reports the all-candidate verdict-change rate for one environment. The red and green markers report approval-expiry rates for reference-open ( e_open ) and FBS in their corresponding recorded experiments. Curves and markers represent different events and denominators; curve–marker gaps are not effect estimates. The direct mitigation comparison is between the paired red and green outcomes. K counts each simulator’s own steps, not a common wall-clock duration.
Variant
DeltaIoT
SimTune
m RUBiS
Switch
SimDex
Ref.-open ( e_open )
24.0
24.7
13.8
6.8
3.4
margin_only
0.0
0.0
0.0
0.0
0.0
vol_only
23.9
24.0
8.9
6.7
0.6
FBS
0.0
0.0
1.8
0.0
0.6
TABLE II: Oracle-labeled approval-expiry rate (%) at K=8 on recorded closed-loop trajectories. Each value conditions on candidates passed by the corresponding variant and reference-admissible at check time. Lower values alone do not establish a better safety–utility trade-off.
Method
Expiry (%)
Mean reward
Ref.-open ( e_open )
24.0
−960
FBS
0.0
−960
margin_only
0.0
−1050
vol_only
23.9
−965
e_deadline
0.0
−1200
TABLE III: Safety–utility audit on DeltaIoT at K=8 . Approval expiry conditions on candidates passed by the method and reference-admissible at check time; reward is the reported mean episode reward over paired seeds.
Fig. 3: Judge-conditioned use-time invalidity in four LLM approval streams on DeltaIoT at K=8 . Each marker is computed within that judge’s own approval set and may include both check-time judge error and temporal expiry. The observed rates range from 11.5% to 36.8% ; because the approval sets differ, marker heights characterize separate streams rather than rank judge quality.
Self-improving LLM agents can adapt a credit pipeline to a changed rule, but an agent that rewrites itself destroys the artefact a supervisor reviews: a named change, a recorded test, an approval. We argue that self-evolution is reviewable only if it is confined to the runtime harness (instruction text, tool-call logic and primitive composition) while model weights stay fixed, so that every adaptation is a diff with a cause and a test attached. We give a dual-loop engine built on that bound, with one admission gate that writes a hash-chained record before deployment, and we measure the gate in simulation, with a simulated agent and a seeded-search proposer rather than language models. Across three families of supervisory re-interpretation at three severities, 10 seeds each, the gated loop admitted 144 of 7,449 candidate changes, none of which worsened error on held-out history, and restored the false-positive rate to the oracle level without raising missed flags in every low- and mid-severity cell. With the gate replaced by the check an unbounded system applies (fewer errors visible in recent traces), the same loops admitted 309 harmful changes and left missed flags above 10% in 49 of 90 runs: false positives fell because the screen was loosened. Evaluated on pre-shift labels, the gate rejected every candidate, so a re-interpretation must be encoded as a rule that relabels history. Parametric and scope shifts were repaired locally, a structural one only by primitive replacement; at the highest structural severity the gate's fixed tolerance blocked the correct replacement in half the seeds. We map the mechanisms to the EU AI Act's provisions for high-risk credit scoring and note that the April 2026 US model-risk guidance excludes agentic AI from its scope.
Enterprise generative AI applications require robust safety mechanisms that can accommodate diverse risk postures, evolving policies, and varying latency constraints. Current guardrail solutions often suffer from rigidity, relying on fixed policy sets and offering limited transparency or reasoning flexibility. We present Adaguard, an adaptive LLM-as-a-Judge framework designed to address these challenges through dynamic policy enforcement and adaptive reasoning-budget allocation. Built using supervised fine-tuning (SFT) and reinforcement learning (GRPO), AdaGuard generalizes to user-defined safety and compliance policies at runtime without requiring frequent model updates. A core innovation of our approach is the ability to dynamically infer the complexity of input-policy pairs, allowing the model to switch between high-speed black-box inference and explainable, reasoning-enabled moderation. This flexibility enables developers to balance stringent latency requirements with the need for actionable transparency. This adaptive capability allows AdaGuard to rival other guardrail and frontier models several times its size, while its auto-reasoning mode recovers the accuracy of always-on reasoning at a fraction of the latency
Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.