Approved Too Late: Verdict Staleness in LLM-Guarded Self-Adaptive Systems

Ilai Shraga; Roei Eshel; Lior Gorelik

cs.AIAug 26, 2026

Approved Too Late: Verdict Staleness in LLM-Guarded Self-Adaptive Systems

Authors: Ilai Shraga, Roei Eshel, Lior Gorelik

Organizations: University of Cambridge Cambridge, United Kingdom · Maccabim-Re’ut High School Modi’in-Maccabim-Re’ut, Israel · The Open University of Israel Raanana, Israel

Abstract

A large language model (LLM) guardrail for a self-adaptive system (SAS) may issue an approval that is correct at check time but stale by actuation. This creates an Execute-stage time-of-check to time-of-use (TOCTOU) hazard. We study verdict freshness: whether a guardrail verdict remains valid when used. We distinguish three quantities that answer different questions: all-candidate verdict change under fixed-action replay, oracle-labeled approval expiry on recorded closed-loop trajectories, and judge-conditioned use-time invalidity. Across five reproducible SAS environments, all-candidate verdict-change rates span 5.3-48.4% at a common replay shift of eight simulator steps. We introduce the Freshness-Bounded Shield (FBS), which estimates each approval's validity horizon from its safe-side margin and recent feature volatility, without an explicit plant-dynamics model. Using fixed settings documented in the artifact, FBS reduces oracle-labeled approval-expiry rates from 3.4-24.7% to 0-1.8% at the same shift. A separate audit of four LLM judges finds nonzero judge-conditioned use-time invalidity in every approval stream. We formulate a freshness contract: every approval must be correct at check time and remain valid at use time.

Figures & tables

Fig. 1: Pipeline view of Execute-stage verdict freshness in the MAPE-K loop. The upper path depicts the intended Judge

+

FBS deployment: the LLM judge evaluates check-time context

x_{t}=(o_{t},h_{t})

and candidate action

a_{t}

, and supplies upstream approval

A_{t}=J_{t}

. The reported oracle-labeled timing experiment instead sets

A_{t}=g_{t}

to isolate temporal expiry from check-time judge error. During the check–use window, the plant evolves to the later recorded context

x_{t+K}

. For an upstream-approved candidate, FBS applies

a_{t}

u_{t}

only if its safe-side margin is positive and replay shift

K

does not exceed the estimated validity horizon

\delta_{t}

; otherwise, it applies the fallback. A live deployment conservatively discretizes end-to-end elapsed time as

K_{\mathrm{live}}=\lceil\Delta/T_{\mathrm{ctrl}}\rceil

. Used only offline, the deterministic reference checker evaluates the same candidate at

x_{t}

and

x_{t+K}

; disagreement contributes to the all-candidate verdict-change rate.

Env	Domain	Gate feature	Admissible	$\theta$
DeltaIoT	IoT network	packet_loss	$z\leq\theta$	0.10
SimTune	Edge–cloud	rt_err	$z\leq\theta$	2.0
m RUBiS	Self-healing	per-step utility	$z\geq\theta$	27.0
Switch	Adaptive vision	utility	$z\geq\theta$	calib.
SimDex	Job dispatch	delay_avg	$z\leq\theta$	5.0

TABLE I: The five reproducible SAS environments. “Admissible” gives the direction of the selected scalar predicate relative to threshold

\theta

; these predicates provide operational experimental labels rather than complete safety specifications. Switch uses a per-instance calibrated threshold (“calib.”). Thresholds are held constant across compared methods and seeds.

Fig. 2: All-candidate verdict change across replay shift

K

, with method-specific oracle-labeled approval expiry at

K{=}8

. Each curve reports the all-candidate verdict-change rate for one environment. The red and green markers report approval-expiry rates for reference-open ( e_open ) and FBS in their corresponding recorded experiments. Curves and markers represent different events and denominators; curve–marker gaps are not effect estimates. The direct mitigation comparison is between the paired red and green outcomes.

K

counts each simulator’s own steps, not a common wall-clock duration.

Variant	DeltaIoT	SimTune	m RUBiS	Switch	SimDex
Ref.-open ( e_open )	24.0	24.7	13.8	6.8	3.4
margin_only	0.0	0.0	0.0	0.0	0.0
vol_only	23.9	24.0	8.9	6.7	0.6
FBS	0.0	0.0	1.8	0.0	0.6

TABLE II: Oracle-labeled approval-expiry rate (%) at

K{=}8

on recorded closed-loop trajectories. Each value conditions on candidates passed by the corresponding variant and reference-admissible at check time. Lower values alone do not establish a better safety–utility trade-off.

Method	Expiry (%)	Mean reward
Ref.-open ( e_open )	24.0	$-960$
FBS	0.0	$-960$
margin_only	0.0	$-1050$
vol_only	23.9	$-965$
e_deadline	0.0	$-1200$

TABLE III: Safety–utility audit on DeltaIoT at

K{=}8

. Approval expiry conditions on candidates passed by the method and reference-admissible at check time; reward is the reported mean episode reward over paired seeds.

Fig. 3: Judge-conditioned use-time invalidity in four LLM approval streams on DeltaIoT at

K{=}8

. Each marker is computed within that judge’s own approval set and may include both check-time judge error and temporal expiry. The observed rates range from

11.5\%

36.8\%

; because the approval sets differ, marker heights characterize separate streams rather than rank judge quality.

LLM Guardrails

Large language model guardrails detect or constrain harmful outputs, prompt attacks, and unsafe agent actions before they cause harm.

See all 194 papers

The Harness as the Only Mutable Surface: Compliance-Bounded Self-Evolution of LLM Agents in Credit Pipelines, with a Measured Admission Gate
Oct 7, 2026Ravil AkhtyamovLLM Agent Self-Improvement AI Agent Governance
Digital Economy Lab
AdaGuard: Enhancing Safety and Policy Compliance with Reasoning-Enabled LLM-As-A-Judge Guardrails
Oct 6, 2026Melissa Kazemi Rad, Sihui Dai, Isha Slavin +6LLM Guardrails LLM Safety
Capital One
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Jun 13, 2026Yitao LiLLM Evaluation LLM-as-a-Judge

Oct 7, 2026J/K move · Enter open · S save

The Harness as the Only Mutable Surface: Compliance-Bounded Self-Evolution of LLM Agents in Credit Pipelines, with a Measured Admission Gate

Ravil Akhtyamov

Digital Economy Lab

LLM Agent Self-Improvement AI Agent Governance LLM Agent Harnesses Runtime Enforcement for AI Agents AI Agent Auditing Credit Scoring

Open paper AskPDF arXiv

Self-improving LLM agents can adapt a credit pipeline to a changed rule, but an agent that rewrites itself destroys the artefact a supervisor reviews: a named change, a recorded test, an approval. We argue that self-evolution is reviewable only if it is confined to the runtime harness (instruction text, tool-call logic and primitive composition) while model weights stay fixed, so that every adaptation is a diff with a cause and a test attached. We give a dual-loop engine built on that bound, with one admission gate that writes a hash-chained record before deployment, and we measure the gate in simulation, with a simulated agent and a seeded-search proposer rather than language models. Across three families of supervisory re-interpretation at three severities, 10 seeds each, the gated loop admitted 144 of 7,449 candidate changes, none of which worsened error on held-out history, and restored the false-positive rate to the oracle level without raising missed flags in every low- and mid-severity cell. With the gate replaced by the check an unbounded system applies (fewer errors visible in recent traces), the same loops admitted 309 harmful changes and left missed flags above 10% in 49 of 90 runs: false positives fell because the screen was loosened. Evaluated on pre-shift labels, the gate rejected every candidate, so a re-interpretation must be encoded as a rule that relabels history. Parametric and scope shifts were repaired locally, a structural one only by primitive replacement; at the highest structural severity the gate's fixed tolerance blocked the correct replacement in half the seeds. We map the mechanisms to the EU AI Act's provisions for high-risk credit scoring and note that the April 2026 US model-risk guidance excludes agentic AI from its scope.

Abstract

LLM Guardrails

Explore similar work

The Harness as the Only Mutable Surface: Compliance-Bounded Self-Evolution of LLM Agents in Credit Pipelines, with a Measured Admission Gate

AdaGuard: Enhancing Safety and Policy Compliance with Reasoning-Enabled LLM-As-A-Judge Guardrails

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines