Clean Sheets and Checkpoints: Evidence Complexity for Certifying Sequential Agent Risk

Rohith Yanambaka Venkata; Don McBride; Tiago Sequeira; Md. Al Amin

cs.LGOct 29, 2026

Clean Sheets and Checkpoints: Evidence Complexity for Certifying Sequential Agent Risk

Authors: Rohith Yanambaka Venkata, Don McBride, Tiago Sequeira, Md. Al Amin

Organizations: Independent Researcher, United States · Cyber Defense Center, Nokia.

Abstract

Dangerous-capability evaluations often test an AI agent from prepared states at ordered milestones. A clean sheet records no successes. What it certifies about a campaign of $B$ attempts depends on what a failed attempt erases. We model the task as a chain of $k$ milestones divided into checkpoint segments and ask how many zero audit responses certify, at level $α$ , a completion probability of at most $ε$ . For segments chosen before the audit, counting possible completion windows gives a certificate for any specified budget. The bound also covers adaptive deployment processes that follow the chain's transitions, provided their success probabilities after every complete history are capped by the corresponding audit success probabilities. The audit does not test this transport assumption. In rare-risk phase interiors, each segment is either rarely completed within the budget (active) or nearly always completed (free). The minimum expected number of zero responses is asymptotic to $\log(1/α)\,(B^a/(a!\,ε))^{1/K_A}$ , where the $a$ active segments contain $K_A$ milestones and are those at least as long as a budget-dependent threshold. The certificate attains this constant. Every asymptotically optimal evaluator must divide its expected response count equally among active milestones in the limit, with a vanishing share elsewhere. Where a single segment passes from active to free, each of its milestones receives a smaller limiting share than an active milestone; a crossover-aware certificate attains the optimal constant. For selected segments with known or unqueried prefix steps, counting failed cycles can sharpen the certificate.

Figures & tables

Figure 1: Reset semantics for

k=4

milestones. A state counts the milestones passed; shaded states start a segment, so progress up to them is retained. An attempt from state

i

succeeds with probability

p_{i+1}

; a failure returns the agent to the start of its current segment (arrows; a failure at a segment start leaves the state unchanged). Right: the completion-window bound of Theorem 3.1 with every segment selected; above it, the budget factor in the rare-risk asymptotic cost (Theorem 4.6 ).

Figure 2: Zero-world evidence to certify

V_{B}\leq 10^{-4}

\alpha=1/40

with

k=4

milestones; its minimum lies between the lower benchmark

c_{\alpha}/t

and the smallest valid prospective certificate. (a) Growth exponent

\Delta\log(c_{\alpha}/t)/\Delta\log B

, cut off near

B=k

; dotted: rare-risk phase-interior exponents

a/K_{A}

\epsilon\to 0

(Theorem 4.6 ), for reference alongside the fixed-

\epsilon

curves. (b) Smallest valid certificate (Theorem 3.1 or Proposition 4.9 ) over

c_{\alpha}/t

; dashed: Theorem 3.1 alone for

(3,1)

; the dot (

B=285

) marks where the smaller unrounded cost switches from both segments to the long one. Shaded:

1\leq Bp_{*}\leq 10

for

(3,1)

, where the short segment crosses over and the lower benchmark’s growth exponent falls from

1/2

1/3

$B$	$p_{\star}$	$\tau/t$	$\tau^{\prime}/t$	$c_{\alpha}/t$	window	cycle
					responses
100	0.0600927	0.78045	0.97234	59.52	78	63
185	0.0485094	0.78238	0.98159	74.18	96	78

Table 1: Six-transition family of Corollary 5.2 ,

\epsilon=1/100

\alpha=1/40

B

counts primitive charged attempts, including the three sure transitions; responses are those at the queried milestones,

3\lceil c_{\alpha}/(3\tau)\rceil

(window) and

3\lceil c_{\alpha}/(3\tau^{\prime})\rceil

(cycle-aware). The restricted diagonal threshold

p_{\star}

lies in exact rational root brackets; other entries are binary64.

Reset rule (unit)	$B_{\max}$	Statement
Full reset	169	Analytic upper-bound certificate
Full reset	185	Computer-assisted exact-risk certificate
Full retention	10	Exact full-retention result
Full reset (trials)	166	Exact complete-trial result

Table 2: Conditional reanalysis of a four-milestone record with ten zero responses per milestone (

\alpha=1/40

\epsilon=1/100

) under the model of Section 2 ;

B_{\max}

counts primitive charged attempts, or complete trials where stated.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Unrounded certificate cost relative to the lower benchmark,

t/\tau

(window coefficient

B-5

) and

t/\tau^{\prime}

(cycle-aware coefficient

B/2

), for the six-transition family at

\alpha=1/40

and

\epsilon=10^{-2}

(thick) or

10^{-4}

(thin); dots mark the budgets of Table 1 . The limits

2^{1/3}

and

1

hold along phase-interior sequences; at fixed

\epsilon

the curves approach values near them. The oscillation at small

B

is a parity effect: most failed cycles have length two.

setting	$k$	$n$	$\epsilon$	$\alpha$	$\bar{p}$	certificate	limit	certificate	ratio
						full reset		full retention
baseline	4	10	$1/100$	$1/40$	0.0881	169	185	10	16.9
$n=5$	4	5	$1/100$	$1/40$	0.1684	15	17	6	2.5
$n=20$	4	20	$1/100$	$1/40$	0.0451	2,427	2,554	19	127.7
$n=40$	4	40	$1/100$	$1/40$	0.0228	37,061	38,116	37	1,001.6
$k=3$	3	10	$1/100$	$1/40$	0.1157	8	9	4	2.0

Appendix

Table 3: Largest certified budget

B

(primitive charged attempts) from

n

zero responses at each of

k

milestones, one parameter changed at a time; Proposition 3.3 at level

\alpha

, risk level

\epsilon

. Full reset: analytic window certificate, and the limit set by the exact risk at the symmetric point (no budget certified by this rule can exceed it). Full retention: exact certificate. Ratio: full-reset certificate over full-retention certificate. Reproduced by sensitivity.py .

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 269 papers

Limits of Statistical Assurance for Embodied and Agentic AI under Adaptive Adversaries
Oct 13, 2026Inderjeet SinghAI Assurance AI Safety Evaluation
Fujitsu Research of Europe, Slough, United Kingdom
Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents
Oct 1, 2026Ali Atiah AlzahraniAI Agent Auditing Agent Evaluation
Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents
Aug 4, 2026Zhenpeng LiIoT Intrusion Detection Cybersecurity
Guangzhou Health Science College, No. 248 Guangyuan Middle Road, Guangzhou, Guangdong 510405, China

Oct 13, 2026J/K move · Enter open · S save

Limits of Statistical Assurance for Embodied and Agentic AI under Adaptive Adversaries

Inderjeet Singh

Fujitsu Research of Europe, Slough, United Kingdom

AI Assurance AI Safety Evaluation AI Agent Monitoring AI Agent Security Benchmarks Evaluation Awareness in Language Models

Open paper AskPDF arXiv

Finite safety assessments cannot exclude an adversary that preserves unattacked behavior until certification and activates afterward. We characterize this limitation for embodied and tool-use AI. An adversary that matches the complete observation law until observable, irrevocable certification and retains intervention access afterward attains

v\mathbb{E}[(n-S_0)_+]

expected post-certification events over

n

trials, where

v

is the active event probability and

S_0

the unattacked certification time. The result covers adaptive and privately randomized evaluators. Switching at certification is exactly optimal for a fixed likelihood-ratio test under memoryless, event-increasing interventions without a binding intervention budget. A positive clean-certification requirement forces linear worst-case expected excess when intervention raises event rates over an authorized deployment horizon, even if certification is deferred to the deadline. In simulated RoCo collaboration, switching yields commanded-separation violations in 34 of 36 post-certification episodes. Preregistered randomized AgentDojo trials estimate 1.98 additional security violations per four-task continuation (95% interval [1.67, 2.29]). An adversarial language-model controller infers hidden certification from noisy evaluation receipts and times intervention accordingly. With permanent revocation, a mixture likelihood-ratio monitor bounds cumulative excess above a prespecified reference rate at square-root order, up to logarithmic factors, on every sample path. A single fixed alternative can permit linear expected excess under stationary attacks. Against a controller observing the monitor's rule and statistic, live mixture monitoring reduces violations from 12.00 to 3.81 per 24 assigned task opportunities. Revocation also reduces tasks executed.

Abstract

AI Agent Auditing

Explore similar work

Limits of Statistical Assurance for Embodied and Agentic AI under Adaptive Adversaries

Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents

Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents