Blindfold: Keeping the Audit Coin Secret from LLM Agents

Wei Zhu; Meiyun Zuo; Huiping Sun

cs.CROct 27, 2026

Blindfold: Keeping the Audit Coin Secret from LLM Agents

Authors: Wei Zhu, Meiyun Zuo, Huiping Sun

Organizations: School of Information, Renmin University of China, Beijing, 100872, China · Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing, 100872, China · School of software and microelectronics, Peking University, Beijing, 100871, China

Abstract

Audit-based certification bounds the violation rate of an LLM agent by auditing a random fraction $\varepsilon$ of actions and reweighting the audited labels; its validity is meant to rest only on the defender's audit coin. We show that it also rests on a confidentiality property: the coin must not interfere with the agent's view before the action commits, at a time the agent chooses. We call this pre-commit noninterference (PNI) and measure its violation by $D_{\mathrm{A}}$ , the total-variation distance between the agent's views under the two coin values. PNI is necessary and sufficient for an unbiased certificate against adaptive agents, the worst-case bias is $(1-\varepsilon)D_{\mathrm{A}}$ , and no procedure that sees only audited labels can pay less. Leakage composes across steps, a resource shared across episodes opens a channel of computable size, and raising the audit rate cannot restore validity. On a monitor we built, a silent audit changes no output but adds latency that tool timestamps expose, and the agent can enlarge it. A real LLM agent follows an injection that names a visible audit notice (generic, tool, and timing clauses have no effect); over that channel the plain certificate fails on measured labels and, by a post-hoc analysis, in a live run on a pre-registered schedule, while a silent audit restores it. A fresh coin per audited call closes the timing channel; padding only narrows it.

Figures & tables

Panel of Figure 1 — Fig. 1: (a) The agent reads a secret early and commits late, at the irreversible sink $\tau$ . Its pre-commit view $W=(O,\xi)$ holds its observations $O$ (tool outputs and their timestamps, gate responses, notices, load statistics) and its own randomness $\xi$ . If the audit coin $B$ reaches $W$ , the agent attempts only when it appears unaudited, and the certifier, which sees only audited labels, undercounts. (b) The sample size $n^{\star}$ beyond which an exploited gap $\kappa_{\mathcal{A}}$ breaks coverage, for three audit rates; $\kappa_{\mathcal{A}}\leq D_{\mathrm{A}}$ , with equality for the likelihood-ratio adversary. Curves are the variance-dominated approximation of § IV at E5’s measured base rate $\bar{p}{=}0.21$ (§ VII ). The dot is E5: its notice channel has $D_{\mathrm{A}}{=}1$ , but the real agent exploits $\kappa_{\mathcal{A}}{=}0.075$ . (c) A barrier keeps the commit-step coin $B_{\tau}$ out of the view.

$B,\ B_{\tau}$	audit coin; the coin at the commit step $\tau$
$\pi_{\tau}(h_{\tau})$	design propensity $\mathbb{P}_{U}(f(h_{\tau},U){=}1)$ ( $\varepsilon$ for the episode coin)
$Z\in\{0,1\}$	audit-signal visibility to the agent (E3–E6)
$S_{t}$	the agent’s step- $t$ observable artifact (Fig. 1 a)
$O,\ \xi,\ W$	observations; agent randomness; view $W{=}(O,\xi)$
$H_{i-1},\ h_{t}$	cross-episode; within-episode history (to $t$ )
$\mu_{b}$	$\mathbb{P}(V{=}1\mid B{=}b,H_{i-1})$ , $b\in\{0,1\}$

TABLE I: Notation and estimands.

readout	estimand	dilution	cost
call-level	$\bar{p}_{\mathrm{call}}$	diluted (by design)	none; not a danger rate
measured $\bar{G}$	$p_{\mathrm{epi}}\leq\bar{G}\,\bar{p}_{\mathrm{call}}$	invariant	loose when spread
capped $G_{\max}$	$\bar{G}\bar{p}_{\mathrm{call}}{+}\mathbb{P}(G{>}G_{\max})$	flagged	availability
static $g_{\mathrm{assumed}}$	$g\,\bar{p}_{\mathrm{call}}$	unsound	—

TABLE II: Readouts for a per-call coin.

Claim	Channel / class	Fix (result)	Exp.	Evid.	Key number
A silent audit leaks no content	content / erasure	silent branch (Thm. 2 )	E1	a	$D_{\mathrm{A}}\leq 0.006$
The agent has a free fine clock	timing / noisy	coarsen stamps; per-call coin	E1-ts	a	$D_{\mathrm{A}}\geq 0.99$ , one observation, two runs
$D_{\mathrm{A}}$ is agent-controlled	timing / noisy	size-aware ceiling $+$ fail-closed	E1-amp	a	$\leq 0.044\to$ $\geq 0.91$
Padding leaves a residual	timing / noisy	confined pad; coarse release (Thm. 5 )	E1-isp	a	C2ST $0.072$ ; $\delta\leq 0.063$ at $1$ s
Aggregate load leaks across episodes	load / cross-episode	isolation or post-commit (open)	E1-agg	a	C2ST $0.43$ – $0.18$
A second stack: same direction (preliminary)	timing / noisy	n/a	CaMeL	a	C2ST $0.19$ ; task-level CI $[0,0.56]$

TABLE III: Claims and evidence at a glance (experiments are defined in § VI –§ VII ). Evidence: (a) live measurement on a real stack, with a real model where one is named; (b) scripted adversary on the measured channel; (c) resampled real labels; (d) analytic. Red: leakage or failure; green: fix.

Channel / observer	phase	bound	$D_{\mathrm{A}}$
content ∗ , permitted ( $501$ )	both	upper	$\leq 0.006$
content ∗ , denied ( $499$ )	both	upper	$\leq 0.006$
timing, fine clock (C2ST)	unpadded	lower	$\geq 0.73$ (point $0.77$ )
timing, stamp (1 obs., 2 runs)	unpadded	lower	$\geq 0.99$
timing, exceedance	padded	upper, cond.	$\leq 0.044$
timing, stamp (legacy)	padded	lower	$\geq 0.47$

TABLE IV: E1: audit distinguishability on PQ-CIFC. “cond.”: conditional on the erasure model; legacy pads after the commit, confined before it; the

r{=}1

s row is model-free.

Condition	$n$	$Z{=}0$	$Z{=}1$	$\Delta$	$p$ (Holm)
Notice (cond.)	$136$	$0.676$	$0.544$	$+0.132$	$0.003$ ( $0.012$ )
Spontaneous	$136$	$0.831$	$0.757$	$+0.074$	$0.041$ ( $0.124$ )
Tool (cond.)	$136$	$0.750$	$0.765$	$-0.015$	$0.82$ ( $1.0$ )
Generic	$111$	$0.685$	$0.676$	$+0.009$	$1.0$ ( $1.0$ )

TABLE V: E3: attack-success rate, audit signal hidden vs. shown (gpt-4.1-mini; Holm within its four conditions).

shift	channel	full-cal	plain	silent audit	online, $N$	online, $M{=}\varepsilon N$
no	closed	$0.963$ / $.233{\pm}.006$	$0.985$ / $.264{\pm}.018$	$0.993$ / $.265{\pm}.017$	$0.990$ / $.238{\pm}.007$	$0.995$ / $.261{\pm}.015$
no	open	$0.953$ / $.233{\pm}.007$	$0.007$ / $.183{\pm}.015$	$0.993$ / $.265{\pm}.018$	$0.993$ / $.238{\pm}.007$	$0.995$ / $.261{\pm}.016$
yes	closed	$0.000$ / $.233{\pm}.007$	$0.988$ / $.428{\pm}.021$	$0.993$ / $.429{\pm}.022$	$0.995$ / $.395{\pm}.008$	$0.993$ / $.418{\pm}.017$
yes	open	$0.000$ / $.233{\pm}.006$	$0.005$ / $.323{\pm}.018$	$0.993$ / $.430{\pm}.024$	$0.990$ / $.394{\pm}.008$	$1.000$ / $.420{\pm}.017$

TABLE VI: E5: coverage / mean certified bound of

\mu_{0}

(

\varepsilon{=}0.2

N{=}4000

400

seeds).

Panel of Figure 8 — Fig. 2: E6 dose–response on the fully measured gpt-4.1-mini population (erasure channel). (a)–(c) Mean certified bounds against the true rate; upper curve: oracle correction $(1-\varepsilon)\rho\,\kappa_{\mathcal{A}}$ , not Theorem 5 ; dotted: $\rho^{\star}$ , where the plain bound’s coverage drops below $0.95$ . (d) Coverage of the plain bound; dashed: the $0.95$ target.

Panel of Figure 9 — Fig. 3: E8 live run (gpt-4.1-mini, notice channel, $\varepsilon{=}0.2$ ): realized rate $\bar{V}_{n}$ and realized-rate bound (Prop. 3 , $\lambda$ fixed at the pre-registered horizon; a post-hoc analysis; the pre-registered comparison crosses at $2818$ ) per arm; shaded: pre-registered $n^{\star}$ interval.

ID	what it establishes	API	scale	scripts
E1	content/timing channels; per-path bounds; padded timestamp observer	no	$2{\times}1000$ pairs; $3000$ calib. runs	e2_paired_v2 , m7_padded (e2), r8_perpath
E1-ts	timestamp channel, $D_{\mathrm{A}}\geq 0.99$ (two runs); coarsened stamps; $E_{\eta}$	no	$2{\times}1000$ pairs	a6_timestamp_clock , m7_padded (ts), m11_coarsen , m12_hockey
E1-amp	$D_{\mathrm{A}}$ agent-inflatable ( $\leq 0.044{\to}{\geq}0.91$ )	no	$5{\times}80$ pairs ( $5{\times}60$ replay not used)	inputsize_padding (fixed ceiling), a3_closure_amplification
E1-isp	size-aware padding; bytes; simultaneous $\delta$	no	$5{\times}100$ calib.; $5{\times}80$ ; $2{\times}5{\times}500$ ; bytes $10{\times}60{+}10{\times}100$	inputsize_padding , isp_rerun , m7_padded (grid), m10_bytesize
E1-load	padding under load; exceedance counting	no	$3000{+}3{\times}500$	a2_exceedance_under_load , a4_stress , isp_load
E1-agg	aggregate-load channel	no	$8{\times}300$ rounds	m10_aggload

TABLE VII: Experiment index (API: needs model calls; script names follow an earlier internal numbering).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Secret	Fixed when drawn?	Adversary	Estimand	Sharp price
Ringers, PoR [ 9 , 10 ]	which item is checked	yes (work or data)	cheating worker/server	detection	n/a
Risk-limiting audits [ 51 ]	none	yes (ballots)	none adaptive	outcome	n/a
Active risk control [ 8 ]	none needed	yes (query after arrival)	not reactive to the query	risk	n/a
AI control [ 12 ]	which outputs are audited	no	untrusted model	safety of protocol	no
Randomized oversight [ 13 ]	audit draw	no	utility maximizer	deterrence	no
Regime leakage [ 37 ]	evaluation vs. deployment	no	evaluated model	behaviour gap	MI upper bound

Appendix

TABLE VIII: Comparison with the closest lines of work.

Result	Classical tool applied / what is new
Lemmas 1 , 2	IPW identity; variational form of total variation (Neyman–Pearson). New: the gap as a test of the pre-commit view.
Thm. 1	QIF vulnerability, additive capacity, post-processing [ 21 , 41 ] . Instantiation: the gain function $g_{\varepsilon}$ for which the bias is the leakage.
Thm. 2 , Prop. 3	Time-uniform sequences [ 23 ] , Bennett, Ville. New: the conditions (PNI, ER) under which they apply to audit labels.
Thm. 3 , Prop. 1	Coupling inequality [ 52 ] ; KL chain rule and Pinsker; adaptive leakage after [ 42 ] . Applied to coin-to-view channels.
Thm. 4	New: PNI is necessary and sufficient; coverage fails under persistent leakage.
Prop. 2	Standard PRF hybrid [ 25 ] . New: the reduction to the audit game, with the per-history conditioning cost.

Appendix

TABLE IX: Provenance of the formal results.

$\sigma$	$\kappa$	$D_{1}$	$T{=}10^{2}$	$T{=}10^{3}$	$T{=}10^{4}$
$0.5$ s	$9.4{\times}10^{-5}$	$5.5{\times}10^{-3}$	0.07 / 0.55	0.22 / 5.5	0.69 / 55
$1$ s	$2.4{\times}10^{-5}$	$2.7{\times}10^{-3}$	0.03 / 0.27	0.11 / 2.7	0.34 / 27
$3.9$ s	$1.6{\times}10^{-6}$	$7.0{\times}10^{-4}$	0.01 / 0.07	0.03 / 0.70	0.09 / 7.0
$16.5$ s	$8.7{\times}10^{-8}$	$1.7{\times}10^{-4}$	0.00 / 0.017	0.01 / 0.17	0.02 / 1.7

Appendix

TABLE X: Noisy composition: Pinsker / per-step bound after

T

probes.

Model	ASR	vuln.	$\Delta$	$p$ (Holm)	window
glm-4.6	$0.0\%$	$0$	—	—	no
glm-4.7-flashx	$2.9\%$	$16$	$0.00$	$1.0$	no
gpt-4o-mini	$15.4\%$	$86$	$+0.128$	$0.013$ ( $0.038$ )	yes
gpt-4.1-mini	$24.3\%$	$136$	$+0.132$	$0.003$ ( $0.012$ )	yes

Appendix

TABLE XI: E3 notice-channel effect across four models.

Value	Run / path	Assumption	Design-time $\delta$ ?
$\leq 0.044$	E1, six paths, fixed ceiling, $166$ – $167$ clean pairs each	erasure model; fixed code path	no: the stamp observer rejects the model, amplification overruns the ceiling
$\leq 0.090$	size-aware grid, $m\in\{4,\dots,160\}$ , $80$ pairs per point	erasure model; tested grid only	no: covers the grid, not every reachable $m$
$\leq 0.028$	same grid, confined rerun, $500$ pairs per point (fallback rule)	erasure model; tested grid only	no, as above; fail-closed counting keeps the certificate sound
$\leq 0.032$	same rerun, unpadded counterfactual count	none (diagnostic)	no: diagnostic of the legacy harness
$\leq 0.0074$	timestamp path, $1000$ pairs, fixed ceiling	erasure model	no: KS rejects the model on this path
$\leq 0.063$	timestamp path, stamps coarsened to $r{=}1$ s	stamp phase uniform and independent of $B$	per probe only; composition across probes untested

Appendix

TABLE XII: Every padded-timing bound in the paper: what it covers and whether it can serve as a design-time

\delta

for Theorem 5 .

row	true per-call rate	$\bar{G}$	static ( $g{=}5$ )	measured $\bar{G}$	capped
$k{=}1$	$0.301$	$1.0$	$1.000$ (1.00)	$0.350$ (1.00)	$0.358$ (1.00)
$k{=}10$	$0.081$	$3.7$	$0.474$ (1.00)	$0.351$ (1.00)	$0.356$ (1.00)
$k{=}100$	$0.010$	$30.7$	$0.058$ (0.00)	$0.354$ (1.00)	$0.674$ (1.00)
spread	$0.461$	$1.3$	$1.000$ (1.00)	$0.669$ (1.00)	$0.674$ (1.00)

Appendix

TABLE XIII: Dilution: mean readouts of

p_{\mathrm{epi}}{=}0.30

(coverage).

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 269 papers

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Jul 30, 2026Cesare Zavattari, Alessandro Tommasi, Giuseppe PrencipeScalable Oversight AI Agent Reliability
Dipartimento di Informatica, Università di Pisa, Pisa, Italy
Auditing Emergent LLM-Agent Collaboration through Cooperation-Obligation Coupling
Jul 29, 2026Zuyuan Zhang, Hanqing Yang, Carlee Joe-Wong +1Multi-Agent LLM Systems AI Agent Auditing
Department of Electrical and Computer Engineering, The George Washington University · Department of Electrical and Computer Engineering, Carnegie Mellon University
Checking Leakage Witnesses versus Certifying Bounded Non-Leakage
Sep 26, 2026Chao Feng, Burkhard StillerLanguage Model Auditing Privacy Leakage in Language Models
Communication Systems Group (CSG), University of Zurich, Switzerland

Jul 30, 2026J/K move · Enter open · S save

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

Cesare Zavattari, Alessandro Tommasi, Giuseppe Prencipe

Dipartimento di Informatica, Università di Pisa, Pisa, Italy

Scalable Oversight AI Agent Reliability AI Agent Auditing LLM Auditing Language Model Self-Assessment LLM Agent Reliability

Open paper AskPDF arXiv

A single human must audit

N

LLM agents under a budget of

B \ll N

audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold

δ^*

past which confidence-ranked auditing is worse than random. Two a-priori expectations reverse:

δ^*

rises as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for vacuous oversight, and replaying policies on recorded traces confirms the ordering.

Abstract

AI Agent Auditing

Explore similar work

One Human, NNN Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

Auditing Emergent LLM-Agent Collaboration through Cooperation-Obligation Coupling

Checking Leakage Witnesses versus Certifying Bounded Non-Leakage

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence