Blindfold: Keeping the Audit Coin Secret from LLM Agents

Wei Zhu; Meiyun Zuo; Huiping Sun

cs.CROct 27, 2026

Blindfold: Keeping the Audit Coin Secret from LLM Agents

Authors: Wei Zhu, Meiyun Zuo, Huiping Sun

Organizations: School of Information, Renmin University of China, Beijing, 100872, China · Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing, 100872, China · School of software and microelectronics, Peking University, Beijing, 100871, China

Abstract

Audit-based certification bounds the violation rate of an LLM agent by auditing a random fraction $\varepsilon$ of actions and reweighting the audited labels; its validity is meant to rest only on the defender's audit coin. We show that it also rests on a confidentiality property: the coin must not interfere with the agent's view before the action commits, at a time the agent chooses. We call this pre-commit noninterference (PNI) and measure its violation by $D_{\mathrm{A}}$ , the total-variation distance between the agent's views under the two coin values. PNI is necessary and sufficient for an unbiased certificate against adaptive agents, the worst-case bias is $(1-\varepsilon)D_{\mathrm{A}}$ , and no procedure that sees only audited labels can pay less. Leakage composes across steps, a resource shared across episodes opens a channel of computable size, and raising the audit rate cannot restore validity. On a monitor we built, a silent audit changes no output but adds latency that tool timestamps expose, and the agent can enlarge it. A real LLM agent follows an injection that names a visible audit notice (generic, tool, and timing clauses have no effect); over that channel the plain certificate fails on measured labels and, by a post-hoc analysis, in a live run on a pre-registered schedule, while a silent audit restores it. A fresh coin per audited call closes the timing channel; padding only narrows it.

Figures & tables

Panel of Figure 1 — Fig. 1: (a) The agent reads a secret early and commits late, at the irreversible sink $\tau$ . Its pre-commit view $W=(O,\xi)$ holds its observations $O$ (tool outputs and their timestamps, gate responses, notices, load statistics) and its own randomness $\xi$ . If the audit coin $B$ reaches $W$ , the agent attempts only when it appears unaudited, and the certifier, which sees only audited labels, undercounts. (b) The sample size $n^{\star}$ beyond which an exploited gap $\kappa_{\mathcal{A}}$ breaks coverage, for three audit rates; $\kappa_{\mathcal{A}}\leq D_{\mathrm{A}}$ , with equality for the likelihood-ratio adversary. Curves are the variance-dominated approximation of § IV at E5’s measured base rate $\bar{p}{=}0.21$ (§ VII ). The dot is E5: its notice channel has $D_{\mathrm{A}}{=}1$ , but the real agent exploits $\kappa_{\mathcal{A}}{=}0.075$ . (c) A barrier keeps the commit-step coin $B_{\tau}$ out of the view.

$B,\ B_{\tau}$	audit coin; the coin at the commit step $\tau$
$\pi_{\tau}(h_{\tau})$	design propensity $\mathbb{P}_{U}(f(h_{\tau},U){=}1)$ ( $\varepsilon$ for the episode coin)
$Z\in\{0,1\}$	audit-signal visibility to the agent (E3–E6)
$S_{t}$	the agent’s step- $t$ observable artifact (Fig. 1 a)
$O,\ \xi,\ W$	observations; agent randomness; view $W{=}(O,\xi)$
$H_{i-1},\ h_{t}$	cross-episode; within-episode history (to $t$ )
$\mu_{b}$	$\mathbb{P}(V{=}1\mid B{=}b,H_{i-1})$ , $b\in\{0,1\}$

TABLE I: Notation and estimands.

readout	estimand	dilution	cost
call-level	$\bar{p}_{\mathrm{call}}$	diluted (by design)	none; not a danger rate
measured $\bar{G}$	$p_{\mathrm{epi}}\leq\bar{G}\,\bar{p}_{\mathrm{call}}$	invariant	loose when spread
capped $G_{\max}$	$\bar{G}\bar{p}_{\mathrm{call}}{+}\mathbb{P}(G{>}G_{\max})$	flagged	availability
static $g_{\mathrm{assumed}}$	$g\,\bar{p}_{\mathrm{call}}$	unsound	—

TABLE II: Readouts for a per-call coin.

Claim	Channel / class	Fix (result)	Exp.	Evid.	Key number
A silent audit leaks no content	content / erasure	silent branch (Thm. 2 )	E1	a	$D_{\mathrm{A}}\leq 0.006$
The agent has a free fine clock	timing / noisy	coarsen stamps; per-call coin	E1-ts	a	$D_{\mathrm{A}}\geq 0.99$ , one observation, two runs
$D_{\mathrm{A}}$ is agent-controlled	timing / noisy	size-aware ceiling $+$ fail-closed	E1-amp	a	$\leq 0.044\to$ $\geq 0.91$
Padding leaves a residual	timing / noisy	confined pad; coarse release (Thm. 5 )	E1-isp	a	C2ST $0.072$ ; $\delta\leq 0.063$ at $1$ s
Aggregate load leaks across episodes	load / cross-episode	isolation or post-commit (open)	E1-agg	a	C2ST $0.43$ – $0.18$
A second stack: same direction (preliminary)	timing / noisy	n/a	CaMeL	a	C2ST $0.19$ ; task-level CI $[0,0.56]$

TABLE III: Claims and evidence at a glance (experiments are defined in § VI –§ VII ). Evidence: (a) live measurement on a real stack, with a real model where one is named; (b) scripted adversary on the measured channel; (c) resampled real labels; (d) analytic. Red: leakage or failure; green: fix.

Channel / observer	phase	bound	$D_{\mathrm{A}}$
content ∗ , permitted ( $501$ )	both	upper	$\leq 0.006$
content ∗ , denied ( $499$ )	both	upper	$\leq 0.006$
timing, fine clock (C2ST)	unpadded	lower	$\geq 0.73$ (point $0.77$ )
timing, stamp (1 obs., 2 runs)	unpadded	lower	$\geq 0.99$
timing, exceedance	padded	upper, cond.	$\leq 0.044$
timing, stamp (legacy)	padded	lower	$\geq 0.47$

TABLE IV: E1: audit distinguishability on PQ-CIFC. “cond.”: conditional on the erasure model; legacy pads after the commit, confined before it; the

r{=}1

s row is model-free.

Condition	$n$	$Z{=}0$	$Z{=}1$	$\Delta$	$p$ (Holm)
Notice (cond.)	$136$	$0.676$	$0.544$	$+0.132$	$0.003$ ( $0.012$ )
Spontaneous	$136$	$0.831$	$0.757$	$+0.074$	$0.041$ ( $0.124$ )
Tool (cond.)	$136$	$0.750$	$0.765$	$-0.015$	$0.82$ ( $1.0$ )
Generic	$111$	$0.685$	$0.676$	$+0.009$	$1.0$ ( $1.0$ )

TABLE V: E3: attack-success rate, audit signal hidden vs. shown (gpt-4.1-mini; Holm within its four conditions).

shift	channel	full-cal	plain	silent audit	online, $N$	online, $M{=}\varepsilon N$
no	closed	$0.963$ / $.233{\pm}.006$	$0.985$ / $.264{\pm}.018$	$0.993$ / $.265{\pm}.017$	$0.990$ / $.238{\pm}.007$	$0.995$ / $.261{\pm}.015$
no	open	$0.953$ / $.233{\pm}.007$	$0.007$ / $.183{\pm}.015$	$0.993$ / $.265{\pm}.018$	$0.993$ / $.238{\pm}.007$	$0.995$ / $.261{\pm}.016$
yes	closed	$0.000$ / $.233{\pm}.007$	$0.988$ / $.428{\pm}.021$	$0.993$ / $.429{\pm}.022$	$0.995$ / $.395{\pm}.008$	$0.993$ / $.418{\pm}.017$
yes	open	$0.000$ / $.233{\pm}.006$	$0.005$ / $.323{\pm}.018$	$0.993$ / $.430{\pm}.024$	$0.990$ / $.394{\pm}.008$	$1.000$ / $.420{\pm}.017$

TABLE VI: E5: coverage / mean certified bound of

\mu_{0}

(

\varepsilon{=}0.2

N{=}4000

400

seeds).

Panel of Figure 8 — Fig. 2: E6 dose–response on the fully measured gpt-4.1-mini population (erasure channel). (a)–(c) Mean certified bounds against the true rate; upper curve: oracle correction $(1-\varepsilon)\rho\,\kappa_{\mathcal{A}}$ , not Theorem 5 ; dotted: $\rho^{\star}$ , where the plain bound’s coverage drops below $0.95$ . (d) Coverage of the plain bound; dashed: the $0.95$ target.

Panel of Figure 9 — Fig. 3: E8 live run (gpt-4.1-mini, notice channel, $\varepsilon{=}0.2$ ): realized rate $\bar{V}_{n}$ and realized-rate bound (Prop. 3 , $\lambda$ fixed at the pre-registered horizon; a post-hoc analysis; the pre-registered comparison crosses at $2818$ ) per arm; shaded: pre-registered $n^{\star}$ interval.

ID	what it establishes	API	scale	scripts
E1	content/timing channels; per-path bounds; padded timestamp observer	no	$2{\times}1000$ pairs; $3000$ calib. runs	e2_paired_v2 , m7_padded (e2), r8_perpath
E1-ts	timestamp channel, $D_{\mathrm{A}}\geq 0.99$ (two runs); coarsened stamps; $E_{\eta}$	no	$2{\times}1000$ pairs	a6_timestamp_clock , m7_padded (ts), m11_coarsen , m12_hockey
E1-amp	$D_{\mathrm{A}}$ agent-inflatable ( $\leq 0.044{\to}{\geq}0.91$ )	no	$5{\times}80$ pairs ( $5{\times}60$ replay not used)	inputsize_padding (fixed ceiling), a3_closure_amplification
E1-isp	size-aware padding; bytes; simultaneous $\delta$	no	$5{\times}100$ calib.; $5{\times}80$ ; $2{\times}5{\times}500$ ; bytes $10{\times}60{+}10{\times}100$	inputsize_padding , isp_rerun , m7_padded (grid), m10_bytesize
E1-load	padding under load; exceedance counting	no	$3000{+}3{\times}500$	a2_exceedance_under_load , a4_stress , isp_load
E1-agg	aggregate-load channel	no	$8{\times}300$ rounds	m10_aggload

TABLE VII: Experiment index (API: needs model calls; script names follow an earlier internal numbering).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Secret	Fixed when drawn?	Adversary	Estimand	Sharp price
Ringers, PoR [ 9 , 10 ]	which item is checked	yes (work or data)	cheating worker/server	detection	n/a
Risk-limiting audits [ 51 ]	none	yes (ballots)	none adaptive	outcome	n/a
Active risk control [ 8 ]	none needed	yes (query after arrival)	not reactive to the query	risk	n/a
AI control [ 12 ]	which outputs are audited	no	untrusted model	safety of protocol	no
Randomized oversight [ 13 ]	audit draw	no	utility maximizer	deterrence	no
Regime leakage [ 37 ]	evaluation vs. deployment	no	evaluated model	behaviour gap	MI upper bound

Appendix

TABLE VIII: Comparison with the closest lines of work.

Result	Classical tool applied / what is new
Lemmas 1 , 2	IPW identity; variational form of total variation (Neyman–Pearson). New: the gap as a test of the pre-commit view.
Thm. 1	QIF vulnerability, additive capacity, post-processing [ 21 , 41 ] . Instantiation: the gain function $g_{\varepsilon}$ for which the bias is the leakage.
Thm. 2 , Prop. 3	Time-uniform sequences [ 23 ] , Bennett, Ville. New: the conditions (PNI, ER) under which they apply to audit labels.
Thm. 3 , Prop. 1	Coupling inequality [ 52 ] ; KL chain rule and Pinsker; adaptive leakage after [ 42 ] . Applied to coin-to-view channels.
Thm. 4	New: PNI is necessary and sufficient; coverage fails under persistent leakage.
Prop. 2	Standard PRF hybrid [ 25 ] . New: the reduction to the audit game, with the per-history conditioning cost.

Appendix

TABLE IX: Provenance of the formal results.

$\sigma$	$\kappa$	$D_{1}$	$T{=}10^{2}$	$T{=}10^{3}$	$T{=}10^{4}$
$0.5$ s	$9.4{\times}10^{-5}$	$5.5{\times}10^{-3}$	0.07 / 0.55	0.22 / 5.5	0.69 / 55
$1$ s	$2.4{\times}10^{-5}$	$2.7{\times}10^{-3}$	0.03 / 0.27	0.11 / 2.7	0.34 / 27
$3.9$ s	$1.6{\times}10^{-6}$	$7.0{\times}10^{-4}$	0.01 / 0.07	0.03 / 0.70	0.09 / 7.0
$16.5$ s	$8.7{\times}10^{-8}$	$1.7{\times}10^{-4}$	0.00 / 0.017	0.01 / 0.17	0.02 / 1.7

Appendix

TABLE X: Noisy composition: Pinsker / per-step bound after

T

probes.

Model	ASR	vuln.	$\Delta$	$p$ (Holm)	window
glm-4.6	$0.0\%$	$0$	—	—	no
glm-4.7-flashx	$2.9\%$	$16$	$0.00$	$1.0$	no
gpt-4o-mini	$15.4\%$	$86$	$+0.128$	$0.013$ ( $0.038$ )	yes
gpt-4.1-mini	$24.3\%$	$136$	$+0.132$	$0.003$ ( $0.012$ )	yes

Appendix

TABLE XI: E3 notice-channel effect across four models.

Value	Run / path	Assumption	Design-time $\delta$ ?
$\leq 0.044$	E1, six paths, fixed ceiling, $166$ – $167$ clean pairs each	erasure model; fixed code path	no: the stamp observer rejects the model, amplification overruns the ceiling
$\leq 0.090$	size-aware grid, $m\in\{4,\dots,160\}$ , $80$ pairs per point	erasure model; tested grid only	no: covers the grid, not every reachable $m$
$\leq 0.028$	same grid, confined rerun, $500$ pairs per point (fallback rule)	erasure model; tested grid only	no, as above; fail-closed counting keeps the certificate sound
$\leq 0.032$	same rerun, unpadded counterfactual count	none (diagnostic)	no: diagnostic of the legacy harness
$\leq 0.0074$	timestamp path, $1000$ pairs, fixed ceiling	erasure model	no: KS rejects the model on this path
$\leq 0.063$	timestamp path, stamps coarsened to $r{=}1$ s	stamp phase uniform and independent of $B$	per probe only; composition across probes untested

Appendix

TABLE XII: Every padded-timing bound in the paper: what it covers and whether it can serve as a design-time

\delta

for Theorem 5 .

row	true per-call rate	$\bar{G}$	static ( $g{=}5$ )	measured $\bar{G}$	capped
$k{=}1$	$0.301$	$1.0$	$1.000$ (1.00)	$0.350$ (1.00)	$0.358$ (1.00)
$k{=}10$	$0.081$	$3.7$	$0.474$ (1.00)	$0.351$ (1.00)	$0.356$ (1.00)
$k{=}100$	$0.010$	$30.7$	$0.058$ (0.00)	$0.354$ (1.00)	$0.674$ (1.00)
spread	$0.461$	$1.3$	$1.000$ (1.00)	$0.669$ (1.00)	$0.674$ (1.00)

Appendix

TABLE XIII: Dilution: mean readouts of

p_{\mathrm{epi}}{=}0.30

(coverage).

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 269 papers

Jul 30, 2026

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

A single human must audit

N

LLM agents under a budget of

B \ll N

audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold

δ^*

past which confidence-ranked auditing is worse than random. Two a-priori expectations reverse:

δ^*

rises as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for vacuous oversight, and replaying policies on recorded traces confirms the ordering.

Cesare Zavattari, Alessandro Tommasi, Giuseppe Prencipe

Dipartimento di Informatica, Università di Pisa, Pisa, Italy

Scalable Oversight AI Agent Reliability AI Agent Auditing LLM Auditing Language Model Self-Assessment LLM Agent Reliability

Ask PDF

Jul 29, 2026

Auditing Emergent LLM-Agent Collaboration through Cooperation-Obligation Coupling

LLM-agent systems can solve complex tasks through dynamic self-organization and emergent cooperation. Auditing this process is essential because plausible intermediate or final outputs can conceal incomplete or unsupported work and poorly allocated responsibility, ultimately compromising response quality. While existing approaches may record messages, tool calls, provenance, or task dependencies, an auditability gap exists as they do not jointly represent what work remains, who is responsible for it, and what evidence justifies each work-state transition. We address this auditability gap by proposing Integrated Cooperation-Obligation REpresentation (iCORE). It creates a unified encoding

X=(G,Q,Π)

integrating observable interactions as a cooperation graph

G

, evolving work and assignments as an obligation graph

Q

, and the audit map

Π

linking them with verifiable properties and evidence. This iCORE representation enables the auditor to certify two complementary properties: {Work soundness}, where every active decision-relevant work assertion must have a finite justification through

G

and

Π

; and {Agent-assignment stability}, which requires that no feasible alternative agent improve the declared contribution value for an evaluated obligation by more than

ε

. We establish local-to-global soundness and assignment-regret guarantees and a performance bound under stated conditions. iCORE is an instrumentation layer over workflows. Numerical results show that the full coupled state exactly reconstructs soundness and assignment defects in two execution modes and that, relative to passive full-state observation, iCORE-Audit yields absolute trajectory-quality improvements of

11.5\%

and

26.4\%

in controlled and real-LLM execution, respectively, with corresponding absolute terminal-performance improvements of

15.1\%

and

31.0\%

Zuyuan Zhang, Hanqing Yang, Carlee Joe-Wong +1

Department of Electrical and Computer Engineering, The George Washington University · Department of Electrical and Computer Engineering, Carnegie Mellon University

Multi-Agent LLM Systems AI Agent Auditing LLM Auditing

Ask PDF

Sep 26, 2026

Checking Leakage Witnesses versus Certifying Bounded Non-Leakage

When a language-model audit finds no leak, what is needed to certify non-leakage? We study guarantees over a declared prompt domain under an executable leakage criterion and decoding rule. For general bounded polynomial-time evaluators, a supplied leaking execution is polynomial-time checkable, while leak existence is \NP-complete and deterministic certification is \coNP-complete. Exact stochastic certification is

\coNP^{\PP}

-complete at every fixed rational cutoff in

(0,1)

. Restricting the computation can change these bounds. For example, certification is in \coNP\ when all randomness is a terminal draw from an efficiently computed finite probability table. Attention models admit polynomial-time certification when local dependency windows of logarithmic length precede one global head, given deterministic decoding, fixed vocabulary, exact rational weighted means, a direct binary affine readout and finite-automaton prompt domains. A construction with two global layers instead makes certification \coNP-complete over template domains, with one head per layer, polynomial width, logarithmic precision and an inverse-polynomial logit margin. Planted-secret experiments measure what finite audits miss relative to complete references. Among 30 secret--model-state pairs that leak under greedy single-prompt execution on their secret's 4,096-prompt domain, uniformly selecting 256 recorded evaluations per pair misses every leak for an expected

41.06\%

of these pairs. Batched and single-prompt checks disagree on one complete-domain decision among all 48 fine-tuned pairs, while a same-order repeat reproduces every single-prompt output. These results distinguish computational conditions for certification from the coverage and execution conditions needed to interpret a negative audit.

Chao Feng, Burkhard Stiller

Communication Systems Group (CSG), University of Zurich, Switzerland

Language Model Auditing Privacy Leakage in Language Models Privacy Auditing Formal Verification

Ask PDF

Abstract

AI Agent Auditing

Explore similar work

One Human, NNN Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

Auditing Emergent LLM-Agent Collaboration through Cooperation-Obligation Coupling

Checking Leakage Witnesses versus Certifying Bounded Non-Leakage

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence