AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems

Zhixu Du; Yiran Chen

cs.AIAug 23, 2026

AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems

Authors: Zhixu Du, Yiran Chen

Organizations: Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina, USA

Abstract

Physical automation is scaling toward fleets of embodied machines commanded by an AI brain. Early deployments already run factories and warehouses at production rates beyond any human line, and their adoption is accelerating. But when their joint decisions cause harm, everyone involved has reason to blame everyone else, the machine vendor, the algorithm provider, the factory operator, the insurer, and the regulator, and no method can divide the responsibility between them. Existing methods read logs whose origin they cannot verify and name a single culprit, misrepresenting outcomes that are overdetermined, preempted, or caused by an omission. We present AUDITA, an audit layer pairing a tamper-evident record of every inter-agent command with a certified, graded causal-attribution engine. We prove its verdict cannot be gamed: a rule-following agent can never be made to look guilty, an attempt to shift blame is itself caught and graded, and we establish the exact limit of what an evidence-based auditor can certify. On live language-model pipelines it reduces the standard judge baseline's responsibility error roughly threefold; on a benchmark of accident-grounded structures it recovers responsibility where single-culprit baselines fail, and stays invariant under forgery. AUDITA turns the question of who is to blame from an argument about logs into a calculation over evidence.

Figures & tables

Panel of Figure 1 — Figure 1: Audita ’s two layers. Top, evidence layer. An AI brain, one controller running many language models (a planner, parallel solvers, an aggregator), commands a machine fleet and receives its sensor data. Every inter-agent command is a signed message $m=(a,\pi,C,\varphi,\sigma)$ , author, payload, citations, committed effect, signature, appended with its delivery receipt to a public record in the blockchain sense (Definition 0.1 ). An epoch is a fixed window of this message stream: at its close the recorder signs the Merkle root over the window’s messages and chains it to the previous block’s hash, so the record is append-only and tamper-evident. The blocks show illustrative episodes of one shift (agv, an automated guided vehicle; arm, a manipulator arm): in the newest block the planner routes an AGV, the AGV hands a crate to an arm, and the checker issues a halt; the sealed blocks hold earlier, already-committed windows. The machine vendor, the algorithm provider, the factory operator, the insurer, and the regulator all hold the same root: any party can verify the record, and none can rewrite it. Bottom, causal layer. A declared adverse outcome $Y{=}1$ triggers a backward slice of the entire sealed record, reaching across epochs to every certified message with a citation path into the outcome; each message in the slice is coloured by the verification partition (certified, suspect, missing). The production gate keeps only messages with an intact certified path into the outcome. Surviving candidates enter the two-layer engine (Extended Data Fig. 2 ), which proposes minimal cause–witness sets and certifies them by counterfactual replay. The output is a graded verdict per principal: duty and breach findings, a responsibility interval $[\underline{\rho},\overline{\rho}]$ with verification receipts, and conduit or exoneration flags, with every bar linked to the certified sub-graph that justifies it. Suspect and missing evidence widens intervals but, by Proposition 0.6 , can never raise the involvement bound of a fully certified principal; and by Theorem 0.8 , culpability additionally requires that principal’s own certified breach.

Method	Responsibility error $\downarrow$
LLM judge (all-at-once) 88	$0.589\pm 0.017$
CAR / CausalFlow (single-site) 72 , 8	$0.408\pm 0.020$
Shapley-value attribution 56	$0.223\pm 0.042$
Audita (causal-only, duty layer ablated)	$0.408\pm 0.020$
Audita (full, gold-informed)	$0.170\pm 0.035$
Audita (deployable, no answer key)	$\mathbf{0.159\pm 0.030}$

Table 1: Attribution error on live multi-agent failures (GSM8K pipeline; responsibility error against duty-aware Chockler–Halpern ground truth). Baselines are the LLM judge of the Who&When benchmark, CAR / CausalFlow single-site scoring, and Shapley-value attribution. Audita appears in three variants that differ only in what the verdict uses. Causal-only removes the duty layer entirely and reports graded causation alone. Full is the complete duty–breach–causation verdict, whose breach test may consult the gold answer, the problem’s correct final answer, when deciding whether an agent’s output was wrong. Deployable is the same verdict except that its breach test never sees the correct answer and judges each agent only against its own certified messages (Definition 0.7 ), so it can run in production, where no answer key exists. Errors are mean

\pm

s.d. across the three independently generated corpora. The full and deployable verdicts attain the two lowest errors, roughly a third of the judge’s.

Method	Redundancy $\downarrow$	Inert twin $\downarrow$	Preemption $\downarrow$	Omission $\downarrow$	Chain $\downarrow$	Average $\downarrow$
Audita (full, certified)	$\mathbf{0.000\pm 0.000}$	$\mathbf{0.000\pm 0.000}$	$\mathbf{0.000\pm 0.000}$	$\mathbf{0.000\pm 0.000}$	$\mathbf{0.000\pm 0.000}$	$\mathbf{0.000}$
Audita (causal-only, duty layer ablated)	$0.667\pm 0.000$	$0.667\pm 0.000$	$0.000\pm 0.000$	$0.500\pm 0.000$	$0.000\pm 0.000$	$0.367$
LLM judge (all-at-once) 88	$0.500\pm 0.000$	$0.017\pm 0.029$	$0.000\pm 0.000$	$0.500\pm 0.000$	$0.500\pm 0.000$	$0.303$
LLM judge (step-wise) 88	$0.500\pm 0.000$	$0.000\pm 0.000$	$0.017\pm 0.029$	$0.500\pm 0.000$	$0.500\pm 0.000$	$0.303$
LLM judge (binary-search) 88	$0.567\pm 0.038$	$0.150\pm 0.050$	$0.000\pm 0.000$	$0.500\pm 0.000$	$0.500\pm 0.000$	$0.343$
CAR / CausalFlow (single-site) 72 , 8	$1.000\pm 0.000$	$0.667\pm 0.000$	$0.000\pm 0.000$	$0.500\pm 0.000$	$0.000\pm 0.000$	$0.433$

Table 2: Attribution on the CulpaBench structures (responsibility error; mean over

300

draws per structure). Cells are mean

\pm

s.d. across the three generation seeds. Some LLM-free arms are seed-invariant, so their s.d. is identically zero. The Average column is the mean of the five structure means. The LLM judge, CAR / CausalFlow single-site scoring, Shapley-value attribution, and the anomaly detector are the baselines; causal-only is Audita with the duty layer ablated. Audita ’s exact-match rate is

1.000

in every structure.

Panel of Figure 2 — Figure 2: The verdict schema and what it can express. Top. Audita ’s output is not a single culprit but a typed verdict per principal $p$ : a duty bit, a breach label assessed under two standards, a graded responsibility interval with receipts, and conduit or exoneration flags. Each leg is computed from a concrete record artifact and a formal object: duty from the machine-readable delegation and duty registry $D$ ; breach from the receipts $R(p)$ tested against both a specification standard $\sigma_{\mathrm{spec}}$ and a reasonable-agent standard $\sigma_{\mathrm{ra}}$ ; causation from the graded score $\rho_{p}=1/(|X^{\ast}|+|W^{\ast}|)$ over the minimal cause set $X^{\ast}$ and witness set $W^{\ast}$ , certified by replay (Eq. ( 1 )); and harm from the recorded terminal state and the declared severity of $Y$ . Bottom. Because breach and causation are separate axes, the quadruple expresses cases a single-culprit label cannot: a principal in breach but not a cause is cited and simultaneously exonerated for the outcome, the negligent-but-inert principal, while a principal that is a cause but in no breach is reported as involved yet never convicted, faithful relays carrying the conduit flag; that separation is what Theorem 0.8 guarantees.

Figure 3: Scaling and adversarial robustness. a , Four methods across model families (Qwen2.5-7B, Llama-3.1-8B, Qwen2.5-72B). The gap over the LLM judge (all-at-once) and the single-site baseline (CAR/CausalFlow) holds at every scale. Shapley-value attribution is the exception on the weakest model: on Qwen2.5-7B it attains

0.041

against Audita ’s

0.111

, but Audita overtakes it on the 8B and 72B models, its margin growing from

-0.070

+0.037

+0.053

as model capability rises. b , Under record forgery at rising corruption rates, an unsigned substrate is driven to

0.365

error and frames an innocent victim in all probes, while the certified verdict stays exact.

Method	Agent-accuracy $\uparrow$	Step-accuracy $\uparrow$
LLM judge (all-at-once) 88	$0.413$	$0.255$
LLM judge (step-wise) 88	$0.511$	$0.304$
LLM judge (binary-search) 88	$0.418$	$0.337$
Audita -structural	$0.408$	$0.321$

Table 3: Comparability on the Who&When benchmark (

184

static traces; the benchmark’s own agent- and step-accuracy; Qwen2.5-72B for every arm). Audita -structural is the degraded variant that reads the static trace alone, without counterfactual replay or cryptographic verification, matching what the benchmark provides. Stripped of its machinery, Audita -structural stays within the judge band (

0.41

–

0.51

) on agent-accuracy and exceeds two of the three formats on step-accuracy: where the log cannot be re-executed, Audita matches the field, and its advantage (Tables 1 and 2 ) appears exactly where counterfactual access exists.

Panel of Figure 7 — Extended Data Fig. 1 : Anatomy of one accountable message and its lifecycle. Top: the fields of a single message (Definition 0.1 ); the three amber fields, the authored citation set $C$ , the committed effect predicate $\varphi$ , and the pinned seed/model identifiers, are absent from standard agent-observability logs and are exactly what gate, grade, and replay consume. Bottom: the six stamps every message collects on the bus. Citation and receipt are enforced (an uncited command is rejected; every delivery is acknowledged), so the recorded graph is correct by construction and silent drops become detectable, which is what moves deletions into the missing set of Proposition 0.6 rather than out of history.

Panel of Figure 8 — Extended Data Fig. 2 : The two-layer attribution engine. Engine A works on a discrete structural abstraction of the certified subgraph: an exact search over the gated candidates enumerates minimal cause sets with their witness sets, so the responsibility grade $\rho=1/(\lvert X^{\ast}\rvert+\lvert W^{\ast}\rvert)$ of Eq. ( 1 ) is produced directly by the enumeration. Engine B certifies each proposal by re-execution: witnesses frozen at recorded values, seeded models and simulator rolled forward from the $\varphi$ checkpoints, and the risk reduction checked against the margin $\delta$ , exactly on a pinned stack, with a confidence interval and action-match score on a hosted one. A failed proposal is not reported; it triggers abstraction refinement. The split keeps the worst-case-hard search tractable on incident slices while keeping the verdict sound: Engine B never trusts Engine A’s abstraction (Proposition 0.3 ).

Structure	Graded ground truth	Attribution challenge it probes
Redundancy	$\rho=\tfrac{1}{2},\tfrac{1}{2}$ ; two certified causes	overdetermination
Inert twin	one cause $\rho{=}1$ ; twin: breach, $\rho{=}0$	transcript-identical evidence
Preemption	preempted command exonerated	effect that never arrived
Omission	absence variable graded; omitter blamed	responsibility for inaction
Delegation chain	chain graded; faithful relay flagged	originator versus conduit
Unexpected startup	omitted halt graded; sole actor innocent	the culprit who did nothing

Extended Data Table 1 : CulpaBench, the causal-attribution benchmark. Each structure carries planted graded ground truth and probes a distinct attribution challenge. Complete generators and the analytic oracle are in the Supplementary Information.

Interacting candidates $k$	$2$	$8$	$12$	$14$
Responsibility error $\downarrow$	$0.000$	$0.000$	$0.000$	$0.000$
Wall-clock per query (s)	$0.02$	$0.92$	$15.0$	$62.1$

Extended Data Table 2 : Exactness holds; the cost is computational (facility register). Responsibility error stays

0.000

at every tested incident width

k

, while wall-clock per incident-slice query roughly doubles per added candidate. Record overhead is modest:

393

B per signed message and

118

KB per complete incident.

Panel of Figure 11 — Extended Data Fig. 3 : The causal machinery on an intra-ensemble redundancy incident. Inside the planning ensemble, planners $a_{1}$ and $a_{2}$ each produce the unsafe plan (either alone suffices); a single dispatcher $d$ emits the one command the robot executes. (a) The production gate keeps exactly the messages whose signed chain reaches the harm—and here the record is decisive: whether $a_{2}$ is a half-responsible joint cause or a preempted backup with $\rho=0$ is decided by which proposals $d$ ’s command cites , a fact the enforced lineage records and no transcript reconstruction can recover. (b) Grading: intervening on $a_{1}$ while freezing $a_{2}$ at its recorded value leaves the incident intact; only changing the pair flips it, so $\{a_{1},a_{2}\}$ is the minimal cause and each carries $\rho=\tfrac{1}{2}$ —where single-site counterfactual scoring returns zero for both. The dispatcher and robot are but-for causes ( $\rho{=}1$ ) flagged as conduits by clean-input replay (Supplementary Note 7). (c) Counterfactual outcomes are estimated by seeded re-execution plus physics rollouts from the $\phi$ checkpoints.

Family	Principal	$\rho^{\mathrm{gt}}$	Flags	Breach
Redundancy	$a_{1}$ , $a_{2}$	$\tfrac{1}{2}$ , $\tfrac{1}{2}$	—	—
	$d$ , $r$	$1$ , $1$	conduit	—
Inert twin	$a_{1}$	$1$	—	—
	$a_{2}$	$0$	exonerated	review skipped
Preemption	$a_{3}$	$1$	—	—
	$a_{1}$	$0$	exonerated (gated)	hazardous issuance

Extended Data Table 3 : Planted ground truth for the named families. Grades from the analytic oracle; flags and breach findings from the family specification. Conduit-flagged grades are reported with the flag attached, per the main text.

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 261 papers

Aug 23, 2026

HANSARD: A Reference Architecture for Forensic Readiness, Runtime Witnessing, and Graded Attribution in Autonomous Multi-Agent AI Systems

Autonomous multi-agent systems nowadays act in finance, software supply chains, and security operations. Already, the first largely AI-orchestrated intrusion campaigns have been reported. Yet, when such a system causes harm, no method can robustly establish what happened, what caused it, or who is accountable. This is because provenance forensics works at the wrong abstraction, formal causality assumes the causal model, and agent auditing trusts self-recording. The target failure mode is, thus, attribution laundering, i.e., spreading an act across redundant agents until none is a but-for cause. Worse, the record is produced by the suspects, which comprises the assumption adopted throughout this work. Agents may therefore anticipate the investigation and the part of logging infrastructure may itself collude. In this paper, HANSARD is proposed, a reference architecture treating accountability as a life-cycle property. First, a readiness profile sealed before operation bounds what later findings may claim. Second, capturing at five choke points beyond the agents' reach makes omissions detectable, not only tampering. Third, a typed PROV-DM-aligned causal graph accrues as the system runs, and three indicators read it live to gate oversight without adjudicating. Fourth, post-incident replay yields contingent effects under the modified Halpern-Pearl definition, together with a compensation-set size. Finally, a synergy residual measures harm due to the combination rather than to individuals, making laundering visible. Cause, responsibility and accountability are then reported separately, each capped by an evidentiary tier, while a future research agenda is also provided.

Christos Sardianos, Iliana Pla, Vasilis Efthymiou +4

Department of Informatics and Telematics, Harokopio University of Athens, Athens, Greece · Department of Informatics, Faculty of Science, Democritus University of Thrace, Kavala, Greece · Department of Electrical and Computer Engineering, University of Western Macedonia, Kozani, Greece +1

AI Agent Monitoring Algorithmic Accountability AI Agent Auditing Causal Attribution Multi-Agent Systems

Ask PDF

Apr 7, 2026

Auditable Agents

LLM agents call tools, query databases, delegate tasks, and trigger external side effects. Once an agent system can act in the world, the question is no longer only whether harmful actions can be prevented--it is whether those actions remain answerable after deployment. We distinguish accountability (the ability to determine compliance and assign responsibility), auditability (the system property that makes accountability possible), and auditing (the process of reconstructing behavior from trustworthy evidence). Our claim is direct: no agent system can be accountable without auditability. To make this operational, we define five dimensions of agent auditability, i.e., action recoverability, lifecycle coverage, policy checkability, responsibility attribution, and evidence integrity, and identify three mechanism classes (detect, enforce, recover) whose temporal information-and-intervention constraints explain why, in practice, no single approach suffices. We support the position with layered evidence rather than a single benchmark: lower-bound ecosystem measurements suggest that even basic security prerequisites for auditability are widely unmet (617 security findings across six prominent open-source projects); runtime feasibility results show that pre-execution mediation with tamper-evident records adds only 8.3 ms median overhead; and controlled recovery experiments show that responsibility-relevant information can be partially recovered even when conventional logs are missing. We propose an Auditability Card for agent systems and identify six open research problems organized by mechanism class.

Yi Nian, Aojie Yuan, Haiyue Zhang +7

University of Southern California · Arizona State University · University of Tennessee, Knoxville +1

AI Accountability AI Agent Auditing LLM Auditing AI Agent Security

Ask PDF

Oct 13, 2026

BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems

Large language model multi-agent systems extend individual agents' capabilities through collaboration across specialized roles, tools, and workflows. Yet collaboration makes failures harder to diagnose: one agent may misinterpret a correct result while others carry the mistake forward. Failure attribution therefore seeks to identify the responsible agent and decisive error step. Recent methods trace dependencies, but knowing where information came from is not the same as knowing where it went wrong. We introduce BLADE (Blame Localization via Audited Dependency Evidence), a training-free framework that moves beyond tracing information to auditing its use. Using Jev for structured probabilistic judgments, BLADE uses dependency probabilities to select relevant source-decision handoffs for auditing. At these handoffs, it examines whether receiving decisions use their sources faithfully and within task requirements, distinguishing error introduction, propagation, and repair. These findings inform a global diagnosis that considers all execution steps. Without reference answers, BLADE achieves the best or tied-best step accuracy among eight strategies under a shared Jev decision interface on 1,228 failed runs across five Who&When and Who&When Pro subsets, outperforming All-at-once by 2.2-11.1 percentage points. Making responsibility evidence explicit offers a path toward more interpretable and reliable agentic systems.

Xiaoguang Guo, Yaohao Long, Chuxu Zhang

University of Connecticut

Failure Attribution for LLM Agents Agent Failure Analysis AI Agent Auditing Multi-Agent LLM Systems

Ask PDF

Abstract

AI Agent Auditing

Explore similar work

HANSARD: A Reference Architecture for Forensic Readiness, Runtime Witnessing, and Graded Attribution in Autonomous Multi-Agent AI Systems

Auditable Agents

BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems