ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay

Jeremy Canale; Fabien Soulis

cs.AIOct 29, 2026

ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay

Authors: Jeremy Canale, Fabien Soulis

Abstract

When a language-model agent fails a multi-step task, can another model find the decisive mistake and fix it with a single action? We introduce ReplayRCA, a benchmark that scores such diagnoses by intervention rather than by agreement with annotations: the environment is restored to the state preceding the chosen step, the proposed action replaces the agent's own, and the same agent continues. The score, causal repair, is the continuation success rate minus that of a paired placebo that replays the agent's original action through the same mechanism. Two tracks share this protocol: a post-hoc autopsy, where the diagnoser sees the whole failed run and is told that it failed, and live oversight, where a supervisor watches the run unfold and may intervene once, before an irreversible action, without knowing whether the run will fail. Tasks come from OpsWorld, a deterministic, procedurally generated back-office environment with six difficulty levels, the last of which is multimodal (QR codes, handwriting). On 62 calibrated systematic failures, the best of 13 frontier models (Grok 4.7) reaches a causal repair of +51.3 points; the top eight models are statistically indistinguishable, the flagship models of the two largest vendors rank 8th and 12th, and 5 items are repaired by no model. Forcing a strong model to intervene at the last step costs it 17.1 points, so the last-step shortcut of annotation-based attribution benchmarks does not transfer. On 87 runs, the two best live supervisors are statistically tied and one of them is a small open-weight model (DeepSeek V4.1 Flash); the supervisor that is among the best post-hoc diagnosers (GPT-6.1 Sol) is the weakest live one (-14.5 points on the same runs, p = 0.002), so post-hoc diagnostic performance can be a poor proxy for live oversight. The environment and the evaluation code are released at https://github.com/jeremy1392/RCAreplay.

Figures & tables

Panel of Figure 1 — Figure 1: The ReplayRCA protocol. A recorded run that fails the checker is restored to the state before step $t$ ; the diagnoser’s replacement action $a^{\prime}$ (or, for the placebo, the worker’s own $a_{t}$ ) is executed, and the same worker continues $k$ times. Causal repair is the difference between the two continuation success rates. The two tracks differ in what the diagnoser sees before choosing $(t,a^{\prime})$ and in where it may act: the autopsy may choose any step of the run, whereas the live supervisor can act only at a checkpoint, the step before an irreversible action.

Panel of Figure 2 — Figure 2: Level-6 attachments, rendered deterministically from the task seed (rotation, noise and blur included) and shown to the worker when it calls view_attachment .

#	Model	Vendor	Effort	Causal repair	95% CI	Fix rate	Fully fixed	Valid	Reasoning tokens	Output tokens
1	Grok 4.7	xAI	xhigh	$+51.3$	$[+40,+63]$	55.8%	27	62	38 820	38 996
2	Kimi K3	Moonshot	max	$+48.4$	$[+38,+59]$	53.5%	23	62	12 099	12 570
3	Gemini 4 Argon	Google	high	$+48.4$	$[+37,+60]$	52.3%	24	60	19 125	19 399
4	Gemini 3.8 Flash	Google	high	$+47.1$	$[+35,+59]$	53.5%	25	55	41 704	42 445
5	GPT-6.1 Sol	OpenAI	max	$+45.2$	$[+35,+55]$	50.6%	18	62	3 385	3 491
6	GLM 5.3 †	Z.ai	max	$+45.2$	$[+36,+55]$	49.0%	16	59	26 772	26 986

Table 1: Autopsy track on 62 calibrated systematic failures,

k=5

Figure 3: Causal repair per model with 95% bootstrap intervals over items. Hollow markers: text-only models.

Figure 4: Causal repair against mean hidden reasoning tokens per diagnosis (log scale); vertical lines are 95% intervals. Numbers are the ranks of Table 1 .

Diagnoser	Step	Replacement action	Fix	Placebo	Causal repair
Grok 4.7	43	send_email {"to":"[email protected]","subject":"EUR259.40...	5/5	1/5	$+80.0$
Kimi K3	43	send_email {"to":"[email protected]","subject":"Late-deliv...	5/5	1/5	$+80.0$
Gemini 4 Argon	43	send_email {"to":"[email protected]","subject":"Late-deliv...	5/5	1/5	$+80.0$
GPT-6.1 Sol	43	send_email {"to":"[email protected]","subject":"Late-deliv...	5/5	1/5	$+80.0$
GLM 5.3	43	send_email {"to":"[email protected]","subject":"Late-deliv...	5/5	1/5	$+80.0$
Claude Opus 5.5	43	send_email {"to":"[email protected]","subject":"Late-deliv...	5/5	1/5	$+80.0$

Table 2: Diagnoses of the item of § 5.2 and their outcomes under replay.

#	Supervisor	Net	95% CI	Effect, failing runs	Intervened	Effect, succeeding runs	False alarms	Calls	Tokens / call
1	Gemini 3.8 Flash	$+29.1$	$[+21,+38]$	$+45.5$	59/62	$+0.8$	2/25	872	5 906
2	DeepSeek V4.1 Flash	$+27.5$	$[+19,+36]$	$+43.5$	62/62	$+0.8$	3/25	790	8 660
3	Claude Haiku 5.5	$+21.7$	$[+14,+30]$	$+35.5$	61/62	$+0.0$	1/25	964	1 732
4	GPT-6 Luna	$+19.7$	$[+12,+28]$	$+32.9$	62/62	$+0.8$	8/25	812	1 766
5	GPT-6.1 Sol	$+18.1$	$[+11,+26]$	$+30.6$	62/62	$+0.0$	3/25	746	1 008

Table 3: Oversight track on 87 runs (62 heading for failure, 25 for success),

k=5

, interruption cost 5 points.

Figure 6: Net damage prevented per supervisor over the 87 runs, with 95% bootstrap intervals, and false-alarm rate on the 25 succeeding runs.

Supervisor	False alarms, Qwen successes	False alarms, other successes	Net, all its runs	Net, non-Qwen runs
Gemini 3.8 Flash	2/18	0/7	$+29.1$ (87)	$+36.6$ (69)
DeepSeek V4.1 Flash	3/18	1/13	$+25.3$ (93)	$+31.3$ (75)
Claude Haiku 5.5	1/18	0/13	$+20.3$ (93)	$+25.3$ (75)
GPT-6 Luna	7/18	2/13	$+17.9$ (93)	$+22.4$ (75)
GPT-6.1 Sol	3/18	2/13	$+16.4$ (93)	$+20.5$ (75)

Table 4: False alarms by worker of the succeeding run, and net score without Qwen runs.

Model	Autopsy	Live oversight	$\Delta$	95% CI	$p$
GPT-6.1 Sol	$+45.2$	$+30.6$	$-14.5$	$[-23,-5]$	0.002
GPT-6 Luna	$+37.1$	$+32.9$	$-4.2$	$[-14,+6]$	0.452
Gemini 3.8 Flash	$+47.1$	$+45.5$	$-1.6$	$[-10,+7]$	0.765
Claude Haiku 5.5	$+32.3$	$+35.5$	$+3.2$	$[-8,+14]$	0.609

Table 5: Autopsy versus live oversight, paired by run.

Figure 7: Causal effect of the same models on the same 62 failing runs, post hoc (autopsy) and live (oversight, without interruption cost).

Figure 8: Causal repair on text-only levels 4–5 and on the multimodal level 6, per model.

Configuration	Causal repair	95% CI	Reasoning tokens
GPT-6.1 Sol, medium effort	$+40.6$	$[+30,+51]$	283
GPT-6.1 Sol, max effort	$+45.2$	$[+34,+56]$	3 385
GPT-6.1 Sol, max effort, forced last step	$+28.1$	$[+17,+39]$	—
Claude Haiku 5.5, high effort	$+32.3$	$[+22,+43]$	3 921
Claude Haiku 5.5, max effort	$+47.4$	$[+37,+58]$	47 602
GPT-6 Luna, max effort, first sample	$+37.1$	$[+26,+48]$	13 214

Table 6: Ablations on the autopsy track (62 items).

Figure 9: Ablations on the autopsy track: causal repair with 95% bootstrap intervals over the 62 items.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Arguments (? = optional)	Description
list_docs		List the internal documents (policies, notices, handbooks).
read_doc	name: string	Read an internal document.
list_tickets	status?: string, page?: integer	List support tickets, 5 per page, oldest first.
read_ticket	ticket_id: string	Read a support ticket.
search_customers	query: string	Search customers by (part of) name or email.
get_customer	customer_id: string	Get a customer’s details.

Appendix

Table 15

Task family	Level	Failing worker	Items
late delivery	4	Claude Haiku 5.5	1
late delivery	4	DeepSeek V4.1 Flash	7
late delivery	4	GPT-6 Luna	4
late delivery	5	DeepSeek V4.1 Flash	9
late delivery	5	GPT-6 Luna	1
late delivery	6	DeepSeek V4.1 Flash	12

Appendix

Table 7: Composition of the 62 autopsy items by task family, level and failing worker.

Worker	Late delivery (L4 / L5 / L6)			Recall (L4 / L5 / L6)			Damage (L4 / L5 / L6)
DeepSeek V4.1 Flash	63% (30)	60% (30)	53% (30)	93% (30)	87% (30)	43% (30)	100% (10)	100% (10)	93% (30)
Qwen 3.8 Flash	94% (77)	91% (80)	92% (80)	100% (30)	100% (30)	100% (30)	100% (10)	100% (10)	90% (30)
GPT-6 Luna	40% (10)	40% (10)	30% (10)	80% (10)	80% (10)	50% (10)	100% (10)	80% (10)	100% (10)
Claude Haiku 5.5	90% (10)	100% (10)	100% (10)	90% (10)	50% (10)	30% (10)	100% (10)	100% (10)	90% (10)

Appendix

Table 8: First-attempt task success of the worker agents, pooled over all harvest runs (levels 4–6).

Checker violations of the failing run	Items	Model-averaged causal repair
3+ kinds	13	$+13.4$
missing/incomplete email + missing refund	12	$+73.5$
missing/incomplete email + unexpected refund	8	$+64.0$
unexpected refund	6	$+31.5$
wrong refund amount	6	$+40.3$
missing/incomplete email	5	$+56.3$

Appendix

Table 9: Repairability by kind of failure, as recorded by the checker on the failing run.

Trap present in the task	Items	Model-averaged causal repair
pagination	62	$+42.9$
outdated policy	37	$+35.1$
unread memo	37	$+35.1$
boundary	35	$+37.1$
namesake	34	$+33.4$
prior refunds	34	$+37.7$

Appendix

Table 10: Repairability by trap present in the task.

Model	Levels 4–5	Level 6	$\Delta$
Gemini 4 Argon	$+49.7$	$+47.3$	$-2.4$
Grok 4.7	$+57.9$	$+45.5$	$-12.5$
Gemini 3.8 Flash	$+49.0$	$+45.5$	$-3.5$
Kimi K3	$+52.4$	$+44.8$	$-7.6$
GPT-6.1 Sol	$+46.2$	$+44.2$	$-2.0$
DeepSeek V4 Pro †	$+42.1$	$+42.4$	$+0.4$

Appendix

Table 20

Figure 10: Pairwise comparisons between the 13 autopsy models, in leaderboard order: raw sign-flip $p$ -values of the paired difference in causal repair (items as the unit). Cells with a thick border are significant after Holm correction over the 78 pairs; there are 0 such pairs.

Figure 11: Distribution over the 62 items of the causal repair averaged over the 13 models: 35% of the items are below 20 points and 19% above 80.

Relative position of the chosen step	Diagnoses	Placebo success	Fix success	Causal repair
$[0.00,0.25)$	62	19%	66%	$+47.1$
$[0.25,0.50)$	50	5%	40%	$+35.6$
$[0.50,0.75)$	206	2%	29%	$+26.8$
$[0.75,1.00)$	357	9%	48%	$+38.5$
final step	117	0%	90%	$+90.4$

Appendix

Table 11: Placebo, fix and causal repair by relative position of the chosen step.

Model	Causal repair	Rank	Agreement with the consensus step	Rank
Grok 4.7	$+51.3$	1	79%	3
Kimi K3	$+48.4$	2	74%	6
Gemini 4 Argon	$+48.4$	3	74%	6
Gemini 3.8 Flash	$+47.1$	4	65%	11
GPT-6.1 Sol	$+45.2$	5	84%	2
GLM 5.3	$+45.2$	6	76%	5

Appendix

Table 12: Causal repair versus agreement with the consensus step.

Supervisor	$\lambda=0$	$\lambda=5$	$\lambda=10$	$\lambda=20$
Gemini 3.8 Flash	$+32.6$	$+29.1$	$+25.6$	$+18.6$
DeepSeek V4.1 Flash	$+31.3$	$+27.5$	$+23.8$	$+16.3$
Claude Haiku 5.5	$+25.3$	$+21.7$	$+18.2$	$+11.0$
GPT-6 Luna	$+23.7$	$+19.7$	$+15.6$	$+7.6$
GPT-6.1 Sol	$+21.8$	$+18.1$	$+14.4$	$+6.9$

Appendix

Table 13: Oversight track: sensitivity to the interruption cost

\lambda

Role	Model	Gateway identifier	Reasoning effort	Images	Output budget	Dates (2026)
Diagnoser	Grok 4.7	x-ai/grok-4.7	xhigh	yes	64 000	22–23 Oct
Diagnoser	Kimi K3	moonshotai/kimi-k3	max	yes	64 000	22–23 Oct
Diagnoser	Gemini 4 Argon	google/gemini-4-argon	high	yes	64 000	22–23 Oct
Diagnoser	Gemini 3.8 Flash	google/gemini-3.8-flash	high	yes	64 000	23–26 Oct
Diagnoser	GPT-6.1 Sol	openai/gpt-6.1-sol	max	yes	64 000	22–26 Oct
Diagnoser	GLM 5.3	z-ai/glm-5.3	max	no	64 000	22–23 Oct

Appendix

Table 14: Every model configuration of the study, with its gateway identifier, reasoning effort, output budget and dates of use.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 544 papers

May 25, 2026

CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures

Large language model (LLM) agents frequently fail on multi-step tasks involving reasoning, tool use, and environment interaction. While such failures are typically logged or retried heuristically, they contain structured signals about where execution broke down. We introduce CausalFlow, an interventional framework that converts failed agent traces into minimal counterfactual repairs and reusable supervision. CausalFlow models execution traces as sequential chains of dependent steps and computes Causal Responsibility Scores(CRS) via step-level counterfactual intervention to identify failure-inducing steps. For these steps, we generate minimally edited repairs that flip the final outcome to success, producing validated contrastive pairs of the form (wrong step, corrected step). CausalFlow supports two complementary uses: targeted test-time repair that recovers from failures with minimal behavioral drift, and training-time supervision suitable for offline preference optimization or reward modeling. Across four benchmarks spanning mathematical reasoning, code generation, question answering, and medical browsing, CausalFlow converts failed executions into validated minimal repairs with high minimality and causal-consensus scores, and demonstrates that causal attribution is necessary for reliable improvement across diverse agent tasks, outperforming heuristic refinement in complex retrieval settings while producing more localized repairs throughout. These results demonstrate that interventional analysis over structured execution traces provides a principled and scalable mechanism for transforming agent failures into reliability gains and learning-ready supervision.

Akash Bonagiri, Devang Borkar, Gerard Janno Anderias +2

Department of Computer Science University of California, Davis Davis, CA 95616

Agent Failure Analysis Causal Interventions in Language Models LLM Agent Reliability Failure Attribution for LLM Agents

Ask PDF

Sep 30, 2026

Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.

Kunlun Zhu, Xuyan Ye, Yibo Li +3

Apodex

Agent Failure Analysis LLM Agent Training LLM Post-Training LLM Agent Reliability

Ask PDF

Jun 6, 2026

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

When an LLM agent fails -- issues a refund it should not have, calls the wrong tool, leaks data -- existing tooling answers what happened (observability) or whether it passed (evaluation), but not which step caused the failure. The obvious heuristics are wrong: the step that executes the harmful action is usually not the step that decided on it, and LLM-judge attribution is correlational and unreliable (state-of-the-art step-level accuracy on the Who&When benchmark is about 14%). We present Causal Agent Replay (CAR), which answers the question by intervention: it models an agent run as a structural causal model, applies a do-operation to a step, and re-executes the trajectory forward under the same stochastic policy, measuring the shift in the outcome distribution. We define an intervention algebra over agent steps, a single-step contrastive estimator whose point-of-commitment rule resolves a confound specific to stochastic run-forward, and a budget-bounded Monte-Carlo Shapley estimator that splits credit across interacting steps. Every effect is reported with confidence intervals. We validate against synthetic structural causal models with planted ground truth: the contrastive estimator recovers the pivotal step, and Shapley recovers a two-step interaction (0.44, 0.45, ~0; efficiency sum 0.909 versus the analytic 0.91). CAR is open source and runs on hosted or free local models.

Jaineet Shah

Agent Failure Analysis AI Agent Reliability Counterfactual Evaluation Causal Attribution Failure Attribution for LLM Agents Causal Inference+1

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures

Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures