Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Kunlun Zhu; Xuyan Ye; Yibo Li; Cheng Qian; Beibin Li; Heng Ji

cs.AISep 30, 2026

Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Authors: Kunlun Zhu, Xuyan Ye, Yibo Li, Cheng Qian, Beibin Li, Heng Ji

Organizations: Apodex

Abstract

An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.

Figures & tables

\aedoriginalfigurelabel : Failure reuse: correction, diagnosis learning and recovery. (a) Executed corrections outperform their matched original-action controls. (b) AED-trained Qwen3-8B leads the shown references in internal exact-step teacher-label agreement. (c) Supplied-location assistance improves recovery over replay. Panels use separate cohorts; protocols and uncertainty: Section 5 and Appendices J , H.2 .

\aedoriginalfigurelabel : AET construction and training views. Natural failures yield diagnoses, corrections and optional replay evidence. Coverage, historical error profiles and experiments use separate cohorts. Replay bars use first/selected proposals ( $n=3{,}062$ each) and restored-harness selected proposals ( $n=871$ ), each with its own control; they measure correction utility. “Step DPO” denotes an offline action-preference pilot (Appendix D.1 ).

Work	Errors	Envs	Harn.	Models	Records	Step	Evid.	Exec. fix	Ctrl.	Train	Agree.
Who&When ( Zhang et al., 2025b )	nat.	2	2	–	184	$\checkmark$	$\times$	$\times$	$\times$	–	–
Who&When Pro ‡ ( Liu et al., 2026 )	inj.	26	15	–	12,326	$\checkmark$	$\times$	$\times$	$\times$	–	$\kappa$ .73
AgenTracer ( Zhang et al., 2026a )	nat.+inj.	6	6	–	${>}$ 2,000	$\checkmark$	$\times$	$\circ$	$\times$	D	–
AEGIS ( Kong et al., 2026 )	inj.	6	6	1	9,533	–	$\times$	$\times$	$\times$	D	$\kappa$ .81
MAST ( Cemri et al., 2026b )	nat.	8	7	6	1,642	$\times$	$\times$	$\circ$	$\times$	–	$\kappa$ .88
TRAIL ( Deshpande et al., 2025 )	nat.	2	2	2	841	$\checkmark$	$\checkmark$	$\times$	$\times$	–	–

\aedoriginaltablelabel : Failure-record resources for LLM agents. Errors: natural (nat.), injected (inj.) or synthetic (syn.). Envs/Harn./Models: environments, harnesses and generating models. Records: each work’s units (AED: error–diagnosis pairs). Step: localization; Evid.: trace evidence; Exec. fix: executed correction; Ctrl.: same-checkpoint original-action control. Train: debugger (D), agent (A); Agree.: human agreement (AED: raw step,

59/69

; metrics differ).

\circ

partial ; “–” not stated. AED counts use one audited collection snapshot. ‡ Counts include multimodal traces; AED is text-only. Agreement definitions and details: Appendix C.3 .

\aedoriginalfigurelabel : Error–diagnosis pair composition, current audit. Both panels include all

50{,}228

source-linked pairs. (a) Environment shares. (b) Harness counts; small sources are grouped. Across this cohort,

33

environments and

19

harness families meet the ten-source-task floor. These are collection shares, not failure rates or training-admission rates (Appendix C.2 ); the full distributions appear in Appendix B.6 .

Panel of Figure 5 — \aedoriginalfigurelabel : Failure profiles across harnesses. Historical machine labels: $2{,}092$ of $2{,}604$ trajectories classified. The nine displayed groups contain $2{,}588$ trajectories ( $2{,}086$ classified); $16$ trajectories fall outside these groups. Bars show conditional family shares; coverage retains abstentions. Task mixtures differ across harnesses, so this is an exploratory subset comparison, not a causal effect or the full-collection distribution (Appendix G.1 ).

\aedoriginalfigurelabel : Diagnosis learning and training scale. (a) Exact-step agreement with internal teacher labels against prompted references. (b) Four nested training sizes, each with three seeds: means

\pm

sample SD, individual runs (open points) and untrained base (dashed line). All scores are case-level micro averages against recorded teacher labels on the same

943

cases. Checkpoint selection varies across sizes; protocols, score provenance and complete results: Appendix I.11 and Table 48 .

(a) Who&When: agent / exact step (%)
Model or training resource	HC	HC + gold	AG	AG + gold
Qwen3-8B base	$56.90/20.69$	$51.72/18.97$	$56.35/\mathbf{37.30}$	$55.56/\mathbf{38.89}$
AED diagnosis SFT	$50.00/15.52$	$44.83/13.79$	$23.81/15.87$	$11.11/9.52$
$+$ answer-format diversity	$\mathbf{70.69}/\mathbf{24.14}$	$\mathbf{68.97}/\mathbf{22.41}$	$\mathbf{59.52}/34.92$	$\mathbf{58.73}/34.13$
(b) AgentErrorBench: exact step / step+module (%)
Model or training resource	ALFWorld	WebShop	GAIA	Env. macro

\aedoriginaltablelabel : Public attribution under a shared prompt. All-case percentages for the

948

-task, single-seed student and its answer-format continuation. HC/AG: hand-crafted/algorithm-generated; gold is the task answer. Budget-forced continuation is enabled. Protocols and comparison scope: Appendix I.5 .

	Who&When
Model or training resource	HC	HC + gold	AG	AG + gold	AEB
Prompted references, same unified attribution prompt
Gemini 3.7 Flash	$25.86/8.62$	$25.86/10.34$	$38.10/26.19$	$38.89/27.78$	$21.50/25.00$
GPT-5.6 Sol	$51.72/\mathbf{25.86}$	$58.62/\mathbf{34.48}$	$\mathbf{67.46}/\mathbf{55.56}$	$\mathbf{66.67}/\mathbf{49.21}$	$18.50/25.50$
Claude Opus 5	$43.10/24.14$	$50.00/22.41$	$49.21/32.54$	$48.41/33.33$	$16.50/23.50$
Claude Sonnet 5	$56.90/20.69$	$60.34/32.76$	$65.87/49.21$	$65.87/48.41$	$\mathbf{23.00}/\mathbf{27.50}$

\aedoriginaltablelabel : Frontier references under a shared prompt. All-case responsible-agent / exact-step accuracy (%) on the same frozen cases, first call only, with no continuation responses scored. Bold marks per-metric column maxima. Student first calls come from the run in Table 2 . Run provenance, repeat variation and exclusions: Appendix V.1 .

\aedoriginalfigurelabel : Repair-containing recipes: gains and losses across environments. All

18

paired percentage-point contrasts against success-only SFT; filled markers and

*

indicate post-hoc Holm-adjusted significance across tasks, conditional on one training seed. Task pools and exposure differ. Planned denominators include run errors. Absolute scores and the untrained base: Appendix X.2 .

Appendix figures & tables64 assets

Supplementary material from the paper’s appendix.

Appendix

Measure	Count
Error–diagnosis pairs	$50{,}228$
Distinct source blobs	$38{,}278$
Environment-scoped source tasks	$9{,}961$
Environments / harness families / policy models	$33/19/23$
Nonempty environment–harness cells	$280$
Source blobs with multiple diagnoses	$9{,}846$

Appendix

\aedoriginaltablelabel : Current collection coverage and multiplicity. Pair, source-blob and task counts describe different units. Blob identities do not establish independent executions.

\aedoriginalfigurelabel : Two views of corrective evidence. (a) First-or-only proposal and original-action retry on

3{,}062

paired attempts; all four outcomes remain visible. (b) Adjacent attributed-step transitions across three development waves, with each wave’s transition count. Transitions within an attempt are dependent. The panels use separate dated cohorts and aggregate reports; they do not describe the final collection or assign validated root causes.

\aedoriginalfigurelabel : Where the current collection has support. Rows and columns include all $33$ environments and $19$ harness families, ordered by pair count. The logarithmic color scale exposes both large and small cells. Gray cells contain no pairs; they do not establish that the combination is unsupported by the harness.

\aedoriginalfigurelabel : The same historical labels, viewed along two axes. Bars normalize by classified trajectories; adjacent counts retain abstentions in the denominator. We display groups with at least

20

attempted labels. All model groups meet that floor; the environment panel omits

87

attempts in smaller groups. This selected cohort includes simplified and excluded environments. Different task mixtures and selective abstention prevent model rankings or estimates of causal environment effects.

\aedoriginalfigurelabel : Failure symptoms as a share of all observed executions. Each failed execution contributes to one mechanical category; successful executions remain in the denominator. End labels show failed/observed counts. The historical harness implementations predate the typed-tool-schema and native-stop fixes, so this figure describes that development cohort, not current harness quality.

Rank	Environment	Pairs	Share (%)	Source tasks
1	BFCL	5,521	10.99	470
2	ALFWorld	4,838	9.63	1,431
3	AgentBench-DB	4,407	8.77	1,120
4	Mind2Web (static)	4,050	8.06	527
5	AgentBench-OS	3,309	6.59	485
6	$\tau$ -bench retail	3,134	6.24	324

Appendix

\aedoriginaltablelabel : Complete Figure 3 (a): all 33 environments. A dagger marks a category included in an Other group in the main figure. Each share uses all

50,228

pairs as its denominator.

Rank	Harness family	Pairs	Share (%)	Source tasks
1	ReAct (single)	16,646	33.14	4,622
2	Native tool calls	9,614	19.14	3,231
3	smolagents	4,614	9.19	1,899
4	AutoGen	4,368	8.70	1,610
5	OpenHands	4,329	8.62	1,651
6	LangGraph	3,873	7.71	1,406

Appendix

\aedoriginaltablelabel : Complete Figure 3 (b): all 19 harnesses. A dagger marks a category included in an Other group in the main figure. Each share uses all

50,228

pairs as its denominator.

Population	Counting unit and size	Use in this paper
Current collection	$50{,}228$ pairs over $38{,}278$ source blobs and $9{,}961$ source tasks	Source-linked collection; separate from the frozen training release
Collection inventory	$15{,}338$ configuration records with steps and a trace reference	Collection coverage; not an admitted training split
Panel annotation audit	$11{,}116$ attempts; $6{,}381$ pass citation grounding	Annotation validity under automated checks
Frozen diagnosis release	$4{,}319$ rows over $2{,}309$ source tasks in $15$ environments (train, development, holdout); $418$ more rows over $121$ previously exposed tasks ship as an exploratory split	The evaluated dataset; a separate, earlier snapshot
Pilot diagnosis split	$359$ train, $201$ development, $169$ holdout rows	Source for the three-seed pilot ( $168$ shared source tasks per arm)
Final paired cohort	$948$ tasks in $133$ families of the $3157$ -row training split ( $1678$ tasks); $943$ held-out rows over $496$ tasks	Final-scale comparison (Table 25 )

Appendix

\aedoriginaltablelabel : The populations answer different questions; attempt, configuration and source-task counts are not interchangeable, and the annotation audit does not enlarge any training cohort.

\aedoriginalfigurelabel : Three separate measurements, not one pipeline: annotation admission, paired attribution SFT on the pilot holdout, and panel versus solo citation grounding. Each panel has its own denominator and the populations are not successive stages of one funnel.

Question (majority of three seats)	Majority verdict			Split	Unanimous	Fleiss $\kappa$
Attributed step is a defensible decisive step	yes $76$	alternative $26$	no $12$	$6$	$70$	$0.44$
Quoted evidence supports the claim	supports $28$	partially $85$	does not $4$	$3$	$37$	$0.13$
Proposed replacement is plausibly better	yes $82$	unclear $16$	no $17$	$5$	$81$	$0.52$
All three seats: step defensible and evidence at least partial $68$ of $120$ ; step yes and evidence supports $9$ of $120$ (Wilson $95\%$ $4.0$ to $13.6\%$ )

Appendix

\aedoriginaltablelabel : Blinded model-panel adjudication of the

120

-record semantic sample. Majority of three seats per question; Split means no majority; Unanimous counts records on which all three seats gave the same verdict.

Stage	Train	Development	Holdout
Input records	$3{,}157$ ( $1{,}678$ )	$219$ ( $135$ )	$943$ ( $496$ )
After deterministic checks	$283$ ( $208$ )	$20$ ( $12$ )	$87$ ( $70$ )
After semantic filtering	$53$ ( $46$ )	$9$ ( $6$ )	$18$ ( $14$ )

Appendix

\aedoriginaltablelabel : Quality-control ledger. Rows (source tasks) of the frozen release’s evaluated splits that remain after each stage of the fifth rubric version. These are retrospective checks, not the training and evaluation inputs used for the reported model results. The separate historical human audit is reported in Table 44 ; its sampled strata do not form another stage of this split-specific ledger.

Prompt	Both labelled	Mode $\kappa$	Family $\kappa$	Malformed (judge B)
Seed	275	0.258	0.186	88
Rubric	266	0.327	0.285	104

Appendix

\aedoriginaltablelabel : Rubric trial on 387 common items. Agreement conditions on both judges assigning a label within each arm, yielding different valid subsets; malformed responses are judge B parse failures on the common cohort, not semantic abstentions.

			Our holdout ( $n{=}943$ )	AgenTracer test ( $n{=}790$ )
Training resource	Tasks	Seeds	Step	Step	Unit $+$ step
Qwen3-8B base	$0$	—	$48.04$	$30.76$	$21.14$
AED (ours)	$1{,}656$	$3$	$48.14\pm 0.42$	$31.65\pm 0.33$	$25.78\pm 0.19$
AgenTracer released data v1.0.0	$1{,}656$	$3$	$45.71\pm 0.11$	$40.72\pm 0.48$	$36.16\pm 0.48$
$\Delta$ ( AED $-$ AgenTracer data), paired			$+2.44$ [0.21, 4.81]	$-9.07$ [-12.41, -5.70]	$-10.38$ [-14.05, -6.96]

Appendix

\aedoriginaltablelabel : Resource against resource under one base, recipe and scorer: micro exact step (%), and unit

+

step only on AgenTracer’s test split (our held-out cases share one gold agent). Unparseable answers and prompts over the serving window (

1

held-out and

16

AgenTracer test rows) are misses; trained rows are mean

\pm

sd over three seeds.

\Delta

: seed mean of the paired contrast, with the widest per-seed

95\%

bootstrap interval, not an interval for the mean.

Environment	Rows	Source tasks	Harness families	Policy models	Paired replay attempts
ALFWorld	$958$	$763$	$9$	$5$	$205$
AppWorld	$438$	$54$	$9$	$7$	$291$
AgentBench-DB	$426$	$342$	$9$	$5$	$530$
BFCL	$414$	$189$	$9$	$5$	$579$
Mind2Web (static)	$395$	$243$	$9$	$10$	—
$\tau$ -bench retail	$348$	$113$	$9$	$6$	$497$

Appendix

\aedoriginaltablelabel : The frozen v14 diagnosis release by environment. Rows and source tasks are counted over the three evaluated splits; harness families and policy models are distinct values of the stored provenance fields; paired replay attempts are the replay package’s count for the environment (Table 13 ) and “—” means that environment has none.

Environment	Pairs	Orig. pass	Prop. pass	$\Delta$	Disc. $P{:}O$	McNemar $p$
BFCL	$579$	$24.0$	$64.2$	$+40.2$	$248{:}15$	$10^{-55}$
AgentBench-DB	$530$	$10.2$	$55.1$	$+44.9$	$242{:}4$	$10^{-66}$
$\tau$ -bench retail	$497$	$22.7$	$75.3$	$+52.5$	$274{:}13$	$10^{-65}$
AgentBench-OS	$458$	$10.7$	$89.5$	$+78.8$	$365{:}4$	$10^{-102}$
$\tau$ -bench airline	$358$	$3.1$	$45.5$	$+42.5$	$159{:}7$	$10^{-38}$
AppWorld	$291$	$5.2$	$13.7$	$+8.6$	$38{:}13$	$10^{-4}$

Appendix

\aedoriginaltablelabel : Search-selected continuation success by environment. Orig. and Prop. are the pass rates of the original and proposed action; Disc.

P{:}O

counts pairs where only the proposal or only the original passed;

p

is the attempt-level exact McNemar test. These diagnostic

p

values assume independent attempts and do not account for repeated source tasks. For selected-proposal provenance comparisons, use the task-clustered intervals in Table 14 .

Continuation	Pairs	Orig. pass	Prop. pass	$\Delta$ [95% CI]	Disc. $P{:}O$	McNemar $p$
Substituted ReAct (single) continuation	$1846$	$19.2$	$62.7$	$+43.5$ [39.5, 48.2]	$859{:}56$	$10^{-185}$
Same harness, state restored	$871$	$6.1$	$53.2$	$+47.1$ [42.3, 51.7]	$419{:}9$	$10^{-111}$
Multi-agent state partly restored	$345$	$4.1$	$54.8$	$+50.7$ [44.2, 57.8]	$176{:}1$	$10^{-51}$

Appendix

\aedoriginaltablelabel : Search-selected continuation success by continuation provenance, same columns as Table 13 ; the interval is the task-clustered percentile bootstrap.

\aedoriginalfigurelabel : Resource against resource under one base, recipe and scorer. Exact step on each resource’s test set for the untrained base and for students trained on AED or AgenTracer data, followed by AgenTracer’s test split under its own error-source labels.

Evaluation subset	Cases	Gain (pp)	$95\%$ interval
All cases	$184$	$+7.97$	$[1.63,14.49]$
No flagged task overlap	$149$	$+4.92$	$[-2.24,12.08]$
Flagged task overlap	$35$	$+20.95$	$[5.71,36.19]$

Appendix

\aedoriginaltablelabel : Post-hoc Who&When task-overlap sensitivity. Mean exact-step change against base across the larger arm’s three seeds, under the unified prompt. Intervals resample shared evaluation items, conditional on these seeds; they do not measure uncertainty over new training seeds.

Interface	Cohort	Seed 17	Seed 202	Seed 828	Mean $\Delta$ [95% interval]
Shared	HC	$5.17$	$8.62$	$3.45$	$-9.77$ [ $-17.24,-2.87$ ]
	HC + gold	$10.34$	$6.90$	$6.90$	$-12.64$ [ $-24.14,-1.72$ ]
	AG	$9.52$	$11.90$	$3.17$	$-31.48$ [ $-40.21,-22.75$ ]
	AG + gold	$6.35$	$8.73$	$5.56$	$-32.80$ [ $-41.53,-24.34$ ]
	AEB	$15.33$	$22.00$	$15.00$	$+1.78$ [ $-3.67,7.33$ ]
	TEB	$4.12$	$11.73$	$6.58$	$-20.51$ [ $-24.55,-16.39$ ]

Appendix

\aedoriginaltablelabel : All fixed public conditions after replacement-data retraining. Exact-step percentages for each seed and their mean change against the retained raw-verified base. AEB uses an equal-weight mean over its three environments; other rows use case accuracy. Intervals resample source-task groups, conditional on these seeds and fixed environments. They are sensitivity intervals, not multiplicity-adjusted tests or replacements for the pre-specified item-bootstrap result above. Interfaces can differ in inputs, answer contracts and step coordinates; rows do not isolate a prompt-wording effect. HC/AG: hand-crafted/algorithm-generated; gold: supplied task reference answer.

Model	Clipped policy	Full policy	First call	Continued
Base	$87$	$78$	$59$	$65$
SFT, seed 17	$63$	$71$	$71$	$0$
SFT, seed 202	$64$	$62$	$62$	$0$
SFT, seed 828	$92$	$83$	$83$	$0$

Appendix

\aedoriginaltablelabel : Exact-step hits on all

400

\tau^{2}

cases. Both policy conditions use the recorded budget-forcing protocol. First-call hits and continuation counts describe the full-policy run; invalid answers remain misses.

Diagnosis: seeds 17, 202, 828; sequence limit 24,576
Recipe	Tasks	Rows	Examples	Batch	Epochs	Updates
Full diagnosis	$948$	$948$	$948$	$12$	$2$	$158$
Full diagnosis	$1{,}656$	$1{,}656$	$1{,}656$	$12$	$2$	$276$
Actor: seed 17; sequence limit 12,288
Success-only	$1{,}093$	$1{,}093$	$10{,}595$	$16$	$2$	$1{,}326$
Preventive	$1{,}183$	$1{,}790$	$11{,}912$	$16$	$2$	$1{,}490$

Appendix

\aedoriginaltablelabel : Recorded training sizes and optimization budgets. Examples are per epoch; updates cover the full run. Batch denotes examples per optimizer update.

Diagnosis configuration	Pairs	Yield	USD / input
AgentDebugX: all-at-once	$735$	$49.0\%$	$0.0052$
AgentDebugX: binary search	$3$	$0.2\%$	$0.0100$
AgentDebugX: counterfactual	$0$	$0.0\%$	$0.0136$
AgentDebugX: heuristic	$802$	$53.5\%$	$0.0000$
AgentDebugX: ensemble	$437$	$29.1\%$	$0.0152$
AgentDebugX: success reference	$756$	$50.4\%$	$0.0054$

Appendix

\aedoriginaltablelabel : Historical production configurations on the common failure pool. Pairs satisfy the recorded structured-diagnosis and trace-citation rule. This is an output-compatibility measurement, not human-validated accuracy.

Meter scope	Cells	Logged USD
Completed cells: rollout	$1{,}198$	$415.53$
Completed cells: diagnosis	$1{,}198$	$343.44$
Completed cells: combined	$1{,}198$	$758.96$
Stopped cells (settled meters)	$10$	$0.12$
Unsettled meters (logged to date)	$7$	$8.84$

Appendix

\aedoriginaltablelabel : API cost by pipeline stage in two historical collection waves, re-read on September 24. A cell is one environment–harness–policy run configuration. Completed cells supply Table 21 ; stopped and unsettled cells remain in this accounting ledger. Only rollout and diagnosis appear in these meters.

Environment	Runs	Diag.	USD	USD/D
AgentBench DB	$2{,}140$	$418$	$20.62$	$0.049$
AgentBench OS	$2{,}140$	$413$	$18.18$	$0.044$
ALFWorld	$1{,}150$	$754$	$100.43$	$0.133$
BabyAI	$630$	$201$	$15.71$	$0.078$
BFCL	$1{,}600$	$666$	$51.63$	$0.078$
GridWorld	$1{,}000$	$115$	$10.51$	$0.091$

Appendix

\aedoriginaltablelabel : Construction API cost by environment and harness. Both panels partition the same completed cells; they are not additive. Runs includes successful and failed task attempts. Diag. (D) counts runs with a recorded valid-step diagnosis, before independent semantic acceptance. USD includes rollout and diagnosis; USD/D is their combined cost per diagnosis. Model and task mixtures differ across rows, so this is descriptive accounting, not a controlled harness ranking.

Configuration	Calls/input	USD/input	USD/pair	USD/50k pairs
AgentDebugX: all-at-once	$1.01$	$0.0052$	$0.0106$	$528$
AgentDebugX: deep analysis	$4.79$	$0.0143$	$0.0218$	$1{,}089$
Multi-model consensus	$3.18$	$0.0247$	$0.0510$	$2{,}549$
AED citation-first judge	$1.00$	$0.0067$	$0.0072$	$359$

Appendix

\aedoriginaltablelabel : Diagnosis-only production cost on the common failure pool. The same representative configurations as Table 47 ; all configurations and yields appear in Table 19 . Pair means a structured location with a matching trace citation under the historical export rule. The last column is a linear budget scenario for 50,000 such outputs, not measured spending or a quality-certified release estimate.

Audit	Recorded observation	Consequence
Corpus terminal shape	An audit of $124{,}796$ unique traces selects $0$ of $3{,}575$ ALFWorld failures and $79$ of $1{,}486$ ScienceWorld failures. The separate ALFWorld-lite adapter has $374$ of $2{,}038$ .	Unsuccessful runs end with finish , without recorded parser/tool errors or exhaustion. This proxy does not enumerate self-repair opportunities.
Held-out failure events	Text-to-SQL: $317/319$ failures; WebShop-lite: $46/82$ .	No recognized non-infrastructure error event or step-limit termination. Missing event coverage can affect this classification.
Lexical subset	Of those failures, $60$ text-to-SQL and $46$ WebShop-lite tasks contain an earlier observation with a verifier-signal token absent from the task, after stopword filtering.	Fixes task membership for initial-state reevaluation; does not show that the policy saw or recognized a complaint.
Training contamination	The strict parser refused list_tables() on $197$ of $1{,}014$ episodes; none of the $1{,}246$ repair rows was built from that refusal.	Use the corrected parser for all evaluated arms.

Appendix

\aedoriginaltablelabel : Actor-evaluation qualifications. Counts refer to different audits and must not be combined. The fixed subset uses a lexical signal, not a recognition label.

Environment	Strict (%)	Corrected (%)	Outcome changes
Text-to-SQL	$65.2$	$68.5$	$+3.4$ pp; $69$ gained, $35$ lost / $1{,}014$
WebShop-lite	$75.3$	$72.7$	$-2.7$ pp; $44/300$ flipped
ALFWorld-lite	—	$88.0$	$+0.7$ pp; $24/300$ flipped

Appendix

\aedoriginaltablelabel : Sensitivity to the parser condition in untrained-policy runs. Both runs used one server at temperature

0

. Other environments also changed, so this does not isolate the parser’s causal effect. Same-parser replications remain necessary.

Supervision	Train tasks	Macro	Micro	Unit $+$ step
Qwen3-8B base	0	$64.05$	$52.66$	$44.97$
Execution-assisted diagnosis	$168$	$59.80\pm 3.78$	$52.47\pm 0.68$	$44.58\pm 0.90$
Teacher-only diagnosis	$168$	$62.44\pm 1.13$	$51.68\pm 2.46$	$41.22\pm 1.81$

Appendix

\aedoriginaltablelabel : Completed diagnosis-SFT comparisons (frozen protocol, mean

\pm

sd over seeds

17/202/828

). Upper block: the contract-faithful rerun of the pilot on its

169

held-out cases (Appendix S ). Lower block: the final cohort, without unit

+

step because the gold agent is one constant label there;

\Delta

-

J) is the seed mean of the paired family-macro difference.

Benchmark	Cases	Base	Execution-assisted	Teacher-only
Who&When HC	$58$	$13.79$	$14.94$	$14.94$
Who&When AG	$126$	$33.33$	$32.54$	$31.48$
AgentErrorBench	$200$	$10.50$	$7.50$	$7.00$
TrajErrBench	$486$	$23.25$	$24.83$	$24.48$

Appendix

\aedoriginaltablelabel : External attribution after paired diagnosis SFT. Exact-step accuracy (%) over every planned case, including missing or invalid predictions. Trained entries are means over three seeds from the superseded pilot run (target-token-weighted loss), whose rerun found both arms at or below the base on all four splits and shipped no per-split values; the base row is untrained. Descriptive, no significance claimed.

Continuation at the supplied location	Attempts	Seeds	Success	$\Delta$
Original action re-applied	$149$	—	$18.7\%$	ref.
Generic reconsideration	$144$	—	$27.1\%$	$+8.4$ [ $2.7$ , $14.8$ ]
Diagnosis and proposed replacement	$146$	—	$32.2\%$	$+13.5$ [ $7.1$ , $20.1$ ]
Deployed coach text	$147$	—	$36.1\%$	$+17.1$ [ $10.8$ , $23.8$ ]

Appendix

\aedoriginaltablelabel : Located-failure cohort:

150

failures in

11

environments resumed at the pre-action state of the attributed step, varying only what the actor is told; Attempts is the number of planned cases on which the arm reached a verdict. The content-free control carries a nudge because a retry with no text changes the execution path.

\Delta

against ref., task-clustered

95\%

intervals.

\aedoriginalfigurelabel : Actor recovery at a supplied error location. Recovery among each arm’s completed cases with task-clustered

95\%

intervals; the arrow marks the paired diagnosis-minus-reconsideration contrast on the cases both arms completed.

Training tasks	Three seed scores (%)	Mean $\pm$ SD	Source
$240$	$51.54/53.55/54.29$	$53.13\pm 1.43$	Run reports
$480$	$55.78/57.05/58.96$	$57.26\pm 1.60$	Run reports
$948$	$59.81/60.34/61.61$	$60.59\pm 0.93$	Run reports
$1,656$	$62.88/63.31/64.48$	$63.56\pm 0.82$	Run reports

Appendix

\aedoriginaltablelabel : Three-seed diagnosis scaling. Micro exact-step agreement on the same

943

expected cases. Scores are sorted within each row, not ordered by seed. Means and SDs use exact counts from the original per-case reports.

Target	Tasks	Seed	Epoch	Seven-field	Compact
Base	$0$	—	—	$46.66$	$47.19$
Full diagnosis	$240$	$17$	$1$	$47.83$	$51.54$
Full diagnosis	$480$	$17$	$2$	$56.52$	$58.96$
Full diagnosis	$948$	$17$	$2$	$58.64$	$60.34$
Full diagnosis	$1656$	$17$	$2$	$62.14$	$63.31$
Full diagnosis	$948$	$202$	$2$	$60.98$	$61.61$

Appendix

\aedoriginaltablelabel : Full-diagnosis nested ladder and seed replicates. Micro exact step on the same held-out cases under two prompt schemas. The summary is mean

\pm

sample standard deviation across training seeds for both replicated cohorts, not a confidence interval.

Protocol and split	$N$	Base	Full FT	Multiformat	$\Delta$	$p$
Training format: AgentErrorBench	$200$	$15.50$	$20.00$	$21.00$	$+4.50$	$0.151$
Training format: Who&When AG	$126$	$37.30$	$15.87$	$34.92$	$-21.43$	$<0.001$
Training format: Who&When AG, WG	$126$	$38.89$	$9.52$	$34.13$	$-29.37$	$<0.001$
Training format: Who&When HC	$58$	$20.69$	$15.52$	$24.14$	$-5.17$	$0.505$
Training format: Who&When HC, WG	$58$	$18.97$	$13.79$	$22.41$	$-5.17$	$0.546$
Training format: TrajErrBench	$486$	$27.57$	$15.43$	$16.87$	$-12.14$	$<0.001$

Appendix

\aedoriginaltablelabel : Public transfer with a target-format control. Micro exact step (percent);

\Delta

and nominal paired McNemar

p

compare full FT with base. HC/AG: hand-crafted/algorithm-generated; WG: with ground truth; —: arm unavailable.

		Committed (arm on W1)		Re-decoded, same box as base
Protocol and split	$N$	$\Delta$	Holm	$\Delta$	Holm	Contract $\Delta$
Training format: AgentErrorBench	$200$	$+4.50$	$1.000$	$+5.00$	$0.712$	$-3.50$
Training format: Who&When AG	$126$	$-21.43$	$0.002$	$-23.02$	$<0.001$	$-54.76$
Training format: Who&When AG, WG	$126$	$-29.37$	$<0.001$	$-30.16$	$<0.001$	$-71.43$
Training format: Who&When HC	$58$	$-5.17$	$1.000$	$+0.00$	$1.000$	$-15.52$
Training format: Who&When HC, WG	$58$	$-5.17$	$1.000$	$-6.90$	$1.000$	$-17.24$

Appendix

\aedoriginaltablelabel : The thirteen transfer tests before and after same-machine repeat evaluation. Holm is over the declared family of

13

; Contract

\Delta

is the repeat-evaluation change in answer-contract compliance. Both columns select the same three cells, all losses.

Training rows	Seed	Correct / cohort	Exact (%)	Format fails	$p$ vs. base
TrajErrBench
0	—	91 / 486	18.72	0	—
948	17	86 / 486	17.70	0	0.6570
1,656	17	75 / 486	15.43	0	0.0888
1,656	202	75 / 486	15.43	1	0.0805
1,656	828	102 / 486	20.99	0	0.2664

Appendix

\aedoriginaltablelabel : Public localization across training seeds. Two benchmarks under the unified attribution prompt. We divide correct predictions by the full frozen cohort, counting a format-contract failure as a miss; Format fails records those failures. The

p

values are nominal, exact paired McNemar tests against base. The producer records the same training-data hash for the larger-rung checkpoints. We use the producer’s checkpoint mapping, since the reports’ model alias is identical for trained and untrained arms.

Panel of Figure 46 — \aedoriginalfigurelabel : Held-out exact step against training-set size. Micro exact step on the $943$ held-out cases by source tasks (log axis; one seed per nested subset, three at the full arm, base dashed, no fit). The upper table reports internal results at every measured size; the lower table reports available public evaluations. Intermediate sizes have no public evaluation. Replicated cells show mean $\pm$ sample sd, with replicate counts in the table. Missing or invalid predictions count as misses on the full evaluation cohort.

Checkpoint	Pairs	Prefers chosen (%)	Implicit reward $>0$ (%)	Mean margin (nats)	Dev prefers chosen (%)
Qwen3-8B base	$94$	$54.3$	—	$10.21$	—
DPO seed 17	$94$	$55.3$	$67.0$	$1.29$	$52.4$
DPO seed 202	$94$	$54.3$	$60.6$	$1.32$	$52.4$
DPO seed 828	$94$	$54.3$	$62.8$	$1.27$	$52.4$
Mean $\pm$ sd over seeds		$54.6\pm 0.6$	$63.5\pm 3.3$	$1.29\pm 0.02$	$52.4\pm 0.0$

Appendix

\aedoriginaltablelabel : Held-out preference agreement after DPO on measured-effect pairs. Rates over the

94

held-out pairs that compile under the

16384

-token limit (

5

skipped:

4

whose sides differ in format,

1

over length); the base row is the untrained student’s own preference. Dev is a

21

-pair sanity check, not used for selection. Mean margin is

\log p(\text{chosen})-\log p(\text{rejected})

under the scored model for the base row and, for the DPO rows, the implicit reward margin

[\log\pi_{\theta}-\log\pi_{\mathrm{ref}}](\text{chosen})-[\log\pi_{\theta}-\log\pi_{\mathrm{ref}}](\text{rejected})

without the

\beta

factor; the base figure is a raw log-likelihood margin and is not comparable with the reference-relative rows. Offline agreement with a replay-measured preference, not task success.

\aedoriginalfigurelabel : Where recorded corrections help. Paired difference

\Delta

(replacement minus original action) with task-clustered intervals, by replacement kind, attributed-step position and environment (cells with at least

30

pairs).

Comparison	Observed result	Scope
Paired replay	Discordant pairs: $1454$ replacement wins, $66$ losses. Source-harness stratum: $871$ pairs, $+47.1$ points $[42.3,51.7]$ .	Selected replayable attempts; the control reruns the original action. No diagnosis-release row joins this replay study.
Located recovery	Deployed coach versus clean diagnosis: $+4.9$ points $[-0.7,10.4]$ .	Supplied error location, untrained debugger; the interval does not establish a framing benefit.
Compact-target pilot	Three-seed micro: $52.47\%$ execution-assisted, $51.68\%$ teacher-only, $52.66\%$ base.	Construction arms differ in teacher access and engine. All six paired X–J intervals include zero.
Reasoning control	Base thinking-off: $47.51$ exact step, $34.89$ unit+step on $943$ cases.	One run. A single gold unit makes unit+step sensitive to output naming.
Public positional control	Test-selected constant steps score $17.2$ , $27.0$ , $16.0$ , $10.3\%$ .	Diagnostic upper envelope for constant-step guesses; not a deployable baseline.

Appendix

\aedoriginaltablelabel : Interpretation of the supplementary results. Each row retains its own cohort and denominator.

\aedoriginalfigurelabel : Two development analyses with separate populations. (a) Changes in attributed step across

1{,}744

successive diagnosis rounds. (b) Error reporting on the same

50

successful trajectories per model under failure-presupposing and neutral prompts. Successful completion does not imply an error-free trace. Neither panel is a trained-model result.

	Wave 1	Wave 2	Wave 3	Pooled
Diagnosed attempts	109	170	228	507
structurally ineligible	0	0	130	130
certificate-eligible	109	170	98	377
Certificates	60	74	15	149
first earned in round 1	45	40	9	94
first earned in rounds 2–7	15	34	6	55

Appendix

\aedoriginaltablelabel : Iterative construction across three campaigns: 507 diagnosed attempts, of which 377 are eligible for paired replay. Later rounds add observed successful contrasts; no randomized one-round control was run. Attempts stopped while still failing are right-censored.

Environment	Type	Source tasks	Pairs	Frozen release
ALFWorld	Public benchmark	$1{,}431$	$4{,}838$	yes
AgentBench-DB	Public benchmark	$1{,}120$	$4{,}407$	yes
AgentBench-OS	Public benchmark	$485$	$3{,}309$	yes
BFCL	Public benchmark	$470$	$5{,}521$	yes
ScienceWorld	Public benchmark	$334$	$1{,}655$	yes
$\tau$ -bench retail	Public benchmark	$324$	$3{,}134$	yes

Appendix

\aedoriginaltablelabel : Every environment counted in the collection snapshot, by type. One row per environment with at least ten source tasks: its distinct source tasks and error–diagnosis pairs in the snapshot, and whether the frozen diagnosis release contains it.

Witness class	Environments	Basis
Observed agreement	BFCL, ScienceWorld, SWE-bench Verified, $\tau$ -bench airline, $\tau$ -bench retail, Terminal-Bench, AgentBench-DB, AgentBench-OS, SWE-smith, Mind2Web (static), ALFWorld, AppWorld	10 by oracle prefix, 2 (ALFWorld, AppWorld) by recorded trace
Measured divergent (excluded)	MCPMark	Two replays of the same recorded prefix disagreed on the verifier verdict
No deterministic replay	WideSearch, GAIA	Live web interactions do not support deterministic prefix replay

Appendix

\aedoriginaltablelabel : State-restoration checks across fifteen target environments. ALFWorld contributes a potential pool of

3{,}553

tasks and was checked using recorded actions because no reference prefix was available. MCPMark is excluded from verified-repair counts after observed divergence. WideSearch and GAIA retain diagnosis records without deterministic replay.

Audit	Measurement	Consequence
Policy request	Reconstructed manifests cover $67{,}308/67{,}377$ traces; $65$ lack a system turn and $4$ have an unmatched digest.	External-framework system turns were re-rendered; reconstruction is distinct from native request logging. New collection records the first request.
Rules invoked by diagnoses	On $550$ model-reviewed pairs, weighted shares are $39.2\%$ visible rules, $32.2\%$ $[27.3,37.0]$ system-only rules, $12.0\%$ unstated rules, and $5.5\%$ $[3.2,7.8]$ contradicted rules.	Model estimates, not human validation. Historical debugger inputs omitted policy system prompts and tool lists.
Field clipping	Cap: $600$ characters. $3367/4737$ release rows contain a clip; $29.5\%$ of text is removed.	The debugger saw clipped text while citation matching used the unclipped render. Seven of $25{,}674$ unresolved citations recover under a wider scorer render.
Rediagnosis	On $200$ clipped rows, clipped–clipped step agreement is $82.5\%$ ; clipped–unclipped is $80.0\%$ .	Aggregate change is comparable to rerunning. The deterministic voter agrees with its unclipped version on $79\%$ ; $19$ rows move only without clipping.

Appendix

\aedoriginaltablelabel : Audited visibility gaps. Historical diagnoses do not acquire information retroactively when an exporter restores it.

Recipe	Input $\to$ response	Evidence and experimental scope
Diagnosis SFT	Failed trace $\to$ attribution, optionally with rationale, evidence and correction	Main diagnosis experiments; the label need not identify a unique cause.
Preventive actor SFT	Pre-error history $\to$ actions from a passing continuation	Executed correction and branch lineage; mixed with successes in Section 5.3 .
Post-error actor SFT	History, erroneous action and rejection $\to$ correction, with or without reflection	Eligible state-preserving rejections; other repairs retain preventive targets. Section 5.3 .
Action DPO	Shared text prefix $\to$ chosen and rejected actions	Historical offline pilot only; it does not meet the current strict lineage criterion (Appendix I.15 ).

Appendix

\aedoriginaltablelabel : Training views and experimental scope. Context tokens carry no SFT loss. An implementation or exportable view does not establish a training benefit.

\aedoriginalfigurelabel : Learning from the same failed run. Diagnosis observes the completed failed trace; the illustrated preventive recovery view observes the history before an intervention. The post-error extension also retains the erroneous action and feedback as masked context (Appendix Q.3 ). Action comparison additionally requires executed alternatives from the same state. Each arrow denotes a supervision interface, not a measured training benefit.

Construction	Target tokens/epoch	Updates (2 epochs)	Updates (3 epochs)
Execution-assisted	$10{,}553$	$28$	$42$
Teacher-only	$9{,}375$	$28$	$42$

Appendix

\aedoriginaltablelabel : Frozen compact-supervision budgets. Counts are per training run, not experimental outcomes. Both arms use the same

168

failed trajectories; learning-rate candidates share these counts. Model and tokenizer revisions are fixed.

View	Input and target	Admission boundary	v5 development partition
Diagnosis SFT	Failed trace $\to$ attributed step, explanation, citations, proposal	Resolvable evidence; intentionally unattempted replay allowed; target semantics independent of replay	$1{,}154$ rows; reference assistance marked in metadata
Debugger SFT	Failed trace $\to$ recorded attribution or recovering intervention	Stricter replay-state gate; reference-assisted labels excluded by default	$994$ rows; recovery language only with a passing bundled branch
Repair SFT	Pre-action prefix $\to$ passing continuation’s assistant turns	Executed passing branch; erroneous action and failed suffix excluded	$265$ rows in the replay package’s training split; masked prefix and tool observations
Outcome	Prefix and candidate $\to$ replay-derived outcome	Typed execution outcome and direct branch lineage	$1{,}082$ rows; offline prediction, not a demonstrated RL result
Action DPO	Shared prefix with chosen/rejected actions	Valid executed lineage and positive observed preference under the stated protocol	$602$ rows; action-specific causal eligibility reported separately

Appendix

\aedoriginaltablelabel : Implemented training interfaces. The right-hand counts are the v5 development partition, not the frozen release: the release materializes the diagnosis view alone, for the reason stated below the table.

Dimension	ADP ( Song et al., 2026 )	AED
Unit of data	Successful action/observation trajectory.	Natural failure, diagnosis, proposed fix and available replay branches.
Action typing	API, code, message actions; text or web observations.	Same three exports; content map retains speaker, step index, event ID.
Provenance	Source dataset name.	Environment/version; harness/version; policy model; run ID; engine version.
Failure handling	Success-only corpora; no failure, reward, or outcome field.	Natural failures, never injected; outcome, reward, verifier signal stored.
Step-level labels	None.	Blamed step, blamed agent, explanation, alternative steps.
Evidence grounding	None.	Verbatim quotes typed by source region/match status; 53.1% of quotes ground the trajectory.

Appendix

\aedoriginaltablelabel : Stored contents of ADP trajectories and AED failure records; T/C denotes treatment/control, E0–E2 are evidence grades, and the AED quantities come from the fixed conversion census.

Context
Model / condition	W&W-HC	W&W-AG	AEB
	( $n{=}58$ )	( $n{=}126$ )	( $n{=}200$ )
Best constant step (oracle)	$17.2$	$27.0$	$16.0$
Prompted GPT-5 mini §	$22.4$	$43.7$	$19.5$
Frozen native-context protocol, no ground truth
Qwen3-8B base	$15.5$	$31.0$	$8.0$

Appendix

\aedoriginaltablelabel : Earlier public attribution references. Exact step uses the full planned denominator; MAST reports micro-F1 against released model-judge labels.

\ddagger

denotes an author-reported result under another protocol, not a run of ours.

\lx@sectionsign

denotes the historical long-context protocol. These references are separate from the completed paired training comparison in Table 26 .

\aedoriginalfigurelabel : Human error localization and diagnosis review. (a) All human verdicts, split by the historical sampling strata; two judgments per record. (b) Raw step agreement with Wilson 95% intervals: preferred steps where both reviewers select a location (

69

records), and the subset where both select a single step (

50

). Both panels concern AI-assisted review of the same

80

historical records.

Stratum	Records	Preferred step	Single step	Verdict agree.	Joint accept
Filter survivors	48	$39/44$ (88.6%)	$31/33$ (93.9%)	$32/48$ (66.7%)	$30/48$ (62.5%)
Judge rejected	16	$10/14$ (71.4%)	$7/9$ (77.8%)	$7/16$ (43.8%)	$6/16$ (37.5%)
Gate rejected	16	$10/11$ (90.9%)	$8/8$ (100.0%)	$11/16$ (68.8%)	$5/16$ (31.3%)
All records	80	$59/69$ (85.5%)	$46/50$ (92.0%)	$50/80$ (62.5%)	$41/80$ (51.3%)

Appendix

\aedoriginaltablelabel : AI-assisted human review by sampling stratum. Preferred step compares the two selected locations, including preferred choices from multiple candidates; single step restricts to two single-step judgments. Verdict agreement uses four categories; joint accept requires two accept verdicts. All agreement values are raw rates, with denominators shown. These are historical, coverage-selected records, not a random final-release sample.

Decision	Response options
Diagnosis content	Accept : key content is acceptable; revise : changes are needed; reject : key content is unsupported; insufficient information : the available record does not support a decision.
Error localization	Single step ; multiple defensible steps with one preferred step and at least one alternative; no single-step attribution ; no agent error ; or insufficient evidence .
Submission	Explicitly confirm the judgment for each record. Step selection is restricted to events in that record. Save the JSON backup and CSV export.

Appendix

\aedoriginaltablelabel : The two required human decisions. Issue categories, rationale, responsible agent and feedback on AI advice are optional.

Example	Instruction
A wrong query at step 0 is corrected at step 2, which returns $A=12$ ; step 3 submits $A=20$ .	Select step 3; exclude the fully recovered query.
A budget of 30 is reduced by an irreversible wrong purchase of 20 at step 0; the required item costs 20.	Select step 0; finding the right item later does not restore the budget.
A valid read-only query returns HTTP 503 and the harness stops, with no evidence of an alternative tool.	Do not force an agent-error attribution.
A required image is unavailable and the grader reports only failure.	Select insufficient information.
Step 0 computes $7+5=13$ and step 1 submits 13 without correction.	Prefer step 0; step 1 may be an alternative. Propagation is not recovery.
The agent guesses a PIN; the correct value appears only in later feedback.	Identify the unsupported guess. A correction cannot use the future value; search first only if a search tool was available.

Appendix

\aedoriginaltablelabel : English translations of the six fictional teaching cases. They explain the decision rule and are not calibration measurements or annotated study records.

\aedoriginalfigurelabel : AI-assisted review workbench, English paper view. The original v6 trace renderer and human-decision controls show a historical item. Model summaries are translated for display, complete original reports remain available, and panels are arranged in three columns for the figure. The models disagree on the record verdict while naming the same primary step. Human choices remain blank; this illustrative derivative cannot submit annotations and does not change the issued packets.

Continuation from the checkpoint	Pass (%) $\uparrow$	Gain over retry (pp)
Original-action retry	$18.4$	—
AET: first/only correction	$\mathbf{51.1}$	$\mathbf{+32.7}$ $[28.4,37.0]$

Appendix

\aedoriginaltablelabel : Correction utility and diagnosis production. Panel (a): pass rates over

3{,}062

paired debugging attempts; brackets give the task-clustered

95\%

interval for the paired gain. Panel (b): all

1{,}500

input failures remain in each method’s yield and cost denominators. The panels use different populations; neither measures independent label accuracy.

(a) Prompt-only baselines
Model	Macro $\uparrow$	Micro $\uparrow$
Qwen3-8B	$50.98$	$47.19$
GPT-6 Astra	$58.15$	$50.69$
Claude Sonnet 5	$56.59$	$51.64$
Gemini 3.8 Flash	$57.39$	$53.23$
Claude Opus 5	$57.80$	$54.72$

Appendix

\aedoriginaltablelabel : Diagnosis learning: complete internal and external scores. Panels (a,b): exact-step agreement on the same

943

cases. Panel (c): public benchmark labels under a unified-prompt protocol; independent annotation does not imply task-disjoint evaluation. Trained rows report mean

\pm

sample SD when multiple seeds were run; scores use recorded teacher labels; the separate historical human audit does not relabel this test set.

Supervision	Text-to- SQL	ALFWorld lite	TextQuest	WebShop lite	GridWorld	Warehouse
Qwen3-8B base	$68.54$	$88.00$	$67.67$	$72.67$	$5.00$	$2.00$
Success-only	$78.50$	$98.00$	$\mathbf{93.67}$	$90.33$	$88.00$	$\mathbf{17.00}$
+ preventive repair	$77.61$ [-1pt] $-0.89$	$\mathbf{98.33}$ [-1pt] $+0.33$	$84.67$ [-1pt] $-9.00^{*}$	$95.00$ [-1pt] $+4.67$	$94.00$ [-1pt] $+6.00$	$14.00$ [-1pt] $-3.00$
+ post-error actions	$\mathbf{78.99}$ [-1pt] $+0.49$	$97.00$ [-1pt] $-1.00$	$85.33$ [-1pt] $-8.33^{*}$	$\mathbf{97.00}$ [-1pt] $+6.67^{*}$	$\mathbf{96.00}$ [-1pt] $+8.00$	$13.00$ [-1pt] $-4.00$
+ actions and reflection	$77.12$ [-1pt] $-1.38$	$97.67$ [-1pt] $-0.33$	$84.00$ [-1pt] $-9.67^{*}$	$\mathbf{97.00}$ [-1pt] $+6.67^{*}$	$90.00$ [-1pt] $+2.00$	$\mathbf{17.00}$ [-1pt] $0.00$
Planned tasks	$1{,}014$	$300$	$300$	$300$	$100$	$100$

Appendix

\aedoriginaltablelabel : Repair training gains and losses depend on the environment. Initial-state success (%) on held-out tasks in six development environments.

*

: post-hoc Holm-adjusted paired McNemar

p<0.05

versus success-only across all

18

contrasts. Run errors remain in the planned denominator. Recipes differ in training-task coverage, exposure and update count.

(a) Admission ablation: one candidate pool
Admission rule	Retained yield	Tasks / envs	Cost / row
No quality checks	$19{,}904$ (ref.)	$4{,}976$ / $31$	$0.04$
Grounding checks only	$2{,}628$ ( $13.2\%$ )	$1{,}139$ / $26$	$0.31$
AET: grounding + semantic review	$767$ ( $3.9\%$ )	$420$ / $24$	$1.25$

Appendix

\aedoriginaltablelabel : Admission yield and correction effectiveness. The two panels use separate cohorts. Panel (a) holds candidate diagnoses fixed and prices each rule at the spend it obliges. Panel (b) compares verifier pass rates from a shared checkpoint; intervals are task-clustered

95\%

bootstrap intervals.

Reference	$\Delta$ [95% CI]
Qwen3-8B base	$+16.12$ [ $10.61$ , $20.15$ ]
GPT-6 Astra, prompted	$+12.62$ [ $5.29$ , $18.47$ ]
Claude Sonnet 5, prompted	$+11.66$ [ $7.28$ , $15.95$ ]
Gemini 3.8 Flash, prompted	$+10.07$ [ $6.30$ , $15.00$ ]
Claude Opus 5, prompted	$+8.59$ [ $3.81$ , $13.24$ ]
same target, nested subset	$+2.97$ [ $0.85$ , $4.75$ ]

Appendix

\aedoriginaltablelabel : Internal paired contrasts: largest arm’s seed-

17

checkpoint minus each reference in micro exact-step agreement, with nominal task-family-bootstrap 95% intervals.

(a) Who&When source prompt: agent / exact step (%)
Model or training resource	HC	HC + gold	AG	AG + gold
Qwen3-8B base	$\mathbf{53.45}/\mathbf{6.90}$	$\mathbf{50.00}/\mathbf{5.17}$	$58.73/\mathbf{21.43}$	$60.32/\mathbf{16.67}$
AED diagnosis SFT	$51.72/1.72$	$\mathbf{50.00}/3.45$	$\mathbf{63.49}/11.90$	$61.11/11.11$
$+$ answer-format diversity	$41.38/1.72$	$48.28/3.45$	$59.52/7.94$	$\mathbf{61.90}/8.73$
(b) AgentErrorBench unified prompt: exact step (%)
Model or training resource	ALFWorld	WebShop	GAIA	Env. macro

Appendix

\aedoriginaltablelabel : Public attribution under alternative prompts. These scores use the same evaluated checkpoints as Table 2 . The unified AgentErrorBench prompt requests only a step, so no module score is defined.

Descriptive episode statistics; not a causal mechanism
Environment	Error-bearing steps, un-repaired policy	Step limit	$\Delta$ mean steps	$\Delta$ success (pp)
WebShop-lite	$33.4\%$	$20$	$-1.31$	$+6.00$
GridWorld	$18.3\%$	$12$	$-1.09$	$+5.33$
TextQuest	$12.4\%$	$42$	$+1.73$	$-9.00$
Text-to-SQL	$6.1\%$	$24$	$+0.03$	$-0.59$
ALFWorld-lite	$3.2\%$	$36$	$+0.35$	$-0.33$

Appendix

\aedoriginaltablelabel : Reference-policy error density and observed changes after repair training. Changes average the three repair arms. Association does not establish a budget mechanism.

Environment	Preventive	Action-only	Reflective	R/A	Session gap
WebShop-lite	0.0436	0.000821	0.000535	1	2.67
GridWorld	0.21	0.0574	0.804	0.146	1.00
TextQuest	7.43e-06	7.03e-05	2.43e-06	0.627	1.67
Text-to-SQL	0.531	0.735	0.295	0.09	3.35
ALFWorld-lite	1	0.607	1	0.727	0.67
Warehouse	0.607	0.424	1	0.454	0.00

Appendix

\aedoriginaltablelabel : Nominal exact McNemar tests. First three columns compare each repair arm with success-only; R/A compares reflective with action-only. Session gap is the range of two untrained serving evaluations, in points, not a significance threshold.

Explore similar work

Jul 21, 2026cs.AI

AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents

LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it. Existing observability tools replay execution traces but provide little support for identifying the root cause or translating diagnosis into recovery. We present AgentDebugX, an open-source debugging framework that organizes debugging as a closed loop of Detect, Attribute, Recover, and Rerun. At its core, DeepDebug performs multi-turn root-cause diagnosis through global trajectory understanding, structure-guided investigation, and cross-examination. On the Who and When benchmark, DeepDebug achieves the best strict attribution accuracy among the evaluated methods on both tested open-weight backbones, reaching 28.8 percent exact agent-and-step accuracy on qwen3.5-9b versus 21.7 percent for the strongest single-pass baseline. On GAIA, DeepDebug repairs 13 of 73 failed tasks in a single rerun, compared with 4 to 6 for three decoupled self-correction baselines, improving overall accuracy from 55.8 percent to 63.6 percent. AgentDebugX exposes this workflow through a Python library, CLI, web console, and installable agentic skill, and provides an opt-in Error Hub for sharing scrubbed failure-diagnosis-repair bundles and reusing them as debugging memory.

Kunlun Zhu, Xuyan Ye, Zhiguang Han +9

University of Illinois Urbana-Champaign · University of Toronto · Google +1

Model Debugging Large Language Model Agents Partial Observability Open-Source

Ask PDF

Sep 1, 2026cs.AI

EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems

Large language model (LLM) agent failures often contain multiple related errors rather than a single mistake. Existing attribution methods usually identify a responsible agent, step, or root cause, but do not explicitly model dependency between errors. We introduce EDGE, an Error Dependency Graph-guided multi-Error attribution framework. EDGE constructs an error dependency graph from observed error events and validates a reliable causal subset through counterfactual rollout. The inference graph guides a two-stage LLM-as-judge detector for error attribution, and the intervention-validated subgraph provides a more reliable basis for explanation and repair analysis. Experiments on TRAIL and MAST show that EDGE improves category-level multi-error attribution across most evaluated models and settings. Experiments with adapted Who&When-style prompts show that the graph helps across prompting strategies. These results suggest that dependency structure is a useful diagnostic prior for agent failures beyond isolated root-cause prediction.

Jun Hou, Priya Pitre, Yi Fang +1

Virginia Tech

Multi-Agent Large Language Model Systems Large Language Models Fail Dependency Graph Edge

Ask PDF

Jun 8, 2026cs.AI

REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces

Large language model (LLM) agents now solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind, especially in the \emph{silent failure} regime. Existing approaches predict suspect steps via classifiers or LLM judges, or recover correct answers via retry, but none feed the intervention outcome back to \emph{refine the attribution itself}. We propose \methodname, a method that closes this gap by diagnosing a candidate error step, testing it through controlled replay with a diagnosis-specific patch, and using the verified outcome flip as contrastive evidence to refine the final attribution. Across four localization benchmarks spanning multi-hop reasoning across domains, \methodname achieves the highest localization accuracy among same-auditor methods across all four benchmarks, with the largest gains on structured tool-use traces, while providing actionable localization even when ground-truth answers are unavailable.

Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok +4

University of California, Los Angeles, USA · Cisco, USA · University of Mississippi, USA

Large Language Models Fail Traces Reflections Localization

Ask PDF