Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Myeongsoo Kim; Dingmin Wang; Siwei Cui; Farima Farmahinifarahani; Terry Yue Zhuo; Shweta Garg; Baishakhi Ray; Rajdeep Mukherjee

cs.SEMar 25, 2026

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Authors: Myeongsoo Kim, Dingmin Wang, Siwei Cui, Farima Farmahinifarahani, Terry Yue Zhuo, Shweta Garg, Baishakhi Ray, Rajdeep Mukherjee,

Organizations: AWS AI Labs · Monash University

Abstract

Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).

Figures & tables

Figure 1: Overview of TrajEval . Each agent execution trace is automatically decomposed into three stages— search (file-level), read (function-level), and edit (function-level)—and compared against the reference patch to yield stage-wise precision and recall.

SWE-bench Verified
Model	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	163	6.7	4.3	20.9	68.1
GPT-5-mini	162	8.6	8.0	14.2	69.1
Qwen3-Coder-480B	187	9.1	10.7	13.4	66.8
Qwen3-Coder-30B	231	11.7	10.8	10.8	66.7
Qwen3-235B	249	17.7	7.2	10.4	64.7

Table 1: Failure-mode distribution among failed trajectories (Pass@1 = 0), SWE-Agent. Buckets are mutually exclusive and exhaustive; each row sums to 100%.

n

counts failed trajectories for which all six stage features could be extracted; trajectories where the agent terminated before producing parseable observations or edits are excluded, most strongly affecting weaker models.

Theme	Sub-type	$n$	%
Coherence	Confused Thrashing	266	29.1
Collapse	Near-Correct Corrupted	97	10.6
Semantic	Wrong Branch Point	170	18.6
Error	Partial Understanding	67	7.3
	Wrong API Call	37	4.0
	Right Loc., Wrong Logic	28	3.1

Table 2: Taxonomy of Edit-Quality failures (n=914, SWE-Agent on SWE-bench Verified). Sub-types are mutually exclusive (cascading precedence; Appendix A.18 ).

Classification	Count	%
EXACT	5	5.2
NEAR	25	25.8
PARTIAL	66	68.0
DIFFERENT	1	1.0
Total	97	100.0

Table 3: Diff-hunk validation of the 97 Near-Correct Corrupted trajectories. The 5 EXACT cases ground the existence proof in Section 4.3 ; 30 strong-match (EXACT + NEAR) bound the recovery ceiling.

Benchmark	Overall	Py	Java	JS	TS
SWE-bench (n=914)	39.7	39.7	—	—	—
PolyBench (n=455)	32.3	26.4	36.3	42.1	32.1

Table 4: Cross-benchmark and per-language Coherence Collapse rate (% of Edit-Quality failures), SWE-Agent. JavaScript on PolyBench (42.1%) exceeds the SWE-bench Python baseline and Java (36.3%) is comparable, indicating the pattern is not Python- or benchmark-specific. Full taxonomy in Appendix A.4 .

Condition	Pass@1	$\Delta$
Baseline	66.0% (330)	—
Oracle	70.6% (353)	+4.6 ( $p<.001$ )
$\textsc{Cons}_{3}$	69.0% (345)	+3.0 ( $p=.08$ )

Table 5: Consensus measurement on GPT-5, SWE-Agent + SWE-bench (n=500). Oracle uses gold patch;

\textsc{Cons}_{3}

uses

\geq 3

-of-6 non-GPT-5 model agreement (no reference-patch access).

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Table 6: Behavioral patterns and efficiency metrics on SWE-bench Verified.

Model	Coh.	Sem.	Loc.	Scope	Gap
GPT-5	33.3	34.2	20.7	10.8	0.9
GPT-5-mini	31.2	32.1	22.3	13.4	0.9
Qwen3-Coder-480B	44.0	32.0	15.2	7.2	1.6
Qwen3-Coder-30B	38.3	36.4	17.5	6.5	1.3
Qwen3-235B	39.1	37.3	14.9	6.8	1.9
Qwen3-32B	48.5	27.2	14.7	5.1	4.4

Appendix

Table 7: Edit-Quality failure themes by model (% of each model’s Edit-Quality failures). Coh. = Coherence Collapse; Sem. = Semantic Error; Loc. = Intra-File Mislocalization; Scope = Scope/Completeness; Gap = Execution Gap.

Theme	SWE-bench	PolyBench	$\Delta$
	(n=914)	(n=455)	(pp)
Coherence Collapse	39.7%	32.3%	$-7.4$
Semantic Error	33.0%	33.9%	$+0.9$
Intra-File Mislocalization	17.6%	16.9%	$-0.7$
Scope/Completeness	7.9%	8.3%	$+0.4$
Execution Gap	1.8%	8.6%	$+6.8$

Appendix

Table 8: Edit-Quality theme distribution: SWE-bench Verified vs. PolyBench Verified (SWE-Agent scaffold). Coherence Collapse remains the largest single theme on both benchmarks; Semantic Error and Intra-File Mislocalization are within

\pm 1

pp across benchmarks. The cross-benchmark drop in Coherence Collapse is driven by PolyBench Python (Table 9 ) rather than by a multilingual effect.

Language	$n$	Coh.	Sem.	Loc.	Scope	Gap
Java	135	36.3	36.3	17.8	4.4	5.2
JavaScript	76	42.1	39.5	11.8	3.9	2.6
Python	216	26.4	31.0	18.1	12.5	12.0
TypeScript	28	32.1	28.6	17.9	7.1	14.3
All	455	32.3	33.9	16.9	8.3	8.6

Appendix

Table 9: PolyBench Edit-Quality taxonomy by language (SWE-Agent). JavaScript (42.1%) exceeds and Java (36.3%) is comparable to the SWE-bench Python baseline (39.7%), contradicting the “monolingual vs. multilingual” hypothesis. The aggregate cross-benchmark drop is concentrated in PolyBench Python (26.4%).

Agent	$\mu_{\text{tool}}$	$\mu_{\text{LLM}}$	$\mu_{\text{bash}}$	$P_{\text{pipe}}$	Loc.	Loc.%
GPT-5	30.71	30.71	15.36	0.22	2558	33.3
GPT-5-mini	32.16	32.16	16.84	0.24	3493	41.5
Qwen3-235B	42.53	42.53	15.25	0.20	1682	22.1
Qwen3-Coder-30B	50.45	50.45	27.44	0.42	2815	20.5
Qwen3-32B	48.59	48.59	13.62	0.23	2469	36.2
Qwen3-Coder-480B	50.30	50.30	24.80	0.36	2900	23.4

Appendix

Table 10: Complete Efficiency Metrics (SWE-bench Verified). Columns:

\mu_{\text{tool}}

mean tool calls per trajectory;

\mu_{\text{LLM}}

mean LLM invocations per trajectory;

\mu_{\text{bash}}

mean bash invocations per trajectory;

P_{\text{pipe}}

mean pipe ( | ) operators per bash command (a count, not a probability; values

>1

indicate multi-pipe chains); Loc. total localization-class commands across all trajectories; Loc.% fraction of bash commands classified as localization. Qwen3-8B was run on SWE-Agent + SWE-bench (n=500 trajectories, included in the failure-mode partition Table 1 ) but is excluded from per-tool aggregate tables due to its shorter average trajectories (7–10 tool calls vs. 20–40 for other models), which yield insufficient per-tool-type samples for reliable aggregation.

Tool		GPT-5	GPT-5-mini	Q-235B	Q-30B	Q-32B	Q-480B
bash	In	76.0	96.8	41.8	87.0	44.7	54.2
bash	Out	5210	1497	906	315	1301	593
view	In	25.7	27.1	27.4	26.6	29.0	27.2
view	Out	1682	1406	902	776	448	715
str_replace	In	322	327	560	616	190	555
str_replace	Out	316	379	418	457	132	432

Appendix

Table 11: Complete Token Consumption (SWE-bench Verified). Qwen3-8B excluded from per-tool aggregates; see Table 10 caption.

Tool	GPT-5	GPT-5-mini	Q-235B	Q-30B	Q-32B	Q-480B
bash	96.2%	96.4%	96.3%	97.3%	94.5%	97.7%
	(7213)	(7953)	(7149)	(13225)	(6417)	(11905)
view	99.7%	99.9%	99.9%	99.8%	99.2%	99.7%
	(4549)	(4200)	(4843)	(5970)	(7521)	(6165)
str_replace	98.8%	98.9%	97.2%	99.5%	92.6%	98.4%
	(1467)	(1656)	(4665)	(2086)	(8137)	(2748)

Appendix

Table 12: Complete Tool Success Rates (SWE-bench Verified). Qwen3-8B excluded from per-tool aggregates; see Table 10 caption.

Agent	$\mu_{\text{tool}}$	$\mu_{\text{LLM}}$	$\mu_{\text{bash}}$	$P_{\text{pipe}}$	Loc.	Loc.%
GPT-5	38.38	38.38	18.06	0.34	2204	35.6
GPT-5-mini	34.62	34.62	18.89	0.37	2447	44.4
Qwen3-235B	43.12	43.12	18.26	0.41	2081	32.3
Qwen3-32B	48.45	48.45	30.26	0.64	7644	71.6
Qwen3-8B	24.89	24.89	24.89	0.02	362	4.1
Qwen3-Coder-30B	46.05	46.05	23.66	0.57	2149	25.7

Appendix

Table 13: Complete Efficiency Metrics (PolyBench Verified)

Tool		GPT-5	GPT-5-mini	Q-235B	Q-32B	Q-8B	Q-30B	Q-480B
bash	In	88.7	102.0	45.0	64.0	51.7	53.5	49.2
bash	Out	12297	21460	3983	4344	968	2472	1539
view	In	26.8	30.1	32.1	30.8	—	28.3	29.0
view	Out	1800	1501	1012	711	—	1244	1068
str_replace	In	306	314	438	190	—	597	565
str_replace	Out	1396	1313	611	130	—	544	555

Appendix

Table 14: Complete Token Consumption (PolyBench Verified)

Tool	GPT-5	GPT-5-mini	Q-235B	Q-32B	Q-8B	Q-30B	Q-480B
bash	96.2%	95.9%	94.5%	92.9%	82.8%	95.9%	96.3%
	(5921)	(5266)	(6152)	(10514)	(8786)	(8043)	(8368)
view	100%	99.9%	99.8%	96.6%	—	99.9%	99.9%
	(4239)	(2846)	(4177)	(2993)		(4676)	(5391)
str_replace	99.4%	99.8%	98.2%	83.5%	—	99.9%	99.2%
	(1757)	(1177)	(2946)	(3032)		(1381)	(1957)

Appendix

Table 15: Complete Tool Success Rates (PolyBench Verified)

Agent	$\mu_{\text{tool}}$	$\mu_{\text{LLM}}$	$\mu_{\text{bash}}$	$P_{\text{pipe}}$	Loc.	Loc.%
GPT-5	24.18	24.18	18.84	0.56	7048	74.8
GPT-5-mini	20.06	20.06	15.15	0.49	5627	74.3
Qwen3-235B	26.26	26.26	9.65	1.10	3650	75.7
Qwen3-32B	13.98	13.98	4.49	0.42	2192	97.6
Qwen3-8B	26.55	26.55	8.97	0.45	4188	93.4
Qwen3-Coder-480B	45.71	45.71	20.61	0.74	6354	61.7

Appendix

Table 16: OpenHands Efficiency Metrics (SWE-bench Verified)

Tool		GPT-5	GPT-5-mini	Q-235B	Q-32B	Q-8B	Q-480B	Q-30B
executeBash	In	31.6	34.3	28.1	33.0	22.9	31.6	41.0
executeBash	Out	1046	1555	549	4503	1962	195	208
fsRead	In	18.7	17.4	18.4	18.7	18.7	18.2	18.5
fsRead	Out	1694	1610	1305	510	500	695	753
fsWrite	In	277	417	324	230	374	593	694
fsWrite	Out	332	250	267	141	131	206	244

Appendix

Table 17: OpenHands Token Consumption (SWE-bench Verified)

Tool	GPT-5	GPT-5-mini	Q-235B	Q-32B	Q-8B	Q-480B	Q-30B
executeBash	95.4%	96.3%	91.5%	98.0%	97.0%	94.9%	95.1%
	(9418)	(7575)	(4823)	(2246)	(4483)	(10304)	(9462)
fsRead	87.4%	63.2%	98.2%	59.8%	95.5%	97.8%	98.2%
	(952)	(960)	(2953)	(3260)	(4159)	(7930)	(5345)
fsWrite	89.0%	74.1%	76.1%	44.5%	29.0%	97.7%	95.6%
	(1168)	(953)	(927)	(483)	(1143)	(2803)	(1817)

Appendix

Table 18: OpenHands Tool Success Rates (SWE-bench Verified)

Agent	$\mu_{\text{tool}}$	$\mu_{\text{LLM}}$	$\mu_{\text{bash}}$	$P_{\text{pipe}}$	Loc.	Loc.%
GPT-5	29.03	29.03	20.18	0.56	5884	76.3
GPT-5-mini	25.45	25.45	19.59	0.51	5449	72.8
Qwen3-235B	30.40	30.40	14.68	0.84	4511	80.5
Qwen3-32B	18.03	18.03	8.83	0.93	2786	82.6
Qwen3-8B	25.43	25.43	10.98	0.30	3463	82.6
Qwen3-Coder-480B	47.45	47.45	19.58	0.66	4956	66.2

Appendix

Table 19: OpenHands Efficiency Metrics (PolyBench Verified)

Tool		GPT-5	GPT-5-mini	Q-235B	Q-32B	Q-8B	Q-480B	Q-30B
executeBash	In	40.3	38.6	30.8	41.5	26.5	26.0	35.3
executeBash	Out	2329	3086	1016	1290	6628	524	623
fsRead	In	23.5	24.4	24.2	23.9	25.0	23.0	23.1
fsRead	Out	1516	1122	1348	575	498	690	823
fsWrite	In	311	383	363	279	472	572	606
fsWrite	Out	308	253	240	154	124	270	285

Appendix

Table 20: OpenHands Token Consumption (PolyBench Verified)

Tool	GPT-5	GPT-5-mini	Q-235B	Q-32B	Q-8B	Q-480B	Q-30B
executeBash	98.2%	97.8%	96.2%	98.6%	98.1%	95.3%	96.0%
	(7710)	(7483)	(5607)	(3374)	(4195)	(7481)	(6641)
fsRead	89.2%	61.0%	97.3%	66.3%	90.0%	93.2%	97.7%
	(1217)	(677)	(2883)	(2401)	(2465)	(7016)	(4621)
fsWrite	66.7%	55.4%	54.5%	39.8%	20.6%	90.5%	77.8%
	(1722)	(1140)	(1264)	(530)	(880)	(2087)	(1540)

Appendix

Table 21: OpenHands Tool Success Rates (PolyBench Verified)

Agent	$\mu_{\text{tool}}$	$\mu_{\text{LLM}}$	$\mu_{\text{bash}}$	$P_{\text{pipe}}$	Loc.	Loc.%
GPT-5	13.45	13.45	13.45	0.75	4224	62.8
GPT-5-mini	14.67	14.67	14.67	0.74	4545	61.9
Qwen3-235B	43.76	43.76	43.76	0.26	14447	66.0
Qwen3-Coder-30B	42.38	42.38	42.38	0.44	10782	50.9
Qwen3-Coder-480B	50.56	50.56	50.56	0.21	16793	66.4

Appendix

Table 22: LiveSWEAgent Efficiency Metrics (SWE-bench Verified). Qwen3-8B and Qwen3-32B excluded due to OpenRouter API errors (Section 3 , coverage note).

Tool		GPT-5	GPT-5-mini	Q-235B	Q-30B	Q-480B
executeBash	In	158.8	96.7	122.3	79.8	97.5
executeBash	Out	528	430	14688	817	152

Appendix

Table 23: LiveSWEAgent Token Consumption (SWE-bench Verified). Qwen3-8B and Qwen3-32B excluded due to OpenRouter API errors.

Tool	GPT-5	GPT-5-mini	Q-235B	Q-30B	Q-480B
executeBash	94.3%	95.8%	97.8%	92.9%	98.2%
	(5434)	(6010)	(20880)	(17701)	(24474)

Appendix

Table 24: LiveSWEAgent Tool Success Rates (SWE-bench Verified). Qwen3-8B and Qwen3-32B excluded due to OpenRouter API errors.

Agent	$\mu_{\text{tool}}$	$\mu_{\text{LLM}}$	$\mu_{\text{bash}}$	$P_{\text{pipe}}$	Loc.	Loc.%
GPT-5	15.23	15.23	15.23	0.89	3906	67.1
GPT-5-mini	14.28	14.28	14.28	0.86	3674	67.3
Qwen3-235B	25.10	25.10	25.10	0.31	6299	65.7
Qwen3-Coder-30B	42.33	42.33	42.33	0.46	7703	47.6
Qwen3-Coder-480B	46.78	46.78	46.78	0.26	12311	68.9

Appendix

Table 25: LiveSWEAgent Efficiency Metrics (PolyBench Verified). Qwen3-8B and Qwen3-32B excluded due to OpenRouter API errors.

Tool		GPT-5	GPT-5-mini	Q-235B	Q-30B	Q-480B
executeBash	In	148.8	95.5	143.6	89.3	101.1
executeBash	Out	501	466	230	131	191

Appendix

Table 26: LiveSWEAgent Token Consumption (PolyBench Verified). Qwen3-8B and Qwen3-32B excluded due to OpenRouter API errors.

Tool	GPT-5	GPT-5-mini	Q-235B	Q-30B	Q-480B
executeBash	94.5%	93.3%	95.3%	97.9%	97.6%
	(4653)	(4235)	(8657)	(14358)	(17184)

Appendix

Table 27: LiveSWEAgent Tool Success Rates (PolyBench Verified). Qwen3-8B and Qwen3-32B excluded due to OpenRouter API errors.

Model	$\tau$	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	0.3	163	5.5	4.9	19.6	69.9
	0.5	163	6.7	4.3	20.9	68.1
	0.7	163	18.4	4.9	20.9	55.8
GPT-5-mini	0.3	162	6.8	7.4	14.2	71.6
	0.5	162	8.6	8.0	14.2	69.1
	0.7	162	16.7	8.6	16.0	58.6

Appendix

Table 28: Failure-mode distribution under varying recall thresholds

\tau

, SWE-Agent on SWE-bench Verified. Each row sums to 100%. Bold entries mark the modal bucket per row. Edit-Quality remains the modal bucket for all five capable models at every threshold (53–72%); Qwen3-32B and Qwen3-8B move between Search-modal and Edit-Quality-modal but stay in the upstream-failure regime (Search + Read

\geq

50% at

\tau=0.7

Model	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	158	11.4	3.2	17.7	67.7
GPT-5-mini	204	10.3	4.4	24.5	60.8
Qwen3-Coder-480B	196	9.7	10.2	15.3	64.8
Qwen3-Coder-30B	278	8.6	12.2	14.7	64.4
Qwen3-235B	276	14.9	9.8	10.9	64.5
Qwen3-32B	445	41.8	18.9	17.5	21.8

Appendix

Table 29: OpenHands on SWE-bench Verified.

Model	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	176	5.1	11.9	15.3	67.6
GPT-5-mini	214	8.4	13.1	15.4	63.1
Qwen3-Coder-480B	238	15.5	40.3	4.6	39.5
Qwen3-Coder-30B	386	33.4	30.3	3.1	33.2
Qwen3-235B	363	12.1	19.8	10.5	57.6
Weighted avg.	1377	17.2	24.3	8.8	49.7

Appendix

Table 30: LiveSWEAgent on SWE-bench Verified. Qwen3-8B and Qwen3-32B were excluded due to OpenRouter API errors with the bash-only LiveSWEAgent harness (Section 3 , coverage note).

Model	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	208	18.8	41.3	8.7	31.2
GPT-5-mini	216	27.8	30.1	8.8	33.3
Qwen3-Coder-480B	261	28.0	39.8	11.1	21.1
Qwen3-Coder-30B	287	26.8	36.6	11.5	25.1
Qwen3-235B	295	38.3	25.8	11.2	24.7
Qwen3-32B	339	61.4	15.3	10.9	12.4

Appendix

Table 31: SWE-Agent on PolyBench Verified (also reproduced inline as Table 1 , bottom).

Model	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	202	19.8	36.6	9.9	33.7
GPT-5-mini	233	27.5	33.5	10.7	28.3
Qwen3-Coder-480B	244	24.2	39.8	5.7	30.3
Qwen3-Coder-30B	293	25.6	38.9	6.1	29.4
Qwen3-235B	301	30.9	33.6	7.6	27.9
Qwen3-32B	348	52.9	25.9	2.9	18.4

Appendix

Table 32: OpenHands on PolyBench Verified.

Model	$n$	Search	Read	E.-Tgt.	E.-Qual.
GPT-5	239	16.7	54.8	4.6	23.8
GPT-5-mini	279	20.8	53.8	6.1	19.4
Qwen3-Coder-480B	297	31.0	59.9	1.3	7.7
Qwen3-Coder-30B	348	50.6	42.5	1.1	5.7
Qwen3-235B	329	24.0	55.0	3.0	17.9
Weighted avg.	1492	29.8	52.8	3.1	14.3

Appendix

Table 33: LiveSWEAgent on PolyBench Verified. Qwen3-8B and Qwen3-32B were excluded due to OpenRouter API errors with the bash-only LiveSWEAgent harness (Section 3 , coverage note).

Metric	SWE-bench	PolyBench
Total instances	500	382
Languages	1 (Python)	4 (Py/Java/TS/JS)
Unique repositories	12	20
Avg files per patch	1.4	1.6
Avg functions per patch	2.1	2.4
Avg lines changed	42	38

Appendix

Table 34: Complete dataset statistics.

$N$	Precision	Recall	Oracle recovered	Analytical ceiling
2	35.2%	95.5%	99.0% (485/490)	+4.6 pp
3	52.4%	94.2%	97.3% (477/490)	+4.5 pp
4	66.8%	91.5%	92.4% (453/490)	+4.3 pp
5	75.6%	86.2%	82.4% (398/483)	+3.8 pp

Appendix

Table 35: Consensus signal validation. For each threshold

N

, we compute the set of files viewed by

\geq N

of the 6 non-GPT-5 models on each instance. Precision/recall are against gold edit locations. “Oracle recovered” is the fraction of the 490 instances where the oracle would fire that consensus also covers at least one gold file. Analytical ceiling = recovery rate

\times

4.6 pp (the oracle gain), achieved only under perfect-precision triggering.

Figure 2: Relationship between trajectory length and Coherence Collapse. (a) Failure theme composition by trajectory-length quartile; Coherence Collapse (red+orange) grows from 22% to 64% (aggregate $\rho=0.32$ ). (b) Dissociation: Confused Thrashing (red) scales with length ( $\rho=0.34$ ) while Near-Correct Corrupted (orange) is flat ( $\rho=0.001$ ).

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 152 papers

Teaching Agents to Code Reliably
Oct 2, 2026Muhammad Ahmed Mohsin, Myeongsoo Kim, Kangrui Ruan +3Repository-Level Program Repair AI Coding Agents
Stanford University · AWS AI Labs
FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents
Aug 9, 2026Dongyi Lv, Fushun E, Aichen Cai +8Fine-Tuning Software Engineering Agents
Xi’an Jiaotong University · Joy Future Academy, JD
CodeTracer: Towards Traceable Agent States
Apr 13, 2026Han Li, Yifan Yao, Letian Zhu +13AI Coding Agents Agent Trajectory Analysis
Nanjing University · Alibaba Group · Renmin University of China +3

Oct 2, 2026J/K move · Enter open · S save

Teaching Agents to Code Reliably

Muhammad Ahmed Mohsin, Myeongsoo Kim, Kangrui Ruan, Shweta Garg+2

Stanford University · AWS AI Labs

Repository-Level Program Repair AI Coding Agents LLM Agent Reliability Automated Program Repair Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Autonomous coding agents solve repository issues by reading code, running commands, editing files, and submitting patches. Extra inference-time compute yields gains only when it produces a useful repair and supplies reliable evidence for choosing one. Three behaviors decide both, and we argue they are teachable rather than byproducts of scale, so a policy can carry them instead of a scaffold. Location diversity remains narrow, since attempts return to the same site and extra samples add no coverage. Edit diversity is left unexploited, since methodologies that differ resolve complementary issues no single run reaches. Verification misleads, since a test the agent writes for its own patch accepts many incorrect ones. Directing search by execution feedback and scoring each patch against its own reverted tree resolves 52.8% of SWE-bench Verified using 48.1% of the agent-steps an eight-sample baseline spends. Training moves these behaviors into the policy. On the 270 issues held out from SFT and RL training, weighted supervised fine-tuning raises pass@1 from 31.9% to 35.2% and pass@8 from 46.7% to 51.1%. A reinforcement objective then trains the verifier against gold-labeled repairs and incorrect variants, crediting the assertions that detect them. It raises pass@1 to 43.0% and pass@8 to 60.7%, lifts verifier precision from 26.8% to 41.7%, and more than halves false acceptance. Resolution improves on two of three out-of-distribution suites and verifier precision on all three, and the gains hold at 7B, 14B, and 30B against published coder baselines.

Abstract

Agent Failure Analysis

Explore similar work

Teaching Agents to Code Reliably

FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents

CodeTracer: Towards Traceable Agent States