Agentic Harness for Real-World Compilers

Yingwei Zheng; Cong Li; Shaohua Li; Yuqun Zhang; Zhendong Su

cs.SEMar 20, 2026

Agentic Harness for Real-World Compilers

Authors: Yingwei Zheng, Cong Li, Shaohua Li, Yuqun Zhang, Zhendong Su

Organizations: Southern University of Science and Technology · ETH Zurich · The Chinese University of Hong Kong

Abstract

Compilers are critical to modern computing, yet fixing compiler bugs is difficult. While recent large language model (LLM) advancements enable automated bug repair, compiler bugs pose unique challenges due to their complexity, deep cross-domain expertise requirements, and sparse, non-descriptive bug reports, necessitating compiler-specific harnesses. To bridge the gap, we introduce llvm-harness, the first harness designed to assist LLM agents in understanding and fixing compiler bugs. Our current focus is on the middle end of LLVM, one of the most widely used compiler infrastructures. Central to llvm-harness are agent-friendly LLVM tools, a benchmark llvm-bench of 334 reproducible LLVM middle-end bugs, and a tailored mini agent llvm-autofix-mini for fixing LLVM middle-end bugs automatically. We evaluate five frontier models and find that they exhibit a performance decline when tackling compiler bugs with the state-of-the-art agent. With llvm-harness' enhancement, their performance improves by 62%. Our specialized mini agent llvm-autofix-mini further outperforms the llvm-harness-enhanced state-of-the-art by 22%. This emphasizes the necessity for specialized harnesses like ours to assist LLMs in compiler engineering tasks. Despite promising results, our expert review also reveals several open challenges that remain when applying LLMs for compiler engineering tasks. GitHub: https://github.com/dtcxzyw/llvm-harness

Figures & tables

Figure 1: Compiler issues, lacking descriptive information, are challenging to diagnose and repair . This compares two LLVM issues (crash and miscompilation) and a common software issue from Django; all issues are simplified for brevity.

				Tests		Fixes
Name	Split	#Bugs (#C/#M)	#Comp	#Rep	#Reg	#Lines	#Funcs	#Files
full	full	334 (222/112)	64	1.4	722.2	17.1	1.6	1.2
	easy	255 (176/79)	48	1.4	734.1	9.1	1.0	1.0
	medium	44 (26/18)	18	1.4	626.5	38.8	2.6	1.0
	hard	35 (20/15)	25	1.7	756.6	47.6	4.9	2.7
live	full	229 (160/69)	43	1.5	743.7	18.5	1.7	1.2

Table 1: llvm-bench includes 334 reproducible issues . “#C/#M” is the number of crashes/miscompilations; “#Comp” the number of affected components. “#Rep” and “#Reg” are the average number of reproducers and component-specific regression tests per issue. “Fixes” report the average edited lines, functions, and files per golden patch.

Figure 2: Distribution of Affected Components in llvm-bench

	%Passed					$Cost
Model	SWEV	full	easy	medium	hard	SWEV	full
GPT 4o	21.6	8.3 (-61.6%)	9.9	5.9	0.0	1.53	1.59
GPT 5	65.0	21.0 (-67.8%)	23.3	17.6	8.7	0.28	0.74
Gemini 2.5 Pro	53.6	9.2 (-82.9%)	10.5	5.9	4.3	0.29	0.81
Qwen 3 Max	69.6	24.4 (-64.9%)	29.1	17.6	0.0	–	4.32
DeepSeek V3.2	60.0	38.9 (-35.2%)	43.0	32.4	17.4	0.03	0.08

Table 2: llvm-bench live is a challenging benchmark for current LLMs . Fixing LLVM issues is more difficult than fixing common software issues with mini-SWE-agent . “ SWEV ” denotes SWE-bench Verified; the data are copied verbatim from the SWE-bench (Bash Only) leaderboard or the respective paper. “ full ”, “ easy ”, “ medium ”, and “ hard ” are splits.

		Difficulty			Bug Type
Harness	full	easy	medium	hard	crash	miscomp
$\times$	24.0	26.2	17.6	17.4	28.1	14.5
$\checkmark$	38.9	43.0	32.4	17.4	41.9	31.9
Improv.	1.6 $\times$	1.6 $\times$	1.8 $\times$	1.0 $\times$	1.5 $\times$	2.2 $\times$

Table 3: llvm-harness improves mini-SWE-agent ’s performance significantly. With llvm-harness , mini-SWE-agent (with DeepSeek V3.2) achieves a 62% improvement in %Passed when addressing LLVM middle-end bugs.

	mini-SWE-agent		llvm-autofix-mini
Model	%Passed	$Cost	%Passed	$Cost
GPT 4o	8.3	1.59	12.2 (1.5 $\times$ )	2.18
GPT 5	21.0	0.74	51.5 (2.5 $\times$ )	0.59
Gemini 2.5 Pro	9.2	0.81	14.4 (1.6 $\times$ )	1.16
Qwen 3 Max	24.4	4.32	35.8 (1.5 $\times$ )	5.67
DeepSeek V3.2	38.9	0.08	10.5 (0.3 $\times$ )	0.15

Table 4: llvm-autofix-mini is a more promising baseline in repairing LLVM issues than mini-SWE-agent . Detailed data regarding the cost and overhead is presented in Table 5 .

Figure 3: llvm-autofix-mini outperforms mini-SWE-agent at each split, but as splits increase in difficulty, frontier models tend to struggle or fail . The Y-axis is “%Passed”. G4 , G5 , GM , QW , and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2.

Figure 4: llvm-autofix-mini outperforms mini-SWE-agent for each bug type, and miscompilations are more challenging than crashes . The Y-axis is “%Passed”. G4 , G5 , GM , QW , and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2.

Figure 5: llvm-autofix-mini is more robust as token limit increases than mini-SWE-agent . The Y-axis is “%Passed”. For mini-SWE-agent , nearly all models converge after reaching three million tokens. G4 , G5 , GM , QW , and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2.

				Token Cost (million)
M	T/m	#Tests	#Rounds	#Tot	#In	#Out	#Cache	%Cache	$Cost
mini-SWE-agent
G4	7.1	13.0	74.4	2.2	2.2	6.9K	2.2	96.4	1.6
G5	14.6	9.5	66.9	2.6	2.6	28.2K	2.5	94.4	0.7
GM	8.8	7.4	51.7	1.6	1.6	21.3K	1.4	85.8	0.8
QW	18.1	13.1	77.7	2.4	2.4	15.3K	1.1	43.9	4.3

Table 5: Cost and Overhead of llvm-autofix-mini and mini-SWE-agent . M: Model; T/m: Time/min; #Tot/#In/#Out/#Cache: The total/input/output/cached tokens consumed (measured in millions unless otherwise specified); G4 , G5 , GM , QW , and DS : GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2.

Figure 6: Failure distribution of unresolved issues. G4 , G5 , GM , QW , and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2, respectively.

	mini-SWE-agent		llvm-autofix-mini
Model	%Passed	%Valid	%Passed	%Valid
GPT 4o	8.3	1.7 (21.0%)	12.2	3.9 (32.1%)
GPT 5	21.0	6.6 (31.3%)	51.5	20.1 (39.0%)
Gemini 2.5 Pro	9.2	2.2 (23.8%)	14.4	5.2 (36.4%)
Qwen 3 Max	24.4	8.7 (35.7%)	35.8	13.1 (36.6%)
DeepSeek V3.2	38.9	14.4 (37.1%)	10.5	4.4 (41.7%)

Table 6: Current LLMs lack satisfactory, true capability in fixing LLVM middle-end issues : Despite passing all tests, fewer than 42% of passed patches are valid. “%Valid = #ValidPatches / #Issues” is the percentage of valid patches.

Figure 7: As splits increase in difficulty, frontier models’ true capability degrades fast : No models can handle the hard split except GPT 5 ( llvm-autofix-mini ). G4 , G5 , GM , QW , and DS are short for GPT 4o, GPT 5, Gemini 2.5 Pro, Qwen 3 Max, and DeepSeek V3.2, respectively. Blue bullets (

\cdot

) represent the rate of truly resolved issues, i.e., “%Valid” in Table 6 .

		mini-SWE-agent		llvm-autofix-mini
Type	Model	%Passed	%Valid	%Passed	%Valid
crash	GPT 4o	11.9	2.5 (21.1%)	17.5	5.6 (32.1%)
	GPT 5	24.4	6.9 (28.2%)	56.9	18.1 (31.9%)
	Gemini 2.5 Pro	11.2	2.5 (22.2%)	15.6	6.2 (40.0%)
	Qwen 3 Max	32.5	11.2 (34.6%)	43.1	15.0 (34.8%)
	DeepSeek V3.2	41.9	11.9 (28.4%)	12.5	4.4 (35.0%)

Table 7: Miscompilations are more challenging than crashes . Valid miscompilation patches are usually fewer than valid crash patches. “ miscomp ” stands for miscompilation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Signature	Description
Generic Tools
find ( pat , dir )	Find files matching the pattern pat in the directory dir recursively
grep ( args )	Find text matching patterns and files mentioned in arguments args
list ( dir )	List files and directories in the directory dir
read ( file , pos , k )	Read k lines of content starting from line pos of the file file
edit ( file , text , rep )	Replace the text text with the text rep in the file file

Appendix

Table 15

Signature	Description
Generic Tools
write ( file , txt )	Write txt into file
bash ( cmd , tmo )	Execute command cmd in a Bash environment with tmo as timeout
todo ( act , args )	Perform a TODO action act with args
LLVM-specific Tools
build ()	Build a debuggable LLVM

Appendix

Table 16

Model	#01	#10	#11	#00	$\chi^{2}$	$p$ -value	Significant?
GPT 5	77	7	41	104	56.68	0.0000	$\checkmark$
Gemini 2.5 Pro	23	11	10	185	3.56	0.0296	$\checkmark$
Qwen 3 Max	42	16	40	131	10.78	0.0005	$\checkmark$

Appendix

Table 8: With frontier models, llvm-autofix-mini significantly outperforms mini-SWE-agent . Matrix #ij: i and j denote whether mini-SWE-agent and llvm-autofix-mini pass or fail for a particular issue, respectively. Significance level:

\alpha=0.05

			mini-SWE-agent				llvm-autofix-mini
			%Passed		%Valid		%Passed		%Valid
Model	Release Date	#Post ( e / m / h )	Pre	Post	Pre	Post	Pre	Post	Pre	Post
GPT 4o	2024-07-18	229 (172/34/23)	–	8.3	–	1.7	–	12.2	–	3.9
GPT 5	2025-08-07	14 (9/3/2)	20.9	21.4	7.0	0.0	49.3	85.7	19.5	28.6
Gemini 2.5 Pro	2025-07-17	21 (14/4/3)	9.1	9.5	2.4	0.0	13.5	23.8	4.8	9.5
Qwen 3 Max	2025-09-05	0 (0/0/0)	24.5	–	8.7	–	35.8	–	13.1	–

Appendix

Table 10: The data leakage problem does not appear to have a substantial impact on GPT 5 and Gemini 2.5 Pro when fixing LLVM issues. “#Post ( e / m / h )” indicates the number of issues ( easy / medium / hard ) that were fixed post the release date of the respective model. “Pre” and “Post” denote the resolution rate on issues pre- and post-release, respectively.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 381 papers

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
Jul 1, 2026Zhao Tian, Yingquan Zhao, Chenyao Suo +2LLM Evaluation Automated Program Repair
School of Computer Software, Tianjin University, Tianjin, China · School of Cybersecurity, Tianjin University, Tianjin, China · School of Computer Science, University of Bristol, Bristol, UK
AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning
Jun 18, 2026Zepeng Li, Jie Ren, Zhanyong Tang +2Compiler Optimization Large Language Model-Guided Optimization
Shaanxi Normal University · China · Northwest University +2
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
Apr 16, 2026Fan Cui, Hongyuan Hou, Zizhang Luo +2Automated Program Repair Electronic Design Automation
Peking University · Beijing, China

Jul 1, 2026J/K move · Enter open · S save

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang+1

School of Computer Software, Tianjin University, Tianjin, China · School of Cybersecurity, Tianjin University, Tianjin, China · School of Computer Science, University of Bristol, Bristol, UK

LLM Evaluation Automated Program Repair Language Model Ensembles

Open paper AskPDF arXiv

LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.

Abstract

LLM Agent Harnesses

Explore similar work

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks