ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair

Jia Li; Zeyang Zhuang; Zhuangbin Chen; Yuxin Su; Wei Meng; Michael R. Lyu

cs.SEMar 28, 2026

ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair

Authors: Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu

Organizations: The Chinese University of Hong Kong, Hong Kong, China · School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Sun Yat-sen University, Guangzhou, China

Abstract

Compilation errors pose pervasive and critical challenges in software development, significantly hindering productivity. Therefore, Automated Compilation Error Repair (ACER) techniques are proposed to mitigate these issues. Despite recent advancements in ACER, its real-world performance remains poorly evaluated. This can be largely attributed to the limitations of existing benchmarks, \ie decontextualized single-file data, lack of authentic source diversity, and biased local task modeling that ignores crucial repository-level complexities. To bridge this critical gap, we propose ComBench, the first repository-level, reproducible real-world benchmark for C/C++ compilation error repair. ComBench is constructed through a novel, automated framework that systematically mines real-world failures from the GitHub CI histories of large-scale open-source projects. Our framework contributes techniques for the high-precision identification of ground-truth repair patches from complex version histories and a high-fidelity mechanism for reproducing the original, ephemeral build environments. To ensure data quality, all samples in ComBench are execution-verified -- guaranteeing reproducible failures and build success with ground-truth patches. Using ComBench, we conduct a comprehensive evaluation of 12 modern LLMs under both direct and agent-based repair settings. Our experiments reveal a significant gap between a model's ability to achieve syntactic correctness (a 73% success rate for GPT-5) and its ability to ensure semantic correctness (only 41% of its patches are valid). We also find that different models exhibit distinct specializations for different error types. ComBench provides a robust and realistic platform to guide the future development of ACER techniques capable of addressing the complexities of modern software development.

Figures & tables

Figure 1. Overview of ComBench construction workflow: (1) Error Collection via dual-filtering and context tracing; (2) Patch Identification through multi-source search and dependency-based filtering; and (3) Environment Construction using Docker-based emulation and job configuration inference for automated reproduction with act .

Category		Description
Warning as Error		Warnings that are elevated to errors by compiler flags ( e.g. , -Werror ).
Declaration Error		Problems with undefined, redeclared, or conflicting identifiers ( e.g. , symbols, namespaces).
Type Error		Violations of the C++ type system, such as mismatches, invalid conversions, or incomplete types.
Syntax Error		Issues related to code grammar and structure, like unexpected tokens or missing punctuation.
Template Error		Errors specific to C++ templates, such as instantiation or argument deduction issues.
Member Error		Issues with accessing or defining members of classes, including access specifier violations.

Table 1. The Eight Major Error Categories in Our Taxonomy, Ordered by Matching Priority ( Warning as Error is excluded from the released benchmark).

Repository	Language	LoC	Files	Commits	Workflow Runs	Compilation Errors	Avg. Dependent Files per Error
Bitcoin ( Bitcoin Core developers, 2009 )	C++	$376,201$	$1452$	$46,307$	$11,388$	$12$	$47$
OpenSSL ( The OpenSSL Project, 1998 )	C	$217,931$	$3809$	$37,991$	$100,788$	$19$	$13$
RocksDB ( Facebook, 2012 )	C++	$724,296$	$1314$	$13,420$	$4706$	$64$	$24$
LLVM ( The LLVM Project, 2003 )	C++	$99,585$	$62,932$	$551,028$	$1,251,657$	$105$	$366$
Total		1,418,014	69,507	648,746	1,368,539	200	204

Table 2. Overview of Projects Analyzed in ComBench .

Panel of Figure 2 — Figure 2. Comparison of Error Distributions Across Projects and Error Types.

Project	Raw Log for Failed Job	Extracted Compilation Error	Repair Patch Identification	Deduplication	Success Reproduction
Bitcoin ( Bitcoin Core developers, 2009 )	585	121	76	42	12
OpenSSL ( The OpenSSL Project, 1998 )	3938	162	27	19	19
RocksDB ( Facebook, 2012 )	2240	265	146	86	64
LLVM ( The LLVM Project, 2003 )	5004	4412	168	115	105
Total	11,767	4,960	417	262	200

Table 3. Overview of Data Entry Collected at Each Phase for Each Project.

Open-source Models				Closed-source Models
Model	Size	Context	Output	Model	Size	Context	Output
DeepSeek-V3 ( AI, 2025 )	671B	128K	8,192	GPT-5 ( OpenAI, 2025b )	-	272K	128,000
Qwen3-235B-A22b ( Yang et al., 2025 )	235B (22B active)	128K	8,192	GPT-4.1 ( OpenAI, 2025a )	-	1M	32,768
Qwen3-30B-A3b ( Yang et al., 2025 )	30B (3B active)	128K	8,192	Claude 4 Sonnet ( Anthropic, 2025c )	-	200K	16,384
Qwen3 (4/8/14/32B) ( Yang et al., 2025 )	4/8/14/32B	128K	8,192	Claude 3.7 Sonnet ( Anthropic, 2025b )	-	128K	16,384
				Gemini 2.5 Pro ( Google, 2025a )	-	1M	64,000

Table 4. Baseline LLMs Used in Our Evaluation

Model	bitcoin (12)			openssl (19)			rocksdb (64)			llvm (105)			Total (200)
Model	CS	SC	EM	CS	SC	EM	CS	SC	EM	CS	SC	EM	CS	SC	EM
Proprietary Models
GPT-5	9	3	1	15	8	6	41	23	12	81	48	21	146	82	40
GPT-4.1	4	2	0	12	7	6	37	20	12	61	35	15	114	64	33
Claude-Sonnet-4	7	3	1	12	8	6	36	22	13	59	35	21	114	68	41
Claude 3.7 Sonnet	7	1	0	10	5	5	35	19	11	55	32	16	107	57	32

Table 5. Performance across different LLMs on ComBench . We report Compile Success (CS), Semantic Correctness (SC), and Exact Match (EM). Bold indicates the best performance in each category.

Figure 3. Performance of All LLMs Across Different Types The bar chart compares the number of compilation errors resolved by each LLM across various types, highlighting differences in performance.

Automated Program Repair

Automated program repair develops methods that locate software faults and produce validated fixes with limited human effort.

See all 158 papers

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Sep 22, 2026Om Nepal, Sushant Aryal, Oluseyi Olukola +1Automated Program Repair LLMs for Cybersecurity
School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA
BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
Sep 27, 2025Zehua Zhang, Ati Priya Bajaj, Divij Handa +14Software Engineering Agents Software Engineering Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimization
Apr 7, 2026Changxin Ke, Rui Zhang, Jiaming Guo +10Automated Program Repair LLM Code Generation
State Key Lab of Processors, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Institute of Microelectronics, CAS

Sep 22, 2026J/K move · Enter open · S save

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi

School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA

Automated Program Repair LLMs for Cybersecurity Code Generation Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.

Abstract

Automated Program Repair

Explore similar work

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software

QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimization