SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding

Songcheng Cai; Zhiheng Lyu; Yuansheng Ni; Xiangchao Chen; Baichuan Zhou; Shenzhe Zhu; Yi Lu; Haozhe Wang

cs.SEMar 17, 2026

SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding

Authors: Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, +8 more

Organizations: University of Waterloo · University of Toronto · The Hong Kong University of Science and Technology · McGill University & MILA · Verdent AI, Inc.

Abstract

Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often overlook the long tail topics and rely on popular repositories where Large Language Models (LLMs) can cheat via memorized knowledge. To address this, we introduce SWE-QA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments. We enforce topical balance via issue-driven clustering to cover under-represented task types and apply a rigorous difficulty calibration process: questions solvable by direct-answer baselines are filtered out. This results in a dataset where agentic workflows significantly outperform direct answering (e.g., a ~13-point gap for Claude Sonnet 4.5), confirming the necessity of agentic codebase exploration. Furthermore, to tackle the scarcity of training data for such complex behaviors, we propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF). This approach allows small open models to learn efficient tool usage and reasoning. Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art proprietary models, demonstrating both the validity of our evaluation and the effectiveness of our agentic training workflow.

Figures & tables

Panel of Figure 1 — Figure 1: SWE-QA-Pro Benchmark and Training Pipeline.

Benchmark	Repo-level	Repo Nav.	Multi-hop	Semantic Coverage	Diff. Calibration	Test Size
CodeQueries ( Sahu et al., 2024 )	✗	✗	✔	✗	✗	29033
InfiBench ( Li et al., 2024 )	✗	✗	✗	✗	✔	234
CodeReQA ( Hu et al., 2024 )	✔	✗	✗	✗	✗	1563
LongCodeQA ( Rando et al., 2025 )	✔	✗	✔	✗	✗	443
SWE-QA ( Peng et al., 2025 )	✔	✔	✔	✗	✗	576
SWE-QA-Pro	✔	✔	✔	✔	✔	260

Table 1: Comparison of representative code benchmarks.

Panel of Figure 2 — Figure 2: t-SNE visualization of semantic distributions. (a) Comparison of the original issue spaces, showing the broad coverage of SWE-Rebench compared to the manually curated SWE-Verified. (b) The distribution of our synthesized datasets (Training and Test) demonstrates high diversity and alignment with the semantic clusters of existing benchmarks.

Figure 3: Difficulty Comparison between SWE-QA and SWE-QA-Pro. Higher difficulty indicates harder questions.

Model	Evaluation Metrics					Overall
Model	Correctness	Completeness	Relevance	Clarity	Reasoning	Overall
Proprietary LLMs
Gemini 2.5 Pro	2.51	2.13	8.66	8.02	4.16	25.48
Gemini 2.5 Pro + Agent	7.12	6.25	8.91	9.34	7.84	39.46
GPT-4.1	3.42	2.38	9.02	9.23	4.68	28.74
GPT-4.1 + Agent	6.86	5.90	8.89	9.13	7.68	38.47

Table 2: SWE-QA-Pro Bench evaluation results. “+agent” denotes models using the SWE-QA-Pro agent framework. Best results per scale are shown in bold , with second-best underlined.

Figure 4: Tool usage behavior across models on SWE-QA-Pro.

Figure 5: Effect of Training Strategy. We compare SFT at different scales and a two-stage SFT→RLAIF setting.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Cluster ID	Cluster Name	Original Ratio (%)	Test Ratio (%)	Coverage ( $\times$ )	Origin Count	Test Count
0.0	Unicode / formatting / parsing / data validation	2.04	0.00	0.00	17,185	0
0.1	SQL / structured grammar / templating Engine	2.67	1.89	0.71	22,489	10
0.2	CLI args / syntax / regex / command completion	2.08	3.22	1.55	17,518	17
0.3	file import/export / metadata / sorting / recurrence	1.89	0.19	0.10	15,940	1
1.0	data formats / FITS / Astropy / units / WCS	0.93	8.14	8.78	7,825	43
1.1	pandas / parquet / datetime / Dask / table schema	1.77	0.57	0.32	14,917	3

Appendix

Table 3: Cluster Coverage (1.0 - 4.4): SWE-Rebench (Original) vs. SWE-QA (Test). [Back to Appendix Contents]

Cluster ID	Cluster Name	Original Ratio (%)	Test Ratio (%)	Coverage ( $\times$ )	Origin Count	Test Count
5.0	asyncio / async context / resource cleanup	2.11	0.19	0.09	17,811	1
5.1	multiprocessing / advanced runtime / concurrency / heterogeneous compute	1.07	0.00	0.00	9,034	0
5.2	runtime error handling / DB transactions / retry / logging system	2.48	0.76	0.31	20,891	4
5.3	threading / execution limits / scheduling / memory / timeout	2.17	0.76	0.35	18,278	4
5.4	connection lifecycle / protocol handling / low-level failures	2.08	0.19	0.09	17,569	1
5.5	parallel execution / distributed frameworks / task graphs	1.72	0.00	0.00	14,511	0

Appendix

Table 4: Cluster Coverage (5.0 - 9.3): SWE-Rebench (Original) vs. SWE-QA (Test). [Back to Appendix Contents]

Hyperparameter	Value
Precision	bfloat16
Max sequence length	32,768
Optimizer	AdamW
Learning rate	$5\times 10^{-6}$
Weight decay	0.05
LR scheduler	Cosine

Appendix

Table 5: Hyperparameters for SFT of SWE-QA-Pro 8B. [Back to Appendix Contents]

Hyperparameter	Value
Max turns	25
Max prompt length	2,048
Max response length	8,192
Max observation length	28,000
Temperature	1.0
Top-p	1.0

Appendix

Table 6: Hyperparameters for RL of SWE-QA-Pro 8B. [Back to Appendix Contents]

Cluster	Subcluster	Description
Input / Parsing / Data Conversion	Unicode / formatting / parsing / data validation	Character encoding, string normalization, and file format sanitization; focuses on correctness of raw text and low-level input structure.
	SQL / structured grammar / templating engine	SQL syntax, AST grammars, and templating systems; covers grammar rules and templated string generation.
	CLI args / syntax / regex / command completion	Argument parsing, Bash/Zsh completion, and regex issues; applies when user input must be parsed or matched interactively.
	File import/export / metadata / sorting / recurrence	File loading, metadata extraction, sorting logic, and recurrence handling for structured data transfer.
Data / Array / Image / Coordinate	Data formats / FITS / Astropy / units / WCS	Scientific data formats and astronomy-specific coordinate systems, including units and world coordinate systems.
	Pandas / parquet / datetime / Dask / table schema	Tabular data manipulation, schema handling, and time-indexed datasets.

Appendix

Table 7: Question Cluster taxonomy used in SWE-QA-Pro [Back to Appendix Contents]


Type	Intention	Definition
What	Architecture exploration	Identify components or structural organization of the system.
	Concept / Definition	Understand the meaning or semantics of code elements.
	Dependency tracing	Identify relationships or dependencies among code elements.
Why	Design rationale	Explain why certain design decisions are made.
	Purpose exploration	Understand the intended purpose of a function or component.

Appendix

Table 8: Taxonomy of Repository-Level Question Intentions [Back to Appendix Contents]

Cluster ID	Cluster Name	Count
0.0	Unicode / formatting / parsing / data validation	5
0.1	SQL / structured grammar / templating Engine	6
0.2	CLI args / syntax / regex / command completion	8
0.3	file import/export / metadata / sorting / recurrence	6
1.0	data formats / FITS / Astropy / units / WCS	4
1.1	pandas / parquet / datetime / Dask / table schema	5

Appendix

Table 9: Cluster Statistics (Counts per Question Cluster) [Back to Appendix Contents]

Class Name	Sub-class Name	Count	Total Num
Why (Causal Queries)	Performance & Scalability	33	65
	Design Rationale	24
	Purpose & Role	8
What (Factual Queries)	Architecture & Components	20	51
	Dependency & Inheritance	17
	Concepts & Definitions	14

Appendix

Table 10: QA Type Statistics [Back to Appendix Contents]

QA Type Name	Claude Sonnet 4.5	Gemini 2.5 Pro	GPT 4.1	GPT-4o	DeepSeek V3.2	Avg.
Why (Causal Queries)	38.72	38.18	37.65	32.50	36.87	36.78
Performance & Scalability	37.27	37.32	37.69	29.34	35.30	35.39
Design Rationale	41.38	38.49	37.99	35.76	38.68	38.46
Purpose & Role	36.48	40.83	36.46	35.72	37.92	37.48
What (Factual Queries)	39.27	38.59	37.66	30.93	37.95	36.88
Architecture & Components	38.38	38.52	37.82	30.41	37.48	36.52

Appendix

Table 11: Results across Different Question Types by SWE-QA-Pro [Back to Appendix Contents]

QA Type Name	Qwen3 8B	Qwen3 32B	Devstral Small-2-24B-Ins	Llama 3.3-70B-Ins	SWE-QA-Pro 8B (SFT)	SWE-QA-Pro 8B (SFT+RL)	Avg.
Why (Causal Queries)	31.52	32.83	35.52	26.66	32.35	33.54	32.07
Performance & Scalability	29.55	30.95	33.49	24.64	30.40	30.62	29.94
Design Rationale	33.74	34.68	38.56	28.39	34.66	37.42	34.57
Purpose & Role	33.04	35.00	34.75	29.71	33.46	33.96	33.32
What (Factual Queries)	29.28	30.11	36.50	21.69	32.95	34.17	30.78
Architecture & Components	28.33	29.52	36.53	20.78	32.97	34.87	30.50

Appendix

Table 12: Results across Different Question Types (Open Models and SWE-QA-Pro Variants) [Back to Appendix Contents]

Repo Name	Claude Sonnet 4.5	Gemini 2.5 Pro	GPT 4.1	GPT-4o	DeepSeek V3.2	Avg.
PSyclone	38.37	37.97	34.53	25.68	35.20	34.35
Pillow	39.90	37.33	39.33	33.73	37.93	37.64
cekit	42.47	39.77	39.37	32.80	38.47	38.58
checkov	37.10	36.90	34.30	28.40	36.27	34.59
docker-py	44.50	44.23	41.43	39.17	43.13	42.49
dwave-cloud-client	39.87	36.47	36.03	30.25	36.87	35.90

Appendix

Table 13: Results across Different Repositories by SWE-QA-Pro [Back to Appendix Contents]

Repo Name	Qwen3 8B	Qwen3 32B	Devstral Small-2-24B-Ins	Llama 3.3-70B-Ins	SWE-QA-Pro 8B (SFT)	SWE-QA-Pro 8B (SFT+RL)	Avg.
PSyclone	27.20	26.10	37.47	23.86	31.73	33.87	30.04
Pillow	29.60	31.80	36.27	24.50	35.37	32.70	31.71
cekit	28.07	33.27	39.57	23.07	35.87	33.83	32.28
checkov	27.47	29.20	34.43	22.68	33.62	32.73	30.02
docker-py	36.03	40.00	41.47	27.59	40.20	39.70	37.50
dwave-cloud-client	32.37	30.73	33.80	24.52	35.27	36.13	32.14

Appendix

Table 14: Results across Different Repositories (Open Models and SWE-QA-Pro Variants) [Back to Appendix Contents]

Cluster ID	Name	Claude Sonnet 4.5	Gemini 2.5 Pro	GPT 4.1	GPT-4o	DeepSeek V3.2	Avg.
0.0	Unicode / formatting / parsing / data validation	42.73	41.07	38.07	34.20	38.00	38.81
0.1	SQL / structured grammar / templating Engine	43.06	42.28	38.11	38.00	39.22	40.13
0.2	CLI args / syntax / regex / command completion	40.67	37.21	35.17	33.88	39.08	37.20
0.3	file import/export / metadata / sorting / recurrence	36.20	37.17	34.06	29.75	39.17	35.27
1.0	data formats / FITS / Astropy / units / WCS	44.58	40.25	36.00	37.00	40.58	39.68
1.1	pandas / parquet / datetime / Dask / table schema	39.08	42.27	37.67	30.25	40.93	38.04

Appendix

Table 15: Results across Question Clusters (Closed Models) [Back to Appendix Contents]

Cluster ID	Name	Qwen3 8B	Qwen3 32B	Devstral Small-2-24B-Ins	Llama 3.3-70B-Ins	SWE-QA-Pro 8B (SFT)	SWE-QA-Pro 8B (SFT+RL)	Avg.
0.0	Unicode / formatting / parsing / data validation	31.47	33.40	38.87	24.17	39.53	38.40	34.31
0.1	SQL / structured grammar / templating Engine	30.00	32.78	39.44	21.76	35.33	38.89	33.03
0.2	CLI args / syntax / regex / command completion	29.79	30.29	31.12	22.72	33.54	35.54	30.50
0.3	file import/export / metadata / sorting / recurrence	31.33	35.61	36.28	23.67	34.00	33.00	32.31
1.0	data formats / FITS / Astropy / units / WCS	30.50	28.42	38.42	20.38	34.17	42.83	32.45
1.1	pandas / parquet / datetime / Dask / table schema	25.67	27.20	40.07	24.45	35.80	35.87	31.51

Appendix

Table 16: Results across Question Clusters (Open Models and SWE-QA-Pro Variants) [Back to Appendix Contents]

Figure 6: Example of a comparison of GPT-4o and SWE-QA-8B (SFT+RL) answers against the ground-truth. [Back to Appendix Contents]

Figure 7: Example of Comparison between Reference Answer and Human Annotated Answer (web3.py). [Back to Appendix Contents]

Figure 8: Example of a comparison of SWE-QA-8B (SFT) and SWE-QA-8B (SFT+RL) answers. [Back to Appendix Contents]

Figure 9: Example of Comparison between Reference Answer and Human Annotated Answer (fritionless-py). [Back to Appendix Contents]

Figure 10: Example of Comparison between Reference Answer and Human Annotated Answer (hy). [Back to Appendix Contents]

Explore similar work

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Sep 29, 2026Hantian Ding, Chloe Bi, Jiacheng Zhu +5Software Engineering Agents Repository-Level Code Generation
Meta Superintelligence Labs
Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering
Mar 27, 2026Yoseph Berhanu Alebachew, Hunter Leary, Swanand Vaishampayan +1LLM Evaluation Software Engineering Benchmarks
Department of Computer Science, Virginia Tech Blacksburg, Virginia, USA
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
May 8, 2026Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12Coding Agents Software Engineering Benchmarks
1Scale AI

Sep 29, 2026J/K move · Enter open · S save

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang+4

Meta Superintelligence Labs

Software Engineering Agents Repository-Level Code Generation Code Generation Benchmarks

Open paper AskPDF arXiv

Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.

Abstract

Explore similar work

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution