SWEPatchTest: Evaluating Software Tests Beyond Reproduction

Hua Lu; Pengcheng Wang; Han Li; Zichen Tian; Yanyu Chen; Huang He; Bingjin Chen

cs.SEOct 13, 2026

SWEPatchTest: Evaluating Software Tests Beyond Reproduction

Authors: Hua Lu, Pengcheng Wang, Han Li, Zichen Tian, Yanyu Chen, Huang He, Bingjin Chen

Organizations: Baidu Inc. · Nanjing University · The Chinese University of Hong Kong

Abstract

Coding agents use generated tests to decide whether a proposed patch satisfies an issue's requirements. A suite that fails on the buggy program and passes after applying the reference patch can still accept an incorrect patch or reject a correct patch. We introduce SWEPATCHTEST, a test-generation benchmark with 619 tasks from SWE-bench Pro, SWE-bench Multilingual, and DeepSWE and 82,579 candidate patches collected from coding-agent trajectories. By replaying each suite on the buggy and patched codebases, \benchmark jointly measures issue reproduction, acceptance of correct patches, and rejection of incorrect patches. Joint Validation Rate (JVR), which requires all three criteria on the same task, exposes failures hidden by single-reference evaluation: the best-performing configuration achieves only 9.21% JVR despite reproducing the reference transition on 59.29% of tasks. We further use validation scores to select and scale demonstrations for supervised fine-tuning (SFT). The resulting training recipe approximately triples JVR and improves downstream repair across three SWE-bench benchmarks. Patch discrimination thus provides both a diagnostic measure of test quality and an execution-based signal for curating agent training data.

Figures & tables

Benchmark	F2P	Correct pool	Incorrect pool	Joint Validation	Tasks	Test scale	Multilingual
TestEval	✗	✗	✗	✗	210	Function	✗
TestGenEval	✗	✗	✓	✗	1,210	File	✗
CLOVER	✗	✗	✗	✗	845	Block/file	✗
MultiFileTest	✗	✗	✗	✗	60	Multi-file	3
SWT-Bench	✓	✗	✗	✗	1,983	Repo.	✗
TDD-Bench Verified	✓	✗	✗	✗	449	Repo.	✗

Table 1: Comparison of test-generation benchmarks.

Figure 1: Patch-discrimination profiles of coding-agent configurations on SWEPatchTest .

Figure 2: Overview of test generation, benchmark construction, and SFT in SWEPatchTest .

Model	Reference	Patch-level			Complete-task
Model	GRR	WDR	CAR	F1	CT-WDR	CT-CAR	CT-F1	JVR
GPT-5.6-sol	59.29	88.57	50.20	64.08	47.66	28.59	35.74	9.21
Claude-Fable-5.1	65.11	83.74	62.19	71.37	34.89	30.37	32.47	7.92
Claude-Opus-5	68.17	85.91	66.76	75.13	32.47	31.34	31.89	7.11
Kimi-K3	33.12	80.24	40.64	53.95	49.43	22.94	31.34	5.98
Qwen3.8-max	49.60	84.69	61.68	71.38	32.47	30.21	31.30	5.17

Table 2: Patch-discrimination performance of representative coding agents on SWEPatchTest .

Figure 3: Patch-discrimination case study for Caddy’s default_bind option.

Configuration	SWEPatchTest								Verified		Multilingual		Pro
	GRR	WDR	CAR	F1	CT-WDR	CT-CAR	CT-F1	JVR	AVG	Pass	AVG	Pass	AVG	Pass
SWE SFT	8.89	20.75	25.28	22.79	9.21	19.39	12.49	1.29	67.53	79.20	55.08	72.00	46.56	60.05
+ TestGen Supervision	39.26 +30.37	59.00 +38.25	43.98 +18.70	50.39 +27.60	13.73 +4.52	28.11 +8.72	18.45 +5.96	2.26 +0.97	69.47 +1.94	80.00 +0.80	62.05 +6.97	76.25 +4.25	48.97 +2.41	60.47 +0.42
+ Validation-guided TestGen	40.39 +31.50	64.37 +43.62	50.27 +24.99	56.45 +33.66	25.69 +16.48	31.50 +12.11	28.30 +15.81	3.23 +1.94	70.24 +2.71	80.60 +1.40	63.02 +7.94	76.33 +4.33	49.18 +2.62	60.68 +0.63
+ Scaled TestGen Supervision	41.20 +32.31	72.14 +51.39	51.62 +26.34	60.18 +37.39	31.34 +22.13	32.96 +13.57	32.13 +19.64	3.88 +2.59	71.84 +4.31	82.00 +2.80	63.30 +8.22	78.00 +6.00	50.70 +4.14	61.70 +1.65

Table 3: Test-generation and downstream repair performance across SFT configurations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Initial	Removed	Retained (%)
SWE-bench Pro	731	389	342 (46.8)
SWE-bench Multilingual	300	133	167 (55.7)
DeepSWE	113	3	110 (97.3)
Overall	1,144	525	619 (54.1)

Appendix

Table 4: Task-level curation by benchmark source.

Panel of Figure 4 — Figure 4: Simplified system prompt for the task-curation judge.

Model family	Model configurations
Claude	Claude-Fable-5.1 , Claude-Opus-5 , Claude-Sonnet-5 , Claude-Fable-5 , Claude-Opus-4.8
GPT	GPT-6-Astra , GPT-5.6-Sol , GPT-5.4 , GPT-5.2
DeepSeek	DeepSeek-V4-Pro , DeepSeek-V4-Flash
Gemini	Gemini-3.8-Flash , Gemini-3.1-Pro , Gemini-3-Pro , Gemini-2.5-Pro , Gemini-2.5-Flash
GLM	GLM-5.3 , GLM-5.2
Qwen	Qwen3.8-Max , Qwen3.8-27B , Qwen3.6-35B-A3B , Qwen3.5-35B-A3B

Appendix

Table 5: Model families used for candidate patch collection.

Patch label	Count	Average per task
Correct patches	27,749	44.83
Incorrect patches	54,830	88.58
All candidate patches	82,579	133.41

Appendix

Table 6: Candidate-pool composition by correctness class.

Reference and patch-level metrics
Model	GRR	WDR	CAR	F1
GPT-5.6-sol	59.29 [55.74, 62.84]	88.57 [86.59, 90.44]	50.20 [47.20, 53.20]	64.08 [61.50, 66.54]
Claude-Fable-5.1	65.11 [61.55, 68.66]	83.74 [81.59, 85.82]	62.19 [59.14, 65.18]	71.37 [69.14, 73.51]
Claude-Opus-5	68.17 [64.62, 71.57]	85.91 [84.15, 87.59]	66.76 [63.96, 69.54]	75.13 [73.34, 76.90]
Kimi-K3	33.12 [29.56, 36.51]	80.24 [77.47, 82.95]	40.64 [37.33, 43.87]	53.95 [50.91, 56.80]
Qwen3.8-max	49.60 [45.88, 53.31]	84.69 [82.81, 86.46]	61.68 [58.70, 64.65]	71.38 [69.38, 73.25]

Appendix

Table 7: Bootstrap estimates and 95% confidence intervals for the eight RQ1 coding-agent configurations.

Model	$\Delta\mathrm{JVR}$ [95% CI]	Discordant tasks	$p_{\mathrm{Holm}}$	First (%)	Rank [95% interval]
GPT-5.6-sol	+0.00 [+0.00, +0.00]	–	–	84.5	1.0 [1.0, 2.5]
Claude-Fable-5.1	-1.29 [-3.72, +0.97]	32 / 24	0.350	13.3	2.0 [1.0, 4.0]
Claude-Opus-5	-2.10 [-4.36, +0.16]	35 / 22	0.222	2.0	3.0 [2.0, 4.0]
Kimi-K3	-3.23 [-5.65, -0.81]	41 / 21	0.045	0.2	4.0 [2.0, 6.0]
Qwen3.8-max	-4.04 [-6.46, -1.62]	42 / 17	0.006	0.0	5.0 [4.0, 7.0]
DeepSeek-V4-Pro	-4.68 [-7.11, -2.26]	44 / 15	0.001	0.0	6.0 [4.0, 8.0]

Appendix

Table 8: Paired JVR comparisons and bootstrap rank stability for RQ1.

Source	Configuration	JVR	GRR	WDR	CAR	CT-WDR	CT-CAR
Multilingual ( $n{=}167$ )	GPT-5.6-sol	16.17	87.43	82.57	79.19	38.92	50.30
Pro ( $n{=}342$ )	Claude-Fable-5	8.77	67.25	84.23	66.19	30.70	32.16
DeepSWE ( $n{=}110$ )	GPT-5.6-sol	2.73	38.18	96.82	34.26	44.55	12.73

Appendix

Table 9: Configurations with the highest JVR in each task source.

Reference and patch-level metrics
Configuration	GRR	WDR	CAR	F1
SWE SFT	8.89 [6.79, 11.15]	20.75 [17.91, 23.66]	25.28 [22.10, 28.41]	22.79 [20.44, 25.08]
TestGen Supervision	39.26 [35.70, 42.81]	59.00 [56.40, 61.61]	43.98 [40.76, 47.16]	50.39 [48.05, 52.70]
Validation-guided TestGen	40.39 [36.83, 43.94]	64.37 [61.40, 67.27]	50.27 [47.03, 53.54]	56.45 [53.99, 58.81]
Scaled TestGen Supervision	41.20 [37.64, 44.91]	72.14 [69.62, 74.69]	51.62 [48.19, 55.05]	60.18 [57.75, 62.50]
Complete-task metrics

Appendix

Table 10: Bootstrap estimates and 95% confidence intervals for the four RQ2 SFT configurations.

Configuration	$\Delta\mathrm{JVR}$ [95% CI]	Discordant tasks	$p_{\mathrm{Holm}}$	First (%)	Rank [95% interval]
SWE SFT	+0.00 [+0.00, +0.00]	–	–	0.0	4.0 [3.0, 4.0]
TestGen Supervision	+0.97 [-0.16, +2.10]	3 / 9	0.146	0.4	3.0 [2.0, 4.0]
Validation-guided TestGen	+1.94 [+0.81, +3.23]	2 / 14	0.008	17.2	2.0 [1.0, 3.0]
Scaled TestGen Supervision	+2.58 [+1.29, +4.04]	2 / 18	0.001	82.4	1.0 [1.0, 2.0]

Appendix

Table 11: Paired JVR comparisons and bootstrap rank stability for RQ2.

Software Engineering Benchmarks

Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.

See all 155 papers

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum
Oct 12, 2026Junjielong Xu, Huachao Zhu, Alyssa Wu +2RL for Code Generation Software Engineering Agents
School of Data Science, The Chinese University of Hong Kong, Shenzhen · Tencent
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI
ExecCritic: Learn to Test, Test to Improve for Coding Agents
Sep 8, 2026Leitian Tao, Baolin Peng, Haorui Wang +7RL for Code Generation Software Engineering Agents
University of Wisconsin–Madison · Microsoft Research · Georgia Tech

Oct 12, 2026J/K move · Enter open · S save

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

Junjielong Xu, Huachao Zhu, Alyssa Wu, Zhichao Hu+1

School of Data Science, The Chinese University of Hong Kong, Shenzhen · Tencent

RL for Code Generation Software Engineering Agents Reward Shaping Curriculum RL

Open paper AskPDF arXiv

The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.

Abstract

Software Engineering Benchmarks

Explore similar work

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

ExecCritic: Learn to Test, Test to Improve for Coding Agents