CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

Junjielong Xu; Huachao Zhu; Alyssa Wu; Zhichao Hu; Pinjia He

cs.LGOct 12, 2026

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

Authors: Junjielong Xu, Huachao Zhu, Alyssa Wu, Zhichao Hu, Pinjia He

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen · Tencent

Abstract

The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.

Figures & tables

Figure 1: Overview of CodeLift. [Top] Behavior balancing : groups tests and patches by their outcomes, weights test groups by the patch groups they kill, and scores patches using these weights. [Bottom] Snapshot slicing : uses self-tests to select candidate checkpoints, scores intermediate snapshots and combines the snapshot deltas

\Delta_{i}

with final advantage

A_{\mathrm{final}}

to obtain slice advantage

A_{i}

Figure 2: Cumulative test-suite distributions on R2E-Gym. [Left] Tests per parameterized family. [Middle] F2P/P2P per issue distribution. [Right] final-patch test pass frequency distribution.

Method	SWE-bench Verified				SWE-bench Pro (Py)
	pass@1	pass@2	pass@4	pass@8	pass@1	pass@2	pass@4	pass@8
Base	26.00	35.14	43.52	50.60	12.17	18.96	27.00	35.71
Binary	34.63	43.04	50.36	56.80	16.54	25.00	34.16	41.73
PR-Harmonic	34.45	43.76	51.62	58.00	16.92	24.70	32.56	38.72
PR-VeRPO	34.45	42.41	49.43	55.40	19.50	28.24	36.62	42.86
PR-Process	35.55	44.11	51.60	58.40	19.36	27.48	35.79	42.48

Table 1: Main results on SWE benchmarks. Entries are pass@

k

(%); Base is the initial model. Deltas show CodeLift gains in percentage points, and bold marks the highest score. We report raw PassRate and PR-Reweighted through their final evaluated checkpoint (step 50) in Figure 3 b.

Figure 3: Training dynamics. Scores monitored during training: (a,b) SWE-bench Verified scores (every 10 steps) and (c) R2E-Gym training rollouts (10-step moving average).

Table 5

Figure 4: Training signal density : rollout groups with nonzero advantages (%).

Figure 5: Progress-Regression preferences : (a) Ratio of regression penalty to progress reward. (b) Positive credit by final-patch test pass rate: improved (green), unchanged (gray), worsened (red).

Figure 6: Positive reward changes for fully passing a test family.

Figure 7: Test weights training dynamics . (a,b) Test-weight shares within rollout groups under PassRate and CodeLift. (c) Share of rollout groups where every final patch passes all tests.

Figure 8: Test groups training dynamics . (a) Groups per issue. (b) Average weight per group.

Figure 9: Regressions training dynamics . (a) Pass-rate decreases count . (b) decrease magnitude .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Issues per batch	64
Rollouts per issue	8
Minibatch (issues)	16
Update epochs	1
Learning rate	$10^{-6}$
Clip width	$4\times 10^{-4}$

Appendix

Table 4: Shared training parameters.

Method	pass@1	pass@2	pass@4	pass@8
Binary	+3.67[+2.10,+5.28]	+4.49[+2.61,+6.38]	+5.48[+3.12,+7.85]	+6.00[+2.80,+9.20]
PR-Harmonic	+3.85[+2.20,+5.55]	+3.77[+1.88,+5.70]	+4.22[+1.89,+6.63]	+4.80[+1.60,+8.00]
PR-Process	+2.75[+1.23,+4.30]	+3.42[+1.69,+5.17]	+4.24[+1.91,+6.55]	+4.40[+1.00,+7.80]
PR-VeRPO	+3.85[+2.25,+5.48]	+5.12[+3.22,+7.03]	+6.41[+3.97,+8.92]	+7.40[+4.00,+10.80]
CL-Outcome	+2.47[+0.95,+4.03]	+3.44[+1.66,+5.25]	+4.47[+2.22,+6.75]	+5.00[+2.00,+8.20]

Appendix

Table 5: CodeLift’s gains on Verified with 95% paired bootstrap intervals (pp).

Credit rule	Share (%)
Binary	48.07
CL-Outcome	74.99
CodeLift	75.02

Appendix

Table 6: Nonzero-advantage rollout groups on the same CodeLift rollouts.

	Failure frequency $d_{g}$		Weight $w_{g}$
Passing rollouts	Before	After	Before	After
More	0.609	0.439	0.460	0.434
Fewer	0.439	0.598	0.395	0.471
Same	0.475	0.475	0.347	0.361

Appendix

Table 7: Changes in test groups with fixed membership across training windows.

Rollout source	CodeLift	CL-Outcome
Intermediate slices: $A_{\mathrm{final}}\leq 0$ to $A_{i}>0$	23.4	17.1
Intermediate slices: $A_{\mathrm{final}}>0$ to $A_{i}<0$	23.9	24.1

Appendix

Table 8: Effect of slicing on recorded rollouts (%).

Endpoint	Score $S(s_{i})$	Correction $\Delta_{i}$	Advantage $A_{i}$
First snapshot	0.5	$-0.2$	$+0.25$
Second snapshot	0.8	$+0.1$	$+0.55$
Final patch	0.7	$0$	$+0.45$

Appendix

Table 9: Slice credit for rollout 5 in Figure 1 .

Test outcomes (passed / total)
Tested behavior	Group weight	Final patch A	Final patch B
Deprecation warnings	0.375	$\checkmark$ (3/3)	$\times$ (0/3)
Module namespace	0.375	$\times$ (0/1)	$\checkmark$ (1/1)
Block construction	0.250	$\times$ (0/1)	$\checkmark$ (1/1)
Other shared tests	0.000	4/4	4/4
Total tests passed		7/9	6/9

Appendix

Table 10: Test outcomes and final rewards for two pandas patches. Counts report passed/total tests;

\checkmark

\times

mark passing/failing test groups. Weights are computed from all eight final patches in the rollout group. The higher reward under each rule is bold.

Case	State	Tests passed	Score $S(s_{i})$	Slice credit $A_{i}^{j}$
NumPy	Intermediate snapshot	8/8	1.000	+0.143
	Final patch	3/8	0.333	-0.524
aiohttp	Intermediate snapshot	63/66	0.000	-0.857
	Final patch	66/66	1.000	+0.143

Appendix

Table 11: Intermediate and final credit in two recorded rollouts.

Panel of Figure 10 — Figure 10: System and user prompt templates (part 1 of 3).

Panel of Figure 11 — Figure 11: User prompt template (part 2 of 3).

Panel of Figure 12 — Figure 12: User prompt template (part 3 of 3).

RL for Code Generation

Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.

See all 150 papers

SWEPatchTest: Evaluating Software Tests Beyond Reproduction
Oct 13, 2026Hua Lu, Pengcheng Wang, Han Li +4Software Engineering Benchmarks Automated Test Generation
Baidu Inc. · Nanjing University · The Chinese University of Hong Kong
ExecCritic: Learn to Test, Test to Improve for Coding Agents
Sep 8, 2026Leitian Tao, Baolin Peng, Haorui Wang +7RL for Code Generation Software Engineering Agents
University of Wisconsin–Madison · Microsoft Research · Georgia Tech
TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble
Oct 7, 2026Pengfei He, Jiayuan Zhou, Shaowei Wang +1Automated Test Generation Automated Software Testing
University of Manitoba, Winnipeg, Canada · Huawei Canada, Markham, Canada · Huawei Technologies, Hangzhou, China

Oct 13, 2026J/K move · Enter open · S save

SWEPatchTest: Evaluating Software Tests Beyond Reproduction

Hua Lu, Pengcheng Wang, Han Li, Zichen Tian+3

Baidu Inc. · Nanjing University · The Chinese University of Hong Kong

Software Engineering Benchmarks Automated Test Generation Benchmark Validity Automated Program Repair

Open paper AskPDF arXiv

Coding agents use generated tests to decide whether a proposed patch satisfies an issue's requirements. A suite that fails on the buggy program and passes after applying the reference patch can still accept an incorrect patch or reject a correct patch. We introduce SWEPATCHTEST, a test-generation benchmark with 619 tasks from SWE-bench Pro, SWE-bench Multilingual, and DeepSWE and 82,579 candidate patches collected from coding-agent trajectories. By replaying each suite on the buggy and patched codebases, \benchmark jointly measures issue reproduction, acceptance of correct patches, and rejection of incorrect patches. Joint Validation Rate (JVR), which requires all three criteria on the same task, exposes failures hidden by single-reference evaluation: the best-performing configuration achieves only 9.21% JVR despite reproducing the reference transition on 59.29% of tasks. We further use validation scores to select and scale demonstrations for supervised fine-tuning (SFT). The resulting training recipe approximately triples JVR and improves downstream repair across three SWE-bench benchmarks. Patch discrimination thus provides both a diagnostic measure of test quality and an execution-based signal for curating agent training data.

Abstract

RL for Code Generation

Explore similar work

SWEPatchTest: Evaluating Software Tests Beyond Reproduction

ExecCritic: Learn to Test, Test to Improve for Coding Agents

TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble