Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning
Authors: Dong Huang, Mark Harman, Jie M. Zhang, Zhijiang Guo, Mingzhe Du, See Kiong Ng
Organizations: National University of Singapore, Singapore · University College London, London, UK · King’s College London, London, UK · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, CN
We introduce Ockhamareto, a single-shot GRPO framework for unit-test generation and selection, based on the principles of Ockham's Razor and Pareto Optimality. Ockhamareto has two principal components: (i)a Pareto-gated Bonus that rewards only rollouts non-dominated in(mutation, −#tests) space, and (ii)Token-level Segment Credit, which attributes each test's marginal mutation kills back to the tokens of its unit-test block. On the UnLeakedTestBench~(ULT), Ockhamareto strictly Pareto-dominates the strongest RL baseline(MIST-RL). Furthermore, it dominates on {\em each and all} optimization objectives, catching more bugs (49.9% vs 31.3% mutation score at N=5), using fewer tests (2.60 vs 4.67 on average), thereby achieving 3.4× the per-test trade-off improvement. The advantage is found in all four benchmarks~(HumanEval+, MBPP+, CodeContests, TestGenEval-Lite): Ockhamareto leads both mutation and coverage metrics on every one, always with the smallest suite. Ockhamareto also outperforms the state-of-the-art at all model scales, adding +30--35~pp mutation at 4B, 9B, and 27B model sizes. We also show that the knee point of the optimal trade-off between efficiency and effectiveness on the Pareto front is not correlated with obvious more easily computed proxy metrics, such as function size. This finding motivates the Pareto front computation; it is needed to identify this crucial engineering trade-off for each function under test.
Figures & tables
Figure 1. Overview of Ockhamareto . (A) For each task the policy πθ samples a GRPO group of K complete pytest suites in a single shot, and each suite is executed in the sandbox against a prebuilt mutant pool. (B) Pareto gate (§ 3.3 ): only rollouts non-dominated in (mutation, − #tests) space receive the conciseness bonus, so R4 , which catches fewer mutants than R1 with more tests, is dominated and earns nothing, while R1 – R3 are on the frontier. (C) Segment credit (§ 3.4 ): the mutants each test is the first to kill give a per-test score Δk , which is zero-meaned across the suite and mapped onto that test’s own token span via the tokenizer’s offset mapping. The update uses per-token advantages Ai,t=Ai+oi,t under a KL constraint to πref , so high-yield tests are reinforced and redundant ones penalized within a single trajectory.
Benchmark
Method
Mut. (%)
Stmt. (%)
Branch (%)
Corr. (%)
nactual
Eff. (%)
ULT
Qwen3.5-4B
14.5
21.6
20.5
27.6
2.53
5.7
+ GRPO
18.9 (+4.4)
31.0 (+9.4)
28.7 (+8.2)
42.0
3.43
5.5
+ MIST-RL
31.3 (+16.8)
49.1 (+27.5)
45.9 (+25.4)
72.7
4.67
6.7
+ Ockhamareto
49.9 (+35.4)
63.1 (+41.5)
62.7 (+42.2)
68.8
2.60
19.2
HumanEval+
Qwen3.5-4B
69.1
54.1
44.2
86.2
4.70
14.7
+ GRPO
69.2 (+0.1)
53.5 (-0.6)
45.2 (+1.0)
86.5
4.60
15.0
Table 1. End-to-end results at N=5 across five held-out benchmarks. All methods are Qwen3.5-4B single-shot fewshot; only the training method differs. CodeContests uses each task’s provided incorrect Python submissions as the fault source (in place of cosmic-ray mutants). TGE-Lite is reported on the shared usable set where every method produces at least one passing test on the reference module. Ockhamareto leads mutation, statement, and branch coverage on every benchmark with the smallest suite in every row.
Figure 2. First- N union statement coverage, branch coverage, and mutation score on ULT ( N∈{1,…,5} ). Ockhamareto saturates by N=3 and dominates all three metrics at every budget; its first test alone ( 33.3% mutation) already exceeds MIST-RL’s fifth ( 31.3% ).
Variant
Mut. (%)
Stmt. (%)
Branch (%)
Corr. (%)
nactual
Eff. (%)
Ockhamareto
49.9
63.1
62.7
68.8
2.60
19.2
w/o segment credit ( Wseg=0 )
41.2
57.8
54.2
77.1
3.07
13.4
w/o conciseness bonus ( WP=WN=0 )
40.0
56.7
55.6
67.2
3.21
12.5
Table 2. Component ablation on ULT, Qwen3.5-4B at N=5 . Each row removes one of the two Ockhamareto components while holding the rest fixed: the token-level segment credit or the Pareto-gated conciseness bonus. “Eff.” is mutants killed per test (Mutation ÷nactual ).
Scale
Model
Mut. (%)
Stmt. (%)
Branch (%)
Corr. (%)
nactual
Eff. (%)
Δ mut
4B
Base
14.5
21.6
20.5
27.6
2.53
5.7
—
Ockhamareto
49.9
63.1
62.7
68.8
2.60
19.2
+35.4
9B
Base
19.8
29.0
27.1
38.3
3.22
6.2
—
Ockhamareto
54.3
66.2
65.5
64.2
2.71
20.0
+34.5
27B
Base
30.5
39.9
38.6
54.6
3.58
8.5
—
Ockhamareto
61.0
70.5
69.8
60.1
2.82
21.6
+30.5
Table 3. Base vs. Ockhamareto at three model scales ( N=5 ): the same fixed configuration delivers a +30 – 35 pp mutation lift at every scale, with suite size compressed ( nactual<3 ) while base suites grow; base-27B remains +19.4 pp behind Ockhamareto-4B.
Table 4. Hyper-parameter sensitivity (Qwen3.5-4B, N=5 ). Each panel varies one weight with the other held at its default; the default rows repeat the headline model. The Wseg=0 and WP=0 rows match the corresponding ablation rows of Table 2 (the WP analysis ties WN to the same value throughout).
Front metric
Min
Max
Mean
Med.
Mode
Knee location (#tests)
1
14
3.6
3
3
Front size (#points)
1
5
2.2
2
2
Dominated suites
30
198
159.6
168
166
Dominated : front ratio
3.4
198
66.9
47.0
–
Table 5. Distribution of per-function Pareto-front metrics over the 100 -task study ( ≈165 valid sampled suites per task). The median function has a knee at 3 tests and a front of only 2 points, and 97% of sampled suites are dominated: without front extraction, a randomly chosen suite is almost surely suboptimal.
Figure 3. The (mutation, #tests) Pareto trade-off on a 100 -task random ULT subset: K=50 suites per policy sampled at T=0.7 ; each curve shows the best mutation achievable with at most n tests, averaged over tasks. The dashed empirical Pareto front pools all policies’ suites and takes the per-task best; Ockhamareto tracks it within 3 – 5 pp and supplies 60.8% of front points.
Figure 4. Fifteen per-function (mutation, #tests) sample clouds and pooled Pareto fronts, stratified by knee location (rows: knee at 1 , 2 , 3 (the median), 4 , and the maximum end n≥12 ; within each row, the three functions with the highest best-achievable mutation among fronts with at least two points). Knee points are circled; front vertices are colored by the supplying policy. The low-budget region of every front belongs to Ockhamareto; baselines appear only on the diminishing-returns tail.
Figure 5. Fifteen per-function fronts stratified by front size (rows: 1 point, the degenerate no-trade-off case, through 5 points, the maximum observed; within each row, the three functions with the highest best-achievable mutation). Knee points are circled where defined; front vertices are colored by the supplying policy. Degenerate fronts are single Ockhamareto points at 100% mutation; larger fronts offer the engineer a menu of discrete operating points.
Diagnostic
Value
seg_active rate (token map OK)
67.8%
mean segments / suite
3.49
sandbox_fail rate
0.15%
Table 6. Segment-credit signal health over the training run; rollouts where re-tokenization does not round-trip ( ∼ 32%) train under scalar GRPO.
Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate large numbers of trivial, redundant, or equivalent mutants that limit their practical use for identifying gaps in a test suite. While recent large language model (LLM)-based approaches generate more realistic faults, most remain test-blind: The model sees only the source code and cannot reason about what existing tests already cover. We propose test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests. We evaluate this approach across a set of five LLMs -- Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.1 Codex Mini, GPT 4.1 Mini, Qwen3-32B -- on the HumanEval and MBPP benchmarks. The extended EvalPlus test suites serve as an automated oracle to verify whether surviving mutants represent genuine bugs. Test-aware prompting yields verified fault rates of 87.7% (HumanEval) and 79.1% (MBPP), meaning these mutants pass all base tests but are caught by the oracle. This vastly outperforms the matched test-blind prompting (which yields only 12.2% and 23.0%, respectively) and the traditional rule-based tool mutmut (4.4% and 5.7%). While fault subtlety (the fraction of extended tests a mutant fails) remains comparable across all three methods, test-awareness minimizes the computational cost per verified fault, compared to test-blind prompting. Exposing an LLM to existing unit tests shifts mutant generation from untargeted bug injection toward effective discovery of weaknesses in an existing test suite. Our work establishes a concrete foundation for future research to scale test-aware mutant generation to production-level environments.
Nils Kiele, Zainab Saad, Zirui Wang +2
Department of Electrical and Software Engineering, University of Calgary, AB, Canada
Large Language Models (LLMs) often fail to generate correct code on the first attempt, which requires using generated unit tests as verifiers to validate the solutions. Despite the success of recent verification methods, they remain constrained by a "scaling-by-quantity" paradigm. This brute-force approach suffers from a critical limitation: it yields diminishing returns in fault detection while causing severe test redundancy. To address this, we propose MIST-RL (Mutation-based Incremental Suite Testing via Reinforcement Learning), a framework that shifts the focus to "scaling-by-utility". We formulate test generation as a sequential decision process optimized via Group Relative Policy Optimization (GRPO). Specifically, we introduce a novel incremental mutation reward combined with dynamic penalties, which incentivizes the model to discover new faults while it suppresses functionally equivalent assertions. Experiments on HumanEval+ and MBPP+ demonstrate that MIST-RL outperforms state-of-the-art baselines. It achieves a +28.5% higher mutation score while reducing the number of test cases by 19.3%. Furthermore, we show that these compact, high-utility tests serve as superior verifiers, which improves downstream code reranking accuracy on HumanEval+ by 3.05% over the SOTA baseline with 10 candidate samples. The source code and data are provided in the supplementary material.
Sicheng Zhu, Jiajun Wang, Jiawei Ai +1
Fudan University, Shanghai, China · University of Science and Technology of China, Hefei, China · Xi’an Jiaotong University, Xi’an, China +1
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using large language models to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with large language models for software engineering.