Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning

Dong Huang; Mark Harman; Jie M. Zhang; Zhijiang Guo; Mingzhe Du; See Kiong Ng

cs.SEAug 25, 2026

Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning

Authors: Dong Huang, Mark Harman, Jie M. Zhang, Zhijiang Guo, Mingzhe Du, See Kiong Ng

Organizations: National University of Singapore, Singapore · University College London, London, UK · King’s College London, London, UK · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, CN

Abstract

We introduce Ockhamareto, a single-shot GRPO framework for unit-test generation and selection, based on the principles of Ockham's Razor and Pareto Optimality. Ockhamareto has two principal components: (i)~~a Pareto-gated Bonus that rewards only rollouts non-dominated in~~(mutation, $-$ #tests) space, and (ii)Token-level Segment Credit, which attributes each test's marginal mutation kills back to the tokens of its unit-test block. On the UnLeakedTestBench~(ULT), Ockhamareto strictly Pareto-dominates the strongest RL baseline(MIST-RL). Furthermore, it dominates on {\em each and all} optimization objectives, catching more bugs ( $49.9\%$ vs $31.3\%$ mutation score at $N{=}5$ ), using fewer tests ( $2.60$ vs $4.67$ on average), thereby achieving $3.4\times$ the per-test trade-off improvement. The advantage is found in all four benchmarks~(HumanEval+, MBPP+, CodeContests, TestGenEval-Lite): Ockhamareto leads both mutation and coverage metrics on every one, always with the smallest suite. Ockhamareto also outperforms the state-of-the-art at all model scales, adding $+30$ -- $35$ ~pp mutation at 4B, 9B, and 27B model sizes. We also show that the knee point of the optimal trade-off between efficiency and effectiveness on the Pareto front is not correlated with obvious more easily computed proxy metrics, such as function size. This finding motivates the Pareto front computation; it is needed to identify this crucial engineering trade-off for each function under test.

Figures & tables

Figure 1. Overview of Ockhamareto . (A) For each task the policy $\pi_{\theta}$ samples a GRPO group of $K$ complete pytest suites in a single shot, and each suite is executed in the sandbox against a prebuilt mutant pool. (B) Pareto gate (§ 3.3 ): only rollouts non-dominated in (mutation, $-$ #tests) space receive the conciseness bonus, so $R_{4}$ , which catches fewer mutants than $R_{1}$ with more tests, is dominated and earns nothing, while $R_{1}$ – $R_{3}$ are on the frontier. (C) Segment credit (§ 3.4 ): the mutants each test is the first to kill give a per-test score $\Delta_{k}$ , which is zero-meaned across the suite and mapped onto that test’s own token span via the tokenizer’s offset mapping. The update uses per-token advantages $A_{i,t}=A_{i}+o_{i,t}$ under a KL constraint to $\pi_{\text{ref}}$ , so high-yield tests are reinforced and redundant ones penalized within a single trajectory.

Benchmark	Method	Mut. (%)	Stmt. (%)	Branch (%)	Corr. (%)	$n_{\text{actual}}$	Eff. (%)
ULT	Qwen3.5-4B	14.5	21.6	20.5	27.6	2.53	5.7
	+ GRPO	18.9 (+4.4)	31.0 (+9.4)	28.7 (+8.2)	42.0	3.43	5.5
	+ MIST-RL	31.3 (+16.8)	49.1 (+27.5)	45.9 (+25.4)	72.7	4.67	6.7
	+ Ockhamareto	49.9 (+35.4)	63.1 (+41.5)	62.7 (+42.2)	68.8	2.60	19.2
HumanEval+	Qwen3.5-4B	69.1	54.1	44.2	86.2	4.70	14.7
	+ GRPO	69.2 (+0.1)	53.5 (-0.6)	45.2 (+1.0)	86.5	4.60	15.0

Table 1. End-to-end results at

N{=}5

across five held-out benchmarks. All methods are Qwen3.5-4B single-shot fewshot; only the training method differs. CodeContests uses each task’s provided incorrect Python submissions as the fault source (in place of cosmic-ray mutants). TGE-Lite is reported on the shared usable set where every method produces at least one passing test on the reference module. Ockhamareto leads mutation, statement, and branch coverage on every benchmark with the smallest suite in every row.

Figure 2. First-

N

union statement coverage, branch coverage, and mutation score on ULT (

N\!\in\!\{1,\dots,5\}

). Ockhamareto saturates by

N{=}3

and dominates all three metrics at every budget; its first test alone (

33.3\%

mutation) already exceeds MIST-RL’s fifth (

31.3\%

Variant	Mut. (%)	Stmt. (%)	Branch (%)	Corr. (%)	$n_{\text{actual}}$	Eff. (%)
Ockhamareto	49.9	63.1	62.7	68.8	2.60	19.2
$w/o$ segment credit ( $W_{\text{seg}}{=}0$ )	41.2	57.8	54.2	77.1	3.07	13.4
$w/o$ conciseness bonus ( $W_{\text{P}}{=}W_{\text{N}}{=}0$ )	40.0	56.7	55.6	67.2	3.21	12.5

Table 2. Component ablation on ULT, Qwen3.5-4B at

N{=}5

. Each row removes one of the two Ockhamareto components while holding the rest fixed: the token-level segment credit or the Pareto-gated conciseness bonus. “Eff.” is mutants killed per test (Mutation

\div

n_{\text{actual}}

Scale	Model	Mut. (%)	Stmt. (%)	Branch (%)	Corr. (%)	$n_{\text{actual}}$	Eff. (%)	$\Delta$ mut
4B	Base	14.5	21.6	20.5	27.6	2.53	5.7	—
4B	Ockhamareto	49.9	63.1	62.7	68.8	2.60	19.2	+35.4
9B	Base	19.8	29.0	27.1	38.3	3.22	6.2	—
9B	Ockhamareto	54.3	66.2	65.5	64.2	2.71	20.0	+34.5
27B	Base	30.5	39.9	38.6	54.6	3.58	8.5	—
27B	Ockhamareto	61.0	70.5	69.8	60.1	2.82	21.6	+30.5

Table 3. Base vs. Ockhamareto at three model scales (

N{=}5

): the same fixed configuration delivers a

+30

–

35

pp mutation lift at every scale, with suite size compressed (

n_{\text{actual}}\!<\!3

) while base suites grow; base-27B remains

+19.4

pp behind Ockhamareto-4B.

Segment-credit weight $W_{\text{seg}}$ ( $W_{\text{P}}{=}W_{\text{N}}{=}0.15$ fixed)
	Mut. (%)	Stmt. (%)	Branch (%)	Corr. (%)	$n_{\text{actual}}$	Eff. (%)
$0.00$	41.2	57.8	54.2	77.1	3.07	13.4
$0.25$	45.4	64.2	63.3	74.6	2.88	15.8
$0.50$ (def.)	49.9	63.1	62.7	68.8	2.60	19.2
$0.75$	43.1	54.7	55.0	62.5	2.18	19.8
Conciseness-bonus weight $W_{\text{P}}{=}W_{\text{N}}$ ( $W_{\text{seg}}{=}0.5$ fixed)

Table 4. Hyper-parameter sensitivity (Qwen3.5-4B,

N{=}5

). Each panel varies one weight with the other held at its default; the default rows repeat the headline model. The

W_{\text{seg}}{=}0

and

W_{\text{P}}{=}0

rows match the corresponding ablation rows of Table 2 (the

W_{\text{P}}

analysis ties

W_{\text{N}}

to the same value throughout).

Front metric	Min	Max	Mean	Med.	Mode
Knee location (#tests)	1	14	3.6	3	3
Front size (#points)	1	5	2.2	2	2
Dominated suites	30	198	159.6	168	166
Dominated : front ratio	3.4	198	66.9	47.0	–

Table 5. Distribution of per-function Pareto-front metrics over the

100

-task study (

\approx 165

valid sampled suites per task). The median function has a knee at

3

tests and a front of only

2

points, and

97\%

of sampled suites are dominated: without front extraction, a randomly chosen suite is almost surely suboptimal.

Figure 3. The (mutation, #tests) Pareto trade-off on a

100

-task random ULT subset:

K{=}50

suites per policy sampled at

T{=}0.7

; each curve shows the best mutation achievable with at most

n

tests, averaged over tasks. The dashed empirical Pareto front pools all policies’ suites and takes the per-task best; Ockhamareto tracks it within

3

–

5

pp and supplies

60.8\%

of front points.

Figure 4. Fifteen per-function (mutation, #tests) sample clouds and pooled Pareto fronts, stratified by knee location (rows: knee at

1

2

3

(the median),

4

, and the maximum end

n\!\geq\!12

; within each row, the three functions with the highest best-achievable mutation among fronts with at least two points). Knee points are circled; front vertices are colored by the supplying policy. The low-budget region of every front belongs to Ockhamareto; baselines appear only on the diminishing-returns tail.

Figure 5. Fifteen per-function fronts stratified by front size (rows:

1

point, the degenerate no-trade-off case, through

5

points, the maximum observed; within each row, the three functions with the highest best-achievable mutation). Knee points are circled where defined; front vertices are colored by the supplying policy. Degenerate fronts are single Ockhamareto points at

100\%

mutation; larger fronts offer the engineer a menu of discrete operating points.

Diagnostic	Value
seg_active rate (token map OK)	67.8%
mean segments / suite	3.49
sandbox_fail rate	0.15%

Table 6. Segment-credit signal health over the training run; rollouts where re-tokenization does not round-trip (

\sim

32%) train under scalar GRPO.

Automated Test Generation

Automated test generation creates inputs, scenarios, and checks to expose software and system defects with less manual effort.

See all 133 papers

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models
Sep 24, 2026Nils Kiele, Zainab Saad, Zirui Wang +2Automated Software Testing Mutation Testing
Department of Electrical and Software Engineering, University of Calgary, AB, Canada
MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning
Mar 2, 2026Sicheng Zhu, Jiajun Wang, Jiawei Ai +1Automated Test Generation Mutation Testing
Fudan University, Shanghai, China · University of Science and Technology of China, Hefei, China · Xi’an Jiaotong University, Xi’an, China +1
EvoOtter: Evolutionary Reproduction Test Generator
Jul 3, 2026Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1Evolutionary Optimization LLM-Based Program Synthesis
IBM

Sep 24, 2026J/K move · Enter open · S save

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

Nils Kiele, Zainab Saad, Zirui Wang, Steve Drew+1

Department of Electrical and Software Engineering, University of Calgary, AB, Canada

Automated Software Testing Mutation Testing LLM Code Generation

Open paper AskPDF arXiv

Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate large numbers of trivial, redundant, or equivalent mutants that limit their practical use for identifying gaps in a test suite. While recent large language model (LLM)-based approaches generate more realistic faults, most remain test-blind: The model sees only the source code and cannot reason about what existing tests already cover. We propose test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests. We evaluate this approach across a set of five LLMs -- Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.1 Codex Mini, GPT 4.1 Mini, Qwen3-32B -- on the HumanEval and MBPP benchmarks. The extended EvalPlus test suites serve as an automated oracle to verify whether surviving mutants represent genuine bugs. Test-aware prompting yields verified fault rates of 87.7% (HumanEval) and 79.1% (MBPP), meaning these mutants pass all base tests but are caught by the oracle. This vastly outperforms the matched test-blind prompting (which yields only 12.2% and 23.0%, respectively) and the traditional rule-based tool mutmut (4.4% and 5.7%). While fault subtlety (the fraction of extended tests a mutant fails) remains comparable across all three methods, test-awareness minimizes the computational cost per verified fault, compared to test-blind prompting. Exposing an LLM to existing unit tests shifts mutant generation from untargeted bug injection toward effective discovery of weaknesses in an existing test suite. Our work establishes a concrete foundation for future research to scale test-aware mutant generation to production-level environments.

Abstract

Automated Test Generation

Explore similar work

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning

EvoOtter: Evolutionary Reproduction Test Generator