MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning

Sicheng Zhu; Jiajun Wang; Jiawei Ai; Xin Li

cs.AIMar 2, 2026

MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning

Authors: Sicheng Zhu, Jiajun Wang, Jiawei Ai, Xin Li

Organizations: Fudan University, Shanghai, China · University of Science and Technology of China, Hefei, China · Xi’an Jiaotong University, Xi’an, China · South China University of Technology, Guangzhou, China

Abstract

Large Language Models (LLMs) often fail to generate correct code on the first attempt, which requires using generated unit tests as verifiers to validate the solutions. Despite the success of recent verification methods, they remain constrained by a "scaling-by-quantity" paradigm. This brute-force approach suffers from a critical limitation: it yields diminishing returns in fault detection while causing severe test redundancy. To address this, we propose MIST-RL (Mutation-based Incremental Suite Testing via Reinforcement Learning), a framework that shifts the focus to "scaling-by-utility". We formulate test generation as a sequential decision process optimized via Group Relative Policy Optimization (GRPO). Specifically, we introduce a novel incremental mutation reward combined with dynamic penalties, which incentivizes the model to discover new faults while it suppresses functionally equivalent assertions. Experiments on HumanEval+ and MBPP+ demonstrate that MIST-RL outperforms state-of-the-art baselines. It achieves a +28.5% higher mutation score while reducing the number of test cases by 19.3%. Furthermore, we show that these compact, high-utility tests serve as superior verifiers, which improves downstream code reranking accuracy on HumanEval+ by 3.05% over the SOTA baseline with 10 candidate samples. The source code and data are provided in the supplementary material.

Figures & tables

Figure 1: Motivation: Quality Over Quantity. (a) Existing “scaling-by-quantity” methods (Blue/Green lines), including the SOTA CodeRM-8B and the larger Qwen3-14B, exhibit rapid logarithmic saturation in Fault Detection Capability, indicating severe Semantic Redundancy (shaded area). In contrast, MIST-RL (Orange) maintains a steep growth trajectory, creating a significant Utility Gap . (b) This utility translates directly to downstream effectiveness. MIST-RL solves more problems (% Problems Solved) with fewer test suites compared to baselines, validating that optimizing for marginal utility is more efficient than brute-force scaling.

Figure 2: Illustration of the Incremental Reward Mechanism. The agent receives positive rewards only for killing new mutants (grey circles). Redundant tests (e.g., T3) that fail to reduce the surviving mutant pool trigger a dynamic penalty (

-\rho

Dataset	Model	Avg. Length	Source Pass (%)	Mutant Kill (%)
HumanEval+	Llama-3-8B-Instruct	2.33	40.99	29.18
	CodeRM-8B	7.61	70.61	45.53
	Qwen3-14B	5.12	74.31	58.69
	MIST-RL (Ours)	6.14	74.86	74.03
MBPP+	Llama-3-8B-Instruct	2.35	42.51	22.54
	CodeRM-8B	6.55	72.67	61.08

Table 1: Main Results on HumanEval+, MBPP+, and DS-1000. MIST-RL consistently outperforms both the base model and strong baselines (including the larger Qwen3-14B) in fault detection (Mutant Kill Rate). It achieves these results while generating test suites that are significantly more concise than CodeRM-8B. The best results for Source Pass and Mutant Kill are highlighted in bold .

Panel of Figure 3 — Figure 3: Marginal Utility Analysis. The figures visualize fault-detection efficiency across datasets. MIST-RL (Red) demonstrates a significantly steeper utility curve compared to CodeRM-8B (Blue), confirming that our approach efficiently prioritizes high-utility test cases early in the generation process.

Verifier Model	Reranked Pass@1 (%)
Verifier Model	Samples (N=10)	Samples (N=20)
Random Selection	$\sim$ 22.50	$\sim$ 22.50
CodeRM-8B	45.73	61.59
Qwen3-14B	44.51	55.49
MIST-RL (Ours)	48.78	62.80
Oracle (Upper Bound)	70.12	75.00

Table 2: Test-Driven Code Reranking on HumanEval+. Comparison of Pass@1 accuracy when using test suites generated by different models to select the best code solution from 10 and 20 candidates, respectively.

Figure 4: Ablation Study on HumanEval+. (a) Effectiveness: The Incremental Reward is essential for high mutation scores. (b) Efficiency: The Dynamic Penalty significantly reduces test suite length (preventing bloat) while maintaining performance.

Problem: move_one_ball(arr) (Check if sortable by rotation)

Hard Mutant: Change loop range(1, len)

\to

range(2, len) (Skips shift-by-1 check)

CodeRM-8B Test Case (Fails to Kill):

self.assertTrue(move_one_ball([3, 4, 5, 1, 2]))

Analysis: Requires 2 shifts. The mutant loop starts at index 2, so it still catches this. Result: Mutant Alive.

MIST-RL Test Case (Kills Mutant):

self.assertTrue(move_one_ball([2, 1]))

Figure 5: Case Study on move_one_ball . CodeRM generates complex but redundant tests that miss the “shift-by-one” boundary. MIST-RL generates the minimal counter-example [2, 1] , successfully killing the loop-range mutant.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Case Study: Problem move_one_ball (HumanEval/109)
1. Source Code & Problem Description	2. The ”Hard” Mutant (Off-by-One Error)
def move_one_ball(arr): """ Determine if it is possible to get a sorted array by performing right shift operations. Ex: [3, 4, 5, 1, 2] -> True (2 shifts) Ex: [3, 5, 4, 1, 2] -> False """ if len(arr) == 0: return True sorted_arr = sorted(arr) if arr == sorted_arr: return True # Check all possible rotations for i in range(1, len(arr)): if arr[i:] + arr[:i] == sorted_arr: return True return False	The mutation engine generates an ”Off-by-One” error by modifying the loop range. This simulates a common developer bug where the boundary condition is mishandled. Mutated Code Snippet: # Original: range(1, len(arr)) # Mutated: range(2, len(arr)) for i in range(2, len(arr)): # The loop now skips the case i=1 # (shifting by exactly 1 position). if arr[i:] + arr[:i] == sorted_arr: return True Effect: The function now correctly handles arrays needing 0 shifts or $\geq 2$ shifts, but fails for arrays needing exactly 1 shift.
3. CodeRM-8B Generated Tests (Baseline)	4. MIST-RL Generated Tests (Ours)
Status: FAILED to kill the mutant. CodeRM generates many tests, but they are redundant or ”lucky” (requiring 0 or $\geq 2$ shifts). def test_code_rm(self): # Case 1: Already sorted (0 shifts) # Mutant result: True (Correct) -> Alive self.assertTrue(move_one_ball([1..5])) # Case 2: Requires 2 shifts # Loop starts at 2, so it finds this! # Mutant result: True (Correct) -> Alive self.assertTrue(move_one_ball([3..2])) # Case 3: Large Random # Mutant result: Matches Original -> Alive self.assertTrue(move_one_ball([10..100]))	Status: SUCCESSFULLY killed the mutant. MIST-RL identifies the specific boundary case that requires exactly 1 shift. def test_mist_rl(self): # ... (omitted common tests) ... # CRITICAL TEST CASE FOUND BY MIST-RL # Input: [2, 1] # Sorted: [1, 2] # Requires: Exactly 1 right shift. # Under Mutant (range starts at 2): # The loop range(2, 2) is empty. # Loop is skipped -> Returns False. # Assertion Fails -> MUTANT KILLED self.assertTrue(move_one_ball([2, 1]))

Appendix

Figure 6: Qualitative Analysis on move_one_ball . The figure compares the test suites generated by CodeRM-8B and MIST-RL against a subtle ”Off-by-One” mutant. (Left) The source code and CodeRM’s redundant tests which fail to trigger the bug. (Right) The mutant logic and MIST-RL’s targeted test case ( [2, 1] ), which precisely hits the ”blind spot” of the mutated loop (index 1), demonstrating the model’s ability to learn high-utility boundary tests.

Optimization
Hyperparameter	Value
Optimizer	AdamW
Learning Rate	1.5e-5
LR Scheduler	Cosine
Warmup Ratio	0.03
Global Batch Size	64 (2 GPUs $\times$ 2 batch $\times$ 16 accum)

Appendix

Table 3: Hyperparameters for MIST-RL Training.

Parameter	Symbol	Value
Quality Weight	$\alpha$	0.05
Marginal Utility Weight	$\beta$	3.0
Base Redundancy Penalty	$\rho_{base}$	0.5
Penalty Growth Rate	$\gamma$	1.0
Max Sequence Reference	$K_{max}$	10
Execution Failure Penalty	$R_{fail}$ (Suite)	-100.0

Appendix

Table 4: Coefficients for Incremental Reward Mechanism.

Category	AST Node Type	Implemented Transformations
AOR	ast.Add (+)	$\to$ Sub (-) , Mult (*)
(Arithmetic)	ast.Sub (-)	$\to$ Add (+) , Mult (*)
	ast.Mult (*)	$\to$ Div (/) , Add (+) , Pow (**)
	ast.Div (/)	$\to$ Mult (*) , FloorDiv (//)
	ast.Mod (%)	$\to$ Mult (*) , Add (+)
ROR	ast.Eq (==)	$\to$ NotEq (!=)

Appendix

Table 5: Full Specification of Mutation Operators implemented in FullMutator .

Automated Test Generation

Automated test generation creates inputs, scenarios, and checks to expose software and system defects with less manual effort.

See all 132 papers

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models
Sep 24, 2026Nils Kiele, Zainab Saad, Zirui Wang +2Automated Software Testing Mutation Testing
Department of Electrical and Software Engineering, University of Calgary, AB, Canada
Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping
Aug 25, 2026Yiwen Zhang, Xiaodong Yan, Zhenyu Huang +6RL for Code Generation Reinforcement Learning with Verifiable Rewards
Zhejiang University · Ant Group
Confidence-Gated Transductive Test Generation for Code Reranking
Sep 11, 2026Sungjae Lee, Youngsik Yoon, Seockbean Song +3LLM Reranking Automated Test Generation
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, China

Sep 24, 2026J/K move · Enter open · S save

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

Nils Kiele, Zainab Saad, Zirui Wang, Steve Drew+1

Department of Electrical and Software Engineering, University of Calgary, AB, Canada

Automated Software Testing Mutation Testing LLM Code Generation

Open paper AskPDF arXiv

Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate large numbers of trivial, redundant, or equivalent mutants that limit their practical use for identifying gaps in a test suite. While recent large language model (LLM)-based approaches generate more realistic faults, most remain test-blind: The model sees only the source code and cannot reason about what existing tests already cover. We propose test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests. We evaluate this approach across a set of five LLMs -- Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.1 Codex Mini, GPT 4.1 Mini, Qwen3-32B -- on the HumanEval and MBPP benchmarks. The extended EvalPlus test suites serve as an automated oracle to verify whether surviving mutants represent genuine bugs. Test-aware prompting yields verified fault rates of 87.7% (HumanEval) and 79.1% (MBPP), meaning these mutants pass all base tests but are caught by the oracle. This vastly outperforms the matched test-blind prompting (which yields only 12.2% and 23.0%, respectively) and the traditional rule-based tool mutmut (4.4% and 5.7%). While fault subtlety (the fraction of extended tests a mutant fails) remains comparable across all three methods, test-awareness minimizes the computational cost per verified fault, compared to test-blind prompting. Exposing an LLM to existing unit tests shifts mutant generation from untargeted bug injection toward effective discovery of weaknesses in an existing test suite. Our work establishes a concrete foundation for future research to scale test-aware mutant generation to production-level environments.

Abstract

Automated Test Generation

Explore similar work

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping

Confidence-Gated Transductive Test Generation for Code Reranking