Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping

Yiwen Zhang; Xiaodong Yan; Zhenyu Huang; Deng Zhao; Liang Jiang; Qing Cui; Zujie Wen; Zhiqiang Zhang

cs.AIAug 25, 2026

Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping

Authors: Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang,

Organizations: Zhejiang University · Ant Group

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging "near-correct" faulty codes, RobustTests captures latent logical discrepancies and employs validator agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+ dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at https://huggingface.co/datasets/sid6/RobustTests.

Figures & tables

Figure 1: Illustration of current methods for test cases synthesis by LLM.(a) Generate Test cases Directly , instructing the LLM to directly synthesize a suite of test cases compliant with the problem specifications.(b) Generate Test cases by Generator Program , where LLM is first instructed to generate generator programs designed to automate the synthesis of test case inputs.

Figure 2: Overview of the proposed RobustTests framework. The framework consists of three core components: (i) Automated Test case Generation in 3.1 , aims to generate high-quality test cases inputs capable of effectively distinguishing correct code from faulty codes; (ii) Test case Validation and Selection in 3.2 , ensures that the generated test cases are semantically sound and capable of diagnosing a broad spectrum of faulty code while simultaneously maximizing set parsimony; and (iii) Dense Reward Function Design in 3.3 , accounts for the potential fallibility of the synthesized test case suite.

Method	Livecodebench	Codeforces
Method	Score	Score	Rating	Percentile
Naive LLM Generation	65.75	35.41	83.36	91.70
HardTests	65.25	35.65	83.58	91.35
CodeContests +	65.41	35.56	83.96	91.45
CodeContests-O	66.21	36.43	84.35	91.81
RobustTests(Ours)	68.39	38.50	85.99	94.67

Table 1: Performance of baselines and RobustTests on the LiveCodeBench and CodeForces benchmarks. The best result is bold and the second result is underline.

Figure 3: Correctness distribution of Qwen3-32B over ten trials across all CodeContests + problems, evaluated using the CodeContests + test case suite. The x-axis represents the distribution of pass rates over ten trials(pass@10), while the y-axis denotes the number of problems within each pass rate interval.

Panel of Figure 4 — Figure 4: Relationship between TSP values and LiveCodeBench performance across different test cases.

Method	Livecodebench	Codeforces
Method	Score	Score	Rating	Percentile
RobustTests*(Ours)	67.84	38.47	85.16	93.97
w/o Test case Synthesis	66.91	36.41	84.4	92.12
w/o Diversity-Driven Selection	66.82	36.23	84.12	91.96
w/o Both Module	65.41	35.56	83.96	91.45

Table 2: Ablation study on the test case synthesis strategy of RobustTests* on LiveCodeBench and CodeForces benchmarks. RobustTests* denotes a variant of the RobustTests framework that excludes the dense reward mechanism. We investigate the performance impact of without Test case Generation (the second row), without Diversity-Driven Selection (the third row) and without Both Module(the forth row). The best results are in bold.

Figure 5: The pass rate distribution of Qwen3-32B over ten trials across all

\text{CodeContests}^{+}_{\text{train}}

problems, evaluated against the test cases provided by RobustTests and without Test case Generation and its configuration without Diversity-Driven Selection.

Figure 6: Correlation between the TSP values of test cases employed during training and the coding performance of LLM on the LiveCodeBench benchmark.

Dataset	Method	LiveCodeBench	Codeforces
Dataset	Method	Score	Score	Rating	Percentile
$\text{CodeContests}^{+}_{\text{train}}$	Sparse Reward	65.41	83.96	35.56	91.45
$\text{CodeContests}^{+}_{\text{train}}$	Dense Reward	66.30	84.24	37.74	92.05
RobustTests*	Sparse Reward	67.84	85.16	38.47	93.97
RobustTests*	Dense Reward	68.39	85.99	38.50	94.67
RobustTests* w/o validator	Sparse Reward	66.02	83.82	35.91	91.28

Table 3: Ablation study on the Reward Module of RobustTests on LiveCodeBench and CodeForces benchmarks, using both RobustTests* and CodeContests + as the training datasets. The best results are in bold.

Panel of Figure 7 — Figure 7: Ablation study on the reward scale in the dense reward function on LiveCodeBench and Codeforces benchmarks. The scale is varied from $0$ to $0.20$ using RobustTests ∗ as the training dataset.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Optimizer	AdamW
Learning Rate ( $\eta$ )	$5\times 10^{-6}$
Adam $\beta_{1},\beta_{2}$	$0.9,0.95$
Weight Decay	$0.01$
Gradient Clipping	$1.0$
Global Batch Size	$256$

Appendix

Table 4: Hyperparameters for Experiment

Panel of Figure 8 — Figure 8: One of prompts on Faulty Code Generation Stage

Panel of Figure 9 — Figure 9: One of prompts on Faulty Code Generation Stage

Panel of Figure 10 — Figure 10: One of prompts on Faulty Code Generation Stage

Panel of Figure 11 — Figure 11: Prompt on Faulty-Code-Driven Test Case Generation Stage

Panel of Figure 12 — Figure 12: Prompt on Input Validator Generation Stage

RL for Code Generation

Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.

See all 150 papers

EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning
Mar 13, 2026Chi Ruan, Dongfu Jiang, Huaye Zeng +2RL for Code Generation Reinforcement Learning with Verifiable Rewards
University of Waterloo · Vector Institute · Harvard University
Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback
May 28, 2026Egor Skopin, Evgeny KotelnikovRL for Code Generation Reward Shaping
Noisy Test-Time Reinforcement Learning for Code LLMs
Sep 26, 2026Xikai Yang, Hieu Trung Nguyen, Dunyuan Xu +4Language Model Robustness RL for Code Generation
The Chinese University of Hong Kong · Huazhong University of Science and Technology

Mar 13, 2026J/K move · Enter open · S save

EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning

Chi Ruan, Dongfu Jiang, Huaye Zeng, Ping Nie+1

University of Waterloo · Vector Institute · Harvard University

RL for Code Generation Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) is a promising approach for improving code generation in large language models, but its effectiveness is limited by weak and static verification signals in existing coding RL datasets. In this paper, we propose a solution-conditioned and adversarial verification framework that iteratively refines test cases based on the execution behaviors of candidate solutions, with the goal of increasing difficulty, improving discriminative power, and reducing redundancy. Based on this framework, we introduce EvolveCoder-22k, a large-scale coding reinforcement learning dataset constructed through multiple rounds of adversarial test case evolution. Empirical analysis shows that iterative refinement substantially strengthens verification, with pass@1 decreasing from 43.80 to 31.22. Reinforcement learning on EvolveCoder-22k yields stable optimization and consistent performance gains, improving Qwen3-4B by an average of 4.2 points across four downstream benchmarks and outperforming strong 4B-scale baselines. Our results highlight the importance of adversarial, solution-conditioned verification for effective and scalable reinforcement learning in code generation.

Abstract

RL for Code Generation

Explore similar work

EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learning

Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback

Noisy Test-Time Reinforcement Learning for Code LLMs