Modern code generation models exhibit longer outputs, accelerated capability growth, and changed training dynamics, rendering traditional training methodologies, algorithms, and datasets ineffective for improving their performance. To address these training bottlenecks, we propose MicroCoder-GRPO, an improved Group Relative Policy Optimization approach with three innovations: conditional truncation masking to improve long output potential while maintaining training stability, diversity-determined temperature selection to maintain and encourage output diversity, and removal of KL loss with high clipping ratios to facilitate solution diversity. MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6, with more pronounced gains under extended context evaluation. Additionally, we release MicroCoder-Dataset, a more challenging training corpus that achieves 3x larger performance gains than mainstream datasets on LiveCodeBench v6 within 300 training steps, and MicroCoder-Evaluator, a robust framework with approximately 25% improved evaluation accuracy and around 40% faster execution. Through comprehensive analysis across more than thirty controlled experiments, we reveal 34 training insights across seven main aspects, demonstrating that properly trained models can achieve competitive performance with larger counterparts.
Figures & tables
Figure 1: Cross-Model Training Dynamics . Performance and response length across Qwen 2.5 and Qwen 3 models, illustrating generation-specific training behaviors and output characteristics.
Figure 2: Truncation Masking Effects on Training. Performance trends under different masking strategies, comparing no mask, complete masking, and conditional masking at various rates.
Figure 3: Temperature Dynamics and Scheduling Analysis. Temperature robustness trends during training showing increasing stability at higher temperatures, convergence of output diversity across different temperature settings, training failure when low temperatures cause initial diversity to fall below convergence values, and better performance of dynamic temperature scheduling (low-to-high transition) compared to static temperature approaches.
Figure 4: Influence of KL Loss and Clip Ratio on Training Dynamics . Performance comparison between standard KL loss and removed KL loss with high clipping, illustrating the relationship between diversity maintenance and sustained performance improvement.
Figure 5: Dataset Quality Comparison. Training dynamics comparing MicroCoder and DeepCoder datasets across accuracy, critic reward, and response length metrics, demonstrating learning effectiveness on challenging problems.
Figure 6: Training Performance with Different Code Evaluators. Accuracy and reward during training using MicroCoder-Evaluator versus LiveCodeBench Evaluator, demonstrating the benefits of robust output validation for coding task training. The right subfigure shows the efficiency improvements of MicroCoder-Evaluator through parallel processing optimization compared to the original DeepCoder single-threaded version.
Figure 7: Illustrative performance comparison between MicroCoder-GRPO and baseline approaches across different model scales and output lengths, illustrating training accuracy, response length, and output diversity dynamics, demonstrating the better stability and sustained improvements of the proposed method under both preset and extended context lengths.
AtCoder
LeetCode
LiveCodeBench
Easy
Medium
Hard
All
Easy
Medium
Hard
All
Easy
Medium
Hard
All
Not trained
Qwen3-1.7B Instruct (4K)
69.2
10.6
1.3
19.2
33.8
3.8
0
10.7
55.2
7.2
0.9
16.1
Qwen3-1.7B Instruct (8K)
69.2
10.6
1.3
19.2
33.8
3.8
0
10.7
55.2
7.2
0.9
16.1
Qwen3-4B-Instruct (4K)
95.2
44.2
9.2
37.3
73.5
19.2
2.5
28.6
86.6
31.7
7.5
34.1
Qwen3-4B-Instruct (8K)
95.2
45.2
10.0
37.9
76.5
18.3
2.5
29.0
87.8
31.7
8.1
34.7
Table 1: Quantitative evaluation results across different datasets comparing MicroCoder-GRPO against baseline methods on various model sizes, difficulty levels, and output budgets to assess algorithmic improvements and extended reasoning capabilities.
Figure 8: On-Policy versus Off-Policy Training Effects . Performance comparison across different train batch size settings, demonstrating that off-policy configurations increase training stability while intermediate settings achieve optimal performance.
Figure 9: Context Length Influence on Training Dynamics. Performance trends across different maximum output length settings showing accuracy, response length growth, and output diversity, demonstrating irreversible effects of early-stage length limitations on model capabilities.
Reinforcement learning with verifiable rewards (RLVR) trains language models using programmatically checkable signals such as unit-test outcomes, enabling direct optimization for functional correctness in code generation. We conduct an empirical study of RLVR for Python code generation on the MBPP benchmark using two small models (Qwen3-0.6B and Llama3.2-1B) with LoRA fine-tuning. Across multiple reward formulations such as: unit-test-only rewards, static-analysis-only shaping via the Ruff linter, and a combined reward, we compare group-based policy optimization variants (GRPO and GSPO) and evaluate both functional correctness and behavioral diagnostics. In our experimental setting, RLVR improves pass@1 on MBPP test by up to 13 percentage points under proposed combined reward configuration. However, we find that reward shaping can induce systematic behavioral shifts: using only static-analysis penalties may bias the policy toward shorter completions that reduce lint errors without reliably improving functional correctness. In contrast, combined rewards mitigate this degeneration and yield more stable trade-offs between correctness and style constraints. Overall, our results highlight that RLVR effectiveness for code generation is highly sensitive to reward design and optimization granularity, and that diagnostics beyond pass@1, including generation length, Ruff severity profiles, and execution error types are useful for identifying failure modes.
Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning, but relies on ground-truth supervision that is costly or infeasible, especially in coding tasks. Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores, achieving notable success on mathematical reasoning benchmarks. However, code generation poses distinct challenges: programs are structurally complex, semantically equivalent solutions may differ syntactically, and verification typically requires execution. Whether these intrinsic reward methods transfer effectively to code remains unexplored. In this work, we present a systematic empirical study of intrinsic reward methods for code generation. We conduct extensive experiments on LiveCodeBench, systematically evaluating representative certainty-based Reinforcement Learning from Internal Feedback (RLIF) approaches under different training scenarios and hyperparameter settings. Our experiments reveal that certainty-based methods yield early gains but inevitably collapse: models progressively shorten outputs and lose reasoning capability, with collapse speed sensitive to sample size and temperature. When used to initialize RLVR training, RLIF pre-training offers no significant improvement over training from scratch. We also provide actionable recommendations for using intrinsic rewards for training code reasoning models. Our study shows both the promise and limitations of intrinsic reward methods for code, informing future work on code models and agents.
Xiaolong Jin, Xuandong Zhao, Wenbo Guo +2
Purdue University · UC Berkeley · UC Santa Barbara
Reinforcement Learning from Verifiable Rewards (RLVR) has driven recent progress in code large language models by leveraging execution-based feedback from unit tests, but its scalability is fundamentally constrained by the availability and reliability of high-quality test cases. We propose CodeScaler, a reward model designed to scale both reinforcement learning training and test-time inference for code generation. CodeScaler is trained on carefully curated preference data derived from verified code problems and incorporates syntax-aware code extraction and validity-preserving reward shaping to ensure stable and robust optimization. Across four coding benchmarks, CodeScaler consistently outperforms execution-based RL by +1.55 points on Qwen3-8B-Base and +4.23 points on Qwen3-14B-Base. By further scaling to 44K problems with additional synthetic data, CodeScaler yields +14.64 points improvement over the base model without requiring any test cases. At inference time, CodeScaler serves as an effective test-time scaling method, achieving performance comparable to unit test approaches while providing a 10-fold reduction in latency. Moreover, CodeScaler surpasses existing reward models on RM-Bench not only in the code domain (+3.3 points), but also in general and reasoning domains (+2.7 points on average).