cs.LGMar 25, 2026
SaveA Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula
Organizations: University of Tübingen · Max Planck Institute for Intelligent Systems · Meta FAIR
Abstract
Reinforcement learning (RL) has emerged as a powerful paradigm for improving large language models beyond supervised fine-tuning, yet sustaining performance gains at scale remains an open challenge, as data diversity and structure, rather than volume alone, become the limiting factor. We address this by introducing a scalable multi-turn synthetic data generation pipeline in which a teacher model iteratively refines problems based on in-context student performance summaries, producing structured difficulty progressions without any teacher fine-tuning. Compared to single-turn generation, this multi-turn approach substantially improves the yield of valid synthetic problems and naturally produces stepping stones, i.e. easier and harder variants of the same core task, that support curriculum-based training. We systematically study how task difficulty, curriculum scheduling, and environment diversity interact during RL training across the Llama3.1-8B Instruct and Qwen3-8B Base model families, with additional scaling experiments on Qwen2.5-32B. Our results show that synthetic augmentation consistently improves in-domain code and in most cases out-of-domain math performance, and we provide empirical insights into how curriculum design and data diversity jointly shape RL training dynamics.
Figures & tables
Figure 1: Overview of the multi-turn synthetic data pipeline. A seed snippet, sampled from random code or real coding puzzles, serves as inspiration for the teacher. In the first turn, the teacher generates an initial problem according to the current environment’s rules, and the student attempts to solve it multiple times. In later turns, the teacher receives a summary of the student’s performance (pass rate and representative solutions) and adapts the problem accordingly. Invalid or redundant generations are filtered and deduplicated before inclusion in the dataset.
Figure 2: Example of multi-turn data generation. The top-left panel shows the seed snippet provided to the teacher, taken from a real coding puzzle. In turn 1, the teacher generates a puzzle with a student pass rate of 0.875 ( ). In turn 2, after observing the student’s performance, the teacher produces a harder variant with a pass rate of 0.25.
Figure 3: Scaling with real data in Qwen3-8B Base. We compare RL training on 25K and 81K real coding problems using GRPO (3 seeds). Performance is tracked on in-domain ( LCB ) and out-of-domain ( Math500 and AIME2024 ) benchmarks throughout training. Performance gains plateau early, indicating limited benefit from scaling real data alone.
Figure 4: Synthetic data augmentations in Llama3.1-8B Instruct. RL training on 25K real code-contest problems (baseline) versus 25K real plus 20K synthetic problem augmentations seeded from solved real questions (3 seeds). Synthetic augmentation improves performance across both in-domain (code: LCB ) and out-of-domain (math: Math500 , AIME2024 ) benchmarks.
Figure 5: Synthetic data augmentations in Qwen3-8B Base. RL training on real code-contest problems (baseline) versus with synthetic problem augmentation (3 seeds). Synthetic data are seeded either with answers to real questions ( SYNTH-Real-Aug ) or with random code snippets from starcoderdata ( SYNTH*-Aug ). Performance improves primarily on in-domain (code: LCB ), while out-of-domain (math: Math500 , AIME2024 ) benchmark performance remains comparable or slightly lower.
Figure 6: Synthetic data augmentations in Qwen2.5-32B Base. RL training on 25K real code-contest problems (baseline) versus 25K real plus 20K synthetic problem augmentations seeded from solved real questions (2 seeds). Synthetic augmentation shows improved performance trends for both in-domain (code: LCB ) and out-of-domain (math: Math500 , AIME2024 ) benchmarks.
Figure 7: RL with pure synthetic data in Qwen3-8B Base. Instead of augmenting the 25K real code-contest questions, RL is performed solely on synthetic induction problems generated with two seeding strategies: (1) real questions as inspiration seeds ( SYNTH-Real-Aug ), and (2) random code snippets from starcoderdata ( SYNTH*-Aug ). Results shown for 3 seeds. Performance is comparable across both setups, with slightly faster in-domain convergence ( LCB ) when seeding with real data.
Figure 8: Examples of easy – medium – hard question chains. Each chain illustrates a progression in task complexity within the same conceptual domain, derived by starting from a hard question and generating simpler variants that capture its core subproblems. This reverse construction ensures that easy and medium questions correspond to meaningful components of the original hard task rather than trivial simplifications.
Figure 9: Hard curriculum
Figure 12: Hard curriculum: hard transitions between easy – medium – hard splits.
Figure 16: Qwen3-8B Base: RL on easy , medium , hard splits. Comparison of RL training when restricted to questions of a single difficulty level (3036 per level) across easy , medium , and hard subsets, with the uniform chain baseline (3036 total, 1012 per level) for reference. Performance shown on in-domain (code: LCB ) and out-of-domain (math: Math500 , AIME2024 ) benchmarks across 3 seeds.
Figure 17: Comparing different curriculum strategies in Qwen3-8B Base. The figure compares RL training with different curriculum strategies over extended difficulty splits ( easy , easy–medium , medium , hard ). Curves show pass@1 and pass@10 on the in-domain benchmark ( LCB , including LCBv5 - easy , medium , and hard splits) and pass@1 and pass@5 on the out-of-domain benchmarks ( Math500 , AIME2024 ), averaged across three seeds. Reverse curricula starting from medium -level questions mitigate overfitting to easy tasks and maintain stronger performance on harder splits.
Figure 18: Reversed curriculum experiments in Qwen3-8B Base. The figure compares RL training with different reverse curriculum strategies over extended difficulty splits ( easy , easy–medium , medium , hard ). We evaluate two variants: starting with the medium split (“medium-start”) and starting with the hard split (“hard-start”). Curves show pass@1 and pass@10 on the in-domain benchmark ( LCB , including LCBv5 - easy , medium , and hard splits) and pass@1 and pass@5 on the out-of-domain benchmarks ( Math500 , AIME2024 ), averaged across three seeds. The medium-start reverse curriculum shows faster convergence and lower variance across seeds compared to the hard-start variant.
Figure 19: Scaling the number of RL environments in Qwen3-8B Base. The figure compares RL training on 20K synthetic problems from a single environment ( induction ) with training on an equal total of 20K synthetic problems distributed across four environments: induction , abduction , deduction , and fuzzing . Curves show pass@1 and pass@10 on in-domain ( LCB ) and pass@1 and pass@5 out-of-domain ( Math500 , AIME2024 ) benchmarks throughout RL training. Distributing the data budget across environments improves out-of-domain generalization and yields higher pass@10 scores in-domain, while reducing overfitting on the easy split.
Figure 20: Scaling the number of RL environments in Llama3.1-8B Instruct compared to real data. The figure compares RL training on 25K real code-contest problems with training on a total of 20K synthetic problems distributed across four environments: induction , abduction , deduction , and fuzzing . Curves show pass@1 and pass@10 on the in-domain benchmark ( LCB ) and pass@1 and pass@5 on out-of-domain benchmarks ( Math500 , AIME2024 ) throughout RL training. Despite using less data, training across multiple synthetic environments improves out-of-domain generalization and yields higher in-domain pass@10 scores compared to real-only training.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 21: RL with synthetic data augmentation seeded from real problems in Qwen3-8B Base. RL training on 25K real code-contest questions (baseline) versus the same data augmented with 20K synthetic induction problems generated using real questions as inspiration seeds ( Real 25K + SYNTH-Real-Aug 20K ), as well as training solely on the synthetic partition ( SYNTH-Real-Aug 20K ). Performance shown on in-domain ( LCB ) and out-of-domain ( Math500 , AIME2024 ) benchmarks.
Figure 22: RL with synthetic data augmentation seeded from random code snippets in Qwen3-8B Base. RL training on 25K real code-contest questions (baseline) versus the same data augmented with 20K synthetic induction problems generated using random snippets from starcoderdata ( Real 25K + SYNTH* Induction 20K ), as well as training solely on the synthetic partition ( SYNTH* Induction 20K ). Performance shown on in-domain ( LCB ) and out-of-domain ( Math500 , AIME2024 ) benchmarks.
Figure 23: RL with pure synthetic data in Qwen3-8B Base. Instead of augmenting the 25K real code-contest questions, RL is performed solely on synthetic induction problems generated with two seeding strategies: (1) real questions as inspiration seeds ( SYNTH-Real-Aug ), and (2) random code snippets from starcoderdata ( SYNTH*-Aug , 3 seeds).
Figure 24: Scaling with real data vs. synthetic augmentation in Qwen3-8B Base. Comparison of RL training on 25K and 81K real code-contest problems, and on 25K real data augmented with 20K synthetic problems generated from random starcoderdata snippets ( Real 25K + SYNTH*-Aug 20K ). Performance is shown on in-domain ( LCB ) and out-of-domain ( Math500 and AIME2024 ) benchmarks throughout training. Synthetic augmentation yields additional gains over the 81K real-data baseline across all LCB splits except medium .
Figure 25: Chains with hard curriculum in Qwen3-8B Base. Extended results for RL training on easy–medium–hard question chains using the hard curriculum schedule. Shown are in-domain results ( LCB , including easy, medium, and hard splits) and out-of-domain results ( Math500 , AIME2024 ) with pass@1, pass@5/pass@10 metrics plotted over RL training steps.
Figure 26: Chains with soft curriculum in Qwen3-8B Base. Extended results for RL training on easy–medium–hard question chains using the soft curriculum schedule. Performance is reported for LCB (easy, medium, hard splits) and out-of-domain benchmarks ( Math500 , AIME2024 ) with pass@1, pass@5/pass@10 metrics over training steps.
Figure 27: Chains without curriculum in Qwen3-8B Base. Extended results for RL training on easy–medium–hard question chains under uniform sampling across difficulty levels. Plots show in-domain ( LCB ) and out-of-domain ( Math500 , AIME2024 ) performance, including easy, medium, and hard splits, with pass@1, pass@5/pass@10 metrics over RL training steps.
| Difficulty | Pass Rate Range | Number of Problems |
|---|---|---|
| easy | 0.85–0.97 | 10000 |
| easy–medium | 0.61–0.85 | 7479 |
| medium | 0.26–0.61 | 5268 |
| hard | 0.10–0.26 | 2220 |
Table 3: Difficulty bins and data split sizes used in curriculum experiments.
Figure 28: Scaling RL environments with real-data baseline in Qwen3-8B Base. Comparison of RL training on 20K problems from a single environment ( induction ), 20K problems distributed across four environments ( induction , abduction , deduction , fuzzing ), and the 25K real code-contest baseline. Shown are in-domain results ( LCB , pass@1 and pass@10) and out-of-domain results ( Math500 , AIME2024 , pass@1). Distributing the data budget across multiple environments improves out-of-domain generalization and results in competitive or higher (in terms of pass@10) in-domain performance compared to the real-data baseline.
More in this topic
RL for Code Generation
Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.
See all 150 papers