Reinforcement learning (RL) has emerged as a powerful paradigm for improving large language models beyond supervised fine-tuning, yet sustaining performance gains at scale remains an open challenge, as data diversity and structure, rather than volume alone, become the limiting factor. We address this by introducing a scalable multi-turn synthetic data generation pipeline in which a teacher model iteratively refines problems based on in-context student performance summaries, producing structured difficulty progressions without any teacher fine-tuning. Compared to single-turn generation, this multi-turn approach substantially improves the yield of valid synthetic problems and naturally produces stepping stones, i.e. easier and harder variants of the same core task, that support curriculum-based training. We systematically study how task difficulty, curriculum scheduling, and environment diversity interact during RL training across the Llama3.1-8B Instruct and Qwen3-8B Base model families, with additional scaling experiments on Qwen2.5-32B. Our results show that synthetic augmentation consistently improves in-domain code and in most cases out-of-domain math performance, and we provide empirical insights into how curriculum design and data diversity jointly shape RL training dynamics.
Figures & tables
Figure 1: Overview of the multi-turn synthetic data pipeline. A seed snippet, sampled from random code or real coding puzzles, serves as inspiration for the teacher. In the first turn, the teacher generates an initial problem according to the current environment’s rules, and the student attempts to solve it multiple times. In later turns, the teacher receives a summary of the student’s performance (pass rate and representative solutions) and adapts the problem accordingly. Invalid or redundant generations are filtered and deduplicated before inclusion in the dataset.
Figure 2: Example of multi-turn data generation. The top-left panel shows the seed snippet provided to the teacher, taken from a real coding puzzle. In turn 1, the teacher generates a puzzle with a student pass rate of 0.875 ( M=8 ). In turn 2, after observing the student’s performance, the teacher produces a harder variant with a pass rate of 0.25.
Figure 3: Scaling with real data in Qwen3-8B Base. We compare RL training on 25K and 81K real coding problems using GRPO (3 seeds). Performance is tracked on in-domain ( LCB ) and out-of-domain ( Math500 and AIME2024 ) benchmarks throughout training. Performance gains plateau early, indicating limited benefit from scaling real data alone.
Figure 4: Synthetic data augmentations in Llama3.1-8B Instruct. RL training on 25K real code-contest problems (baseline) versus 25K real plus 20K synthetic problem augmentations seeded from solved real questions (3 seeds). Synthetic augmentation improves performance across both in-domain (code: LCB ) and out-of-domain (math: Math500 , AIME2024 ) benchmarks.
Figure 5: Synthetic data augmentations in Qwen3-8B Base. RL training on real code-contest problems (baseline) versus with synthetic problem augmentation (3 seeds). Synthetic data are seeded either with answers to real questions ( SYNTH-Real-Aug ) or with random code snippets from starcoderdata ( SYNTH*-Aug ). Performance improves primarily on in-domain (code: LCB ), while out-of-domain (math: Math500 , AIME2024 ) benchmark performance remains comparable or slightly lower.
Figure 6: Synthetic data augmentations in Qwen2.5-32B Base. RL training on 25K real code-contest problems (baseline) versus 25K real plus 20K synthetic problem augmentations seeded from solved real questions (2 seeds). Synthetic augmentation shows improved performance trends for both in-domain (code: LCB ) and out-of-domain (math: Math500 , AIME2024 ) benchmarks.
Figure 7: RL with pure synthetic data in Qwen3-8B Base. Instead of augmenting the 25K real code-contest questions, RL is performed solely on synthetic induction problems generated with two seeding strategies: (1) real questions as inspiration seeds ( SYNTH-Real-Aug ), and (2) random code snippets from starcoderdata ( SYNTH*-Aug ). Results shown for 3 seeds. Performance is comparable across both setups, with slightly faster in-domain convergence ( LCB ) when seeding with real data.
Figure 8: Examples of easy – medium – hard question chains. Each chain illustrates a progression in task complexity within the same conceptual domain, derived by starting from a hard question and generating simpler variants that capture its core subproblems. This reverse construction ensures that easy and medium questions correspond to meaningful components of the original hard task rather than trivial simplifications.
Figure 9: Hard curriculum
Figure 12: Hard curriculum: hard transitions between easy – medium – hard splits.
Figure 16: Qwen3-8B Base: RL on easy , medium , hard splits. Comparison of RL training when restricted to questions of a single difficulty level (3036 per level) across easy , medium , and hard subsets, with the uniform chain baseline (3036 total, 1012 per level) for reference. Performance shown on in-domain (code: LCB ) and out-of-domain (math: Math500 , AIME2024 ) benchmarks across 3 seeds.
Figure 17: Comparing different curriculum strategies in Qwen3-8B Base. The figure compares RL training with different curriculum strategies over extended difficulty splits ( easy , easy–medium , medium , hard ). Curves show pass@1 and pass@10 on the in-domain benchmark ( LCB , including LCBv5 - easy , medium , and hard splits) and pass@1 and pass@5 on the out-of-domain benchmarks ( Math500 , AIME2024 ), averaged across three seeds. Reverse curricula starting from medium -level questions mitigate overfitting to easy tasks and maintain stronger performance on harder splits.
Figure 18: Reversed curriculum experiments in Qwen3-8B Base. The figure compares RL training with different reverse curriculum strategies over extended difficulty splits ( easy , easy–medium , medium , hard ). We evaluate two variants: starting with the medium split (“medium-start”) and starting with the hard split (“hard-start”). Curves show pass@1 and pass@10 on the in-domain benchmark ( LCB , including LCBv5 - easy , medium , and hard splits) and pass@1 and pass@5 on the out-of-domain benchmarks ( Math500 , AIME2024 ), averaged across three seeds. The medium-start reverse curriculum shows faster convergence and lower variance across seeds compared to the hard-start variant.
Figure 19: Scaling the number of RL environments in Qwen3-8B Base. The figure compares RL training on 20K synthetic problems from a single environment ( induction ) with training on an equal total of 20K synthetic problems distributed across four environments: induction , abduction , deduction , and fuzzing . Curves show pass@1 and pass@10 on in-domain ( LCB ) and pass@1 and pass@5 out-of-domain ( Math500 , AIME2024 ) benchmarks throughout RL training. Distributing the data budget across environments improves out-of-domain generalization and yields higher pass@10 scores in-domain, while reducing overfitting on the easy split.
Figure 20: Scaling the number of RL environments in Llama3.1-8B Instruct compared to real data. The figure compares RL training on 25K real code-contest problems with training on a total of 20K synthetic problems distributed across four environments: induction , abduction , deduction , and fuzzing . Curves show pass@1 and pass@10 on the in-domain benchmark ( LCB ) and pass@1 and pass@5 on out-of-domain benchmarks ( Math500 , AIME2024 ) throughout RL training. Despite using less data, training across multiple synthetic environments improves out-of-domain generalization and yields higher in-domain pass@10 scores compared to real-only training.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 21: RL with synthetic data augmentation seeded from real problems in Qwen3-8B Base. RL training on 25K real code-contest questions (baseline) versus the same data augmented with 20K synthetic induction problems generated using real questions as inspiration seeds ( Real 25K + SYNTH-Real-Aug 20K ), as well as training solely on the synthetic partition ( SYNTH-Real-Aug 20K ). Performance shown on in-domain ( LCB ) and out-of-domain ( Math500 , AIME2024 ) benchmarks.
Figure 22: RL with synthetic data augmentation seeded from random code snippets in Qwen3-8B Base. RL training on 25K real code-contest questions (baseline) versus the same data augmented with 20K synthetic induction problems generated using random snippets from starcoderdata ( Real 25K + SYNTH* Induction 20K ), as well as training solely on the synthetic partition ( SYNTH* Induction 20K ). Performance shown on in-domain ( LCB ) and out-of-domain ( Math500 , AIME2024 ) benchmarks.
Figure 23: RL with pure synthetic data in Qwen3-8B Base. Instead of augmenting the 25K real code-contest questions, RL is performed solely on synthetic induction problems generated with two seeding strategies: (1) real questions as inspiration seeds ( SYNTH-Real-Aug ), and (2) random code snippets from starcoderdata ( SYNTH*-Aug , 3 seeds).
Figure 24: Scaling with real data vs. synthetic augmentation in Qwen3-8B Base. Comparison of RL training on 25K and 81K real code-contest problems, and on 25K real data augmented with 20K synthetic problems generated from random starcoderdata snippets ( Real 25K + SYNTH*-Aug 20K ). Performance is shown on in-domain ( LCB ) and out-of-domain ( Math500 and AIME2024 ) benchmarks throughout training. Synthetic augmentation yields additional gains over the 81K real-data baseline across all LCB splits except medium .
Figure 25: Chains with hard curriculum in Qwen3-8B Base. Extended results for RL training on easy–medium–hard question chains using the hard curriculum schedule. Shown are in-domain results ( LCB , including easy, medium, and hard splits) and out-of-domain results ( Math500 , AIME2024 ) with pass@1, pass@5/pass@10 metrics plotted over RL training steps.
Figure 26: Chains with soft curriculum in Qwen3-8B Base. Extended results for RL training on easy–medium–hard question chains using the soft curriculum schedule. Performance is reported for LCB (easy, medium, hard splits) and out-of-domain benchmarks ( Math500 , AIME2024 ) with pass@1, pass@5/pass@10 metrics over training steps.
Figure 27: Chains without curriculum in Qwen3-8B Base. Extended results for RL training on easy–medium–hard question chains under uniform sampling across difficulty levels. Plots show in-domain ( LCB ) and out-of-domain ( Math500 , AIME2024 ) performance, including easy, medium, and hard splits, with pass@1, pass@5/pass@10 metrics over RL training steps.
Difficulty
Pass Rate Range
Number of Problems
easy
0.85–0.97
10000
easy–medium
0.61–0.85
7479
medium
0.26–0.61
5268
hard
0.10–0.26
2220
Appendix
Table 3: Difficulty bins and data split sizes used in curriculum experiments.
Figure 28: Scaling RL environments with real-data baseline in Qwen3-8B Base. Comparison of RL training on 20K problems from a single environment ( induction ), 20K problems distributed across four environments ( induction , abduction , deduction , fuzzing ), and the 25K real code-contest baseline. Shown are in-domain results ( LCB , pass@1 and pass@10) and out-of-domain results ( Math500 , AIME2024 , pass@1). Distributing the data budget across multiple environments improves out-of-domain generalization and results in competitive or higher (in terms of pass@10) in-domain performance compared to the real-data baseline.
Reinforcement learning for code generation trains models to write, repair, and optimize programs using execution-based and other task-specific feedback.
Reinforcement Learning from Verifiable Rewards (RLVR) has driven recent progress in code large language models by leveraging execution-based feedback from unit tests, but its scalability is fundamentally constrained by the availability and reliability of high-quality test cases. We propose CodeScaler, a reward model designed to scale both reinforcement learning training and test-time inference for code generation. CodeScaler is trained on carefully curated preference data derived from verified code problems and incorporates syntax-aware code extraction and validity-preserving reward shaping to ensure stable and robust optimization. Across four coding benchmarks, CodeScaler consistently outperforms execution-based RL by +1.55 points on Qwen3-8B-Base and +4.23 points on Qwen3-14B-Base. By further scaling to 44K problems with additional synthetic data, CodeScaler yields +14.64 points improvement over the base model without requiring any test cases. At inference time, CodeScaler serves as an effective test-time scaling method, achieving performance comparable to unit test approaches while providing a 10-fold reduction in latency. Moreover, CodeScaler surpasses existing reward models on RM-Bench not only in the code domain (+3.3 points), but also in general and reasoning domains (+2.7 points on average).
Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.
Anmol Kabra, Yilun Yin, Albert Gong +6
Cornell University · University of Cambridge · Stanford University
Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia. Improving Small Language Models~(SLMs) for these languages faces a trilemma: Supervised Fine-Tuning~(SFT) is bottlenecked by data scarcity, inference-time scaling is too expensive for deployment, and Reinforcement Learning from scratch yields near zero advantages. We propose a three-phase pipeline that resolves this trilemma by decoupling syntax acquisition from algorithmic reasoning. First, we left-shift inference-time compute to an offline data synthesis engine that uses iterative compiler and test feedback to generate verified training examples. Second, we fine-tune an SLM on this synthetic, verified data to embed strong syntactic priors. Third, we apply Reinforcement Learning with Verifiable Reward~(RLVR) grounded by language-agnostic Input/Output tests, where the SFT prior constrains exploration away from syntax errors. Applied to Qwen3-8B, our pipeline improves pass@1 by up to +7.6 points on MultiPL-E and +14.2 points on the Agnostics LiveCodeBench for Julia compared to SOTA results. Furthermore, the pipeline only used 31 data and 61 cost over the previous state-of-the-art. We further demonstrate that the pipeline generalizes to Ballerina achieving 49.7% MultiPL-E Pass@1, a language with near-zero pretraining representation. Ablations confirm that both the SFT phase and execution-grounded rewards are necessary for stable training.