Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints. Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves by only +1.1 and its best filtered real data subset by +5.8. In finance, we synthesize over 14K verifiable problems from only 340 solutions and improve over base by 6.9 and 2.4 points on SecQue and FinanceBench, respectively.
Figures & tables
Figure 1: MagicBeanStalk synthesizes RLVR data from hard seeds to escape the zero-advantage trap. (a) For a fixed policy π and budget L , all-failure and all-success rollout groups yield zero advantage; ≈88% of a 78 K math corpus falls in this trap for gpt-oss-120b at 32K. (b) Three empirical principles: hard seeds specify what to synthesize; synthetic problems must be substantively new yet preserve the seed’s reasoning operations and their composition; and usefulness is relative to π and L . (c) MBS applies them in order: the seed’s verified teacher trace is decomposed top-down into subproblems that help a student solve their parent (1); harder new leaves are synthesized and recursively composed bottom-up into parallel trees p′ , p′′ (2); synthetic root problems with mixed target-policy rollouts and a complete correct response within L are kept (3).
Figure 2: Effective RLVR data synthesis must account for the problem, target policy, and response budget. Difficulty distributions (left): Empirical success-rate distributions from 8 gpt-oss-120b rollouts per problem in math and finance. We denote problems with 0/8 , 1 – 7/8 , and 8/8 correct responses as hard , Goldilocks , and stale/mastered , respectively; each bar height is the fraction of problems in that row. Roughly 88% of the corpus falls in the zero-advantage trap providing no learning signal, whereas MagicBeanStalk places 58.2% of its math problems and 56.7% of its finance problems in the Goldilocks band. Policy and budget dependence (right): The two panels on (b) use the problem subset that GPT-5-nano does not solve in every rollout excluding most of the stale mass on the left. Holding the problems and 32K response limit fixed, changing the target policy reclassifies 32.3% of problems. Holding gpt-oss-120b fixed, increasing the response limit from 16K to 64K reduces the fraction of hard problems from 33.1% to 10.6% . Black curve reports avg accuracy. Difficulty depends on the problem, policy, and rollout budget.
Figure 3: MBS improves beyond seed training and direct synthesis. Filtering shows that more data alone is not enough but discards the hard problems the target policy is not able to solve yet. (a) Starting from the same base policy, we compare one epoch of training on fixed seed problems and the same amount of qualified seed-synthesis and MBS problems. All derived from the same fixed seed population. Seed synthesis does not surpass seed-only training, whereas MBS provides a clear additional gain. In this control experiment, the seed population mixes goldilocks and hard problems, so that seed-only training is a meaningful baseline. (b) Holding the base policy, training recipe, and evaluation fixed, we train on a 6,922 -problem proxy that preserves the source mixture’s difficulty distribution, on its 862 -problem Goldilocks subset, and on the 696 -problem subset that additionally demonstrates in-budget solvability. Both filtered subsets outperform the full proxy while using fewer problems.
Figure 4: Direct prompting a teacher to generate variants is limited by both the problems it constructs and the number of useful candidates it yields. Left: We evaluate 300 same-seed pairs in math and 100 in finance for each approach. A judge scores each generated problem for reasoning-blueprint preservation and problem novelty. Blueprint preservation measures if the solutions to the synthetic problem requires the seed’s essential reasoning operations and dependencies. Novelty measures if the synthetic problem is new rather than a paraphrase, contextual rewrite, or numerical substitution. Higher is better. Right: Cumulative qualified problems as source seeds are added, (both methods evaluated on the same seeds). Qualification requires 1 – 7/8 correct and at least one correct response that terminates normally . In finance, we use same qualification. Finance curves are observed through 340 seeds and linearly projected thereafter. MBS produces more qualified descendants per seed while more often preserving the blueprint in a new problem.
Table 5Table 6
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Samples
Blueprint
Novelty
Goldilocks
All three
Direct synthesis
64
48/64
17/64
32/64
3/64
Direct synthesis (oversampling)
387
61/64
29/64
56/64
13/64
MBS
64
52/64
45/64
48/64
28/64
Appendix
Table 7: Direct synthesis under the MBS construction budget. Samples is the number of evaluated candidates. Blueprint and novelty require judge scores of at least three, and Goldilocks denotes 1 – 7 correct responses among N=8 gpt-oss-120b rollouts at 16K.
Table 8
Component
Without component
With component
Change
Teacher pool
Single teacher: 63.0%
Equal-call pool: 71.5%
+8.5 pp seed coverage
Effectivity test
22 teacher-valid decompositions
6 effective decompositions
16 non-helpful decompositions removed
Harder-leaf synthesis
Ordinary variants: 1/10
Harder variants: 3/10
+2 qualified seeds
Recursive depth
D=1 : 1,345 roots / 840 seeds
D≤3 : 1,533 roots / 956 seeds
+188 roots / +116 seeds
Appendix
Table 10: Component analyses. Teacher coverage uses 100 hard seeds under an equal four-call budget. The leaf comparison uses 10 fixed seeds. Qualified problems produce mixed outcomes over N=8 gpt-oss-120b rollouts and at least one complete correct response within 16K.
Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most informative training frontier. We show that these otherwise-inert problems can be unlocked via teacher-guided curriculum learning: partial reasoning traces from a stronger model create a graded difficulty landscape, and a backward-chaining curriculum progressively withdraws guidance until the model solves problems unaided. Training on only 128 unsolvable problems matches or exceeds GRPO trained on a full 2,000-problem corpus (~16x data efficiency) on the nine-benchmark average for both base models, while substantially expanding the reasoning boundary measured by pass@k at large k. Furthermore, we identify a distribution-shift cost that is particularly acute in the unsolvable-only regime and propose Monotone Frontier Curriculum (MFC), a method that monotonically drives training toward unguided solving, consistently outperforming existing curriculum methods.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs
Data synthesis for training large reasoning models offers a scalable alternative to limited, human-curated datasets, enabling the creation of high-quality data. However, existing approaches face several challenges: (i) indiscriminate generation that ignores the solver's ability and yields low-value problems, or reliance on complex data pipelines to balance problem difficulty; and (ii) a lack of reasoning in problem generation, leading to shallow problem variants. In this paper, we develop a problem generator that reasons explicitly to plan problem directions before synthesis and adapts difficulty to the solver's ability. We recover problem-design CoT from human-authored multi-part questions, using later subquestions as targets to infer the design rationale. These traces initialize the generator, which is then optimized with a seed-conditioned reward based on solver answer consistency. The reward guides difficulty adjustment relative to each seed while favoring problems near the solver's competence boundary. Across 10 mathematical and general reasoning benchmarks, our method outperforms seed-only training by 2.69 and 3.39 percentage points after one and three rounds, respectively. The framework also extends to vision-language models. Our code is publicly available at https://github.com/WalkerWorldPeace/Reasoning-Synthesis.
Yongxian Wei, Yilin Zhao, Li Shen +7
Tsinghua University · Tencent · Sun Yat-sen University +1