MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

Ximing Lu; Aviral Kumar; Jan Kautz; Yejin Choi; David Acuna

cs.AIOct 8, 2026

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

Authors: Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi, David Acuna

Organizations: University of Washington · Carnegie Mellon University · NVIDIA · Stanford University

Abstract

Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints. Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves by only +1.1 and its best filtered real data subset by +5.8. In finance, we synthesize over 14K verifiable problems from only 340 solutions and improve over base by 6.9 and 2.4 points on SecQue and FinanceBench, respectively.

Figures & tables

Figure 1: MagicBeanStalk synthesizes RLVR data from hard seeds to escape the zero-advantage trap. (a) For a fixed policy

\pi

and budget

L

, all-failure and all-success rollout groups yield zero advantage;

\approx 88\%

of a

78

K math corpus falls in this trap for gpt-oss-120b at 32K. (b) Three empirical principles: hard seeds specify what to synthesize; synthetic problems must be substantively new yet preserve the seed’s reasoning operations and their composition; and usefulness is relative to

\pi

and

L

. (c) MBS applies them in order: the seed’s verified teacher trace is decomposed top-down into subproblems that help a student solve their parent (1); harder new leaves are synthesized and recursively composed bottom-up into parallel trees

p^{\prime}

p^{\prime\prime}

(2); synthetic root problems with mixed target-policy rollouts and a complete correct response within

L

are kept (3).

Panel of Figure 2 — Figure 2: Effective RLVR data synthesis must account for the problem, target policy, and response budget. Difficulty distributions (left): Empirical success-rate distributions from 8 gpt-oss-120b rollouts per problem in math and finance. We denote problems with $0/8$ , $1$ – $7/8$ , and $8/8$ correct responses as hard , Goldilocks , and stale/mastered , respectively; each bar height is the fraction of problems in that row. Roughly $\mathbf{88\%}$ of the corpus falls in the zero-advantage trap providing no learning signal, whereas MagicBeanStalk places $58.2\%$ of its math problems and $56.7\%$ of its finance problems in the Goldilocks band. Policy and budget dependence (right): The two panels on (b) use the problem subset that GPT-5-nano does not solve in every rollout excluding most of the stale mass on the left. Holding the problems and 32K response limit fixed, changing the target policy reclassifies $32.3\%$ of problems. Holding gpt-oss-120b fixed, increasing the response limit from 16K to 64K reduces the fraction of hard problems from $33.1\%$ to $10.6\%$ . Black curve reports avg accuracy. Difficulty depends on the problem, policy, and rollout budget.

Panel of Figure 3 — Figure 3: MBS improves beyond seed training and direct synthesis. Filtering shows that more data alone is not enough but discards the hard problems the target policy is not able to solve yet. (a) Starting from the same base policy, we compare one epoch of training on fixed seed problems and the same amount of qualified seed-synthesis and MBS problems. All derived from the same fixed seed population. Seed synthesis does not surpass seed-only training, whereas MBS provides a clear additional gain. In this control experiment, the seed population mixes goldilocks and hard problems, so that seed-only training is a meaningful baseline. (b) Holding the base policy, training recipe, and evaluation fixed, we train on a $6{,}922$ -problem proxy that preserves the source mixture’s difficulty distribution, on its $862$ -problem Goldilocks subset, and on the $696$ -problem subset that additionally demonstrates in-budget solvability. Both filtered subsets outperform the full proxy while using fewer problems.

Panel of Figure 4 — Figure 4: Direct prompting a teacher to generate variants is limited by both the problems it constructs and the number of useful candidates it yields. Left: We evaluate $300$ same-seed pairs in math and $100$ in finance for each approach. A judge scores each generated problem for reasoning-blueprint preservation and problem novelty. Blueprint preservation measures if the solutions to the synthetic problem requires the seed’s essential reasoning operations and dependencies. Novelty measures if the synthetic problem is new rather than a paraphrase, contextual rewrite, or numerical substitution. Higher is better. Right: Cumulative qualified problems as source seeds are added, (both methods evaluated on the same seeds). Qualification requires $1$ – $7/8$ correct and at least one correct response that terminates normally . In finance, we use same qualification. Finance curves are observed through 340 seeds and linearly projected thereafter. MBS produces more qualified descendants per seed while more often preserving the blueprint in a new problem.

Table 5

Table 6

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Samples	Blueprint	Novelty	Goldilocks	All three
Direct synthesis	64	$48/64$	$17/64$	$32/64$	$3/64$
Direct synthesis (oversampling)	387	$61/64$	$29/64$	$56/64$	$13/64$
MBS	64	$52/64$	$45/64$	$48/64$	$\mathbf{28/64}$

Appendix

Table 7: Direct synthesis under the MBS construction budget. Samples is the number of evaluated candidates. Blueprint and novelty require judge scores of at least three, and Goldilocks denotes

1

–

7

correct responses among

N=8

gpt-oss-120b rollouts at 16K.

Table 8

Component	Without component	With component	Change
Teacher pool	Single teacher: $63.0\%$	Equal-call pool: $71.5\%$	$+8.5$ pp seed coverage
Effectivity test	22 teacher-valid decompositions	6 effective decompositions	16 non-helpful decompositions removed
Harder-leaf synthesis	Ordinary variants: $1/10$	Harder variants: $3/10$	$+2$ qualified seeds
Recursive depth	$D=1$ : 1,345 roots / 840 seeds	$D\leq 3$ : 1,533 roots / 956 seeds	$+188$ roots / $+116$ seeds

Appendix

Table 10: Component analyses. Teacher coverage uses 100 hard seeds under an equal four-call budget. The leaf comparison uses 10 fixed seeds. Qualified problems produce mixed outcomes over

N=8

gpt-oss-120b rollouts and at least one complete correct response within 16K.

Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards trains models using outcomes checked by code, proofs, simulators, or structured criteria.

See all 685 papers

Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR
Sep 12, 2026Yukang Zhu, Zhen HanRL for Language Model Reasoning Curriculum RL
Independent Researcher · Amazon
ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models
Feb 23, 2026Andre He, Nathaniel Weir, Kaj Bostrom +4Synthetic Data Generation for Language Models RL for Language Model Reasoning
Carnegie Mellon University · Amazon Web Services
Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis
Nov 13, 2025Yongxian Wei, Yilin Zhao, Li Shen +7Synthetic Data Generation Synthetic Data Generation for Language Models
Tsinghua University · Tencent · Sun Yat-sen University +1

Sep 12, 2026J/K move · Enter open · S save

Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR

Yukang Zhu, Zhen Han

Independent Researcher · Amazon

RL for Language Model Reasoning Curriculum RL LLM Reasoning Reinforcement Learning with Verifiable Rewards LLM-Guided RL

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most informative training frontier. We show that these otherwise-inert problems can be unlocked via teacher-guided curriculum learning: partial reasoning traces from a stronger model create a graded difficulty landscape, and a backward-chaining curriculum progressively withdraws guidance until the model solves problems unaided. Training on only 128 unsolvable problems matches or exceeds GRPO trained on a full 2,000-problem corpus (~16x data efficiency) on the nine-benchmark average for both base models, while substantially expanding the reasoning boundary measured by pass@k at large k. Furthermore, we identify a distribution-shift cost that is particularly acute in the unsolvable-only regime and propose Monotone Frontier Curriculum (MFC), a method that monotonically drives training toward unguided solving, consistently outperforming existing curriculum methods.

Abstract

Reinforcement Learning with Verifiable Rewards

Explore similar work

Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR

ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis