When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora

Esther Xin

cs.CLSep 20, 2026

When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora

Authors: Esther Xin

Organizations: Independent Researcher

Abstract

Several recent pipelines build RLVR training data by masking a span of real corpus text and asking a language model to invent plausible wrong answers around it. The correct option is therefore genuine human prose; every distractor is synthetic. Correctness and provenance become entangled, and a policy could in principle learn the second instead of the first. We audit that possibility in GooseReason-0.7M. First we ask whether the asymmetry is visible at all: a classifier reading only five surface statistics (never the meaning) reaches AUROC 0.562 over 315,499 options, barely above chance. The aggregate hides something, though. Code sits at 0.416, below chance, and manual inspection explains why: code distractors turn out to be single-operator mutations of the gold answer rather than freely written alternatives, so the two classes are nearly identical by construction. Detecting a signal is not the same as showing a model uses it, so we then run an intervention. We build a paraphrase-matched control corpus, hold training-set size identical across arms, and train two policies under one fixed budget. The exploitation gap does not favour the unmodified-data arm: 0.021 against 0.027 for the control. Under our budget, in other words, a detectable artifact went unexploited. We think that dissociation, along with the domain-specific construction finding, is worth knowing for anyone curating corpora of this kind, and we release the audit as a mostly CPU-only protocol.

Figures & tables

Split	GBM AUROC	LogReg AUROC	$n$
overall	0.562	0.551	315,499
math	0.584	0.598	101,926
stem	0.583	0.569	102,414
code	0.416	0.503	111,159

Table 1: Held-out AUROC for a classifier separating gold spans from distractors using five surface-statistical features alone (rare-token rate, word- and sentence-length burstiness, character entropy, and POS-tag KL-divergence against a reference distribution). No feature inspects semantic content. code is the only split below the chance line of 0.500.

Group	Orig.	Neutr.	Advers.	Gap
treatment	0.283	0.212	0.191	0.021
control	0.299	0.233	0.207	0.027

Table 2: Overall four-option MCQ accuracy (chance

=0.25

) for the treatment-trained (original data) and control-trained (paraphrase-matched data) policies on three held-out test variants, and the resulting artifact exploitation gap (Neutr.

-

Advers.). The control gap is not smaller than the treatment gap, giving no evidence that training on unmodified data induces additional shortcut reliance relative to training on artifact-neutralised data.

Group	Domain	Orig.	Neutr.	Advers.	Gap
treatment	code	0.300	0.220	0.187	0.033
treatment	math	0.153	0.103	0.137	$-$ 0.033
treatment	stem	0.397	0.313	0.250	0.063
control	code	0.330	0.263	0.207	0.057
control	math	0.167	0.123	0.150	$-$ 0.027
control	stem	0.400	0.313	0.263	0.050

Table 3: Exploitation gap decomposed by domain. The negative gap in math is not an artefact of a single arm: it recurs, at similar magnitude, in both the treatment and control policies, indicating a domain-level effect independent of training-data provenance rather than sampling noise in one arm.

Figure 1: Stage 1 classifier AUROC by domain and classifier type. The dashed line marks chance (0.5). code is the only domain falling below it.

Figure 2: Exploitation gap by domain, treatment vs. control policy. No domain shows a treatment gap substantially exceeding its control counterpart.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Base policy model	Qwen3-1.7B
Paraphrase model (control construction)	Qwen2.5-3B-Instruct
Fine-tuning method	4-bit NF4 QLoRA, $r{=}16$ , $\alpha{=}32$
LoRA target modules	q_proj, k_proj, v_proj, o_proj
RL algorithm	GRPO ( Shao et al., 2024 )
Training steps per arm	60

Appendix

Table 4: Training, evaluation, and compute configuration for Phase 2. Train-set size is matched exactly across the treatment and control arms to avoid confounding a data-authenticity effect with a data-quantity effect. All settings are released with the code so the run can be reproduced or extended to a larger compute budget.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 239 papers

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
Oct 8, 2026Ximing Lu, Aviral Kumar, Jan Kautz +2Reinforcement Learning with Verifiable Rewards Synthetic Data Generation for Language Models
University of Washington · Carnegie Mellon University · NVIDIA +1
When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR
Jul 13, 2026Chuyifei ZhangAlgorithmic Auditing Reinforcement Learning with Verifiable Rewards
Beijing Jiaotong University
RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data
May 26, 2026Hsiu-Yuan Huang, Weijie Liu, Chenming Tang +5Data Provenance Reinforcement Learning with Verifiable Rewards
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent

Oct 8, 2026J/K move · Enter open · S save

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi+1

University of Washington · Carnegie Mellon University · NVIDIA+1

Reinforcement Learning with Verifiable Rewards Synthetic Data Generation for Language Models Synthetic Data Generation Synthetic Data Evaluation Numerical Reasoning in Language Models

Open paper AskPDF arXiv

Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints. Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves by only +1.1 and its best filtered real data subset by +5.8. In finance, we synthesize over 14K verifiable problems from only 340 solutions and improve over base by 6.9 and 2.4 points on SecQue and FinanceBench, respectively.

Abstract

Synthetic Data Evaluation

Explore similar work

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data