Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved reasoning in formal domains such as mathematics and code, but extending these gains beyond STEM remains challenging. Extending RLVR beyond STEM is fundamentally constrained by the lack of high-quality verifiable training data. In this work, we introduce SUPERNOVA, a framework for curating RLVR data from natural instruction datasets, which are a rich source of expert-annotated data but are underexplored for RLVR training. Through 100+ controlled RL experiments, we systematically study how to utilize these dataset for RLVR and how data curation decisions affect downstream reasoning performance . In particular, we investigate three data designs: (a) source task selection, (b) task mixing, and (c) synthetic interventions. Our analysis reveals that source task selection has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance and synthetic interventions do not improve reasoning. Guided by these insights, we construct SUPERNOVA, a high-quality RLVR dataset of 25K instances curated from natural instruction datasets. We show that training Qwen3-0.6B on SUPERNOVA outperforms the base Qwen3-0.6B, yielding a relative gain of 64.4pp on BigBench Extra Hard (BBEH), a challenging benchmark comprising 23 complex reasoning tasks. Importantly, we find that gains from SUPERNOVA generalize to unseen benchmarks, larger model scales, and newer model families. Overall, our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. Models, Data, Code at https://github.com/asuvarna31/supernova.
Figures & tables
Figure 1: SuperNova shows strong reasoning performance. Training with our curated SuperNova data leads to consistent pass@k improvements on a challenging benchmark, BBEH-test. (a) SuperNova shows consistent performance gains across various model sizes from the Qwen3 family. (b) SuperNova shows superior performance to existing reasoning datasets across varying values of k under compute-matched comparisons.
Figure 2: SuperNova Framework: We study the impact of various data design choices for RLVR data curation from natural instruction datasets. First, we study the impact of task selection on downstream reasoning performance. Then, we explore strategies to mix diverse tasks in source data. Finally, we examine whether synthetic data interventions can enhance data quality and improve downstream reasoning.
Figure 3: SuperNova Distribution. SuperNova comprises of 25K training samples that target 9 diverse reasoning types.
Figure 4: Impact of Task Selection. We train the baseline (Qwen3-0.6B) on each task individually under compute-matched settings. We report relative pass@8 gains on BBEH-mini for each task and highlight the tasks that improve and degrade the baseline.
Figure 5: (Left) Correlation between semantic similarity and task performance. (Middle) Correlation between lexical similarity and task performance. (Right) Correlation between win rate and downstream reasoning performance.
Top 1
Top 2
Top 4
Top 8
Top 16
Micro Mixing
pass@1
7.5
8.9
7.5
7.6
7.5
pass@8
18.3
22.8
18.7
18.0
20.2
Macro Mixing
pass@1
7.6
8.2
6.6
6.4
7.5
pass@8
21.5
21.7
17.4
17.0
18.3
Table 1: Impact of mixing. We mix questions from tasks following two strategies: micro mixing and macro mixing. We find that micro mixing with top 2 tasks achieves the best performance (bold) .
Model
MMLU-Pro
BBH
Zebralogic
MATH500
Average
Qwen3-0.6B
55.3
52.4
34.4
71.9
53.5
RL w/ General-Reasoner Ma et al. (2025)
54.4
64.3
45.4
66.3
57.6
RL w/ Nemotron-Crossthink Akter et al. (2026)
55.7
69.9
49.1
70.0
61.2
RL w/ SuperNova
56.2
81.5
49.4
71.4
64.6
Table 2: SuperNova beats reasoning datasets on reasoning benchmarks . We report pass@8 across four benchmarks that were unseen during data curation. We find that LLMs trained on SuperNova show improved pass@8 over the models trained on baseline datasets under compute-matched settings.
Figure 6: SuperNova generalizes to OOD Benchmarks. We report pass@8 across four reasoning benchmarks for Qwen3-4B and SuperNova -4B.
Model
Avg.
Brd.
Mov.
Dis.
Bool.
Geo.
T.Ar.
WoL
Word
Shuf.
Zebra
NYCC
M.Ar.
Hyp.
OLMo-3-7B-Think
15.1
6.1
78.4
48.5
0
0
16.2
2.4
24.4
0
0
20.8
0
0
OpenReasoning-7B
9.7
4.1
19.6
60.6
0
0
0
2.4
9.8
0
0
29.2
0
0
OpenThinker3-7B
8.8
12.2
9.8
54.5
0
0
8.1
0
17.1
0
0
12.5
0
0
Qwen3-8B
26.5
59.2
51
57.6
5.4
2.9
67.6
31
36.6
2.3
10
20.8
0
0
Qwen3-4B
25.8
65.3
64.7
51.5
5.4
0
48.6
26.2
36.6
0
10
25
2.2
0
SuperNova-4B
46.8
71.4
65.7
65.2
56.8
54.4
51.4
47.6
42.7
41.9
33
30.2
25.6
22.2
Table 3: SuperNova models exhibit strong performance across challenging reasoning tasks. We report the pass@8 results (%) on 13 tasks from BBEH-test. Best per column is bolded and second-best in underlined . Avg. is computed across the 13 tasks shown. ( Brd.=Boardgame QA, Mov.=Movie Recommendation, Dis.=Disambiguation QA, Bool.=Boolean Expressions, Geo.=Geometric Shapes, T.Ar.=Time Arithmetic, WoL=Web of Lies, Word=Word Sorting, Shuf.=Shuffled Objects, Zebra=Zebra Puzzles, M.Ar.=Multistep Arithmetic, Hyp.=Hyperbaton.)
Figure 7: We train Qwen3.5-2B and LLaMA3.2-3B-Instruct with SuperNova and show relative gains over their respective baseline models on 23 complex reasoning tasks.
Figure 8: We show the performance comparison between the baseline model and SuperNova -0.6B by scaling values of k up to 128 on 23 complex reasoning tasks.
Shared raw task definition: “In this task you will be given a claim and a perspective. You should determine whether that
perspective supports or undermines the claim. If the perspective could possibly convince someone with different view,
it is supporting, otherwise it is undermining.”
Raw
claim: Children should not be allowed to inherit vast wealth as this damages them and society. perspective: Inherited wealth demotivates the recipients so that they put less effort into training, education and social skills. Output: support
Reformat (MCQ, 2-way)
Claim: Children should not be allowed to inherit vast wealth as this damages them and society. Perspective: Inherited wealth demotivates the recipients so that they put less effort into training, education and social skills. Question: Does the perspective support or undermine the claim? Choose one option: (A) Supports (B) Undermines Answer: A
Raw
claim: Domestic intelligence agencies have a legitimate role to play in democracy. perspective: The government does not have the right to spy on its citizens. Output: undermine
Appendix
Table 4: Examples from SuperNova dataset. We show examples from our top performing task task738. In the reformatting step of our pipeline, we employs LLMs to (i) inline the shared task description in each sample, (ii) produce both open-ended and multiple-choice variants, and (iii) vary MCQ option counts from 2-way to 10-way. This reformatting across diverse tasks, normalizes the answer format and allows for rule-base verification.
Figure 9: Pass@k accuracy of task-specific models across various values of k.
Dimension
Description
Many-hop reasoning
Add information that increases the number of reasoning steps needed to reach the answer.
Going against strong prior
Add context that creates a misleading prior belief which conflicts with the correct answer, tempting the model to answer incorrectly based on surface-level associations.
Learning on the fly
Introduce a new rule, definition, or convention within the problem that must be understood and applied to solve it.
Long-context
Pad the problem with additional (but non-answer-changing) context to increase overall length.
Finding errors in reasoning traces
Include a flawed reasoning chain within the problem that the model must recognize as incorrect.
Inductive reasoning
Provide a set of examples that establish a pattern, requiring the model to induce and apply the pattern.
Appendix
Table 5: Following Kazemi et al. (2025) , we design these interventions to improve the data quality. We provide the interventions and their definitions here.
Intervention
pass@8
Micro-Top2
22.8
Going Against Prior
22.6
Long-Context
21.3
Inductive Reasoning
20.4
Finding Errors
20.0
Many-hop Reasoning
20.0
Appendix
Table 6: Impact of interventions. We compare the performance of models trained on datasets transformed using synthetic interventions. We find that the base dataset is superior to all interventions.
Figure 10: (a)We categorize the source tasks based on target reasoning skill and task type. We report Mean Pass@8 across each task category and highlight the categories which degrade baseline (Qwen3-0.6b) model on BBEH-mini.
Figure 11: Verbatim output from SuperNova -4B on a Boardgame-QA instance with 30 rules and preference orderings, together with the step-by-step inference trace it implies. The model produces the correct answer (proved) by chaining 13 rule applications and correctly resolves four rule-preference conflicts.
BBEH Task
Rank
Task ID
Training Task
movie recommendation
1
task827
copa_commonsense_reasoning
2
task069
abductivenli_classification
3
task212
logic2text_classification
4
task1297
qasc_question_answering
5
task1209
atomic_classification_objectuse
word sorting
1
task828
copa_commonsense_cause_effect
Appendix
Table 7: Top-5 training tasks per BBEH task.
Task Name
Summary
task738 perspectrum classification
Decide whether the given perspective supports or undermines the given claim.
task003 mctaco question generation event duration
Writing questions that involve commonsense understanding of “event duration”.
task717 mmmlu answer generation logical fallacies
Answering multiple choice questions on logical fallacies.
task249 enhanced wsc pronoun disambiguation
Given a sentence and a pronoun, decide which one of the choices the pronoun is referring to.
task1385 anli r1 entailment
Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise.
task1296 wiki hop question answering
Given a subject, a relation, and a context, find the object with that relation to the subject.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs
Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations, achieving performance comparable to learning from clean data. In this work, we show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data. After rectifying the dataset with a rigorous re-verification pipeline, we demonstrate that noise is destructive to RLVR. We show that existing RLVR algorithm improvements fail to mitigate the impact of noise, achieving similar performance to that of the basic GRPO. Furthermore, we find that the model trained on truly incorrect annotations performs 8-10% worse than the model trained on clean data across mathematical reasoning benchmarks. Finally, we show that these findings hold for real-world noise in Text2SQL tasks, where training on real-world, human annotation errors cause 5-12% lower accuracy than clean data. Our results show that current RLVR methods cannot yet compensate for poor data quality. High-quality data remains essential.
Reinforcement learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models, particularly in mathematics and coding. While recent efforts have extended this paradigm to broader general scientific (STEM) domains, the complex interplay between supervised fine-tuning (SFT) and RL in these contexts remains underexplored. In this paper, we conduct controlled experiments revealing a critical challenge: for general STEM domains, RL applied directly to base models is highly sample-inefficient and is consistently surpassed by supervised fine-tuning (SFT) on moderate-quality responses. Yet sequential SFT followed by RL can further improve performance, suggesting that the two stages play complementary roles, and that how training data is allocated between them matters. Therefore, we propose DeReason, a difficulty-based data decoupling strategy for general reasoning. DeReason partitions training data by reasoning intensity estimated via LLM-based scoring into reasoning-intensive and non-reasoning-intensive subsets. It allocates broad-coverage, non-reasoning-intensive problems to SFT to establish foundational domain knowledge, and reserves a focused subset of difficult problems for RL to cultivate complex reasoning. We demonstrate that this principled decoupling yields better performance than randomly splitting the data for sequential SFT and RL. Extensive experiments on general STEM and mathematical benchmarks demonstrate that our decoupled curriculum training significantly outperforms SFT-only, RL-only, and random-split baselines. Our work provides a systematic study of the interplay between SFT and RL for general reasoning, offering a highly effective and generalized post-training recipe.
Hanxu Hu, Yuxuan Wang, Maggie Huan +4
University of Zurich · University of Pennsylvania · ETH Zurich +2