cs.AIApr 9, 2026
SaveSUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
Organizations: University of California, Los Angeles
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved reasoning in formal domains such as mathematics and code, but extending these gains beyond STEM remains challenging. Extending RLVR beyond STEM is fundamentally constrained by the lack of high-quality verifiable training data. In this work, we introduce SUPERNOVA, a framework for curating RLVR data from natural instruction datasets, which are a rich source of expert-annotated data but are underexplored for RLVR training. Through 100+ controlled RL experiments, we systematically study how to utilize these dataset for RLVR and how data curation decisions affect downstream reasoning performance . In particular, we investigate three data designs: (a) source task selection, (b) task mixing, and (c) synthetic interventions. Our analysis reveals that source task selection has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance and synthetic interventions do not improve reasoning. Guided by these insights, we construct SUPERNOVA, a high-quality RLVR dataset of 25K instances curated from natural instruction datasets. We show that training Qwen3-0.6B on SUPERNOVA outperforms the base Qwen3-0.6B, yielding a relative gain of 64.4pp on BigBench Extra Hard (BBEH), a challenging benchmark comprising 23 complex reasoning tasks. Importantly, we find that gains from SUPERNOVA generalize to unseen benchmarks, larger model scales, and newer model families. Overall, our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. Models, Data, Code at https://github.com/asuvarna31/supernova.
Figures & tables
Figure 1: SuperNova shows strong reasoning performance. Training with our curated SuperNova data leads to consistent pass@k improvements on a challenging benchmark, BBEH-test. (a) SuperNova shows consistent performance gains across various model sizes from the Qwen3 family. (b) SuperNova shows superior performance to existing reasoning datasets across varying values of k under compute-matched comparisons.
Figure 2: SuperNova Framework: We study the impact of various data design choices for RLVR data curation from natural instruction datasets. First, we study the impact of task selection on downstream reasoning performance. Then, we explore strategies to mix diverse tasks in source data. Finally, we examine whether synthetic data interventions can enhance data quality and improve downstream reasoning.
Figure 3: SuperNova Distribution. SuperNova comprises of 25K training samples that target 9 diverse reasoning types.
Figure 4: Impact of Task Selection. We train the baseline (Qwen3-0.6B) on each task individually under compute-matched settings. We report relative pass@8 gains on BBEH-mini for each task and highlight the tasks that improve and degrade the baseline.
Figure 5: (Left) Correlation between semantic similarity and task performance. (Middle) Correlation between lexical similarity and task performance. (Right) Correlation between win rate and downstream reasoning performance.
| Top 1 | Top 2 | Top 4 | Top 8 | Top 16 | |
|---|---|---|---|---|---|
| Micro Mixing | |||||
| pass@1 | 7.5 | 8.9 | 7.5 | 7.6 | 7.5 |
| pass@8 | 18.3 | 22.8 | 18.7 | 18.0 | 20.2 |
| Macro Mixing | |||||
| pass@1 | 7.6 | 8.2 | 6.6 | 6.4 | 7.5 |
| pass@8 | 21.5 | 21.7 | 17.4 | 17.0 | 18.3 |
Table 1: Impact of mixing. We mix questions from tasks following two strategies: micro mixing and macro mixing. We find that micro mixing with top 2 tasks achieves the best performance (bold) .
| Model | MMLU-Pro | BBH | Zebralogic | MATH500 | Average |
|---|---|---|---|---|---|
| Qwen3-0.6B | 55.3 | 52.4 | 34.4 | 71.9 | 53.5 |
| RL w/ General-Reasoner Ma et al. (2025) | 54.4 | 64.3 | 45.4 | 66.3 | 57.6 |
| RL w/ Nemotron-Crossthink Akter et al. (2026) | 55.7 | 69.9 | 49.1 | 70.0 | 61.2 |
| RL w/ SuperNova | 56.2 | 81.5 | 49.4 | 71.4 | 64.6 |
Table 2: SuperNova beats reasoning datasets on reasoning benchmarks . We report pass@8 across four benchmarks that were unseen during data curation. We find that LLMs trained on SuperNova show improved pass@8 over the models trained on baseline datasets under compute-matched settings.
Figure 6: SuperNova generalizes to OOD Benchmarks. We report pass@8 across four reasoning benchmarks for Qwen3-4B and SuperNova -4B.
| Model | Avg. | Brd. | Mov. | Dis. | Bool. | Geo. | T.Ar. | WoL | Word | Shuf. | Zebra | NYCC | M.Ar. | Hyp. |
| OLMo-3-7B-Think | 15.1 | 6.1 | 78.4 | 48.5 | 0 | 0 | 16.2 | 2.4 | 24.4 | 0 | 0 | 20.8 | 0 | 0 |
| OpenReasoning-7B | 9.7 | 4.1 | 19.6 | 60.6 | 0 | 0 | 0 | 2.4 | 9.8 | 0 | 0 | 29.2 | 0 | 0 |
| OpenThinker3-7B | 8.8 | 12.2 | 9.8 | 54.5 | 0 | 0 | 8.1 | 0 | 17.1 | 0 | 0 | 12.5 | 0 | 0 |
| Qwen3-8B | 26.5 | 59.2 | 51 | 57.6 | 5.4 | 2.9 | 67.6 | 31 | 36.6 | 2.3 | 10 | 20.8 | 0 | 0 |
| Qwen3-4B | 25.8 | 65.3 | 64.7 | 51.5 | 5.4 | 0 | 48.6 | 26.2 | 36.6 | 0 | 10 | 25 | 2.2 | 0 |
| SuperNova-4B | 46.8 | 71.4 | 65.7 | 65.2 | 56.8 | 54.4 | 51.4 | 47.6 | 42.7 | 41.9 | 33 | 30.2 | 25.6 | 22.2 |
Table 3: SuperNova models exhibit strong performance across challenging reasoning tasks. We report the pass@8 results (%) on 13 tasks from BBEH-test. Best per column is bolded and second-best in underlined . Avg. is computed across the 13 tasks shown. ( Brd.=Boardgame QA, Mov.=Movie Recommendation, Dis.=Disambiguation QA, Bool.=Boolean Expressions, Geo.=Geometric Shapes, T.Ar.=Time Arithmetic, WoL=Web of Lies, Word=Word Sorting, Shuf.=Shuffled Objects, Zebra=Zebra Puzzles, M.Ar.=Multistep Arithmetic, Hyp.=Hyperbaton.)
Figure 7: We train Qwen3.5-2B and LLaMA3.2-3B-Instruct with SuperNova and show relative gains over their respective baseline models on 23 complex reasoning tasks.
Figure 8: We show the performance comparison between the baseline model and SuperNova -0.6B by scaling values of k up to 128 on 23 complex reasoning tasks.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Task A: task738_perspectrum_classification — claim/perspective stance detection (classification, support/undermine). | |
|---|---|
| Shared raw task definition: “In this task you will be given a claim and a perspective. You should determine whether that | |
| perspective supports or undermines the claim. If the perspective could possibly convince someone with different view, | |
| it is supporting, otherwise it is undermining.” | |
| Raw | claim: Children should not be allowed to inherit vast wealth as this damages them and society. perspective: Inherited wealth demotivates the recipients so that they put less effort into training, education and social skills. Output: support |
| Reformat (MCQ, 2-way) | Claim: Children should not be allowed to inherit vast wealth as this damages them and society. Perspective: Inherited wealth demotivates the recipients so that they put less effort into training, education and social skills. Question: Does the perspective support or undermine the claim? Choose one option: (A) Supports (B) Undermines Answer: A |
| Raw | claim: Domestic intelligence agencies have a legitimate role to play in democracy. perspective: The government does not have the right to spy on its citizens. Output: undermine |
Table 4: Examples from SuperNova dataset. We show examples from our top performing task task738. In the reformatting step of our pipeline, we employs LLMs to (i) inline the shared task description in each sample, (ii) produce both open-ended and multiple-choice variants, and (iii) vary MCQ option counts from 2-way to 10-way. This reformatting across diverse tasks, normalizes the answer format and allows for rule-base verification.
Figure 9: Pass@k accuracy of task-specific models across various values of k.
| Dimension | Description |
|---|---|
| Many-hop reasoning | Add information that increases the number of reasoning steps needed to reach the answer. |
| Going against strong prior | Add context that creates a misleading prior belief which conflicts with the correct answer, tempting the model to answer incorrectly based on surface-level associations. |
| Learning on the fly | Introduce a new rule, definition, or convention within the problem that must be understood and applied to solve it. |
| Long-context | Pad the problem with additional (but non-answer-changing) context to increase overall length. |
| Finding errors in reasoning traces | Include a flawed reasoning chain within the problem that the model must recognize as incorrect. |
| Inductive reasoning | Provide a set of examples that establish a pattern, requiring the model to induce and apply the pattern. |
Table 5: Following Kazemi et al. (2025) , we design these interventions to improve the data quality. We provide the interventions and their definitions here.
| Intervention | pass@8 |
|---|---|
| Micro-Top2 | 22.8 |
| Going Against Prior | 22.6 |
| Long-Context | 21.3 |
| Inductive Reasoning | 20.4 |
| Finding Errors | 20.0 |
| Many-hop Reasoning | 20.0 |
Table 6: Impact of interventions. We compare the performance of models trained on datasets transformed using synthetic interventions. We find that the base dataset is superior to all interventions.
Figure 10: (a)We categorize the source tasks based on target reasoning skill and task type. We report Mean Pass@8 across each task category and highlight the categories which degrade baseline (Qwen3-0.6b) model on BBEH-mini.
Figure 11: Verbatim output from SuperNova -4B on a Boardgame-QA instance with 30 rules and preference orderings, together with the step-by-step inference trace it implies. The model produces the correct answer (proved) by chaining 13 rule applications and correctly resolves four rule-preference conflicts.
| BBEH Task | Rank | Task ID | Training Task |
|---|---|---|---|
| movie recommendation | 1 | task827 | copa_commonsense_reasoning |
| 2 | task069 | abductivenli_classification | |
| 3 | task212 | logic2text_classification | |
| 4 | task1297 | qasc_question_answering | |
| 5 | task1209 | atomic_classification_objectuse | |
| word sorting | 1 | task828 | copa_commonsense_cause_effect |
Table 7: Top-5 training tasks per BBEH task.
| Task Name | Summary |
|---|---|
| task738 perspectrum classification | Decide whether the given perspective supports or undermines the given claim. |
| task003 mctaco question generation event duration | Writing questions that involve commonsense understanding of “event duration”. |
| task717 mmmlu answer generation logical fallacies | Answering multiple choice questions on logical fallacies. |
| task249 enhanced wsc pronoun disambiguation | Given a sentence and a pronoun, decide which one of the choices the pronoun is referring to. |
| task1385 anli r1 entailment | Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. |
| task1296 wiki hop question answering | Given a subject, a relation, and a context, find the object with that relation to the subject. |
Table 8: Task descriptions.
More in this topic
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,040 papers