Preventing Curriculum Collapse in Self-Evolving Reasoning Systems

Vaibhav Mishra

cs.LGMar 3, 2026

Preventing Curriculum Collapse in Self-Evolving Reasoning Systems

Authors: Vaibhav Mishra

Abstract

Self-evolving reasoning frameworks let LLMs improve their reasoning capabilities by iteratively generating and solving problems without external supervision, using verifiable rewards. Ideally, such systems are expected to explore a diverse problem space and propose new challenges of high learning value. While prior work has largely focused on solver-side optimisation and verification, recent evidence suggests that self-evolving systems can exhibit diversity collapse in posing new problems after just a few iterations, even when surface-level variation is preserved. We introduce Prism, a question-centric self-evolution method that directly tackles this collapse. Prism defines a persistent diversity signal over an embedding-induced semantic partition of mathematical problems and uses it to encourage balanced exploration of underrepresented regions across iterations. This coverage signal is combined with a Zone-of-Proximal-Development (ZPD) gate to preserve edge-of-solvability difficulty. Evaluated on seven widely used mathematical reasoning benchmarks against five self-evolving baselines, Prism achieves the highest accuracy on six out of seven tasks, achieving gains of +3.98 absolute points over R-Zero on AMC and +3.68 on Minerva Math. Prism also generates semantically diverse and challenging questions across iterations, resulting in the construction of the Prism-Math dataset comprising 100k mathematical questions. These results demonstrate that cross-iteration semantic coverage is a high-leverage and under-explored axis for building more capable self-evolving reasoners. We release the code, dataset, and models to facilitate further research.

Figures & tables

Panel of Figure 1 — Figure 1: Prism performance overview. (a) Pass@1 accuracy comparison with baseline across benchmarks. (b) Math-500 test accuracy across iterations.

Figure 2: Iteration-wise progression of the Prism self-evolution loop.

Model	GSM8K	MATH-500	AMC	Minerva	OlyBench	AIME 2024	AIME 2025
Qwen3-4B-Base	72.60	68.20	47.50	42.30	34.80	6.70	10.30
R-Zero [ 10 ]	92.12	79.60	57.27	52.94	44.59	13.40	9.60
R-Few (5%) [ 28 ]	92.60	78.00	52.40	53.20	42.80	14.50	9.90
R-Few (1%) [ 28 ]	92.30	77.80	52.70	52.10	42.40	13.60	9.10
Absolute Zero [ 29 ]	89.30	76.20	52.50	38.20	38.50	13.30	13.30
SPICE [ 30 ]	92.70	78.00	57.50	51.90	42.70	12.20	19.10

Table 1: Pass@1 accuracy across seven mathematical reasoning benchmarks. All methods use Qwen3-4B-Base as the base model. Bold indicates the best result per benchmark.

Questioner	Active	Std	Entropy	Norm. Ent.	Gini	Top-10 %
Base model	89	33.3	4.96	0.71	0.81	61.2%
R-Zero	65	91.4	3.71	0.53	0.90	79.5%
Prism	107	36.0	5.81	0.83	0.66	42.4%

Table 2: Semantic coverage statistics over

K{=}128

clusters. Higher entropy, lower Gini, and lower top-10 share indicate more uniform coverage.

Panel of Figure 3 — Figure 3: Curriculum diversity visualisation. Left: Per-cluster frequency distributions. R-Zero (top) is dominated by a single spike at cluster 44 (951 questions); Prism (bottom) covers 107 of 128 clusters with no dominant mode. Right: Lorenz curves for R-Zero (blue) and Prism (orange). A curve closer to the diagonal indicates more equitable cluster coverage.

Div. Reward	SolverInit	MATH-500
✗	✗	79.60
✓	✗	80.46
✗	✓	79.12
✓	✓	81.02

Table 3: Component ablation (Pass@1 accuracy). “Div.” = cluster-diversity reward; “SolverInit” = Solver-initialised Questioner.

$\lambda$	MATH-500
1.0	78.32
3.0	79.80
5.0	81.02
8.0	78.80

Table 4: Sensitivity to the diversity weight

\lambda

Component	R-Zero	Prism
Questioner GRPO training	38 min	38 min
Embedding + cluster assignment	N/A	2.4 min
Question generation & filtering	25 min	25 min
Solver GRPO training	82 min	82 min
Total per iteration	145 min	147.4 min
Overhead	N/A	+1.7%

Table 5: Wall-clock overhead per co-evolution iteration.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Setting	Value
General	Base model	Qwen3-4B-Base
	Co-evolution iterations ( $T$ )	4
	Hardware	8 $\times$ H100 node
	Precision	BFloat16
Questioner	GRPO steps per iteration	6
	Roll-outs ( $n$ )	4

Appendix

Table 6: Training hyperparameters for Prism.

Mathematical Problem Generation

See all 14 papers

Oct 3, 2026

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.

Jinyuan Li, Chengsong Huang, Langlin Huang +4

Washington University in St. Louis · University of Michigan, Ann Arbor

Language Model Self-Improvement Self-Training for Language Models Text Generation Mathematical Reasoning

Ask PDF

Apr 16, 2026

CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning

Large Language Models (LLMs) exhibit strong mathematical reasoning when trained on high-quality Chain-of-Thought (CoT) that articulates intermediate steps, yet costly CoT curation hinders further progress. While existing remedies such as distillation from stronger LLMs and self-synthesis based on test-time search alleviate this issue, they often suffer from diminishing returns or high computing overhead.In this work, we propose CoTEvol, a genetic evolutionary framework that casts CoT generation as a population-based search over reasoning trajectories.Candidate trajectories are iteratively evolved through reflective global crossover at the trajectory level and local mutation guided by uncertainty at the step level, enabling holistic recombination and fine-grained refinement. Lightweight, task-aware fitness functions are designed to guide the evolutionary process toward accurate and diverse reasoning. Empirically, CoTEvol improves correct-CoT synthesis success by over 30% and enhances structural diversity, with markedly improved efficiency. LLMs trained on these evolutionary CoT data achieve an average gain of 6.6% across eight math benchmarks, outperforming previous distillation and self-synthesis approaches. These results underscore the promise of evolutionary CoT synthesis as a scalable and effective method for mathematical reasoning tasks.

Zhuo Wang, Zhuo Zhang, Yafu Li +3

1Fudan University · 5Independent Researcher · 3The Chinese University of Hong Kong +3

Evolutionary Optimization LLM Training LLM Mathematical Reasoning

Ask PDF

Nov 1, 2025

OpenSIR: Open-Ended Self-Improving Reasoner

Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving by at least +4.4 points on reasoning models.

Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3

University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1

Self-Play RL RL for Language Model Reasoning Language Model Self-Improvement LLM Mathematical Reasoning Large Reasoning Models

Ask PDF

Abstract

Mathematical Problem Generation

Explore similar work

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning

OpenSIR: Open-Ended Self-Improving Reasoner