Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Jinyuan Li; Chengsong Huang; Langlin Huang; Donghong Cai; Shiping Gao; Yuyi Yang; Jiaxin Huang

cs.LGOct 3, 2026

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Authors: Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang

Organizations: Washington University in St. Louis · University of Michigan, Ann Arbor

Abstract

Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.

Figures & tables

Figure 1: Average performance on seven math benchmarks. R-Quest exceeds the peak of R-Zero in every round and peaks at round ten, leading by 17.32 points.

Figure 2: Two question-quality problems in self-evolution and how R-Quest addresses them: (a) R-Quest can recognize and reject invalid questions; (b) R-Quest can detect repeated mathematical tasks despite differences in expression.

Panel of Figure 3 — Figure 3: Question validity rate and solver performance on Qwen3-4B-Base. (a) Valid-question rates before and after R-Zero’s answer-consistency filter. (b) Average math reasoning performance of the solver trained on matched original and repaired datasets.

Figure 4: Repeated question distribution and BLEU fragmentation in the fifth round of R-Zero. BLEU separates the illustrated mathematically equivalent questions into multiple different clusters.

Qwen3-4B-Base
Method	Avg.	AMC	Minerva	MATH	GSM8K	Olympiad	AIME25	AIME24

Base model	46.17	47.34	51.84	74.20	88.63	39.11	11.35	10.73
Base model (validity init.)	47.07	49.14	53.68	74.40	89.76	41.04	10.73	10.73

R-Zero	49.10	54.84	54.78	76.20	92.42	42.37	12.29	10.83

Table 1: Mathematical reasoning results (%) across seven benchmarks. Avg. denotes the mean across benchmarks. For each backbone, the best score in each column is shown in bold .

Figure 5: Average performance on seven benchmarks over ten rounds on Qwen3-4B-Base. Yellow and orange denote R-Quest without novelty or validity feedback.

Qwen3-4B-Base
Method	Code generation			General-domain reasoning
	Avg.	HumanEval+	MBPP+	Avg.	SuperGPQA	MMLU-Pro	BBEH

Base model	58.79	54.88	62.70	28.79	26.21	51.41	8.74
Base model (validity init.)	59.63	55.49	63.76	29.40	26.28	51.58	10.35

Table 2: Code generation (pass@1) and general-domain reasoning results (%). Avg. denotes the mean within each domain. Best scores within each backbone are in bold .


Variant	Math	Code	General
R-Quest	51.64	62.69	32.75
w/o novelty	51.02	61.98	31.80
w/o validity	50.27	61.28	31.53
w/o both (R-Zero)	49.10	59.27	31.21

Table 3: Component ablations on Qwen3-4B-Base. Scores are averaged within each domain.

Figure 6: Validity assessment, question and answer quality during self-evolution on Qwen3-4B-Base. Panels (a) and (b) show R-Quest’s improved validity assessment on held-out questions. Panels (c) and (d) show that R-Quest maintains high question validity and better preserves majority-answer accuracy on valid questions. Bands in (c) show pointwise 95% Wilson confidence intervals.

Figure 7: Question type clusters (bars, left axis) and mathematical performance (lines, right axis) on Qwen3-4B-Base. Numbers above the bars denote top-five group proportion.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Error type	Count	Percentage
Contradiction	380	28.9%
No solution	362	27.5%
Ambiguity	315	23.9%
Missing information	181	13.7%
Poorly formed questions	72	5.5%
Other	7	0.5%

Appendix

Table 5: Error types among the 1,317 invalid questions in the training and validation splits.

Setting	Value
Training steps	15
Questions per rollout batch	512
Responses per question	8
Actor global batch size	128
Learning rate	$10^{-6}$
KL coefficient	$10^{-2}$

Appendix

Table 6: Training settings for validity-aware solver initialization.

Figure 8: Approximate pass probability

(1-p)^{K}

for the default comparison budget

K=8

and the additional setting

K=16

Table 14


Variant	Mathematical reasoning
	Avg.	AMC	Minerva	MATH	GSM8K	Olympiad	AIME25	AIME24
R-Quest	51.64	57.89	58.82	77.60	92.57	46.67	13.44	14.48
w/o novelty	51.02	55.94	56.62	78.00	91.81	45.63	15.42	13.75
w/o validity	50.27	56.48	58.82	77.00	91.74	41.19	13.54	13.12
w/o both (R-Zero)	49.10	54.84	54.78	76.20	92.42	42.37	12.29	10.83

Appendix

Table 9: Per-benchmark results for the gate ablations on Qwen3-4B-Base.

Explore similar work

INFUSER: Influence-Guided Self-Evolution Improves Reasoning
Jun 8, 2026Siyu Chen, Miao Lu, Beining Wu +7RL for Language Model Reasoning Group Relative Policy Optimization
Yale University · Stanford University · University of Chicago +2
Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Models
Sep 1, 2026Xincheng Wei, Yifan Ding, Fucheng Xiong +6Self-Training for Language Models Language Model Self-Improvement
The Chinese University of Hong Kong, Shenzhen · Meituan, LongCat Team · Peking University +2
Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula
Aug 30, 2026Vinoth Selvendran, Zhanming ZhangReward Modeling RL for Language Model Reasoning
Independent Researcher · Palo Alto, CA, USA · New York, NY, USA

Jun 8, 2026, cs.LGJ/K move · Enter open · S save

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen+6

Yale University · Stanford University · University of Chicago+2

RL for Language Model Reasoning Group Relative Policy Optimization Curriculum Learning Language Model Self-Improvement

Open paper AskPDF arXiv

Self-evolution offers a scalable path to stronger reasoning: a pretrained language model improves itself with only minimal external supervision. Yet existing methods either depend on extensively curated or teacher-generated training data, or, when the generator runs unsupervised, reward it by a difficulty heuristic that need not improve the solver. We introduce INFUSER, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them. The solver is trained with standard correctness rewards against the generator-provided answers, while the generator is rewarded by an optimizer-aware influence score that measures whether each proposed question would actually improve the solver on the target distribution. Because this continuous, noisy influence score is poorly served by standard GRPO, we propose DuGRPO, a dual-normalized variant of GRPO, for generator training. Together, these turn the document pool into an adaptive curriculum that favors questions useful to the current solver, not just hard ones. On Qwen3-8B-Base, INFUSER outperforms strong self-evolution baselines with over 20% relative improvement on Olympiad and SuperGPQA benchmarks, and an 8B INFUSER co-evolving generator outperforms a frozen 32B thinking generator on math and coding. Ablations confirm each design choice is necessary, and two extensions, applying INFUSER to an instruction-finetuned anchor and augmenting it with rule-verifiable RLVR data, further demonstrate the flexibility and generalizability of the framework. Code is available at https://github.com/FFishy-git/INFUSER.

Abstract

Explore similar work

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Models

Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula