Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

Yuyang Deng; Yu Wang; Jiayun Wang

cs.LGSep 28, 2026

Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

Authors: Yuyang Deng, Yu Wang, Jiayun Wang

Organizations: Accenture, Center for Advanced AI · Georgia Institute of Technology

Abstract

Self-evolving language models improve by generating tasks and learning from their own feedback, but adapting the task generator often requires a separate challenger-training loop. Can we generate tasks adapted to the current solver without explicitly training a challenger? We introduce \textbf{D}irect Self-\textbf{E}volving \textbf{O}ptimization (DEO), which replaces challenger parameter updates with solver-guided task sampling. The KL-regularized challenger objective defines an exponential tilt of a fixed base task distribution. DEO uses this distribution as a sampling target: a frozen LLM generates and mutates tasks, the solver scores them, and an approximate Metropolis selection rule refines the training pool. Only the solver is trained. Theoretically, for an idealized variant that samples exactly from the tilted distribution, and under regularity, local gradient-dominance, and initialization conditions, we show that DEO learns distributionally robust reasoning ability. In experiments, DEO achieves reasoning performance competitive with R-Zero while using over $50\%$ less wall-clock training time, and improves reasoning accuracy over a no-walk ablation. Replacing the task generator with a frozen API-only LLM further improves the local solver, illustrating a capability enabled by removing challenger training.

Figures & tables

Figure 1: Comparison between traditional challenger training self-evolving method and our DEO. (A) Conventional self-evolving method updates both a task proposer and a solver. (B) DEO keeps the generator frozen and uses solver feedback to select task mutations; only the solver receives parameter updates. DEO uses more than half less wall-clock time than R-Zero while yielding competitive performance.

OctoThinker-3B-Hybrid-Base
Method	MATH 500	GSM8K	AMC	Minerva	Olympiad Bench	AIME 2024	AIME 2025	AVG7	HARD5	MMLU-Pro	SuperGPQA	BBEH	AVG3
Base	44.60	61.49	17.42	19.12	14.67	6.67	0.00	23.42	11.58	8.20	5.33	0.97	4.83
No walk	48.40	73.24	25.00	27.21	18.67	0.00	0.00	27.50	14.18	17.73	9.70	3.30	10.24
R-Zero	50.20	73.09	12.50	27.21	15.26	3.33	0.00	25.94	11.66	19.10	10.47	4.00	11.19
DEO	52.60	74.75	25.08	26.47	15.85	3.33	0.00	28.30	14.15	17.97	10.83	3.70	10.83
Qwen3-4B-Base

Table 1: Base checkpoints and iteration-three results (accuracy, %). DEO and no walk use 2000 initial candidates; the Qwen R-Zero runs retain their native larger pools. Implementation settings are in Appendix E . Base denotes the pretrained checkpoint before self-evolution. Bold marks the best score in each column within each base-model block, including ties.

	In-band			Retained
Method	Iter 1	Iter 2	Iter 3	Iter 1	Iter 2	Iter 3
No walk	1029	970	1044	977	913	1002
DEO	1386	1358	1393	1246	1208	1228

Table 2: Question counts for Qwen3-4B, iterations 1–3. In-band counts precede the pseudo-label and format checks; retained counts follow them. DEO and no walk each start with 2000 candidates.

Figure 3: Accepted LLM mutations add reasoning requirements (Qwen3-4B, iteration one). A and B show the seed and first accepted mutation, with the added reasoning explained on the right; C shows both accepted mutations. Each shown mutation raises the uncertainty score and is accepted with probability one.

Generator	MATH 500	GSM8K	AMC	Minerva	Olympiad Bench	AIME 2024	AIME 2025	AVG7	HARD5
Qwen3-4B	76.80	91.43	57.34	45.96	41.48	6.67	13.33	47.57	32.96
Claude	76.80	91.51	52.50	50.37	42.67	20.00	13.33	49.60	35.77

Table 3: DEO with a Qwen3-4B versus Claude Haiku 4.5 generator/mutator; Qwen3-4B solver, iteration three (%). Bold: better score, including ties. For the Claude one, the pseudo-labels for training were still produced by last-round solver

\theta_{t}

. The Claude generator+mutator can notably boost the performance, while the R-Zero type of method cannot leverage that.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Base models	Qwen3-4B/8B-Base; OctoThinker-3B-Hybrid-Base
Candidate pool per round	$N=2000$ , freshly generated
Mutation sweeps	$L_{\mathrm{w}}=5$
MH temperature	Fixed $\beta=0.1$
Agreement rollouts per evaluated question	$m=9$
Agreement sampling	Temperature $1.0$ , top- $p=1.0$ , top- $k=40$

Appendix

Table 4: Configuration of the local-generator DEO runs. The agreement rollouts and GRPO training rollouts serve different purposes.

Model	Base	No walk	R-Zero	DEO
OctoThinker-3B	9.50	12.80	13.10	12.80
Qwen3-4B	22.00	25.50	24.75	24.80
Qwen3-8B	23.55	26.55	26.65	27.00

Appendix

Table 5: Omni-MATH accuracy (%) on the fixed 2000-question subset. All trained methods use iteration three. Bold marks the best trained-method score in each row.

		AVG7 ( $\uparrow$ )			HARD5 ( $\uparrow$ )
Model	Method	Iter 1	Iter 2	Iter 3	Iter 1	Iter 2	Iter 3
4B	No walk	45.49	47.73	45.68	30.02	32.84	30.49
	R-Zero	46.46	48.13	45.93	31.38	33.96	31.16
	DEO	45.09	47.71	47.57	29.02	33.18	32.96
8B	No walk	51.91	50.64	51.78	38.04	36.28	37.49
	R-Zero	51.96	51.70	52.88	37.88	37.68	38.97

Appendix

Table 6: DEO, R-Zero and the no-walk ablation through iteration three. DEO and no walk use 2000 initial candidates per round; R-Zero uses its native larger question pool. DEO uses fixed

\beta=0.1

, five sweeps and plain noisy acceptance, without CD or auxiliary interventions. Bold marks the best score among the three methods at each scale and iteration.

Explore similar work

INFUSER: Influence-Guided Self-Evolution Improves Reasoning
Jun 8, 2026Siyu Chen, Miao Lu, Beining Wu +7Self-Evolution Iterative Co-Training
Yale University · Stanford University · University of Chicago +2
A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2LLM Reasoning Strategies Mathematical Reasoning Benchmarks
1The Chinese University of Hong Kong, Shenzhen · 2Shanghai Jiao Tong University · 3The Chinese University of Hong Kong
Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Models
Sep 1, 2026Xincheng Wei, Yifan Ding, Fucheng Xiong +6Self-Evolution Mathematical Reasoning Benchmarks
The Chinese University of Hong Kong, Shenzhen · Meituan, LongCat Team · Peking University +2

Jun 8, 2026, cs.LGJ/K move · Enter open · S save

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen+6

Yale University · Stanford University · University of Chicago+2

Self-Evolution Iterative Co-Training Reasoning Skills Data Generation

Open paper AskPDF arXiv

Self-evolution offers a scalable path to stronger reasoning: a pretrained language model improves itself with only minimal external supervision. Yet existing methods either depend on extensively curated or teacher-generated training data, or, when the generator runs unsupervised, reward it by a difficulty heuristic that need not improve the solver. We introduce INFUSER, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them. The solver is trained with standard correctness rewards against the generator-provided answers, while the generator is rewarded by an optimizer-aware influence score that measures whether each proposed question would actually improve the solver on the target distribution. Because this continuous, noisy influence score is poorly served by standard GRPO, we propose DuGRPO, a dual-normalized variant of GRPO, for generator training. Together, these turn the document pool into an adaptive curriculum that favors questions useful to the current solver, not just hard ones. On Qwen3-8B-Base, INFUSER outperforms strong self-evolution baselines with over 20% relative improvement on Olympiad and SuperGPQA benchmarks, and an 8B INFUSER co-evolving generator outperforms a frozen 32B thinking generator on math and coding. Ablations confirm each design choice is necessary, and two extensions, applying INFUSER to an instruction-finetuned anchor and augmenting it with rule-verifiable RLVR data, further demonstrate the flexibility and generalizability of the framework. Code is available at https://github.com/FFishy-git/INFUSER.

Abstract

Explore similar work

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Models