Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Yujun Zhou; Zhenwen Liang; Haolin Liu; Wenhao Yu; Kishan Panaganti; Linfeng Song; Dian Yu; Xiangliang Zhang

cs.LGSep 18, 2025

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Authors: Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang,

Organizations: Tencent AI Lab · University of Notre Dame · University of Virginia

Abstract

Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: https://github.com/YujunZhou/EVOL-RL.

Figures & tables

Figure 1: Training dynamics of TTRL and Evol-RL on Qwen3-4B-Base. TTRL improves pass@1, but pass@16, response length, and policy entropy decline; Evol-RL preserves broader coverage.

Figure 2: Overview of Evol-RL . Majority voting provides the selection signal, while trace novelty refines rewards within each selection group. Entropy regularization maintains rollout diversity, and asymmetric clipping preserves rare, high-advantage updates. Values are illustrative.

Training Dataset	Model	MATH	AIME24	AIME25	AMC	GPQA
Qwen3-4B-Base
–	Base Model	67.4/89.6	10.0/32.4	5.5/30.0	39.3/75.2	34.4/85.6
MATH-TRAIN	w/TTRL	75.4/86.9	12.1/23.2	6.8/28.6	42.5/75.2	36.5/81.4
	w/ Evol-RL	80.0/93.3	20.7/47.6	17.5/39.9	51.4/80.3	37.2/88.7
	$\Delta$	+4.6 / +6.4	+8.6 / +24.4	+10.7 / +11.3	+8.9 / +5.1	+0.7 / +7.3
MATH-500	w/TTRL	79.3/83.2	10.0/28.0	7.2/29.9	47.6/72.0	36.2/75.9

Table 1: TTRL and Evol-RL across training settings. Each cell reports pass@1/pass@16 averaged over 32 rollouts;

\Delta

denotes Evol-RL minus TTRL.

Method	MATH	AIME24	AIME25
Base Model	67.4/89.6	10.0/32.4	5.5/30.0
TTRL	79.3/83.2	10.0/28.0	7.2/29.9
EMPO	79.9 /88.9	11.8/31.6	9.5/26.3
Co-Reward	77.6/89.2	13.6/25.9	10.5/30.3
ECHO	78.5/86.4	10.6/30.6	10.5/31.7
Evol-RL	79.8/ 93.8	19.0/43.2	16.1/41.9

Table 2: Matched comparison on Qwen3-4B-Base trained on MATH-500. Each cell reports pass@1/pass@16.

Training Dataset	Model	MATH	AIME24	AIME25	AMC	GPQA
MATH-500	w/ Evol-RL	79.8 / 93.8	19.0 / 43.2	16.1 / 41.9	50.3 / 82.2	38.8 / 89.1
	-ClipHigh	75.1/91.8	12.2/31.8	11.4/31.3	42.7/73.9	32.3/81.8
	-Ent	79.5/93.4	18.3/38.5	14.7/34.3	48.3/78.6	38.6/87.0
	-ClipHigh-Ent	76.3/92.6	12.8/38.8	12.5/37.4	46.2/77.4	35.6/88.8
	-Novelty Reward	79.3/88.7	12.1/27.0	11.1/34.8	47.6/73.3	37.9/81.4
AIME24	w/ Evol-RL	79.6 / 93.6	20.6 / 40.9	17.1 / 42.0	49.9 / 80.9	38.0 / 87.8

Table 3: Performance of Qwen3-4B-Base with Evol-RL and its ablations. Each cell reports pass@1/pass@16 accuracy.

Figure 3: Training dynamics of TTRL and Evol-RL . Columns correspond to training on MATH-TRAIN, MATH-500, and AIME24; rows show held-out AIME25 pass@1, response length, and policy entropy.

	TTRL	Evol-RL
Correct-minority	45.1 $\rightarrow$ 38.3	45.3 $\rightarrow$ 55.5
MATH pass@16	89.0 $\rightarrow$ 88.5	89.8 $\rightarrow$ 93.1

Table 4: Training dynamics from step 40 to 105 on Qwen3-4B-Base with MATH-500.

Figure 4: Performance of Evol-RL components applied to supervised GRPO. Qwen3-4B-Base is trained on MATH with a ground-truth verifier. Outer bars show pass@16 and inner bars show pass@1.

Model	MMLU-Pro		SuperGPQA		BBEH
	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4
Qwen3-8B-Base	47.3	74.5	26.5	54.1	10.4	24.0
w/TTRL	53.4	73.9	29.7	53.3	12.1	24.1
w/EVOL-RL	55.3	78.5	30.2	57.0	11.5	24.9

Table 5: Transfer performance of the Qwen3-8B-Base model on broader reasoning benchmarks after label-free training on MATH-TRAIN.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Train Batch Size	8
PPO Mini-Batch Size	1 (effective size of 32)
PPO Micro-Batch Size	2
Rollouts for Majority Vote	64
Rollouts Used for Training	32
Generation Temperature	1.0

Appendix

Table 6: General hyperparameters for label-free training, following TTRL.

Hyperparameter	Value
Asymmetric Clipping High ( $\epsilon_{\text{high}}$ )	0.28
Entropy Regularizer Coefficient ( $\lambda_{\text{ent}}$ )	0.003
Novelty Score Mixing Coefficient ( $\alpha$ )	0.5

Appendix

Table 7: Key hyperparameters specific to the Evol-RL framework.

Model	MATH	AIME24	AIME25	AMC	GPQA
OctoThinker-8B-Hybrid-Base
Base Model	33.8/79.8	1.5/13.5	1.3/13.9	16.2/56.9	26.3/85.7
w/TTRL	63.8/76.4	2.8/10.8	2.1/11.0	27.9/54.7	31.9/71.5
w/ Evol-RL	63.2/86.3	9.0/30.3	7.2/22.4	34.1/65.6	33.2/85.7
$\Delta$	-0.6 / +9.9	+6.2 / +19.5	+5.1 / +11.4	+6.2 / +10.9	+1.3 / +14.2

Appendix

Table 8: Comparison of models trained with TTRL and Evol-RL on MATH-500 on OctoThinker-8B-Hybrid-Base. Each cell shows pass@1/pass@16 (averaged over 32 rollouts).

\Delta

uses red (+) for positive and blue for negative values.

Model	AIME24	AIME25
Qwen3.5-35B-A3B-Base
Base Model	57.0/80.8	44.9/67.6
w/TTRL	63.0/86.2	45.3/67.9
w/ Evol-RL	63.6/88.4	48.2/71.4
$\Delta$	+0.6 / +2.2	+2.9 / +3.5

Appendix

Table 9: Short-horizon comparison of TTRL and Evol-RL on Qwen3.5-35B-A3B-Base trained on AIME24 for 30 steps using LoRA. Each cell shows pass@1/pass@16.

\Delta

uses red (+) for positive and blue for negative values.

Model	MATH	AIME24	AIME25	AMC	GPQA
Qwen3-4B-Base
Base Model	67.4/89.6	10.0/32.4	5.5/30.0	39.3/75.2	34.4/85.6
w/TTRL	75.4/86.9	12.1/23.2	6.8/28.6	42.5/75.2	36.5/81.4
w/EM-RL-Token	76.0/90.6	12.5/31.3	10.5/30.8	46.6/77.7	36.8/82.6
w/EM-RL-Sequence	67.4/89.9	10.6/31.4	7.1/28.8	39.6/73.7	34.5/86.2
w/Self-Consistency	76.0/89.7	12.5/30.4	10.4/33.6	48.1/78.1	35.9/81.2

Appendix

Table 10: Extended comparison of baseline methods on Qwen3-4B-Base, trained in a label-free setting using MATH-Train. Each cell shows pass@1/pass@16, and the highest value in each column is bolded.

Metric	TTRL	Evol-RL
MATH pass@1	75.4 $\rightarrow$ 77.8	76.1 $\rightarrow$ 78.6
MATH pass@16	89.0 $\rightarrow$ 88.5	89.8 $\rightarrow$ 93.1
Correct-minority availability	45.1 $\rightarrow$ 38.3	45.3 $\rightarrow$ 55.5
Mean response length	524 $\rightarrow$ 488	593 $\rightarrow$ 1,983

Appendix

Table 11: Training dynamics from step 40 to 105 on Qwen3-4B-Base trained on MATH-500.

Checkpoint	Different-Approach Rate	Correct Pairs
Base	11.1%	25
Before evolving point	14.3%	45
Final	22.2%	51

Appendix

Table 12: Reasoning-strategy diversity across training checkpoints on AIME24. The analysis includes only pairs in which both solutions are correct.

Figure 5: Majority-vote accuracy (maj@16) on AIME25 throughout training. The models are trained on ( Left ) MATH-TRAIN, ( Middle ) MATH-500, and ( Right ) AIME24.

Figure 6: Training dynamics of TTRL and Evol-RL on Qwen3-8B-Base. Columns correspond to training on MATH-TRAIN, MATH-500, and AIME24; rows show held-out AIME25 pass@1, average response length, and policy entropy.

Figure 7: Pass@16 training curves on AIME24 and AIME25 across three training settings (MATH-TRAIN / MATH-500 / AIME24). TTRL’s pass@16 consistently declines, while Evol-RL ’s steadily recovers and grows after the evolving point.

Figure 8: Wall-clock time (seconds) per training step for Novelty Reward Calculation (Red) versus the Total Training Process (Blue) across three datasets. While both costs naturally increase as the model evolves to generate longer reasoning paths, the reward calculation overhead remains a minor fraction of the total computational load.

Figure 9: Case study of the reasoning path cosine similarity matrix for 8 rollouts on a single AIME25 problem. The axes are labeled with the final_answer(index_i) for each rollout. The 8 rollouts produced three distinct final answers: "19" (6 times), "31" (1 time), and "7221" (1 time).

Length	# Q	High-sim $\to$ SAME	Low-sim $\to$ DIFF	Agreement	Sim. Gap
$<3$ K	12	$66.7\%$	$83.3\%$	$75.0\%$	$+0.301$
3K–4.5K	20	$80.0\%$	$91.7\%$	$85.8\%$	$+0.466$
4.5K–6K	13	$69.2\%$	$89.7\%$	$79.5\%$	$+0.378$
6K+	15	$88.9\%$	$95.6\%$	$92.2\%$	$+0.550$
Total	60	$\mathbf{77.2\%}$	$\mathbf{90.6\%}$	$\mathbf{83.9\%}$	$\mathbf{+0.436}$

Appendix

Table 13: LLM-judge agreement with cosine-similarity ranking, grouped by response length.

Comparison	Cosine Similarity
Refined vs. original (within-pair)	$0.834$
Cross-solution (different reasoning)	$0.681$
Gap	$\mathbf{+0.154}$

Appendix

Table 14: Surface-rewriting robustness on

300

solutions (

30

AIME24/25 questions

\times

5

each). Refined solutions remain close to their originals, while cross-solution pairs are clearly more distant.

Novelty Signal	MATH p@1/p@16	AIME24 p@1/p@16	AIME25 p@1/p@16
Qwen3-Embedding-4B	79.8/93.8	19.0/43.2	16.1/41.9
OpenAI text-embedding-3-small	79.3/93.8	16.2/39.7	15.6/40.8
No novelty ( $-$ Novelty in Table 3 )	79.3/88.7	12.1/27.0	11.1/34.8

Appendix

Table 15: Cross-family embedding ablation; both embeddings improve multi-sample coverage over the no-novelty ablation.

$\beta$	MATH	AIME24	AIME25	GPQA
0.0	79.3/88.7	12.1/27.0	11.1/34.8	37.9/81.4
0.5	79.8 / 93.8	19.0 / 43.2	16.1/ 41.9	38.8 / 89.1
1.0	77.4/93.5	17.1/35.7	16.8 /36.1	35.4/81.7
1.5	59.6/89.0	5.8/24.0	7.0/31.2	33.5/80.2
2.0	36.4/84.4	4.4/16.4	4.3/23.0	33.0/84.3

Appendix

Table 16: Sensitivity to novelty reward strength

\beta

on Qwen3-4B-Base trained with MATH-500.

\beta=0.5

is the default setting. Each cell shows pass@1 / pass@16 .

$\alpha$	MATH	AIME24	AIME25	AMC	GPQA
0.00	78.4/92.4	17.1/39.8	14.4/38.0	48.7/79.8	37.8/87.3
0.25	79.4/93.4	18.5/42.1	15.7/40.9	49.9/81.7	38.5/88.6
0.50	79.8/93.8	19.0/43.2	16.1/41.9	50.3/82.2	38.8/89.1
0.75	79.5/93.6	18.6/42.6	15.9/41.5	50.1/82.0	38.6/88.9
1.00	79.1/93.0	18.2/41.4	15.3/40.2	49.4/81.2	38.3/88.4

Appendix

Table 17: Sensitivity to mixing coefficient

\alpha

. Qwen3-4B-Base trained on MATH-500. Each cell shows pass@1 / pass@16 .

$T$	MATH	AIME24	AIME25	AMC	GPQA
0.0	82.0/82.0	23.3/23.3	16.7/16.7	54.2/54.2	36.0/36.0
0.2	80.7/92.9	20.2/42.2	16.6/39.9	52.7/80.6	39.0/84.7
0.4	80.3/93.1	20.2/40.5	16.4/41.4	52.3/79.5	38.7/87.2
0.6	80.3/93.4	19.3/42.3	18.3/42.3	51.5/80.1	38.2/89.0
0.8	79.9/93.6	19.6/43.1	18.5/40.2	52.1/81.3	37.6/91.2
1.0	79.3/93.8	18.9/42.6	16.0/43.4	51.5/83.4	37.5/91.3

Appendix

Table 18: Sensitivity to sampling temperature

T

for the trained Evol-RL model (Qwen3-4B-Base, MATH-TRAIN). Each cell shows pass@1 / pass@16 .

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Language Models Learn to Improve Language Models
Oct 16, 2026Minhua Lin, Zijun Wang, Zewen Liu +7Language Model Self-Improvement Large Language Model-Guided Optimization
The Pennsylvania State University · UC Santa Cruz · Emory University +2
EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
May 5, 2026Shuyue Stella Li, Rui Xin, Teng Xiao +8RL for Language Models LLM Evaluation
University of Washington · 2Allen Institute for AI · University of Pennsylvania
Label-Free Reinforcement Learning via Cross-Model Entropy
May 27, 2026Matt Gorbett, Hossein ShiraziReinforcement Learning Instruction Following
Independent Researcher · San Diego State University

Oct 16, 2026J/K move · Enter open · S save

Language Models Learn to Improve Language Models

Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng+6

The Pennsylvania State University · UC Santa Cruz · Emory University+2

Language Model Self-Improvement Large Language Model-Guided Optimization RL for Language Models Reward Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

Abstract

Language Model Self-Improvement

Explore similar work

Language Models Learn to Improve Language Models

EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

Label-Free Reinforcement Learning via Cross-Model Entropy