Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

Jacob Dineen; Aswin RRV; Zhikun Xu; Ben Zhou

cs.CLApr 3, 2026

Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

Authors: Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou

Organizations: Arizona State University

Abstract

Co-evolutionary self-play, where one language model generates problems and another solves them, promises curriculum learning without human supervision. The promise breaks down early in practice. The proposer converges to a narrow distribution of problems that satisfy the reward function, and the collapsed curriculum teaches the solver little, stalling the loop. We introduce vocabulary dropout, a lightweight intervention that randomly masks the proposer's output logits during both policy training and curriculum generation. The mask is hard and non-stationary, so the proposer cannot lock into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, vocabulary dropout sustains proposer diversity throughout training across lexical, semantic, and functional measures, and improves the solver by an average of +4.4 points at 8B with the largest gains on competition-level benchmarks. Explicit action-space constraints, filling the structural role that game rules fill in classical self-play, can keep co-evolution in language productive. Vocabulary dropout is one simple way to impose them.

Figures & tables

Figure 1: Training pipeline. Left : Vocabulary dropout masks a random subset of output logits, constraining the proposer’s token distribution. Orange denotes masked tokens. Right : The co-evolution loop. In Phase 1 (proposer training), the proposer generates $K$ problems, the frozen solver attempts each $M$ times, and the proposer is rewarded based on solver uncertainty. In Phase 2 (solver training), the frozen proposer generates a curriculum of $K$ problems and the solver is rewarded for matching the correct answer.

Qwen3-4B
	Pass@1 (%)
Setting	MATH500	GSM8K	AMC	Olympiad	AIME’24	AIME’25	Avg.
Base (no training)	54.1 $\pm$ 2.2	63.5 $\pm$ 5.2	29.2 $\pm$ 1.8	20.6 $\pm$ 1.3	12.2 $\pm$ 0.9	1.1 $\pm$ 0.9	30.1
KL-Cov	60.8 $\pm$ 1.9	82.6 $\pm$ 2.0	42.0 $\pm$ 3.8	23.0 $\pm$ 0.8	9.3 $\pm$ 2.6	4.0 $\pm$ 1.1	37.0
Clip-Cov	65.8 $\pm$ 1.1	85.4 $\pm$ 1.0	39.5 $\pm$ 3.4	24.5 $\pm$ 0.2	12.7 $\pm$ 1.7	4.0 $\pm$ 1.7	38.7
Baseline ( $\alpha{=}1.0$ )	64.2 $\pm$ 0.8	85.2 $\pm$ 0.7	41.7 $\pm$ 1.4	23.2 $\pm$ 0.3	8.9 $\pm$ 0.9	6.7 $\pm$ 1.6	38.3

Table 1: Solver pass@1 accuracy (%) after 5 co-evolution iterations. Vocabulary dropout (VD) is applied to the proposer only, and the solver trains on the resulting curriculum.

\alpha

controls the fraction of the vocabulary retained. Phase ablations decompose

\alpha{=}0.75

into train-only and gen-only. Best per model in green bold .

Panel of Figure 2 — Figure 2: Question profile at iteration 5 (% change from baseline). VD85 VD75.

Panel of Figure 3 — Figure 3: Diversity and curriculum quality over co-evolution iterations ( $\alpha{=}0.75$ , both phases). Top: collapse signals. Bottom: diversity metrics. Dashed = baseline, solid = dropout. Semantic metrics (b, d, e) use text-embedding-3-small ( OpenAI, 2024 ) .

Trained on	MATH500	GSM8K	AMC	Olympiad	AIME’24	AIME’25	Minerva	Avg.
Base (no training)	55.0	75.1	25.0	20.0	6.7	3.3	25.4	30.1
Baseline	71.0	88.6	40.0	28.2	3.3	6.7	29.8	38.2
KL-Cov	69.8	88.6	45.0	27.6	10.0	3.3	31.6	39.4
Clip-Cov	66.6	86.7	40.0	24.2	10.0	3.3	32.0	37.5
VD	71.0	89.8	50.0	27.8	3.3	10.0	37.1	41.3

Table 2: Solver pass@1 (%) after GRPO on each arm’s externally-verified iteration-5 curriculum, training a fresh Qwen3-4B-Base. Curricula are taken from the 8B co-evolution runs and verified with GPT-5-mini. Best trained arm per column in green bold .

Scale	Method	Filter pass	Pass rate	Well-posed	Correct	$\Delta$ Vendi (WP)	$\Delta$ Vendi (C)
4B	Baseline	946	11.8%	850	275	—	—
4B	VD75	1,222	15.3%	925	269	$+11.4$	$+7.1$
8B	Baseline	510	6.4%	465	163	—	—
8B	VD75	1,408	17.6%	1,112	325	$+7.5$	$+2.8$

Table 3: Curriculum composition and diversity at the final iteration. Filter pass and Pass rate count generated problems landing in the difficulty band

[0.3,0.7]

, Well-posed and Correct are GPT-4.1-mini judgments, and

\Delta

Vendi is VD75 minus baseline on the well-posed (WP) and correct (C) subsets.

Panel of Table 4 — Table 4: Left : zero-shot out-of-domain transfer of the 8B solver (VD at $\alpha{=}0.75$ ), evaluated under the protocol of Table 1 . Best between Baseline and VD per row in green bold . Right : policy mean per-token entropy over the first 100 GRPO steps of single-policy training (Qwen3-4B-Base on GSM8K), baseline vs. VD ( $\alpha{=}0.95$ ).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Hyperparameter	Value
KL-Cov	$k$ (top-covariance token fraction)	$2\times 10^{-3}$
	$\beta$ (KL coefficient on selected tokens)	1.0
Clip-Cov	$\rho$ (masked-token fraction)	$2\times 10^{-4}$
	$\omega_{\text{low}},\omega_{\text{high}}$ (covariance bounds)	1.0, 5.0
Both	Entropy bonus coefficient	0
	Scope	Proposer only

Appendix

Table 5: Entropy-control hyperparameters, following Cui et al. (2025) ’s 7B reference configuration.

Model	Setting	It. 1	It. 2	It. 3	It. 4	It. 5
Qwen3-8B	Baseline	81.4	45.5	42.2	39.1	38.0
	Train-only	79.0	43.2	42.0	40.4	39.0
	Gen-only	62.0	38.3	33.7	33.4	31.9
	VD75	61.8	34.9	33.1	36.7	32.4
Qwen3-4B	Baseline	75.6	38.1	38.0	42.2	37.7
	Train-only	76.0	37.2	44.2	38.0	38.2

Appendix

Table 6: Proposer answer correctness (%) over co-evolution iterations, judged by GPT-4.1-mini solving each problem independently.

Setting	MATH500	GSM8K	AMC	Olympiad	AIME’24	AIME’25	Avg.
	Pass@1 (%)
Qwen3-4B
Base (no training)	54.1 $\pm$ 2.2	63.5 $\pm$ 5.2	29.2 $\pm$ 1.8	20.6 $\pm$ 1.3	12.2 $\pm$ 0.9	1.1 $\pm$ 0.9	30.1
Baseline ( $\alpha{=}1.0$ )	56.0 $\pm$ 1.1	72.7 $\pm$ 3.4	33.3 $\pm$ 3.0	20.3 $\pm$ 0.9	10.0 $\pm$ 1.6	4.4 $\pm$ 0.9	32.8
VD85 ( $\alpha{=}0.85$ )	58.4 $\pm$ 0.8	60.5 $\pm$ 4.1	37.5 $\pm$ 4.1	19.3 $\pm$ 1.1	11.1 $\pm$ 0.9	4.4 $\pm$ 2.4	31.9
Qwen3-8B

Appendix

Table 7: Proposer reasoning ability (pass@1 %) on standard benchmarks. VD is applied to the proposer during co-evolution. Co-evolutionary training does not improve the proposer’s own problem-solving ability.

Panel of Figure 5 — Figure 5: Cumulative Vendi score (questions pooled across iterations 1– $N$ , subsampled to 2,000). VD75 maintains higher diversity at both scales.

Model	Setting	Value	$\Delta$	Value	$\Delta$
		Semantic $\uparrow$		Logical $\uparrow$
Qwen3-4B	Baseline	$0.665\pm{<}0.001$	—	$+0.241\pm 0.013$	—
Qwen3-4B	VD	${\color[rgb]{0,0.4688,0.3125}\textbf{0.692}}\pm{<}0.001$	$+4.1\%$	${\color[rgb]{0,0.4688,0.3125}\textbf{+0.405}}\pm 0.014$	$+68.1\%$
Qwen3-8B	Baseline	$0.638\pm{<}0.001$	—	$+0.315\pm 0.013$	—
Qwen3-8B	VD	${\color[rgb]{0,0.4688,0.3125}\textbf{0.677}}\pm{<}0.001$	$+6.1\%$	${\color[rgb]{0,0.4688,0.3125}\textbf{+0.376}}\pm 0.016$	$+19.4\%$

Appendix

Table 8: NLI and cosine diversity at iteration 5, on the curriculum-filtered problem sets the solver trains on. Higher is better for both. Standard errors bootstrapped from 100 resamples. Best per scale in green bold .

Model	Setting	It. 1	1–2	1–3	1–4	1–5	Growth
		Cumulative Vendi Score $\uparrow$
Qwen3-4B	Baseline	44.6	46.1	48.9	49.5	52.1	+7.5
	Train-only	46.1	51.0	52.0	53.7	55.7	+9.6
	Gen-only	59.7	62.9	63.9	63.9	64.3	+4.6
	Both	61.5	63.9	68.6	69.7	70.2	+8.7
Qwen3-8B	Baseline	40.0	41.5	42.1	43.2	42.2	+2.2

Appendix

Table 9: Cumulative question diversity (Vendi score) over co-evolution iterations via text-embedding-3-small embeddings. Growth is the gain from iteration 1 to iterations 1–5 pooled.

Panel of Figure 6 — Figure 6: Embedding diversity by dropout phase ( $\alpha{=}0.75$ ). Both phases combined achieves the highest diversity. All metrics use text-embedding-3-small .

Proposer	Setting	MATH500	GSM8K	AMC	Olympiad	AIME’24	AIME’25	Avg.
		Pass@1 (%)
4B	Baseline	64.2 $\pm$ 0.8	85.2 $\pm$ 0.7	41.7 $\pm$ 1.4	23.2 $\pm$ 0.3	8.9 $\pm$ 0.9	6.7 $\pm$ 1.6	38.3
4B	+ VD85	65.7 $\pm$ 0.9	81.8 $\pm$ 0.4	50.0 $\pm$ 2.4	24.7 $\pm$ 0.9	8.9 $\pm$ 1.8	4.4 $\pm$ 0.9	39.3
8B	Baseline	63.5 $\pm$ 2.7	82.7 $\pm$ 2.5	44.2 $\pm$ 2.4	24.4 $\pm$ 0.1	11.1 $\pm$ 1.6	6.7 $\pm$ 5.4	38.8
8B	+ VD85 (gen-only)	66.1 $\pm$ 1.1	83.4 $\pm$ 2.7	35.8 $\pm$ 4.3	23.4 $\pm$ 0.7	13.3 $\pm$ 2.7	2.2 $\pm$ 1.6	37.4

Appendix

Table 10: Cross-scale co-evolution: 8B proposer

\to

4B solver, compared with symmetric 4B

\to

4B from Table 1 . Best per group in green bold .

Model	MATH500	AMC
Base (no training)	47.7	25.3
Baseline solver	47.3	26.0
VD75 solver	47.4	25.2

Appendix

Table 11: Co-evolution on Qwen2.5-1.5B-Instruct (5 iterations).

	Qwen3-4B		Qwen3-8B
Setting	Unique tok.	# Q	Unique tok.	# Q
Baseline	2,717	6,771	1,881	7,044
VD75	3,697	3,377	2,868	4,864
$\Delta$	+36%		+52%

Appendix

Table 12: Unique tokens in iteration-5 generated questions.

Panel of Figure 7 — Figure 7: Qwen3-8B solver accuracy across iterations under fixed vs. annealed ( $0.75{\to}1.0$ ) vocabulary dropout. Teal $\alpha$ values below each tick show the anneal schedule. (a) Mean of MATH500, GSM8K, OlympiadBench, and Minerva Math.

Setting	MATH500	GSM8K	AMC	Olympiad	AIME’24	AIME’25	Minerva
Baseline ( $\alpha{=}1.0$ )	70.6	86.3	47.5	23.6	6.7	3.3	29.8
VD ( $\alpha{=}0.95$ )	63.0	87.9	45.0	24.1	10.0	0.0	34.9

Appendix

Table 13: Vocabulary dropout applied to a single GRPO policy, with no co-evolution: Qwen3-4B-Base trained on GSM8K with a binary correctness reward, with (

\alpha{=}0.95

) and without (

\alpha{=}1.0

) vocabulary dropout, evaluated zero-shot (pass@1 %). Best per benchmark in green bold .

Benchmark	# Examples	Domain
GSM8K ( Cobbe et al., 2021 )	1,319	Grade-school math
MATH500 ( Hendrycks et al., 2021 )	500	Competition math
AMC ( Mathematical Association of America, 2023 )	40	Competition math
OlympiadBench ( He et al., 2024 )	910	Olympiad math
AIME 2024 ( Mathematical Association of America, 2024 )	30	Olympiad math
AIME 2025 ( Mathematical Association of America, 2024 )	30	Olympiad math

Appendix

Table 14: Evaluation benchmarks.

Parameter	Proposer	Solver
Base model	Qwen3-4B / Qwen3-8B
Optimizer	AdamW
Learning rate	$1\times 10^{-6}$
Weight decay	$1\times 10^{-2}$
LR warmup	0
Max grad norm	1.0

Appendix

Table 15: GRPO training hyperparameters for proposer and solver.

Parameter	Value
Embedding model	text-embedding-3-small
Embedding dimension	1536
Normalization	L2 (cosine kernel)
Max samples (eigendecomp.)	5,000
Subsampling seed	42
Eigenvalue threshold	$10^{-12}$

Appendix

Table 16: Vendi score hyperparameters.

Parameter	Value
Observer model	(same as base)
LoRA rank ( $r$ )	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q/k/v/o_proj, gate/up/down_proj
Max sequence length	512

Appendix

Table 17: Epiplexity computation hyperparameters.

Curriculum RL

Curriculum reinforcement learning organizes tasks, examples, or training conditions to help agents learn complex behaviors efficiently.

See all 121 papers

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
Sep 18, 2025Yujun Zhou, Zhenwen Liang, Haolin Liu +7Language Model Self-Improvement RL for Language Models
Tencent AI Lab · University of Notre Dame · University of Virginia
DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolution
Aug 12, 2026Siheng Xiong, Ali Payani, Oguzhan Gungordu +1Language Model Self-Improvement
Georgia Institute of Technology · Cisco Research
Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula
Aug 30, 2026Vinoth Selvendran, Zhanming ZhangReward Modeling RL for Language Model Reasoning
Independent Researcher · Palo Alto, CA, USA · New York, NY, USA

Sep 18, 2025J/K move · Enter open · S save

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu+6

Tencent AI Lab · University of Notre Dame · University of Virginia

Language Model Self-Improvement RL for Language Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: https://github.com/YujunZhou/EVOL-RL.

Abstract

Curriculum RL

Explore similar work

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolution

Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula