Selection Is Not Learning: Choosing Model Futures

Haoyi Yang; Kshitij Mishra; Nils Lukas; Salem Lahlou

cs.LGOct 15, 2026

Selection Is Not Learning: Choosing Model Futures

Authors: Haoyi Yang, Kshitij Mishra, Nils Lukas, Salem Lahlou

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Abstract

Self-evolving language models train on tasks they generate, so every round must decide which updated model the next round inherits. Current loops such as R-Zero, Agent0, and Skill Self-Play make this decision through the task, admitting tasks that look learnable and inheriting every resulting update. We identify a selection-to-update gap in these loops, where a task that looks learnable need not produce a useful update. Most updates on verified tasks fail to make correct answers more likely, yet these loops inherit them all. HeirCommit freezes the updated model and uses fresh evaluation questions to decide whether to inherit it or retain the current model. EvoHeir separates parameter inheritance from experience inheritance, using current and archived experience to repair failures and preserve successes before final confirmation. Our analysis characterizes selection-induced optimism and decomposes regret into selection, absorption, and confirmation errors. HeirCommit cuts the harmful share of accepted updates by 68-83% relative to random inheritance at the same acceptance rate. Across four backbones, EvoHeir achieves the best mathematics and general-reasoning averages, raising the math average by 5.3% relative to the strongest baseline, Agent0. The framework lets useful experience persist even when the model that produced it is not inherited.

Figures & tables

Panel of Figure 1 — Figure 1: Selecting a learnable task does not guarantee learning, so EvoHeir judges model futures and keeps experience. Top: task-centric loops inherit every update, yet 60 of 96 updates on verified tasks show no gain, and 42–52% of admitted tasks leave the frontier under fresh responses. Bottom: EvoHeir explores, screens a recipient, repairs its failures with current and archived references while keeping its successes, and commits the result only if it beats the current model on fresh probes. The archive keeps experience either way. HeirCommit cuts the harmful share of commits on every host, and over ten rounds EvoHeir keeps climbing to $+5.7$ math points while R-Zero peaks at round 8 and slips (means over four backbones). All plots show measured data.

Figure 2: Self-evolution environment. Two agents co-evolve through the Evolution Controller.

Panel of Figure 3 — Figure 3: One EvoHeir round. Steps follow Algorithm 1 , and the archive keeps qualified experience whether or not the round commits.

Figure 4: Selection does not predict learning, and HeirCommit repairs inheritance. (a) Selected-target optimism, the support a target shows on the responses that selected it minus its support on fresh responses, against fixed-target controls. (b) Correct-target NLL gain of 96 real Skill-SP updates on exact-valid tasks, sorted by gain. (c,d) Harmful share of commits and pool-mean NLL gain over 64 native updates per host. Matched random commits HeirCommit ’s fraction of updates at random. Appendix D gives the full audit.

	Math Avg. (7 benchmarks) $\uparrow$						General Avg. (5 benchmarks) $\uparrow$
Backbone	Base	R-Zero	Agent0	No Rec.	EvoHeir	vs. Agent0	Base	R-Zero	Agent0	No Rec.	EvoHeir
Qwen3-4B-Base	25.81	28.79 $\pm$ 0.65	30.01 $\pm$ 0.57	30.29 $\pm$ 0.40	31.82 $\pm$ 0.55	$+$ 6.0%	27.00	28.82	29.51	29.82	30.52
Qwen3-8B-Base	28.06	32.65 $\pm$ 0.65	34.11 $\pm$ 0.63	34.35 $\pm$ 0.38	36.51 $\pm$ 0.67	$+$ 7.0%	32.34	33.84	35.24	35.75	36.48
OLMo-3-7B	19.92	21.73 $\pm$ 0.61	23.35 $\pm$ 0.44	23.50 $\pm$ 0.56	24.94 $\pm$ 0.48	$+$ 6.8%	18.05	18.66	20.05	20.34	21.40
Gemma-4-E4B-it	43.77	45.11 $\pm$ 0.50	46.66 $\pm$ 0.59	46.53 $\pm$ 0.42	47.21 $\pm$ 0.44	$+$ 1.2%	38.88	39.01	40.37	40.73	40.93

Table 1: Ten-round self-evolution across four backbones (%). Each average weights its benchmarks equally, and

\pm

is the sample SD of Math Avg. over three seeds. “vs. Agent0” is the relative Math Avg. gain of EvoHeir (Full) over Agent0. Figure 5 and Appendix E give per-benchmark results.

Figure 5: Per-benchmark gain of EvoHeir (Full) over Agent0 at round 10 (pp, three-seed means). 45 of 48 cells are positive, and the three negative cells are 30-problem AIME sets.

Figure 6: Ten-round Math Avg. trajectories, as change from round 0 (pp). Lines and bands show the mean and

\pm 1

SD over three seeds. The bar at round 10 marks EvoHeir ’s lead over Agent0, and each panel has its own y scale.

Figure 7: Experience value under fixed-recipient interventions (pp, 1,024 items each, before confirmation). (a) Donor gain against absorption gain for 16 donors. The shaded quadrant holds weaker donors whose experience helps, and

+

marks their mean. (b) Mean new (up) and lost (down) successes from the same 64 references, with black bars at the mean net gain and thin lines pairing the two methods at each recipient. (c) Absorption gain as history eligibility widens at a fixed 64-reference budget, with thin lines for the four recipients. (d) Paired contribution of non-inherited over same-age inherited history at 64 records, with pointwise 95% hierarchical bootstrap intervals over three seed groups. Age zero is an identical-material control.

Panel of Figure 8 — Figure 8: Qualitative analysis on Qwen3-4B-Base at round 5. (a) A round-4 archived reference from a proposal that was not inherited recovers $x=16$ after the recipient drops the cross term; Recovery reproduces it without the reference. (b) Under the same 64-reference budget, Recovery keeps the remainder that Cache SFT loses. The fresh-probe gain is $-0.31$ pp, so HeirCommit retains the root.

Variant	Math Avg. $\uparrow$	General Avg. $\uparrow$
EvoHeir (Full)	31.82 $\pm$ 0.55	30.52 $\pm$ 0.35
Same-Lineage	30.77 $\pm$ 0.57	29.99 $\pm$ 0.35
Random Experience	30.54 $\pm$ 0.44	29.87 $\pm$ 0.43
Current-Round Only	30.85 $\pm$ 0.69	30.04 $\pm$ 0.35
NoConfirm	29.80 $\pm$ 1.13	29.33 $\pm$ 0.65

Table 2: Component ablations (%, mean

\pm

SD over three seeds).

Figure 9: Equal-budget comparison and sensitivity on Qwen3-4B-Base . (a–c) Math Avg. after ten rounds, mean

\pm

SD over three training seeds. (a) Four methods at equal accounting budgets, relative to Full’s default of 123.7 units. (b,c) Candidate count and confirmation-pool size at budget 1.0, where native training absorbs the remaining budget. Open rings mark the defaults, which reuse the main runs. (d) Net absorption gain of a fixed recipient on 2,400 problems as the preservation weight changes, over three absorption-order seeds. Accounting units are not measured GPU-hours or FLOPs.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Host	Persistence rule	Coverage	Useful retained	Harmful commits	Mean gain ( $10^{-3}$ NLL)	$\Delta$ Native ( $10^{-3}$ NLL)
Skill-SP	Native commit-all	100.00%	100.00%	57.81%	$-3.47$	$0.00$
	Matched random	39.06%	39.06%	57.81%	$-1.36$	$+2.11$
	HeirCommit	39.06%	77.78%	16.00%	$+2.49$	$+5.96$
R-Zero	Native commit-all	100.00%	100.00%	76.56%	$-23.17$	$0.00$
	Matched random	23.44%	23.44%	76.56%	$-5.43$	$+17.74$
	HeirCommit	23.44%	86.67%	13.33%	$+10.19$	$+33.36$

Appendix

Table 4: Complete HeirCommit audit across native update sources. Matched random uses HeirCommit ’s coverage within each host. Gain and

\Delta

Native are in

10^{-3}

NLL units; higher is better.

Figure 10: Complete self-evolution results (%). Every benchmark, backbone, and method after ten rounds. Numbers are three-seed means, and color shows the change from each backbone’s starting checkpoint (Base, gray rows).

Figure 11: Supplementary interventions. (a) Objective controls on a fixed recipient and 2,400 problems, with mean

\pm

SD net gain over three absorption-order seeds. (b) New and lost successes as the preservation weight changes. (c) Absorption gain with non-inherited or same-age inherited history at 64 records. (d) Net gain of Recovery and Cache SFT at each recipient checkpoint.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
Sep 18, 2025Yujun Zhou, Zhenwen Liang, Haolin Liu +7Language Model Self-Improvement RL for Language Models
Tencent AI Lab · University of Notre Dame · University of Virginia
When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain
Sep 29, 2026Chenxu Wang, Chaozhuo Li, Xinze Shi +6Language Model Self-Improvement LLM Training
Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence · Central South University +2
Learning to Self-Evolve
Mar 19, 2026Xiaoyin Chen, Canwen Xu, Yite Wang +3Language Model Self-Improvement RL for Language Models
Mila – Quebec AI Institute · University of Montreal · Snowflake

Sep 18, 2025J/K move · Enter open · S save

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu+6

Tencent AI Lab · University of Notre Dame · University of Virginia

Language Model Self-Improvement RL for Language Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: https://github.com/YujunZhou/EVOL-RL.

Abstract

Language Model Self-Improvement

Explore similar work

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain

Learning to Self-Evolve