Self-evolving language models train on tasks they generate, so every round must decide which updated model the next round inherits. Current loops such as R-Zero, Agent0, and Skill Self-Play make this decision through the task, admitting tasks that look learnable and inheriting every resulting update. We identify a selection-to-update gap in these loops, where a task that looks learnable need not produce a useful update. Most updates on verified tasks fail to make correct answers more likely, yet these loops inherit them all. HeirCommit freezes the updated model and uses fresh evaluation questions to decide whether to inherit it or retain the current model. EvoHeir separates parameter inheritance from experience inheritance, using current and archived experience to repair failures and preserve successes before final confirmation. Our analysis characterizes selection-induced optimism and decomposes regret into selection, absorption, and confirmation errors. HeirCommit cuts the harmful share of accepted updates by 68-83% relative to random inheritance at the same acceptance rate. Across four backbones, EvoHeir achieves the best mathematics and general-reasoning averages, raising the math average by 5.3% relative to the strongest baseline, Agent0. The framework lets useful experience persist even when the model that produced it is not inherited.
Figures & tables
Figure 1: Selecting a learnable task does not guarantee learning, so EvoHeir judges model futures and keeps experience. Top: task-centric loops inherit every update, yet 60 of 96 updates on verified tasks show no gain, and 42–52% of admitted tasks leave the frontier under fresh responses. Bottom: EvoHeir explores, screens a recipient, repairs its failures with current and archived references while keeping its successes, and commits the result only if it beats the current model on fresh probes. The archive keeps experience either way. HeirCommit cuts the harmful share of commits on every host, and over ten rounds EvoHeir keeps climbing to +5.7 math points while R-Zero peaks at round 8 and slips (means over four backbones). All plots show measured data.
Figure 2: Self-evolution environment. Two agents co-evolve through the Evolution Controller.
Figure 3: One EvoHeir round. Steps follow Algorithm 1 , and the archive keeps qualified experience whether or not the round commits.
Figure 4: Selection does not predict learning, and HeirCommit repairs inheritance. (a) Selected-target optimism, the support a target shows on the responses that selected it minus its support on fresh responses, against fixed-target controls. (b) Correct-target NLL gain of 96 real Skill-SP updates on exact-valid tasks, sorted by gain. (c,d) Harmful share of commits and pool-mean NLL gain over 64 native updates per host. Matched random commits HeirCommit ’s fraction of updates at random. Appendix D gives the full audit.
Math Avg. (7 benchmarks) ↑
General Avg. (5 benchmarks) ↑
Backbone
Base
R-Zero
Agent0
No Rec.
EvoHeir
vs. Agent0
Base
R-Zero
Agent0
No Rec.
EvoHeir
Qwen3-4B-Base
25.81
28.79 ± 0.65
30.01 ± 0.57
30.29 ± 0.40
31.82 ± 0.55
+ 6.0%
27.00
28.82
29.51
29.82
30.52
Qwen3-8B-Base
28.06
32.65 ± 0.65
34.11 ± 0.63
34.35 ± 0.38
36.51 ± 0.67
+ 7.0%
32.34
33.84
35.24
35.75
36.48
OLMo-3-7B
19.92
21.73 ± 0.61
23.35 ± 0.44
23.50 ± 0.56
24.94 ± 0.48
+ 6.8%
18.05
18.66
20.05
20.34
21.40
Gemma-4-E4B-it
43.77
45.11 ± 0.50
46.66 ± 0.59
46.53 ± 0.42
47.21 ± 0.44
+ 1.2%
38.88
39.01
40.37
40.73
40.93
Table 1: Ten-round self-evolution across four backbones (%). Each average weights its benchmarks equally, and ± is the sample SD of Math Avg. over three seeds. “vs. Agent0” is the relative Math Avg. gain of EvoHeir (Full) over Agent0. Figure 5 and Appendix E give per-benchmark results.
Figure 5: Per-benchmark gain of EvoHeir (Full) over Agent0 at round 10 (pp, three-seed means). 45 of 48 cells are positive, and the three negative cells are 30-problem AIME sets.
Figure 6: Ten-round Math Avg. trajectories, as change from round 0 (pp). Lines and bands show the mean and ±1 SD over three seeds. The bar at round 10 marks EvoHeir ’s lead over Agent0, and each panel has its own y scale.
Figure 7: Experience value under fixed-recipient interventions (pp, 1,024 items each, before confirmation). (a) Donor gain against absorption gain for 16 donors. The shaded quadrant holds weaker donors whose experience helps, and + marks their mean. (b) Mean new (up) and lost (down) successes from the same 64 references, with black bars at the mean net gain and thin lines pairing the two methods at each recipient. (c) Absorption gain as history eligibility widens at a fixed 64-reference budget, with thin lines for the four recipients. (d) Paired contribution of non-inherited over same-age inherited history at 64 records, with pointwise 95% hierarchical bootstrap intervals over three seed groups. Age zero is an identical-material control.
Figure 8: Qualitative analysis on Qwen3-4B-Base at round 5. (a) A round-4 archived reference from a proposal that was not inherited recovers x=16 after the recipient drops the cross term; Recovery reproduces it without the reference. (b) Under the same 64-reference budget, Recovery keeps the remainder that Cache SFT loses. The fresh-probe gain is −0.31 pp, so HeirCommit retains the root.
Variant
Math Avg. ↑
General Avg. ↑
EvoHeir (Full)
31.82 ± 0.55
30.52 ± 0.35
Same-Lineage
30.77 ± 0.57
29.99 ± 0.35
Random Experience
30.54 ± 0.44
29.87 ± 0.43
Current-Round Only
30.85 ± 0.69
30.04 ± 0.35
NoConfirm
29.80 ± 1.13
29.33 ± 0.65
Table 2: Component ablations (%, mean ± SD over three seeds).
Figure 9: Equal-budget comparison and sensitivity on Qwen3-4B-Base . (a–c) Math Avg. after ten rounds, mean ± SD over three training seeds. (a) Four methods at equal accounting budgets, relative to Full’s default of 123.7 units. (b,c) Candidate count and confirmation-pool size at budget 1.0, where native training absorbs the remaining budget. Open rings mark the defaults, which reuse the main runs. (d) Net absorption gain of a fixed recipient on 2,400 problems as the preservation weight changes, over three absorption-order seeds. Accounting units are not measured GPU-hours or FLOPs.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Host
Persistence rule
Coverage
Useful retained
Harmful commits
Mean gain ( 10−3 NLL)
Δ Native ( 10−3 NLL)
Skill-SP
Native commit-all
100.00%
100.00%
57.81%
−3.47
0.00
Matched random
39.06%
39.06%
57.81%
−1.36
+2.11
HeirCommit
39.06%
77.78%
16.00%
+2.49
+5.96
R-Zero
Native commit-all
100.00%
100.00%
76.56%
−23.17
0.00
Matched random
23.44%
23.44%
76.56%
−5.43
+17.74
HeirCommit
23.44%
86.67%
13.33%
+10.19
+33.36
Appendix
Table 4: Complete HeirCommit audit across native update sources. Matched random uses HeirCommit ’s coverage within each host. Gain and Δ Native are in 10−3 NLL units; higher is better.
Figure 10: Complete self-evolution results (%). Every benchmark, backbone, and method after ten rounds. Numbers are three-seed means, and color shows the change from each backbone’s starting checkpoint (Base, gray rows).
Figure 11: Supplementary interventions. (a) Objective controls on a fixed recipient and 2,400 problems, with mean ± SD net gain over three absorption-order seeds. (b) New and lost successes as the preservation weight changes. (c) Absorption gain with non-inherited or same-age inherited history at 64 records. (d) Net gain of Recovery and Cache SFT at each recipient checkpoint.
Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: https://github.com/YujunZhou/EVOL-RL.
Yujun Zhou, Zhenwen Liang, Haolin Liu +7
Tencent AI Lab · University of Notre Dame · University of Virginia
Self-evolution lets large language models (LLMs) improve iteratively using their own generated data, but often suffers from self-evolution degeneration: performance improves, plateaus, then declines. Existing methods address this issue at the component level, targeting either the Questioner or the Solver, and overlook that self-evolution is a tightly coupled system. We propose a holistic framework based on learnable information gain, which measures how much novel, parameterizable information a round provides relative to the previous round. Theoretically, this gain equals the Kullback-Leibler divergence between the two rounds' data distributions plus their entropy change. Practically, it is estimated by fitting a small language model to the previous round and scoring new data via negative log-likelihood. Based on this diagnostic, we propose ATRI (Adaptive Training Regulation via Information-gain), which reweights samples within a round and halts training across rounds when information gain remains low. Experiments on popular datasets demonstrate the superiority of our proposal.
Chenxu Wang, Chaozhuo Li, Xinze Shi +6
Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence · Central South University +2
We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.
Xiaoyin Chen, Canwen Xu, Yite Wang +3
Mila – Quebec AI Institute · University of Montreal · Snowflake