When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain
Authors: Chenxu Wang, Chaozhuo Li, Xinze Shi, Songyang Liu, Kyrie You Wu, Ziluowen Luo, Shun Zhang, Chenxi Li, +1 more
Organizations: Beijing University of Posts and Telecommunications · Beijing Academy of Artificial Intelligence · Central South University · Graduate School of China Academy of Engineering Physics · The Chinese University of Hong Kong, Shenzhen
Self-evolution lets large language models (LLMs) improve iteratively using their own generated data, but often suffers from self-evolution degeneration: performance improves, plateaus, then declines. Existing methods address this issue at the component level, targeting either the Questioner or the Solver, and overlook that self-evolution is a tightly coupled system. We propose a holistic framework based on learnable information gain, which measures how much novel, parameterizable information a round provides relative to the previous round. Theoretically, this gain equals the Kullback-Leibler divergence between the two rounds' data distributions plus their entropy change. Practically, it is estimated by fitting a small language model to the previous round and scoring new data via negative log-likelihood. Based on this diagnostic, we propose ATRI (Adaptive Training Regulation via Information-gain), which reweights samples within a round and halts training across rounds when information gain remains low. Experiments on popular datasets demonstrate the superiority of our proposal.
Figures & tables
Figure 1: Previous self-evolution loops monitor each component separately and drift into decline (left). Our ATRI model tracks system-level diagnostic (right).
Figure 2: Analysis on Qwen3-4B-Base with MATH. (a) Partial signals cannot reflect performance degradation. (b) Lower learnable information gain can signal later performance degradation.
Figure 3: The overview of the proposed ATRI framework.
Method
Math AVG
Overall AVG
Mathematical Reasoning Benchmarks
General Reasoning
GSM8K
MATH
AMC
Minerva
Olymp.
AIME24
AIME25
MMLU-Pro
SuperGPQA
BBEH
Qwen3-4B-Base
Base Model
42.58
36.34
87.86
67.96
45.59
38.10
41.16
11.03
6.35
37.17
20.84
7.33
STaR ( Zelikman et al., 2022 )
43.96
37.74
88.36
71.20
47.96
39.65
41.92
11.47
7.16
39.54
22.55
7.60
SPIN ( Chen et al., 2024 )
45.26
39.45
89.35
73.60
50.02
41.03
42.53
11.97
8.35
44.60
24.82
8.25
AZR ( Zhao et al., 2025 )
46.51
41.38
89.49
76.31
52.47
42.20
42.50
12.23
10.36
52.66
27.28
8.34
Table 1: Main results: ATRI versus existing self-evolution methods on Qwen3-4B-Base and Qwen3-8B-Base across seven mathematical reasoning benchmarks and three general reasoning benchmarks. Bold : best per column. Underline : second best.
Figure 5
Figure 5: Three-phase lifecycle across self-evolution methods (Qwen3-4B-Base / MATH). Ct falls below the alarm threshold τ one to four rounds before the acc peak and then fluctuates around zero.
Figure 7Table 8
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Task
Base model
Peak round
Alarm round
Peak acc.
Round-15 acc.
Reference
MATH
Qwen3-4B
4
3
42.99
29.22
New model family
MATH
Llama-3.1-8B
5
4
45.16
37.84
New task type
MBPP
Qwen3-4B
6
5
59.10
51.70
Appendix
Table 6: Lifecycle and alarm behaviour across base models and tasks. Accuracy is held-out performance at the listed round.
Method
Round 15
Validation stopping
Oracle checkpointing
Passive Ct stopping
Vanilla
28.93 (15)
42.99 (4)
42.99 (4)
42.99 (4)
STaR
36.37 (15)
45.18 (8)
45.43 (9)
44.87 (6)
SPIN
37.41 (15)
47.55 (7)
47.55 (7)
46.61 (6)
AZR
41.58 (15)
48.11 (8)
48.27 (9)
47.22 (7)
R-Zero
44.32 (15)
48.62 (10)
48.74 (9)
47.86 (8)
R-Diverse
45.47 (15)
49.58 (9)
49.91 (8)
48.36 (7)
Appendix
Table 7: MATH accuracy of the checkpoint returned by four stopping rules on the six baseline runs of Section 4.7 . The selected round is shown in parentheses.
Samples per round
Trials matching full-data round
Maximum shift
100%
94.2%
1 round
75%
91.3%
1 round
50%
85.8%
1 round
25%
77.6%
2 rounds
Appendix
Table 8: Bootstrap stability of the stopping round at four per-round data sizes. Each row summarises 1,000 trials.
Setting
Correlation
Alarm shift
Stopping shift
Same stopping round
Original
1.000
0
0
100%
Length matched
0.982±0.007
0
0
95.6%
Format standardised
0.969
0
0
100%
Both controls
0.958±0.010
0
0
93.1%
Appendix
Table 9: Stopping behaviour under surface-level controls. Correlation is Spearman against the original Ct trajectory. Ranges are standard deviations over the 100 length-matching repetitions.
Setting
Proxy data
Alarm round
Peak round
Rounds before peak
MATH
Previous MATH round
3
4
1
MBPP
Previous MBPP round
5
6
1
MATH → MBPP
Final MATH round
8
6
−2
MATH → MBPP
First MBPP round
5
6
1
Appendix
Table 10: Alarm timing under task shifts. The last column counts the rounds by which the alarm precedes the performance peak.
Base model
Main loop
Proxy +Ct
Relative overhead
Qwen3-4B-Base
8.2
0.35
4.1%
Qwen3-8B-Base
15.5
0.35
2.3%
Qwen3-14B-Base
26.8
0.35
1.3%
Appendix
Table 11: Measured per-round cost in GPU-hours at three base-model scales.
Self-evolving large language model (LLM) systems repeatedly propose, evaluate, and incorporate updates to prompts, skills, or other persistent artifacts. Despite their growing effectiveness, these systems typically operate under a predetermined iteration or compute budget, without a principled criterion to determine when further evolution is no longer worthwhile. This can lead to two undesirable consequences: unnecessary computation after performance has saturated and the risk of returning late updates that overfit or exploit the evaluation signal. These issues motivate us to study two fundamental questions: when should a self-evolving system stop, and what should it output once it stops? We address the first by formulating an online sequential testing problem and constructing an anytime-valid restart detector using the per-item paired evaluation outcomes already produced by self-evolving LLM systems. We address the second by formulating a change-point estimation problem and using the estimated transition to select an earlier artifact for output. The resulting procedure is plug-and-play and requires no modification of the underlying self-evolving algorithms. Across two self-evolving frameworks, three LLM model families, and five benchmarks, our method substantially reduces computation costs while maintaining comparable unseen-test performance. For example, on SearchQA with SkillOpt and DeepSeek V4 Flash, our method stops at round 4 rather than the full budget of 40, reducing token usage by 91.6% while achieving 82.43% unseen-test accuracy versus 82.00% under the full-budget run.
Enoch Yin, Bin Liu, Zhengling Qi
Department of Statistics, The George Washington University · Department of Statistics and Data Science, School of Management, Fudan University · Department of Decision Sciences, School of Business, The George Washington University
We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.
Xiaoyin Chen, Canwen Xu, Yite Wang +3
Mila – Quebec AI Institute · University of Montreal · Snowflake
Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the self-evolution algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework: single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that self-evolution consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non-trivial gap to oracle supervision. We find that multi-turn critic-revision with large models can reach strong self-evolution performance, with Gemma 12B nearly matching oracle-supervised training. Beyond Knights and Knaves, we also evaluate self-evolution on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop self-evolution can help and show how internally generated supervision remains insufficient under this minimal formulation.
Zhenting Qi, Susanna Maria Baby, Stefanie Anna Baby +5