Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. Existing continual learning approaches rely on external supervision, such as labeled data, demonstrations, or reward signals, which can be costly or unavailable in practice. Test-time adaptation removes this dependence by learning from self-generated supervision, but existing methods derive that supervision from the single policy being updated, so once the policy drifts, its own learning signal drifts with it, leading to policy collapse, degraded reasoning, and catastrophic forgetting. In this paper, we propose Evolution Strategy Reinforcement Learning (ESRL), a continual test-time learning framework that draws its self-generated supervision from a population of policies instead of the single policy being trained. ESRL perturbs an anchor policy in parameter space to form a population of nearby policies, samples one trajectory from each, and aggregates their answers into label-free group-relative learning signals for the central student. The anchor is held fixed within each block and refreshed from the student at block boundaries, so the population stays centered on a stable yet current policy and continues to supply diverse and reliable supervision as the student moves. In single-task test-time adaptation across mathematical reasoning, general knowledge, and medical domains, ESRL improves average accuracy by up to 31.7% relative to the base model and 16.6% relative to the strongest test-time adaptation baseline. Notably, ESRL substantially reduces catastrophic forgetting and achieves the strongest final performance in continual learning.
Figures & tables
Figure 1: ESRL comparisons, continual learning performance, and overall performance summary. Top left: Comparison with existing methods for continual test-time learning across three key dimensions. Top right: Continual learning performance after training on MATH-500. Bottom: ESRL consistently improves the base models across domain-specific benchmarks and in-domain transfer tasks. All results are reported as accuracy (%).
Figure 2: Overview of ESRL . ESRL first constructs a population of perturbed policies around a shared anchor and generates one rollout from each policy. Majority voting provides label-free rewards, which are converted into group-relative advantages. The student is then updated with an advantage-weighted log-likelihood objective. The anchor remains fixed within each block of B prompts and is refreshed from the latest student at the next block boundary.
Model
Method
Math
General Knowledge
Medical
Avg.
MATH-500
AMC23
AIME25
MMLU
ARC-C
GPQA-D
MedQA
MedMCQA
PubMedQA
Qwen3-0.6B
Base model
5.0
6.0
0.0
44.0
65.5
26.3
23.5
37.0
45.5
28.1
TTRL
10.0
13.3
3.3
44.5
70.0
25.8
32.5
33.5
52.5
31.7
ES
6.0
6.0
0.0
47.0
68.5
27.3
28.0
37.5
45.5
29.5
SFT
0.0
0.0
0.0
43.0
60.5
28.3
22.5
23.0
60.0
26.4
ECHO
17.0
9.9
3.3
46.0
69.5
23.7
18.5
37.5
45.5
30.1
Table 1: Pass@1 results for source-benchmark evaluation and in-domain generalization across different backbone models.
Figure 3: Sequential training of Qwen3-0.6B on MATH-500, MMLU, and MedQA. Pass@1 is reported across 3000 training steps for (a) MATH-500, (b) MMLU, and (c) MedQA.
Figure 4: Training dynamics during continual learning. The first row shows the mean response length, the second row shows Pass@32, the third row shows Majority@32, where ESRL reports population-level majority voting, and the fourth row shows answer entropy. The three columns correspond to MATH-500, MMLU, and MedQA, respectively. All metrics are evaluated every 250 training steps. Detailed definitions of these metrics are provided in Appendix A.2
Figure 5: Comparison of Pass@1 on MATH-500 under different batch sizes.
B
Math
General Knowledge
Medical
Avg.
MATH-500
AMC23
AIME25
MMLU
ARC-C
GPQA-D
MedQA
MedMCQA
PubMedQA
2
23.0
16.9
6.7
48.0
67.0
30.3
31.0
41.0
48.0
34.7
10
27.0
19.3
6.7
49.5
70.5
30.8
34.0
38.0
57.0
37.0
20
25.0
15.3
6.7
49.5
65.5
32.3
30.5
35.0
54.5
34.9
50
17.0
22.9
0.0
46.0
63.5
27.3
34.0
39.0
47.5
33.0
Table 2: Pass@1 under different block sizes B using Qwen3-0.6B.
Group Size N
Math
Avg.
MATH-500
AMC23
AIME25
16
74.0
51.8
16.7
47.5
32
72.0
50.6
16.7
46.4
64
75.0
53.0
16.7
48.2
Table 3: Pass@1 under different rollout group sizes N using Qwen3-4B.
Loss Function
Math
General Knowledge
Medical
Avg.
MATH-500
AMC23
AIME25
MMLU
ARC-C
GPQA-D
MedQA
MedMCQA
PubMedQA
GRPO-Style Clipped Loss
67.0
44.6
16.7
65.5
87.5
29.3
49.0
54.0
61.5
52.8
Advantage-Weighted Loss
74.0
51.8
16.7
68.5
89.0
33.3
53.0
56.0
66.0
56.5
Table 4: Pass@1 under different optimization objectives using Qwen3-4B.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Math
General Knowledge
Medical
Method
MATH-500
AMC23
AIME25
MMLU
ARC-C
GPQA-D
MedQA
MedMCQA
PubMedQA
Avg.
Base model
45.0
33.0
6.7
73.0
85.0
63.9
81.0
83.0
90.7
62.4
TTRL
31.0
36.1
6.7
75.0
92.0
54.0
90.0
73.0
90.0
60.9
EVOL-RL
4.0
41.0
0.0
86.0
92.0
59.1
86.0
68.0
84.0
57.8
ECHO
43.0
37.3
3.6
62.0
79.0
34.8
88.0
72.0
87.0
56.3
ESRL
45.0
38.6
13.3
82.0
92.0
57.4
88.0
88.0
90.0
66.0
Appendix
Table 5: Pass@32 results for source-benchmark evaluation and in-domain generalization using Qwen3-0.6B.
σ
General Knowledge
Avg.
MMLU
ARC-C
GPQA-D
0.0
45.5
68.5
24.7
46.2
0.0005
47.0
69.5
27.8
48.1
0.001
49.5
70.5
30.8
50.3
0.005
46.5
70.5
31.3
49.4
0.01
48.5
70.5
27.3
48.8
Appendix
Table 6: Pass@1 results under different perturbation scale σ using Qwen3-0.6B.
Method
Runtime (min)
Prompts/h
Peak Mem. (GiB)
GPU Util.
Energy (Wh)
TTRL
78.85
304.38
100.94
67.6%
303.3
ESRL
89.90
266.96
100.78
76.2%
376.0
Appendix
Table 7: Runtime and computational cost comparison between ESRL and TTRL using Qwen3-0.6B on the Math domain. Results are measured over three independent runs.
Method
General Knowledge
Avg.
MMLU
ARC-C
GPQA-D
Base model
44.0±0.0
65.5±0.0
26.3±0.3
45.3
TTRL
44.5±1.5
70.0±1.6
25.8±2.8
46.8
ECHO
46.0±1.0
69.5±1.0
23.7±1.1
46.4
Evol-RL
46.5±2.3
69.5±0.3
27.3±1.3
47.8
ESRL
49.5±1.3
70.5±0.3
30.8±1.3
50.3
Appendix
Table 8: Pass@1 performance of different methods on the General Knowledge domain, reported as mean ± standard deviation over three seeds using Qwen3-0.6B.
Setting
Math
Avg.
MATH-500
AMC23
AIME25
Base Model
5.0
6.0
0.0
3.7
Batch Size = 128
6.0
6.0
0.0
4.0
Appendix
Table 9: Failed attempts with a large batch size using Qwen3-0.6B.
We introduce EvoLib, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision. Instead of adapting model parameters, our approach maintains a shared library of knowledge abstractions, including modular skills and reflective insights, automatically extracted from the model's own inference trajectories. To support continual improvement, we introduce a principled weighting and consolidation mechanism that jointly optimizes for immediate utility and long-term value. This allows simple, instance-specific abstractions to evolve into more general and reusable ones over time. Across challenging benchmarks in mathematical reasoning, code generation, and multi-turn agentic environments, EvoLib improves substantially over the top test-time scaling and learning methods without ground-truth feedback.
Test-Time Training offers a promising way to improve the reasoning ability of large language models (LLMs) by adapting the model using only the test questions. However, existing methods struggle with difficult reasoning problems for two reasons: raw test questions are often too difficult to yield high-quality pseudo-labels, and the limited size of test sets makes continuous online updates prone to instability. To address these limitations, we propose TTCS, a co-evolving test-time training framework. Specifically, TTCS initializes two policies from the same pretrained model: a question synthesizer and a reasoning solver. These policies evolve through iterative optimization: the synthesizer generates progressively challenging question variants conditioned on the test questions, creating a structured curriculum tailored to the solver's current capability, while the solver updates itself using self-consistency rewards computed from multiple sampled responses on both original test and synthetic questions. Crucially, the solver's feedback guides the synthesizer to generate questions aligned with the model's current capability, and the generated question variants in turn stabilize the solver's test-time training. Experiments show that TTCS consistently strengthens the reasoning ability on challenging mathematical benchmarks and transfers to general-domain tasks across different LLM backbones, highlighting a scalable path towards dynamically constructing test-time curricula for self-evolving. Our code and implementation details are available at https://github.com/XMUDeepLIT/TTCS.
Chengyi Yang, Zhishang Xiang, Yunbo Tang +5
Xiamen University · Washington University in St. Louis · Renmin University of China
Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient but prone to severe forgetting, while Reinforcement Learning (RL) preserves prior capabilities but converges slowly without expert guidance. To bridge this gap, we introduce Trigger-based Supervised Reinforcement Learning (TSRL), a framework that integrates the strengths of both paradigms through an adaptive, state-dependent expert supervision mechanism. TSRL employs RL as the optimization backbone, activating SFT selectively on "hard states" where the policy fails. Theoretically, we show that restricting supervision to failure states derives an upper bound on the cumulative parameter updates, thereby reducing forgetting compared to full SFT. Empirically, TSRL achieves consistent gains across three agent benchmarks (ALFWorld, WebShop, and Deep Search) at both 7B and 32B scales. In continual learning scenarios, our method retains up to 9.6 points more capability on prior tasks than SFT->RL baselines, while matching or exceeding target-task performance.
Yuqian Fu, Kaiwen Jiang, Zijun Song +3
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · University of Science and Technology of China