Continual Test-Time Learning with Evolution Strategies

Guangya Hao; Yiting Duan; Longyan Tan; Yunbo Long; Hao Wu; Ge Zheng; Alexandra Brintrup; Zhuokai Zhao

cs.LGOct 16, 2026

Continual Test-Time Learning with Evolution Strategies

Authors: Guangya Hao, Yiting Duan, Longyan Tan, Yunbo Long, Hao Wu, Ge Zheng, Alexandra Brintrup, Zhuokai Zhao

Organizations: University of Cambridge · Western Sydney University · The Alan Turing Institute · University of Chicago

Abstract

Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. Existing continual learning approaches rely on external supervision, such as labeled data, demonstrations, or reward signals, which can be costly or unavailable in practice. Test-time adaptation removes this dependence by learning from self-generated supervision, but existing methods derive that supervision from the single policy being updated, so once the policy drifts, its own learning signal drifts with it, leading to policy collapse, degraded reasoning, and catastrophic forgetting. In this paper, we propose Evolution Strategy Reinforcement Learning (ESRL), a continual test-time learning framework that draws its self-generated supervision from a population of policies instead of the single policy being trained. ESRL perturbs an anchor policy in parameter space to form a population of nearby policies, samples one trajectory from each, and aggregates their answers into label-free group-relative learning signals for the central student. The anchor is held fixed within each block and refreshed from the student at block boundaries, so the population stays centered on a stable yet current policy and continues to supply diverse and reliable supervision as the student moves. In single-task test-time adaptation across mathematical reasoning, general knowledge, and medical domains, ESRL improves average accuracy by up to 31.7% relative to the base model and 16.6% relative to the strongest test-time adaptation baseline. Notably, ESRL substantially reduces catastrophic forgetting and achieves the strongest final performance in continual learning.

Figures & tables

Figure 1: ESRL comparisons, continual learning performance, and overall performance summary. Top left: Comparison with existing methods for continual test-time learning across three key dimensions. Top right: Continual learning performance after training on MATH-500. Bottom: ESRL consistently improves the base models across domain-specific benchmarks and in-domain transfer tasks. All results are reported as accuracy (%).

Figure 2: Overview of ESRL . ESRL first constructs a population of perturbed policies around a shared anchor and generates one rollout from each policy. Majority voting provides label-free rewards, which are converted into group-relative advantages. The student is then updated with an advantage-weighted log-likelihood objective. The anchor remains fixed within each block of

B

prompts and is refreshed from the latest student at the next block boundary.

Model	Method	Math			General Knowledge			Medical			Avg.
Model	Method	MATH-500	AMC23	AIME25	MMLU	ARC-C	GPQA-D	MedQA	MedMCQA	PubMedQA	Avg.
Qwen3-0.6B	Base model	5.0	6.0	0.0	44.0	65.5	26.3	23.5	37.0	45.5	28.1
	TTRL	10.0	13.3	3.3	44.5	70.0	25.8	32.5	33.5	52.5	31.7
	ES	6.0	6.0	0.0	47.0	68.5	27.3	28.0	37.5	45.5	29.5
	SFT	0.0	0.0	0.0	43.0	60.5	28.3	22.5	23.0	60.0	26.4
	ECHO	17.0	9.9	3.3	46.0	69.5	23.7	18.5	37.5	45.5	30.1

Table 1: Pass@1 results for source-benchmark evaluation and in-domain generalization across different backbone models.

Figure 3: Sequential training of Qwen3-0.6B on MATH-500, MMLU, and MedQA. Pass@1 is reported across 3000 training steps for (a) MATH-500, (b) MMLU, and (c) MedQA.

Figure 4: Training dynamics during continual learning. The first row shows the mean response length, the second row shows Pass@32, the third row shows Majority@32, where ESRL reports population-level majority voting, and the fourth row shows answer entropy. The three columns correspond to MATH-500, MMLU, and MedQA, respectively. All metrics are evaluated every 250 training steps. Detailed definitions of these metrics are provided in Appendix A.2

Figure 5: Comparison of Pass@1 on MATH-500 under different batch sizes.

$B$	Math			General Knowledge			Medical			Avg.
	MATH-500	AMC23	AIME25	MMLU	ARC-C	GPQA-D	MedQA	MedMCQA	PubMedQA
2	23.0	16.9	6.7	48.0	67.0	30.3	31.0	41.0	48.0	34.7
10	27.0	19.3	6.7	49.5	70.5	30.8	34.0	38.0	57.0	37.0
20	25.0	15.3	6.7	49.5	65.5	32.3	30.5	35.0	54.5	34.9
50	17.0	22.9	0.0	46.0	63.5	27.3	34.0	39.0	47.5	33.0

Table 2: Pass@1 under different block sizes

B

using Qwen3-0.6B.

Group Size $N$	Math			Avg.
Group Size $N$	MATH-500	AMC23	AIME25	Avg.
16	74.0	51.8	16.7	47.5
32	72.0	50.6	16.7	46.4
64	75.0	53.0	16.7	48.2

Table 3: Pass@1 under different rollout group sizes

N

using Qwen3-4B.

Loss Function	Math			General Knowledge			Medical			Avg.
	MATH-500	AMC23	AIME25	MMLU	ARC-C	GPQA-D	MedQA	MedMCQA	PubMedQA
GRPO-Style Clipped Loss	67.0	44.6	16.7	65.5	87.5	29.3	49.0	54.0	61.5	52.8
Advantage-Weighted Loss	74.0	51.8	16.7	68.5	89.0	33.3	53.0	56.0	66.0	56.5

Table 4: Pass@1 under different optimization objectives using Qwen3-4B.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MATH-500	AMC23	AIME25	MMLU	ARC-C	GPQA-D	MedQA	MedMCQA	PubMedQA	Avg.
	Math			General Knowledge			Medical
Base model	45.0	33.0	6.7	73.0	85.0	63.9	81.0	83.0	90.7	62.4
TTRL	31.0	36.1	6.7	75.0	92.0	54.0	90.0	73.0	90.0	60.9
EVOL-RL	4.0	41.0	0.0	86.0	92.0	59.1	86.0	68.0	84.0	57.8
ECHO	43.0	37.3	3.6	62.0	79.0	34.8	88.0	72.0	87.0	56.3
ESRL	45.0	38.6	13.3	82.0	92.0	57.4	88.0	88.0	90.0	66.0

Appendix

Table 5: Pass@32 results for source-benchmark evaluation and in-domain generalization using Qwen3-0.6B.

$\sigma$	General Knowledge			Avg.
$\sigma$	MMLU	ARC-C	GPQA-D	Avg.
0.0	45.5	68.5	24.7	46.2
0.0005	47.0	69.5	27.8	48.1
0.001	49.5	70.5	30.8	50.3
0.005	46.5	70.5	31.3	49.4
0.01	48.5	70.5	27.3	48.8

Appendix

Table 6: Pass@1 results under different perturbation scale

\sigma

using Qwen3-0.6B.

Method	Runtime (min)	Prompts/h	Peak Mem. (GiB)	GPU Util.	Energy (Wh)
TTRL	78.85	304.38	100.94	67.6%	303.3
ESRL	89.90	266.96	100.78	76.2%	376.0

Appendix

Table 7: Runtime and computational cost comparison between ESRL and TTRL using Qwen3-0.6B on the Math domain. Results are measured over three independent runs.

Method	General Knowledge			Avg.
Method	MMLU	ARC-C	GPQA-D	Avg.
Base model	$44.0_{\scriptscriptstyle\pm 0.0}$	$65.5_{\scriptscriptstyle\pm 0.0}$	$26.3_{\scriptscriptstyle\pm 0.3}$	$45.3$
TTRL	$44.5_{\scriptscriptstyle\pm 1.5}$	$70.0_{\scriptscriptstyle\pm 1.6}$	$25.8_{\scriptscriptstyle\pm 2.8}$	$46.8$
ECHO	$46.0_{\scriptscriptstyle\pm 1.0}$	$69.5_{\scriptscriptstyle\pm 1.0}$	$23.7_{\scriptscriptstyle\pm 1.1}$	$46.4$
Evol-RL	$46.5_{\scriptscriptstyle\pm 2.3}$	$69.5_{\scriptscriptstyle\pm 0.3}$	$27.3_{\scriptscriptstyle\pm 1.3}$	$\underline{47.8}$
ESRL	$49.5_{\scriptscriptstyle\pm 1.3}$	$70.5_{\scriptscriptstyle\pm 0.3}$	$30.8_{\scriptscriptstyle\pm 1.3}$	$\mathbf{50.3}$

Appendix

Table 8: Pass@1 performance of different methods on the General Knowledge domain, reported as mean

\pm

standard deviation over three seeds using Qwen3-0.6B.

Setting	Math			Avg.
Setting	MATH-500	AMC23	AIME25	Avg.
Base Model	5.0	6.0	0.0	3.7
Batch Size = 128	6.0	6.0	0.0	4.0

Appendix

Table 9: Failed attempts with a large batch size using Qwen3-0.6B.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 223 papers

Test-Time Learning with an Evolving Library
May 14, 2026Weijia Xu, Alessandro Sordoni, Chandan Singh +4Memory-Augmented Language Models Continual Test-Time Adaptation
Microsoft Research
TTCS: Test-Time Curriculum Synthesis for Self-Evolving
Jan 30, 2026Chengyi Yang, Zhishang Xiang, Yunbo Tang +5Test-Time Training Language Model Self-Improvement
Xiamen University · Washington University in St. Louis · Renmin University of China
TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training
Oct 13, 2026Yuqian Fu, Kaiwen Jiang, Zijun Song +3Self-Supervised RL Continual Learning for LLM Agents
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · University of Science and Technology of China

May 14, 2026J/K move · Enter open · S save

Test-Time Learning with an Evolving Library

Weijia Xu, Alessandro Sordoni, Chandan Singh, Zelalem Gero+3

Microsoft Research

Memory-Augmented Language Models Continual Test-Time Adaptation Language Model Self-Improvement

Open paper AskPDF arXiv

We introduce EvoLib, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision. Instead of adapting model parameters, our approach maintains a shared library of knowledge abstractions, including modular skills and reflective insights, automatically extracted from the model's own inference trajectories. To support continual improvement, we introduce a principled weighting and consolidation mechanism that jointly optimizes for immediate utility and long-term value. This allows simple, instance-specific abstractions to evolve into more general and reusable ones over time. Across challenging benchmarks in mathematical reasoning, code generation, and multi-turn agentic environments, EvoLib improves substantially over the top test-time scaling and learning methods without ground-truth feedback.

Abstract

Continual Learning for LLMs

Explore similar work

Test-Time Learning with an Evolving Library

TTCS: Test-Time Curriculum Synthesis for Self-Evolving

TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training