ACLArena: Agent Continue Learning in Multi-stage Post-training

Haixin Wang; Xiaoxuan Wang; Junkai Zhang; Han Zhang; Renliang Sun; Alexander K Taylor; Yidan Shi; Haoran Deng

cs.AISep 21, 2026

ACLArena: Agent Continue Learning in Multi-stage Post-training

Authors: Haixin Wang, Xiaoxuan Wang, Junkai Zhang, Han Zhang, Renliang Sun, Alexander K Taylor, Yidan Shi, Haoran Deng, +4 more

Organizations: University of California, Los Angeles · University of California, Santa Cruz

Abstract

Building general-purpose agents for industrial deployment requires integrating multiple capabilities, each typically acquired at a distinct stage of training. Yet there is currently no well-established recipe for Agent Continual Learning (ACL), with little understanding of the trade-offs among existing integration paradigms. To address this gap, we introduce ACLArena, a framework for comprehensively studying, analyzing, and evaluating ACL. We first build a sequential training pipeline and conduct an in-depth analysis that explains the mechanisms of forgetting and generalization from two complementary perspectives, the model level and the token level. Guided by these analyses, we systematically compare multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging to assess their ability to recover previously learned capabilities while preserving newly acquired ones. Through extensive experiments, we develop a detailed understanding of how capabilities transfer across stages. Finally, we propose a new ACL recipe that combines offline replay over high-quality trajectories with a routed network of multiple LoRA experts each specialized via RL, substantially improving the agent's ability to learn across multiple domains. Comprehensive experiments on four reasoning and agentic tasks, evaluated under both in-domain and out-of-domain settings, demonstrate the value of our analysis and the effectiveness of our approach.

Figures & tables

Figure 1: Overview of the four mainstream paradigms for capability consolidation: sequential training, multi-teacher mixed on-policy distillation (MMOPD), self-distilled fine-tuning (SDFT), and model merging (MM).

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Per-stage performance (Math $\rightarrow$ Search $\rightarrow$ E-commerce $\rightarrow$ Instruction Following)
Base	$6.46$	$13.3_{\scriptscriptstyle\pm 0.2}$	$3.5_{\scriptscriptstyle\pm 0.4}$	$46.0_{\scriptscriptstyle\pm 0.9}$	$30.8_{\scriptscriptstyle\pm 1.6}$	$65.1_{\scriptscriptstyle\pm 0.3}$	$22.3_{\scriptscriptstyle\pm 0.0}$	$11.1_{\scriptscriptstyle\pm 0.6}$	$17.5_{\scriptscriptstyle\pm 1.0}$	$5.0_{\scriptscriptstyle\pm 2.5}$	$14.4_{\scriptscriptstyle\pm 0.2}$
Seq- Math	${\color[rgb]{1,0,0}\mathbf{25.83}}$	$22.0_{\scriptscriptstyle\pm 0.2}$	$2.3_{\scriptscriptstyle\pm 0.5}$	$43.3_{\scriptscriptstyle\pm 0.6}$	${\color[rgb]{1,0,0}\mathbf{40.2_{\scriptscriptstyle\pm 2.5}}}$	$79.0_{\scriptscriptstyle\pm 0.2}$	$38.7_{\scriptscriptstyle\pm 0.2}$	$22.8_{\scriptscriptstyle\pm 0.5}$	$13.1_{\scriptscriptstyle\pm 2.1}$	$3.3_{\scriptscriptstyle\pm 3.8}$	$17.1_{\scriptscriptstyle\pm 0.2}$
Seq- Search	$23.33$	${\color[rgb]{1,0,0}\mathbf{45.2_{\scriptscriptstyle\pm 0.4}}}$	$2.0_{\scriptscriptstyle\pm 0.3}$	$39.8_{\scriptscriptstyle\pm 0.2}$	$37.2_{\scriptscriptstyle\pm 0.7}$	$79.2_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{52.9_{\scriptscriptstyle\pm 0.1}}}$	${\color[rgb]{1,0,0}\mathbf{37.4_{\scriptscriptstyle\pm 0.8}}}$	$9.1_{\scriptscriptstyle\pm 3.9}$	$30.8_{\scriptscriptstyle\pm 1.4}$	$15.8_{\scriptscriptstyle\pm 0.2}$

Table 1: Main sequential training results. Single-hop Search reports the average performance on PopQA and TriviaQA. Multi-hop Search reports the average performance on 2Wiki, Bamboogle, HotpotQA, and MuSiQue. Dashed boxes indicate the training stage corresponding to each task, while changes in the subsequent rows reflect generalization and forgetting. We denote the checkpoints after each training stage as Seq-Math , Seq-Search , Seq-E-commerce , and Seq-IF , respectively; Seq-IF is the final sequential checkpoint and is referred to as Seq-Final in subsequent experiments. The best score in each column is highlighted in red . All results are reported as mean

\pm

standard deviation over three independent runs.

Panel of Figure 2 — Figure 2: Diagnosing sequential training. Rows (top to bottom) follow the training order; columns correspond to the evaluation of each domain, and marginal bars show mean performance gap $\Delta$ per metric and per stage. The projection in (c) reveals both that the tasks occupy distinct optimization directions and that it progressively fails to recover them. (d) provides token-level analysis of original rollout with different entropy.

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Per-Task Oracle Training
Expert ${}_{\text{Math}}$	${\color[rgb]{1,0,0}\mathbf{25.83}}$	$22.0_{\scriptscriptstyle\pm 0.2}$	$2.3_{\scriptscriptstyle\pm 0.5}$	$43.3_{\scriptscriptstyle\pm 0.6}$	$40.2_{\scriptscriptstyle\pm 2.5}$	$79.0_{\scriptscriptstyle\pm 0.2}$	$38.7_{\scriptscriptstyle\pm 0.2}$	$22.8_{\scriptscriptstyle\pm 0.5}$	$13.1_{\scriptscriptstyle\pm 2.1}$	$3.3_{\scriptscriptstyle\pm 3.8}$	$17.1_{\scriptscriptstyle\pm 0.2}$
Expert ${}_{\text{Search}}$	$9.79$	${\color[rgb]{1,0,0}\mathbf{49.9_{\scriptscriptstyle\pm 1.7}}}$	$2.7_{\scriptscriptstyle\pm 0.2}$	$62.4_{\scriptscriptstyle\pm 0.2}$	$40.2_{\scriptscriptstyle\pm 1.9}$	$79.2_{\scriptscriptstyle\pm 0.1}$	$\underline{{56.0_{\scriptscriptstyle\pm 1.3}}}$	${36.7_{\scriptscriptstyle\pm 2.3}}$	$6.4_{\scriptscriptstyle\pm 0.7}$	$36.7_{\scriptscriptstyle\pm 10.1}$	$14.7_{\scriptscriptstyle\pm 0.3}$
Expert ${}_{\text{E-commerce}}$	$6.25$	$19.3_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{33.2_{\scriptscriptstyle\pm 1.1}}}$	$40.1_{\scriptscriptstyle\pm 0.4}$	$33.2_{\scriptscriptstyle\pm 2.6}$	$70.1_{\scriptscriptstyle\pm 0.1}$	$32.2_{\scriptscriptstyle\pm 0.2}$	$19.6_{\scriptscriptstyle\pm 0.2}$	${\color[rgb]{1,0,0}\mathbf{46.8_{\scriptscriptstyle\pm 1.3}}}$	$65.8_{\scriptscriptstyle\pm 1.3}$	$15.7_{\scriptscriptstyle\pm 0.3}$

Table 2: Main result. Each per-task expert is trained independently on a single task. Seq-Final denotes the final checkpoint obtained after sequentially training. Starting from this checkpoint, we further apply MMOPD and SDFT . MM uniformly averages the parameters. Dashed boxes indicate the training stage corresponding to each task.

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Single-teacher distillation (Student: Seq- E-commerce $\leftarrow$ Teacher: Seq- Search )
Teacher ${}_{\text{Search}}$	${\color[rgb]{1,0,0}\mathbf{23.33}}$	$45.2_{\scriptscriptstyle\pm 0.4}$	$2.0_{\scriptscriptstyle\pm 0.3}$	$39.8_{\scriptscriptstyle\pm 0.2}$	${\color[rgb]{1,0,0}\mathbf{37.2_{\scriptscriptstyle\pm 0.7}}}$	${\color[rgb]{1,0,0}\mathbf{79.2_{\scriptscriptstyle\pm 0.1}}}$	$52.9_{\scriptscriptstyle\pm 0.1}$	$37.4_{\scriptscriptstyle\pm 0.8}$	$9.1_{\scriptscriptstyle\pm 3.9}$	$30.8_{\scriptscriptstyle\pm 1.4}$	$15.8_{\scriptscriptstyle\pm 0.2}$
Student ${}_{\text{E-commerce}}$	$6.04$	$14.6_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{34.0_{\scriptscriptstyle\pm 0.9}}}$	$41.3_{\scriptscriptstyle\pm 0.3}$	$36.0_{\scriptscriptstyle\pm 1.4}$	$67.3_{\scriptscriptstyle\pm 0.3}$	$13.2_{\scriptscriptstyle\pm 0.1}$	$9.4_{\scriptscriptstyle\pm 0.2}$	${\color[rgb]{1,0,0}\mathbf{46.3_{\scriptscriptstyle\pm 1.6}}}$	${\color[rgb]{1,0,0}\mathbf{74.2_{\scriptscriptstyle\pm 1.4}}}$	$19.3_{\scriptscriptstyle\pm 0.9}$
Student ${}_{\text{MOPD}}$	$15.62$	${\color[rgb]{1,0,0}\mathbf{48.4_{\scriptscriptstyle\pm 0.0}}}$	$15.8_{\scriptscriptstyle\pm 0.6}$	${\color[rgb]{1,0,0}\mathbf{53.6_{\scriptscriptstyle\pm 0.2}}}$	$35.1_{\scriptscriptstyle\pm 1.4}$	$78.7_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{53.8_{\scriptscriptstyle\pm 0.2}}}$	${\color[rgb]{1,0,0}\mathbf{39.0_{\scriptscriptstyle\pm 0.2}}}$	$21.7_{\scriptscriptstyle\pm 1.4}$	$65.0_{\scriptscriptstyle\pm 2.0}$	${\color[rgb]{1,0,0}\mathbf{19.7_{\scriptscriptstyle\pm 0.0}}}$

Table 3: Ablation on single-teacher distillation. The teacher is the search-stage checkpoint from sequential training, and the student is the subsequent E-commerce-stage checkpoint. Student

{}_{\text{MOPD}}

is obtained by distilling the search-stage teacher into the student. Dashed boxes indicate the training stage corresponding to each task.

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Seq + SDFT
Random-Mix	${\color[rgb]{1,0,0}\mathbf{22.92}}$	$48.3$	${\color[rgb]{1,0,0}\mathbf{25.0}}$	$54.0$	${\color[rgb]{1,0,0}\mathbf{44.4}}$	$79.2$	$55.7$	$37.0$	${\color[rgb]{1,0,0}\mathbf{23.9}}$	$65.0$	${\color[rgb]{1,0,0}\mathbf{21.3}}$
Balanced-Mix	$22.29$	${\color[rgb]{1,0,0}\mathbf{48.4}}$	$22.3$	$53.8$	$38.26$	$78.86$	$55.6$	${\color[rgb]{1,0,0}\mathbf{38.2}}$	$20.0$	${\color[rgb]{1,0,0}\mathbf{67.5}}$	$19.7$
Base + SDFT

Table 4: Comparison of SDFT variants with different initializations and data mixing strategies. SDFT is initialized either from the sequentially trained checkpoint (

\pi_{K}

) or the base model (

\pi_{0}

), using oracle trajectories collected from the corresponding policy. Random-Mix samples training examples randomly, whereas Balanced-Mix enforces uniform sampling.

Panel of Figure 3 — Figure 3: Magnitude and direction of parameter updates across post-training pipelines. (a) PCA of model weights. MMOPD remains close to the final sequential checkpoint, whereas SDFT moves farther in parameter space. (b) Relative update magnitude ( $\|\Delta\theta\|/\|\theta\|$ ). SFT-based methods produce much larger parameter updates than RL-based methods, with SDFT exhibiting the largest changes.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Initialization	Training source	Training method
Base	Pretrained	–	–
Oracle Training
Expert ${}_{\text{Math}}$ (=Seq- Math )	Base	DAPO-Math-17K	RL
Expert ${}_{\text{Search}}$	Base	Natural Question	SFT $+$ RL
Expert ${}_{\text{E-commerce}}$	Base	$\tau^{3}$ -Bench (retail)	SFT $+$ RL
Expert ${}_{\text{IF}}$	Base	Instruction Following	RL

Appendix

Table 5: Overview of the models and capability-consolidation methods. Per-task experts are trained independently from the pretrained base model, whereas sequential checkpoints inherit all preceding training stages. MMOPD and SDFT are initialized from the final sequential checkpoint. Model merging (MM) is training-free and directly averages the four sequential training checkpoints. MLE initializes from SDFT and trains a separate LoRA expert for each domain using RL.

Figure 4: Training collapse of vanilla OPD on the agentic search task. The green, purple, and yellow curves represent Mixed OPD, reverse-KL-based OPD, and forward-KL-based OPD, respectively. The green curve clearly demonstrates the stable training dynamics of Mixed OPD.

Panel of Figure 5 — Figure 5: Failure modes of direct RL without prior SFT on search tasks. Direct RL leads to (a) frequent format violations and (b) increasingly truncated responses , highlighting the role of SFT in establishing valid task behavior before RL refinement.

Merge weights	Weights					ID evaluation ( $\uparrow$ )				Avg ( $\uparrow$ )
Merge weights	Base	Math	Search	E-commerce	IF	AIME26 (avg@16)	Search	$\tau^{3}$ -Retail	IF	Avg ( $\uparrow$ )
Uniform	0.2	0.2	0.2	0.2	0.2	10.62	37.0	11.11	67.8	31.63
Base-heavy	0.6	0.1	0.1	0.1	0.1	6.25	22.0	5.84	53.4	21.87
Math-heavy	0.1	0.6	0.1	0.1	0.1	18.12	18.5	1.97	49.7	22.07
Search-heavy	0.1	0.1	0.6	0.1	0.1	10.83	44.5	0.89	61.9	29.53
E-commerce-heavy	0.1	0.1	0.1	0.6	0.1	8.54	16.3	17.56	40.3	20.68

Appendix

Table 6: Model merging weight sweep. We merge five checkpoints into one model by linear weight averaging. Rows correspond to different merge-weight allocations. Balanced merging tests whether equal task contribution is sufficient, while biased merging tests whether assigning a dominant weight to one checkpoint improves average performance. Higher Avg indicates better overall performance across tasks.

Figure 6: Cosine similarity between domain task vectors $\Delta\theta_{k}=\theta_{k}-\theta_{0}$ .

Figure 7: Token-level prediction consistency along the training chain, split at

H=0.5

, for all three benchmarks. The leftmost point is the math checkpoint itself and is

1.0

by definition; panel headings give the low- / high-entropy share of positions.

Benchmark	Entropy bin	$n$	% of tokens	Seq- SeaSFT	Seq- Search	Seq- E-comSFT	Seq- E-commerce	Seq- IF
AIME (348,828 positions)
	$[0,0.01)$	298,125	85.5%	0.9998	0.9998	0.9997	0.9997	0.9997
	$[0.01,0.05)$	16,947	4.9%	0.9980	0.9980	0.9943	0.9945	0.9941
	$[0.05,0.2)$	9,198	2.6%	0.9935	0.9927	0.9776	0.9776	0.9764
	$[0.2,0.5)$	7,720	2.2%	0.9600	0.9567	0.9118	0.9130	0.9088
	$H<0.5$	331,990	95.2%	0.9986	0.9985	0.9968	0.9968	0.9967

Appendix

Table 7: Token-level prediction consistency by entropy bin. Each entry is the fraction of positions in that bin at which the checkpoint still ranks the math checkpoint’s token first. The two shaded rows per benchmark are the aggregates plotted in Figure 7 ; they are token-weighted over the bins above them, not averages of the per-bin rates, note how few positions the highest bins hold. Bins marked

\dagger

have fewer than

500

positions and are not reliable.

Explore similar work

Jun 4, 2026cs.LG

CLaaS: Continual learning as a service for sample efficient online learning

Deployed large language model agents must adapt to distribution shift in dynamic environments. Ideally, adaptation can be performed from accumulated agent experiences and retain prior capabilities while transferring to future tasks. However, agent actions and environmental transitions can only be sampled once per scenario, as real-world environments cannot be trivially reset. To this end, we investigate an experiential and online continual learning setting in which agents learn from a stream of scenarios. We propose continual learning as-a-service (CLaaS), a system which enables agents to improve during deployment, abstracted behind a chat API. To increase sample efficiency, CLaaS stores rollouts in an experience replay buffer for gradient reuse during asynchronous training. We evaluate CLaaS on an adversarial task, demonstrating that parametric updates lead to superior forward transfer and less forgetting than in-context learning, with replay being a critical choice for sample efficiency.

Kion Fallah, Silen Naihin, Barak Widawsky +1

1Resolute Labs · 2South Park Commons · 3Incept Labs

Continual Learning for LLM Agents Continual Learning Replay-Based Continual Learning Online Continual Learning Data-Efficient Learning

Ask PDF

Aug 19, 2026cs.LG

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Continual Learning for LLM Agents Continual Learning Agent Harness Optimization

Ask PDF

Jun 1, 2026cs.AI

AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes. Continual learning expects an agent to accumulate reusable experience across a stream of tasks, improve over time, and avoid interference from irrelevant experiences. Unfortunately, existing benchmarks struggle to evaluate continual learning in language agents rigorously. Most efforts focus on retrieval and reasoning over long-context conversations or documents, while recent lifelong-adaptation benchmarks often rely on naive task streams with limited analysis of cross-task relationships, making it difficult to understand what an agent learns and reuses over time. This paper presents an evaluation framework AgentCL for continual learning in agents, centered on controlled task streams and metrics for transfer gains. AgentCL constructs compositional streams where earlier sub-solutions, evidence, or workflows are intentionally reusable in later tasks, and contrasts them with naive streams where such reusability is not guaranteed. We use the benchmark to evaluate non-parametric memory designs for continual learning. To diagnose how memory design choices affect continual learning, we develop MemProbe, a probing method that stores interactions, insights, and skills, while filtering unreliable experiences during consolidation. Empirical analysis across coding, deep research, and language understanding/reasoning tasks shows that naive streams offer limited ability to distinguish memory designs, whereas controlled streams more clearly distinguish their plasticity. Meanwhile, naive and held-out settings often yield limited gains and can expose memory-induced degradation. These results highlight the need for stronger memory designs that balance plasticity and stable reuse.

Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda +3

The Ohio State University · Johns Hopkins University · Intuit AI Research

Continual Learning for LLM Agents AI Agent Evaluation Memory-Augmented Language Model Agents

Ask PDF