ACLArena: Agent Continue Learning in Multi-stage Post-training

Haixin Wang; Xiaoxuan Wang; Junkai Zhang; Han Zhang; Renliang Sun; Alexander K Taylor; Yidan Shi; Haoran Deng

cs.AISep 21, 2026

ACLArena: Agent Continue Learning in Multi-stage Post-training

Authors: Haixin Wang, Xiaoxuan Wang, Junkai Zhang, Han Zhang, Renliang Sun, Alexander K Taylor, Yidan Shi, Haoran Deng, +4 more

Organizations: University of California, Los Angeles · University of California, Santa Cruz

Abstract

Building general-purpose agents for industrial deployment requires integrating multiple capabilities, each typically acquired at a distinct stage of training. Yet there is currently no well-established recipe for Agent Continual Learning (ACL), with little understanding of the trade-offs among existing integration paradigms. To address this gap, we introduce ACLArena, a framework for comprehensively studying, analyzing, and evaluating ACL. We first build a sequential training pipeline and conduct an in-depth analysis that explains the mechanisms of forgetting and generalization from two complementary perspectives, the model level and the token level. Guided by these analyses, we systematically compare multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging to assess their ability to recover previously learned capabilities while preserving newly acquired ones. Through extensive experiments, we develop a detailed understanding of how capabilities transfer across stages. Finally, we propose a new ACL recipe that combines offline replay over high-quality trajectories with a routed network of multiple LoRA experts each specialized via RL, substantially improving the agent's ability to learn across multiple domains. Comprehensive experiments on four reasoning and agentic tasks, evaluated under both in-domain and out-of-domain settings, demonstrate the value of our analysis and the effectiveness of our approach.

Figures & tables

Figure 1: Overview of the four mainstream paradigms for capability consolidation: sequential training, multi-teacher mixed on-policy distillation (MMOPD), self-distilled fine-tuning (SDFT), and model merging (MM).

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Per-stage performance (Math $\rightarrow$ Search $\rightarrow$ E-commerce $\rightarrow$ Instruction Following)
Base	$6.46$	$13.3_{\scriptscriptstyle\pm 0.2}$	$3.5_{\scriptscriptstyle\pm 0.4}$	$46.0_{\scriptscriptstyle\pm 0.9}$	$30.8_{\scriptscriptstyle\pm 1.6}$	$65.1_{\scriptscriptstyle\pm 0.3}$	$22.3_{\scriptscriptstyle\pm 0.0}$	$11.1_{\scriptscriptstyle\pm 0.6}$	$17.5_{\scriptscriptstyle\pm 1.0}$	$5.0_{\scriptscriptstyle\pm 2.5}$	$14.4_{\scriptscriptstyle\pm 0.2}$
Seq- Math	${\color[rgb]{1,0,0}\mathbf{25.83}}$	$22.0_{\scriptscriptstyle\pm 0.2}$	$2.3_{\scriptscriptstyle\pm 0.5}$	$43.3_{\scriptscriptstyle\pm 0.6}$	${\color[rgb]{1,0,0}\mathbf{40.2_{\scriptscriptstyle\pm 2.5}}}$	$79.0_{\scriptscriptstyle\pm 0.2}$	$38.7_{\scriptscriptstyle\pm 0.2}$	$22.8_{\scriptscriptstyle\pm 0.5}$	$13.1_{\scriptscriptstyle\pm 2.1}$	$3.3_{\scriptscriptstyle\pm 3.8}$	$17.1_{\scriptscriptstyle\pm 0.2}$
Seq- Search	$23.33$	${\color[rgb]{1,0,0}\mathbf{45.2_{\scriptscriptstyle\pm 0.4}}}$	$2.0_{\scriptscriptstyle\pm 0.3}$	$39.8_{\scriptscriptstyle\pm 0.2}$	$37.2_{\scriptscriptstyle\pm 0.7}$	$79.2_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{52.9_{\scriptscriptstyle\pm 0.1}}}$	${\color[rgb]{1,0,0}\mathbf{37.4_{\scriptscriptstyle\pm 0.8}}}$	$9.1_{\scriptscriptstyle\pm 3.9}$	$30.8_{\scriptscriptstyle\pm 1.4}$	$15.8_{\scriptscriptstyle\pm 0.2}$

Table 1: Main sequential training results. Single-hop Search reports the average performance on PopQA and TriviaQA. Multi-hop Search reports the average performance on 2Wiki, Bamboogle, HotpotQA, and MuSiQue. Dashed boxes indicate the training stage corresponding to each task, while changes in the subsequent rows reflect generalization and forgetting. We denote the checkpoints after each training stage as Seq-Math , Seq-Search , Seq-E-commerce , and Seq-IF , respectively; Seq-IF is the final sequential checkpoint and is referred to as Seq-Final in subsequent experiments. The best score in each column is highlighted in red . All results are reported as mean

\pm

standard deviation over three independent runs.

Panel of Figure 2 — Figure 2: Diagnosing sequential training. Rows (top to bottom) follow the training order; columns correspond to the evaluation of each domain, and marginal bars show mean performance gap $\Delta$ per metric and per stage. The projection in (c) reveals both that the tasks occupy distinct optimization directions and that it progressively fails to recover them. (d) provides token-level analysis of original rollout with different entropy.

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Per-Task Oracle Training
Expert ${}_{\text{Math}}$	${\color[rgb]{1,0,0}\mathbf{25.83}}$	$22.0_{\scriptscriptstyle\pm 0.2}$	$2.3_{\scriptscriptstyle\pm 0.5}$	$43.3_{\scriptscriptstyle\pm 0.6}$	$40.2_{\scriptscriptstyle\pm 2.5}$	$79.0_{\scriptscriptstyle\pm 0.2}$	$38.7_{\scriptscriptstyle\pm 0.2}$	$22.8_{\scriptscriptstyle\pm 0.5}$	$13.1_{\scriptscriptstyle\pm 2.1}$	$3.3_{\scriptscriptstyle\pm 3.8}$	$17.1_{\scriptscriptstyle\pm 0.2}$
Expert ${}_{\text{Search}}$	$9.79$	${\color[rgb]{1,0,0}\mathbf{49.9_{\scriptscriptstyle\pm 1.7}}}$	$2.7_{\scriptscriptstyle\pm 0.2}$	$62.4_{\scriptscriptstyle\pm 0.2}$	$40.2_{\scriptscriptstyle\pm 1.9}$	$79.2_{\scriptscriptstyle\pm 0.1}$	$\underline{{56.0_{\scriptscriptstyle\pm 1.3}}}$	${36.7_{\scriptscriptstyle\pm 2.3}}$	$6.4_{\scriptscriptstyle\pm 0.7}$	$36.7_{\scriptscriptstyle\pm 10.1}$	$14.7_{\scriptscriptstyle\pm 0.3}$
Expert ${}_{\text{E-commerce}}$	$6.25$	$19.3_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{33.2_{\scriptscriptstyle\pm 1.1}}}$	$40.1_{\scriptscriptstyle\pm 0.4}$	$33.2_{\scriptscriptstyle\pm 2.6}$	$70.1_{\scriptscriptstyle\pm 0.1}$	$32.2_{\scriptscriptstyle\pm 0.2}$	$19.6_{\scriptscriptstyle\pm 0.2}$	${\color[rgb]{1,0,0}\mathbf{46.8_{\scriptscriptstyle\pm 1.3}}}$	$65.8_{\scriptscriptstyle\pm 1.3}$	$15.7_{\scriptscriptstyle\pm 0.3}$

Table 2: Main result. Each per-task expert is trained independently on a single task. Seq-Final denotes the final checkpoint obtained after sequentially training. Starting from this checkpoint, we further apply MMOPD and SDFT . MM uniformly averages the parameters. Dashed boxes indicate the training stage corresponding to each task.

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Single-teacher distillation (Student: Seq- E-commerce $\leftarrow$ Teacher: Seq- Search )
Teacher ${}_{\text{Search}}$	${\color[rgb]{1,0,0}\mathbf{23.33}}$	$45.2_{\scriptscriptstyle\pm 0.4}$	$2.0_{\scriptscriptstyle\pm 0.3}$	$39.8_{\scriptscriptstyle\pm 0.2}$	${\color[rgb]{1,0,0}\mathbf{37.2_{\scriptscriptstyle\pm 0.7}}}$	${\color[rgb]{1,0,0}\mathbf{79.2_{\scriptscriptstyle\pm 0.1}}}$	$52.9_{\scriptscriptstyle\pm 0.1}$	$37.4_{\scriptscriptstyle\pm 0.8}$	$9.1_{\scriptscriptstyle\pm 3.9}$	$30.8_{\scriptscriptstyle\pm 1.4}$	$15.8_{\scriptscriptstyle\pm 0.2}$
Student ${}_{\text{E-commerce}}$	$6.04$	$14.6_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{34.0_{\scriptscriptstyle\pm 0.9}}}$	$41.3_{\scriptscriptstyle\pm 0.3}$	$36.0_{\scriptscriptstyle\pm 1.4}$	$67.3_{\scriptscriptstyle\pm 0.3}$	$13.2_{\scriptscriptstyle\pm 0.1}$	$9.4_{\scriptscriptstyle\pm 0.2}$	${\color[rgb]{1,0,0}\mathbf{46.3_{\scriptscriptstyle\pm 1.6}}}$	${\color[rgb]{1,0,0}\mathbf{74.2_{\scriptscriptstyle\pm 1.4}}}$	$19.3_{\scriptscriptstyle\pm 0.9}$
Student ${}_{\text{MOPD}}$	$15.62$	${\color[rgb]{1,0,0}\mathbf{48.4_{\scriptscriptstyle\pm 0.0}}}$	$15.8_{\scriptscriptstyle\pm 0.6}$	${\color[rgb]{1,0,0}\mathbf{53.6_{\scriptscriptstyle\pm 0.2}}}$	$35.1_{\scriptscriptstyle\pm 1.4}$	$78.7_{\scriptscriptstyle\pm 0.1}$	${\color[rgb]{1,0,0}\mathbf{53.8_{\scriptscriptstyle\pm 0.2}}}$	${\color[rgb]{1,0,0}\mathbf{39.0_{\scriptscriptstyle\pm 0.2}}}$	$21.7_{\scriptscriptstyle\pm 1.4}$	$65.0_{\scriptscriptstyle\pm 2.0}$	${\color[rgb]{1,0,0}\mathbf{19.7_{\scriptscriptstyle\pm 0.0}}}$

Table 3: Ablation on single-teacher distillation. The teacher is the search-stage checkpoint from sequential training, and the student is the subsequent E-commerce-stage checkpoint. Student

{}_{\text{MOPD}}

is obtained by distilling the search-stage teacher into the student. Dashed boxes indicate the training stage corresponding to each task.

	In-domain evaluation ( $\uparrow$ )				Out-of-domain evaluation ( $\uparrow$ )
	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	Math		Single-hop Search	Multi-hop Search	E-commerce		IF
Pipeline	AIME26 (avg@16)	NQ	$\tau^{3}$ -Retail	IF-Eval	GPQA	MMLU	Avg.	Avg.	$\tau^{3}$ -Telecom	$\tau^{3}$ -Mock	IF-Bench
Seq + SDFT
Random-Mix	${\color[rgb]{1,0,0}\mathbf{22.92}}$	$48.3$	${\color[rgb]{1,0,0}\mathbf{25.0}}$	$54.0$	${\color[rgb]{1,0,0}\mathbf{44.4}}$	$79.2$	$55.7$	$37.0$	${\color[rgb]{1,0,0}\mathbf{23.9}}$	$65.0$	${\color[rgb]{1,0,0}\mathbf{21.3}}$
Balanced-Mix	$22.29$	${\color[rgb]{1,0,0}\mathbf{48.4}}$	$22.3$	$53.8$	$38.26$	$78.86$	$55.6$	${\color[rgb]{1,0,0}\mathbf{38.2}}$	$20.0$	${\color[rgb]{1,0,0}\mathbf{67.5}}$	$19.7$
Base + SDFT

Table 4: Comparison of SDFT variants with different initializations and data mixing strategies. SDFT is initialized either from the sequentially trained checkpoint (

\pi_{K}

) or the base model (

\pi_{0}

), using oracle trajectories collected from the corresponding policy. Random-Mix samples training examples randomly, whereas Balanced-Mix enforces uniform sampling.

Panel of Figure 3 — Figure 3: Magnitude and direction of parameter updates across post-training pipelines. (a) PCA of model weights. MMOPD remains close to the final sequential checkpoint, whereas SDFT moves farther in parameter space. (b) Relative update magnitude ( $\|\Delta\theta\|/\|\theta\|$ ). SFT-based methods produce much larger parameter updates than RL-based methods, with SDFT exhibiting the largest changes.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Initialization	Training source	Training method
Base	Pretrained	–	–
Oracle Training
Expert ${}_{\text{Math}}$ (=Seq- Math )	Base	DAPO-Math-17K	RL
Expert ${}_{\text{Search}}$	Base	Natural Question	SFT $+$ RL
Expert ${}_{\text{E-commerce}}$	Base	$\tau^{3}$ -Bench (retail)	SFT $+$ RL
Expert ${}_{\text{IF}}$	Base	Instruction Following	RL

Appendix

Table 5: Overview of the models and capability-consolidation methods. Per-task experts are trained independently from the pretrained base model, whereas sequential checkpoints inherit all preceding training stages. MMOPD and SDFT are initialized from the final sequential checkpoint. Model merging (MM) is training-free and directly averages the four sequential training checkpoints. MLE initializes from SDFT and trains a separate LoRA expert for each domain using RL.

Figure 4: Training collapse of vanilla OPD on the agentic search task. The green, purple, and yellow curves represent Mixed OPD, reverse-KL-based OPD, and forward-KL-based OPD, respectively. The green curve clearly demonstrates the stable training dynamics of Mixed OPD.

Panel of Figure 5 — Figure 5: Failure modes of direct RL without prior SFT on search tasks. Direct RL leads to (a) frequent format violations and (b) increasingly truncated responses , highlighting the role of SFT in establishing valid task behavior before RL refinement.

Merge weights	Weights					ID evaluation ( $\uparrow$ )				Avg ( $\uparrow$ )
Merge weights	Base	Math	Search	E-commerce	IF	AIME26 (avg@16)	Search	$\tau^{3}$ -Retail	IF	Avg ( $\uparrow$ )
Uniform	0.2	0.2	0.2	0.2	0.2	10.62	37.0	11.11	67.8	31.63
Base-heavy	0.6	0.1	0.1	0.1	0.1	6.25	22.0	5.84	53.4	21.87
Math-heavy	0.1	0.6	0.1	0.1	0.1	18.12	18.5	1.97	49.7	22.07
Search-heavy	0.1	0.1	0.6	0.1	0.1	10.83	44.5	0.89	61.9	29.53
E-commerce-heavy	0.1	0.1	0.1	0.6	0.1	8.54	16.3	17.56	40.3	20.68

Appendix

Table 6: Model merging weight sweep. We merge five checkpoints into one model by linear weight averaging. Rows correspond to different merge-weight allocations. Balanced merging tests whether equal task contribution is sufficient, while biased merging tests whether assigning a dominant weight to one checkpoint improves average performance. Higher Avg indicates better overall performance across tasks.

Figure 6: Cosine similarity between domain task vectors $\Delta\theta_{k}=\theta_{k}-\theta_{0}$ .

Figure 7: Token-level prediction consistency along the training chain, split at

H=0.5

, for all three benchmarks. The leftmost point is the math checkpoint itself and is

1.0

by definition; panel headings give the low- / high-entropy share of positions.

Benchmark	Entropy bin	$n$	% of tokens	Seq- SeaSFT	Seq- Search	Seq- E-comSFT	Seq- E-commerce	Seq- IF
AIME (348,828 positions)
	$[0,0.01)$	298,125	85.5%	0.9998	0.9998	0.9997	0.9997	0.9997
	$[0.01,0.05)$	16,947	4.9%	0.9980	0.9980	0.9943	0.9945	0.9941
	$[0.05,0.2)$	9,198	2.6%	0.9935	0.9927	0.9776	0.9776	0.9764
	$[0.2,0.5)$	7,720	2.2%	0.9600	0.9567	0.9118	0.9130	0.9088
	$H<0.5$	331,990	95.2%	0.9986	0.9985	0.9968	0.9968	0.9967

Appendix

Table 7: Token-level prediction consistency by entropy bin. Each entry is the fraction of positions in that bin at which the checkpoint still ranks the math checkpoint’s token first. The two shaded rows per benchmark are the aggregates plotted in Figure 7 ; they are token-weighted over the bins above them, not averages of the per-bin rates, note how few positions the highest bins hold. Bins marked

\dagger

have fewer than

500

positions and are not reliable.

Explore similar work

CLaaS: Continual learning as a service for sample efficient online learning
Jun 4, 2026Kion Fallah, Silen Naihin, Barak Widawsky +1Continual Learning for LLM Agents Continual Learning
1Resolute Labs · 2South Park Commons · 3Incept Labs
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
Aug 19, 2026Borui Kang, Jinrui Gu, Junhan Lv +3Continual Learning for LLM Agents Continual Learning
State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia
AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents
Jun 1, 2026Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda +3Continual Learning for LLM Agents AI Agent Evaluation
The Ohio State University · Johns Hopkins University · Intuit AI Research

Jun 4, 2026, cs.LGJ/K move · Enter open · S save

CLaaS: Continual learning as a service for sample efficient online learning

Kion Fallah, Silen Naihin, Barak Widawsky, Qingqing Mao

1Resolute Labs · 2South Park Commons · 3Incept Labs

Continual Learning for LLM Agents Continual Learning Replay-Based Continual Learning Online Continual Learning Data-Efficient Learning

Open paper AskPDF arXiv

Deployed large language model agents must adapt to distribution shift in dynamic environments. Ideally, adaptation can be performed from accumulated agent experiences and retain prior capabilities while transferring to future tasks. However, agent actions and environmental transitions can only be sampled once per scenario, as real-world environments cannot be trivially reset. To this end, we investigate an experiential and online continual learning setting in which agents learn from a stream of scenarios. We propose continual learning as-a-service (CLaaS), a system which enables agents to improve during deployment, abstracted behind a chat API. To increase sample efficiency, CLaaS stores rollouts in an experience replay buffer for gradient reuse during asynchronous training. We evaluate CLaaS on an adversarial task, demonstrating that parametric updates lead to superior forward transfer and less forgetting than in-context learning, with replay being a critical choice for sample efficiency.

Abstract

Explore similar work

CLaaS: Continual learning as a service for sample efficient online learning

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents