Routed Continual Learning

Michael Philip; Mahesh Balakrishnan; David Geraghty; David Mao; Vidhya Venkat; Ilya Mironov

cs.LGOct 28, 2026

Routed Continual Learning

Authors: Michael Philip, Mahesh Balakrishnan, David Geraghty, David Mao, Vidhya Venkat, Ilya Mironov

Organizations: FAIR at Meta

Abstract

Agents deployed at scale repeatedly encounter related problems, and it is natural to let them learn from each other's experience rather than starting over each time. Unfortunately, there is no single way to retain that experience that works well across settings, and no way to know in advance which future problems a given experience will help with. A global history preserves every detail but becomes expensive and mixes together experience that may not belong together; compressing that history into one fixed representation limits the cost but can throw away valuable details. We introduce Routed Continual Learning (Routed CL), an architecture that decides how to partition and retain experience. Past trajectories are grouped into clusters, each of which maintains state using one memory system from a fixed library. Before each run, an LLM-based router either reuses an existing cluster or creates a new one, basing its decision on the problem statement, memory system descriptions, and cluster summaries. We also introduce Context Memory Adapter, which mediates between the memory system and the agentic loop. We evaluate the architecture on a heterogeneous stream that interleaves 301 instances from the six CL-Bench tasks with 97 banking instances from $τ^3$ -Bench. Across four language models and five matched rollouts, Routed Continual Learning raises normalized reward over a single global in-context learning (ICL) history by 15-30% of the maximum attainable normalized reward, achieves higher mean normalized reward than ICL-only clustering, and reduces estimated API cost by 65-72% relative to Global ICL. These results support approaching continual learning as a routing problem that chooses both a learning mechanism and a state cluster rather than committing to one representation globally.

Figures & tables

Model	Stateless ICL	Global ICL	Routed ICL	Routed CL	Ground-truth assignment
Gemini 3.7 Flash $[0,0]$	$-0.055$ $[-0.066,-0.041]$	$-0.030$ $[-0.055,-0.022]$	0.211 $[0.199,0.245]$	0.213 $[0.111,0.272]$	0.284 $[0.236,0.333]$
GPT-5.6 Sol $[0,0]$	0.130 $[0.122,0.137]$	0.168 $[0.125,0.189]$	0.387 $[0.333,0.424]$	0.464 $[0.442,0.501]$	0.478 $[0.441,0.531]$
Opus 4.7 $[0,0]$	0.060 $[0.051,0.067]$	0.092 $[0.077,0.104]$	0.209 $[0.189,0.242]$	0.243 $[0.184,0.309]$	0.312 $[0.276,0.370]$
Opus 5 $[0,0]$	0.178 $[0.155,0.196]$	0.169 $[0.152,0.198]$	0.340 $[0.302,0.383]$	0.401 $[0.356,0.433]$	0.408 $[0.392,0.423]$

Table 1: Normalized reward across five matched rollouts. Each cell reports the mean and the range as

[\min,\max]

. Stateless-reference gain is reported separately in Table 4 . Higher is better; bold marks the best of the four staged conditions. Ground-truth assignment uses source-task identity, unavailable to the router, to apply a fixed memory-system–cluster mapping specified before confirmatory evaluation.

Figure 1: Overview of Routed Continual Learning. Before each instance, the router chooses a memory system and either reuses or creates a stateful cluster. Context Memory Adapter reads its state and augments the task context before execution; the selected cluster is updated from the trajectory, while all other clusters remain unchanged. The displayed clusters are illustrative; the pool grows as clusters are created and reused online.

Memory system	Retained state and observation structure
Retained trajectories
In-context Learning (ICL)	Ordered visible trajectories
Distilled reasoning memory
ReasoningBank (RB)	Strategies induced from self-judged successful and failed trajectories
Structured numerical state
Hierarchical Regression (HR)	Repeated keyed scalar outcomes

Table 2: Five memory systems grouped by the form of state they retain.

Figure 2: Mean normalized reward versus mean estimated API cost across the five matched rollouts in Table 1 . Estimated costs include every model call under the frozen pricing schedule. Within each model, lines connect Global ICL, Routed ICL, and Routed CL; Stateless ICL appears separately. Table 6 reports the exact mean costs. Routed CL achieves the highest mean reward among the four staged conditions for every model and lies on the empirical reward–cost Pareto frontier.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Instances	Instance objective	Reusable cross-instance information
Codebase Adaptation	19	Resolve repository issues under a command budget	Repository structure, test workflows, and contributor conventions
Cohort Studies	20	Estimate event-free curves for 36 population cohorts	Population structure across partially observed and differently sampled studies
Database Exploration	40	Answer database questions using limited SQL queries	Schema, joins, encodings, and migration-induced changes
Exploitable Poker	120	Maximize profit over independent poker hands	Opponent behavior and changes in strategy
Sales Prediction	12	Forecast five years of product–location demand	Shared growth, product and location effects, and forecast calibration
Blind Spectrum Monitoring	90	Infer long-run transmitter occupancy from noisy scans	Persistent transmitters observed intermittently across scans

Appendix

Table 3: Source tasks used to construct the heterogeneous evaluation stream.

Model	Global ICL	Routed ICL	Routed CL	Ground-truth assignment
Gemini 3.7 Flash	0.019 $[0.001,0.038]$	0.229 $[0.193,0.271]$	0.265 $[0.150,0.341]$	0.313 $[0.281,0.356]$
GPT-5.6 Sol	0.043 $[-0.010,0.074]$	0.307 $[0.252,0.357]$	0.381 $[0.364,0.431]$	0.401 $[0.352,0.453]$
Opus 4.7	0.028 $[0.017,0.037]$	0.159 $[0.137,0.201]$	0.207 $[0.147,0.281]$	0.273 $[0.236,0.332]$
Opus 5	$-0.029$ $[-0.084,0.007]$	0.193 $[0.151,0.279]$	0.265 $[0.237,0.318]$	0.270 $[0.256,0.283]$

Appendix

Table 4: Gain relative to matched model-specific Stateless ICL across five rollouts. Each cell reports the mean, with

[\min,\max]

beneath. Higher is better; bold marks the best of the four staged conditions.

Model	Routed CL $-$ Routed ICL Mean $[\min,\max]$	Positive rollouts
Gemini 3.7 Flash	0.002 $[-0.090,0.073]$	4/5
GPT-5.6 Sol	0.076 $[0.060,0.110]$	5/5
Opus 4.7	0.034 $[-0.009,0.085]$	4/5
Opus 5	0.061 $[0.044,0.072]$	5/5

Appendix

Table 5: Paired normalized-reward differences between Routed CL and Routed ICL across the five matched stream seeds. The second column reports the mean with

[\min,\max]

beneath; positive values favor Routed CL.

Figure 3: Mean normalized reward versus mean estimated API cost for the ReasoningBank comparison over three matched rollouts. Within each model, lines connect Global RB, Routed RB, and Routed CL. Routed CL is recomputed over these three rollouts and therefore differs from the five-rollout values in Table 1 . Points toward the upper left are better.

Figure 4: Stateless-reference gain through successive prefixes of the heterogeneous stream. Lines show means across five matched rollouts and shading spans

[\min,\max]

; the endpoints reproduce the full-stream gains in Table 4 .

Model	Stateless ICL	Global ICL	Routed ICL	Routed CL	Ground-truth assignment
Gemini 3.7 Flash	$39.59	$200.14	$91.31	$61.08	$59.20
GPT-5.6 Sol	$89.24	$456.22	$170.40	$125.99	$117.08
Opus 4.7	$128.86	$708.78	$212.37	$222.72	$212.04
Opus 5	$161.77	$914.36	$424.54	$316.61	$265.10

Appendix

Table 6: Mean estimated API cost in USD per 398-instance rollout across five matched seeds. Costs include every model call under the accounting policy in Appendix A.7 ; Table 7 reports the corresponding token breakdown. Lower is better.

Model	Condition	Fresh input	Cache-read input	Output	Usage total
Gemini 3.7 Flash	Stateless ICL	34.1	91.1	0.7	127.2
	Global ICL	237.4	192.8	0.7	432.2
	Routed ICL	101.1	137.3	0.6	239.7
	Routed CL	58.8	105.5	0.9	166.7
	GT assignment	57.7	101.3	0.9	161.2
GPT-5.6 Sol	Stateless ICL	10.2	79.0	0.8	90.1

Appendix

Table 7: Mean token usage (millions) per 398-instance rollout across the same five matched rollouts as Table 6 . The three input columns partition total input; cached input is not counted again in the total. Lower is better, and comparisons are most direct within a model because tokenizers differ.

Figure 5: Five-seed mean change in stateless-reference gain by source task. The panels compare successive conditions: Global ICL minus Stateless ICL, Routed ICL minus Global ICL, and Routed CL minus Routed ICL. Blue indicates improvement and orange regression.

Figure 6: Five-seed mean API-cost ratio between Routed CL and Routed ICL by source task. Each cell reports $C_{\mathrm{Routed\ CL}}/C_{\mathrm{Routed\ ICL}}$ using mean task-level costs across matched rollouts. Values below one (blue) indicate that Routed CL is cheaper; values above one (orange) indicate that it is more expensive. Costs include all model calls attributed to instances from that source task.

(a) Memory-system assignment
Source task	Ground truth	Gemini 3.7 Flash	GPT-5.6 Sol	Opus 4.7	Opus 5
Code	ICL	RB 4, ICL 1	ICL 5	RB 3, ICL 2	ICL 4 , RB 1
Cohort	HDE	HDE 5	HDE 5	HDE 4 , RB 1	HDE 5
Database	ICL	ICL 5	ICL 5	ICL 3 , RB 2	ICL 5
Poker	ICL	ICL 5	ICL 5	ICL 5	ICL 5
Sales	HR	ICL 4, RB 1	HR 5	HR 4 , ICL 1	HR 5

Appendix

Table 8: Memory-system and cluster assignments across four models and five matched rollouts per model. Panel (a) counts assignments to each memory system; panel (b) lists the number of clusters in rollout order. Bold entries match the corresponding component of Ground-truth assignment; cluster agreement requires the complete partition, not only its size. Source-task identities and grouping keys are used only for post-hoc evaluation. Memory-system abbreviations follow Table 2 .

Figure 7: Ground-truth assignment used for diagnostic comparisons. The mapping was fixed before confirmatory evaluation and is not an optimized policy or upper bound. It shows the joint memory-system and cluster assignment for each source task.

Continual Learning for LLM Agents

Continual learning for LLM agents studies how agents adapt from experience while retaining skills and knowledge across tasks.

See all 202 papers

Mar 2, 2026

Position: Modular Memory is the Key to Continual Learning Agents

Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in continuous operation, experience accumulation, and personalization, capabilities that are central to adaptive intelligence. While continual learning research has long targeted these goals, its historical focus on in-weight learning (IWL), i.e., updating a single model's parameters to absorb new knowledge, has rendered catastrophic forgetting a persistent challenge. Our position is that combining the strengths of In-Weight Learning (IWL) and the newly emerged capabilities of In-Context Learning (ICL) through the design of modular memory is the missing piece for continual adaptation at scale. We outline a conceptual framework for modular memory-centric architectures that leverage ICL for rapid adaptation and knowledge accumulation, and IWL for stable updates to model capabilities, charting a practical roadmap toward continually learning agents.

Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov +21

Toyota Motor Europe · University of Bremen · University of Florence +17

Continual Learning for LLM Agents Memory-Augmented Agents

Ask PDF

Aug 19, 2026

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Continual Learning for LLM Agents Continual Learning Agent Harness Optimization

Ask PDF

Jun 4, 2026

CLaaS: Continual learning as a service for sample efficient online learning

Deployed large language model agents must adapt to distribution shift in dynamic environments. Ideally, adaptation can be performed from accumulated agent experiences and retain prior capabilities while transferring to future tasks. However, agent actions and environmental transitions can only be sampled once per scenario, as real-world environments cannot be trivially reset. To this end, we investigate an experiential and online continual learning setting in which agents learn from a stream of scenarios. We propose continual learning as-a-service (CLaaS), a system which enables agents to improve during deployment, abstracted behind a chat API. To increase sample efficiency, CLaaS stores rollouts in an experience replay buffer for gradient reuse during asynchronous training. We evaluate CLaaS on an adversarial task, demonstrating that parametric updates lead to superior forward transfer and less forgetting than in-context learning, with replay being a critical choice for sample efficiency.

Kion Fallah, Silen Naihin, Barak Widawsky +1

1Resolute Labs · 2South Park Commons · 3Incept Labs

Continual Learning for LLM Agents Continual Learning Replay-Based Continual Learning Online Continual Learning Data-Efficient Learning

Ask PDF

Abstract

Continual Learning for LLM Agents

Explore similar work

Position: Modular Memory is the Key to Continual Learning Agents

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

CLaaS: Continual learning as a service for sample efficient online learning