Routed Continual Learning

Michael Philip; Mahesh Balakrishnan; David Geraghty; David Mao; Vidhya Venkat; Ilya Mironov

cs.LGOct 28, 2026

Routed Continual Learning

Authors: Michael Philip, Mahesh Balakrishnan, David Geraghty, David Mao, Vidhya Venkat, Ilya Mironov

Organizations: FAIR at Meta

Abstract

Agents deployed at scale repeatedly encounter related problems, and it is natural to let them learn from each other's experience rather than starting over each time. Unfortunately, there is no single way to retain that experience that works well across settings, and no way to know in advance which future problems a given experience will help with. A global history preserves every detail but becomes expensive and mixes together experience that may not belong together; compressing that history into one fixed representation limits the cost but can throw away valuable details. We introduce Routed Continual Learning (Routed CL), an architecture that decides how to partition and retain experience. Past trajectories are grouped into clusters, each of which maintains state using one memory system from a fixed library. Before each run, an LLM-based router either reuses an existing cluster or creates a new one, basing its decision on the problem statement, memory system descriptions, and cluster summaries. We also introduce Context Memory Adapter, which mediates between the memory system and the agentic loop. We evaluate the architecture on a heterogeneous stream that interleaves 301 instances from the six CL-Bench tasks with 97 banking instances from $τ^3$ -Bench. Across four language models and five matched rollouts, Routed Continual Learning raises normalized reward over a single global in-context learning (ICL) history by 15-30% of the maximum attainable normalized reward, achieves higher mean normalized reward than ICL-only clustering, and reduces estimated API cost by 65-72% relative to Global ICL. These results support approaching continual learning as a routing problem that chooses both a learning mechanism and a state cluster rather than committing to one representation globally.

Figures & tables

Model	Stateless ICL	Global ICL	Routed ICL	Routed CL	Ground-truth assignment
Gemini 3.7 Flash $[0,0]$	$-0.055$ $[-0.066,-0.041]$	$-0.030$ $[-0.055,-0.022]$	0.211 $[0.199,0.245]$	0.213 $[0.111,0.272]$	0.284 $[0.236,0.333]$
GPT-5.6 Sol $[0,0]$	0.130 $[0.122,0.137]$	0.168 $[0.125,0.189]$	0.387 $[0.333,0.424]$	0.464 $[0.442,0.501]$	0.478 $[0.441,0.531]$
Opus 4.7 $[0,0]$	0.060 $[0.051,0.067]$	0.092 $[0.077,0.104]$	0.209 $[0.189,0.242]$	0.243 $[0.184,0.309]$	0.312 $[0.276,0.370]$
Opus 5 $[0,0]$	0.178 $[0.155,0.196]$	0.169 $[0.152,0.198]$	0.340 $[0.302,0.383]$	0.401 $[0.356,0.433]$	0.408 $[0.392,0.423]$

Table 1: Normalized reward across five matched rollouts. Each cell reports the mean and the range as

[\min,\max]

. Stateless-reference gain is reported separately in Table 4 . Higher is better; bold marks the best of the four staged conditions. Ground-truth assignment uses source-task identity, unavailable to the router, to apply a fixed memory-system–cluster mapping specified before confirmatory evaluation.

Figure 1: Overview of Routed Continual Learning. Before each instance, the router chooses a memory system and either reuses or creates a stateful cluster. Context Memory Adapter reads its state and augments the task context before execution; the selected cluster is updated from the trajectory, while all other clusters remain unchanged. The displayed clusters are illustrative; the pool grows as clusters are created and reused online.

Memory system	Retained state and observation structure
Retained trajectories
In-context Learning (ICL)	Ordered visible trajectories
Distilled reasoning memory
ReasoningBank (RB)	Strategies induced from self-judged successful and failed trajectories
Structured numerical state
Hierarchical Regression (HR)	Repeated keyed scalar outcomes

Table 2: Five memory systems grouped by the form of state they retain.

Figure 2: Mean normalized reward versus mean estimated API cost across the five matched rollouts in Table 1 . Estimated costs include every model call under the frozen pricing schedule. Within each model, lines connect Global ICL, Routed ICL, and Routed CL; Stateless ICL appears separately. Table 6 reports the exact mean costs. Routed CL achieves the highest mean reward among the four staged conditions for every model and lies on the empirical reward–cost Pareto frontier.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Instances	Instance objective	Reusable cross-instance information
Codebase Adaptation	19	Resolve repository issues under a command budget	Repository structure, test workflows, and contributor conventions
Cohort Studies	20	Estimate event-free curves for 36 population cohorts	Population structure across partially observed and differently sampled studies
Database Exploration	40	Answer database questions using limited SQL queries	Schema, joins, encodings, and migration-induced changes
Exploitable Poker	120	Maximize profit over independent poker hands	Opponent behavior and changes in strategy
Sales Prediction	12	Forecast five years of product–location demand	Shared growth, product and location effects, and forecast calibration
Blind Spectrum Monitoring	90	Infer long-run transmitter occupancy from noisy scans	Persistent transmitters observed intermittently across scans

Appendix

Table 3: Source tasks used to construct the heterogeneous evaluation stream.

Model	Global ICL	Routed ICL	Routed CL	Ground-truth assignment
Gemini 3.7 Flash	0.019 $[0.001,0.038]$	0.229 $[0.193,0.271]$	0.265 $[0.150,0.341]$	0.313 $[0.281,0.356]$
GPT-5.6 Sol	0.043 $[-0.010,0.074]$	0.307 $[0.252,0.357]$	0.381 $[0.364,0.431]$	0.401 $[0.352,0.453]$
Opus 4.7	0.028 $[0.017,0.037]$	0.159 $[0.137,0.201]$	0.207 $[0.147,0.281]$	0.273 $[0.236,0.332]$
Opus 5	$-0.029$ $[-0.084,0.007]$	0.193 $[0.151,0.279]$	0.265 $[0.237,0.318]$	0.270 $[0.256,0.283]$

Appendix

Table 4: Gain relative to matched model-specific Stateless ICL across five rollouts. Each cell reports the mean, with

[\min,\max]

beneath. Higher is better; bold marks the best of the four staged conditions.

Model	Routed CL $-$ Routed ICL Mean $[\min,\max]$	Positive rollouts
Gemini 3.7 Flash	0.002 $[-0.090,0.073]$	4/5
GPT-5.6 Sol	0.076 $[0.060,0.110]$	5/5
Opus 4.7	0.034 $[-0.009,0.085]$	4/5
Opus 5	0.061 $[0.044,0.072]$	5/5

Appendix

Table 5: Paired normalized-reward differences between Routed CL and Routed ICL across the five matched stream seeds. The second column reports the mean with

[\min,\max]

beneath; positive values favor Routed CL.

Figure 3: Mean normalized reward versus mean estimated API cost for the ReasoningBank comparison over three matched rollouts. Within each model, lines connect Global RB, Routed RB, and Routed CL. Routed CL is recomputed over these three rollouts and therefore differs from the five-rollout values in Table 1 . Points toward the upper left are better.

Figure 4: Stateless-reference gain through successive prefixes of the heterogeneous stream. Lines show means across five matched rollouts and shading spans

[\min,\max]

; the endpoints reproduce the full-stream gains in Table 4 .

Model	Stateless ICL	Global ICL	Routed ICL	Routed CL	Ground-truth assignment
Gemini 3.7 Flash	$39.59	$200.14	$91.31	$61.08	$59.20
GPT-5.6 Sol	$89.24	$456.22	$170.40	$125.99	$117.08
Opus 4.7	$128.86	$708.78	$212.37	$222.72	$212.04
Opus 5	$161.77	$914.36	$424.54	$316.61	$265.10

Appendix

Table 6: Mean estimated API cost in USD per 398-instance rollout across five matched seeds. Costs include every model call under the accounting policy in Appendix A.7 ; Table 7 reports the corresponding token breakdown. Lower is better.

Model	Condition	Fresh input	Cache-read input	Output	Usage total
Gemini 3.7 Flash	Stateless ICL	34.1	91.1	0.7	127.2
	Global ICL	237.4	192.8	0.7	432.2
	Routed ICL	101.1	137.3	0.6	239.7
	Routed CL	58.8	105.5	0.9	166.7
	GT assignment	57.7	101.3	0.9	161.2
GPT-5.6 Sol	Stateless ICL	10.2	79.0	0.8	90.1

Appendix

Table 7: Mean token usage (millions) per 398-instance rollout across the same five matched rollouts as Table 6 . The three input columns partition total input; cached input is not counted again in the total. Lower is better, and comparisons are most direct within a model because tokenizers differ.

Figure 5: Five-seed mean change in stateless-reference gain by source task. The panels compare successive conditions: Global ICL minus Stateless ICL, Routed ICL minus Global ICL, and Routed CL minus Routed ICL. Blue indicates improvement and orange regression.

Figure 6: Five-seed mean API-cost ratio between Routed CL and Routed ICL by source task. Each cell reports $C_{\mathrm{Routed\ CL}}/C_{\mathrm{Routed\ ICL}}$ using mean task-level costs across matched rollouts. Values below one (blue) indicate that Routed CL is cheaper; values above one (orange) indicate that it is more expensive. Costs include all model calls attributed to instances from that source task.

(a) Memory-system assignment
Source task	Ground truth	Gemini 3.7 Flash	GPT-5.6 Sol	Opus 4.7	Opus 5
Code	ICL	RB 4, ICL 1	ICL 5	RB 3, ICL 2	ICL 4 , RB 1
Cohort	HDE	HDE 5	HDE 5	HDE 4 , RB 1	HDE 5
Database	ICL	ICL 5	ICL 5	ICL 3 , RB 2	ICL 5
Poker	ICL	ICL 5	ICL 5	ICL 5	ICL 5
Sales	HR	ICL 4, RB 1	HR 5	HR 4 , ICL 1	HR 5

Appendix

Table 8: Memory-system and cluster assignments across four models and five matched rollouts per model. Panel (a) counts assignments to each memory system; panel (b) lists the number of clusters in rollout order. Bold entries match the corresponding component of Ground-truth assignment; cluster agreement requires the complete partition, not only its size. Source-task identities and grouping keys are used only for post-hoc evaluation. Memory-system abbreviations follow Table 2 .

Figure 7: Ground-truth assignment used for diagnostic comparisons. The mapping was fixed before confirmatory evaluation and is not an optimized policy or upper bound. It shows the joint memory-system and cluster assignment for each source task.

Continual Learning for LLM Agents

Continual learning for LLM agents studies how agents adapt from experience while retaining skills and knowledge across tasks.

See all 202 papers

Position: Modular Memory is the Key to Continual Learning Agents
Mar 2, 2026Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov +21Continual Learning for LLM Agents Memory-Augmented Agents
Toyota Motor Europe · University of Bremen · University of Florence +17
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
Aug 19, 2026Borui Kang, Jinrui Gu, Junhan Lv +3Continual Learning for LLM Agents Continual Learning
State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia
CLaaS: Continual learning as a service for sample efficient online learning
Jun 4, 2026Kion Fallah, Silen Naihin, Barak Widawsky +1Continual Learning for LLM Agents Continual Learning
1Resolute Labs · 2South Park Commons · 3Incept Labs

Mar 2, 2026J/K move · Enter open · S save

Position: Modular Memory is the Key to Continual Learning Agents

Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov, Lucas Caccia+20

Toyota Motor Europe · University of Bremen · University of Florence+17

Continual Learning for LLM Agents Memory-Augmented Agents

Open paper AskPDF arXiv

Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in continuous operation, experience accumulation, and personalization, capabilities that are central to adaptive intelligence. While continual learning research has long targeted these goals, its historical focus on in-weight learning (IWL), i.e., updating a single model's parameters to absorb new knowledge, has rendered catastrophic forgetting a persistent challenge. Our position is that combining the strengths of In-Weight Learning (IWL) and the newly emerged capabilities of In-Context Learning (ICL) through the design of modular memory is the missing piece for continual adaptation at scale. We outline a conceptual framework for modular memory-centric architectures that leverage ICL for rapid adaptation and knowledge accumulation, and IWL for stable updates to model capabilities, charting a practical roadmap toward continually learning agents.

Abstract

Continual Learning for LLM Agents

Explore similar work

Position: Modular Memory is the Key to Continual Learning Agents

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

CLaaS: Continual learning as a service for sample efficient online learning