Position: Modular Memory is the Key to Continual Learning Agents
Authors: Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov, Lucas Caccia, Antonio Carta, Laurent Charlin, Barbara Hammer, Tyler L. Hayes, +16 more
Organizations: Toyota Motor Europe · University of Bremen · University of Florence · Microsoft Research · Università di Pisa · HEC Montreal, Mila–Quebec AI Institute, Canada CIFAR AI Chair · Bielefeld University · Georgia Institute of Technology · KU Leuven · University of Rochester · University of Texas at San Antonio · Nankai University · LUISS University · Stony Brook University · University of Tübingen · University of Trento, FBK · University of Groningen · Tsinghua University · Computer Vision Center Barcelona · Seoul National University
Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in continuous operation, experience accumulation, and personalization, capabilities that are central to adaptive intelligence. While continual learning research has long targeted these goals, its historical focus on in-weight learning (IWL), i.e., updating a single model's parameters to absorb new knowledge, has rendered catastrophic forgetting a persistent challenge. Our position is that combining the strengths of In-Weight Learning (IWL) and the newly emerged capabilities of In-Context Learning (ICL) through the design of modular memory is the missing piece for continual adaptation at scale. We outline a conceptual framework for modular memory-centric architectures that leverage ICL for rapid adaptation and knowledge accumulation, and IWL for stable updates to model capabilities, charting a practical roadmap toward continually learning agents.
Figures & tables
Figure 1: An illustration of the role of memory in unlocking continual learning through the construction of distinct modules. The framework comprises (1) a core model for perception and reasoning, (2) a working memory module for temporarily storing information relevant to the current interaction round, and (3) a long-term memory module that accumulates extracted experiences, facts, and observations. Knowledge stored in long-term memory is selectively retrieved into working memory and consolidated into the core model, steadily improving its capabilities.
Memory Type
Per-item Storage
Capacity
Modality
Update Speed
Interference Risk
Selective Forgetting
Retrieval
Generalization
Slot-based Memory
↳ Raw Data (summaries, exemplars, coresets)
Low
Unbounded
Uni-modal
Fast
Low
Easy
Grows with #items
Limited
↳ Embeddings (KV cache, activations, learned)
Medium
Unbounded
Uni/ Multi-modal
Fast
Low
Easy
Grows with #items
Implicit (learned encodings)
Neural Memory ( e.g. , DNNs, Hopfield, SDM)
High
Bounded
Uni/ Multi-modal
Slow/ Fast
High
Difficult
Depends on network size
Explicit (shared params)
Table 1: Desiderata and assessment of current memory representations (as defined in Sec. 4.1 ). Storage is more compact for neural memory while Capacity is unbounded for slot-based, raw data is typically unimodal, update speed is varied based on specific design choices, interference is high in neural memory, selective forgetting is easier for slot based. Finally generalization is poor in raw-data, implicitly encoded in embeddings, while explicitly optimized in neural memory.
Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.
Borui Kang, Jinrui Gu, Junhan Lv +3
State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia
Achieving self-evolution in intelligent agents requires the continual accumulation of new knowledge across changing task sequences without forgetting previously acquired abilities. Existing approaches either internalize knowledge by updating model parameters, which induces catastrophic forgetting, or rely on external memory, which fails to genuinely enhance the model's intrinsic capabilities. We propose MoLEM, a generative mixture of latent memory framework based on a dynamic mixture-of-experts (MoE). We treat multiple experts as independent carriers to generate memory. A router selects and weights experts through key-query matching, and the aggregated latent memory is injected into the reasoning process. The base model for reasoning remains entirely frozen, with all experiential knowledge internalized into the additional modules, avoiding catastrophic forgetting. For continual learning, each training stage is paired with a lightweight autoencoder that selects the appropriate routing group at inference, and inputs that match no stage fall back to the pretrained model. Experiments train the framework on continual-learning sequences spanning math, science, and code domains. After training, we evaluate the framework on the corresponding test sets to measure task learning and competence preservation across continual adaptation stages. After the full continual-learning sequence, our method improves the average accuracy by 10.40% over the Vanilla pretrained baseline, while none of the competing methods consistently exceed this baseline across different training orders.
Dianzhi Yu, Vireo Zhang, Hongru Wang +7
The Chinese University of Hong Kong · Nanyang Technological University · University of Edinburgh +2
In this paper, I formulate a Continual Learning problem and propose a method named "Homeostatic Continual Learning" that enables an AI agent to learn continuously in a changing environment without catastrophic forgetting. The core of the method is to find outliers in the environment data when the agent experiences an outlier in its output. Through this method, the agent gradually completes its model and policy and performs well in more and more contexts. I also suggest that we may use the method to build a world model where the agent factorizes the objects in the world into features, abstract objects into comparable instances of concepts and map concepts to intents through features. I discuss the works needed to render the method practical, the connections to many fields in Artificial Intelligence and the broader implications of the method.