Cortex: Consolidating Context into Parametric Memory

Jintian Zhang; Feng He; Xinnian Liang; Ziwen Xu; Haoming Xu; Haojing Huang; Shuangzhi Wu; Ningyu Zhang

cs.CLOct 14, 2026

Cortex: Consolidating Context into Parametric Memory

Authors: Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang

Organizations: Zhejiang University · ByteDance Seed · Tsinghua University

Abstract

Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 236 papers

Self-Consolidating Language Models: Continual Knowledge Incorporation from Context
May 8, 2026Zekun Wang, Anant Gupta, Zihan Dong +1Continual Learning for LLMs Long-Context Language Modeling
Georgia Institute of Technology
Cache2LoRA: Understanding and Improving Context Internalization
Oct 16, 2026Yi Fang, Gaurav Srivastava, Zhenyu Bi +2LLM Context Compression Knowledge Augmentation for Language Models
Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning
Oct 2, 2026Yehya Farhat, Michael Desmond, Anastasios KyrillidisContinual Learning for LLM Agents Continual Learning for LLMs
Rice University, Department of Computer Science Houston, TX, USA · IBM Research Yorktown Heights, NY, USA

May 8, 2026J/K move · Enter open · S save

Self-Consolidating Language Models: Continual Knowledge Incorporation from Context

Zekun Wang, Anant Gupta, Zihan Dong, Christopher J. MacLellan

Georgia Institute of Technology

Continual Learning for LLMs Long-Context Language Modeling Incremental Learning LLM Post-Training

Open paper AskPDF arXiv

Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context workflows. While longer context windows expose more evidence, they do not ensure that useful information is preserved and reused. We study continual context consolidation: writing current context into model weights while limiting interference with previously consolidated information. We propose Self-Consolidating Language Models (SCoL), a post-training framework in which, given current context, an LLM learns to generate textual update instructions specifying which of its own Transformer layers should be updated. Because committed updates change the model that later generates future selections, we train SCoL with meta-reinforcement learning over an evolving model state. We instantiate SCoL with supervised QA rewards on SQuAD knowledge incorporation and intrinsic likelihood-based rewards for LongBench v2 long-context consolidation. Across both settings, SCoL improves acquisition and retention over prompting, summarization, batch test-time training, and sequential finetuning baselines. Analysis of learned selection patterns shows that SCoL encourages the LLM to generate sparse update locations that align with layers of high Fisher information, suggesting that the model learns to route plasticity toward loss-sensitive regions while limiting interference. Moreover, SCoL transfers from shorter meta-training streams to longer LongBench v2 streams at evaluation, suggesting that our framework supports scalable streaming consolidation.

Abstract

Continual Learning for LLMs

Explore similar work

Self-Consolidating Language Models: Continual Knowledge Incorporation from Context

Cache2LoRA: Understanding and Improving Context Internalization

Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning