cs.LGFeb 25, 2026

Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns

Authors: Afshin Khadangi

Organizations: SnT, University of Luxembourg

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models deployed in the wild must adapt to evolving data, user behavior, and task mixtures without erasing previously acquired capabilities. In practice, this remains difficult: sequential updates induce catastrophic forgetting, while many stabilization methods rely on external procedures that are costly, brittle, or difficult to scale. We present TRC2^{2} (Thalamically Routed Cortical Columns), a decoder-only architecture that makes continual learning a property of the backbone itself. TRC2^{2} combines stacked cortical columns with a thalamic modulatory pathway for selective inter-column communication and a hippocampal pathway for event selective retrieval, delayed surprise-based writing, and replay-driven consolidation. This design localizes fast plasticity while preserving a slower stable computation pathway. We further introduce a causal memory-update scheme and an online replay controller that adjusts consolidation strength from measured forgetting. Across a task-sequential language-modeling stream over C4, WikiText-103, and GSM8K, TRC2^{2} consistently improves task-boundary modeling quality and substantially reduces cumulative forgetting relative to Transformer, Mamba, MoE, DeepSeek and continual learning baselines trained under the same pipeline. Ablations show that the thalamic and hippocampal components are central to the retention gains, while the full model remains competitive in throughput and training cost.

Figures & tables

Figure 1: Overview of the TRC 2 architecture and its main empirical trends.
Figure 2: Task-boundary perplexity versus model size. Points show means across 3 seeds and shaded bands show 95% confidence intervals. Lower values are better in all panels. C4 is reported at the 10k boundary, WikiText-103 at the 20k boundary, and GSM8K at the 22k boundary; the GSM8K axis is shown on a log scale.
Figure 3: Secondary teacher-forced text metrics versus model size. Points show means across 3 seeds and shaded bands show 95% confidence intervals. Higher values are better in all panels. For C4 and WikiText-103 we report BLEU at steps 10k and 20k, respectively; for GSM8K we report token accuracy at step 22k.
Figure 4: Efficiency frontier for task-boundary quality versus compute budget, using the Memory ×\times Hour/GPU statistic. Points show means across 3 seeds and error bars denote 95% confidence intervals. Higher values are better on all three panels.
Figure 5: Efficiency retention tradeoff measured by throughput and perplexity forgetting. Points show model configurations, the horizontal axis reports throughput in thousands of tokens per second, and the vertical axis reports PPL AUFC at step 22k, where lower values indicate better retention. Error bars denote 95% confidence intervals across seeds. The preferred region is the lower right corner, corresponding to higher throughput and lower cumulative forgetting.
Last-step scores at 22k
Thal.Hippo.PPL ↓\downarrowText score ↑\uparrowTokens/s ↑\uparrowMem×HourGPU↓\frac{\mathrm{Mem}\times\mathrm{Hour}}{\mathrm{GPU}}\downarrow
C4WikiGSMC4WikiGSM (%)
✓✓990.2668.7031.284.0515.0357.19∼\sim 103000134 GB⋅h134\,\mathrm{GB\cdot h}
✗✓1284.7955.2630.933.2315.5257.15∼\sim 109000131 GB⋅h131\,\mathrm{GB\cdot h}
✓✗2015.73613.2927.441.823.2857.74∼\sim 14700095 GB⋅h95\,\mathrm{GB\cdot h}
✗✗2050.02550.0826.792.304.5058.51∼\sim 15400087 GB⋅h87\,\mathrm{GB\cdot h}
Table 1: Ablation study on the TRC 2 d768-l4 configuration (top row in each panel). Rows are identified by the presence or absence of the thalamic and hippocampal modules. The upper panel reports the last logged held-out scores at step 22k for perplexity, BLEU, and GSM8K token accuracy. The lower panel reports area-under-forgetting-curve (AUFC) values at steps 20k and 22k.
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1 : Full overview of the TRC 2 architecture. Tokens are processed through stacked cortical columns. Intermediate cortical state drives a thalamic modulatory pathway and a hippocampal pathway with event-selective retrieval, delayed surprise based writing, and replay based consolidation. The thalamic pathway shapes subsequent cortical processing, while the hippocampal pathway injects retrieved contextual information into later columns and supports continual adaptation across the task stream.
ModelParamsddLLPPL ↓\downarrowTask-boundary text score ↑\uparrowTokens/s ↑\uparrowMem×HourGPU↓\frac{\mathrm{Mem}\times\mathrm{Hour}}{\mathrm{GPU}}\downarrow
C4WikiGSMC4WikiGSM (%)
Transformer144M5122893.1733.831.7e31.7{\mathrm{e}}37.9016.1551.42∼\sim 93000157 GB⋅h157\,\mathrm{GB\cdot h}
Transformer228M7682076.2729.942.8e32.8{\mathrm{e}}38.0616.7853.09∼\sim 84000202 GB⋅h202\,\mathrm{GB\cdot h}
Transformer321M10241670.6828.902.8e32.8{\mathrm{e}}37.7816.6953.41∼\sim 73000266 GB⋅h266\,\mathrm{GB\cdot h}
MoE345M5121699.2835.712.3e32.3{\mathrm{e}}37.3215.9651.01∼\sim 76000238 GB⋅h238\,\mathrm{GB\cdot h}
MoE441M7681094.3234.476.4e36.4{\mathrm{e}}36.8314.9150.09∼\sim 78000235 GB⋅h235\,\mathrm{GB\cdot h}
Appendix
Table D.1 : Task boundary performance and efficiency compared across baselines. For C4 and WikiText-103, we report the last score at the corresponding task boundary, namely steps 10k and 20k. For GSM8K, we report the last score at the final task boundary, step 22k. Tokens/s is the last logged throughput value. We report BLEU at the C4 and WikiText-103 task boundaries, and token accuracy on GSM8K. Here dd denotes model width and LL the number of backbone blocks; for TRC 2 these blocks are cortical columns.
ModelParamsddLLPPL AUFC ↓\downarrowBLEU AUFC ↓\downarrowTokAcc AUFC ↓\downarrowTokens/s ↑\uparrowMem×HourGPU↓\frac{\mathrm{Mem}\times\mathrm{Hour}}{\mathrm{GPU}}\downarrow
20k22k20k22k20k22k
Transformer144M512281.161.152.211.330.210.15∼\sim 93000157 GB⋅h157\,\mathrm{GB\cdot h}
Transformer228M768201.231.302.161.300.200.15∼\sim 84000202 GB⋅h202\,\mathrm{GB\cdot h}
Transformer321M1024161.151.301.881.160.190.14∼\sim 73000266 GB⋅h266\,\mathrm{GB\cdot h}
MoE345M512161.251.242.111.260.210.15∼\sim 76000238 GB⋅h238\,\mathrm{GB\cdot h}
MoE441M768101.181.291.921.150.200.14∼\sim 78000235 GB⋅h235\,\mathrm{GB\cdot h}
Appendix
Table D.2 : Continual learning performance at steps 20k and 22k. We report time-normalized area-under-forgetting-curve (AUFC) values. For perplexity, forgetting is measured in log-perplexity space; for BLEU and token accuracy, we additionally apply gain normalization when the required baseline denominator is available. Here dd denotes model width and LL the number of backbone blocks; for TRC 2 these blocks are cortical columns.
Figure D.2 : Efficiency frontier for task boundary quality versus compute budget, using the Memory ×\times Hour/GPU statistic reported in Table D.1 . Points show means across 3 seeds and error bars denote 95% confidence intervals. Higher values are better on all three panels. The figure highlights the quality-efficiency tradeoff across architectures at the task boundaries.
VariantParametersMemory
Col. (%)Hippo. (%)Thal. (%)Col. (%)Hippo. (%)Thal. (%)
TRC 2 d256-l2499.670.310.0188.5211.460.01
TRC 2 d512-l1099.240.720.0389.2210.750.03
TRC 2 d768-l497.941.960.0982.7817.140.08
TRC 2 d768-l898.960.990.0590.589.370.04
TRC 2 d1024-l698.671.270.0691.038.910.06
Appendix
Table E.3 : Static subsystem budget for TRC 2 variants. The parameter columns report the number of trainable parameters assigned to the three logged subsystems: cortical columns, thalamic pathway, and hippocampal pathway. The memory columns report the corresponding static device footprint in GB, computed from trainable parameter storage together with persistent buffer storage, with shared tensors counted only once. The separate token embedding and output head group used in optimization is not included in this table. Percentages are computed relative to the sum of the three logged subsystems shown here.
Figure E.3 : Hippocampal write dynamics across the task stream for TRC 2 variants. Vertical spikes in the traces mark task boundaries at 1010 k and 2020 k optimizer steps. Surprise statistics and the adaptive threshold τ\tau rise after task changes and then relax within task, while the keep fraction and raw write fraction remain sparse for most of training.
Figure E.4 : Additional auxiliary traces for TRC 2 variants. The logged router auxiliary is stable across the stream, while Ltd\mathcal{L}_{\mathrm{td}} and Lrep\mathcal{L}_{\mathrm{rep}} show transient increases but do not drift upward over time.
Figure F.5 : Verification summary for causality and state semantics in TRC 2 . The checks cover forward causality, gradient causality, prefix consistency, delayed write behavior, memory persistence, and replay buffer update semantics.
Figure H.6 : Per-dimension comparison of task boundary results grouped by model width dmd_{m} . Bars show means across 3 seeds and error bars show 95% confidence intervals. The top row reports perplexity, lower is better, and the GSM8K panel is shown on a log scale. The bottom row reports task boundary text metrics, higher is better.
Figure H.7 : Normalized multi-metric comparison using the best selected configuration per architecture. Outer radius is better on all axes, with perplexity axes inverted after normalization so that larger values always indicate stronger performance. The highlighted polygon corresponds to the TRC 2 408M configuration with dm=768d_{m}=768 and nb=8n_{b}=8 . This figure is intended as a qualitative aggregate summary, since the normalization compresses absolute metric differences.
Figure H.8 : Configuration level 95% confidence intervals across all task boundary results. Each point shows the mean over 3 seeds and horizontal bars denote 95% confidence intervals. Lower values are better for perplexity, whereas higher values are better for BLEU and GSM8K token accuracy. This view exposes uncertainty at the individual configuration level and complements the aggregated scale based comparisons in the main text.
Figure I.9 : Best PPL AUFC per architecture at steps 20k and 22k. Each bar reports the best performing configuration for that model family, with lower values indicating better retention. Error bars denote 95% confidence intervals across seeds. The summary highlights that TRC 2 attains the strongest retention profile at both checkpoints.
Figure J.10 : Cumulative forgetting trajectories across the five additional task-order shuffles. Each panel corresponds to one permutation of the original three-task stream, with shaded background bands indicating the task currently being trained and dashed vertical lines marking task boundaries. The vertical axis reports perplexity AUFC, where lower values indicate better retention. Across all five shuffled streams, TRC 2 remains in the lowest forgetting band throughout training, while most baselines accumulate substantially larger forgetting after task transitions and especially when a task is revisited late in the stream.
Figure J.11 : Order robustness of perplexity-based retention across shuffled task streams. Left: heatmap of PPL AUFC at step 2222 k for each model family and each of the five additional shuffles, with lower values indicating better retention. Top right: mean PPL AUFC across shuffles together with the min–max range for each model family. Bottom: ratio between the best baseline PPL AUFC and the corresponding TRC 2 PPL AUFC for each shuffle, where values above 11 indicate an advantage for TRC 2 . The figure shows that TRC 2 is not only best on average but also the most order-robust family under this stress test.
Figure J.12 : Multi-metric retention across shuffled task orders. The three heatmaps report PPL AUFC, BLEU AUFC, and token-accuracy AUFC across the five additional shuffles, with lower values better in every panel. The bar plot on the right summarizes the order-averaged retention profile of each model family for the three metrics. TRC 2 remains strongest across all three axes, indicating that its order robustness is not specific to one retention statistic but extends to both loss-based and text-based continual-learning summaries.
Figure J.13 : Per-task perplexity at task-window completion across the five additional shuffled streams. Each panel corresponds to one task, and each grouped set of bars corresponds to one shuffled order; the background color behind each group indicates whether that task appeared first, second, or third in the stream. The vertical axis is on a log scale. The main pattern is that several baselines deteriorate sharply when a task is encountered late or revisited after strong distribution shift, whereas TRC 2 remains in a comparatively low perplexity regime across all orders and all three tasks.
Figure K.14 : Task-boundary perplexity trajectories across training. Solid lines show the mean across runs, and shaded bands indicate the standard error.
Figure K.15 : Task boundary text metrics across training. Solid lines for C4 and WikiText-103 show Gaussian smoothed means across runs, and shaded bands indicate the standard error.
Figure K.16 : Continual learning retention measured by area under the forgetting curve. Solid lines show the mean across runs, and shaded bands indicate one standard deviation. Lower values are better.
More in this topic

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Explore similar work

CardsList
May 7, 2026

CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning

Large language models (LLMs) can learn new tasks through fine-tuning, but learning tasks sequentially often leads to catastrophic forgetting of earlier ones. This paper proposes CRAFT, a continual learning framework that keeps model weights frozen and instead learns low-rank interventions on hidden representations. Similar tasks share an intervention to help knowledge transfer, with similarity judged by how each task alters the model's output distribution rather than by a learned gating network. Within a shared intervention, a Kullback-Leibler (KL) penalty keeps each new task close to the intervention's existing behavior, which limits forgetting of earlier tasks, while a small, task-private intervention in the deepest layers preserves dedicated capacity that later tasks cannot overwrite. The learned interventions also make it possible to identify the task at test time, so task identity labels are not needed. Across multiple benchmarks and model scales, CRAFT improves overall performance and reduces forgetting compared with strong LoRA-based methods, while remaining robust to task ordering.

Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz +2

Iowa State University · Maro Systems, USA · University of California, Berkeley +1

Continual Learning for LLMsLow-Rank AdaptationIncremental LearningCatastrophic Forgetting
AskPDF
Sep 7, 2026

Continual Learning Mechanisms Compose for Long-Horizon Memorization

Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.

Zheyuan Zhang, Alvin Zhang, Daniel Khashabi +1

Johns Hopkins University

Continual Learning for LLMsMemorization in Language ModelsCatastrophic ForgettingLow-Rank Adaptation
AskPDF
Oct 15, 2026

Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models

Biological intelligence continually adapts to new information while retaining prior skills through localized plasticity. In contrast, continual learning in large language models (LLMs) remains vulnerable to catastrophic forgetting. A promising direction is to identify functionally specialized subnetworks within LLMs that enable targeted and resource-efficient adaptation. Using gradient-attribution profiling, we localize cognition-associated functional subnetworks exhibiting extremely low mutual overlap. Building on this finding, we propose Function-isolated Memory-Efficient Optimization (CiMEO), a continual learning framework that routes each training example to its corresponding functional subnetwork, restricting parameter updates and optimizer states to selected regions. Across 1.4 million sequential instructions, CiMEO achieves the highest average performance among the compared methods on five generative benchmarks spanning programming, examination-style tasks, and mathematical reasoning, while mitigating catastrophic forgetting. Subspace-overlap and principal-angle analyses further indicate that localized updates help preserve pre-trained representations, highlighting functional subnetwork localization as a practical approach to efficient and stable continual learning in LLMs.

Shuoran Jiang, Wentao Zhu, Yang Xiang +8

School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China · China Academy of Engineering Physics, Mianyang, China · Peng Cheng Laboratory, Shenzhen, China +2

Continual Learning for LLMsIncremental LearningCatastrophic Forgetting
AskPDF