Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns

Afshin Khadangi

cs.LGFeb 25, 2026

Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns

Authors: Afshin Khadangi

Organizations: SnT, University of Luxembourg

Abstract

Large language models deployed in the wild must adapt to evolving data, user behavior, and task mixtures without erasing previously acquired capabilities. In practice, this remains difficult: sequential updates induce catastrophic forgetting, while many stabilization methods rely on external procedures that are costly, brittle, or difficult to scale. We present TRC $^{2}$ (Thalamically Routed Cortical Columns), a decoder-only architecture that makes continual learning a property of the backbone itself. TRC $^{2}$ combines stacked cortical columns with a thalamic modulatory pathway for selective inter-column communication and a hippocampal pathway for event selective retrieval, delayed surprise-based writing, and replay-driven consolidation. This design localizes fast plasticity while preserving a slower stable computation pathway. We further introduce a causal memory-update scheme and an online replay controller that adjusts consolidation strength from measured forgetting. Across a task-sequential language-modeling stream over C4, WikiText-103, and GSM8K, TRC $^{2}$ consistently improves task-boundary modeling quality and substantially reduces cumulative forgetting relative to Transformer, Mamba, MoE, DeepSeek and continual learning baselines trained under the same pipeline. Ablations show that the thalamic and hippocampal components are central to the retention gains, while the full model remains competitive in throughput and training cost.

Figures & tables

Figure 1: Overview of the TRC 2 architecture and its main empirical trends.

Figure 2: Task-boundary perplexity versus model size. Points show means across 3 seeds and shaded bands show 95% confidence intervals. Lower values are better in all panels. C4 is reported at the 10k boundary, WikiText-103 at the 20k boundary, and GSM8K at the 22k boundary; the GSM8K axis is shown on a log scale.

Figure 3: Secondary teacher-forced text metrics versus model size. Points show means across 3 seeds and shaded bands show 95% confidence intervals. Higher values are better in all panels. For C4 and WikiText-103 we report BLEU at steps 10k and 20k, respectively; for GSM8K we report token accuracy at step 22k.

Figure 4: Efficiency frontier for task-boundary quality versus compute budget, using the Memory $\times$ Hour/GPU statistic. Points show means across 3 seeds and error bars denote 95% confidence intervals. Higher values are better on all three panels.

Figure 5: Efficiency retention tradeoff measured by throughput and perplexity forgetting. Points show model configurations, the horizontal axis reports throughput in thousands of tokens per second, and the vertical axis reports PPL AUFC at step 22k, where lower values indicate better retention. Error bars denote 95% confidence intervals across seeds. The preferred region is the lower right corner, corresponding to higher throughput and lower cumulative forgetting.

Last-step scores at 22k
Thal.	Hippo.	PPL $\downarrow$			Text score $\uparrow$			Tokens/s $\uparrow$	$\frac{\mathrm{Mem}\times\mathrm{Hour}}{\mathrm{GPU}}\downarrow$
Thal.	Hippo.	C4	Wiki	GSM	C4	Wiki	GSM (%)	Tokens/s $\uparrow$
✓	✓	990.26	68.70	31.28	4.05	15.03	57.19	$\sim$ 103000	$134\,\mathrm{GB\cdot h}$
✗	✓	1284.79	55.26	30.93	3.23	15.52	57.15	$\sim$ 109000	$131\,\mathrm{GB\cdot h}$
✓	✗	2015.73	613.29	27.44	1.82	3.28	57.74	$\sim$ 147000	$95\,\mathrm{GB\cdot h}$
✗	✗	2050.02	550.08	26.79	2.30	4.50	58.51	$\sim$ 154000	$87\,\mathrm{GB\cdot h}$

Table 1: Ablation study on the TRC 2 d768-l4 configuration (top row in each panel). Rows are identified by the presence or absence of the thalamic and hippocampal modules. The upper panel reports the last logged held-out scores at step 22k for perplexity, BLEU, and GSM8K token accuracy. The lower panel reports area-under-forgetting-curve (AUFC) values at steps 20k and 22k.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1 : Full overview of the TRC 2 architecture. Tokens are processed through stacked cortical columns. Intermediate cortical state drives a thalamic modulatory pathway and a hippocampal pathway with event-selective retrieval, delayed surprise based writing, and replay based consolidation. The thalamic pathway shapes subsequent cortical processing, while the hippocampal pathway injects retrieved contextual information into later columns and supports continual adaptation across the task stream.

Model	Params	$d$	$L$	PPL $\downarrow$			Task-boundary text score $\uparrow$			Tokens/s $\uparrow$	$\frac{\mathrm{Mem}\times\mathrm{Hour}}{\mathrm{GPU}}\downarrow$
Model	Params	$d$	$L$	C4	Wiki	GSM	C4	Wiki	GSM (%)	Tokens/s $\uparrow$
Transformer	144M	512	28	93.17	33.83	$1.7{\mathrm{e}}3$	7.90	16.15	51.42	$\sim$ 93000	$157\,\mathrm{GB\cdot h}$
Transformer	228M	768	20	76.27	29.94	$2.8{\mathrm{e}}3$	8.06	16.78	53.09	$\sim$ 84000	$202\,\mathrm{GB\cdot h}$
Transformer	321M	1024	16	70.68	28.90	$2.8{\mathrm{e}}3$	7.78	16.69	53.41	$\sim$ 73000	$266\,\mathrm{GB\cdot h}$
MoE	345M	512	16	99.28	35.71	$2.3{\mathrm{e}}3$	7.32	15.96	51.01	$\sim$ 76000	$238\,\mathrm{GB\cdot h}$
MoE	441M	768	10	94.32	34.47	$6.4{\mathrm{e}}3$	6.83	14.91	50.09	$\sim$ 78000	$235\,\mathrm{GB\cdot h}$

Appendix

Table D.1 : Task boundary performance and efficiency compared across baselines. For C4 and WikiText-103, we report the last score at the corresponding task boundary, namely steps 10k and 20k. For GSM8K, we report the last score at the final task boundary, step 22k. Tokens/s is the last logged throughput value. We report BLEU at the C4 and WikiText-103 task boundaries, and token accuracy on GSM8K. Here

d

denotes model width and

L

the number of backbone blocks; for TRC 2 these blocks are cortical columns.

Model	Params	$d$	$L$	PPL AUFC $\downarrow$		BLEU AUFC $\downarrow$		TokAcc AUFC $\downarrow$		Tokens/s $\uparrow$	$\frac{\mathrm{Mem}\times\mathrm{Hour}}{\mathrm{GPU}}\downarrow$
Model	Params	$d$	$L$	20k	22k	20k	22k	20k	22k	Tokens/s $\uparrow$
Transformer	144M	512	28	1.16	1.15	2.21	1.33	0.21	0.15	$\sim$ 93000	$157\,\mathrm{GB\cdot h}$
Transformer	228M	768	20	1.23	1.30	2.16	1.30	0.20	0.15	$\sim$ 84000	$202\,\mathrm{GB\cdot h}$
Transformer	321M	1024	16	1.15	1.30	1.88	1.16	0.19	0.14	$\sim$ 73000	$266\,\mathrm{GB\cdot h}$
MoE	345M	512	16	1.25	1.24	2.11	1.26	0.21	0.15	$\sim$ 76000	$238\,\mathrm{GB\cdot h}$
MoE	441M	768	10	1.18	1.29	1.92	1.15	0.20	0.14	$\sim$ 78000	$235\,\mathrm{GB\cdot h}$

Appendix

Table D.2 : Continual learning performance at steps 20k and 22k. We report time-normalized area-under-forgetting-curve (AUFC) values. For perplexity, forgetting is measured in log-perplexity space; for BLEU and token accuracy, we additionally apply gain normalization when the required baseline denominator is available. Here

d

denotes model width and

L

the number of backbone blocks; for TRC 2 these blocks are cortical columns.

Figure D.2 : Efficiency frontier for task boundary quality versus compute budget, using the Memory $\times$ Hour/GPU statistic reported in Table D.1 . Points show means across 3 seeds and error bars denote 95% confidence intervals. Higher values are better on all three panels. The figure highlights the quality-efficiency tradeoff across architectures at the task boundaries.

Variant	Parameters			Memory
Variant	Col. (%)	Hippo. (%)	Thal. (%)	Col. (%)	Hippo. (%)	Thal. (%)
TRC 2 d256-l24	99.67	0.31	0.01	88.52	11.46	0.01
TRC 2 d512-l10	99.24	0.72	0.03	89.22	10.75	0.03
TRC 2 d768-l4	97.94	1.96	0.09	82.78	17.14	0.08
TRC 2 d768-l8	98.96	0.99	0.05	90.58	9.37	0.04
TRC 2 d1024-l6	98.67	1.27	0.06	91.03	8.91	0.06

Appendix

Table E.3 : Static subsystem budget for TRC 2 variants. The parameter columns report the number of trainable parameters assigned to the three logged subsystems: cortical columns, thalamic pathway, and hippocampal pathway. The memory columns report the corresponding static device footprint in GB, computed from trainable parameter storage together with persistent buffer storage, with shared tensors counted only once. The separate token embedding and output head group used in optimization is not included in this table. Percentages are computed relative to the sum of the three logged subsystems shown here.

Panel of Figure 12 — Figure E.3 : Hippocampal write dynamics across the task stream for TRC 2 variants. Vertical spikes in the traces mark task boundaries at $10$ k and $20$ k optimizer steps. Surprise statistics and the adaptive threshold $\tau$ rise after task changes and then relax within task, while the keep fraction and raw write fraction remain sparse for most of training.

Panel of Figure 13 — Figure E.4 : Additional auxiliary traces for TRC 2 variants. The logged router auxiliary is stable across the stream, while $\mathcal{L}_{\mathrm{td}}$ and $\mathcal{L}_{\mathrm{rep}}$ show transient increases but do not drift upward over time.

Figure F.5 : Verification summary for causality and state semantics in TRC 2 . The checks cover forward causality, gradient causality, prefix consistency, delayed write behavior, memory persistence, and replay buffer update semantics.

Figure H.6 : Per-dimension comparison of task boundary results grouped by model width $d_{m}$ . Bars show means across 3 seeds and error bars show 95% confidence intervals. The top row reports perplexity, lower is better, and the GSM8K panel is shown on a log scale. The bottom row reports task boundary text metrics, higher is better.

Figure H.7 : Normalized multi-metric comparison using the best selected configuration per architecture. Outer radius is better on all axes, with perplexity axes inverted after normalization so that larger values always indicate stronger performance. The highlighted polygon corresponds to the TRC 2 408M configuration with $d_{m}=768$ and $n_{b}=8$ . This figure is intended as a qualitative aggregate summary, since the normalization compresses absolute metric differences.

Figure H.8 : Configuration level 95% confidence intervals across all task boundary results. Each point shows the mean over 3 seeds and horizontal bars denote 95% confidence intervals. Lower values are better for perplexity, whereas higher values are better for BLEU and GSM8K token accuracy. This view exposes uncertainty at the individual configuration level and complements the aggregated scale based comparisons in the main text.

Figure I.9 : Best PPL AUFC per architecture at steps 20k and 22k. Each bar reports the best performing configuration for that model family, with lower values indicating better retention. Error bars denote 95% confidence intervals across seeds. The summary highlights that TRC 2 attains the strongest retention profile at both checkpoints.

Figure J.10 : Cumulative forgetting trajectories across the five additional task-order shuffles. Each panel corresponds to one permutation of the original three-task stream, with shaded background bands indicating the task currently being trained and dashed vertical lines marking task boundaries. The vertical axis reports perplexity AUFC, where lower values indicate better retention. Across all five shuffled streams, TRC 2 remains in the lowest forgetting band throughout training, while most baselines accumulate substantially larger forgetting after task transitions and especially when a task is revisited late in the stream.

Figure J.11 : Order robustness of perplexity-based retention across shuffled task streams. Left: heatmap of PPL AUFC at step $22$ k for each model family and each of the five additional shuffles, with lower values indicating better retention. Top right: mean PPL AUFC across shuffles together with the min–max range for each model family. Bottom: ratio between the best baseline PPL AUFC and the corresponding TRC 2 PPL AUFC for each shuffle, where values above $1$ indicate an advantage for TRC 2 . The figure shows that TRC 2 is not only best on average but also the most order-robust family under this stress test.

Figure J.12 : Multi-metric retention across shuffled task orders. The three heatmaps report PPL AUFC, BLEU AUFC, and token-accuracy AUFC across the five additional shuffles, with lower values better in every panel. The bar plot on the right summarizes the order-averaged retention profile of each model family for the three metrics. TRC 2 remains strongest across all three axes, indicating that its order robustness is not specific to one retention statistic but extends to both loss-based and text-based continual-learning summaries.

Figure J.13 : Per-task perplexity at task-window completion across the five additional shuffled streams. Each panel corresponds to one task, and each grouped set of bars corresponds to one shuffled order; the background color behind each group indicates whether that task appeared first, second, or third in the stream. The vertical axis is on a log scale. The main pattern is that several baselines deteriorate sharply when a task is encountered late or revisited after strong distribution shift, whereas TRC 2 remains in a comparatively low perplexity regime across all orders and all three tasks.

Panel of Figure 23 — Figure K.14 : Task-boundary perplexity trajectories across training. Solid lines show the mean across runs, and shaded bands indicate the standard error.

Panel of Figure 24 — Figure K.15 : Task boundary text metrics across training. Solid lines for C4 and WikiText-103 show Gaussian smoothed means across runs, and shaded bands indicate the standard error.

Panel of Figure 25 — Figure K.16 : Continual learning retention measured by area under the forgetting curve. Solid lines show the mean across runs, and shaded bands indicate one standard deviation. Lower values are better.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning
May 7, 2026Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz +2Continual Learning for LLMs Low-Rank Adaptation
Iowa State University · Maro Systems, USA · University of California, Berkeley +1
Continual Learning Mechanisms Compose for Long-Horizon Memorization
Sep 7, 2026Zheyuan Zhang, Alvin Zhang, Daniel Khashabi +1Continual Learning for LLMs Memorization in Language Models
Johns Hopkins University
Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models
Oct 15, 2026Shuoran Jiang, Wentao Zhu, Yang Xiang +8Continual Learning for LLMs Incremental Learning
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China · China Academy of Engineering Physics, Mianyang, China · Peng Cheng Laboratory, Shenzhen, China +2

May 7, 2026J/K move · Enter open · S save

CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning

Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz, Tanya Roosta+1

Iowa State University · Maro Systems, USA · University of California, Berkeley+1

Continual Learning for LLMs Low-Rank Adaptation Incremental Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Large language models (LLMs) can learn new tasks through fine-tuning, but learning tasks sequentially often leads to catastrophic forgetting of earlier ones. This paper proposes CRAFT, a continual learning framework that keeps model weights frozen and instead learns low-rank interventions on hidden representations. Similar tasks share an intervention to help knowledge transfer, with similarity judged by how each task alters the model's output distribution rather than by a learned gating network. Within a shared intervention, a Kullback-Leibler (KL) penalty keeps each new task close to the intervention's existing behavior, which limits forgetting of earlier tasks, while a small, task-private intervention in the deepest layers preserves dedicated capacity that later tasks cannot overwrite. The learned interventions also make it possible to identify the task at test time, so task identity labels are not needed. Across multiple benchmarks and model scales, CRAFT improves overall performance and reduces forgetting compared with strong LoRA-based methods, while remaining robust to task ordering.

Abstract

Continual Learning for LLMs

Explore similar work

CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning

Continual Learning Mechanisms Compose for Long-Horizon Memorization

Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models