Routing without Forgetting

Alessio Masano; Giovanni Bellitto; Dipam Goswani; Joost Van de Weijer; Concetto Spampinato

cs.LGMar 10, 2026

Routing without Forgetting

Authors: Alessio Masano, Giovanni Bellitto, Dipam Goswani, Joost Van de Weijer, Concetto Spampinato

Organizations: University of Catania · Department of Computer Science, Universitat Autònoma de Barcelona Computer Vision Center, Barcelona

Abstract

Continual learning in transformers is commonly addressed through parameter-efficient adaptation: prompts, adapters, or LoRA modules are specialized per task while the backbone remains frozen. Although effective in controlled multi-epoch settings, these approaches rely on gradual gradient-based specialization and struggle in Online Continual Learning (OCL), where data arrive as a non-stationary stream and each sample may be observed only once. We recast continual learning in transformers as a routing problem: under strict online constraints, the model must dynamically select the appropriate representational subspace for each input without explicit task identifiers or repeated optimization. We thus introduce Routing without Forgetting (RwF), a transformer architecture augmented with energy-based associative retrieval layers inspired by Modern Hopfield Networks. Instead of storing or merging task-specific prompts, RwF generates dynamic prompts through single-step associative retrieval over the transformer token embeddings at each layer. Retrieval corresponds to the closed-form minimization of a strictly convex free-energy functional, enabling input-conditioned routing within each forward pass, independently of iterative gradient refinement. Across challenging class-incremental benchmarks, RwF improves over existing prompt-based methods. On Split-ImageNet-R and Split-ImageNet-S, RwF outperforms prior prompt-based approaches by a large margin, even in few-shot learning regimes. These results indicate that embedding energy-based associative routing directly within the transformer backbone provides a principled and effective foundation for OCL.

Figures & tables

Figure 1: RwF layer: routing-augmented transformer block. Given input tokens $Z_{\ell}$ , a Hopfield-based associative retrieval module generates input-conditioned routing prompts $P_{\ell}$ via energy-based pooling over token features. The retrieved $P_{\ell}$ are concatenated with the $Z_{\ell}$ tokens and processed by the standard Multi-Head Self-Attention (MHSA). After MHSA, only the backbone tokens $\tilde{Z}_{\ell}$ are propagated to the MLP blocks and then to the next RwF transformer layer $\ell+1$ , while $\tilde{P}_{\ell}$ are discarded.

	Split-CIFAR100		Split-ImageNet-R		Split-ImageNet-S
Method	$A_{\text{Final}}\,(\uparrow)$	Forgetting $(\downarrow)$	$A_{\text{Final}}\,(\uparrow)$	Forgetting $(\downarrow)$	$A_{\text{Final}}\,(\uparrow)$	Forgetting $(\downarrow)$
Joint	89.50 $\pm$ 0.04	–	76.78 $\pm$ 0.44	–	63.82 $\pm$ 0.02	–
Finetune	15.83 $\pm$ 0.25	79.68 $\pm$ 0.08	12.06 $\pm$ 0.34	79.61 $\pm$ 0.04	7.09 $\pm$ 0.08	26.45 $\pm$ 0.14
AGEM [ 11 ]	12.67 $\pm$ 1.87	82.51 $\pm$ 2.27	5.60 $\pm$ 2.42	53.97 $\pm$ 1.97	0.16 $\pm$ 0.04	9.42 $\pm$ 0.17
ER [ 12 ]	44.85 $\pm$ 1.83	44.67 $\pm$ 4.29	40.99 $\pm$ 3.96	32.38 $\pm$ 0.89	30.21 $\pm$ 0.70	37.14 $\pm$ 1.83
EWC++ [ 19 ]	10.61 $\pm$ 0.74	84.10 $\pm$ 1.11	3.86 $\pm$ 2.02	36.95 $\pm$ 1.46	0.32 $\pm$ 0.28	22.46 $\pm$ 4.69

Table 1: OCL performance comparison. Final Average Accuracy

A_{\text{Final}}(\uparrow)

and Forgetting

(\downarrow)

on Split-CIFAR100, Split-ImageNet-R, and Split-ImageNet-S. Best: bold, second: underline, third: italic. RwF achieves top performance on large-scale ImageNet benchmarks while ranking competitively on CIFAR100.

	Split-ImageNet-R
Method	100%	70%	50%	20%
L2P [ 2 ]	57.97 $\pm$ 0.31	56.15 $\pm$ 0.13	53.71 $\pm$ 0.32	44.79 $\pm$ 1.14
CODA-Prompt [ 4 ]	66.16 $\pm$ 0.39	64.60 $\pm$ 1.04	61.63 $\pm$ 0.28	54.79 $\pm$ 1.16
DualPrompt [ 3 ]	60.88 $\pm$ 0.62	58.67 $\pm$ 0.81	56.87 $\pm$ 0.33	47.68 $\pm$ 0.52
InfLoRA [ 8 ]	62.20 $\pm$ 0.87	51.94 $\pm$ 0.67	37.90 $\pm$ 0.98	6.65 $\pm$ 0.08
APER [ 5 ]	55.12 $\pm$ 0.02	55.04 $\pm$ 0.27	54.16 $\pm$ 0.45	49.72 $\pm$ 0.13

Table 2: Performance under few-shot data regimes (Split-ImageNet-R). Final Average Accuracy (

A_{\text{Final}}(\uparrow)

) as the fraction of available training samples decreases from 100% to 20%. Best: bold, second: underline, third: italic. RwF maintains the highest accuracy across all evaluated data regimes.

Figure 2: Performance scaling with increasing task fragmentation (Split-ImageNet-R). Final Average Accuracy ( $\mathrm{A}_{\text{Final}}$ ) $(\uparrow)$ as the number of sequential tasks $t$ increases from 5 to 40.

$k$	%	Split-CIFAR100		Split-ImageNet-R		Split-ImageNet-S
$k$	%	First- $k$	Last- $k$	First- $k$	Last- $k$	First- $k$	Last- $k$
0	0.0	15.83 $\pm$ 0.25		12.06 $\pm$ 0.34		7.09 $\pm$ 0.08
1	0.7	79.57 $\pm$ 0.54	65.98 $\pm$ 0.73	67.78 $\pm$ 1.02	60.10 $\pm$ 0.61	58.36 $\pm$ 0.55	49.19 $\pm$ 0.60
3	2.1	82.48 $\pm$ 0.31	60.81 $\pm$ 4.12	74.09 $\pm$ 0.29	55.92 $\pm$ 0.61	61.37 $\pm$ 0.43	48.88 $\pm$ 0.62
5	3.5	82.89 $\pm$ 0.47	61.56 $\pm$ 3.16	74.68 $\pm$ 0.04	60.64 $\pm$ 2.00	62.19 $\pm$ 0.48	51.83 $\pm$ 0.29
7	4.9	81.65 $\pm$ 1.06	64.77 $\pm$ 0.59	75.39 $\pm$ 0.17	63.95 $\pm$ 0.52	62.88 $\pm$ 0.57	53.83 $\pm$ 0.42

Table 3: Effect of routing depth on Split-CIFAR100, Split-ImageNet-R and Split-ImageNet-S. Final Average Accuracy (

\mathrm{A}_{\text{Final}}(\uparrow)

) as the number of HopfieldPooling (HP) layers

k

increases. First-

k

inserts HP layers in the first

k

transformer block; Last-

k

inserts them in last

k

blocks (

12\dots 12-k+1

). Best: bold, second: underline, third: italic. Green cells indicate chosen configuration used for all paper results.

Model	# params
L2P [ 2 ]	0.47%
DualPrompt [ 3 ]	0.58%
CODA-Prompt [ 4 ]	5.00%
InfLoRA [ 8 ]	1.70%
APER [ 5 ]	1.50%
EASE [ 6 ]	1.32%

Table 4: Trainable parameters (%) relative to the full ViT backbone. RwF maintains a low parameter footprint, comparable to other parameter-efficient CL methods.

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 106 papers

Information-Theoretic Decoupled Prompt Tuning for Continual Learning
Aug 13, 2026Yunfei Zhang, Wen Wen, Tieliang Gong +1Continual Learning for VLMs Continual Learning for LLMs
National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
Is Prompt Selection Necessary for Task-Free Online Continual Learning?
Apr 6, 2026Seoyoung Park, Haemin Lee, Hankook LeeTask-Free Continual Learning Continual Learning
Sungkyunkwan University
Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning
May 6, 2026William T. Redman, Erik C. Johnson, Brian RobinsonCompositional Reasoning Continual Learning
Johns Hopkins Applied Physics Lab

Aug 13, 2026J/K move · Enter open · S save

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Yunfei Zhang, Wen Wen, Tieliang Gong, Weizhan Zhang

National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China

Continual Learning for VLMs Continual Learning for LLMs VLM Adaptation

Open paper AskPDF arXiv

Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning has attracted increasing attention as an efficient approach for adapting pre-trained models to CL tasks. However, existing prompt design paradigms commonly suffer from retrieval dependence and classifier bias, which make model adaptation sensitive to prompt selection and bias predictions toward newly arrived classes. To address these challenges, we propose Decoupled Prompt Tuning for Continual Learning (DPT4CL), which decouples the CLIP textual prompt into a task-shared prompt distribution and class-specific prompts. The task-shared prompt distribution is derived by optimizing an Information Bottleneck objective to facilitate cross-task knowledge transfer and alleviate classifier bias, while class-specific prompts enhance inter-class separability without relying on explicit prompt retrieval. Furthermore, we establish a unified excess risk bound from an information-theoretic perspective, providing theoretical support for the robust generalization and forgetting mitigation of the proposed framework. Extensive experiments on standard CL benchmarks demonstrate that DPT4CL achieves state-of-the-art performance. The source code is available at https://github.com/Cloudfly-Z/DPT4CL

Abstract

Online Continual Learning

Explore similar work

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Is Prompt Selection Necessary for Task-Free Online Continual Learning?

Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning