Anchor Behavior, Not Parameters: Rehearsal-Free Continual Learning with a Single Shared Prompt
Authors: Duy-Cuong Bui, Trung-Anh Dang, Vincent Nguyen, Duc V. Le, Ngoc-Son Vu
Organizations: ETIS - CY Cergy Paris Université, ENSEA, CNRS UMR 8051, France · LIFO UR 4022, Université d’Orléans, INSA CVL, France · University of Twente, Netherlands · LIST3N - Université de Technologie de Troyes, France
Prompt-based continual learning adapts a frozen pre-trained vision transformer by tuning a small set of prompt tokens. Recently, one-prompt methods which share one single prompt across all tasks make this paradigm maximally parameter efficient. The open question is how a shared prompt can keep the knowledge of old classes while being updated for new ones. Existing methods protect old knowledge in parameter space, where they freeze task-specific prompts, expand new components, or sparsely gate which tokens receive gradients. However, this method sacrifices plasticity, yet the drift it still permits in shared tokens can corrupt the prompt's function on past classes. What must be preserved is the prompt's behavior. We propose TEA-Prompt, which couples (i) a token evaluation module that assigns each prompt token a continuous, input-conditional relevance score injected into the attention logits, softly specializing tokens across tasks without discarding any of them, and (ii) an attention-behavior anchoring loss that constrains each token's attention output around class-specific query regions using only compact Gaussian query statistics without stored old images, weighted by the token's past relevance. Anchoring the attention value allows the parameters to continue updating freely, as long as their interaction remains unchanged. TEA-Prompt achieves state-of-the-art accuracy with markedly lower forgetting on CIFAR-100, ImageNet-R, and CUB-200 with only 0.38M parameters. Code is publicy available at https://github.com/cuonga1cvp/TEA-Prompt.git
Figures & tables
Figure 1: Overview of TEA-Prompt. Left: Inside each MSA layer of the frozen ViT, prompt tokens (PK,PV) are prepended to the frozen keys and values. The reduced query scores every prompt token (Eq. ( 2 )); a thresholded sigmoid turns the scores into evaluation weights (Eq. ( 3 )) whose logarithm shifts the prompt attention logits Lprompt (Eq. ( 4 )), modulating each token’s attention mass before the joint softmax with the input-token logits Lpre-trained (Eq. ( 5 )). Right: At the end of task t−1 the prompt is snapshotted and class-conditional Gaussian query statistics are stored (Eq. ( 6 )); while learning task t , queries sampled from these statistics (Eq. ( 7 )) probe the prompt attention of the frozen snapshot ( A(t−1) ) and of the current prompt ( A(t) ), and LAD (Eq. ( 9 )) anchors the current behavior to the old one, weighted by each token’s past relevance.
Method
Params (M)
CIFAR-100
ImageNet-R
CUB-200
Avg Acc( ↑ )
Forgetting( ↓ )
Avg Acc( ↑ )
Forgetting( ↓ )
Avg Acc( ↑ )
Forgetting( ↓ )
Joint training
–
91.38
–
82.06
–
88.00
–
LAE [ 13 ]
0.24
86.09 ± 0.55
6.22 ± 0.17
73.88 ± 0.43
5.08 ± 0.33
–
–
InfLoRA [ 27 ]
3.84
86.91 ± 0.34
4.22 ± 0.37
76.13 ± 0.44
5.22 ± 0.48
–
–
EASE [ 49 ]
11.90
87.28 ± 0.71
4.81 ± 0.42
75.82 ± 0.54
5.83 ± 0.37
–
–
L2P [ 45 ]
0.24
83.86 ± 0.28
7.35 ± 0.38
61.57 ± 0.66
9.73 ± 0.47
71.22 ± 0.54
10.68 ± 0.61
Table 1: Performance comparison on CIFAR-100, ImageNet-R, and CUB-200 datasets. Params counts the learnable parameters on ImageNet-R with 10 tasks. ( ”-” means Not Available)
Pre-trained model
Method
CIFAR-100
ImageNet-R
iBOT-1K
L2P
75.57±0.41
60.97±0.26
DualPrompt
76.63±0.05
61.51±1.05
CODA-Prompt
76.42±0.40
66.56±0.68
RainbowPrompt
76.20±0.34
65.82±0.52
VQ-Prompt
75.80±0.34
71.68±0.72
SMoPE
76.10±0.34
72.17±0.52
Table 2: Average accuracy on CIFAR-100 and ImageNet-R with self-supervised pre-trained models iBOT-1K and DINO-1K. Best results are in bold .
Training Strategy
Avg Acc ( ↑ )
Forgetting ( ↓ )
Baseline
83.44 ± 0.19
6.48 ± 0.42
+ Token Evaluation
86.73 ± 0.45
5.32 ± 0.36
+ Task-Adaptive Prediction
88.73 ± 0.29
3.74 ± 0.43
+ First-Task Stabilization
89.25 ± 0.21
3.68 ± 0.38
+ Anchoring Loss LAD
90.59 ± 0.31
2.68 ± 0.25
Table 3: Ablating different components of our method on CIFAR-100.
Dataset
∣Cseen∣
LAD /step
Memory
CIFAR-100
90
0.94 GFLOPs
4.61 MB
ImageNet-R / CUB-200
180
1.88 GFLOPs
8.29 MB
Table 4: Estimated anchor overhead (FP32, ViT-B/16, Lp=6 , H=12 , D=64 , M=25 , r=10 ). Per-step cost is for learning the final task, probing the ∣Cseen∣ previously seen classes; memory is the complete anchor state after training (statistics of all classes plus the snapshot Pold ).
Anchor weighting
Avg Acc ( ↑ )
Forgetting ( ↓ )
Inverse relevance ( 1−γold )
89.05 ± 0.20
4.55 ± 0.35
Uniform ( γold≡1 )
90.10 ± 0.35
3.22 ± 0.27
Past relevance (ours)
90.59 ± 0.31
2.68 ± 0.25
Table 5: Ablation study for the interaction between the token evaluation module and anchoring loss on CIFAR-100, 10 tasks.
Hyperparameter
Description
CIFAR-100
ImageNet-R
CUB-200
E
Number of training epochs
50
50
50
η~
Learning rate
0.005
0.0025
0.0025
bsz
Batch size
128
64
128
n
Token prompt length
50
50
50
τ(s)
Threshold for the soft weighting function
5-th highest score
5-th highest score
5-th highest score
τsoft
Temperature for the soft weighting function
0.2
0.5
0.2
Table 6: Hyperparameters used in TEA-Prompt.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
τsoft
0.01
0.05
0.1
0.2 †
0.4
0.8
1.6
3.2
6.4
Avg Acc ( ↑ )
89.78
90.02
90.36
90.59
90.04
89.36
89.20
90.23
89.53
Forgetting ( ↓ )
3.25
3.00
2.90
2.68
3.16
3.23
3.22
2.83
3.09
Appendix
Table 7: Sensitivity of TEA-Prompt to τsoft , K in τ(s) , r , and λ on CIFAR-100 (10 tasks, single seed). Each sweep varies one hyperparameter with the other three fixed at the shared optimum ( τsoft=0.2 , K=5 , r=10 , λ=60 , marked † ), which corresponds to the main-paper configuration.
CIFAR-100
ImageNet-R
CUB-200
Token weighting strategy
Avg Acc ( ↑ )
Forgetting ( ↓ )
Avg Acc ( ↑ )
Forgetting ( ↓ )
Avg Acc ( ↑ )
Forgetting ( ↓ )
Sparse selection (Top- K )
84.53 ± 0.27
5.58 ± 0.33
75.01 ± 0.30
6.39 ± 0.21
68.87 ± 0.42
11.62 ± 0.61
Softmax weighting
85.16 ± 0.42
5.91 ± 0.35
75.24 ± 0.33
5.36 ± 0.27
69.43 ± 0.43
11.75 ± 0.34
Sigmoid weighting
86.73 ± 0.45
5.32 ± 0.36
75.46 ± 0.32
5.21 ± 0.35
69.59 ± 0.54
11.52 ± 0.42
Appendix
Table 8: Performance comparison between soft token weighting and sparse selection. TAP is turned off during this experiment.
Anchor statistics over
qˉ
Q
Avg Acc ( ↑ )
90.59
90.02
Forgetting ( ↓ )
2.68
3.01
Statistics memory (MB)
3.7
3.7
Appendix
Table 9: Variable on which the anchor statistics are computed (CIFAR-100, 10 tasks, single seed).
Figure 2: Prompt-token utilization pooled over all 10 tasks and all 12 heads of each injected layer (last panel: all layers), the dashed line marks the uniform level 1/25 .
Figure 3: Mean evaluation weight γ on Task 2, per head at each of the six injected layers; each cell averages the weight over all Task-2 training images.
γ interval
Share (%)
Cumulative (%)
[0,0.1)
70.28
70.28
[0.1,0.2)
6.60
76.88
[0.2,0.3)
1.81
78.69
[0.3,0.4)
0.79
79.48
[0.4,0.5)
0.52
80.00
[0.5,1]
20.00
100.00
Appendix
Table 10: Distribution of the evaluation weights γ over all (input, head, token) activations on CIFAR-100. The 20% mass above 0.5 equals K/M by construction of the threshold.
thr
0 (none)
0.1
0.2
0.3
0.4
0.5
Removed (%)
0
70.28
76.88
78.69
79.48
80.00
Avg Acc ( ↑ )
90.59
89.95
89.75
90.25
89.52
89.10
Forgetting ( ↓ )
2.68
2.87
3.14
2.96
3.20
3.24
Appendix
Table 11: Removing low-weight prompt tokens on CIFAR-100 (10 tasks). Tokens whose evaluation weight falls below thr are discarded (weight set to zero) throughout training and inference; thr=0 is the unmodified model. The second row gives the fraction of activations removed (from Table 10 ).
TEA-Prompt
Avg Acc ( ↑ )
Forgetting ( ↓ )
without LAD
89.25 ± 0.21
3.68 ± 0.38
with LAD (task-specific query)
89.74 ± 0.25
3.57 ± 0.29
with LAD (class-specific query)
90.59 ± 0.31
2.68 ± 0.25
Appendix
Table 12: Comparison of task-specific query and class-specific query for LAD on CIFAR-100.
Sampling model
Diag. †
Mean only
Full-cov.
GMM-2
Avg Acc ( ↑ )
87.37
87.02
87.25
86.15
Forgetting ( ↓ )
3.96
4.09
4.08
3.98
GFLOPs/step
2.19
0.22
2.72
2.20
Appendix
Table 13: Density model used to sample anchor queries q~ in LAD (CIFAR-100, 10 tasks, single seed). GFLOPs measure one anchor evaluation per training step at the final task. † : diagonal Gaussian (ours); Mean only = σ=0 ; Full covariance Gaussian; GMM-2 with 2 diagonal Gaussian components. TAP is turn-off during this experiment.
Institute for Artificial Intelligence and Data Science, University at Buffalo, Buffalo, NY, USA · Department of Computer Science and Engineering, University at Buffalo, Buffalo, NY, USA
LIFO UR 4022 Université d’Orléans, INSA CVL, Orléans, 45067, France · ETIS CY Cergy Paris University, ENSEA, CNRS, Cergy, 95000, France · LIST3N Université de Technologie de Troyes, Troyes, 10010, France