Anchor Behavior, Not Parameters: Rehearsal-Free Continual Learning with a Single Shared Prompt
Organizations: ETIS - CY Cergy Paris Université, ENSEA, CNRS UMR 8051, France · LIFO UR 4022, Université d’Orléans, INSA CVL, France · University of Twente, Netherlands · LIST3N - Université de Technologie de Troyes, France
Abstract
Prompt-based continual learning adapts a frozen pre-trained vision transformer by tuning a small set of prompt tokens. Recently, one-prompt methods which share one single prompt across all tasks make this paradigm maximally parameter efficient. The open question is how a shared prompt can keep the knowledge of old classes while being updated for new ones. Existing methods protect old knowledge in parameter space, where they freeze task-specific prompts, expand new components, or sparsely gate which tokens receive gradients. However, this method sacrifices plasticity, yet the drift it still permits in shared tokens can corrupt the prompt's function on past classes. What must be preserved is the prompt's behavior. We propose TEA-Prompt, which couples (i) a token evaluation module that assigns each prompt token a continuous, input-conditional relevance score injected into the attention logits, softly specializing tokens across tasks without discarding any of them, and (ii) an attention-behavior anchoring loss that constrains each token's attention output around class-specific query regions using only compact Gaussian query statistics without stored old images, weighted by the token's past relevance. Anchoring the attention value allows the parameters to continue updating freely, as long as their interaction remains unchanged. TEA-Prompt achieves state-of-the-art accuracy with markedly lower forgetting on CIFAR-100, ImageNet-R, and CUB-200 with only 0.38M parameters. Code is publicy available at https://github.com/cuonga1cvp/TEA-Prompt.git
Figures & tables
| Method | Params (M) | CIFAR-100 | ImageNet-R | CUB-200 | |||
| Avg Acc( ) | Forgetting( ) | Avg Acc( ) | Forgetting( ) | Avg Acc( ) | Forgetting( ) | ||
| Joint training | – | 91.38 | – | 82.06 | – | 88.00 | – |
| LAE [ 13 ] | 0.24 | 86.09 0.55 | 6.22 0.17 | 73.88 0.43 | 5.08 0.33 | – | – |
| InfLoRA [ 27 ] | 3.84 | 86.91 0.34 | 4.22 0.37 | 76.13 0.44 | 5.22 0.48 | – | – |
| EASE [ 49 ] | 11.90 | 87.28 0.71 | 4.81 0.42 | 75.82 0.54 | 5.83 0.37 | – | – |
| L2P [ 45 ] | 0.24 | 83.86 0.28 | 7.35 0.38 | 61.57 0.66 | 9.73 0.47 | 71.22 0.54 | 10.68 0.61 |
| Pre-trained model | Method | CIFAR-100 | ImageNet-R |
| iBOT-1K | L2P | ||
| DualPrompt | |||
| CODA-Prompt | |||
| RainbowPrompt | |||
| VQ-Prompt | |||
| SMoPE |
| Training Strategy | Avg Acc ( ) | Forgetting ( ) |
| Baseline | 83.44 0.19 | 6.48 0.42 |
| + Token Evaluation | 86.73 0.45 | 5.32 0.36 |
| + Task-Adaptive Prediction | 88.73 0.29 | 3.74 0.43 |
| + First-Task Stabilization | 89.25 0.21 | 3.68 0.38 |
| + Anchoring Loss | 90.59 0.31 | 2.68 0.25 |
| Dataset | /step | Memory | |
| CIFAR-100 | 90 | 0.94 GFLOPs | 4.61 MB |
| ImageNet-R / CUB-200 | 180 | 1.88 GFLOPs | 8.29 MB |
| Anchor weighting | Avg Acc ( ) | Forgetting ( ) |
| Inverse relevance ( ) | 89.05 0.20 | 4.55 0.35 |
| Uniform ( ) | 90.10 0.35 | 3.22 0.27 |
| Past relevance (ours) | 90.59 0.31 | 2.68 0.25 |
| Hyperparameter | Description | CIFAR-100 | ImageNet-R | CUB-200 |
| Number of training epochs | 50 | 50 | 50 | |
| Learning rate | 0.005 | 0.0025 | 0.0025 | |
| Batch size | 128 | 64 | 128 | |
| Token prompt length | 50 | 50 | 50 | |
| Threshold for the soft weighting function | 5-th highest score | 5-th highest score | 5-th highest score | |
| Temperature for the soft weighting function | 0.2 | 0.5 | 0.2 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| 0.01 | 0.05 | 0.1 | 0.2 † | 0.4 | 0.8 | 1.6 | 3.2 | 6.4 | |
| Avg Acc ( ) | 89.78 | 90.02 | 90.36 | 90.59 | 90.04 | 89.36 | 89.20 | 90.23 | 89.53 |
| Forgetting ( ) | 3.25 | 3.00 | 2.90 | 2.68 | 3.16 | 3.23 | 3.22 | 2.83 | 3.09 |
| CIFAR-100 | ImageNet-R | CUB-200 | ||||
| Token weighting strategy | Avg Acc ( ) | Forgetting ( ) | Avg Acc ( ) | Forgetting ( ) | Avg Acc ( ) | Forgetting ( ) |
| Sparse selection (Top- ) | 84.53 0.27 | 5.58 0.33 | 75.01 0.30 | 6.39 0.21 | 68.87 0.42 | 11.62 0.61 |
| Softmax weighting | 85.16 0.42 | 5.91 0.35 | 75.24 0.33 | 5.36 0.27 | 69.43 0.43 | 11.75 0.34 |
| Sigmoid weighting | 86.73 0.45 | 5.32 0.36 | 75.46 0.32 | 5.21 0.35 | 69.59 0.54 | 11.52 0.42 |
| Anchor statistics over | ||
| Avg Acc ( ) | 90.59 | 90.02 |
| Forgetting ( ) | 2.68 | 3.01 |
| Statistics memory (MB) | 3.7 | 3.7 |
| interval | Share (%) | Cumulative (%) |
| 70.28 | 70.28 | |
| 6.60 | 76.88 | |
| 1.81 | 78.69 | |
| 0.79 | 79.48 | |
| 0.52 | 80.00 | |
| 20.00 | 100.00 |
| 0 (none) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | |
| Removed (%) | 0 | 70.28 | 76.88 | 78.69 | 79.48 | 80.00 |
| Avg Acc ( ) | 90.59 | 89.95 | 89.75 | 90.25 | 89.52 | 89.10 |
| Forgetting ( ) | 2.68 | 2.87 | 3.14 | 2.96 | 3.20 | 3.24 |
| TEA-Prompt | Avg Acc ( ) | Forgetting ( ) |
| without | 89.25 0.21 | 3.68 0.38 |
| with (task-specific query) | 89.74 0.25 | 3.57 0.29 |
| with (class-specific query) | 90.59 0.31 | 2.68 0.25 |
| Sampling model | Diag. † | Mean only | Full-cov. | GMM-2 |
| Avg Acc ( ) | 87.37 | 87.02 | 87.25 | 86.15 |
| Forgetting ( ) | 3.96 | 4.09 | 4.08 | 3.98 |
| GFLOPs/step | 2.19 | 0.22 | 2.72 | 2.20 |