cs.LGNov 9, 2026

Anchor Behavior, Not Parameters: Rehearsal-Free Continual Learning with a Single Shared Prompt

Authors: Duy-Cuong Bui, Trung-Anh Dang, Vincent Nguyen, Duc V. Le, Ngoc-Son Vu

Organizations: ETIS - CY Cergy Paris Université, ENSEA, CNRS UMR 8051, France · LIFO UR 4022, Université d’Orléans, INSA CVL, France · University of Twente, Netherlands · LIST3N - Université de Technologie de Troyes, France

Abstract

Prompt-based continual learning adapts a frozen pre-trained vision transformer by tuning a small set of prompt tokens. Recently, one-prompt methods which share one single prompt across all tasks make this paradigm maximally parameter efficient. The open question is how a shared prompt can keep the knowledge of old classes while being updated for new ones. Existing methods protect old knowledge in parameter space, where they freeze task-specific prompts, expand new components, or sparsely gate which tokens receive gradients. However, this method sacrifices plasticity, yet the drift it still permits in shared tokens can corrupt the prompt's function on past classes. What must be preserved is the prompt's behavior. We propose TEA-Prompt, which couples (i) a token evaluation module that assigns each prompt token a continuous, input-conditional relevance score injected into the attention logits, softly specializing tokens across tasks without discarding any of them, and (ii) an attention-behavior anchoring loss that constrains each token's attention output around class-specific query regions using only compact Gaussian query statistics without stored old images, weighted by the token's past relevance. Anchoring the attention value allows the parameters to continue updating freely, as long as their interaction remains unchanged. TEA-Prompt achieves state-of-the-art accuracy with markedly lower forgetting on CIFAR-100, ImageNet-R, and CUB-200 with only 0.38M parameters. Code is publicy available at https://github.com/cuonga1cvp/TEA-Prompt.git

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Information-Theoretic Decoupled Prompt Tuning for Continual Learning

    Aug 13, 2026Yunfei Zhang, Wen Wen, Tieliang Gong +1Continual Learning for VLMsContinual Learning for LLMs

  2. Turning Back Without Forgetting: Selective Backward Refinement for Parameter-Efficient Continual Learning

    May 31, 2026Anushka Tiwari, Kaiyi JiStorage-Efficient Continual LearningParameter-Efficient Fine-Tuning

  3. GAP-Prompt: Gated Adaptive Prompting for Efficient Continual Learning

    Aug 24, 2026Trung-Anh Dang, Duy-Cuong Bui, Ngoc-Son Vu +2Prompt-Based Continual LearningContinual Learning