[CLS] and [EOS] Tokens are not Enough for CLIP-based Class-Incremental Learning
Organizations: School of Artificial Intelligence and the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Huawei Noah’s Ark Lab
Abstract
Class-Incremental Learning (CIL) aims to empower models to continuously acquire new knowledge while mitigating catastrophic forgetting. With the advent of vision-language models like CLIP, leveraging their cross-modal alignment capabilities to address CIL challenges has emerged as a dominant paradigm. Existing methods leverage CLIP's global representations (i.e., the visual [CLS] token and the textual [EOS] token) for cross-modal alignment. While achieving effective classification, they overlook the rich patch-level visual features and token-level textual semantics inherent in CLIP. For example, the [CLS] token aggregates local information to capture the general concept of "cat", while the patch-level details retain specific localized features essential for recognition, such as the tail, fur, and face. We propose PACE (PAtch-token Cross-modal Enhancement) to uncover the long-neglected local representations within CLIP. First, based on the global representations like [CLS] token and [EOS] token, we identify the top-k most crucial visual patches and textual tokens. Subsequently, to better utilize the extracted key local features, we design a cross-modal fusion module. This module leverages local textual semantics to refine visual representations, while augmenting textual semantics with local visual features. As the fusion module is shared across all tasks, directly fine-tuning it on new tasks would interfere with the knowledge learned from previous tasks, leading to catastrophic forgetting. To alleviate this, we employ a projection-based update strategy to effectively preserve the critical parameter space of previous tasks during the adaptation to new tasks. Extensive experiments demonstrate that PACE significantly outperforms state-of-the-art methods across multiple CIL benchmarks.
Figures & tables
| Method | Aircraft | CIFAR100 | Cars | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| B0 Inc10 | B50 Inc10 | B0 Inc10 | B50 Inc10 | B0 Inc10 | B50 Inc10 | ||||||||
| Finetune | 3.16 | 0.96 | 1.72 | 1.05 | 7.84 | 4.44 | 5.30 | 2.46 | 3.14 | 1.10 | 1.54 | 1.13 | |
| CoOp [ 62 ] | 14.54 | 7.14 | 13.05 | 7.77 | 47.00 | 24.24 | 41.23 | 24.12 | 36.46 | 21.65 | 37.40 | 20.87 | |
| SimpleCIL [ 65 ] | 59.24 | 48.09 | 53.05 | 48.09 | 84.15 | 76.63 | 80.20 | 76.63 | 92.04 | 86.85 | 88.96 | 86.85 | |
| ZS-CLIP [ 15 ] | 26.66 | 17.22 | 21.70 | 17.22 | 81.81 | 71.38 | 76.49 | 71.38 | 82.60 | 76.37 | 78.32 | 76.37 | |
| Dataset | |||||
|---|---|---|---|---|---|
| Aircraft | 64.26 | 64.67 | 64.84 | 65.26 | 64.40 |
| CIFAR100 | 86.43 | 86.74 | 86.83 | 86.92 | 86.66 |
| Patch Selection | Cars B0 Inc10 | ImageNet-R B0 Inc20 | ||
|---|---|---|---|---|
| Random- | 89.33 | 82.48 | 81.16 | 72.79 |
| Attention-only | 92.14 | 88.42 | 84.30 | 78.40 |
| Top- (ours) | 93.94 | 89.35 | 85.03 | 79.38 |
| Method | Eval Time (min) | Peak Memory (GB) | Infer. (ms/img) | FLOPs (G) | |
|---|---|---|---|---|---|
| ZS-CLIP [ 15 ] | 82.60 | 0.96 | 1.04 | 2.56 | 11.29 |
| DualPrompt [ 54 ] | 76.26 | 3.22 | 6.95 | 8.58 | 28.96 |
| CODA-Prompt [ 51 ] | 80.21 | 3.59 | 15.53 | 9.58 | 35.33 |
| RAPF [ 58 ] | 82.89 | 1.34 | 6.43 | 3.59 | 11.29 |
| CLG-CBM [ 59 ] | 78.94 | 1.13 | 1.05 | 3.01 | 11.29 |
| PROOF [ 20 ] | 90.74 | 2.41 | 1.16 | 6.43 | 11.85 |
| Method | Aircraft B0 Inc10 | Cars B0 Inc10 | ImageNet-R B0 Inc20 | |||
|---|---|---|---|---|---|---|
| SimpleCIL [ 65 ] | 61.05 1.28 | 48.09 0.00 | 91.54 0.81 | 86.85 0.00 | 80.98 0.57 | 74.48 0.00 |
| RAPF [ 58 ] | 45.81 1.64 | 23.78 1.11 | 80.05 1.34 | 69.70 1.36 | 82.93 0.47 | 76.13 0.58 |
| CLG-CBM [ 59 ] | 47.48 1.57 | 38.60 0.80 | 78.66 0.96 | 74.45 0.34 | 79.42 0.24 | 72.00 0.87 |
| PROOF [ 20 ] | 66.34 1.50 | 50.82 0.70 | 92.18 0.97 | 88.38 0.17 | 83.63 0.46 | 78.71 0.12 |
| PACE | 67.55 1.27 | 51.66 0.58 | 93.88 0.91 | 89.63 0.20 | 84.15 0.57 | 79.16 0.24 |
| Method | Aircraft | CIFAR100 | ImageNet-R | SUN | ||||
|---|---|---|---|---|---|---|---|---|
| B0 Inc10 | B50 Inc10 | B0 Inc10 | B50 Inc10 | B0 Inc20 | B100 Inc20 | B0 Inc30 | B150 Inc30 | |
| CODA-Prompt [ 51 ] | 25.09 | 20.03 | 21.43 | 19.48 | 16.60 | 12.27 | 22.71 | 17.69 |
| RAPF [ 58 ] | 25.68 | 19.09 | 8.14 | 4.57 | 9.95 | 4.48 | 9.86 | 4.81 |
| CLG-CBM [ 59 ] | 14.41 | 5.32 | 13.96 | 12.57 | 11.44 | 9.50 | 14.62 | 12.26 |
| PROOF [ 20 ] | 14.67 | 7.43 | 16.47 | 15.27 | 9.06 | 6.80 | 15.16 | 5.01 |
| PACE | 15.09 | 6.24 | 13.67 | 8.45 | 7.88 | 4.63 | 8.78 | 4.43 |
| Prompt Template | Cars B0 Inc10 | UCF B0 Inc10 | ||
|---|---|---|---|---|
| A photo of a [CLASS] | 93.94 | 89.35 | 94.40 | 89.47 |
| A photo of the [CLASS] | 93.92 | 89.30 | 95.07 | 90.00 |
| A good photo of a [CLASS] | 93.96 | 89.44 | 94.84 | 90.22 |
| A blurry photo of a [CLASS] | 93.91 | 89.27 | 94.11 | 89.05 |
| Method | ImageNet-100 B0 Inc10 | ImageNet-1K B0 Inc100 | ||
|---|---|---|---|---|
| SimpleCIL [ 65 ] | 91.10 | 86.88 | 75.94 | 68.37 |
| RAPF [ 58 ] | 91.92 | 87.02 | 73.09 | 65.59 |
| PROOF [ 20 ] | 88.44 | 82.88 | 70.52 | 63.35 |
| PACE | 92.21 | 87.78 | 77.43 | 69.93 |
| Dataset | ||||||||
|---|---|---|---|---|---|---|---|---|
| Aircraft | 64.89 | 64.90 | 65.12 | 65.26 | 64.84 | 64.74 | 65.28 | 64.85 |
| Cars | 93.80 | 93.91 | 93.86 | 93.94 | 93.92 | 93.84 | 93.88 | 93.58 |
| ImageNet-R | 84.50 | 84.59 | 84.80 | 85.03 | 84.86 | 84.54 | 85.07 | 84.38 |
Incremental Learning
Incremental learning studies how models update from sequential data while retaining earlier knowledge as classes, tasks, or domains change.
See all 151 papers