cs.CVNov 16, 2026

[CLS] and [EOS] Tokens are not Enough for CLIP-based Class-Incremental Learning

Authors: Hao Sun, Tao Hu, Shixiong Kai, Siyuan Xu, Wenqian Zhao, Mingxuan Yuan, Da-Wei Zhou

Organizations: School of Artificial Intelligence and the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Huawei Noah’s Ark Lab

Abstract

Class-Incremental Learning (CIL) aims to empower models to continuously acquire new knowledge while mitigating catastrophic forgetting. With the advent of vision-language models like CLIP, leveraging their cross-modal alignment capabilities to address CIL challenges has emerged as a dominant paradigm. Existing methods leverage CLIP's global representations (i.e., the visual [CLS] token and the textual [EOS] token) for cross-modal alignment. While achieving effective classification, they overlook the rich patch-level visual features and token-level textual semantics inherent in CLIP. For example, the [CLS] token aggregates local information to capture the general concept of "cat", while the patch-level details retain specific localized features essential for recognition, such as the tail, fur, and face. We propose PACE (PAtch-token Cross-modal Enhancement) to uncover the long-neglected local representations within CLIP. First, based on the global representations like [CLS] token and [EOS] token, we identify the top-k most crucial visual patches and textual tokens. Subsequently, to better utilize the extracted key local features, we design a cross-modal fusion module. This module leverages local textual semantics to refine visual representations, while augmenting textual semantics with local visual features. As the fusion module is shared across all tasks, directly fine-tuning it on new tasks would interfere with the knowledge learned from previous tasks, leading to catastrophic forgetting. To alleviate this, we employ a projection-based update strategy to effectively preserve the critical parameter space of previous tasks during the adaptation to new tasks. Extensive experiments demonstrate that PACE significantly outperforms state-of-the-art methods across multiple CIL benchmarks.

Figures & tables

Explore similar work

CardsList
  1. Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning

    May 13, 2026Hao Sun, Zi-Jun Ding, Da-Wei ZhouClass-Incremental LearningVLM Adaptation

  2. BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

    Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental LearningVLM Adaptation

  3. Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning

    Nov 19, 2025Tao Hu, Lan Li, Zhen-Hao Xie +1Class-Incremental LearningContinual Learning for VLMs