Attribute-Evidence Learning with Multimodal Prompt for Generalized Category Discovery
Authors: Wei Zhang, Zhu Teng, Zhixiang Chi, Junnan Zou, Wenxin Luo, Linfeng Ye, Jianping Fan, Konstantions N. Plataniotis
Organizations: the Beijing Jiaotong University, Beijing 100044, China · University of Toronto, Toronto, ON M5S 2E8, Canada · the FuDan University, Shanghai 200433, China
Generalized category discovery (GCD) aims to discover novel categories from unlabeled data by leveraging limited labeled samples from old classes. Existing methods that rely solely on visual cues often neglect the multimodal nature of human perception, prompting recent approaches to incorporate textual information such as category tags and attributes. However, these approaches typically entangle global tags and local attributes within a shared global image representation, rendering attributes unreliable and potentially providing misleading cues that can drive the model toward incorrect predictions. To this end, we propose AptM-GCD, a novel multimodal framework that exploits apt textual guidance through attribute evidence learning and multimodal prompt reasoning. Specifically, we design an Attribute-Evidence Learning (AEL) mechanism that treats attributes as localized visual evidence rather than global category descriptors. By interacting attribute embeddings with patch-level visual tokens, AEL identifies reliable attribute-related regions and extracts discriminative local evidence, thereby reducing the impact of noisy or irrelevant attribute cues. Building upon the extracted evidence, we further develop a Multimodal Prompt Learning (MPL) mechanism to bridge local attribute semantics and global visual understanding. MPL converts fine-grained textual evidence into visual prompts and injects them into a multimodal prompt transformer, enabling attribute information to enhance category discrimination through contextual reasoning without enforcing direct global alignment between modalities. Extensive experiments on six datasets demonstrate that AptM-GCD consistently achieves SoTA performance across both fine-grained and generic benchmarks. Moreover, the proposed framework exhibits strong generalization ability, delivering consistent gains under unimodal and class-imbalanced GCD settings.
Figures & tables
Fig. 1: Illustration of multimodal generalized category discovery. Images include labeled data from old classes and unlabeled data from both old and novel categories, while a heterogeneous textual dictionary of category tags and attribute descriptions introduces semantic guidance for category discovery.
Fig. 2: Overview of challenges in existing multimodal GCD methods. (a) Attribute-based GCD pipeline: Illustration of how mixed-granularity semantic binding can cause spurious attribute-image associations, contaminating the category-level decision space. (b) Empirical misclassification statistics: For attribute-based methods [ 16 ] , showing the proportion of samples where retrieved attributes are irrelevant to the ground-truth class and lead to incorrect predictions. (c) Direct addition of attributes to tag-based GCD: Naively adding attributes to a tag-based method can degrade performance on novel classes, whereas our proposed method improves results.
Fig. 3: Overall framework of AptM-GCD. AptM-GCD uses Attribute-Evidence Learning (AEL) and Multimodal Prompt Learning (MPL) to harness fine-grained attributes as apt textual guidance for representation refinement. AEL selects image-specific attributes and extracts attribute-conditioned visual evidence from object-aware patch tokens. MPL converts this evidence into multimodal prompts and injects them into the visual transformer to integrate local attribute cues with global visual context. The refined representations are finally used for tag-based multimodal classification with category tags as global semantic prototypes. The right panels provide detailed illustrations of adaptive attribute selection and visual evidence modeling.
Fig. 4: Object-Guided Patch Calibration. We extract the last-layer [CLS] -to-patch attention from DINO and normalize it into a soft object prior. Each prior value measures the estimated object relevance of a patch, which is used to suppress background responses during adaptive attribute selection.
Fig. 5: Multimodal Prompt Generation. The left part shows the Recurrent Self-generation Mechanism, where the first forward pass constructs attribute-conditioned visual evidence through AEL and generates image-specific prompts that are reused in the second forward pass to guide the model. The right part shows the structure of the Multimodal Prompt Generator.
CUB
Aircraft
All ↑
Old ↑
New ↑
M ↓ (MB)
T ↓ (s)
All ↑
Old ↑
New ↑
M ↓ (MB)
T ↓ (s)
TextGCD w. tag
69.9
74.9
67.3
5256
48
48.1
46.3
49.0
5256
52
TextGCD w. tag + attribute
76.6
80.6
74.7
7038
62
50.8
44.9
53.8
7038
66
TextGCD w. our framework
TABLE IV: Quantitative analysis of AptM-GCD on attribute-based method [ 16 ]
Fig. 6: Qualitative analysis of AptM-GCD on attribute-based methods. Last-layer attention maps and t-SNE feature distributions show that directly using retrieved attributes can cause spurious attention and entangled category boundaries. AptM-GCD alleviates this issue by transforming attributes into reliable visual evidence, guiding the model to focus on discriminative object regions and produce clearer category separation.
AEL
MPL
CUB
Stanford Cars
Aircraft
All
Old
New
All
Old
New
All
Old
New
✗
✗
78.0
80.5
76.4
87.0
93.3
84.2
61.3
65.4
59.3
✓
✗
79.2
80.4
78.6
88.8
93.5
86.5
63.3
65.9
62.0
✗
✓
80.1
81.1
79.6
90.2
93.6
88.4
64.7
65.7
64.3
✓
✓
81.5
80.6
81.9
91.0
94.0
89.2
65.3
65.5
65.3
TABLE V: Component ablation of AEL and MPL in AptM-GCD
Fig. 7: Qualitative analysis of component in AptM-GCD. AEL localizes attribute evidence, while MPL integrates such local evidence with global visual context through soft visual prompts, leading to more reliable attention.
Prior
CUB
Stanford Scars
Aircraft
All
Old
New
All
Old
New
All
Old
New
✗
80.3
80.5
80.2
90.7
93.8
88.6
64.0
65.3
63.3
✓
81.5
80.6
81.9
91.0
94.0
89.2
65.3
65.5
65.3
TABLE VI: Quantitative ablation of Object-guided Patch Calibration
Fig. 8: Visualization of the object prior. Patch colors indicate the estimated object relevance, changing from red to blue as the prior value decreases. The prior highlights foreground object regions while suppressing background patches, enabling more reliable attribute selection.
Fig. 9: Sensitivity of the nucleus threshold ρ in Adaptive Attribute Selection. We vary ρ from 0.5 to 0.9 on three fine-grained datasets and ImageNet-100. ρ=0.8 providing the best balance between attribute coverage and selection reliability
ρ
CUB
Stanford Cars
Aircraft
0.5
14
9
6
0.6
16
11
8
0.7
23
13
9
0.8
29
15
10
0.9
37
17
11
TABLE VIII: Average number of selected attributes under different nucleus thresholds ρ .
Fig. 10: Effect of attribute-patch interaction strategies in Visual Evidence Modeling. We compare different ways of modeling the correspondence between selected attributes and visual patches. Arrows denote the query direction in cross-attention, where attribute → patch uses attributes as queries to retrieve visual evidence from patches.
Fig. 11: Effect of MPL as a visual prompting method within AptM-GCD. Without AEL or textual attributes, MPL outperforms representative visual prompt methods, including VPT [ 31 ] , Global [ 58 ] , and SPM [ 59 ] , showing its strong ability to refine visual representations. Adding AEL further improves performance.
Method
SSB-AVG
ImageNet-100
All
Old
New
All
Old
New
TextGCD (ECCV’24)
71.4
71.0
71.7
91.7
95.7
89.7
+ Our MLP
72.6 +1.2
72.0 +1.0
73.5 +1.8
93.2 +1.5
96.1 +0.4
91.9 +2.2
GET (CVPR’25)
71.5
74.8
69.8
88.0
92.4
85.2
+ Our MLP
73.3 +1.8
74.8 +0.0
72.3 +2.5
90.9 +2.9
92.7 +0.3
88.1 +2.9
SpectralGCD (ICLR’26)
77.1
79.7
75.8
93.4
96.1
92.0
TABLE IX: Transferability of MLP on multimodal GCD methods
SSB-Avg
CIFAR-100
Method
All
Old
New
All
Old
New
CMS [ 60 ]
57.1
65.9
52.7
79.6
83.2
72.3
+ Our MLP
60.5 +3.4
68.3 +2.4
57.5 +4.8
82.0 +2.4
84.2 +1.0
76.6 +4.3
CiPR [ 22 ]
52.1
60.1
47.9
81.5
82.4
79.7
+ Our MLP
56.2 +4.1
63.9 +3.8
53.1 +5.2
81.9 +0.4
82.5 +0.1
81.9 +2.2
LegoGCD [ 25 ]
58.7
69.7
53.3
81.8
81.4
82.5
TABLE X: Transferability of MLP in unimodal GCD
p=20
p=100
Method
Type
New
Old
All
New
Old
All
SimGCD [ 8 ] (ICCV-23)
Standard GCD
78.9
77.4
77.6
55.9
77.4
67.9
+ Our MLP
85.0
83.2
84.1
73.1
81.6
77.4
CMS [ 60 ] (CVPR-24)
Standard GCD
75.0
84.6
78.8
66.4
71.6
69.1
+ Our MLP
84.1
86.8
86.6
83.0
74.5
78.8
TABLE XI: Transferability of MLP under class-imbalanced GCD on CIFAR-10-LT
CUB
Stanford Cars
Aircraft
Fuse
All
Old
New
All
Old
New
All
Old
New
✗
80.5
80.1
80.8
90.3
93.6
88.4
64.6
64.6
64.6
✓
81.5
80.6
81.9
91.0
94.0
89.2
65.3
65.5
65.3
TABLE XII: Effect of the feature fuse in multimodal prompt generator.
Fig. 12: Effect of prompt placement in the Multimodal Prompt Transformer. We inject multimodal prompts into different transformer blocks to study layer-wise performance on three fine-grained datasets.
SSB-Avg
ImageNet-100
Training Strategy
All
Old
New
All
Old
New
Decoupling
77.7
79.8
76.2
93.6
95.8
92.6
Alternating
78.1
80.4
77.8
94.1
96.1
93.0
Joint
79.3
78.4
78.8
95.1
96.8
94.4
TABLE XIII: Ablation study of training strategy of Prompt Transformer and Prompt Generator on the SSB and ImageNet.
Method
Backbone
Extra Modules
Total Params (M)
TextGCD [ 16 ]
CLIP ViT-B/16
classifiers
∼ 150
SpectralGCD [ 20 ]
CLIP ViT-B/16
Projection
∼ 156
AptM-GCD
CLIP ViT-B/16
Projection + AEL + MPL
∼ 170
TABLE XIV: Comparison of model size with multimodal GCD methods.