Organizations: School of Traffic and Transportation, Beijing Jiaotong University, Beijing 100044, China · Foundation Modal Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Queen Mary School Hainan, Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Computer Science, NUS School of Computing, Singapore 117417
Few-shot fine-grained visual classification (FGVC) aims to discriminate between subtly different categories with very limited data. While Multimodal Large Language Models (MLLM) exhibit strong capabilities in both fine-grained visual understanding and open-world category recognition, directly fine-tuning them on few-shot FGVC tasks typically leads to overfitting on sparse samples, which erodes their valuable open-world knowledge and compromises generalization. To fundamentally overcome this limitation, we present UniFGVC, a universal training-free framework that shifts the paradigm from parameter fine-tuning to training-free retrieval-based inference. UniFGVC reframes few-shot FGVC as a multimodal retrieval problem using predefined category templates. Its core is the Category-Discriminative Visual Captioner (CDV-Captioner), an attribute-aware module designed to leverage MLLM for generating structured textual descriptions that explicitly pinpoint discriminative visual attributes between similar classes. By integrating chain-of-thought reasoning and visual references, CDV-Captioner mitigates hallucination and maximizes the discriminative power of textual captions, thereby converting images into attribute-enriched image-description pairs. Subsequently, utilizing arbitrary pre-trained visual and textual encoders in a plug-and-play manner, UniFGVC extracts features from both modalities and computes their similarity with all category templates for recognition. Extensive experiments on 12 FGVC datasets demonstrate UniFGVC's universal compatibility and consistent superiority. It not only outperforms existing few-shot CLIP-based methods but also surpasses several fully-supervised MLLM-based approaches, confirming the general advantage of the proposed retrieval paradigm.
Figures & tables
Fig. 1: Overview of different few-shot FGVC paradigms. (a) CLIP-based methods rely on fine-tuning to achieve fine-grained discrimination but show limited cross-domain generalization. (b) MLLM-based methods leverage generated image captions in model training to enhance fine-grained recognition, but the captions are often generic or hallucinated. (c) Our proposed UniFGVC shifts the paradigm from parameter fine-tuning to training-free retrieval-based inference, reframing the task as a multimodal retrieval problem with predefined category templates. Central to UniFGVC is the CDV-Captioner, which leverages chain-of-thought reasoning and visual references to mitigate hallucination and produce structured attribute-enriched textual descriptions.
Fig. 2: An overview of the proposed UniFGVC. UniFGVC is a universal, training-free framework for few-shot fine-grained visual classification, which reformulates the task as a multimodal retrieval problem using structured attribute-aware representations. The CDV-Captioner progressively prompts the MLLM to output the structured fine-grained attribute-aware feature description of the target image, by integrating the category-related linguistic priors inherent in the MLLM and visual priors derived from reference images.
Methods
Venue
Setting
Img.
Cal.
DTD
Eur.
Air.
Flo.
Food
Pets
Cars
SUN
UCF
avg.
CLIP
ICML2021
Zero-shot
58.18
86.29
42.32
37.56
17.28
66.14
77.31
85.77
55.61
58.52
61.46
58.77
CoOp
IJCV2022
1-shot
57.15
87.53
44.39
50.63
9.64
68.12
74.32
85.89
55.59
60.29
61.92
59.59
Tip-Adapter
ECCV2022
60.88
88.80
50.49
50.34
19.01
81.17
76.22
86.04
56.78
61.23
66.19
63.38
SgVA-CLIP
TMM2023
61.11
89.67
49.97
58.18
22.34
76.17
78.16
88.05
58.17
64.81
64.43
64.64
CaFo
CVPR2023
63.81
91.82
53.14
68.24
25.01
81.29
77.98
89.21
62.12
64.87
68.71
67.84
GDA
ICLR2024
61.07
87.36
47.27
58.07
17.62
72.12
77.41
85.51
56.87
59.97
62.81
62.37
TABLE I: Comparative performance(%) of different few-shot FGVC methods across 1/2/4/8/16-shot settings on 11 fine-grained classification datasets. Bold and underline indicates the best and second best performance.
Model
Venue
#P
Flowers102
OxfordPets
FGVCAircraft
StandfordCars
CUBbirds
Avg.
LLaVA-Next
CVPR2024
7B
43.91
53.72
32.49
49.48
34.88
42.90
FineR
ICLR2024
7B
63.82
72.87
-
63.46
69.54
-
Idefics2
NeurIPS2024
8B
72.78
81.28
56.23
80.25
47.17
67.54
CascadeVLM
EMNLP2024
8B
91.62
-
63.95
92.04
80.80
-
Finedefics
ICLR2025
8B
89.88
92.18
63.82
84.67
57.61
77.63
UniFGVC
7B
95.74
90.16
58.06
94.63
77.48
83.21
TABLE II: Comparative performance (%) of different fully-supervised MLLM-based models on 5 datasets . Bold and underline indicates the best and second best performance. #P denotes parameters count.
Setting
1
2
4
8
16
avg.
Image-Only
37.88
45.76
52.18
57.14
60.72
50.74
Raw-Desc
58.38
66.52
67.28
70.94
71.56
66.94
Struct-Desc
65.42
70.96
73.66
74.20
76.20
72.09
Sum-Desc
66.27
71.84
75.31
76.26
78.11
73.56
Random-Ref
69.16
73.12
76.64
79.08
80.24
75.65
Similar-Ref
70.18
74.42
78.00
80.24
81.08
76.78
TABLE III: Ablation study of CDV-Captioner on ImageNet under 1/2/4/8/16-shot settings. Different variants are evaluated by varying the description structure and reference guidance.
Fig. 3: Visualization examples of structured attribute descriptions generated by Raw-Desc, Sum-Desc and Similar-Ref approaches.
Fig. 4: Visualization of the step-by-step refinement of the structured attribute description generated by CDV-Captioner.
s
1
2
4
8
16
avg.
1
66.24
67.22
71.15
72.28
73.62
70.10
2
67.25
71.08
74.97
77.44
78.33
73.81
3
70.18
74.42
78.00
80.24
81.08
76.78
4
70.24
74.44
77.56
80.83
80.62
76.74
5
70.21
74.32
77.98
81.00
82.18
77.14
TABLE IV: Ablation study on the number of regions (s) under the few-shot setting on ImageNet.
t
1
2
4
8
16
avg.
0
66.27
71.84
75.31
76.26
78.11
73.56
1
69.03
74.30
77.76
79.38
81.33
76.36
2
69.25
74.42
77.04
80.23
80.32
76.25
3
70.63
74.68
77.34
80.21
80.20
76.61
4
70.18
74.42
78.00
80.24
81.08
76.78
TABLE V: Ablation study on the number of reference samples (t) under the few-shot setting on ImageNet.
Encoders
1
2
4
8
16
avg.
Bge-m3
57.90
59.02
62.28
66.90
67.44
62.71
CLIP
59.08
62.92
66.08
66.94
67.82
64.57
RADIO
64.64
68.96
70.38
71.56
72.60
69.63
Unicom
70.18
74.42
78.00
80.24
81.08
76.78
TABLE VI: ablation study on the compatibility of UniFGVC with different vision encoders under the few-shot setting on ImageNet.
MLLM
1
2
4
8
16
avg.
InternVL-8B
60.17
66.10
70.94
72.62
74.70
68.91
QwenVL-2B
66.46
70.31
74.23
75.99
78.07
73.01
QwenVL-7B
70.18
74.42
78.00
80.24
81.08
76.78
GPT-4o
73.21
76.54
81.06
82.31
84.03
79.43
TABLE VII: ablation study on the compatibility of UniFGVC with different MLLM under the few-shot setting on ImageNet.
Setting
1
2
4
8
16
avg.
Max pooling
64.23
65.88
70.91
72.34
73.12
69.15
Averaging
64.36
65.68
71.06
72.31
74.03
69.49
Concatenation
66.46
69.72
74.69
76.21
77.48
72.91
Hybridization
67.48
70.92
75.31
76.20
77.62
73.51
TABLE VIII: Comparison of different multimodal fusion strategies on ImageNet.
Setting
MLLM
Hall. Score
Hall. Level
Raw-Desc
Qwen2.5-VL-7B
67.37
Medium
Sum-Desc
81.72
Low
Similar-Ref
82.04
Low
Raw-Desc
GPT-4o
78.62
Low
Sum-Desc
81.94
Low
Similar-Ref
82.07
Low
TABLE IX: Hallucination scores of CDV-Captioner variants on ImageNet.
Models
Setting
Accuracy
Training Time
Infer. speed/Image
Trainable Params
CoOp
16-shot
62.95
205min
6.90ms
2028
Tip-Adapter
65.44
198min
6.52ms
16M
SgVA-CLIP
65.80
218min
7.05ms
16M
Finedefics
fully-supervised
75.32
672min
14.45ms
18M
CascadeVLM
80.61
646min
18.36ms
32M
UniFGVC
16-shot
81.08
186min
18.11ms
0
TABLE X: Efficiency analysis for different methods on ImageNet. CLIP-based methods are evaluated under the 16-shot setting, MLLM-based methods under the fully-supervised setting, and UniFGVC under the 16-shot setting. For UniFGVC, the reported training time corresponds to the time required for multimodal category template construction.
Hall. Injection
Hall. Level
Accuracy
0
Low
81.08
1
Low
80.91
2-3
Medium
80.27
4-8
High
79.87
TABLE XI: Robustness evaluation under injected hallucination on ImageNet.
Retrieval Paradigm
mAP
R@10
Accuracy
Retr
Infer
Nearest-Neighbor
64.28
76.87
81.08
8.21
19.38
Clustering-Based
60.16
75.44
77.03
6.03
12.71
Hash-Based
56.33
70.57
71.80
4.83
8.43
TABLE XII: Retrieval paradigm comparison on ImageNet, where “Retr” denotes per-image retrieval latency and “Infer” denotes end-to-end inference latency (ms).
Fig. 5: Qualitative failure cases of UniFGVC caused by MLLM hallucination or attribute misinterpretation. Erroneous textual cues are highlighted in red, while the corresponding correct visual evidence is highlighted in green.
Fig. 6: Visualization of CDV-Captioner on scene-centric and non-object datasets including EuroSAT, DTD, and SUN.
Setting
1
2
4
8
16
avg.
Image-Only
42.51
58.67
66.22
68.91
76.33
62.53
Raw-Desc
43.21
60.05
67.26
72.52
76.69
63.95
Similar-Ref
56.74
67.24
75.65
81.69
84.57
73.18
TABLE XIII: Ablation study of CDV-Captioner on EuroSAT under 1/2/4/8/16-shot settings.
Fine-grained visual classification (FGVC) requires distinguishing between highly similar categories by focusing on subtle, localized visual differences. Recent multimodal methods have demonstrated that textual attribute descriptions can provide valuable semantic supervision by explicitly describing discriminative object characteristics. However, these descriptions are typically unavailable at test time, limiting their direct use in deployment. We propose FG-LUPI, a Learning Using Privileged Information (LUPI)-based framework that leverages attribute descriptions during training. We train a multimodal teacher that integrates image patches and textual attributes via gated cross-attention, allowing semantic attributes to condition visual representations. We distill this knowledge into an image-only student using multi-level distillation at the prediction, patch, and attention levels, aligning global predictions, patch representations, and attention patterns with the teacher. This encourages the student to inherit attention structure learned from multimodal supervision while maintaining image-only inference. Experiments on multiple FGVC benchmarks show consistent improvements in classification accuracy over strong ViT-based baselines. Furthermore, evaluation using Expected Calibration Error (ECE) shows that FG-LUPI also produces better-calibrated predictions across all six benchmarks. These results suggest that textual attribute supervision can benefit not only fine-grained recognition performance but also the reliability of model confidence, while preserving the efficiency and practicality of image-only inference.
Aishwarya Venkataramanan, Christian Ickler, Joachim Denzler
Computer Vision Group, Friedrich Schiller University Jena
Large-scale Vision-Language Models have demonstrated impressive transfer learning capabilities across a wide range of tasks. For few-shot classification, we observe that VLMs exhibit a notable ability to filter candidate categories and thus achieve high Top-K accuracy. However, they often struggle with fine-grained discrimination among visually similar categories, resulting in unsatisfactory Top-1 performance, as shown in Figure 1. Existing studies on VLM adapters generally focus on global alignment between visual and textual representations in the feature space, but fail to exploit semantically similar categories to refine fine-grained visual representations. Based on these observations, we propose a novel coarse-to-fine VLM fine-tuning approach for few-shot learning that leverages quantum computation, termed the Multi-Modal Quantum Adapter (MQAdapter). Specifically, MQAdapter first retrieves the Top-K category candidates most similar to the input image and uses them as semantic anchors. It then employs a cross-modal quantum learning mechanism to refine visual features under the guidance of these anchors. The core of this mechanism is the encoding of visual and textual features into quantum states. By leveraging quantum entanglement and superposition in a high-dimensional Hilbert space, MQAdapter effectively models higher-order cross-modal interactions, producing more discriminative representations than traditional Euclidean adapters. MQAdapter is parameter-efficient and can be integrated with various existing fine-tuning algorithms to achieve further performance gains. Evaluations on 15 datasets demonstrate the effectiveness of MQAdapter while requiring fewer trainable parameters.
Yumiao Zhao, Bo Jiang, Min Lu +2
Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, China · Origin Quantum Computing Company Limited, Hefei, Anhui 230026, China
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.
Adhemar de Senneville, Xavier Bou, Jérémy Anger +2
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France