Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label
Authors: Cheng Chen, Yifan Zhao, Jia Li
Organizations: State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing, 100191, China
Learning multi-label image classification with incomplete annotations is a challenging task that has been widely studied for its superior trade-off between high efficiency and less labor consumption on large-scale datasets. Predominant methods rely on strong prior assumptions to recover the missing semantics from partial annotations. However, these statistic priors suffer from unstable semantic mistakes and thus lead to catastrophic overfitting. Toward this end, we propose a Language-driven Dense Semantic Adaptor (LDSA) that excavates prior-adaptive relationships from multimodal pretrained CLIP models. In our approach, the densely contrastive adaptor is first proposed to construct dense visual contrastive constraints, transferring the task-specific knowledge to visual domains. We then propose a language-driven interactive decoder with the help of class-specific prompt tuning, which adapts language proxies with visual domains. With the collaborative learning of proposed modules, experimental results demonstrate our proposed LDSA achieves a new state of the art on public multi-label classification benchmarks, and interpretable analyses reveal that our LDSA discovers implicit semantic relationships with the prior-adaptive learning scheme.
Figures & tables
Fig. 1: The motivation of our approach. a) Existing works learn semantic relationships with prior assumptions, e.g. , label co-occurrence, which leads to semantic mistakes in specific scenarios. b) Our approach aims to excavate adaptive relationships from generalized knowledge in CLIP and transfer them to downstream partial multi-label learning.
Fig. 2: The proposed LDSA framework consists of two major steps: 1) The densely contrastive adaptor transfers and constrains the semantic representations from generalized CLIP visual encoders. 2) Pretrained language knowledge adapts to visual domains with prompt learning and transferring the text priors with the language-driven interactive decoder.
Fig. 3: Illustrations of the densely contrastive adaptor. The adaptor consists of dense feature regularizers and transformer encoders, connected with residual connections. Initially, the strong and weak versions of the same image do not exhibit the expected correspondence in latent space, leading to inconsistent distribution.
Fig. 4: Illustration of language-guided learning. Our proposed method first uses class-specific learnable prompts to guide the alignment of visual and text domains. Then we construct a cross-modal interaction to guide the representation of dense visual features by exploiting the transformer architecture. We finally construct the compact semantic prediction instead of the widely used fc layers.
Fig. 5: The illustration of feature space alignment with proposed LDSA on the MS-COCO dataset, where the distributions of the real image feature and prompt prototypes are visualized for each step. 1) With pretrained CLIP, prototypes are distributed wisely while contrastive weak and hard views of images are incorrectly distributed in more than one manifold. 2) After adapting, image features are correctly clustered, however, the prototypes are corrupted . 3) With language-guided learning, LDSA learns the optimal representations, capturing the correspondence of images, as well as relationships between labels.
Benchmarks
Methods
10%
20%
30%
40%
50%
60%
70%
80%
90%
Avg.
MS-COCO [ 38 ]
SSGRL [ 11 ]
62.5
70.5
73.2
74.5
76.3
76.5
77.1
77.9
78.4
74.1
GCN-ML [ 14 ]
63.8
70.9
72.8
74.0
76.7
77.1
77.3
78.3
78.6
74.4
KGGR [ 8 ]
66.6
71.4
73.8
76.7
77.5
77.9
78.4
78.7
79.1
75.6
Curriculum Learning [ 21 ]
26.7
31.8
51.5
65.4
70.0
71.9
74.0
77.4
78.0
60.7
Partial BCE [ 21 ]
61.6
70.5
74.1
76.3
77.2
77.7
78.2
78.4
78.5
74.7
SST [ 10 ]
68.1
73.5
75.9
77.3
78.1
78.9
79.2
79.6
79.9
76.7
TABLE I: Experimental comparison using mAP (%) with the state-of-the-art methods on the PASCAL VOC, MS-COCO, and VG-200 benchmarks.
DF
Adaptor
LG
Prompt
mAP
Δ
Fine-tuned CLIP
34.0
{\color[rgb]{0,0,0}(-33.3)}
+BYOL
71.2
(+3.9)
+MoCo
75.0
(+7.7)
+DetCo
59.2
(−8.1)
+UP-DETR
68.5
(+1.2)
+ORL
39.7
(−27.6)
TABLE II: Ablation study for different modules on MS-COCO dataset. DF: Resolution of Dense Feature. LG: Language Guidance.
Adaptor
Augumentations
mAP
Δ
×
weak
77.1
{\color[rgb]{0,0,0}(+0.0)}
×
strong
74.1
{\color[rgb]{0,0,1}(-3.0)}
×
weak + strong
77.9
{\color[rgb]{1,0,0}(+0.8)}
✓
weak
77.9
{\color[rgb]{1,0,0}(+0.8)}
✓
strong
70.8
{\color[rgb]{0,0,1}(-6.3)}
✓
weak + strong
79.1
{\color[rgb]{1,0,0}(+2.0)}
TABLE III: Ablation study for detailed designs of densely contrastive adaptor on MS-COCO dataset.
Regularizers
Augmentations
mAP
Δ
DE
BN
Weak
Strong
✓
77.1
{\color[rgb]{0,0,0}(+0.0)}
✓
✓
77.9
{\color[rgb]{1,0,0}(+0.8)}
✓
✓
76.3
{\color[rgb]{0,0,1}(-0.8)}
✓
✓
✓
78.6
{\color[rgb]{1,0,0}(+1.5)}
✓
✓
77.2
{\color[rgb]{1,0,0}(+0.1)}
TABLE IV: Detailed ablation study for dense feature regularizers. DE: Dense encoder. BN: Dense batch norm.
Methods
mAP
OP
OR
OF1
CP
CR
CF1
AP
56.1
78.5
51.1
61.9
64.7
40.9
50.1
FP
73.9
80.1
67.8
73.4
78.2
60.5
68.2
CP
79.1
84.5
72.6
78.1
81.4
67.5
73.8
TABLE V: Ablation study for semantic-aware compact prediction. CP: Semantic-aware compact prediction. AP: Average prediction. FP: Fully-connected prediction.
Benchmarks
Methods
OP
OR
OF1
CP
CR
CF1
MS-COCO [ 38 ]
SSGRL [ 11 ]
86.3
64.8
73.9
82.1
58.4
68.1
GCN-ML [ 14 ]
85.2
64.2
73.1
81.8
58.9
68.4
KGGR [ 8 ]
84.0
65.6
73.7
81.4
60.9
69.7
Curriculum Learning [ 21 ]
87.8
51.0
61.9
60.9
40.4
48.3
Partial BCE [ 21 ]
86.7
64.7
74.0
83.1
58.9
68.8
SST [ 10 ]
86.3
67.7
75.8
82.8
62.6
71.2
TABLE VI: Different evaluation metrics (OP, OR, OF1, CP, CR, CF1) on the PASCAL VOC [ 22 ] , MS-COCO [ 38 ] , and VG-200 [ 34 ] benchmarks.
TABLE VII: The statistics for the inference time of ours and three state-of-the-art methods on the MS-COCO dataset.
Fig. 6: The mAP during training under different contrastive configurations.
Fig. 7: Visualization of class activation maps on MS-COCO [ 38 ] via GradCAM++ [ 6 ] . CAMs in the first and second lines of each group are exported from our LDSA and the baseline respectively. The confidences predicted by models are labeled on CAMs. The labels that are not present in images are colored red.
Fig. 8: The visualization of distances and distributions of language prototypes.
Fig. 9: The feature manifolds of augmented images. Our model in c) clusters samples by its semantics while the ablated model w/o adaptor focuses on the visual appearances with a clear decision boundary in f).
CL
w/o Regularizers
w/ Regularizers
Ours Implicit
77.9{\color[rgb]{0,0,0}(+0.0)}
79.1{\color[rgb]{0,0,0}(+0.0)}
+ InfoNCE
78.0{\color[rgb]{1,0,0}(+0.1)}
78.7{\color[rgb]{0,0,1}(-0.4)}
+ L1 Loss
77.8{\color[rgb]{0,0,1}(-0.1)}
79.1{\color[rgb]{0,0,0}(+0.0)}
TABLE VIII: Ablation study for contrastive effects on MS-COCO. CL: Contrastive learning. Regularizers: Dense feature regularizers.
Fig. 10: Semantic relationships of representative labels on MS-COCO. The labels are connected by the cosine similarity of our language prototypes ( LDSA ), label CO-occurrence exported from the training set ( CO. ), and cosine similarity of encoded prompts by vanilla CLIP ( CLIP ), respectively. We highlight three edges and analyze their relationships.
Methods
1-shot
5-shot
LaSO [ 2 ]
45.3
58.1
ML-FSL [ 48 ]
54.4
63.6
CoOp [ 66 ]
46.9
55.6
Tip-Adapter [ 61 ]
53.8
59.7
Ours
56.3
60.0
TABLE IX: Few-shot experimental comparison using mAP (%) on the MS-COCO dataset.