Mutually Causal Semantic Distillation Network for Zero-Shot Learning
Authors: Shiming Chen, Shuhuang Chen, Guo-Sen Xie, Xinge You
Organizations: Huazhong University of Science and Technology, Wuhan, China · National Anti-Counterfeit Engineering Research Center, Wuhan, Chna · Nanjing University of Science and Technology, Nanjing, China
Zero-shot learning (ZSL) aims to recognize the unseen classes in the open-world guided by the side-information (e.g., attributes). Its key task is how to infer the latent semantic knowledge between visual and attribute features on seen classes, and thus conducting a desirable semantic knowledge transfer from seen classes to unseen ones. Prior works simply utilize unidirectional attention within a weakly-supervised manner to learn the spurious and limited latent semantic representations, which fail to effectively discover the intrinsic semantic knowledge (e.g., attribute semantic) between visual and attribute features. To solve the above challenges, we propose a mutually causal semantic distillation network (termed MSDN++) to distill the intrinsic and sufficient semantic representations for ZSL. MSDN++ consists of an attribute→visual causal attention sub-net that learns attribute-based visual features, and a visual→attribute causal attention sub-net that learns visual-based attribute features. The causal attentions encourages the two sub-nets to learn causal vision-attribute associations for representing reliable features with causal visual/attribute learning. With the guidance of semantic distillation loss, the two mutual attention sub-nets learn collaboratively and teach each other throughout the training process. Extensive experiments on three widely-used benchmark datasets (e.g., CUB, SUN, AWA2, and FLO) show that our MSDN++ yields significant improvements over the strong baselines, leading to new state-of-the-art performances.
Figures & tables
Figure 1: Four investigated ZSL paradigms. (a) Embedding-based method. (b) Generative method. (c) Common space learning method. (d) Ours proposed mutually causal semantic distillation network (MSDN++). The semantic space S is represented by the class semantic vector annotated by humans based on the attribute descriptions. The visual space V is learned by a network backbone ( e.g. , ResNet101 ( He et al., 2016 ) ). The common space O is a shared latent space between visual mapping and semantic mapping. The attribute space A is learned by a language model ( e.g. , Glove ( Pennington et al., 2014 ) ). Filled triangles, circles, squares and diamonds denote the sample features in S , V , O and A , respectively.
Figure 2: The pipeline of MSDN++. MSDN++ consists of an attribute → visual causal attention sub-net (AVCA) and visual → attribute causal attention sub-net (VACA). AVCA learns the attribute-based visual features F with attribute-based visual learning and causal visual learning, while VACA discovers the vision-based attribute features S with vision-based attribute learning and causal attribute learning. Then, tow mapping functions M1 and M2 map the visual features and attribute features into semantic space as semantic representations ψ(x) and Ψ(x) , respectively. A semantic distillation loss Ldistill to match the probability estimates of the two sub-nets for semantic distillation ( i.e. , p1 and p2 ), enabling MSDN++ to learn intrinsic semantic knowledge. During inference, we fuse the predictions of the two sub-nets to take full use of the complementary semantic representations.
Figure 3: Part of samples on four challenge datasets, including three fine-grained datasets ( i.e. , (a) CUB ( Welinder et al., 2010 ) , (b) SUN ( Patterson and Hays, 2012 ) , and FLO Nilsback and Zisserman (2008) ), and one coarse-grained dataset ( i.e. , AWA2 ( Xian et al., 2017 ) ). Each image is sampled from different classes. We find that the images of different classes in fine-grained dataset are very similar and not easy to be distinguished, while the images in the coarse-grained dataset are more easy to be recognized. For example, CUB includes similar bird images, but AWA2 consists of various animal images.
Methods
CUB
SUN
AWA2
CZSL
GZSL
CZSL
GZSL
CZSL
GZSL
acc
U
S
H
acc
U
S
H
acc
U
S
H
Generative Methods
f-CLSWGAN ( Xian et al., 2018 )
57.3
43.7
57.7
49.7
60.8
42.6
36.6
39.4
68.2
57.9
61.4
59.6
f-VAEGAN-D2 ( Xian et al., 2019 )
61.0
48.4
60.1
53.6
64.7
45.1
38.0
41.3
71.1
57.6
70.6
63.5
Composer ∗ ( Huynh and Elhamifar, 2020b )
69.4
56.4
63.8
59.9
62.6
55.1
22.0
31.4
71.5
62.1
77.3
68.8
Table 1: Results ( % ) of the state-of-the-art CZSL and GZSL models on CUB, SUN, and AWA2, including generative methods, common space-based methods, and embedding-based methods. The best and second-best results are marked in Red and Blue , respectively. The symbol “–” indicates no results. The symbol “ * ” denotes attention-based methods. The symbol † denotes ZSL methods based on large-scale vision-language model.
Figure 4: Visualization of attention maps learned by the first sub-nets of MSDN ( Chen et al., 2022b ) and MSDN++ on CUB. We show the top-10 attention maps focused by models. The red boxes indicate MSDN learns the wrong attention maps that are irrelevant to the corresponding attributes.
Methods
FLO
U
S
H
f-CLSWGAN ( Xian et al., 2018 )
59.3
74.2
65.9
f-VAEGAN-D2 ( Xian et al., 2019 )
56.8
74.9
64.6
TF-VAEGAN ( Narayan et al., 2020 )
62.5
84.1
71.7
EG-part-net ( Chen et al., 2024c )
61.5
81.4
70.1
MSDN ( Conference Version ) ( Chen et al., 2022b )
62.2
81.0
70.3
Table 2: Results ( % ) of the state-of-the-art GZSL models on FLO.
Figure 5: Visualization of attention maps for the two mutual attention sub-nets (i.e, MSDN++(AVCA) and MSDN++(VACA)). Results show that our AVCA and VACA subnets can overally learn the accurate visual localizations, but they also learn few of falure cases.
Figure 6: The attention maps of various causal interventions, including (a) random attention, (b) uniform attention, and (c) reversed attention.
Figure 7: t-SNE visualizations of visual features for (a) seen classes and (b) unseen classes, learned by the baseline, MSDN ( Chen et al., 2022b ) , and MSDN++. The 10 colors denote 10 different seen/unseen classes randomly selected from CUB. Results show that our MSDN++ learn the intrinsic semantic representations both in seen and unseen classes compared to the baseline. Meanwhile, the each subnet of MSDN++ consistently enhances the intra-class compactness and inter-classes separability for MSDN.
Method
CUB
AWA2
acc
U
S
H
acc
U
S
H
baseline
57.4
44.2
55.2
49.1
54.8
30.3
30.7
30.5
baseline w/ LAR
58.5
46.5
54.6
50.2
56.9
20.6
89.7
33.5
MSDN++(AVCA) w/o Ldistill
76.2
68.7
69.1
68.9
71.9
65.5
76.8
70.7
MSDN++(VACA) w/o Ldistill
68.4
57.5
65.6
61.3
68.0
56.5
74.6
64.3
MSDN++(AVCA) w/ Ldistill
77.7
69.6
70.0
69.8
73.0
66.4
79.3
72.3
Table 3: Ablation studies for different components of MSDN++. The baseline is the visual feature extracted from CNN backbone with a global average pooling and then mapped into semantic embedding for ZSL.
Figure 8: The effectiveness of the combination coefficients ( α1,α2 ) between the AVCA and VACA sub-nets. Results show that MSDN++ performs poorly when α1/α2 is set small, and its performances drop down when α1/α2 is set too large. Because the attribute-based visual features and visual-based attribute features are complementary for discriminative semantic embedding representations.
Method
CUB
acc
U
S
H
MSDN ( Chen et al., 2022b )
76.1
68.7
67.5
68.1
MSDN++ w/ (a)
78.5
70.8
70.3
70.6
MSDN++ w/ (b)
78.4
70.8
70.0
70.4
MSDN++ w/ (c)
78.0
71.0
69.9
70.5
MSDN++ w/ (d)
78.5
71.8
69.5
70.6
Table 4: Effects of various causal interventions on CUB, i.e. , (a) Random Attention, (b) Uniform Attention, (c) Reversed Attention, and (d) Random Attention+Reversed Attention.
Figure 9: The effects of λcal , λAR , λcausal , and λdistill on CUB (top-row) and AWA2 (bottom-row). Results show that MSDN++ is robust when the loss weights are set to small and the performance will drop rapidly when loss weights are set to too large. Because the large loss weights will hamper the balance of various losses.
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
College of Computer Science and Technology, Qingdao University · School of Computer Science, Shanghai JiaoTong University · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)