Mutually Causal Semantic Distillation Network for Zero-Shot Learning
Authors: Shiming Chen, Shuhuang Chen, Guo-Sen Xie, Xinge You
Organizations: Huazhong University of Science and Technology, Wuhan, China · National Anti-Counterfeit Engineering Research Center, Wuhan, Chna · Nanjing University of Science and Technology, Nanjing, China
Zero-shot learning (ZSL) aims to recognize the unseen classes in the open-world guided by the side-information (e.g., attributes). Its key task is how to infer the latent semantic knowledge between visual and attribute features on seen classes, and thus conducting a desirable semantic knowledge transfer from seen classes to unseen ones. Prior works simply utilize unidirectional attention within a weakly-supervised manner to learn the spurious and limited latent semantic representations, which fail to effectively discover the intrinsic semantic knowledge (e.g., attribute semantic) between visual and attribute features. To solve the above challenges, we propose a mutually causal semantic distillation network (termed MSDN++) to distill the intrinsic and sufficient semantic representations for ZSL. MSDN++ consists of an attribute→visual causal attention sub-net that learns attribute-based visual features, and a visual→attribute causal attention sub-net that learns visual-based attribute features. The causal attentions encourages the two sub-nets to learn causal vision-attribute associations for representing reliable features with causal visual/attribute learning. With the guidance of semantic distillation loss, the two mutual attention sub-nets learn collaboratively and teach each other throughout the training process. Extensive experiments on three widely-used benchmark datasets (e.g., CUB, SUN, AWA2, and FLO) show that our MSDN++ yields significant improvements over the strong baselines, leading to new state-of-the-art performances.
Figures & tables
Figure 1: Four investigated ZSL paradigms. (a) Embedding-based method. (b) Generative method. (c) Common space learning method. (d) Ours proposed mutually causal semantic distillation network (MSDN++). The semantic space S is represented by the class semantic vector annotated by humans based on the attribute descriptions. The visual space V is learned by a network backbone ( e.g. , ResNet101 ( He et al., 2016 ) ). The common space O is a shared latent space between visual mapping and semantic mapping. The attribute space A is learned by a language model ( e.g. , Glove ( Pennington et al., 2014 ) ). Filled triangles, circles, squares and diamonds denote the sample features in S , V , O and A , respectively.
Figure 2: The pipeline of MSDN++. MSDN++ consists of an attribute → visual causal attention sub-net (AVCA) and visual → attribute causal attention sub-net (VACA). AVCA learns the attribute-based visual features F with attribute-based visual learning and causal visual learning, while VACA discovers the vision-based attribute features S with vision-based attribute learning and causal attribute learning. Then, tow mapping functions M1 and M2 map the visual features and attribute features into semantic space as semantic representations ψ(x) and Ψ(x) , respectively. A semantic distillation loss Ldistill to match the probability estimates of the two sub-nets for semantic distillation ( i.e. , p1 and p2 ), enabling MSDN++ to learn intrinsic semantic knowledge. During inference, we fuse the predictions of the two sub-nets to take full use of the complementary semantic representations.
Figure 3: Part of samples on four challenge datasets, including three fine-grained datasets ( i.e. , (a) CUB ( Welinder et al., 2010 ) , (b) SUN ( Patterson and Hays, 2012 ) , and FLO Nilsback and Zisserman (2008) ), and one coarse-grained dataset ( i.e. , AWA2 ( Xian et al., 2017 ) ). Each image is sampled from different classes. We find that the images of different classes in fine-grained dataset are very similar and not easy to be distinguished, while the images in the coarse-grained dataset are more easy to be recognized. For example, CUB includes similar bird images, but AWA2 consists of various animal images.
Methods
CUB
SUN
AWA2
CZSL
GZSL
CZSL
GZSL
CZSL
GZSL
acc
U
S
H
acc
U
S
H
acc
U
S
H
Generative Methods
f-CLSWGAN ( Xian et al., 2018 )
57.3
43.7
57.7
49.7
60.8
42.6
36.6
39.4
68.2
57.9
61.4
59.6
f-VAEGAN-D2 ( Xian et al., 2019 )
61.0
48.4
60.1
53.6
64.7
45.1
38.0
41.3
71.1
57.6
70.6
63.5
Composer ∗ ( Huynh and Elhamifar, 2020b )
69.4
56.4
63.8
59.9
62.6
55.1
22.0
31.4
71.5
62.1
77.3
68.8
Table 1: Results ( % ) of the state-of-the-art CZSL and GZSL models on CUB, SUN, and AWA2, including generative methods, common space-based methods, and embedding-based methods. The best and second-best results are marked in Red and Blue , respectively. The symbol “–” indicates no results. The symbol “ * ” denotes attention-based methods. The symbol † denotes ZSL methods based on large-scale vision-language model.
Figure 4: Visualization of attention maps learned by the first sub-nets of MSDN ( Chen et al., 2022b ) and MSDN++ on CUB. We show the top-10 attention maps focused by models. The red boxes indicate MSDN learns the wrong attention maps that are irrelevant to the corresponding attributes.
Methods
FLO
U
S
H
f-CLSWGAN ( Xian et al., 2018 )
59.3
74.2
65.9
f-VAEGAN-D2 ( Xian et al., 2019 )
56.8
74.9
64.6
TF-VAEGAN ( Narayan et al., 2020 )
62.5
84.1
71.7
EG-part-net ( Chen et al., 2024c )
61.5
81.4
70.1
MSDN ( Conference Version ) ( Chen et al., 2022b )
62.2
81.0
70.3
Table 2: Results ( % ) of the state-of-the-art GZSL models on FLO.
Figure 5: Visualization of attention maps for the two mutual attention sub-nets (i.e, MSDN++(AVCA) and MSDN++(VACA)). Results show that our AVCA and VACA subnets can overally learn the accurate visual localizations, but they also learn few of falure cases.
Figure 6: The attention maps of various causal interventions, including (a) random attention, (b) uniform attention, and (c) reversed attention.
Figure 7: t-SNE visualizations of visual features for (a) seen classes and (b) unseen classes, learned by the baseline, MSDN ( Chen et al., 2022b ) , and MSDN++. The 10 colors denote 10 different seen/unseen classes randomly selected from CUB. Results show that our MSDN++ learn the intrinsic semantic representations both in seen and unseen classes compared to the baseline. Meanwhile, the each subnet of MSDN++ consistently enhances the intra-class compactness and inter-classes separability for MSDN.
Method
CUB
AWA2
acc
U
S
H
acc
U
S
H
baseline
57.4
44.2
55.2
49.1
54.8
30.3
30.7
30.5
baseline w/ LAR
58.5
46.5
54.6
50.2
56.9
20.6
89.7
33.5
MSDN++(AVCA) w/o Ldistill
76.2
68.7
69.1
68.9
71.9
65.5
76.8
70.7
MSDN++(VACA) w/o Ldistill
68.4
57.5
65.6
61.3
68.0
56.5
74.6
64.3
MSDN++(AVCA) w/ Ldistill
77.7
69.6
70.0
69.8
73.0
66.4
79.3
72.3
Table 3: Ablation studies for different components of MSDN++. The baseline is the visual feature extracted from CNN backbone with a global average pooling and then mapped into semantic embedding for ZSL.
Figure 8: The effectiveness of the combination coefficients ( α1,α2 ) between the AVCA and VACA sub-nets. Results show that MSDN++ performs poorly when α1/α2 is set small, and its performances drop down when α1/α2 is set too large. Because the attribute-based visual features and visual-based attribute features are complementary for discriminative semantic embedding representations.
Method
CUB
acc
U
S
H
MSDN ( Chen et al., 2022b )
76.1
68.7
67.5
68.1
MSDN++ w/ (a)
78.5
70.8
70.3
70.6
MSDN++ w/ (b)
78.4
70.8
70.0
70.4
MSDN++ w/ (c)
78.0
71.0
69.9
70.5
MSDN++ w/ (d)
78.5
71.8
69.5
70.6
Table 4: Effects of various causal interventions on CUB, i.e. , (a) Random Attention, (b) Uniform Attention, (c) Reversed Attention, and (d) Random Attention+Reversed Attention.
Figure 9: The effects of λcal , λAR , λcausal , and λdistill on CUB (top-row) and AWA2 (bottom-row). Results show that MSDN++ is robust when the loss weights are set to small and the performance will drop rapidly when loss weights are set to too large. Because the large loss weights will hamper the balance of various losses.
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
Zero-shot learning (ZSL) aims to recognize unseen classes without visual instances. However, existing methods usually assume clean labels, overlooking real-world label noise and ambiguity, which degrades performance. To bridge this gap, we propose the Dynamic Visual-semantic Alignment (DVSA), a robust ZSL framework for learning from ambiguous labels. DVSA uses a bidirectional visual-semantic alignment module with attention to mutually calibrate visual features and attribute prototypes, and a contrastive optimization grounded in Mutual Information (MI) at the attribute level to strengthen discriminative, semantically consistent attributes. In addition, a dynamic label disambiguation mechanism iteratively corrects noisy supervision while preserving semantic consistency, narrowing the instance-label gap, and improving generalization. Extensive experiments on standard benchmarks verify that DVSA achieves stronger performance under ambiguous supervision.
Jiangnan Li, Linqing Huang, Xiaowen Yan +3
College of Computer Science and Technology, Qingdao University · School of Computer Science, Shanghai JiaoTong University · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)
Zero-Shot Learning (ZSL) facilitates knowledge transfer via shared semantic spaces. However, a critical bottleneck in this paradigm is Semantic Entanglement, where visual representations are inevitably conflated with visually similar semantic concepts, such as distinguishing the intrinsic traits of a Wolf from the shared features of a Husky. Existing global alignment methods often indiscriminately maximize correlations between visual and semantic modalities, leading models to overfit spurious similarities rather than capturing distinctive class identities. To address this fundamental limitation, we propose the Causal-Visual Dynamic Label Refinement (CV-DCLR) framework. Unlike traditional approaches that rely on superficial visual statistics, CV-DCLR recalibrates visual-semantic associations via a Dual-Stream Mutual Correction Mechanism. This includes a Visual Likelihood Stream to model observational patterns and a Causal Importance Stream that verifies the structural necessity of candidate prototypes through Counterfactual Intervention. Acting as a logical filter, our adaptive gating mechanism dynamically modulates feature responses to amplify genuine causal traits while suppressing visually plausible but structurally irrelevant distractors. Extensive experiments on the CUB, SUN, and AWA2 benchmarks under a rigorous Semantic Entanglement Injection protocol demonstrate that CV-DCLR significantly outperforms state-of-the-art methods in high-ambiguity scenarios. Specifically, while existing models suffer catastrophic degradation under entanglement, our framework maintains robust performance, effectively disentangling true class identities from semantic confounders.
Can Wang, Jiangnan Li, Mingyu Li +4
School of Computer Science and Technology, Qingdao University, Qingdao, China · School of Software Engineering, Tongji University, Shanghai, China
Generative zero-shot learning (ZSL) synthesizes visual features for unseen classes by learning a semantic-conditioned generator from seen classes. Since semantic conditions determine the knowledge transfer from seen to unseen classes, their effectiveness is critical for generating plausible features. However, existing class-level semantic prototypes lack semantic diversity to capture intra-class variations, while exhibiting inconsistent inter-class structures with the visual space due to the semantic-visual gap. To address these issues, we propose Adaptive Attribute Distribution and Visual Structure Alignment (AAVS), a unified semantic condition learning framework for generative ZSL. Specifically, the Adaptive Attribute Distribution (AAD) learns dimension-adaptive attribute distributions with discriminability-guided variation calibration to capture transferable semantic diversity, while Visual Structure Alignment (VSA) aligns the diverse attributes with refined visual prototypes to learn visual inter-class structures. By jointly modeling semantic diversity and visual structural consistency, AAVS provides more effective generation conditions for unseen classes to generate visual features. Extensive quantitative and qualitative evaluations on three benchmarks demonstrate that AAVS outperforms existing generative ZSL methods and validate the effectiveness of the learned generation conditions.
Haojie Pu, Zhuoming Li, Yongbiao Gao +3
School of Computer Science and Engineering, Southeast University, Nanjing, China