Recent advances in zero-shot learning (ZSL) have demonstrated the potential of generative models. Typically, generative ZSL synthesizes visual features conditioned on semantic prototypes to model the data distribution of unseen classes, followed by training a classifier on the synthesized data. However, the synthesized features often remain task-agnostic, leading to degraded performance. Moreover, inferring a faithful distribution from semantic prototypes alone is insufficient for classes that are semantically similar but visually distinct. To address these and advance ZSL, we propose RLVC, an outcome-reward reinforcement learning RL framework with visual cues for generative ZSL. At its core, RL empowers the generative model to self-evolve, implicitly enhancing its generation capability. In particular, RLVC updates the generative model using an outcome-based reward, encouraging the synthesis of task-relevant features. Furthermore, we introduce class-wise visual cues that (i) align synthesized features with visual prototypes and (ii) stabilize the RL training updates. For the training process, we present a novel cold-start strategy. Comprehensive experiments and analyses on three prevalent ZSL benchmarks demonstrate that RLVC achieves state-of-the-art results with a 4.7% gain.
Figures & tables
Figure 1: Motivating illustration. (a) Existing generative ZSL methods train with adversarial losses conditioned only on semantic prototypes. This often leads to task-agnostic synthesized features and inter-class overlap. (b) Our RLVC incentivizes the generative model updating via RL reward and visual cues, enabling synthesized features that remain task-relevant and faithfully represent the data distribution.
Figure 2: Model architecture and training of RLVC. The top panel shows how we train the reward model with a visual encoder to produce fine-tuned visual features and reward signals. The bottom panel depicts how we update the policy model Gθ ( i.e . , generator) via outcome-reward reinforcement learning ( blue arrows ) and visual cues ( green arrows ), enabling synthesized features that remain task-relevant and faithfully represent the data distribution. x0 and x~0 denote the real and synthesized features of seen classes, respectively.
Figure 3: The training trends of our RLVC on CUB, including raw reward, EMA-adjusted advantage and ZSL accuracy.
Method
Backbone
CUB
SUN
AWA2
Acc
U
S
H
Acc
U
S
H
Acc
U
S
H
Embedding
CLIP ⋆ [ICML’21] [ 61 ]
ViT
–
55.2
54.8
55.0
–
–
–
–
–
–
–
–
TransZero++ [TPAMI’22] [ 8 ]
ResNet
78.3
67.5
73.6
70.4
67.6
48.6
37.8
42.5
72.6
64.6
82.7
72.5
DUET [AAAI’23] [ 19 ]
ViT
72.3
62.9
72.8
67.5
64.4
45.7
45.8
45.8
69.9
63.7
84.7
72.7
ICIS [ICCV’23] [ 20 ]
ResNet
60.6
45.8
73.7
56.5
51.8
45.2
25.6
32.7
64.6
35.6
93.3
51.6
HAS [ACM MM’23] [ 18 ]
ResNet
76.5
69.6
74.1
71.8
63.2
42.8
38.9
40.8
71.4
63.1
87.3
73.3
Table 1: Compared our RLVC with the SOTA methods in CZSL and GZSL settings on CUB, SUN and AWA2 benchmarks. The symbol “ ⋆ ” indicates the semantic prototypes from the class name. The symbol “–” denotes that no results are provided in the original papers. The bold and underlined markings indicate the best and second-best results, respectively.
Method
Semantic prototype
CUB
SUN
AWA2
Acc
U
S
H
Acc
U
S
H
Acc
U
S
H
Vanilla model
word embedding
61.8
52.4
66.3
58.5
72.0
55.4
46.4
50.5
73.1
62.8
82.3
71.2
RLVC
word embedding
62.8 (+1.0)
51.5
70.9
59.7 (+1.2)
72.1 (+0.1)
55.0
50.3
52.5 (+2.0)
74.5 (+1.4)
61.3
91.7
73.5 (+2.3)
Vanilla model
attribute vector
88.6
71.0
79.8
75.1
75.8
58.3
52.3
55.1
75.7
70.1
76.0
72.8
RLVC
attribute vector
90.1 (+1.5)
80.9
81.4
81.2 (+6.1)
77.7 (+1.9)
59.6
55.6
57.6 (+2.6)
84.0 (+8.3)
78.4
82.4
80.4 (+7.6)
Table 2: Effectiveness validation of RLVC across different semantic prototypes, including word embeddings of class names and expert-annotated attribute vectors. We mark the best results in bold and the accuracy gains (%) in parentheses .
Configuration
CUB
SUN
AWA2
Acc
H
Acc
H
Acc
H
RLVC w/o RL & visual cues
88.6
75.1
75.8
55.1
75.7
72.8
RLVC w/o RL ( i.e . , Eq. ( 11 ))
89.2
80.1
76.1
55.6
79.4
73.9
RLVC w/o visual cues ( i.e . , Eq. ( 13 ))
88.9
79.2
77.0
56.9
74.9
76.6
RLVC w/o fine-tuning visual encoder
89.2
77.5
76.0
56.4
81.1
76.3
RLVC w/o advantage ( i.e . , Eq. ( 9 ))
89.6
79.7
76.0
56.0
82.4
78.2
Table 3: Results of RLVC variants on CUB, SUN and AWA2 datasets. We ablate specific components to assess their effectiveness. The bold marking indicates the best results.
Loss
CUB
SUN
AWA2
Acc
H
Acc
H
Acc
H
KL
88.9
80.1
77.2
58.2
76.4
76.4
ℓ1
89.2
80.9
77.3
57.8
75.9
77.6
LPD
90.1
81.2
77.7
57.6
84.0
80.4
Table 4: Comparison results for different prototype-distillation losses combined with RLVC on CUB, SUN and AWA2 datasets. The bold marking indicates the best results.
Figure 4: Qualitative t-SNE visualization of RLVC on CUB: (a) without RL and visual cues, (b) without visual cues, and (c) full RLVC. We use real features of seen classes and synthetic features of unseen classes. Zoom in for details.
Figure 5: Effect of hyperparameters on CUB, including the epoch of RL cold-start, the coefficient of visual loss, and the number of synthetic unseen samples.
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
Generative zero-shot learning (ZSL) synthesizes visual features for unseen classes by learning a semantic-conditioned generator from seen classes. Since semantic conditions determine the knowledge transfer from seen to unseen classes, their effectiveness is critical for generating plausible features. However, existing class-level semantic prototypes lack semantic diversity to capture intra-class variations, while exhibiting inconsistent inter-class structures with the visual space due to the semantic-visual gap. To address these issues, we propose Adaptive Attribute Distribution and Visual Structure Alignment (AAVS), a unified semantic condition learning framework for generative ZSL. Specifically, the Adaptive Attribute Distribution (AAD) learns dimension-adaptive attribute distributions with discriminability-guided variation calibration to capture transferable semantic diversity, while Visual Structure Alignment (VSA) aligns the diverse attributes with refined visual prototypes to learn visual inter-class structures. By jointly modeling semantic diversity and visual structural consistency, AAVS provides more effective generation conditions for unseen classes to generate visual features. Extensive quantitative and qualitative evaluations on three benchmarks demonstrate that AAVS outperforms existing generative ZSL methods and validate the effectiveness of the learned generation conditions.
Haojie Pu, Zhuoming Li, Yongbiao Gao +3
School of Computer Science and Engineering, Southeast University, Nanjing, China
Zero-shot learning (ZSL) aims to recognize unseen classes without visual instances. However, existing methods usually assume clean labels, overlooking real-world label noise and ambiguity, which degrades performance. To bridge this gap, we propose the Dynamic Visual-semantic Alignment (DVSA), a robust ZSL framework for learning from ambiguous labels. DVSA uses a bidirectional visual-semantic alignment module with attention to mutually calibrate visual features and attribute prototypes, and a contrastive optimization grounded in Mutual Information (MI) at the attribute level to strengthen discriminative, semantically consistent attributes. In addition, a dynamic label disambiguation mechanism iteratively corrects noisy supervision while preserving semantic consistency, narrowing the instance-label gap, and improving generalization. Extensive experiments on standard benchmarks verify that DVSA achieves stronger performance under ambiguous supervision.
Jiangnan Li, Linqing Huang, Xiaowen Yan +3
College of Computer Science and Technology, Qingdao University · School of Computer Science, Shanghai JiaoTong University · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)
Zero-shot learning (ZSL) aims to recognize the unseen classes in the open-world guided by the side-information (e.g., attributes). Its key task is how to infer the latent semantic knowledge between visual and attribute features on seen classes, and thus conducting a desirable semantic knowledge transfer from seen classes to unseen ones. Prior works simply utilize unidirectional attention within a weakly-supervised manner to learn the spurious and limited latent semantic representations, which fail to effectively discover the intrinsic semantic knowledge (e.g., attribute semantic) between visual and attribute features. To solve the above challenges, we propose a mutually causal semantic distillation network (termed MSDN++) to distill the intrinsic and sufficient semantic representations for ZSL. MSDN++ consists of an attribute→visual causal attention sub-net that learns attribute-based visual features, and a visual→attribute causal attention sub-net that learns visual-based attribute features. The causal attentions encourages the two sub-nets to learn causal vision-attribute associations for representing reliable features with causal visual/attribute learning. With the guidance of semantic distillation loss, the two mutual attention sub-nets learn collaboratively and teach each other throughout the training process. Extensive experiments on three widely-used benchmark datasets (e.g., CUB, SUN, AWA2, and FLO) show that our MSDN++ yields significant improvements over the strong baselines, leading to new state-of-the-art performances.
Shiming Chen, Shuhuang Chen, Guo-Sen Xie +1
Huazhong University of Science and Technology, Wuhan, China · National Anti-Counterfeit Engineering Research Center, Wuhan, Chna · Nanjing University of Science and Technology, Nanjing, China