Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.
Figures & tables
Figure 1: (a) CZSL aims to recognize unseen attribute-object compositions by learning from seen compositions. (b) Comparison between CLIP and diffusion embedding spaces. CLIP tends to form object-centric clusters, resulting in limited shared compositional structure across different objects. Diffusion representations exhibit more consistent compositional patterns, leading to a more structured and shared compositional subspace. (c) Prior evidence from the Winoground compositional reasoning benchmark, adapted from ( Li et al., 2023a ) . Diffusion-based representations substantially outperform CLIP on object, relation, and joint compositional reasoning, suggesting that diffusion models capture compositional semantics more effectively than contrastive representations. (d) Overview of our approach: We propose to regularize the CLIP embeddings with diffusion features through representation alignment for better compositional generalization.
Figure 2: Overview of our DIFFCZSL framework. (a) Existing CLIP-based CZSL framework. (b) DIFF-CZSL introduces diffusion-derived features as auxiliary supervision during training. The diffusion branch is removed at inference time, incurring no extra inference cost.
MIT-States ( Isola et al., 2015 )
UT-Zappos ( Yu and Grauman, 2014 )
C-GQA ( Naeem et al., 2021 )
Method
Seen
Unseen
HM
AUC
Seen
Unseen
HM
AUC
Seen
Unseen
HM
AUC
Closed-world Results
CLIP [ICML21]
30.2
46.0
26.1
11.0
15.8
49.1
15.6
5.0
7.5
25.0
8.6
1.4
CoOP [IJCV22]
34.4
47.6
29.8
13.5
52.1
49.3
34.6
18.8
20.5
26.8
17.1
4.4
DFSP(i2t) [CVPR23]
47.4
52.4
37.2
20.7
64.2
66.4
45.1
32.1
35.6
29.3
24.3
8.7
DFSP(BiF) [CVPR23]
47.1
52.8
37.7
20.8
63.3
69.2
47.1
33.5
36.5
32.0
26.2
9.9
Table 1: Quantitative comparison (§ 5.2 ) on three benchmarks under closed-world and open-world settings. * denotes results from our implementation. ‡ represents the methods leveraging the LLM’s knowledge.
Table 4Table 5
Method
MIT-States
UT-Zappos
S
U
HM
AUC
S
U
HM
AUC
Troika
49.0
53.0
39.2
22.1
66.8
73.4
54.6
41.7
+ DINOv3
49.6
53.3
39.6
22.3
70.5
72.5
54.2
42.5
+ DIFF
50.5
53.4
39.9
23.1
72.2
74.0
57.6
46.2
Table 6: Comparison with alternative pretrained supervision on MIT-States and UT-Zappos.
Figure 3: Visual comparison between CSP ( Nayak et al., 2022 ) , Troika ( Huang et al., 2024 ) , CAMS ( Yang et al., 2025 ) , and our method on the MIT-States ( Isola et al., 2015 ) and UT-Zappos ( Yu and Grauman, 2014 ) datasets. Green denotes correct predictions, while red denotes incorrect predictions.
Method
MIT-States
UT-Zappos
C-GQA
CSP
15m 25s
9m 40s
36m 20s
+ DIFF
15m 57s
10m 07s
37m 17s
Troika
59m 26s
19m 36s
199m 35s
+ DIFF
65m 35s
22m 24s
203m 06s
CAMS
7m 35s
6m 36s
23m 39s
+ DIFF
9m 18s
7m 41s
26m 15s
Table 7: Training time per epoch of the three baselines and their DIFF-augmented variants.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
MIT-States
UT-Zappos
C-GQA
Learning rate
5×10−5
5×10−4
5×10−5
Batch size
64
32
16
Gradient accumulation
2
2
4
Epochs
20
20
15
Image alignment weight λimg
0.1
0.5
0.1
Text alignment weight λtxt
1
0.5
0.8
Appendix
Table 9: Implementation details and hyperparameters for DIFF-CSP across different datasets.
Hyperparameter
MIT-States
UT-Zappos
C-GQA
Learning rate
1×10−4
5×10−4
5×10−5
Batch size
8
16
16
Gradient accumulation
8
4
1
Epochs
10
10
15
Image alignment weight λimg
0.7
0.5
0.7
Text alignment weight λtxt
0.3
0.5
0.3
Appendix
Table 10: Implementation details and hyperparameters for DIFF-Troika across different datasets.
Hyperparameter
MIT-States
UT-Zappos
C-GQA
Learning rate
1×10−4
2.5×10−4
1×10−4
Batch size
64
64
16
Gradient accumulation
1
1
1
Epochs
15
15
15
Image alignment weight λimg
0.5
0.5
0.5
Text alignment weight λtxt
0.1
0.1
0.5
Appendix
Table 11: Implementation details and hyperparameters for the DIFF-CAMS across different datasets.
Table 12Table 13
Method
MIT-States
UT-Zappos
S
U
HM
AUC
S
U
HM
AUC
Troika† (baseline)
49.0 ± 0.4
53.0 ± 0.2
39.3 ± 0.2
22.1 ± 0.1
66.8 ± 1.1
73.8 ± 0.6
54.6 ± 0.5
41.7 ± 0.7
DIFF-Troika (ours)
49.9 ± 0.6
53.2 ± 0.2
39.6 ± 0.3
22.6 ± 0.4
71.0 ± 1.2
74.4 ± 0.4
57.7 ± 0.2
45.4 ± 0.8
Appendix
Table 16: Performance comparison between Troika and our method on MIT-States and UT-Zappos, averaged over 3 random seeds with standard deviations. † marks results taken from the original publications.
Method
Mean ↑
Median ↑
Q25 ↑
Q75 ↑
Max ↑
CLIP
0.1887
0.1818
0.1342
0.2310
0.4561
Diffusion
0.2012
0.1935
0.1552
0.2456
0.5654
Appendix
Table 17: Summary statistics of per-attribute consistency for diffusion and CLIP feature spaces on MIT-States.
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
Compositional zero-shot learning (CZSL) aims to recognize novel combinations of previously observed objects and attributes. Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization. The Geodesically Decomposable Embeddings (GDE) framework formalizes this by identifying directions in the tangent space at the Frechet mean of the embedding distribution, which encode primitive visual concepts. Unseen attribute-object compositions can be synthesized through vector addition of these primitives. We revisit the assumption that compositional generalization requires explicit attribute-object pair re-composition. By means of new compositional scoring functions and extensive evaluations, we show that directly using the decomposed primitives suffices for recognizing unseen compositions. In that, we gain insights on how the geometry of the embedding manifold (Euclidean, spherical, or hyperbolic) and dataset structure affect compositional reasoning. Across models (CLIP, DINOv2, MERU, HyCoCLIP), we show that compositional structure consistently emerges as primitive representations, enabling inference on CZSL benchmarks (MIT-States, VAW-CZSL, UT-Zappos50K, Clothing16K) and group robustness on Waterbirds and CelebA. Code: https://github.com/nis-research/Compositionality-with-Primitives.
Puru Vaish, Meelad Dashti, Christoph Brune +3
University of Twente, The Netherlands · Politecnico di Torino, Italy · University of Amsterdam, The Netherlands
Compositional zero-shot learning (CZSL) aims to recognize unseen attribute-object compositions by recombining primitives learned from seen pairs. Recent CZSL methods built on vision-language models (VLMs) typically adopt parameter-efficient fine-tuning (PEFT). They apply visual disentanglers for decomposition and manipulate token-level prompts or prefixes to encode compositions. However, such PEFT-based designs suffer from two fundamental limitations: (1) Implicit Composition Construction, where composition is realized only via token concatenation or branch-wise prompt tuning rather than an explicit operation in the embedding space; (2) Remained Feature Entanglement, where imperfect disentanglement leaves attribute, object, and composition features mutually contaminated. Together, these issues limit the generalization ability of current CZSL models. In this paper, we are the first to systematically study flow matching for CZSL and introduce FlowComposer, a model-agnostic framework that learns two primitive flows to transport visual features toward attribute and object text embeddings, and a learnable Composer that explicitly fuses their velocity fields into a composition flow. To exploit the inevitable residual entanglement, we further devise a leakage-guided augmentation scheme that reuses leaked features as auxiliary signals. We thoroughly evaluate FlowComposer on three public CZSL benchmarks by integrating it as a plug-and-play component into various baselines, consistently achieving significant improvements.
Zhenqi He, Lin Li, Long Chen
The Hong Kong University of Science and Technology
Compositional Zero-Shot Learning (CZSL) aims to combine known attributes and objects as primitives for recognizing previously unseen attribute-object pairs. Prior works either predict attributes and objects independently, missing their strong contextual dependency, or use unidirectional conditional modeling (e.g., object-guided attribute prediction), which is prone to error propagation. We propose PRPC, a Progressive Reasoning framework with Primitive Correction, which explicitly models the bidirectional dependency between attributes and objects via step-wise inference. PRPC performs mutual correction of primitives to suppress prediction errors in earlier steps. Specifically, we formulate CZSL as structured, Q&A-style Chain-of-Thought reasoning process and constrain the MLLM to follow predefined semantic steps to generate intermediate decisions. To further enhance the reliability and logical consistency of intermediate reasoning, we introduce reinforcement learning post-training with a GRPO-based objective, providing step-level rewards aligned with the progressive inference procedure. Extensive experiments on three CZSL benchmarks demonstrate that PRPC achieves state-of-the-art performance, validating the effectiveness of progressive reasoning and bidirectional correction for robust compositional generalization.
Ziyi Chen, Haoyan Shi, Sunhan Xu +1
School of Computer Science and Technology, Beijing Jiaotong University · Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education)