DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

Hangyu Tian; Zhenqi He; Yanghao Wang; Long Chen

cs.CVAug 20, 2026

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

Authors: Hangyu Tian, Zhenqi He, Yanghao Wang, Long Chen

Organizations: The Hong Kong University of Science and Technology

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.

Figures & tables

Figure 1: (a) CZSL aims to recognize unseen attribute-object compositions by learning from seen compositions. (b) Comparison between CLIP and diffusion embedding spaces. CLIP tends to form object-centric clusters, resulting in limited shared compositional structure across different objects. Diffusion representations exhibit more consistent compositional patterns, leading to a more structured and shared compositional subspace. (c) Prior evidence from the Winoground compositional reasoning benchmark, adapted from ( Li et al., 2023a ) . Diffusion-based representations substantially outperform CLIP on object, relation, and joint compositional reasoning, suggesting that diffusion models capture compositional semantics more effectively than contrastive representations. (d) Overview of our approach: We propose to regularize the CLIP embeddings with diffusion features through representation alignment for better compositional generalization.

Figure 2: Overview of our DIFFCZSL framework. (a) Existing CLIP-based CZSL framework. (b) DIFF-CZSL introduces diffusion-derived features as auxiliary supervision during training. The diffusion branch is removed at inference time, incurring no extra inference cost.

	MIT-States ( Isola et al., 2015 )				UT-Zappos ( Yu and Grauman, 2014 )				C-GQA ( Naeem et al., 2021 )
Method	Seen	Unseen	HM	AUC	Seen	Unseen	HM	AUC	Seen	Unseen	HM	AUC
Closed-world Results
CLIP [ICML21]	30.2	46.0	26.1	11.0	15.8	49.1	15.6	5.0	7.5	25.0	8.6	1.4
CoOP [IJCV22]	34.4	47.6	29.8	13.5	52.1	49.3	34.6	18.8	20.5	26.8	17.1	4.4
DFSP(i2t) [CVPR23]	47.4	52.4	37.2	20.7	64.2	66.4	45.1	32.1	35.6	29.3	24.3	8.7
DFSP(BiF) [CVPR23]	47.1	52.8	37.7	20.8	63.3	69.2	47.1	33.5	36.5	32.0	26.2	9.9

Table 1: Quantitative comparison (§ 5.2 ) on three benchmarks under closed-world and open-world settings. * denotes results from our implementation.

\ddagger

represents the methods leveraging the LLM’s knowledge.

Table 4

Table 5

Method	MIT-States				UT-Zappos
Method	S	U	HM	AUC	S	U	HM	AUC
Troika	49.0	53.0	39.2	22.1	66.8	73.4	54.6	41.7
+ DINOv3	49.6	53.3	39.6	22.3	70.5	72.5	54.2	42.5
+ DIFF	50.5	53.4	39.9	23.1	72.2	74.0	57.6	46.2

Table 6: Comparison with alternative pretrained supervision on MIT-States and UT-Zappos.

Figure 3: Visual comparison between CSP ( Nayak et al., 2022 ) , Troika ( Huang et al., 2024 ) , CAMS ( Yang et al., 2025 ) , and our method on the MIT-States ( Isola et al., 2015 ) and UT-Zappos ( Yu and Grauman, 2014 ) datasets. Green denotes correct predictions, while red denotes incorrect predictions.

Method	MIT-States	UT-Zappos	C-GQA
CSP	15m 25s	9m 40s	36m 20s
+ DIFF	15m 57s	10m 07s	37m 17s
Troika	59m 26s	19m 36s	199m 35s
+ DIFF	65m 35s	22m 24s	203m 06s
CAMS	7m 35s	6m 36s	23m 39s
+ DIFF	9m 18s	7m 41s	26m 15s

Table 7: Training time per epoch of the three baselines and their DIFF-augmented variants.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	MIT-States	UT-Zappos	C-GQA
Learning rate	$5\times 10^{-5}$	$5\times 10^{-4}$	$5\times 10^{-5}$
Batch size	64	32	16
Gradient accumulation	2	2	4
Epochs	20	20	15
Image alignment weight $\lambda_{\text{img}}$	0.1	0.5	0.1
Text alignment weight $\lambda_{\text{txt}}$	1	0.5	0.8

Appendix

Table 9: Implementation details and hyperparameters for DIFF-CSP across different datasets.

Hyperparameter	MIT-States	UT-Zappos	C-GQA
Learning rate	$1\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-5}$
Batch size	8	16	16
Gradient accumulation	8	4	1
Epochs	10	10	15
Image alignment weight $\lambda_{\text{img}}$	0.7	0.5	0.7
Text alignment weight $\lambda_{\text{txt}}$	0.3	0.5	0.3

Appendix

Table 10: Implementation details and hyperparameters for DIFF-Troika across different datasets.

Hyperparameter	MIT-States	UT-Zappos	C-GQA
Learning rate	$1\times 10^{-4}$	$2.5\times 10^{-4}$	$1\times 10^{-4}$
Batch size	64	64	16
Gradient accumulation	1	1	1
Epochs	15	15	15
Image alignment weight $\lambda_{\text{img}}$	0.5	0.5	0.5
Text alignment weight $\lambda_{\text{txt}}$	0.1	0.1	0.5

Appendix

Table 11: Implementation details and hyperparameters for the DIFF-CAMS across different datasets.

Table 12

Table 13

Method	MIT-States				UT-Zappos
Method	S	U	HM	AUC	S	U	HM	AUC
Troika† (baseline)	49.0 $\pm$ 0.4	53.0 $\pm$ 0.2	39.3 $\pm$ 0.2	22.1 $\pm$ 0.1	66.8 $\pm$ 1.1	73.8 $\pm$ 0.6	54.6 $\pm$ 0.5	41.7 $\pm$ 0.7
DIFF-Troika (ours)	49.9 $\pm$ 0.6	53.2 $\pm$ 0.2	39.6 $\pm$ 0.3	22.6 $\pm$ 0.4	71.0 $\pm$ 1.2	74.4 $\pm$ 0.4	57.7 $\pm$ 0.2	45.4 $\pm$ 0.8

Appendix

Table 16: Performance comparison between Troika and our method on MIT-States and UT-Zappos, averaged over 3 random seeds with standard deviations. † marks results taken from the original publications.

Method	Mean $\uparrow$	Median $\uparrow$	Q25 $\uparrow$	Q75 $\uparrow$	Max $\uparrow$
CLIP	0.1887	0.1818	0.1342	0.2310	0.4561
Diffusion	0.2012	0.1935	0.1552	0.2456	0.5654

Appendix

Table 17: Summary statistics of per-attribute consistency for diffusion and CLIP feature spaces on MIT-States.

Zero-Shot Learning

Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.

See all 184 papers

Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models
Oct 21, 2026Puru Vaish, Meelad Dashti, Christoph Brune +3Compositional Reasoning in VLMs Zero-Shot Learning
University of Twente, The Netherlands · Politecnico di Torino, Italy · University of Amsterdam, The Netherlands
FlowComposer: Composable Flows for Compositional Zero-Shot Learning
Mar 17, 2026Zhenqi He, Lin Li, Long ChenZero-Shot Learning Compositional Representation Learning
The Hong Kong University of Science and Technology
Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning
Jul 7, 2026Ziyi Chen, Haoyan Shi, Sunhan Xu +1Zero-Shot Learning VLM Reasoning
School of Computer Science and Technology, Beijing Jiaotong University · Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education)

Oct 21, 2026J/K move · Enter open · S save

Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

Puru Vaish, Meelad Dashti, Christoph Brune, Pascal Mettes+2

University of Twente, The Netherlands · Politecnico di Torino, Italy · University of Amsterdam, The Netherlands

Compositional Reasoning in VLMs Zero-Shot Learning Compositional Representation Learning Representation Geometry in VLMs

Open paper AskPDF arXiv

Compositional zero-shot learning (CZSL) aims to recognize novel combinations of previously observed objects and attributes. Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization. The Geodesically Decomposable Embeddings (GDE) framework formalizes this by identifying directions in the tangent space at the Frechet mean of the embedding distribution, which encode primitive visual concepts. Unseen attribute-object compositions can be synthesized through vector addition of these primitives. We revisit the assumption that compositional generalization requires explicit attribute-object pair re-composition. By means of new compositional scoring functions and extensive evaluations, we show that directly using the decomposed primitives suffices for recognizing unseen compositions. In that, we gain insights on how the geometry of the embedding manifold (Euclidean, spherical, or hyperbolic) and dataset structure affect compositional reasoning. Across models (CLIP, DINOv2, MERU, HyCoCLIP), we show that compositional structure consistently emerges as primitive representations, enabling inference on CZSL benchmarks (MIT-States, VAW-CZSL, UT-Zappos50K, Clothing16K) and group robustness on Waterbirds and CelebA. Code: https://github.com/nis-research/Compositionality-with-Primitives.

Abstract

Zero-Shot Learning

Explore similar work

Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

FlowComposer: Composable Flows for Compositional Zero-Shot Learning

Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning