Compositional Zero-Shot Learning (CZSL) aims to recognize novel attribute-object compositions based on the knowledge learned from seen ones. Existing methods suffer from performance degradation caused by the distribution shift of label space at test time, which stems from the inclusion of unseen compositions recombined from attributes and objects. To overcome the challenge, we propose a novel approach that accumulates comprehensive knowledge in both textual and visual modalities from unsupervised data to update multimodal prototypes at test time. Building on this, we further design an adaptive update weight to control the degree of prototype adjustment, enabling the model to flexibly adapt to distribution shift during testing. Moreover, a dynamic priority queue is introduced that stores high-confidence images to acquire visual prototypes from historical images for inference. Since the model tends to favor compositions already stored in the queue during testing, we warm-start the queue by initializing it with training images for visual prototypes of seen compositions and generating unseen visual prototypes using the mapping learned between seen and unseen textual prototypes. Considering the semantic consistency of multimodal knowledge, we align textual and visual prototypes by multimodal collaborative representation learning. To provide a more reliable evaluation for CZSL, we introduce a new benchmark dataset, C-Fashion, and refine the widely used but noisy MIT-States dataset. Extensive experiments indicate that our approach achieves state-of-the-art performance on four benchmark datasets under both closed-world and open-world settings. The source code and datasets are available at https://github.com/xud-yan/WARM-CAT .
Figures & tables
Fig. 1: At test time , existing methods ( top ) fail to adapt using test images, resulting in biased prediction distributions due to label space shift. By contrast, WARM-CAT ( bottom ) progressively accumulates multimodal knowledge from unsupervised test data, enabling effective adaptation to address this challenge.
Fig. 2: Prompt tuning of the text encoder and adapter tuning of the visual encoder during training.
Fig. 3: The overall architecture of our proposed WARM-CAT at test time. The model accumulates multimodal knowledge to update prototypes to overcome the label distribution shift.
Fig. 4: Learning the mapping relationship between seen and unseen textual prototypes ( top ) and applying it to seen visual prototypes to obtain unseen visual prototypes ( bottom ).
Dataset
Composition
Train
Validation
Test
∣A∣
∣O∣
∣A×O∣
∣Cs∣
∣X∣
∣Cs∣
∣Cu∣
∣X∣
∣Cs∣
∣Cu∣
∣X∣
UT-Zappos [ 32 ]
16
12
192
83
22998
15
15
3214
18
18
2914
C-Fashion
76
28
2128
350
19018
60
59
5221
60
58
5466
C-GQA [ 12 ]
413
674
278362
5592
26920
1252
1040
7280
888
923
5098
MIT-States ∗
141
266
37506
1074
9851
200
186
2085
201
184
2143
TABLE I: Summary statistics of the four datasets used in our experiments.
Fig. 5: Image annotation generated by MLLM for C-Fashion benchmark dataset.
Methods
UT-Zappos
C-Fashion
C-GQA
MIT-States ∗
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
CLIP [ 18 ] ( ICML’21 )
5.0
15.6
15.8
49.1
11.8
25.7
27.0
59.4
1.4
8.6
7.5
25.0
19.6
36.6
35.3
64.2
CoOp [ 22 ] ( IJCV’22 )
18.8
34.6
52.1
49.3
39.1
53.7
67.7
64.6
4.4
17.1
20.5
26.8
24.7
40.5
39.2
72.0
CSP [ 19 ] ( ICLR’23 )
33.0
46.6
64.2
66.2
39.4
54.9
66.0
65.7
6.2
20.5
28.8
26.8
27.6
44.2
44.3
70.3
DFSP [ 20 ] ( CVPR’23 )
36.0
47.2
66.7
71.7
44.0
58.2
69.0
70.2
10.5
27.1
38.2
32.0
31.4
46.7
46.2
75.8
Troika [ 21 ] ( CVPR’24 )
41.7
54.6
66.8
73.8
47.5
59.3
74.3
72.8
12.4
29.4
41.0
35.7
36.3
50.6
49.7
80.2
TABLE II: Closed-world results on UT-Zappos, C-Fashion, C-GQA, and MIT-States ∗ . The best results are displayed in boldface , and the second-best results are underlined . The four indicators are explained in Metrics (Sec. IV-C ).
Methods
UT-Zappos
C-Fashion
C-GQA
MIT-States ∗
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
CLIP [ 18 ] ( ICML’21 )
2.2
11.2
15.7
20.6
6.1
18.9
26.9
14.5
0.3
4.0
7.5
4.6
3.4
13.6
35.1
12.4
CoOp [ 22 ] ( IJCV’22 )
13.2
28.9
52.1
31.5
25.4
42.2
67.5
43.0
0.7
5.5
21.0
4.6
3.2
13.0
39.3
9.7
CSP [ 19 ] ( ICLR’23 )
22.7
38.9
64.1
44.1
27.7
44.3
66.0
47.0
1.2
6.9
28.7
5.2
6.2
19.2
44.4
18.5
DFSP [ 20 ] ( CVPR’23 )
30.3
44.0
66.8
60.0
30.7
47.1
69.2
50.1
2.4
10.4
38.3
7.2
9.8
25.5
46.4
26.0
Troika [ 21 ] ( CVPR’24 )
33.0
47.8
66.4
61.2
33.7
48.7
68.4
56.5
2.7
10.9
40.8
7.9
11.6
27.8
49.6
28.4
TABLE III: Open-world results on UT-Zappos, C-Fashion, C-GQA, and MIT-States ∗ . In the open-world setting, a post-training feasibility calibration is employed to eliminate implausible compositions from the expanded search space [ 11 ] .
Module
UT-Zappos
C-Fashion
C-GQA
Priority Queue
Textual KAM
Visual KAM
AUW
PQI for Seen Comp.
PQI for Unseen Comp.
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
43.6
55.5
68.7
74.3
49.6
62.3
71.5
74.7
15.4
33.4
44.3
39.7
✓
43.3
55.7
68.6
74.1
49.3
62.1
71.5
74.4
15.4
33.2
44.5
39.7
✓
45.2
58.1
74.3
68.8
51.4
63.2
79.0
70.6
15.5
33.5
44.5
39.8
✓
✓
43.4
55.7
68.5
74.1
49.3
62.1
71.5
74.4
14.3
31.6
44.9
36.9
✓
✓
✓
46.3
58.5
76.3
68.2
51.1
62.6
78.3
71.6
15.6
33.6
45.6
39.0
TABLE IV: Abaltion study of our proposed modules on UT-Zappos, C-Fashion, and C-GQA. AUW is adaptive update weights. PQI for Seen/Unseen Comp. means priority queue initialization for seen/unseen compositions.
Loss
UT-Zappos
C-Fashion
C-GQA
LPE
LMCRL
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
AUC
HM
Seen
Unseen
43.6
55.5
68.7
74.3
49.6
62.3
71.5
74.7
15.4
33.4
44.3
39.7
✓
45.1
58.1
71.7
71.9
50.4
62.9
74.0
73.5
16.2
34.0
45.6
40.4
✓
36.4
52.1
57.3
70.9
43.8
57.4
68.5
70.5
11.9
28.4
35.0
40.8
✓
✓
52.9
64.3
78.4
73.9
51.5
63.6
76.6
72.5
16.9
35.4
45.6
41.5
TABLE V: Ablation study of our designed loss on UT-Zappos, C-Fashion, and C-GQA.
Test Order
UT-Zappos
C-Fashion
C-GQA
AUC
HM
AUC
HM
AUC
HM
Order 1
52.9
64.3
51.5
63.6
16.9
35.4
Order 2
52.0
63.7
50.9
63.6
16.5
35.2
Order 3
51.7
63.3
51.8
63.7
16.4
35.0
Mean
52.2
63.8
51.4
63.6
16.6
35.2
SD
0.6
0.5
0.5
0.1
0.3
0.2
TABLE VI: Influence of test order on UT-Zappos, C-Fashion, and C-GQA. SD denotes standard deviation.
Prediction Probability
UT-Zappos
C-Fashion
C-GQA
AUC
HM
AUC
HM
AUC
HM
Text-Only
52.5
64.1
50.9
63.5
16.8
35.3
Visual-Only
12.1
27.1
9.1
25.9
2.4
13.7
Multimodal
52.9
64.3
51.5
63.6
16.9
35.4
TABLE VII: Ablation Study of prediction probability during inference on UT-Zappos, C-Fashion, and C-GQA.
Initialization
UT-Zappos
C-Fashion
C-GQA
AUC
HM
AUC
HM
AUC
HM
Uniform Random
43.5
56.1
49.7
62.7
15.6
34.0
Normal Random
51.1
62.3
49.2
62.7
16.3
35.0
Random Walking
51.2
63.3
50.6
63.3
16.9
35.3
All Zeros
52.9
64.3
51.5
63.6
16.9
35.4
TABLE VIII: Influence of initialization strategies of KAMs on UT-Zappos, C-Fashion and C-GQA.
Method
UT-Zappos
C-Fashion
C-GQA
AUC
HM
AUC
HM
AUC
HM
Online-TPS [ 62 ]
44.6
57.3
49.3
61.4
14.8
32.7
TDA [ 58 ]
41.6
54.6
47.7
60.3
14.4
32.0
DPE [ 59 ]
43.3
56.0
49.3
62.1
15.3
33.5
WARM-CAT (Ours)
52.9
64.3
51.5
63.6
16.9
35.4
TABLE IX: Influence of different test-time methods based on base model on UT-Zappos, C-Fashion, and C-GQA.
Table 15
Fig. 6: Influence of multimodal balance factor α on UT-Zappos, C-Fashion, and C-GQA.
Fig. 7: Influence of the update control factor in AUW θ on UT-Zappos, C-Fashion, and C-GQA.
Methods
C-Fashion
MIT-States ∗
Head Acc ↑
Body Acc ↑
Tail Acc ↑
All Acc ↑
All Std ↓
Head Acc ↑
Body Acc ↑
Tail Acc ↑
All Acc ↑
All Std ↓
CLIP [ 18 ] ( ICML’21 )
24.7
25.5
23.9
24.8
26.1
30.4
36.8
40.1
35.9
35.9
CoOp [ 22 ] ( IJCV’22 )
35.2
27.6
27.4
29.8
31.8
33.4
41.8
42.4
39.5
36.5
CSP [ 19 ] ( ICLR’23 )
34.2
25.3
21.7
26.9
30.2
39.5
38.1
32.8
36.9
34.3
DFSP [ 20 ] ( CVPR’23 )
35.3
27.0
27.9
29.7
31.3
39.5
38.5
37.1
38.4
34.2
Troika [ 21 ] ( CVPR’24 )
36.1
30.0
35.8
33.6
33.1
44.8
43.9
41.7
43.5
34.8
TABLE XII: Long-tailed results on C-Fashion and MIT-States ∗ .
Fig. 8: Cumulative classification accuracy with increasing test sample size on three datasets.
Fig. 9: Successful and failure cases on C-Fashion and C-GQA. The predictions of images from WARM-CAT (Ours) with the base model are reported. The successful and failure results are marked in green and red , respectively.
Fig. 10: Comparison of multimodal prototype similarity at the test onset and endpoint on C-GQA. All unseen compositions consisting of the attribute cooked and its corresponding objects ( e . g ., noodles , shrimp …) are selected.
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
Compositional zero-shot learning (CZSL) aims to recognize unseen attribute-object compositions by recombining primitives learned from seen pairs. Recent CZSL methods built on vision-language models (VLMs) typically adopt parameter-efficient fine-tuning (PEFT). They apply visual disentanglers for decomposition and manipulate token-level prompts or prefixes to encode compositions. However, such PEFT-based designs suffer from two fundamental limitations: (1) Implicit Composition Construction, where composition is realized only via token concatenation or branch-wise prompt tuning rather than an explicit operation in the embedding space; (2) Remained Feature Entanglement, where imperfect disentanglement leaves attribute, object, and composition features mutually contaminated. Together, these issues limit the generalization ability of current CZSL models. In this paper, we are the first to systematically study flow matching for CZSL and introduce FlowComposer, a model-agnostic framework that learns two primitive flows to transport visual features toward attribute and object text embeddings, and a learnable Composer that explicitly fuses their velocity fields into a composition flow. To exploit the inevitable residual entanglement, we further devise a leakage-guided augmentation scheme that reuses leaked features as auxiliary signals. We thoroughly evaluate FlowComposer on three public CZSL benchmarks by integrating it as a plug-and-play component into various baselines, consistently achieving significant improvements.
Zhenqi He, Lin Li, Long Chen
The Hong Kong University of Science and Technology
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.
Hangyu Tian, Zhenqi He, Yanghao Wang +1
The Hong Kong University of Science and Technology
Compositional Zero-Shot Learning (CZSL) aims to combine known attributes and objects as primitives for recognizing previously unseen attribute-object pairs. Prior works either predict attributes and objects independently, missing their strong contextual dependency, or use unidirectional conditional modeling (e.g., object-guided attribute prediction), which is prone to error propagation. We propose PRPC, a Progressive Reasoning framework with Primitive Correction, which explicitly models the bidirectional dependency between attributes and objects via step-wise inference. PRPC performs mutual correction of primitives to suppress prediction errors in earlier steps. Specifically, we formulate CZSL as structured, Q&A-style Chain-of-Thought reasoning process and constrain the MLLM to follow predefined semantic steps to generate intermediate decisions. To further enhance the reliability and logical consistency of intermediate reasoning, we introduce reinforcement learning post-training with a GRPO-based objective, providing step-level rewards aligned with the progressive inference procedure. Extensive experiments on three CZSL benchmarks demonstrate that PRPC achieves state-of-the-art performance, validating the effectiveness of progressive reasoning and bidirectional correction for robust compositional generalization.
Ziyi Chen, Haoyan Shi, Sunhan Xu +1
School of Computer Science and Technology, Beijing Jiaotong University · Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education)