WARM-CAT: Warm-Started Test-Time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

Xudong Yan; Songhe Feng; Jiaxin Wang; Xin Su; Yi Jin

cs.CVFeb 26, 2026

WARM-CAT: Warm-Started Test-Time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

Authors: Xudong Yan, Songhe Feng, Jiaxin Wang, Xin Su, Yi Jin

Organizations: School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize novel attribute-object compositions based on the knowledge learned from seen ones. Existing methods suffer from performance degradation caused by the distribution shift of label space at test time, which stems from the inclusion of unseen compositions recombined from attributes and objects. To overcome the challenge, we propose a novel approach that accumulates comprehensive knowledge in both textual and visual modalities from unsupervised data to update multimodal prototypes at test time. Building on this, we further design an adaptive update weight to control the degree of prototype adjustment, enabling the model to flexibly adapt to distribution shift during testing. Moreover, a dynamic priority queue is introduced that stores high-confidence images to acquire visual prototypes from historical images for inference. Since the model tends to favor compositions already stored in the queue during testing, we warm-start the queue by initializing it with training images for visual prototypes of seen compositions and generating unseen visual prototypes using the mapping learned between seen and unseen textual prototypes. Considering the semantic consistency of multimodal knowledge, we align textual and visual prototypes by multimodal collaborative representation learning. To provide a more reliable evaluation for CZSL, we introduce a new benchmark dataset, C-Fashion, and refine the widely used but noisy MIT-States dataset. Extensive experiments indicate that our approach achieves state-of-the-art performance on four benchmark datasets under both closed-world and open-world settings. The source code and datasets are available at https://github.com/xud-yan/WARM-CAT .

Figures & tables

Fig. 1: At test time , existing methods ( top ) fail to adapt using test images, resulting in biased prediction distributions due to label space shift. By contrast, WARM-CAT ( bottom ) progressively accumulates multimodal knowledge from unsupervised test data, enabling effective adaptation to address this challenge.

Fig. 2: Prompt tuning of the text encoder and adapter tuning of the visual encoder during training.

Fig. 3: The overall architecture of our proposed WARM-CAT at test time. The model accumulates multimodal knowledge to update prototypes to overcome the label distribution shift.

Fig. 4: Learning the mapping relationship between seen and unseen textual prototypes ( top ) and applying it to seen visual prototypes to obtain unseen visual prototypes ( bottom ).

Dataset	Composition			Train		Validation			Test
Dataset	$\|A\|$	$\|O\|$	$\|A\times O\|$	$\|C^{s}\|$	$\|\mathcal{X}\|$	$\|C^{s}\|$	$\|C^{u}\|$	$\|\mathcal{X}\|$	$\|C^{s}\|$	$\|C^{u}\|$	$\|\mathcal{X}\|$
UT-Zappos [ 32 ]	16	12	192	83	22998	15	15	3214	18	18	2914
C-Fashion	76	28	2128	350	19018	60	59	5221	60	58	5466
C-GQA [ 12 ]	413	674	278362	5592	26920	1252	1040	7280	888	923	5098
MIT-States ∗	141	266	37506	1074	9851	200	186	2085	201	184	2143

TABLE I: Summary statistics of the four datasets used in our experiments.

Fig. 5: Image annotation generated by MLLM for C-Fashion benchmark dataset.

Methods	UT-Zappos				C-Fashion				C-GQA				MIT-States ∗
Methods	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen
CLIP [ 18 ] ( ICML’21 )	5.0	15.6	15.8	49.1	11.8	25.7	27.0	59.4	1.4	8.6	7.5	25.0	19.6	36.6	35.3	64.2
CoOp [ 22 ] ( IJCV’22 )	18.8	34.6	52.1	49.3	39.1	53.7	67.7	64.6	4.4	17.1	20.5	26.8	24.7	40.5	39.2	72.0
CSP [ 19 ] ( ICLR’23 )	33.0	46.6	64.2	66.2	39.4	54.9	66.0	65.7	6.2	20.5	28.8	26.8	27.6	44.2	44.3	70.3
DFSP [ 20 ] ( CVPR’23 )	36.0	47.2	66.7	71.7	44.0	58.2	69.0	70.2	10.5	27.1	38.2	32.0	31.4	46.7	46.2	75.8
Troika [ 21 ] ( CVPR’24 )	41.7	54.6	66.8	73.8	47.5	59.3	74.3	72.8	12.4	29.4	41.0	35.7	36.3	50.6	49.7	80.2

TABLE II: Closed-world results on UT-Zappos, C-Fashion, C-GQA, and MIT-States ∗ . The best results are displayed in boldface , and the second-best results are underlined . The four indicators are explained in Metrics (Sec. IV-C ).

Methods	UT-Zappos				C-Fashion				C-GQA				MIT-States ∗
Methods	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen
CLIP [ 18 ] ( ICML’21 )	2.2	11.2	15.7	20.6	6.1	18.9	26.9	14.5	0.3	4.0	7.5	4.6	3.4	13.6	35.1	12.4
CoOp [ 22 ] ( IJCV’22 )	13.2	28.9	52.1	31.5	25.4	42.2	67.5	43.0	0.7	5.5	21.0	4.6	3.2	13.0	39.3	9.7
CSP [ 19 ] ( ICLR’23 )	22.7	38.9	64.1	44.1	27.7	44.3	66.0	47.0	1.2	6.9	28.7	5.2	6.2	19.2	44.4	18.5
DFSP [ 20 ] ( CVPR’23 )	30.3	44.0	66.8	60.0	30.7	47.1	69.2	50.1	2.4	10.4	38.3	7.2	9.8	25.5	46.4	26.0
Troika [ 21 ] ( CVPR’24 )	33.0	47.8	66.4	61.2	33.7	48.7	68.4	56.5	2.7	10.9	40.8	7.9	11.6	27.8	49.6	28.4

TABLE III: Open-world results on UT-Zappos, C-Fashion, C-GQA, and MIT-States ∗ . In the open-world setting, a post-training feasibility calibration is employed to eliminate implausible compositions from the expanded search space [ 11 ] .

Module						UT-Zappos				C-Fashion				C-GQA
Priority Queue	Textual KAM	Visual KAM	AUW	PQI for Seen Comp.	PQI for Unseen Comp.	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen
						43.6	55.5	68.7	74.3	49.6	62.3	71.5	74.7	15.4	33.4	44.3	39.7
✓						43.3	55.7	68.6	74.1	49.3	62.1	71.5	74.4	15.4	33.2	44.5	39.7
	✓					45.2	58.1	74.3	68.8	51.4	63.2	79.0	70.6	15.5	33.5	44.5	39.8
✓		✓				43.4	55.7	68.5	74.1	49.3	62.1	71.5	74.4	14.3	31.6	44.9	36.9
✓	✓	✓				46.3	58.5	76.3	68.2	51.1	62.6	78.3	71.6	15.6	33.6	45.6	39.0

TABLE IV: Abaltion study of our proposed modules on UT-Zappos, C-Fashion, and C-GQA. AUW is adaptive update weights. PQI for Seen/Unseen Comp. means priority queue initialization for seen/unseen compositions.

$\mathcal{L}oss$		UT-Zappos				C-Fashion				C-GQA
$\mathcal{L}_{PE}$	$\mathcal{L}_{MCRL}$	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen	AUC	HM	Seen	Unseen
		43.6	55.5	68.7	74.3	49.6	62.3	71.5	74.7	15.4	33.4	44.3	39.7
✓		45.1	58.1	71.7	71.9	50.4	62.9	74.0	73.5	16.2	34.0	45.6	40.4
	✓	36.4	52.1	57.3	70.9	43.8	57.4	68.5	70.5	11.9	28.4	35.0	40.8
✓	✓	52.9	64.3	78.4	73.9	51.5	63.6	76.6	72.5	16.9	35.4	45.6	41.5

TABLE V: Ablation study of our designed loss on UT-Zappos, C-Fashion, and C-GQA.

Test Order	UT-Zappos		C-Fashion		C-GQA
Test Order	AUC	HM	AUC	HM	AUC	HM
Order 1	52.9	64.3	51.5	63.6	16.9	35.4
Order 2	52.0	63.7	50.9	63.6	16.5	35.2
Order 3	51.7	63.3	51.8	63.7	16.4	35.0
Mean	52.2	63.8	51.4	63.6	16.6	35.2
SD	0.6	0.5	0.5	0.1	0.3	0.2

TABLE VI: Influence of test order on UT-Zappos, C-Fashion, and C-GQA. SD denotes standard deviation.

Prediction Probability	UT-Zappos		C-Fashion		C-GQA
Prediction Probability	AUC	HM	AUC	HM	AUC	HM
Text-Only	52.5	64.1	50.9	63.5	16.8	35.3
Visual-Only	12.1	27.1	9.1	25.9	2.4	13.7
Multimodal	52.9	64.3	51.5	63.6	16.9	35.4

TABLE VII: Ablation Study of prediction probability during inference on UT-Zappos, C-Fashion, and C-GQA.

Initialization	UT-Zappos		C-Fashion		C-GQA
Initialization	AUC	HM	AUC	HM	AUC	HM
Uniform Random	43.5	56.1	49.7	62.7	15.6	34.0
Normal Random	51.1	62.3	49.2	62.7	16.3	35.0
Random Walking	51.2	63.3	50.6	63.3	16.9	35.3
All Zeros	52.9	64.3	51.5	63.6	16.9	35.4

TABLE VIII: Influence of initialization strategies of KAMs on UT-Zappos, C-Fashion and C-GQA.

Method	UT-Zappos		C-Fashion		C-GQA
Method	AUC	HM	AUC	HM	AUC	HM
Online-TPS [ 62 ]	44.6	57.3	49.3	61.4	14.8	32.7
TDA [ 58 ]	41.6	54.6	47.7	60.3	14.4	32.0
DPE [ 59 ]	43.3	56.0	49.3	62.1	15.3	33.5
WARM-CAT (Ours)	52.9	64.3	51.5	63.6	16.9	35.4

TABLE IX: Influence of different test-time methods based on base model on UT-Zappos, C-Fashion, and C-GQA.

Table 15

Panel of Figure 16 — Fig. 6: Influence of multimodal balance factor $\alpha$ on UT-Zappos, C-Fashion, and C-GQA.

Panel of Figure 17 — Fig. 7: Influence of the update control factor in AUW $\theta$ on UT-Zappos, C-Fashion, and C-GQA.

Methods	C-Fashion					MIT-States ∗
Methods	Head Acc $\uparrow$	Body Acc $\uparrow$	Tail Acc $\uparrow$	All Acc $\uparrow$	All Std $\downarrow$	Head Acc $\uparrow$	Body Acc $\uparrow$	Tail Acc $\uparrow$	All Acc $\uparrow$	All Std $\downarrow$
CLIP [ 18 ] ( ICML’21 )	24.7	25.5	23.9	24.8	26.1	30.4	36.8	40.1	35.9	35.9
CoOp [ 22 ] ( IJCV’22 )	35.2	27.6	27.4	29.8	31.8	33.4	41.8	42.4	39.5	36.5
CSP [ 19 ] ( ICLR’23 )	34.2	25.3	21.7	26.9	30.2	39.5	38.1	32.8	36.9	34.3
DFSP [ 20 ] ( CVPR’23 )	35.3	27.0	27.9	29.7	31.3	39.5	38.5	37.1	38.4	34.2
Troika [ 21 ] ( CVPR’24 )	36.1	30.0	35.8	33.6	33.1	44.8	43.9	41.7	43.5	34.8

TABLE XII: Long-tailed results on C-Fashion and MIT-States ∗ .

Panel of Figure 19 — Fig. 8: Cumulative classification accuracy with increasing test sample size on three datasets.

Fig. 9: Successful and failure cases on C-Fashion and C-GQA. The predictions of images from WARM-CAT (Ours) with the base model are reported. The successful and failure results are marked in green and red , respectively.

Panel of Figure 21 — Fig. 10: Comparison of multimodal prototype similarity at the test onset and endpoint on C-GQA. All unseen compositions consisting of the attribute cooked and its corresponding objects ( e . g ., noodles , shrimp …) are selected.

Zero-Shot Learning

Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.

See all 184 papers

FlowComposer: Composable Flows for Compositional Zero-Shot Learning
Mar 17, 2026Zhenqi He, Lin Li, Long ChenZero-Shot Learning Compositional Representation Learning
The Hong Kong University of Science and Technology
DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations
Aug 20, 2026Hangyu Tian, Zhenqi He, Yanghao Wang +1Zero-Shot Learning Compositional Representation Learning
The Hong Kong University of Science and Technology
Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning
Jul 7, 2026Ziyi Chen, Haoyan Shi, Sunhan Xu +1Zero-Shot Learning VLM Reasoning
School of Computer Science and Technology, Beijing Jiaotong University · Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education)

Mar 17, 2026J/K move · Enter open · S save

FlowComposer: Composable Flows for Compositional Zero-Shot Learning

Zhenqi He, Lin Li, Long Chen

The Hong Kong University of Science and Technology

Zero-Shot Learning Compositional Representation Learning Flow Matching VLM Adaptation

Open paper AskPDF arXiv

Compositional zero-shot learning (CZSL) aims to recognize unseen attribute-object compositions by recombining primitives learned from seen pairs. Recent CZSL methods built on vision-language models (VLMs) typically adopt parameter-efficient fine-tuning (PEFT). They apply visual disentanglers for decomposition and manipulate token-level prompts or prefixes to encode compositions. However, such PEFT-based designs suffer from two fundamental limitations: (1) Implicit Composition Construction, where composition is realized only via token concatenation or branch-wise prompt tuning rather than an explicit operation in the embedding space; (2) Remained Feature Entanglement, where imperfect disentanglement leaves attribute, object, and composition features mutually contaminated. Together, these issues limit the generalization ability of current CZSL models. In this paper, we are the first to systematically study flow matching for CZSL and introduce FlowComposer, a model-agnostic framework that learns two primitive flows to transport visual features toward attribute and object text embeddings, and a learnable Composer that explicitly fuses their velocity fields into a composition flow. To exploit the inevitable residual entanglement, we further devise a leakage-guided augmentation scheme that reuses leaked features as auxiliary signals. We thoroughly evaluate FlowComposer on three public CZSL benchmarks by integrating it as a plug-and-play component into various baselines, consistently achieving significant improvements.

Abstract

Zero-Shot Learning

Explore similar work

FlowComposer: Composable Flows for Compositional Zero-Shot Learning

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning