Structure-aware Prompt Adaptation from Seen to Unseen for Open-Vocabulary Compositional Zero-Shot Learning

Yihang Duan; Jiong Wang; Pengpeng Zeng; Ji Zhang; Lei Zhao; Chong Wang; Jingkuan Song; Lianli Gao

cs.CVMar 4, 2026

Structure-aware Prompt Adaptation from Seen to Unseen for Open-Vocabulary Compositional Zero-Shot Learning

Authors: Yihang Duan, Jiong Wang, Pengpeng Zeng, Ji Zhang, Lei Zhao, Chong Wang, Jingkuan Song, Lianli Gao

Organizations: University of Electronic Science and Technology of China · Ningbo University · Southwest Jiaotong University · Tongji University

Abstract

The goal of Open-Vocabulary Compositional Zero-Shot Learning (OV-CZSL) is to recognize attribute-object compositions in the open-vocabulary setting, where compositions of both seen and unseen attributes and objects are evaluated. Recently, prompt tuning methods have demonstrated strong generalization capabilities in the closed setting, where only compositions of seen attributes and objects are evaluated, i.e., Compositional Zero-Shot Learning (CZSL). However, directly applying these methods to OV-CZSL may not be sufficient to generalize to unseen attributes, objects and their compositions, as it is limited to seen attributes and objects. Normally, when faced with unseen concepts, humans adopt analogies with seen concepts that have the similar semantics thereby inferring their meaning (e.g., "wet" and "damp", "shirt" and "jacket"). In this paper, we experimentally show that the distribution of semantically related attributes or objects tends to form consistent local structures in the embedding space. Based on the above structures, we propose Structure-aware Prompt Adaptation (SPA) method, which enables models to generalize from seen to unseen attributes and objects. Specifically, in the training stage, we design a Structure-aware Consistency Loss (SCL) that encourages the local structure's consistency of seen attributes and objects in each iteration. In the inference stage, we devise a Structure-guided Adaptation Strategy (SAS) that adaptively aligns the structures of unseen attributes and objects with those of trained seen attributes and objects with similar semantics. Notably, SPA is a plug-and-play method that can be seamlessly integrated into existing CZSL prompt tuning methods. Extensive experiments on OV-CZSL benchmarks demonstrate that SPA achieves competitive closed-set performance while significantly improving open-vocabulary results.

Figures & tables

Fig. 1: Illustration of Compositional Zero-Shot Learning (CZSL) and its extended version Open-Vocabulary Compositional Zero-Shot Learning (OV-CZSL).

Fig. 2: Our core motivation : Semantically related primitives (e.g., ”wet” and ”damp”) form consistent local structures in the embedding space. This structural prior is preserved through training and provides a basis for generalizing from seen to unseen concepts.

Fig. 3: Overview of the proposed Structure-aware Prompt Adaptation (SPA). It comprises two main components: (1) Structure-aware Consistency Loss (SCL) applied during training to preserve the structural distribution of seen primitives in the CLIP embedding space, and (2) Structure-guided Adaptation Strategy dynamically adjusts the representations of unseen primitives by using their semantic proximity to seen ones at test time. SPA operates as a plugin that seamlessly integrates with existing CLIP-based prompt tuning methods.

Methods	Mit-States									C-GQA
Methods	HM $\uparrow$	AUC $\uparrow$	Seen $\uparrow$	Unseen $\uparrow$	$AO$ $\uparrow$	$(AO)^{*}$ $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$	HM $\uparrow$	AUC $\uparrow$	Seen $\uparrow$	Unseen $\uparrow$	$AO$ $\uparrow$	$(AO)^{*}$ $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
Traditional Paradigm (Non-VLM)
LE [ 24 ]	7.64	1.01	16.29	9.46	10.24	11.38	5.98	4.15	2.87	8.39	1.17	19.37	8.36	10.76	6.51	9.53	2.67	1.08
CompCos [ 20 ]	10.22	1.97	26.53	10.29	14.32	21.09	5.86	2.89	0.63	9.64	2.35	40.19	7.25	20.19	20.24	4.47	1.95	0.26
OADis [ 30 ]	9.55	1.83	25.35	10.79	12.18	16.06	6.40	5.41	1.34	9.74	2.33	42.88	7.12	20.86	15.19	6.17	3.47	0.61
SCEN [ 7 ]	9.72	1.73	22.08	8.25	11.85	30.02	3.82	0.33	0.08	9.03	1.97	41.65	7.83	20.65	21.42	3.61	1.08	0.05

TABLE I: Comprehensive results on the MIT-States and C-GQA benchmarks. The top block reports traditional CZSL methods, and the bottom block presents VLM-based prompt tuning approaches with and without SPA. Incorporating SPA consistently improves performance, notably yielding large gains on open-vocabulary splits, such as a +55.1% relative improvement on

A^{*}O^{*}

for C-GQA.

Methods	VAW-CZSL
Methods	HM $\uparrow$	AUC $\uparrow$	$AO$ $\uparrow$	$(AO)^{*}$ $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
Traditional Paradigm (Non-VLM)
LE [ 24 ]	8.27	1.49	15.62	10.48	5.79	2.78	0.98
CompCos [ 20 ]	10.68	2.69	20.21	20.58	5.04	2.48	0.5
OADis [ 30 ]	10.91	2.68	21.19	15.65	6.75	3.16	0.76
SCEN [ 7 ]	10.64	2.53	19.06	20.76	4.52	2.05	0.42

TABLE II: Results on the VAW-CZSL benchmark. Compared with traditional CZSL methods, VLM-based prompting approaches achieve stronger generalization. Incorporating SPA further enhances performance, particularly under fine-grained and open-vocabulary settings, showing consistent gains across most metrics.

Methods	UT-Zappos
Methods	HM $\uparrow$	AUC $\uparrow$	$AO$ $\uparrow$	$(AO)^{*}$ $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
VLM Prompting Paradigm
CSP [ 25 ]	27.33	9.56	16.01	8.58	3.97	7.16	3.33
+SPA	29.01	10.75	15.91	8.86	5.66	9.66	10.57
HPL [ 32 ]	27.83	9.70	16.52	9.10	3.23	6.56	1.98
+SPA	30.13	11.03	15.99	8.99	6.32	7.38	8.67

TABLE III: Comprehensive results on the UT-Zappos benchmarks. Integrating SPA consistently bolsters performance across various baselines, particularly in open-vocabulary scenarios. Such empirical evidence underscores the efficacy of our structural reasoning in capturing nuanced attribute-object relationships.

Neighbors $K$	MIT-States
Neighbors $K$	HM $\uparrow$	AUC $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
$K=0$ (Baseline)	$26.82$	$10.22$	$22.36$	$26.22$	$25.52$
$K=1$	$27.59$	$10.64$	$27.41$	$26.24$	$29.44$
$K=3$	$27.70$	$10.66$	$26.43$	$27.30$	$29.25$
$K=5$	27.80	$10.59$	26.52	27.38	29.44
$K=7$	$27.70$	10.66	$26.43$	$27.30$	$29.25$

TABLE IV: Ablation study on the effect of the number of neighbors (

K

) in SPA. We report key performance metrics on MIT-States, showing that a moderate choice of

K=5

achieves the best trade-off across HM and open-vocabulary metrics.

	Training Overhead		Inference Overhead		Performance Gain
Method	Time (min) $\downarrow$	Memory (MB) $\downarrow$	Speed (ms/img) $\downarrow$	Memory (MB) $\downarrow$	HM $\uparrow$	AO $\uparrow$
Baseline (CSP)	20.33	6388	2.34	4560	26.82	25.52
w SCL	21.38 (+1.05)	6490 (+102)	2.34 (+0.00)	4560 (+0.00)	27.14 (+0.32)	26.15 (+0.63)
w SAS	20.33 (+0.00)	6388 (+0.00)	2.36 (+0.02)	4574 (+0.14)	27.52 (+0.70)	28.05 (+2.53)
SPA (Full)	21.38 (+1.05)	6490 (+102)	2.36 (+0.02)	4574 (+14)	27.80 (+0.98)	29.44 (+3.92)

TABLE V: Component-wise computational cost and performance trade-off analysis of SPA on the MIT-States dataset. The values in parentheses denote the increase in cost or performance over the CSP baseline.

Fig. 4: Qualitative comparison of CSP+SPA against the baseline CSP on samples from MIT-States (top) and C-GQA (bottom). GT denotes Ground Truth. SPA corrects the baseline’s (CSP) prediction errors on challenging open-vocabulary compositions ( $A^{*}O$ , $AO^{*}$ , $A^{*}O^{*}$ ) while maintaining strong performance on seen ones ( $AO$ , $(AO)^{*}$ ). Failure cases for open-vocabulary types are also shown to analyze the method’s boundaries. Predictions marked in red are incorrect; those in green are correct.

Method	MIT-States
Method	HM $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$	Train Memory $\downarrow$
CLIP (Full FT)	$25.09$	$22.36$	$25.73$	$24.95$	$22\,748$
CSP	$26.82$	$22.36$	$26.22$	$25.52$	6388
CSP + NEL	$26.93$	$24.43$	$25.52$	$26.47$	$11\,544$
CSP + SPA	27.80	26.52	27.38	29.44	$6490$

TABLE VI: Comparison of design choices on MIT-States. CSP + SPA outperforms full fine-tuning CLIP, CSP, and CSP + NEL across all metrics while maintaining low training memory.

Setting	Components		MIT-States
Setting	SCL	SAS	HM $\uparrow$	AUC $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
Baseline ( w/o SPA)			26.82	10.22	22.36	26.22	25.52
+SCL	✓		27.14	10.34	23.73	27.33	26.15
+SAS		✓	27.52	10.58	22.94	26.83	28.05
+SCL +SAS (Full SPA)	✓	✓	27.80	10.59	26.52	27.38	29.44

TABLE VII: Ablation study of SPA components on MIT-States [ 10 ] using the CSP [ 25 ] baseline. We incrementally add the Structure-aware Consistency Loss (SCL) and Structure-guided Adaptation Strategy (SAS), showing that each component improves performance, with the full SPA model achieving the best overall results across HM, AUC, and open-vocabulary metrics.

Correlation Measure	Pearson ( $\rho$ )	Spearman ( $r_{s}$ )	P-value
Semantic vs. Visual Distance	0.1548	0.2153	0.0026

TABLE VIII: Quantitative Analysis of Semantic-Visual Alignment.

Neighbor Selection	Avg. Visual Similarity (Mean-Centered)
Random Neighbors	0.0047
Top-5 Semantic Neighbors	0.0642 ( $\uparrow$ 13.7 $\times$ )

TABLE IX: Consistency Check: Visual Similarity of Semantic Neighbors

Fig. 5: Sensitivity analysis of the regularization intensity

\lambda

for Troika on the VAW-CZSL dataset. To ensure transparency and robustness, we report the average performance across five independent runs with different random seeds . As

\lambda

increases, the marginal decrease in Seen performance (

AO

) is significantly outweighed by the sharp gain in Unseen performance (

A^{*}O^{*}

), resulting in the Harmonic Mean (

HM

) reaching its peak at

\lambda=1.0

. This confirms that

\lambda=1.0

is the optimal equilibrium point for balancing empirical fitting and structural manifold preservation.

Isolation Group	Ratio	Avg. Sim	CSP ( $A^{}O^{}$ )	+SPA ( $A^{}O^{}$ )	Gain ( $\Delta$ )
Near	50%	0.78	25.69	30.62	+4.93
Moderate	40%	0.66	25.37	28.63	+3.26
Isolated	10%	0.41	25.27	26.78	+1.51
Total Average	100%	0.70	25.52	29.44	+3.92

TABLE X: Performance Gain vs. Semantic Isolation on MIT-States. Samples are partitioned into three groups based on the semantic similarity to their closest neighbors in the training set. The results reflect the

A^{*}O^{*}

performance for CSP and CSP+SPA.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Datasets	Attributes		Objects		Training Set	Validation Set	Test Set
Datasets	$A$	$A^{*}$	$O$	$O^{*}$	$AO$	$AO$ / $(AO)^{}/A^{}O/AO^{}/A^{}O^{*}$	$AO$ / $(AO)^{}/A^{}O/AO^{}/A^{}O^{*}$
MIT-States [ 10 ]	84	31	182	63	955	236 / 105 / 126 / 177 / 44	289 / 130 / 157 / 218 / 50
C-GQA [ 21 ]	311	102	504	170	4094	1012 / 447 / 525 / 517 / 147	1239 / 542 / 664 / 655 / 176
VAW-CZSL [ 27 ]	330	135	406	110	7142	1767 / 803 / 1420 / 1253 / 412	2161 / 982 / 1737 / 1532 / 504
UT-Zappos [ 36 ]	12	4	9	3	49	8 / 9 / 6 / 8 / 2	8 / 9 / 7 / 8 / 2

Appendix

TABLE S1: Dataset splits for the three OV-CZSL benchmarks. Notation:

A/A^{*}

and

O/O^{*}

denote seen/unseen attributes and objects. The val/test sets are broken down into five compositional types: seen attribute–seen object pairs (

AO

), novel compositions of seen attributes and objects (

(AO)^{*}

), unseen attribute–seen object pairs (

A^{*}O

), seen attribute–unseen object pairs (

AO^{*}

), and fully novel pairs (

A^{*}O^{*}

). This partitioning enables a comprehensive evaluation of generalization to novel compositions and unseen concepts.

Weight $\lambda$	MIT-States
Weight $\lambda$	HM $\uparrow$	AUC $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
$\lambda=0.05$	$27.52$	$10.58$	$23.15$	$26.66$	$28.74$
$\lambda=0.10$	$27.49$	$10.58$	$23.20$	$26.58$	$28.93$
$\lambda=0.50$	$27.52$	$10.58$	$23.03$	$26.69$	$28.68$
$\lambda=1.00$	27.80	10.59	26.52	$27.38$	$29.44$
$\lambda=5.00$	$27.49$	$10.52$	$23.01$	$26.69$	$29.00$

Appendix

TABLE S2: Ablation study of the SCL weight

\lambda

on MIT-States , showing that performance is stable across a wide range of values, with

\lambda=1.0

achieving the best overall performance.

Temperature $\boldsymbol{\tau}$	MIT-States
Temperature $\boldsymbol{\tau}$	HM $\uparrow$	AUC $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
$\tau=0.01$	$27.23$	$10.25$	$26.22$	$26.24$	$28.52$
$\tau=0.05$	$27.38$	$10.42$	$25.55$	$26.69$	$28.50$
$\tau=0.07$	$27.47$	$10.46$	$25.53$	$26.77$	$29.31$
$\tau=0.10$	27.80	10.59	26.52	27.38	29.44
$\tau=0.20$	$27.52$	$10.46$	$23.15$	$26.72$	$28.49$

Appendix

TABLE S3: Ablation study of the temperature

\tau

in SPA on MIT-States , illustrating that performance peaks at

\tau=0.10

, which provides the best HM and open-vocabulary scores.

Neighbors $K$	C-GQA
Neighbors $K$	HM $\uparrow$	AUC $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
$K=0$ (Baseline)	$15.19$	$3.54$	$16.49$	$7.65$	$9.28$
$K=1$	$15.27$	$3.66$	$16.14$	$7.15$	$5.38$
$K=3$	$15.90$	$3.87$	$16.90$	$7.78$	$7.11$
$K=5$	16.70	4.07	18.98	7.12	10.67
$K=7$	$16.65$	$4.03$	$18.45$	$7.34$	$10.44$

Appendix

TABLE S4: Ablation study on the effect of the number of neighbors (

K

) in SPA. We report key performance metrics on C-GQA, showing that a moderate choice of

K=5

achieves the best trade-off across HM and open-vocabulary metrics.

Neighbors $K$	VAW-CZSL
Neighbors $K$	HM $\uparrow$	AUC $\uparrow$	$A^{*}O$ $\uparrow$	$AO^{*}$ $\uparrow$	$A^{}O^{}$ $\uparrow$
$K=0$ (Baseline)	$13.26$	$2.71$	$3.21$	$5.55$	$5.59$
$K=1$	$12.79$	$2.56$	$2.67$	$5.30$	$4.39$
$K=3$	$13.05$	$2.73$	$2.85$	$4.98$	$6.14$
$K=5$	13.92	$2.42$	4.36	6.40	6.70
$K=7$	$13.86$	$2.37$	$4.34$	$6.34$	$6.67$

Appendix

TABLE S5: Ablation study on the effect of the number of neighbors (

K

) in SPA. We report key performance metrics on VAW-CZSL, showing that a moderate choice of

K=5

achieves the best trade-off across HM and open-vocabulary metrics.

Component in SAS	Share of Inference Time (%)
k-NN Search	80.2%
Adaptation & Aggregation	19.8%

Appendix

TABLE S6: Breakdown of the inference time overhead of the SAS module.

Zero-Shot Learning

Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.

See all 184 papers

WARM-CAT: Warm-Started Test-Time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning
Feb 26, 2026Xudong Yan, Songhe Feng, Jiaxin Wang +2Zero-Shot Learning Test-Time Adaptation
School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China
Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning
Jul 7, 2026Ziyi Chen, Haoyan Shi, Sunhan Xu +1Zero-Shot Learning VLM Reasoning
School of Computer Science and Technology, Beijing Jiaotong University · Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education)
DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations
Aug 20, 2026Hangyu Tian, Zhenqi He, Yanghao Wang +1Zero-Shot Learning Compositional Representation Learning
The Hong Kong University of Science and Technology

Feb 26, 2026J/K move · Enter open · S save

WARM-CAT: Warm-Started Test-Time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

Xudong Yan, Songhe Feng, Jiaxin Wang, Xin Su+1

School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China

Zero-Shot Learning Test-Time Adaptation Compositional Representation Learning Multimodal Learning

Open paper AskPDF arXiv

Compositional Zero-Shot Learning (CZSL) aims to recognize novel attribute-object compositions based on the knowledge learned from seen ones. Existing methods suffer from performance degradation caused by the distribution shift of label space at test time, which stems from the inclusion of unseen compositions recombined from attributes and objects. To overcome the challenge, we propose a novel approach that accumulates comprehensive knowledge in both textual and visual modalities from unsupervised data to update multimodal prototypes at test time. Building on this, we further design an adaptive update weight to control the degree of prototype adjustment, enabling the model to flexibly adapt to distribution shift during testing. Moreover, a dynamic priority queue is introduced that stores high-confidence images to acquire visual prototypes from historical images for inference. Since the model tends to favor compositions already stored in the queue during testing, we warm-start the queue by initializing it with training images for visual prototypes of seen compositions and generating unseen visual prototypes using the mapping learned between seen and unseen textual prototypes. Considering the semantic consistency of multimodal knowledge, we align textual and visual prototypes by multimodal collaborative representation learning. To provide a more reliable evaluation for CZSL, we introduce a new benchmark dataset, C-Fashion, and refine the widely used but noisy MIT-States dataset. Extensive experiments indicate that our approach achieves state-of-the-art performance on four benchmark datasets under both closed-world and open-world settings. The source code and datasets are available at https://github.com/xud-yan/WARM-CAT .

Abstract

Zero-Shot Learning

Explore similar work

WARM-CAT: Warm-Started Test-Time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations