Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

Puru Vaish; Meelad Dashti; Christoph Brune; Pascal Mettes; Tatiana Tommasi; Nicola Strisciuglio

cs.CVOct 21, 2026

Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

Authors: Puru Vaish, Meelad Dashti, Christoph Brune, Pascal Mettes, Tatiana Tommasi, Nicola Strisciuglio

Organizations: University of Twente, The Netherlands · Politecnico di Torino, Italy · University of Amsterdam, The Netherlands

Abstract

Compositional zero-shot learning (CZSL) aims to recognize novel combinations of previously observed objects and attributes. Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization. The Geodesically Decomposable Embeddings (GDE) framework formalizes this by identifying directions in the tangent space at the Frechet mean of the embedding distribution, which encode primitive visual concepts. Unseen attribute-object compositions can be synthesized through vector addition of these primitives. We revisit the assumption that compositional generalization requires explicit attribute-object pair re-composition. By means of new compositional scoring functions and extensive evaluations, we show that directly using the decomposed primitives suffices for recognizing unseen compositions. In that, we gain insights on how the geometry of the embedding manifold (Euclidean, spherical, or hyperbolic) and dataset structure affect compositional reasoning. Across models (CLIP, DINOv2, MERU, HyCoCLIP), we show that compositional structure consistently emerges as primitive representations, enabling inference on CZSL benchmarks (MIT-States, VAW-CZSL, UT-Zappos50K, Clothing16K) and group robustness on Waterbirds and CelebA. Code: https://github.com/nis-research/Compositionality-with-Primitives.

Figures & tables

Panel of Figure 1 — Figure 1: Geodesics initiating from the origin across Euclidean, spherical, and hyperbolic manifolds. The plots illustrate the structural differences imposed by space curvature: (a) Euclidean space is flat; (b) spherical space has positive curvature with origin estimated on the Fréchet mean of the dataset embeddings; (c) Hyperbolic space has negative curvature with a well-defined origin, resulting in tree-like hierarchical representation space. Data clusters are differentiated by color and marker type ({ red , blue } $\times$ { square , circle }).

Panel of Figure 2 — Figure 2: Illustration of decomposition and primitive-based methods. For unseen samples of Blue Squares , the goal is to obtain meaningful representations of the concepts Blue and Square from seen training samples. Then, the primitive directions in the tangent space ( $\textbf{v}_{o}$ and $\textbf{v}_{a}$ ) can independently be compared to the test samples projected to the same space. Alternatively, it is possible to calculate the intrinsic mean for each concept label directly on the manifold ( $\mu_{a}$ and $\mu_{o}$ and use it as a reference for the test sample via a manifold-specific inner product to estimate similarity. The pair-based alternatives proposed in previous work either rely on the projection on the manifold of the summed primitives in the tangent space ( $\textbf{v}_{o}+\textbf{v}_{a}$ ), or leverage the textual object-attribute compositional description embedded in the manifold.

		Harmonic Mean (%, $\uparrow$ )					Unseen Pair Accuracy (%, $\uparrow$ )
	Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT-States	CLIP ViT-L/14	8.03	10.56	9.37	10.90	10.40	8.04	13.03	8.62	11.98	11.75
	DINOv2 ViT-L/14	3.00	3.69	6.56	8.22	8.04	2.88	4.78	6.26	9.33	9.42
	CLIP-B (GRIT)	4.61	4.70	4.75	6.64	6.38	4.24	5.36	4.80	7.18	7.12
	MERU-B	4.68	4.74	3.58	6.57	6.36	4.40	5.54	3.40	7.25	7.40
	HyCoCLIP-B	5.26	6.53	4.28	7.94	7.54	4.67	8.20	3.71	8.75	8.76

Table 1: Harmonic Mean (HM) and Unseen Pair Accuracy (UPA) results across models and datasets. Primitive-based inference consistently improves compositional generalization over GDE pair composition, particularly on unseen attribute–object pairs.

	Harmonic Mean (%, $\uparrow$ ) / After Whitening
Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
CLIP ViT-L/14	11.76 / –	9.61 / –	16.41 / 16.69	8.04 / 13.75	9.33 / 22.80
DINOv2 ViT-L/14	2.81 / –	0.38 / –	12.06 / 13.64	3.20 / 12.40	2.49 / 16.17
CLIP-B (GRIT)	2.33 / –	1.61 / –	10.81 / 13.53	9.04 / 12.79	8.95 / 14.65
MERU-B	3.38 / –	3.80 / –	10.39 / 12.79	8.99 / 8.80	9.36 / 16.32
HyCoCLIP-B	4.34 / –	3.94 / –	8.50 / 13.01	5.89 / 8.14	5.86 / 15.11

Table 2: CZSL metrics before and after whitening across models and inference methods for the low-intrinsic variance UT-Zappos dataset . Whitening consistently improves primitive-based inference and restores its advantage over pair composition.

Panel of Figure 3 — Figure 3: Intrinsic dataset variance measured using squared geodesic distance to the dataset Fréchet mean across different geometries. From left to right: DINOv2 ViT-L/14 with Euclidean geometry, CLIP ViT-L/14 with spherical geometry, and HyCoCLIP-B with hyperbolic geometry. UT-Zappos exhibits substantially lower variance than the broader datasets, while geometry-specific whitening increases the variance of UTZ-W toward the same regime.

		Zero Shot			GDE-Primitive			Intrinsic Primitive
	Model	Avg. Acc. $\uparrow$	WG. Acc. $\uparrow$	Gap $\downarrow$	Avg. Acc. $\uparrow$	WG. Acc. $\uparrow$	Gap $\downarrow$	Avg. Acc. $\uparrow$	WG. Acc. $\uparrow$	Gap $\downarrow$
Waterbirds	CLIP ViT-L/14	80.9	48.8	32.1	90.8	85.7	5.13	90.8	85.7	5.17
	DINOv2 ViT-L/14	69.1	42.4	26.7	85.5	48.0	37.5	85.0	49.4	35.6
	CLIP-B (GRIT)	58.4	36.9	21.5	77.1	64.4	12.7	77.0	64.4	12.7
	MERU-B	58.3	31.7	26.6	63.0	39.0	24.0	77.1	62.6	14.6
	HyCoCLIP-B	53.9	20.3	33.7	62.6	38.3	24.3	77.1	62.6	14.5

Table 3: Group-robustness results (%). Average Accuracy (Avg. Acc.), Worst-Group Accuracy (WG. Acc.), and their gap on Waterbirds and CelebA. GDE and intrinsic object primitives consistently improve over zero-shot object inference without retraining or group annotations, with especially large gains for the hyperbolic models MERU and HyCoCLIP.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

		Harmonic Mean (%, $\uparrow$ )			Unseen Pair Accuracy (%, $\uparrow$ )
Dataset	$\mathcal{M}$	GDE Pair	GDE Prim.	Int. Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT	$\mathrm{euc}$	6.56	8.22	8.04	6.26	9.33	9.42
MIT	$\mathrm{sph}$	2.17	4.70	4.85	1.87	4.61	4.75
VAW	$\mathrm{euc}$	1.30	2.05	2.20	1.63	2.60	2.81
VAW	$\mathrm{sph}$	0.28	0.68	0.72	0.28	0.65	0.71

Appendix

Table 4: Evaluation results utilizing the Talk2DINO as Euclidean (

\mathrm{euc}

) or Spherical (

\mathrm{sph}

) embeddings (after

\ell_{2}

normalisation). The native euclidean manifold substantially outperforms spherical.

		CLIP ViT-L/14		DINOv2 ViT-L/14		CLIP-B (GRIT)		HyCoCLIP-B		MERU-B
		CORR	MI	CORR	MI	CORR	MI	CORR	MI	CORR	MI
MIT-States	object-attribute	0.453	13.0	0.456	13.0	0.444	13.0	0.453	13.0	0.473	13.0
	object-pair	0.825	191.0	0.895	320.7	0.831	148.7	0.839	119.7	0.843	119.4
	attribute-pair	0.530	77.9	0.516	129.0	0.511	44.5	0.513	44.2	0.531	44.3
VAW-CZSL	object-attribute	0.187	7.2	0.183	7.2	0.195	6.9	0.191	6.9	0.173	6.9
	object-pair	0.575	38.1	0.713	61.0	0.565	28.1	0.565	26.6	0.576	29.9

Appendix

Table 5: Cross Correlation and Mutual Information estimates between the GDE primitives and the model embedding. Since GDE primtives live in euclidean space, usual statistics calculation methods apply.

		Harmonic Mean (%, $\uparrow$ )		Unseen Pair Accuracy (%, $\uparrow$ )
Dataset	Model	Weighted	Uniform	Weighted	Uniform
MIT	CLIP ViT-L/14	10.90	10.39	11.98	11.47
	DINOv2 ViT-L/14	8.22	8.02	9.33	9.10
	CLIP-B (GRIT)	6.64	6.12	7.18	6.62
	MERU-B	6.57	6.59	7.25	7.14
	HyCoCLIP-B	7.94	7.83	8.75	8.62

Appendix

Table 6: Disabling noise estimation results in minor performance variations across models for GDE primitives validating that weighing carries no directional information.

		Object Accuracy (%, $\uparrow$ )					Attribute Accuracy (%, $\uparrow$ )
	Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT-States	CLIP ViT-L/14	47.33	47.33	43.79	43.68	43.94	15.76	20.05	18.38	20.57	20.68
	DINOv2 ViT-L/14	32.70	29.70	40.13	40.25	40.22	8.49	11.97	16.00	18.45	18.41
	CLIP-B (GRIT)	36.59	35.81	33.91	35.50	35.58	10.35	13.48	14.09	17.41	16.97
	MERU-B	37.31	36.34	34.03	35.39	35.51	10.57	13.22	11.71	17.48	17.07
	HyCoCLIP-B	39.79	39.72	37.41	39.01	39.15	10.95	16.26	12.00	18.09	17.71

Appendix

Table 7: Open-world Compositional Zero-Shot Learning results (%,

\uparrow

). Metrics include Object Accuracy and Attribute Accuracy.

		Seen Pair Accuracy (%, $\uparrow$ )					Overall Pair Accuracy (%, $\uparrow$ )
	Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT-States	CLIP ViT-L/14	29.03	22.23	31.18	25.21	25.88	15.76	20.05	18.38	20.57	20.68
	DINOv2 ViT-L/14	16.64	9.87	27.23	20.63	19.96	8.49	11.97	16.00	18.45	18.41
	CLIP-B (GRIT)	21.72	16.93	22.18	18.99	18.82	10.35	13.48	14.09	17.41	16.97
	MERU-B	22.61	16.76	19.58	18.91	19.20	10.57	13.22	11.71	17.48	17.07
	HyCoCLIP-B	23.87	15.50	21.97	21.34	20.97	10.95	16.26	12.00	18.09	17.71

Appendix

Table 8: Open-world Compositional Zero-Shot Learning results (%,

\uparrow

). Metrics include Seen Pair Accuracy and Overall Pair Accuracy.

Panel of Figure 4 — Figure 4: Radius distributions for HyCoCLIP-B showing attribute primitives, object primitives, and attribute-object compositions in hyperbolic space. Dashed lines indicate mean radii. We find, objects intrinsic primitives are consistently occupy the higher radius stratum compared to attributes, except in the case of CelebA and Waterbirds. The pair embeddings themselves are significantly further away, which is consistent with the hierarchical entailment objective of these models.

Panel of Figure 5 — Figure 5: Radius distributions for MERU showing attribute primitives, object primitives, and attribute-object compositions in hyperbolic space. Dashed lines indicate mean radii. Similar to HycoCLIP, we find, objects intrinsic primitives are consistently occupy the higher radius stratum compared to attributes, except in the case of CelebA and Waterbirds. The pair embeddings themselves are significantly further away, which is consistent with the hierarchical entailment objective of these models.

Panel of Figure 6 — Figure 6: Random compositional examples across datasets.

Compositional Reasoning in VLMs

See all 29 papers

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations
Aug 20, 2026Hangyu Tian, Zhenqi He, Yanghao Wang +1Zero-Shot Learning Compositional Representation Learning
The Hong Kong University of Science and Technology
Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
Feb 27, 2026Arnas Uselis, Andrea Dittadi, Seong Joon OhNeural Representation Geometry Visual Representation Learning
Tübingen AI Center, University of Tübingen · Helmholtz · Technical University of Munich
From Model Patterns to Abstract Semantics in Compositional Zero-Shot Learning
Sep 14, 2026Weize Li, Zhicheng Zhao, Fei SuZero-Shot Learning VLM Reasoning
Beijing University of Posts and Telecommunications · Beijing Key Laboratory of Network System and Network Culture · Key Laboratory of Interactive Technology and Experience System

Aug 20, 2026J/K move · Enter open · S save

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

Hangyu Tian, Zhenqi He, Yanghao Wang, Long Chen

The Hong Kong University of Science and Technology

Zero-Shot Learning Compositional Representation Learning

Open paper AskPDF arXiv

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.

Abstract

Compositional Reasoning in VLMs

Explore similar work

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

From Model Patterns to Abstract Semantics in Compositional Zero-Shot Learning