ARGENT: Adaptive Hierarchical Image-Text Representations

Chuong Huynh; Hossein Souri; Abhinav Kumar; Vitali Petsiuk; Deen Dayal Mohan; Suren Kumar

cs.CVMar 24, 2026

ARGENT: Adaptive Hierarchical Image-Text Representations

Authors: Chuong Huynh, Hossein Souri, Abhinav Kumar, Vitali Petsiuk, Deen Dayal Mohan, Suren Kumar

Organizations: University of Maryland, College Park · Samsung Research America, AI Center – Mountain View

Abstract

Large-scale Vision-Language Models (VLMs) such as CLIP learn powerful semantic representations but operate in Euclidean space, which fails to capture the inherent hierarchical structure of visual and linguistic concepts. Hyperbolic geometry, with its exponential volume growth, offers a principled alternative for embedding such hierarchies with low distortion. However, existing hyperbolic VLMs use entailment losses that are unstable: as parent embeddings contract toward the origin, their entailment cones widen toward a half-space, causing catastrophic cone collapse that destroys the intended hierarchy. Additionally, hierarchical evaluation of these models remains unreliable, being largely retrieval-based and correlation-based metrics and prone to taxonomy dependence and ambiguous negatives. To address these limitations, we propose an adaptive entailment loss paired with a norm regularizer that prevents cone collapse without heuristic aperture clipping. We further introduce an angle-based probabilistic entailment protocol (PEP) for evaluating hierarchical understanding, scored with AUC-ROC and Average Precision. This paper introduces a stronger hyperbolic VLM baseline ARGENT, Adaptive hieRarchical imaGe-tExt represeNTation. ARGENT improves the SOTA hyperbolic VLM by 0.7, 1.1, and 0.8 absolute points on image classification, text-to-image retrieval, and proposed hierarchical metrics, respectively.

Figures & tables

Panel of Figure 1 — Figure 1: ARGENT improves both hierarchical training and evaluation. (a) ARGENT outperforms the HyCoCLIP baseline on downstream tasks and our hierarchical benchmark (PEP AUC). (b) Our new Probabilistic Entailment Score ( $p_{\text{Ent}}$ ) offers a more discriminative evaluation; while both models may achieve 100% correlation, our metric correctly identifies $VLM_{A}$ as superior.

Figure 2: Behavior of Adaptive Entailment $\mathcal{L}_{\text{AdaEnt}}$ and standard Entailment Loss $\mathcal{L}_{\text{Ent}}$ . The figure highlights two cases in a top-down view of hyperbolic space: (1) Inside the norm boundary ( $\|\tilde{\mathbf{y}}\|\leq\frac{2C}{\sqrt{\kappa}}$ ): The standard $\mathcal{L}_{\text{Ent}}$ collapses to zero for all $\mathbf{x}$ in the non-origin half-space, even when the exterior angle $\phi(\mathbf{x},\mathbf{y})$ is large. Our $\mathcal{L}_{\text{AdaEnt}}$ remains active ( $\mathcal{L}_{\text{AdaEnt}}\gg 0$ ), preventing vanishing gradients. (2) Outside the norm boundary : $\mathcal{L}_{\text{Ent}}$ penalizes the likely noisy positive $\mathbf{x}_{2}$ and the true negative $\mathbf{x}_{1}$ with the same value. Our $\mathcal{L}_{\text{AdaEnt}}$ adaptively assigns a lower loss to $\mathbf{x}_{2}$ while strongly penalizing $\mathbf{x}_{1}$ .

Panel of Figure 3 — Figure 3: Analysis of the vanilla and proposed adaptive entailment loss . (a) The norm constraint required by the standard loss. (b) A comparison between loss functions. (c) The effect of our adaptive weight. (d) The behavior of our norm regularizer.

Figure 4: Weighing factor

h(\mathbf{x},\mathbf{y})

(scaling) w.r.t. the distance (negative of similarity) for intra and inter-modality samples in MERU, HyCoCLIP and ARGENT. ARGENT uses adaptive weights compared to constant weights in MERU and HyCoCLIP.

Figure 5: Recall@1 of the BLIP-L model in image-to-text retrieval task on HierarCaps. Level 1 and 4 denotes the most generic and the most specific captions. We restrict the candidate pool to a local set containing ground-truth captions and the top-5 most similar (presumed false negative) captions. The model performance degrades when the text is more generic (Level 1).

Remove top-k	Precision (%) $\relbar\joinrel\mathrel{\RHD}$	Recall (%) $\relbar\joinrel\mathrel{\RHD}$
None	15.36	43.23
k=5	17.10 (+1.74)	46.45 (+3.22)
k=10	18.05 (+2.69)	48.10 (+4.87)

Table 1: HierarCaps image-to-text retrieval performance of

\text{CLIP }^{\text{L}}_{\text{FT}}

increases as false negative captions are progressively removed from the candidate pool. This shows that the current evaluation protocol is flawed.

Model	Cls (Acc $\relbar\joinrel\mathrel{\RHD}$ )		T2I Ret (R@k $\relbar\joinrel\mathrel{\RHD}$ )				I2T Ret (R@k $\relbar\joinrel\mathrel{\RHD}$ )				Hierarchical Metrics
	INet	Avg.	COCO		Flickr		COCO		Flickr		WordNet			PEP
	INet	Avg.	k=5	k=10	k=5	k=10	k=5	k=10	k=5	k=10	TIE $\relbar\joinrel\mathrel{\RHD}$	LCA $\relbar\joinrel\mathrel{\RHD}$	Jac $\relbar\joinrel\mathrel{\RHD}$	AUC $\relbar\joinrel\mathrel{\RHD}$	AP $\relbar\joinrel\mathrel{\RHD}$
CLIP-S [ 30 ]	32.8	35.5	51.4	63.0	78.3	85.7	66.0	76.5	88.5	93.3	4.39	2.50	73.6	–	–
MERU-S [ 9 ]	32.6	34.3	51.2	63.3	77.7	86.3	65.8	76.7	88.2	94.3	4.35	2.48	73.7	58.4	19.9
HyCoCLIP-S [ 26 ]	37.3	39.2	52.0	63.5	78.5	85.9	65.2	75.9	88.7	92.8	3.96	2.33	76.2	96.8	87.7
ARGENT-S	38.8	38.4	53.2	64.7	79.9	87.5	66.9	77.8	90.4	95.6	3.76	2.26	77.7	99.3	91.2

Table 2: Main Results. ARGENT outperforms baselines on almost all metrics , often by a large margin. The adaptive approach not only improves performance on downstream tasks (classification and retrieval) but also reduces hierarchical errors. Results of all 16 image classification datasets are in supplementary material. [Keys: Best , Gain and drop relative to HyCoCLIP baseline [ 26 ] , Cls= Classifcation, T2I= Text-to-Image, I2T= Image-to-Text, Ret= Retrieval]

Panel of Figure 6 — Figure 6: HoroPCA and norm distribution of HyCoCLIP ( left ) and ARGENT ( right ) on HierarCaps and CC3M. ARGENT increases the variance of embedding norms and reduces the overlap between caption and phrase distributions.

Model	Module	Cls.	Ret.	WordNet		PEP
		Acc. $\relbar\joinrel\mathrel{\RHD}$	R@5 $\relbar\joinrel\mathrel{\RHD}$	Err. $\relbar\joinrel\mathrel{\RHD}$	Jacc. $\relbar\joinrel\mathrel{\RHD}$	AUC $\relbar\joinrel\mathrel{\RHD}$	AP $\relbar\joinrel\mathrel{\RHD}$
(a) Adaptive Entailment loss with Different Backbones
MERU-S	Ent	20.8	52.6	5.07	59.2	73.5	45.0
	AdaEnt	20.6	53.4	4.98	59.4	95.5	60.9
HyCoCLIP-S	Ent	22.9	54.8	4.73	62.2	94.6	76.2
	AdaEnt	22.8	55.5	4.60	62.8	97.3	76.9

Table 3: Ablation studies of proposed components. Our proposed

\mathcal{L}_{\text{AdaEnt}}

outperforms standard entailment loss in (a). The adaptive wight is the most efficient when applying on intra-modality pairs in (b). Regularization loss balances the performance of the model on tasks in (d). [Key: Best in each group].

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	General						Fine-grained						Misc				Avg
Model	ImageNet	CIFAR-10	CIFAR-100	SUN397	Caltech-101	STL-10	Food-101	CUB	Cars	Aircraft	Pets	Flowers	DTD	EuroSAT	RESISC45	Country211
CLIP-S [ 30 ]	$32.8$	$71.6$	$39.8$	$45.9$	$69.3$	$90.5$	$44.7$	$10.2$	$6.4$	$2.1$	$38.8$	$14.6$	$19.6$	$37.7$	$38.8$	4.8	$35.5$
MERU-S [ 9 ]	$32.6$	$72.5$	$42.0$	$46.3$	$69.3$	$89.9$	$35.1$	$7.3$	$6.5$	$2.3$	$42.3$	$12.8$	$18.0$	$30.2$	$37.0$	$4.1$	$34.3$
HyCoCLIP-S [ 26 ]	$37.3$	$83.4$	$49.2$	$48.6$	$71.2$	$91.6$	$47.0$	12.6	$8.0$	2.8	47.6	19.4	22.8	42.4	$39.1$	$4.7$	39.2
ARGENT-S	38.8	83.6	50.5	49.2	73.7	91.9	48.0	$10.3$	8.5	$2.6$	$43.8$	$10.9$	$21.6$	$37.0$	39.4	$4.7$	$38.4$
CLIP-B [ 30 ]	$36.9$	$71.7$	$46.4$	$49.1$	$74.6$	$92.6$	$46.2$	$10.7$	$8.1$	$2.0$	$41.8$	$18.1$	$22.1$	$34.0$	$41.8$	$5.2$	$37.6$

Appendix

Table 4: Image Classification Accuracy on 16 datasets, extension of Tab. 2 . Bold highlights the best score in each dataset within each model variant. [Key: Avg= Average]

Panel of Figure 7 — Figure 7: Image-to-Text Retrieval Results on HierarCaps dataset. Our ARGENT can retrieve more accurate texts in correct order than the baseline. Purple highlights the groundtruth marked in HierarCaps. Groundtruths are in correct order from the most specific to the coarsest detailed captions.

Panel of Figure 8 — Figure 8: Image-to-Text Retrieval Results on HierarCaps dataset. (cont.) Our ARGENT can retrieve more accurate texts in correct order than the baseline. Purple highlights the groundtruth marked in HierarCaps. Groundtruths are in correct order from the most specific to the coarsest detail captions.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 722 papers

Aug 29, 2026

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.

Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat +3

hyper3labs, Berlin, Germany · Hasso Plattner Institute, University of Potsdam, Potsdam, Germany · Faculty of Computers and Data Science, Alexandria University, Alexandria, Egypt +1

Vision-Language Models Vision-Language Alignment Vision-Language Retrieval

Ask PDF

Sep 21, 2026

Hierarchical Prompt Learning for Hyperbolic Vision-Language Models

Hyperbolic vision-language models (VLMs) represent image and text features in a geometry naturally suited to hierarchy, but their adaptation to downstream tasks has largely relied on fixed prompts. Existing prompt learning methods, meanwhile, treat class labels as a flat set and do not exploit available taxonomic structure. We address this gap with a hierarchical prompt learning plug-in for frozen hyperbolic VLMs. Given a fixed offline parent-class hierarchy, it augments a class prompt learner with a separate parent prompt learner, parent-level supervision, hyperbolic entailment regularization, and parent-feedback logit fusion. We instantiate the method with CoOp, CoCoOp and MaPLe, yielding HyPLO, CoHyPLO and MaHyPLO. Across the standard 11-dataset benchmark, all variants improve base-to-new generalization and cross-dataset transfer, and remain comparable to their prompt learning baselines under domain shift. Six hierarchical metrics and embedding analyses show that the method produces more taxonomically consistent predictions and induces a hierarchy-consistent organization of parent, class, and image embeddings in hyperbolic space. Its gains are largest when novel classes must be placed within a fixed taxonomy, and smallest for fine-grained confusions among sibling classes or shifts affecting only the image distribution.

Andro Erdelez, Pascal Mettes, Behzad Bozorgtabar

Aarhus University · University of Amsterdam

Vision-Language Models Prompt Learning VLM Adaptation Hierarchical Representation Learning Image Classification Prompt Learning for VLMs

Ask PDF

Mar 23, 2026

Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models

While Vision-Language Models (VLMs) have achieved remarkable performance, their Euclidean embeddings remain limited in capturing hierarchical relationships such as part-to-whole or parent-child structures, and often face challenges in multi-object compositional scenarios. Hyperbolic VLMs mitigate this issue by better preserving hierarchical structures and modeling part-whole relations (i.e., whole scene and its part images) through entailment. However, existing approaches do not model that each part has a different level of semantic representativeness to the whole. We propose UNcertainty-guided Compositional Hyperbolic Alignment (UNCHA) for enhancing hyperbolic VLMs. UNCHA models part-to-whole semantic representativeness with hyperbolic uncertainty, by assigning lower uncertainty to more representative parts and higher uncertainty to less representative ones for the whole scene. This representativeness is then incorporated into the contrastive objective with uncertainty-guided weights. Finally, the uncertainty is further calibrated with an entailment loss regularized by entropy-based term. With the proposed losses, UNCHA learns hyperbolic embeddings with more accurate part-whole ordering, capturing the underlying compositional structure in an image and improving its understanding of complex multi-object scenes. UNCHA achieves state-of-the-art performance on zero-shot classification, retrieval, and multi-label classification benchmarks. Our code and models are available at: https://github.com/jeeit17/UNCHA.git.

Hayeon Kim, Ji Ha Jang, Junghun James Kim +1

Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · INMC & IPAI, Seoul National University, Republic of Korea

Vision-Language Models Hierarchical Representation Learning Vision-Language Alignment Hyperbolic Representation Learning

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

Hierarchical Prompt Learning for Hyperbolic Vision-Language Models

Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models