What Drives Hierarchy-Aware Image Retrieval? Taxonomy Alignment, Objective Choice, and Geometry

Ling Shi

cs.CVSep 22, 2026

What Drives Hierarchy-Aware Image Retrieval? Taxonomy Alignment, Objective Choice, and Geometry

Authors: Ling Shi

Organizations: Southeast University

Abstract

Foundation vision models provide strong generic representations, yet high class-level retrieval accuracy does not necessarily imply that an embedding respects a target semantic taxonomy. We study strict explicit-taxonomy image retrieval on frozen DINOv2 features and ask: when hierarchical retrieval improves, how much of the change is associated with the organization of taxonomy-aware supervision, and how much with the Euclidean-hyperbolic geometry choice? We evaluate higher levels with strict cross-class criteria that exclude finer-grained matches, and compare Euclidean and hyperbolic projections trained with taxonomy-distance regression or a taxonomy-aware supervised contrastive objective. A compute-matched 2 x 2 Geometry x Loss factorial uses the same 768-256-32 projector capacity, optimization schedule, batch order, and fixed 100-epoch budget; the Loss axis denotes the Regression-to-Taxonomy-SupCon objective-family contrast. On CUB, the objective-family contrasts in mean hierarchy mAP (strict middle/high average, excluding Class/Leaf) are +0.0487 in Euclidean space and +0.0414 in hyperbolic space, compared with geometry contrasts of +0.0102 and +0.0030. On NABirds Parent-disjoint retrieval, the corresponding objective-family contrasts are +0.0467 and +0.0440, whereas geometry contrasts are +0.0017 and -0.0009. A semantic-alignment control shows that the true taxonomy substantially outperforms a structure-preserving shuffled hierarchy, while a NABirds curvature/radius control does not support stronger negative curvature as the explanation for the observed hierarchy gains. Across the two taxonomies, the Regression-to-Taxonomy-SupCon contrasts are larger in aggregate than the evaluated geometry contrasts; semantic alignment also matters separately, while geometry remains hierarchy-dependent.

Figures & tables

Figure 1: Overview of the controlled hierarchy-aware retrieval framework. A frozen DINOv2-ViT-B/14 encoder produces 768-D features, followed by a shared 768–256–32 projector family. We cross Euclidean/Hyperbolic geometry with taxonomy-distance Regression/ Taxonomy-SupCon , forming $E_{\mathrm{R}},H_{\mathrm{R}},E_{\mathrm{S}},H_{\mathrm{S}}$ under matched initialization, batch order, optimizer, and fixed 100-epoch compute. Retrieval uses strict leaf-, middle-, and high-level positives that exclude finer-level matches.

Dataset	Arm	Class mAP	Strict middle mAP	Strict high mAP	Mean hierarchy mAP
CUB	$E_{\mathrm{R}}$	$0.6171\pm 0.0028$	$0.4991\pm 0.0017$	$0.5636\pm 0.0010$	$0.5314\pm 0.0013$
	$H_{\mathrm{R}}$	$0.6351\pm 0.0033$	$0.5153\pm 0.0029$	$0.5679\pm 0.0006$	$0.5416\pm 0.0017$
	$E_{\mathrm{S}}$	$0.5144\pm 0.0032$	$0.5480\pm 0.0022$	$0.6121\pm 0.0007$	$0.5800\pm 0.0009$
	$H_{\mathrm{S}}$	$0.4550\pm 0.0016$	$0.5405\pm 0.0022$	$0.6255\pm 0.0006$	$0.5830\pm 0.0009$
NABirds	$E_{\mathrm{R}}$	$0.6831\pm 0.0028$	$0.1994\pm 0.0034$	$0.4830\pm 0.0029$	$0.3412\pm 0.0031$
	$H_{\mathrm{R}}$	$0.6840\pm 0.0052$	$0.1971\pm 0.0019$	$0.4887\pm 0.0024$	$0.3429\pm 0.0005$

Table 1: Controlled Geometry

\times

Loss factorial. The Loss axis denotes the Regression-to- Taxonomy-SupCon objective-family contrast. All cells share projector capacity, initialization within seed, batch sequence, optimizer, and fixed 100-epoch training. CUB reports its registered query-micro endpoint; NABirds Parent-disjoint reports group-macro. Mean hierarchy mAP averages strict middle and strict high mAP only (Class/Leaf excluded). Values are mean

\pm

sample SD over three training seeds.

Figure 2: Simple contrasts of objective family and geometry under matched compute. The figure’s “Loss” labels denote the Regression-to- Taxonomy-SupCon objective-family contrast; $\Delta_{\mathrm{loss}}$ is retained only as shorthand notation. Each small point is one paired training-seed contrast; the larger point is the mean and the horizontal error bar is descriptive sample SD. Mean hierarchy mAP averages strict middle/high levels only and excludes Class/Leaf. CUB uses its query-micro primary endpoint and NABirds Parent-disjoint its group-macro primary endpoint.

Figure 3: Semantic supervision and hierarchy-level redistribution. (a) NABirds Parent-disjoint mAP under Leaf-only, Leaf+Parent, full true-taxonomy, and structure-preserving shuffled-taxonomy supervision. (b) Per-level Hyperbolic $-$ Euclidean effect under Taxonomy-SupCon . In both datasets, the hyperbolic branch decreases finer/middle-level retrieval while improving the highest evaluated level.

Supervision	Class	Parent	Super.	Mean hier.
Leaf only	.6717	.1901	.4231	.3066
Leaf + Parent	.6178	.2441	.4407	.3424
Full taxonomy	.5459	.2408	.5391	.3899
Shuffled taxonomy	.5874	.1559	.2881	.2220

Table 2: Semantic-supervision ablation on NABirds Parent-disjoint. Parent/Supergroup use group-macro mAP. Each arm uses its validation-frozen training duration; this tests semantic alignment rather than the compute-matched Regression-to- Taxonomy-SupCon objective-family contrast.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Geometry	$\tau$	Genus mAP	Family mAP	Mean	Status
Euclidean	0.07	0.537612	0.617534	0.577573
Euclidean	0.10	0.541775	0.618417	0.580096
Euclidean	0.20	0.549530	0.629024	0.589277	winner
Euclidean	0.30	0.518694	0.658657	0.588676	runner-up
Euclidean	0.50	0.424330	0.696066	0.560198
Euclidean	1.00	0.348874	0.712698	0.530786

Appendix

Table 1: CUB temperature selection. The score is the query-micro mean of strict Genus and strict Family mAP.

Geometry	$\tau$	Parent mAP	Supergroup mAP	Mean	Status
Euclidean	0.07	0.228385	0.544684	0.386535
Euclidean	0.10	0.229470	0.547203	0.388336
Euclidean	0.20	0.232159	0.557994	0.395076
Euclidean	0.30	0.219048	0.574292	0.396670	runner-up
Euclidean	0.50	0.201959	0.595510	0.398735	winner
Euclidean	1.00	0.180174	0.603103	0.391639

Appendix

Table 2: NABirds Parent-disjoint temperature selection. Parent and Supergroup are group-macro mAP; the score is their mean.

Arm	Seed	Class	Genus	Family	Mean hierarchy
$E_{R}$	42	0.613884	0.500621	0.564324	0.532472
$E_{R}$	2024	0.618459	0.497207	0.562511	0.529859
$E_{R}$	3407	0.619024	0.499484	0.563990	0.531737
$H_{R}$	42	0.632376	0.517540	0.568281	0.542911
$H_{R}$	2024	0.638752	0.512089	0.567167	0.539628
$H_{R}$	3407	0.634214	0.516354	0.568181	0.542268

Appendix

Table 3: CUB registered query-micro mAP after averaging the three Q/G perturbations within each training seed.

Effect	42	2024	3407	Mean $\pm$ SD
$\Delta_{\rm geom}^{R}$	+0.010438	+0.009769	+0.010531	$+0.010246\pm 0.000416$
$\Delta_{\rm geom}^{S}$	+0.002273	+0.003090	+0.003638	$+0.003001\pm 0.000687$
$\Delta_{\rm loss}^{E}$	+0.048350	+0.049131	+0.048529	$+0.048670\pm 0.000409$
$\Delta_{\rm loss}^{H}$	+0.040184	+0.042452	+0.041636	$+0.041424\pm 0.001149$
$\Delta_{\rm int}$	-0.008165	-0.006678	-0.006892	$-0.007245\pm 0.000804$

Appendix

Table 4: CUB mean-hierarchy simple effects by independent training seed. Final column is mean

\pm

sample SD.

Arm	Seed	Class	Parent	Supergroup	Mean hierarchy
$E_{R}$	42	0.684085	0.201829	0.484932	0.343381
$E_{R}$	2024	0.685218	0.200846	0.484305	0.342575
$E_{R}$	3407	0.679945	0.195478	0.479702	0.337590
$H_{R}$	42	0.685569	0.196332	0.488446	0.342389
$H_{R}$	2024	0.688281	0.199166	0.486469	0.342818
$H_{R}$	3407	0.678195	0.195661	0.491257	0.343459

Appendix

Table 5: NABirds Parent-disjoint group-macro mAP after averaging Q/G perturbations within training seed.

Effect	42	2024	3407	Mean $\pm$ SD
$\Delta_{\rm geom}^{R}$	-0.000992	+0.000242	+0.005869	$+0.001706\pm 0.003657$
$\Delta_{\rm geom}^{S}$	+0.000734	+0.001133	-0.004577	$-0.000904\pm 0.003188$
$\Delta_{\rm loss}^{E}$	+0.044379	+0.043916	+0.051673	$+0.046656\pm 0.004351$
$\Delta_{\rm loss}^{H}$	+0.046105	+0.044806	+0.041226	$+0.044046\pm 0.002527$
$\Delta_{\rm int}$	+0.001726	+0.000890	-0.010446	$-0.002610\pm 0.006799$

Appendix

Table 6: NABirds mean-hierarchy group-macro simple effects by independent training seed.

Arm	Level	R@1	R@5	R@10	R@20
$E_{R}$	Class	$0.7718\pm 0.0025$	$0.9355\pm 0.0057$	$0.9656\pm 0.0049$	$0.9799\pm 0.0031$
$E_{R}$	Genus	$0.2417\pm 0.0066$	$0.6462\pm 0.0104$	$0.8134\pm 0.0072$	$0.9417\pm 0.0066$
$E_{R}$	Family	$0.0829\pm 0.0012$	$0.2292\pm 0.0065$	$0.3329\pm 0.0051$	$0.5128\pm 0.0090$
$H_{R}$	Class	$0.7895\pm 0.0027$	$0.9387\pm 0.0019$	$0.9650\pm 0.0033$	$0.9793\pm 0.0014$
$H_{R}$	Genus	$0.2328\pm 0.0011$	$0.6476\pm 0.0141$	$0.8170\pm 0.0176$	$0.9561\pm 0.0014$
$H_{R}$	Family	$0.0698\pm 0.0046$	$0.1971\pm 0.0103$	$0.2901\pm 0.0025$	$0.4559\pm 0.0087$

Appendix

Table 7: CUB factorial Recall@

K

(query-micro), mean

\pm

sample SD over training seeds.

Arm	Level	R@1	R@5	R@10	R@20
$E_{R}$	Class	$0.8510\pm 0.0070$	$0.9545\pm 0.0059$	$0.9738\pm 0.0050$	$0.9867\pm 0.0035$
$E_{R}$	Parent	$0.0652\pm 0.0098$	$0.2079\pm 0.0083$	$0.3360\pm 0.0126$	$0.5090\pm 0.0188$
$E_{R}$	Supergroup	$0.0848\pm 0.0086$	$0.2561\pm 0.0052$	$0.3602\pm 0.0151$	$0.5410\pm 0.0199$
$H_{R}$	Class	$0.8507\pm 0.0027$	$0.9537\pm 0.0053$	$0.9762\pm 0.0012$	$0.9874\pm 0.0017$
$H_{R}$	Parent	$0.0615\pm 0.0029$	$0.2127\pm 0.0016$	$0.3348\pm 0.0112$	$0.5119\pm 0.0081$
$H_{R}$	Supergroup	$0.0875\pm 0.0087$	$0.2528\pm 0.0240$	$0.3662\pm 0.0199$	$0.5339\pm 0.0158$

Appendix

Table 8: NABirds Parent-disjoint factorial Recall@

K

(group-macro), mean

\pm

sample SD over training seeds.

Arm	Class R@1	Class mAP	Parent mAP	Supergroup mAP	Mean hierarchy
Leaf-only	$.8447\pm.0013$	$.6717\pm.0067$	$.1901\pm.0013$	$.4231\pm.0025$	$.3066\pm.0019$
Leaf+Parent	$.8096\pm.0018$	$.6178\pm.0072$	$.2441\pm.0072$	$.4407\pm.0059$	$.3424\pm.0015$
Full taxonomy	$.7478\pm.0078$	$.5459\pm.0030$	$.2408\pm.0056$	$.5391\pm.0024$	$.3899\pm.0031$
Shuffled taxonomy	$.8126\pm.0043$	$.5874\pm.0034$	$.1559\pm.0105$	$.2881\pm.0113$	$.2220\pm.0107$

Appendix

Table 9: NABirds Parent-disjoint semantic-alignment control. Values are mean

\pm

sample SD over training seeds after Q/G averaging within seed.

Configuration	Parent mAP	Supergroup mAP	Mean hierarchy
Near-flat $c=10^{-6},r=.4$	$.196609\pm.001516$	$.489188\pm.000151$	$.342898\pm.000682$
Base $c=.1,r=.4$	$.196308\pm.001256$	$.488650\pm.000356$	$.342479\pm.000457$
Strong curvature $c=1,r=.4$	$.195269\pm.000472$	$.485061\pm.000216$	$.340165\pm.000173$
Small radius $c=.1,r=.2$	$.197770\pm.001646$	$.471127\pm.003440$	$.334449\pm.001068$
Large radius $c=.1,r=.8$	$.199846\pm.004081$	$.478574\pm.009418$	$.339210\pm.006642$
Matched Euclidean	$.198735\pm.003163$	$.482696\pm.004045$	$.340715\pm.003594$

Appendix

Table 10: NABirds Parent-disjoint curvature/radius mechanism control. Mean

\pm

sample SD over training seeds.

Effect	42	2024	3407	Mean $\pm$ SD
$\Delta_{\rm geom}^{R}$	+0.008632	+0.008104	+0.009976	$+0.008904\pm 0.000965$
$\Delta_{\rm geom}^{S}$	+0.004214	+0.004807	+0.004921	$+0.004647\pm 0.000380$
$\Delta_{\rm loss}^{E}$	+0.061003	+0.062655	+0.062948	$+0.062202\pm 0.001048$
$\Delta_{\rm loss}^{H}$	+0.056585	+0.059358	+0.057892	$+0.057945\pm 0.001387$
$\Delta_{\rm int}$	-0.004418	-0.003296	-0.005055	$-0.004256\pm 0.000890$

Appendix

Table 11: Post-hoc CUB group-macro mean-hierarchy effects.

Check	CUB	NABirds Parent-disjoint
Training samples	5,994	19,021
Trainable parameters/cell	205,600	205,600
Batch size	128	128
Epochs/cell	100	100
Updates/epoch	46	148
Training seeds	42/2024/3407	42/2024/3407

Appendix

Table 12: Final factorial fairness and leakage checks from the accepted pre-test/final audits.

Image Retrieval

Image retrieval finds and ranks images from visual, textual, or combined queries across large and specialized collections.

See all 85 papers

Jul 3, 2026

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization (

HiR^2

), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that

HiR^2

effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2_ICML2026.

Hulingxiao He, Zhi Tan, Yuxin Peng

Wangxuan Institute of Computer Technology, Peking University

Hierarchical Classification Hierarchical Representation Learning Multimodal Large Language Models

Ask PDF

Mar 25, 2026

Light Cones For Vision: Simple Causal Priors For Visual Hierarchy

Standard vision models treat objects as independent points in Euclidean space, unable to capture hierarchical structure like parts within wholes. We introduce Worldline Slot Attention, which models objects as persistent trajectories through spacetime worldlines, where each object has multiple slots at different hierarchy levels sharing the same spatial position but differing in temporal coordinates. This architecture consistently fails without geometric structure: Euclidean worldlines achieve 0.078 level accuracy, below random chance (0.33), while Lorentzian worldlines achieve 0.479-0.661 across three datasets: a 6x improvement replicated over 20+ independent runs. Lorentzian geometry also outperforms hyperbolic embeddings showing visual hierarchies require causal structure (temporal dependency) rather than tree structure (radial branching). Our results demonstrate that hierarchical object discovery requires geometric structure encoding asymmetric causality, an inductive bias absent from Euclidean space but natural to Lorentzian light cones, achieved with only 11K parameters. The code is available at: https://github.com/iclrsubmissiongram/loco.

Manglam Kartik, Neel Tushar Shah

Indian Institute of Technology Bombay, Mumbai, India

Hierarchical Representation Learning Object-Centric Representation Learning Slot Attention

Ask PDF

Mar 24, 2026

ARGENT: Adaptive Hierarchical Image-Text Representations

Large-scale Vision-Language Models (VLMs) such as CLIP learn powerful semantic representations but operate in Euclidean space, which fails to capture the inherent hierarchical structure of visual and linguistic concepts. Hyperbolic geometry, with its exponential volume growth, offers a principled alternative for embedding such hierarchies with low distortion. However, existing hyperbolic VLMs use entailment losses that are unstable: as parent embeddings contract toward the origin, their entailment cones widen toward a half-space, causing catastrophic cone collapse that destroys the intended hierarchy. Additionally, hierarchical evaluation of these models remains unreliable, being largely retrieval-based and correlation-based metrics and prone to taxonomy dependence and ambiguous negatives. To address these limitations, we propose an adaptive entailment loss paired with a norm regularizer that prevents cone collapse without heuristic aperture clipping. We further introduce an angle-based probabilistic entailment protocol (PEP) for evaluating hierarchical understanding, scored with AUC-ROC and Average Precision. This paper introduces a stronger hyperbolic VLM baseline ARGENT, Adaptive hieRarchical imaGe-tExt represeNTation. ARGENT improves the SOTA hyperbolic VLM by 0.7, 1.1, and 0.8 absolute points on image classification, text-to-image retrieval, and proposed hierarchical metrics, respectively.

Chuong Huynh, Hossein Souri, Abhinav Kumar +3

University of Maryland, College Park · Samsung Research America, AI Center – Mountain View

Vision-Language Models Hierarchical Representation Learning Hyperbolic Representation Learning VLM Evaluation

Ask PDF

Abstract

Image Retrieval

Explore similar work

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

Light Cones For Vision: Simple Causal Priors For Visual Hierarchy

ARGENT: Adaptive Hierarchical Image-Text Representations