Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering

Nirmalendu Prakash; Narmeen Fatimah Oozeer; Xin Su; Phillip Howard; Shaan Shah; Zoe Wanying He; Shuang Wu; Shivam Raval

cs.CVApr 13, 2026

Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering

Authors: Nirmalendu Prakash, Narmeen Fatimah Oozeer, Xin Su, Phillip Howard, Shaan Shah, Zoe Wanying He, Shuang Wu, Shivam Raval,

Organizations: Singapore University of Technology and Design · Martian · Thoughtworks · UCSD ·

Abstract

CLIP retrieval is typically framed as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global cross-modal alignment, many retrieval failures arise from local geometric inconsistencies: nearby items are incorrectly ordered, leading to systematic confusions (e.g., pentagon vs. hexagon) and produces diffuse, weakly controlled result sets. Prior work largely optimizes for point wise relevance or finetuning to mitigate these problems. We instead view retrieval as a problem of neighborhood alignment. Our work introduces (1) neighborhood-level re-ranking via Hungarian matching, which rewards structural consistency; (2) query-conditioned local steering, where directions derived from contrastive neighborhoods around the query reshape retrieval. We show that these techniques improve retrieval performance on attribute-binding and compositional retrieval tasks. Together, these methods operate on local neighborhoods but serve different roles: re-ranking rewards alignment whereas local steering controls neighborhood structure. This shows that retrieval quality and controllability depend critically on local structure, which can be exploited at inference time without retraining.

Figures & tables

Figure 1: Local neighborhood structure provides information beyond pointwise similarity. The left panel illustrates how matching the geometry of nearby neighbors can identify the correct cross-modal correspondence when cosine similarity fails. The right panel shows an example from retrieval, where Fused Gromov Wasserstein(FGW)-based reranking exploits this local alignment and corrects the top-1 prediction.

		Recall@ $K$			nDCG@ $K$
Stage 1	Stage 2	R@1	R@5	R@10	nDCG@1	nDCG@5	nDCG@10
CLIP	None	3.5	13.4	21.3	0.4	0.6	0.7
CLIP	Local Hungarian	16.4	—	—	0.2	—	—
CLIP	Local FGW	5.1	14.9	22.5	0.5	0.6	0.8
Ridge	None	72.3	91.5	94.6	0.9	0.8	0.9
Ridge	Local Hungarian	97.6	—	—	1.0	—	—

Table 1: Synthetic Shapes. Best per-metric in bold within each section. “—” = not applicable. Recall@

K

from full retrieval; nDCG@

K

from VLM-relevance aggregation (

n{=}200

queries).

		Recall@ $K$			nDCG@ $K$
Stage 1	Stage 2	R@1	R@5	R@10	nDCG@1	nDCG@5	nDCG@10
CLIP	None	42.8	67.3	75.9	0.7	0.7	0.8
CLIP	Local Hungarian	78.4	83.3	86.1	0.8	—	—
CLIP	Local FGW	56.3	72.7	79.3	0.5	0.6	0.8
Ridge	None	74.3	84.4	87.2	0.8	0.8	0.9
Ridge	Local Hungarian	91.1	—	—	—	—	—

Table 2: Urban1K. Best per-metric in bold within each section. “—” = not applicable. Recall@

K

from full retrieval; nDCG@

K

from VLM-relevance aggregation (

n{=}200

queries).

Dataset	Metric	Baseline CLIP	Multi-object steering (average)	FGW reranking	Hungarian reranking
CLEVR-HOPE	R@1	0.050	0.010	0.682	0.683
	R@5	0.141	0.051	0.682	—
	R@10	0.230	0.094	0.682	—
	nDCG@1	0.054	0.012	0.683	0.695
	nDCG@5	0.103	0.039	0.665	—
	nDCG@10	0.146	0.063	0.682	—

Table 3: Compositional retrieval results on CLEVR-HOPE and Visual Genome. Queries are constructed from multiple object–attribute descriptions, and we compare baseline CLIP retrieval, naive multi-object steering, and FGW/Hungarian reranking. Reranking consistently improves retrieval and ranking metrics, and yields the strongest noun-level grounding (CAS-noun), while naive multi-object steering degrades performance in both settings.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

		Recall@ $K$			nDCG@ $K$
Stage 1	Stage 2	R@1	R@5	R@10	nDCG@1	nDCG@5	nDCG@10
CLIP	None	8.7	34.9	55.5	0.6	0.7	0.8
CLIP	Local Hungarian	43.9	—	—	0.5	—	—
CLIP	Local FGW	8.7	34.9	55.7	0.5	0.6	0.7
Ridge	None	9.2	39.9	62.8	0.6	0.7	0.8
Ridge	Local Hungarian	52.5	—	—	0.5	—	—

Appendix

Table 4: SugarCrepe. Best per-metric in bold . “—” = not applicable. Recall@

K

from full retrieval; nDCG@

K

from VLM-relevance aggregation (

n{=}200

queries).

		Recall@ $K$			nDCG@ $K$
Stage 1	Stage 2	R@1	R@5	R@10	nDCG@1	nDCG@5	nDCG@10
SigLIP	None	8.7	34.9	55.5	0.7	0.8	0.9
SigLIP	Local Hungarian	43.9	—	—	0.4	—	—
SigLIP	Local FGW	7.5	34.2	55.7	0.6	0.7	0.8
Ridge	None	0.7	39.9	62.8	0.7	0.8	0.8
Ridge	Local Hungarian	52.5	—	—	0.5	—	—

Appendix

Table 5: SugarCrepe. Best per metric in bold . “—” = not applicable. Recall@

K

from full retrieval; nDCG@

K

from VLM-relevance aggregation (

n{=}200

queries).

		Recall@ $K$			nDCG@ $K$
Stage 1	Stage 2	R@1	R@5	R@10	nDCG@1	nDCG@5	nDCG@10
SigLIP	None	3.5	13.4	21.3	0.4	0.6	0.7
SigLIP	Local Hungarian	16.4	—	—	0.2	—	—
SigLIP	Local FGW	4.5	16.1	23.9	0.5	0.6	0.8
Ridge	None	72.3	91.5	94.6	0.9	0.8	0.9
Ridge	Local Hungarian	97.6	—	—	1.0	—	—

Appendix

Table 6: Synthetic shapes. Best per metric in bold . “—” = not applicable. nDCG@

K

from VLM-relevance aggregation (full split,

n{=}13244

		Recall@ $K$			nDCG@ $K$
Stage 1	Stage 2	R@1	R@5	R@10	nDCG@1	nDCG@5	nDCG@10
SigLIP	None	42.8	67.3	75.9	0.7	0.7	0.8
SigLIP	Local Hungarian	76.5	—	—	0.8	—	—
SigLIP	Local FGW	18.4	72.3	79.8	0.4	0.6	0.8
Ridge	None	74.3	84.4	87.2	0.8	0.8	0.9
Ridge	Local Hungarian	91.1	—	—	0.9	—	—

Appendix

Table 7: Urban1k. Best per metric in bold . “—” = not applicable. nDCG@

K

from VLM-relevance aggregation (

n{=}200

queries).

Panel of Figure 3 — Figure 3: Representative per-query FGW recovery trajectories on Synthetic Shapes. For each query, we plot the rank of the ground-truth image under FGW as the cosine neighborhood size $k$ varies, with the dashed line marking R@1. FGW recovery is non-monotonic: the ground-truth may briefly reach rank 0 for some values of $k$ , but this behavior is unstable and often disappears as the neighborhood changes.

Figure 4: Shape substitution matrix on Synthetic Shapes. Each entry shows how often a query containing shape $s_{i}$ retrieves an image containing shape $s_{j}$ instead, with color held fixed. Errors are concentrated on a few structured confusions, most notably pentagon–hexagon.

Dataset	#Samples	Domain	Setting
Synthetic Shapes	2,430	Synthetic	Compositional retrieval
Urban1K	1,000	Real-world	Long-caption retrieval
Visual Genome	108,077	Real-world	Dense grounding
SugarCrepe	7,512	Real-world	Compositional retrieval
NAC	2,430	Real-world (controlled)	Attribute binding
CLEVR-HOPE	1,000	Synthetic	Compositional retrieval

Appendix

Table 8: Summary of datasets used in our experiments.

Type	Count (both dirs)	%
color	89,658	87.6
size	4,500	4.4
pose	1,788	1.7
state	1,662	1.6
shape	668	0.7
human	216	0.2

Appendix

Table 9: Distribution of contrastive attribute types in the Visual Genome-derived dataset. Counts are aggregated across both directions of each contrastive pair.

Attribute contrast	Nouns retained (min count)
cluttered $\rightarrow$ tidy	garage (41), basement (15), kitchen (13)
foggy $\rightarrow$ clear	lake (19)
folded $\rightarrow$ unfolded	shirt (23)
frozen $\rightarrow$ thawed	lake (13)
open $\rightarrow$ closed	bucket (54), pot (41), bottle (27), bag (22),
	gate (20), road (20), street (20), door (19),

Appendix

Table 10: Final steerable attribute contrasts after verification and filtering. We retain only noun–attribute pairs with Qwen-VL confidence

\geq 0.9

and at least 10 verified samples, ensuring that retrieval metrics such as R@10 can be computed reliably.

Figure 5: Sample NAC top-5 retrievals: original CLIP text-image matching ( $\alpha=0$ ) versus text embeddings steered to correct the attribute ‘clear’ ( $\alpha>0$ ).

Figure 6: Retrieval quality as a function of steering strength

\alpha

on NAC, Synthetic Shapes, and Visual Genome. For each dataset, we report Recall@K, concept attribution scores, and nDCG@K under query-conditioned steering. Steering is effective in single-concept settings, where moderate values of

\alpha

improve both retrieval and concept fidelity, with especially strong gains on Synthetic Shapes and consistent improvements on NAC. In contrast, performance on Visual Genome degrades as

\alpha

increases.

Figure 7: Sample Synthetic Shapes top-5 retrievals: original CLIP text–image matching ( $\alpha=0$ , left) versus text embeddings steered toward the contrast attribute ( $\alpha>0$ , right).

Figure 8: Sample NAC top-5 retrievals: original CLIP text-image matching ( $\alpha=0$ ) versus text embeddings steered to correct the attribute ‘upright’ ( $\alpha>0$ ).

Figure 9: Sample NAC top-5 retrievals: original CLIP text-image matching ( $\alpha=0$ ) versus text embeddings steered to correct the attribute ‘toppled’ ( $\alpha>0$ ).

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

PlaceSeek: Human-Centered Geospatial Retrieval of Urban Outdoor Places via Semantic Grounding and Affective Alignment
Aug 25, 2026Ziqi Cui, Shangyu LouInformation Retrieval Image Retrieval
University of California, Santa Barbara & San Diego State University California, USA
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
May 22, 2026Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra +1Cross-Modal Learning Cross-Modal Alignment
University of Amsterdam Amsterdam, The Netherlands
CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space
Apr 13, 2026Sohwi Lim, Lee Hyoseok, Jungjoon Park +1Vision-Language Retrieval
KAIST

Aug 25, 2026J/K move · Enter open · S save

PlaceSeek: Human-Centered Geospatial Retrieval of Urban Outdoor Places via Semantic Grounding and Affective Alignment

Ziqi Cui, Shangyu Lou

University of California, Santa Barbara & San Diego State University California, USA

Information Retrieval Image Retrieval Vision-Language Retrieval Visual Grounding

Open paper AskPDF arXiv

People search for urban outdoor places not only by category or function, but also by what activities a place can support and how it is perceived. Existing geospatial retrieval remains largely POIcentric and metadata-driven, making it difficult to satisfy openended, affective, or activity-oriented needs. We present PlaceSeek, a human-centered outdoor place retrieval framework that maps natural-language queries to geolocated street-view imagery. PlaceSeek introduces an intent-aware retrieval mechanism that decomposes user queries into functional and affective sub-intents. A Semantic Grounding Module verifies whether candidate street-view results contain the physical evidence needed to support the intended activity, while an Affective Alignment Module re-ranks physically valid candidates using a LoRA-adapted vision-language model trained on human urban perception judgments. We evaluate PlaceSeek on 31,956 street-view locations in Milan across 10 naturallanguage queries annotated by five human evaluators. PlaceSeek achieves 88.0% Precision@5, a mean match score of 3.39/4.0, and 0.920 nDCG@5, outperforming CLIP, fine-tuned CLIP, SigLIP, and a VQA-based baseline. Ablation results show that physical grounding is essential for retrieval validity, while affective alignment improves ranking quality among physically valid candidates. These findings highlight that complex urban spatial queries require modeling both verifiable visual evidence and human perceptual preferences. PlaceSeek provides a potential framework for human-centered nextgeneration geospatial retrieval systems.

Abstract

Cross-Modal Retrieval

Explore similar work

PlaceSeek: Human-Centered Geospatial Retrieval of Urban Outdoor Places via Semantic Grounding and Affective Alignment

RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?

CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space