Discarded or Unreachable? Diagnosing and Repairing Spatial Relation Failure in Vision--Language Similarity

Mahdi Mostajabdaveh; Zirui Zhou

cs.CVNov 3, 2026

Discarded or Unreachable? Diagnosing and Repairing Spatial Relation Failure in Vision--Language Similarity

Authors: Mahdi Mostajabdaveh, Zirui Zhou

Organizations: Huawei Technologies Canada

Abstract

Contrastive vision-language models score left/right relations near chance, but the symptom has two causes that call for opposite repairs: the relation may survive in the pooled embedding yet be unreachable by native image-text similarity, or pooling may discard it while the dense features retain it. On clean two object photographs, across ten encoders, the lateral polarity is linearly decodable from the flip odd component of the pooled vector at 0.77-0.99, yet a search over a million captions leaves nine of the ten at chance under native scoring: an access failure. On controlled C4 orbits, where the relation is the only thing that varies, pooling keeps the arrangement axis at 0.998-1.000 and loses its polarity, which the patch grid retains: a pooling failure. Both sit downstream of the dense features, so one readout covers both. Our zero parameter, query conditioned spatial moment recomputes the geometry from the patch grid and stores it as a vector beside the native embedding. On a 118,287 image index it raises left/right pairwise accuracy against reversed geometry negatives from 0.51-0.53 to 0.81-0.96 and conditional R@1 among 21 category matched candidates from 0.05 to 0.37-0.75, at under a millisecond per batched query, while reproducing the native ranking exactly for queries that name no relation.

Figures & tables

Figure 1: One near-chance symptom, two failures, one repair. (a) On What’s Up the pooled vector carries the polarity, in its flip-odd component $z^{-}$ (Sec. 3 ), but native scoring cannot reach it; on the $C_{4}$ orbits only the patch grid carries it. Chance is $0.5$ ( $0.25$ for the 4-way row); the ring marks the object that moves. (b) What’s Up left/right under one protocol. Flip test-time augmentation compares a photograph with its mirror; pretraining is $24$ ViT-B/32 CLIPs trained from scratch with up to all captions relational; fine-tuning is three fine-tunes on $256$ k pairs. Ranges are over the ten encoders of Tab. 1 , or over runs.

		Supervised probe (What’s Up, lateral)				Mirror pairs		Mirror ranking
Encoder	Pooling	$z$	$z^{-}$	$\Delta_{\text{lat}}$	$\Delta_{\text{vert}}$	$z^{-}$	$z^{+}$	flip effect	moment
CLIP ViT-B/16	CLS + proj	0.674	0.771	$+0.097$	$-0.015$	1.000	0.500	0.510 [0.476, 0.549]	0.951
CLIP LAION-2B	CLS + proj	0.705	0.878	$+0.174$	$-0.005$	1.000	0.500	0.500 [0.461, 0.539]	0.942
CLIP DataComp	CLS + proj	0.713	0.805	$+0.092$	$-0.015$	1.000	0.500	0.495 [0.466, 0.524]	0.893
MetaCLIP-B	CLS + proj	0.700	0.815	$+0.115$	$+0.000$	0.990	0.500	0.510 [0.485, 0.534]	0.951
DFN-B	CLS + proj	0.660	0.825	$+0.165$	$+0.005$	1.000	0.500	0.510 [0.490, 0.534]	0.942

Table 1: The lateral polarity is in the flip-odd component while the native caption-specific flip effect does not read it. Left: a probe on

z^{-}

beats one on the full pooled vector on every encoder, while the same projection does nothing vertically. Middle:

z^{+}

is exactly chance on mirror pairs, as Lemma 1 requires. Right, both columns under mirror ranking , a photograph against its own flip: the caption-specific flip effect is at chance on every encoder, while the dense spatial moment reads the polarity through that same protocol. Cluster bootstraps over object pairs. In every table, bold marks the best value in a group of directly comparable numbers; here,

z

against

z^{-}

and the flip effect against the moment.

Encoder	best in $V_{128}$	real caps.	$\cos(\delta_{T},w^{\star})$
CLIP ViT-B/16	0.690	0.524	$-0.071$
CLIP LAION-2B	0.728	0.510	$-0.038$
CLIP DataComp	0.835	0.490	$-0.000$
MetaCLIP-B	0.709	0.558	$+0.032$
DFN-B	0.775	0.505	$+0.042$
SigLIP2-B	0.767	0.490	$+0.023$

Table 2: The caption subspace is not the bottleneck; alignment is.

V_{128}

is the top-128 principal subspace of

20

K COCO caption embeddings,

\approx\!85\%

of caption energy. Col. 2: the best bias-free single direction in it, the functional form a caption score has. Col. 3: the best of

22

relational phrasings, selected on training folds and scored held out. Col. 4: its angle to the image-side lateral direction.

	pooled, binary (chance $0.5$ )			4-way (chance $0.25$ )
Encoder	axis	polarity	polarity †	probe	native	moment
CLIP ViT-B/16	1.000	0.507	0.513	0.475	0.252	0.819
CLIP LAION-2B	0.998	0.470	0.557	0.476	0.284	0.877
CLIP DataComp	1.000	0.490	0.497	0.481	0.273	0.886
MetaCLIP-B	1.000	0.503	0.530	0.487	0.286	0.823
DFN-B	1.000	0.520	0.580	0.481	0.266	0.858

Table 3: Pooling keeps the arrangement axis and loses the polarity along it.

300

exact

C_{4}

orbits: two segmented objects at a fixed pair centre, displacement rotated

90^{\circ}

per step, object pixels upright, so the four members differ only in the relation. axis and polarity are directions fitted on these orbits and tested on held-out ones.

\dagger

is the best of an MLP, an RBF-SVM and a cosine kNN on the same rows, orbit groups and folds. On those same rows all three recover an XOR label planted at the polarity channel’s own amplitude at

0.978

–

1.000

, which is what makes this column a null rather than an underpowered reader. A reader that identifies the axis and guesses the polarity scores

0.5

on the 4-way task, which is where the probe column sits. The channel is isotropic (

|\cos(u_{x},u_{y})|\leq 0.019

Figure 2: What is stored, what a query adds, and what their product reads. In (c), the fields are centred patch relevance for each object name; dashed lines mark the first spatial moments used by the score.

	left/right				above/below
	pairwise		cond. R@1		pairwise		cond. R@1
Encoder	nat.	+mom.	nat.	+mom.	nat.	+mom.	nat.	+mom.
CLIP ViT-B/16	0.510	0.839	0.048	0.411	0.497	0.806	0.059	0.312
CLIP LAION-2B	0.516	0.857	0.052	0.451	0.527	0.820	0.049	0.356
CLIP DataComp	0.512	0.844	0.047	0.421	0.523	0.809	0.050	0.356
MetaCLIP-B	0.513	0.808	0.046	0.365	0.506	0.781	0.046	0.289

Table 4: The remedy is not specific to the left/right axis, and it survives as a ranking. Relational retrieval over 118,287 images. Every distractor holds the same two object categories as the gold image with the geometry reversed, so pairwise is the fraction of them a gold image outscores and cond. R@1 is how often it is ranked first among all

21

candidates — that is, how often the top-ranked image satisfies the queried relation.

n=6{,}595

horizontal and

3{,}114

vertical scored queries; cond. R@1 uses the

4{,}812

and

1{,}110

with a full pool, so its chance level is a single number. Both scores here are given the parsed

(A,\text{relation},B)

triple; free-form queries and the parser are evaluated separately in the supplement. Ambiguous splits and the

\beta

grid are in the supplement. Cluster bootstraps over query images and over unordered category pairs, whichever is wider, put every pairwise cell inside

\pm 0.029

; per-encoder intervals, including the paired difference, are in the supplement.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	checkpoint
CLIP ViT-B/16	ViT-B-16 / openai
CLIP LAION-2B	ViT-B-16 / laion2b_s34b_b88k
CLIP DataComp	ViT-B-16 / datacomp_xl_s13b_b90k
MetaCLIP-B	ViT-B-16-quickgelu / metaclip_fullcc
DFN-B	ViT-B-16 / dfn2b
SigLIP2-B	google/siglip2-base-patch16-224

Appendix

Table S1 : The evaluated checkpoints. The ten encoders with a text tower used throughout, plus the three appearing only in the symmetry-transport analysis; pooling heads are in Tab. 1 . A model / tag pair is an OpenCLIP architecture and pretrained tag, a single string a Hugging Face identifier.

		PCA sketch		random projection
$k$	storage	acc.	cost	acc.	cost
16	1.03–1.06 $\times$	0.748	$+0.090$	0.598	$+0.212$
32	1.06–1.12 $\times$	0.774	$+0.056$	0.683	$+0.137$
64	1.11–1.25 $\times$	0.784	$+0.036$	0.696	$+0.113$
128	1.22–1.50 $\times$	0.808	$+0.019$	0.718	$+0.078$
256	1.44–2.00 $\times$	0.820	$+0.005$	0.751	$+0.043$

Appendix

Table S2: What the compressed index costs, over the whole rank range. Median across 10 encoders on What’s Up lateral; storage is stored floats per image as a multiple of the native vector. cost is the drop from the uncompressed

3D

representation, so a negative entry beats it. PCA is fitted on

400

unlabelled photographs available at index-build time; the random projection is the assumption-free construction, reported because it is the one that fails.

	obj.-identity	wrong-cap.	VSR blind	spatial moment: real / shuf.
Encoder	variant	null	prior	(What’s Up lat.)
CLIP ViT-B/16	0.563	0.510	0.501	0.772 / 0.588
CLIP LAION-2B	0.597	0.451	0.501	0.806 / 0.635
CLIP DataComp	0.563	0.398	0.501	0.694 / 0.523
MetaCLIP-B	0.549	0.393	0.501	0.772 / 0.626
DFN-B	0.437	0.621	0.501	0.782 / 0.630

Appendix

Table S3 : The priors and nulls behind every real-image number. Under photo-pair and mirror ranking both competing photographs carry the same caption, so a text-only reader is exactly

0.500

by construction rather than by measurement; VSR is unpaired, so its blind prior is measured (col. 4). Col. 2 scores the lateral relation from object-name directions instead of the relation, the object-identity control. Col. 3 is the shuffled-query null for native scoring: the same mirror test run with the opposite caption, whose difference from the raw value is the flip effect of Tab. 1 . Col. 5 is the shuffled-query null for the spatial moment, its score when given another item’s object names; the real score is above it on every encoder, by

0.108

–

0.225

	from scratch ( $24$ $+$ $6$ runs)	fine-tune ( $3$ )	learned pool ( $60$ )
init	ViT-B/32, random	ViT-B/32, laion2b_s34b_b79k	frozen enc. $+$ head
optimiser	AdamW $(0.9,0.98)$ , $\epsilon{=}10^{-6}$	AdamW	AdamW
lr	$10^{-3}$ , cosine	$10^{-4}$ LoRA, $10^{-6}$ full	$3{\times}10^{-4}$ , constant
warmup	$2{,}000$ steps	none	none
weight decay	$0.2$	$0.1$	$10^{-2}$
batch	$1{,}024$	$128$	$32$

Appendix

Table S4 : Optimisation settings for every training run in the paper, as run. Values not listed are the open_clip defaults; nothing was tuned per cell.

Figure S1 : The decodability–reachability gap under the standard caption set, and the geometry behind it. Left: native scoring is at chance on all ten while a probe on the flip-odd component reads the lateral polarity; the connector is the gap between what the representation holds and what the metric uses. Right: the best bias-free direction in the caption span

V_{128}

is far above chance while the best of the

22

relational caption pairs remains near chance, and the two relation directions are near-orthogonal.

Reader	WU l/r	WU on/under	VSR	$C_{4}$ 4-way
	(0.5)	(0.5)	(0.5)	(0.25)
spatial moment (ours)	0.826	0.662	0.738	0.894
centroid, $T{=}1$	0.827	0.664	0.738	0.895
centroid, $T{=}4$	0.828	0.663	0.738	0.895
centroid, $T{=}16$	0.827	0.662	0.738	0.894
argmax patch ( $k{=}1$ )	0.713	0.602	0.666	0.490

Appendix

Table S5: Simpler readers off the same patch grid, and what they cost. Mean over 10 encoders; chance in italics. Every row reads the same dense features and differs only in how it turns a relevance field into a position. The centroid family is the spatial moment numerically, and

k{=}

all is chance by construction. What separates the spatial moment is not accuracy but that it factorises: every other row needs the patch grid at query time,

24

–

364\times

the bytes and

52

–

754\times

the compute.

	left/right			above/below
Encoder	native	+moment	$\Delta$	native	+moment	$\Delta$
CLIP ViT-B/16	0.510 [0.501, 0.519]	0.839 [0.826, 0.851]	0.329 [0.314, 0.344]	0.497 [0.475, 0.520]	0.806 [0.784, 0.826]	0.309 [0.286, 0.329]
CLIP LAION-2B	0.516 [0.507, 0.525]	0.857 [0.845, 0.868]	0.341 [0.327, 0.354]	0.527 [0.502, 0.552]	0.820 [0.797, 0.841]	0.293 [0.267, 0.318]
CLIP DataComp	0.512 [0.503, 0.522]	0.844 [0.830, 0.856]	0.332 [0.317, 0.346]	0.523 [0.498, 0.547]	0.809 [0.788, 0.829]	0.286 [0.263, 0.311]
MetaCLIP-B	0.513 [0.504, 0.523]	0.808 [0.793, 0.822]	0.294 [0.279, 0.310]	0.506 [0.481, 0.531]	0.781 [0.758, 0.803]	0.276 [0.253, 0.299]
DFN-B	0.512 [0.503, 0.521]	0.837 [0.823, 0.851]	0.325 [0.310, 0.340]	0.526 [0.502, 0.550]	0.827 [0.810, 0.844]	0.302 [0.278, 0.325]

Appendix

Table S6 : Cluster-robust intervals for every cell of the clean retrieval split.

95\%

bootstrap,

2{,}000

resamples. The queries are not independent draws — the 6,595 scored horizontal ones sit on 4,553 images and use 831 unordered category pairs — so we resample clusters, by image and by category pair, and report the wider. Both arms score the same queries, so

\Delta

is a paired difference. Widest half-width here:

\pm 0.029

	left/right			above/below
Encoder	native	+moment	$\Delta$	native	+moment	$\Delta$
CLIP ViT-B/16	0.498 [0.494, 0.503]	0.807 [0.797, 0.817]	0.308 [0.299, 0.318]	0.504 [0.495, 0.513]	0.783 [0.770, 0.795]	0.279 [0.267, 0.291]
CLIP LAION-2B	0.497 [0.492, 0.501]	0.810 [0.801, 0.820]	0.314 [0.304, 0.323]	0.508 [0.498, 0.518]	0.792 [0.780, 0.804]	0.284 [0.271, 0.297]
CLIP DataComp	0.498 [0.493, 0.502]	0.803 [0.793, 0.813]	0.306 [0.296, 0.315]	0.510 [0.501, 0.520]	0.789 [0.776, 0.801]	0.278 [0.265, 0.291]
MetaCLIP-B	0.497 [0.492, 0.501]	0.762 [0.752, 0.772]	0.265 [0.256, 0.275]	0.508 [0.498, 0.518]	0.750 [0.737, 0.762]	0.242 [0.230, 0.254]
DFN-B	0.498 [0.493, 0.502]	0.799 [0.789, 0.810]	0.302 [0.291, 0.311]	0.513 [0.504, 0.522]	0.788 [0.776, 0.800]	0.276 [0.264, 0.288]

Appendix

Table S7 : Cluster-robust intervals for every cell of the ambiguous retrieval split.

95\%

bootstrap,

2{,}000

resamples. The queries are not independent draws — the 27,551 scored horizontal ones sit on 18,210 images and use 1,073 unordered category pairs — so we resample clusters, by image and by category pair, and report the wider. Both arms score the same queries, so

\Delta

is a paired difference. Widest half-width here:

\pm 0.014

	left/right			above/below
Encoder	native	+moment	$\Delta$	native	+moment	$\Delta$
CLIP ViT-B/16	0.048 [0.042, 0.055]	0.411 [0.378, 0.443]	0.362 [0.330, 0.395]	0.059 [0.043, 0.076]	0.312 [0.256, 0.367]	0.253 [0.202, 0.309]
CLIP LAION-2B	0.052 [0.045, 0.059]	0.451 [0.414, 0.485]	0.399 [0.363, 0.435]	0.049 [0.033, 0.067]	0.356 [0.299, 0.413]	0.307 [0.253, 0.362]
CLIP DataComp	0.047 [0.041, 0.053]	0.421 [0.383, 0.458]	0.374 [0.336, 0.412]	0.050 [0.030, 0.072]	0.356 [0.299, 0.414]	0.306 [0.253, 0.360]
MetaCLIP-B	0.046 [0.040, 0.053]	0.365 [0.331, 0.399]	0.319 [0.284, 0.353]	0.046 [0.029, 0.065]	0.289 [0.240, 0.340]	0.243 [0.200, 0.288]
DFN-B	0.045 [0.038, 0.051]	0.420 [0.385, 0.453]	0.376 [0.342, 0.408]	0.050 [0.035, 0.066]	0.343 [0.286, 0.400]	0.294 [0.236, 0.352]

Appendix

Table S8 : Relation-conditioned R@1, with the same cluster-robust intervals as the pairwise cells. How often the gold image is ranked first among the

21

candidates that all hold the two queried categories. Chance is exactly

1/21=0.048

, and native scoring sits on it. Restricted to the 4,812 horizontal and 1,110 vertical queries whose pool is full, so one chance level covers every row; the mixed-pool variant is in the text. A tie does not count as a win: the readout must score the gold image strictly above all

20

	$\beta$ multiplier					one-term $S_{T}$
Encoder	$\times 0.25$	$\times 0.5$	$\times 1$	$\times 2$	$\times 4$	$B$ -only	$A$ -only	native
CLIP ViT-B/16	0.677	0.768	0.839	0.872	0.883	0.695	0.697	0.510
CLIP LAION-2B	0.704	0.793	0.857	0.885	0.892	0.723	0.733	0.516
CLIP DataComp	0.677	0.771	0.844	0.879	0.890	0.708	0.721	0.512
MetaCLIP-B	0.648	0.731	0.808	0.853	0.868	0.650	0.655	0.513
DFN-B	0.678	0.769	0.837	0.870	0.880	0.713	0.722	0.512

Appendix

Table S9 : The repair does not need

\beta

to be right, and it does need both object terms. Pairwise accuracy, clean horizontal split. Multipliers apply to the label-free

\hat{\beta}

, so

\times 1

is the printed setting and the columns span a factor of

16

B

-only and

A

-only keep one term of

S_{T}=t(B)-t(A)

; no multiplier rescues either. The last row is why

\hat{\beta}

is not simply too small: R@1 peaks there while the relational contrast keeps rising.

Encoder	$\hat{\beta}$	$\hat{\beta}$	ratio	$\hat{\beta}$	ratio
	COCO	CC3M		Open Images
CLIP ViT-B/16	0.79	0.75	$0.95\times$	0.68	$0.86\times$
CLIP LAION-2B	1.30	1.18	$0.91\times$	1.07	$0.82\times$
CLIP DataComp	1.01	1.05	$1.04\times$	0.99	$0.98\times$
MetaCLIP-B	0.79	0.83	$1.05\times$	0.70	$0.88\times$
DFN-B	1.03	1.19	$1.16\times$	1.01	$0.99\times$

Appendix

Table S10: The calibration corpus does not matter.

\hat{\beta}

re-estimated by the same unlabelled rule on two corpora that are not COCO. Every value is within

0.82

–

1.33\times

the COCO one, no further than five disjoint

400

-image subsamples of a single corpus move it (

1.06

–

1.28\times

). Open Images ships no captions, so its caption side is CC3M alt-text.

	left/right			above/below
Encoder	templ.	free	parsed	templ.	free	parsed
CLIP ViT-B/16	0.839	0.839	0.610	0.806	0.806	0.663
CLIP LAION-2B	0.857	0.857	0.616	0.820	0.823	0.675
CLIP DataComp	0.844	0.846	0.611	0.809	0.807	0.665
MetaCLIP-B	0.808	0.811	0.602	0.781	0.784	0.647
DFN-B	0.837	0.840	0.616	0.827	0.828	0.666

Appendix

Table S11: A free-form query costs nothing; recovering the triple from it costs everything. Three arms on the clean split. templ. is the printed protocol. free replaces the caption with one Qwen2.5-7B-Instruct wrote in its own words, objects still given, and differs by

-0.0014

+0.0034

across all twenty cells. parsed takes the triple from a vocabulary-free parser run on that sentence, with unparsed queries scored natively rather than dropped.

	left/right		above/below
Encoder	native	+moment	native	+moment
CLIP ViT-B/16	0.498	0.807	0.504	0.783
CLIP LAION-2B	0.497	0.810	0.508	0.792
CLIP DataComp	0.498	0.803	0.510	0.789
MetaCLIP-B	0.497	0.762	0.508	0.750
DFN-B	0.498	0.799	0.513	0.788

Appendix

Table S12 : Pairwise retrieval on the pre-declared ambiguous splits, kept separate from the headline clean results. Horizontal: 28,147 queries; vertical: 22,782.

	trainable	WU zero-shot	VSR	ViewSpatial
	params	lateral	moment	moment
base, none	—	0.505	0.712	0.755
LoRA	0.86 M	0.515	0.699	0.722
NegCLIP	151 M	0.495	0.712	0.712
synthetic	151 M	0.485	0.717	0.736

Appendix

Table S13: Relational fine-tuning does not move the native score. All three start from the same ViT-B/32 checkpoint and are compared against that model; 256k supervised pairs, two of them full fine-tunes. Zero-shot lateral stays inside 0.485–0.515 while the zero-parameter spatial moment reads 0.70–0.76 before and after. At 2000 steps the claim is about this scale, not about supervision in general.

Bench.	Method	Cov.	Accuracy
WU	det. $\tau{=}0.02$	1.000	0.990 [0.971, 1.000]
	det. $\tau{=}0.05$	0.990	0.990 [0.971, 1.000]
	det. $\tau{=}0.1$	0.961	0.990 [0.970, 1.000]
	det. $\tau{=}0.2$	0.883	0.989 [0.967, 1.000]
	det. $\tau{=}0.3$	0.786	0.988 [0.963, 1.000]
	moment	1.000	0.990 [0.971, 1.000]

Appendix

Table S14: An open-vocabulary detector matches the readout in accuracy. It abstains when it cannot ground a named object, so it is reported as a coverage/accuracy curve over its threshold

\tau

, from one detection pass; the readout always answers. On VSR its higher numbers are bought with coverage. What separates them is that it cannot be precomputed into an index.

	native (paired), per seed	mean	probe( $z$ )
$\rho=0$	$0.398$ $0.417$ $0.485$	$0.434$	$0.736$
$\rho=1$	$0.471$ $0.485$ $0.529$	$0.495$	$0.708$

Appendix

Table S15: Counterfactual support does not produce native reachability. Native paired accuracy on What’s Up per seed and its mean, and a linear probe on the pooled vector

z

, with each anchor’s exact left/right counterfactual in its batch (

\rho=1

) and without (

\rho=0

). The rule fixed before the runs counted

\geq 0.70

as support and

\leq 0.55

as null.

Encoder	$\cos(w_{z^{-}}\!,w_{x})$	shuf. p95	mom $\to z^{-}$	$z^{-}\!\to$ mom
clip-datacomp	$-0.037$	$0.113$	$0.466$	$0.533$
dfn-b	$+0.036$	$0.117$	$0.505$	$0.612$
clip-laion2b	$+0.047$	$0.092$	$0.544$	$0.660$
clip-openai	$-0.050$	$0.141$	$0.447$	$0.471$
metaclip-b	$-0.080$	$0.103$	$0.447$	$0.243$
convnext-clip	$-0.092$	$0.170$	$0.451$	$0.353$

Appendix

Table S16 : The spatial moment and the pooled flip-odd probe read different directions. Col. 2: cosine between the two lateral readers; col. 3: the 95th percentile of that cosine when the labels are permuted; cols. 4–5: accuracy of a reader fitted on one representation and scored on the other, cross-fitted. Bold: at or beyond its own shuffled floor.

Captions	seeds	native	$\cos(t_{L},t_{R})$
$p=0$	3	0.503	0.998
$p=0.01$	3	0.494	0.794
$p=0.05$	3	0.508	0.973
$p=0.10$	3	0.510	0.956
$p=0.25$	3	0.502	0.983
$p=0.50$	3	0.481	0.984

Appendix

Table S17: In these from-scratch runs, relational supervision does not produce native reachability, even at

p=1

. Twenty-four CLIP-style encoders, three seeds per cell, trained from scratch on the same 373,528 images for the same

29.06

M samples seen, about

78

passes over the corpus;

p

changes captions only. Native lateral scoring stays at chance across the axis while the pooled vector stays decodable, and at

p=1

the two text embeddings are still almost collinear. The wrong-relation control fixes relation-word frequency and randomises truth.

	fixed	diverse	wrong rel.
$p=0.25$	$0.4903$	$0.4951$	$0.4984$
$p=1.0$	$0.4967$	$0.5049$	$0.5097$
$\cos(t_{\text{left}},t_{\text{right}})$ , $p{=}0.25$	$0.962$	$0.991$	$0.979$

Appendix

Table S18 : Phrasing and truth do not move native reachability. Native lateral accuracy, mean over three seeds, for fixed phrasing, diverse phrasing and wrong-relation captions at two relational frequencies

p

, and the text contrast

\cos(t_{\text{left}},t_{\text{right}})

p=0.25

image-side lateral direction	reads pairs	$\|\cos\|$ with real-pair dir.
fitted in What’s Up, cross-fitted	$0.660$ – $0.961$	$1$
What’s Up’s own mirror contrast	$0.621$ – $0.951$	$0.369$ – $0.752$
$C_{4}$ orbits, frequency-1	$0.456$ – $0.583$	$0.003$ – $0.112$
$C_{4}$ orbits, Fisher	$0.359$ – $0.660$	$0.004$ – $0.069$

Appendix

Table S19 : Three image-side lateral directions do not coincide. Accuracy of each direction on What’s Up photo pairs, and its

|\cos|

with the direction fitted on those pairs. Ranges over the ten encoders.

	zero-parameter		learned attention pool
Encoder	rule	$+$ probe	no coords	$+$ coords	$+$ coords
	(WU)	(WU)	(WU)	(in-dist.)	(WU)
CLIP ViT-B/16	0.772	0.777	0.510	0.997	0.553
CLIP LAION-2B	0.806	0.806	0.519	0.999	0.466
CLIP DataComp	0.694	0.699	0.519	0.998	0.534
MetaCLIP-B	0.772	0.752	0.515	0.998	0.709

Appendix

Table S20 : A learned pooling head solves the training distribution and does not transfer. Query-conditioned readouts trained for 20 epochs on 47,000 lateral items from the composited corpus, then read on What’s Up (WU), which is photographs and which no arm has seen. Pool columns take the best of

k\in\{1,4,16\}

queries, which favours the pool. Without coordinate channels it cannot express a spatial moment and sits at chance; with them it learns its own split and still does not transfer, except on MetaCLIP-B (

0.709

[0.660,0.757]

, one arm of thirty).

family	reads its own held-out instances	capable	chance
wu-vert	$1.000$	$10/10$	$0.500$
wu-mirror	$0.786$ – $0.985$	$10/10$	$0.500$
wu-real	$0.767$ – $0.990$	$10/10$	$0.500$
c4-axis	$0.998$ – $1.000$	$10/10$	$0.500$
c4-mirror	$0.480$ – $0.562$	$0/10$	$0.500$
c4-disp	$0.463$ – $0.587$	$0/10$	$0.500$

Appendix

Table S21 : Which stimulus families carry a signed lateral direction in the pooled vector. Each family’s own direction, cross-fitted, on its held-out instances (range over the ten encoders), and on how many encoders it is capable, meaning it clears

0.60

	Flickr30k test ( $1{,}000$ images)				COCO val2017 ( $5{,}000$ images)
	all captions, R@1		parsed, R@1		all captions, R@1		parsed, R@1
Encoder	native	$+$ moment	native	$+$ moment	native	$+$ moment	native	$+$ moment
CLIP ViT-B/16	0.592	0.587	0.638	0.514	0.302	0.295	0.305	0.205
CLIP LAION-2B	0.698	0.688	0.752	0.523	0.416	0.405	0.407	0.233
CLIP DataComp	0.676	0.665	0.725	0.486	0.398	0.388	0.404	0.250
MetaCLIP-B	0.707	0.702	0.748	0.619	0.414	0.406	0.409	0.295

Appendix

Table S22 : On standard caption retrieval the spatial term costs R@1 on the captions the parser fires on. Text-to-image R@1 with the rule-based parser in front of every caption and

\beta

estimated out of domain on CC3M. The parser returns a triple for

4.4\%

of Flickr and

6.5\%

of COCO captions, almost all vertical; the rest score exactly as native, so all captions differs from native only through the parsed ones.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 426 papers

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Sep 28, 2026Aditya Sharma, Divya SaxenaCross-Modal Retrieval Modality Gap in VLMs
Indian Institute of Technology Jodhpur
Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
Mar 13, 2026Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj +7Spatial Reasoning in Language Models Visual Spatial Reasoning
Cohere Labs Community · Eisai Inc. · Indian Institute of Science, Bangalore +4
When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
Oct 9, 2026Shubh Garg, Chen ShaniFaithfulness in VLMs VLM Evaluation
Thapar Institute of Engineering and Technology · Tel Aviv University

Sep 28, 2026J/K move · Enter open · S save

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Aditya Sharma, Divya Saxena

Indian Institute of Technology Jodhpur

Cross-Modal Retrieval Modality Gap in VLMs Vision-Language Alignment Contrastive Learning for VLMs

Open paper AskPDF arXiv

Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.

Abstract

Vision-Language Grounding

Explore similar work

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors