Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry

Woosang Jeon; Jiwon Yang; Soo Chung; Taehyeong Kim

cs.CVSep 29, 2026

Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry

Authors: Woosang Jeon, Jiwon Yang, Soo Chung, Taehyeong Kim

Organizations: Seoul National University

Abstract

Visual distinctions are often finer than those reflected in linguistic conceptualization. Vision-language models exhibit a similar asymmetry: a distinction can remain discriminable in frozen image geometry while being weakly addressable through the native text interface. We study this gap by separating visual discriminability from linguistic addressability in text-to-image retrieval. Using FactorAtlas, a fully crossed testbed of 23,040 images spanning shape, hue, pattern, and nuisance variation, we compare both readouts on held-out images of the same distinctions. We then derive image-side contrasts that separate each value from its alternatives for matched visual grounding, and test whether this reduces the native-text access gap across factors and models. Direction-specific and visual-absence controls tie these gains to the relevant visual contrast; the gains persist after global alignment and extend to compositional retrieval and natural images. Together, these results show that visual discriminability and linguistic addressability need not coincide, and that matched visual grounding can probe and reduce the resulting access gap.

Figures & tables

Figure 1: From visual discriminability to linguistic access. A visual distinction may remain discriminable in frozen image geometry yet weakly accessed by its native text query; matched visual grounding connects the two through a value-specific visual contrast.

Figure 2: FactorAtlas and its primary held-out protocol. The testbed factorially combines 8 shapes, 12 hues, and 10 surface patterns under balanced nuisance variation. Disjoint nuisance contexts are used for calibration, validation, and held-out testing, while the semantic factors remain fully crossed and balanced.

Figure 3: Factor-value top-1 readout gap. Visual discriminability exceeds native linguistic addressability across all factors.

Figure 4: Matched access gain across individual factor values. Native text retrieval and matched visual grounding are compared for each value of shape, hue, and pattern on held-out images.

Figure 5: Visual contrast matters. Matched contrast outperforms prototype-only and mismatched-direction controls.

Query form	Native	Matched
{v}	.629	.876
a {v} surface	.549	.874
an object with a {v} pattern	.577	.881
a surface that is {v}	.528	.874
a textured {v} surface	.501	.872

Table 1: Prompt robustness in the pattern setting. Matched grounding improves across all tested query forms.

Figure 6: Matched access gain under attenuated visual evidence. Matched access gain remains substantial as the pattern signal weakens and disappears when the visual distinction is removed.

Context holdout, per backbone
	Shape		Hue		Pattern
	mAP	Top-1	mAP	Top-1	mAP	Top-1
CLIP family
CLIP ViT-B/16	$.519\rightarrow\mathbf{.723}$	$.543\rightarrow\mathbf{.690}$	$.649\rightarrow\mathbf{.809}$	$.583\rightarrow\mathbf{.836}$	$.361\rightarrow\mathbf{.830}$	$.425\rightarrow\mathbf{.844}$
EVA02-B/16	$.563\rightarrow\mathbf{.804}$	$.645\rightarrow\mathbf{.767}$	$.586\rightarrow\mathbf{.796}$	$.656\rightarrow\mathbf{.841}$	$.515\rightarrow\mathbf{.843}$	$.544\rightarrow\mathbf{.884}$
FG-CLIP Base	$.599\rightarrow\mathbf{.740}$	$.585\rightarrow\mathbf{.706}$	$.678\rightarrow\mathbf{.892}$	$.720\rightarrow\mathbf{.901}$	$.555\rightarrow\mathbf{.848}$	$.496\rightarrow\mathbf{.882}$

Table 2: Matched access gain across factors, backbones, and held-out protocols. Each cell reports native

\rightarrow

matched full-gallery mAP and factor-value top-1 accuracy. Per-backbone rows use the context holdout; the lower block reports seven-backbone macro results across additional protocols.

Figure 7: Global alignment leaves residual access gaps. ( Left ) Factor-level macro-mAP under the context holdout. ( Right ) Factor-global alignment before and after matched grounding across seven backbones, with native-space matched grounding shown for reference.

Figure 8: Factor-level access gains support exact compositional retrieval. Native and matched queries are compared under the context holdout and three unseen-combination protocols. Matched visual grounding improves exact R@1, R@5, and mAP across all held-out conditions. H

\times

S, P

\times

S, and P

\times

H denote held-out hue–shape, pattern–shape, and pattern–hue combinations.

Policy	Test mAP
Native	.6797
Ground every value	.8220
Validation-guided	.8245

Table 3: Validation-guided selective grounding. Test mAP across the context and unseen-combination protocols.

Dataset	Target distinction	Held-out setting
DTD ( Cimpoi et al., 2014 )	Texture	Official partitions
Fashionpedia ( Jia et al., 2020 )	Pattern / garment length	Category-held-out / image-disjoint
COCO-Facet ( Li et al., 2026 )	Material	Image-disjoint
UT-Zappos ( Yu and Grauman, 2014 )	Material	Product-disjoint

Table 4: Natural-image transfer settings. We evaluate texture, garment pattern and length, and material distinctions using dataset-specific splits that separate evaluation examples from those used to estimate and tune the grounding.

Figure 9: Matched Access Gain beyond FactorAtlas . Seven-backbone macro mAP improves across natural texture, garment, and material distinctions.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

SigLIP2 Base
	Pattern	Hue	Shape
Image prototype	.919	.909	.825
Native text	.676	.603	.692
Seven-backbone macro
Image prototype	.907	.835	.815
Native text	.598	.599	.678

Appendix

Table 5: Intervention-free factor-value top-1.

Backbone	Pattern	Hue	Shape	$\lambda_{\mathrm{P/H/S}}$
CLIP ViT-B/16	$.361\rightarrow.568\rightarrow\mathbf{.830}$	$.649\rightarrow.656\rightarrow\mathbf{.809}$	$.519\rightarrow.584\rightarrow\mathbf{.723}$	$.713/.163/.500$
EVA02-B/16	$.515\rightarrow.643\rightarrow\mathbf{.843}$	$.586\rightarrow.595\rightarrow\mathbf{.796}$	$.563\rightarrow.649\rightarrow\mathbf{.804}$	$.525/.150/.488$
FG-CLIP Base	$.555\rightarrow.689\rightarrow\mathbf{.848}$	$.678\rightarrow.708\rightarrow\mathbf{.892}$	$.599\rightarrow.653\rightarrow\mathbf{.740}$	$.700/.363/.500$
SigLIP SO400M	$.663\rightarrow.745\rightarrow\mathbf{.891}$	$.667\rightarrow.683\rightarrow\mathbf{.825}$	$.730\rightarrow.765\rightarrow\mathbf{.850}$	$.613/.175/.538$
SigLIP2 Base	$.635\rightarrow.716\rightarrow\mathbf{.878}$	$.678\rightarrow.688\rightarrow\mathbf{.854}$	$.702\rightarrow.726\rightarrow\mathbf{.813}$	$.425/.100/.400$
SigLIP2 Large	$.620\rightarrow.732\rightarrow\mathbf{.875}$	$.590\rightarrow.597\rightarrow\mathbf{.736}$	$.744\rightarrow.756\rightarrow\mathbf{.837}$	$.450/.113/.213$

Appendix

Table 6: Target attraction versus target-versus-rest visual contrast. Each factor cell reports native

\rightarrow

target-prototype attraction

\rightarrow

target-versus-rest grounding mAP under the primary context holdout. The final column reports the validation-selected attraction strengths for pattern, hue, and shape.

ID	Pattern	Hue	Shape
T1	{v}	{v}	{v}
T2	a {v} surface	a {v} object	a {v}
T3	an object with a {v} pattern	an object that is {v}	an object shaped like a {v}
T4	a surface that is {v}	a {v} colored object	a geometric {v}
T5	a textured {v} surface	a {v} surface	a three dimensional {v} object

Appendix

Table 7: Fixed factor-specific prompt templates. The placeholder {v} is replaced by every value in the corresponding factor vocabulary.

Template	Pattern	Hue	Shape
T1	$.629\rightarrow.876$	$.657\rightarrow.846$	$.704\rightarrow.813$
T2	$.549\rightarrow.874$	$.636\rightarrow.853$	$.696\rightarrow.813$
T3	$.577\rightarrow.881$	$.646\rightarrow.853$	$.685\rightarrow.813$
T4	$.528\rightarrow.874$	$.656\rightarrow.862$	$.642\rightarrow.813$
T5	$.501\rightarrow.872$	$.585\rightarrow.846$	$.659\rightarrow.813$
Improved	$5/5$	$5/5$	$5/5$

Appendix

Table 8: Prompt robustness on SigLIP2 Base. Each entry reports native

\rightarrow

matched test mAP.

Value	Additional lexical forms	Canonical AP	Lexical AP
Plain	solid, unpatterned	.287	.109
Striped	banded, lined	.375	.367
Wavy	undulating, wave patterned	.666	.558
Checkered	checkerboard, chequered	.999	.976
Dotted	polka dotted, dot patterned	.711	.490
Cracked	fissured, crackle patterned	.936	.744

Appendix

Table 9: Pattern lexical-substitution control. The canonical native ensemble is compared with an equal-weight ensemble of three fixed lexical forms.

$\gamma$	Image query	Native	Matched	Cosine
1.00	.649	.635	.878	1.000
.70	.609	.611	.863	.995
.50	.567	.590	.845	.981
.35	.518	.566	.824	.962
.20	.423	.519	.774	.920
.10	.269	.416	.659	.825

Appendix

Table 10: Pattern attenuation on SigLIP2 Base. Performance is mAP; cosine is measured against the full-signal directions.

Figure 10: Finite grounding and the pure visual endpoint under limited calibration support. Top: test mAP for direction-only retrieval and validation-selected finite grounding. Bottom: corresponding selected grounding strength

\alpha

. Points and shading show mean

\pm 1

standard deviation over 12 balanced calibration resamples; the dashed line denotes native-text retrieval.

	Pattern		Hue		Shape
Backbone	mAP	Top-1	mAP	Top-1	mAP	Top-1
CLIP ViT-B/16	$.371\rightarrow\mathbf{.838}$	$.434\rightarrow\mathbf{.856}$	$.650\rightarrow\mathbf{.820}$	$.579\rightarrow\mathbf{.856}$	$.520\rightarrow\mathbf{.729}$	$.549\rightarrow\mathbf{.707}$
EVA02-B/16	$.533\rightarrow\mathbf{.856}$	$.562\rightarrow\mathbf{.888}$	$.578\rightarrow\mathbf{.816}$	$.645\rightarrow\mathbf{.841}$	$.579\rightarrow\mathbf{.817}$	$.660\rightarrow\mathbf{.787}$
FG-CLIP Base	$.587\rightarrow\mathbf{.864}$	$.529\rightarrow\mathbf{.910}$	$.698\rightarrow\mathbf{.906}$	$.710\rightarrow\mathbf{.950}$	$.606\rightarrow\mathbf{.746}$	$.602\rightarrow\mathbf{.736}$
SigLIP SO400M	$.691\rightarrow\mathbf{.908}$	$.739\rightarrow\mathbf{.921}$	$.662\rightarrow\mathbf{.838}$	$.613\rightarrow\mathbf{.908}$	$.735\rightarrow\mathbf{.862}$	$.756\rightarrow\mathbf{.847}$
SigLIP2 Base	$.673\rightarrow\mathbf{.895}$	$.709\rightarrow\mathbf{.924}$	$.672\rightarrow\mathbf{.872}$	$.601\rightarrow\mathbf{.910}$	$.710\rightarrow\mathbf{.832}$	$.695\rightarrow\mathbf{.810}$

Appendix

Table 11: Per-backbone matched access gains on unseen factor combinations. Each entry reports native

\rightarrow

matched .

Factor	mAP: native $\rightarrow$ matched	Top-1: native $\rightarrow$ matched	Positive endpoints
Pattern	$.5701{\pm}.0014\rightarrow\mathbf{.8618{\pm}.0005}$	$.6031{\pm}.0019\rightarrow\mathbf{.8977{\pm}.0008}$	168/168
Hue	$.6207{\pm}.0026\rightarrow\mathbf{.7877{\pm}.0032}$	$.5961{\pm}.0019\rightarrow\mathbf{.8182{\pm}.0040}$	168/168
Shape	$.6630{\pm}.0026\rightarrow\mathbf{.8093{\pm}.0030}$	$.6871{\pm}.0031\rightarrow\mathbf{.7905{\pm}.0049}$	168/168

Appendix

Table 12: Matched access gains across 24 balanced nuisance partitions. Values are mean

\pm

standard deviation across folds after averaging the seven backbones within each fold.

Pattern			Hue			Shape
Value	Native $\rightarrow$ Matched	$\Delta$	Value	Native $\rightarrow$ Matched	$\Delta$	Value	Native $\rightarrow$ Matched	$\Delta$
plain	$.227\rightarrow.770$	+.542	red	$.825\rightarrow.898$	+.072	cube	$.551\rightarrow.823$	+.271
striped	$.372\rightarrow.854$	+.482	orange	$.789\rightarrow.923$	+.133	sphere	$.895\rightarrow.978$	+.083
wavy	$.592\rightarrow.763$	+.171	yellow	$.767\rightarrow.886$	+.119	cylinder	$.832\rightarrow.908$	+.076
checkered	$.975\rightarrow.998$	+.023	lime	$.669\rightarrow.878$	+.209	cone	$.753\rightarrow.915$	+.161
dotted	$.643\rightarrow.884$	+.241	green	$.623\rightarrow.842$	+.218	torus	$.885\rightarrow.999$	+.114

Appendix

Table 13: Value-resolved AP under the primary context holdout. Each entry reports the seven-backbone macro native

\rightarrow

matched AP and its change. Access gains are broad but value-dependent; teal is the only value with a negative macro change in this protocol.

Factor	Native	Shared	Factor	Matched
Pattern	.567	.779	.806	.862
Hue	.619	.528	.580	.786
Shape	.660	.739	.762	.803
Macro	.616	.682	.716	.817

Appendix

Table 14: Primary-context global comparison. Seven-backbone macro-mAP.

Method	Pattern	Hue	Shape
Native	.635	.678	.702
Factor-specific map	.772	.727	.789
Shared map	.775	.712	.768
Matched grounding	.878	.854	.813

Appendix

Table 15: Additional whole-vocabulary maps on SigLIP2 Base. Values are primary-context test mAP. Validation selects Procrustes for all map scopes in this split.

	Shared global			Factor global
Backbone	Global	+Matched	Improved	Global	+Matched	Improved
CLIP B/16	.680	.743	11/12	.735	.765	10/12
EVA02-B/16	.725	.794	10/12	.738	.791	10/12
FG-CLIP Base	.708	.785	10/12	.753	.797	8/12
SigLIP SO400M	.764	.823	9/12	.775	.827	9/12
SigLIP2 Base	.740	.813	11/12	.753	.816	11/12

Appendix

Table 16: Residual access gains after global alignment. Macro-mAP averages context and three unseen-combination protocols.

Backbone	Pattern	Hue	Shape
CLIP ViT-B/16	$.361\rightarrow.800\rightarrow\mathbf{.830}$	$.649\rightarrow.703\rightarrow\mathbf{.809}$	$.519\rightarrow.677\rightarrow\mathbf{.723}$
EVA02-B/16	$.515\rightarrow.797\rightarrow\mathbf{.843}$	$.586\rightarrow.644\rightarrow\mathbf{.796}$	$.563\rightarrow.740\rightarrow\mathbf{.804}$
FG-CLIP Base	$.555\rightarrow.827\rightarrow\mathbf{.848}$	$.678\rightarrow.767\rightarrow\mathbf{.892}$	$.599\rightarrow.688\rightarrow\mathbf{.740}$
SigLIP SO400M	$.663\rightarrow.847\rightarrow\mathbf{.891}$	$.667\rightarrow.753\rightarrow\mathbf{.825}$	$.730\rightarrow.827\rightarrow\mathbf{.850}$
SigLIP2 Base	$.635\rightarrow.841\rightarrow\mathbf{.878}$	$.678\rightarrow.723\rightarrow\mathbf{.854}$	$.702\rightarrow.760\rightarrow\mathbf{.813}$
SigLIP2 Large	$.620\rightarrow.822\rightarrow\mathbf{.875}$	$.590\rightarrow.657\rightarrow\mathbf{.736}$	$.744\rightarrow.786\rightarrow\mathbf{.837}$

Appendix

Table 17: Full-support cache adaptation across backbones and factors. Each factor cell reports native

\rightarrow

Tip-Adapter-style cache

\rightarrow

target-versus-rest grounding full-gallery mAP under the primary context holdout. All hyperparameters are selected on validation and fixed before test evaluation.

	Native			Matched
Condition	R@1	R@5	mAP	R@1	R@5	mAP
Context	.357	.714	.517	.711	.962	.819
Unseen H $\times$ S	.631	.929	.764	.928	.998	.960
Unseen P $\times$ S	.622	.952	.766	.876	.995	.931
Unseen P $\times$ H	.661	.937	.779	.828	.990	.900

Appendix

Table 18: Exact composition. Seven-backbone macro results.

Protocol	Factor	$\alpha$	Ground/keep	Native	All	Selective	Improved/declined
Context	Pattern	.70	10/0	.635	.878	.878	10/0
	Hue	.65	11/1	.678	.854	.855	11/0
	Shape	2.90	7/1	.702	.813	.813	7/0
Unseen H $\times$ S	Pattern	.70	9/1	.673	.895	.895	9/0
	Hue	.10	7/5	.671	.698	.697	6/1
	Shape	7.35	7/1	.721	.793	.809	7/0

Appendix

Table 19: Selective grounding on SigLIP2 Base.

Images/value	1	2	4	8	16	32	64	128	256	Full
Test mAP	.695	.711	.730	.755	.774	.795	.806	.812	.819	.822
Gain	+.015	+.032	+.051	+.076	+.095	+.115	+.127	+.133	+.139	+.142
Direction cosine	.466	.594	.719	.825	.897	.946	.972	.986	.994	1.000

Appendix

Table 20: Matched access gains with increasing labeled support. Finite-budget results average 12 calibration resamples.

Factor	1	8	32	256	Full
Pattern	.688	.802	.860	.884	.887
Hue	.670	.698	.731	.769	.776
Shape	.728	.767	.793	.805	.803

Appendix

Table 21: Factor-resolved support-size subset. Entries are test mAP.

Backbone	DTD	Fashionpedia		COCO-Facet	UT-Zappos
	Texture	Pattern	Length	Material	Material
CLIP ViT-B/16	.359 $\rightarrow$ .668	.452 $\rightarrow$ .618	.178 $\rightarrow$ .317	.604 $\rightarrow$ .850	.522 $\rightarrow$ .617
EVA02-B/16	.444 $\rightarrow$ .731	.567 $\rightarrow$ .768	.220 $\rightarrow$ .361	.668 $\rightarrow$ .819	.665 $\rightarrow$ .665
FG-CLIP Base	.500 $\rightarrow$ .751	.566 $\rightarrow$ .789	.241 $\rightarrow$ .352	.601 $\rightarrow$ .899	.631 $\rightarrow$ .676
SigLIP SO400M	.621 $\rightarrow$ .803	.631 $\rightarrow$ .799	.297 $\rightarrow$ .396	.615 $\rightarrow$ .869	.677 $\rightarrow$ .751
SigLIP2 Base	.564 $\rightarrow$ .771	.620 $\rightarrow$ .789	.285 $\rightarrow$ .370	.627 $\rightarrow$ .874	.678 $\rightarrow$ .717

Appendix

Table 22: Backbone-resolved natural-image transfer. Each cell reports native

\rightarrow

matched full-gallery macro-mAP. Fashionpedia pattern uses the garment-category holdout, and COCO-Facet material uses the natural localized crop. Matched grounding improves all 28 backbone–setting endpoints in the four stable transfer settings; UT-Zappos is reported separately as a higher-variance boundary.

Backbone	Native	Reused direction	$\Delta$
CLIP ViT-B/16	.414	.490	+.075
EVA02-B/16	.575	.720	+.145
FG-CLIP Base	.568	.800	+.232
SigLIP SO400M	.640	.767	+.127
SigLIP2 Base	.631	.795	+.164
SigLIP2 Large	.628	.750	+.121

Appendix

Table 23: Cross-dataset reuse of shared pattern distinctions. For each backbone, FactorAtlas-derived pattern directions and its fixed FactorAtlas grounding strength are reused without Fashionpedia fitting or validation. Values are four-value macro-mAP averaged over the four Fashionpedia category-held-out partitions.

Explore similar work

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Sep 28, 2026Aditya Sharma, Divya SaxenaModality Gap Cross-Modal
Indian Institute of Technology Jodhpur
Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
May 26, 2026Yifan Jiang, Ruoxi Ning, Sheng Yao +1Representational Collapse Relevance
University of Waterloo · Vector Institute
Mirage Probes: How Vision Models Fake Visual Understanding
Jun 11, 2026Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5Weak Visual Grounding Vision Foundation Models
Columbia University · Intuit · Technion +2

Sep 28, 2026, cs.CVJ/K move · Enter open · S save

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Aditya Sharma, Divya Saxena

Indian Institute of Technology Jodhpur

Modality Gap Cross-Modal Readout

Open paper AskPDF arXiv

Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.

Abstract

Explore similar work

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery

Mirage Probes: How Vision Models Fake Visual Understanding