cs.CLOct 9, 2026

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Authors: Shubh Garg, Chen Shani

Organizations: Thapar Institute of Engineering and Technology · Tel Aviv University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. We introduce EICS (Evaluation-Invariant Concept Stability), a metric for testing whether representations consistently encode concepts under controlled one-concept interventions, and CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models), a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across multiple VLM architectures, we find that spatial relations are represented substantially less faithfully than lexical information, even when the same models score well above chance on compositional benchmarks. Further analysis shows that the relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation, whether through the (EOS) token or by mean pooling. These findings reveal a systematic failure invisible to output-based evaluation: VLMs do not preserve, in the representation used downstream, relational information that is present in their own input.

Figures & tables

BenchmarkEditLevelLexicalLayers
AROword swapsoutput×\times×\times
SugarCrepeLLM negativesoutput✓×\times
Winogroundword orderoutput×\times×\times
What’sUpobject layoutoutput×\times×\times
Oursone conceptrepr.✓✓
Table 1: Prior compositional benchmarks score model outputs; EICS probes the representation, against a lexical baseline, at every layer. ARO ( Yuksekgonul et al., 2023 ) , SugarCrepe ( Hsieh et al., 2023 ) , Winoground ( Thrush et al., 2022 ) , What’sUp ( Kamath et al., 2023 ) ; Ours = CASP-VL + EICS. Level : what is evaluated (model output or internal representation). Lexical : a baseline that sees only the words (text-only scorers for SugarCrepe, the BoW probe for ours). Layers : results reported per layer.
Figure 1: EICS asks whether a representation follows a one-concept change in its input. (I) We edit one concept in a Visual Genome scene graph GG , giving G~\tilde{G} . (II) Both graphs are rendered as captions xx and x~\tilde{x} from one template. (III) The text encoder maps each caption to a hidden state. (IV) A linear probe trained only on original captions (V) is tested on the perturbed ones: EICS is how often it predicts the new label, minus chance. The same probe on token counts (dashed) gives the BoW lexical baseline.
Figure 2: A color flip moves the representation across the probe’s decision boundary; a spatial flip does not. Schematic. Left: flipping “red” to “blue” moves the representation into the blue cluster, so a probe trained on original captions predicts the new label (high EICS). Right: flipping “on” to “beneath” leaves the representation inside the “on” cluster, so the probe still predicts the old label (near-zero EICS).
Figure 3: Color becomes decodable in the last two layers; spatial relations stay far below the lexical baseline at every layer. EICS by layer for CLIP ViT-B/32 (12 color classes, 16 relation classes), averaged over five image-grouped splits; bands span the lowest and highest split. Dashed and dotted lines: the BoW baseline for each concept, averaged over the same splits.
CLIP B/32, L12Color (12)Spatial (16)
[EOS]++ 0.312 ±\pm .019++ 0.070 ±\pm .012
Mean pooling++ 0.309 ±\pm .025++ 0.093 ±\pm .020
Predicate tokens++ 0.917 ±\pm .000++ 0.930 ±\pm .004
BoW++ 0.332 ±\pm .016++ 0.372 ±\pm .022
Orig. acc, CLIP/BoW0.65 / 0.660.24 / 0.63
Table 2: No sentence-level pooling keeps the spatial relation, while the predicate’s own tokens identify it almost perfectly. CLIP ViT-B/32 at L12, mean ±\pm s.d. over five image-grouped splits. Predicate tokens : mean over the edited word’s tokens; its ceiling is 1−1/∣C∣1-1/|\mathcal{C}| (0.917 for color, 0.938 for spatial). Orig. acc : probe accuracy on original captions. Number of classes in parentheses; every value has its 95% CI above zero on every split (split-42 CIs in Table 9 ).
 Accuracy (%)EICS
ModelAROSC RelSC SwapColorSpatial
Chance50.050.050.00.0000.000
BoWn/an/an/a++ 0.359*++ 0.359*
CLIP B/3251.869.361.5++ 0.292*++ 0.066*
CLIP L/1453.365.361.1++ 0.303*++ 0.053*
SigLIP B/1646.370.660.3++ 0.339*++ 0.038*
Table 3: Benchmark accuracy does not predict representational faithfulness: SigLIP has the best SugarCrepe relation accuracy and the worst spatial EICS. ARO : VG-Relation ( Yuksekgonul et al., 2023 ) ; SC Rel / SC Swap : SugarCrepe replace_rel / swap_obj ( Hsieh et al., 2023 ) . All three are two-way choices. EICS is measured on CASP-VL under the main protocol (split 42; Section 4 , Setup), at the final layer of each model: L12 of the CLIP and SigLIP text encoders, and L31 of LLaVA’s language model at the last prompt token (Appendix M ); all ARO and SugarCrepe subtasks are in Appendix C . We computed all accuracies with a single script, scoring each caption by its image–text similarity, rather than copying them from prior papers. We report accuracy over all examples; this is why our CLIP ViT-B/32 ARO score differs from the 59.0% of Yuksekgonul et al. (2023) , which is macro-averaged over relations. For reference, Hsieh et al. (2023) report 97.0% / 99.0% for humans and at most 50.0% for text-only models on SC Rel / SC Swap. BoW is shown for the CLIP split; each model’s own BoW is in Table 6 . * = 95% CI above zero; ‡\ddagger = generative model, contrastive scoring not applicable; n/a = not applicable.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

FlipEmbedding layerL6L12
Color1.0000.9970.943
Relation0.9960.9960.992
Appendix
Table 4: Relation flips barely change CLIP’s text representation, even at the last layer. Mean cosine similarity between original and perturbed [EOS] states (CLIP ViT-B/32). Values near 1.0 mean the representation barely responds to the edit.
BenchmarkSubtaskCLIP B/32CLIP L/14SigLIP B/16
AROVG-Relation51.853.346.3
VG-Attribution61.461.654.0
SugarCrepeadd_att69.371.283.7
replace_att80.579.187.2
swap_att63.961.971.6
add_obj77.078.589.1
Appendix
Table 5: SigLIP beats CLIP ViT-B/32 by 5–14 points on every attribute and object subtask, but not on the two relational ones ( ++ 1.3 on replace_rel , −- 1.2 on swap_obj ). Accuracy (%) on every ARO and SugarCrepe subtask (two-way choice, chance =50%=50\% ). Shaded rows are the relational subtasks used in Table 3 .
ModelColor EICS [95% CI]BoWSpatial EICS [95% CI]BoW
CLIP ViT-B/32++ 0.292 [ ++ 0.251, ++ 0.336]++ 0.359++ 0.066 [ ++ 0.024, ++ 0.107]++ 0.359
CLIP ViT-L/14++ 0.303 [ ++ 0.259, ++ 0.346]++ 0.359++ 0.053 [ ++ 0.016, ++ 0.090]++ 0.359
SigLIP-B/16++ 0.339 [ ++ 0.296, ++ 0.383]++ 0.323++ 0.038 [ ++ 0.000, ++ 0.080]++ 0.331
LLaVA-1.5-7B (L31)++ 0.143 [ ++ 0.107, ++ 0.182]++ 0.359++ 0.115 [ ++ 0.070, ++ 0.165]++ 0.359
Appendix
Table 6: The color–spatial gap is clear for the three contrastive models; LLaVA keeps little of either concept. EICS with 95% bootstrap CIs at the layers of Table 3 , main protocol (split 42), with each model’s BoW on the same split. SigLIP’s 64-token limit removes a few triplets, so its split differs slightly (483 color, 239 spatial test triplets; 482 and 242 otherwise).
 ColorRelation
Triplets2,4001,491
Images1,9641,265
Triplets per image (mean/max)1.22 / 51.18 / 5
Original classes1216
Classes reached by edits815
Captions with 5 sentences96.9%96.0%
Appendix
Table 7: CASP-VL at a glance. Computed from the released triplet file with dataset_stats.py .
Color flips (2,400)Relation flips (1,491)
pink →\to red 106, grey →\to white 106, purple →\to blue 106, gray →\to black 105, brown →\to black 104, white →\to gray 104, black →\to white 102, orange →\to yellow 101, orange →\to red 99, black →\to brown 98, white →\to black 96, brown →\to white 96, gray →\to white 95, pink →\to white 94, grey →\to black 94, purple →\to green 94, yellow →\to red 82, red →\to green 78, green →\to red 73, blue →\to yellow 69, green →\to yellow 68, blue →\to green 66, blue →\to red 65, red →\to yellow 65, yellow →\to green 63, green →\to blue 59, red →\to blue 57, yellow →\to blue 55near →\to far from 100, in front of →\to behind 100, inside →\to outside 100, outside →\to inside 100, beside →\to away from 100, next to →\to far from 100, behind →\to in front of 100, left of →\to right of 63, on top of →\to beneath 55, below →\to over 55, beneath →\to on top of 54, above →\to under 52, over →\to under 51, over →\to below 49, above →\to below 48, beneath →\to on 46, below →\to above 45, on top of →\to under 45, under →\to above 39, on →\to under 38, under →\to over 36, on →\to below 32, on →\to beneath 30, right of →\to left of 28, under →\to on 25
Appendix
Table 8: Every edit in CASP-VL, with counts. Replacements are drawn from a fixed list of alternatives per original label.
CLIP B/32, L12, split 42Color (12)Spatial (16)
[EOS]++ 0.292 [ ++ 0.251, ++ 0.336]*++ 0.066 [ ++ 0.024, ++ 0.107]*
Mean pooling++ 0.328 [ ++ 0.282, ++ 0.371]*++ 0.095 [ ++ 0.049, ++ 0.136]*
Predicate tokens++ 0.917 [ ++ 0.917, ++ 0.917]*++ 0.933 [ ++ 0.925, ++ 0.938]*
BoW++ 0.359 [ ++ 0.315, ++ 0.400]*++ 0.359 [ ++ 0.297, ++ 0.421]*
Orig. acc, CLIP/BoW0.65 / 0.660.21 / 0.64
Appendix
Table 9: Split 42 of Table 2 , with confidence intervals. EICS with 95% bootstrap CIs over the 482 color and 242 spatial held-out triplets (* = CI above zero); number of classes in parentheses. Orig. acc : probe accuracy on original captions. Every single-split analysis in the appendix uses this split.
 ColorSpatial
SeednnCLIPBoWKeptnnCLIPBoWKept
42482++ 0.292++ 0.35981%242++ 0.066++ 0.35918%
1498++ 0.292++ 0.31692%227++ 0.070++ 0.40017%
2498++ 0.316++ 0.32498%218++ 0.070++ 0.36419%
3487++ 0.327++ 0.33498%234++ 0.087++ 0.39122%
4484++ 0.334++ 0.328102%247++ 0.055++ 0.34616%
Appendix
Table 10: The color–spatial gap holds on every split. CLIP ViT-B/32, L12 [EOS] , main protocol, one row per split seed. All EICS values have their 95% CI above zero. Kept : CLIP EICS as a share of BoW EICS. Spatial test sets differ in size because unscorable flips are dropped per split.
new label y~\tilde{y}blackbluebrowngraygreengreyorangepinkpurpleredwhiteyellownnnew %old %other %
black2065211141247366301753
blue228400012563253531136
brown51301010022116193150
gray43262501205131191665
green0611250113421155452233
red2443601082415489462430
Appendix
Table 11: Color: most perturbed captions move toward the new color, and errors spread over neighboring colors. Confusion of the L12 [EOS] probe on perturbed captions (CLIP ViT-B/32, main protocol, seed 42). Rows: the new label y~\tilde{y} ; columns: predicted class; cells: counts, shaded by share of the row (green = correct, red = error). new/old/other % : share predicted as the new label, the original label, or anything else.
new label y~\tilde{y}abovebehindbelowbeneathbesidein front ofinsideleft ofnearnext toonon top ofoutsideoverright ofundernnnew %old %other %
above411400000032120119211168
behind160023100110510021291457
below3204104020115602310694
beneath02002010110421021601288
in front of030004302120210119211663
inside110011300012600117183547
Appendix
Table 12: Spatial: there is no diagonal; the probe rarely reaches the new relation and mostly ignores the edit. Same layout as Table 11 . Only relations that occur as a scorable new label in the test split are shown as rows; columns are the 16 classes of the probe.
ConceptOld label in x~\tilde{x} ?nnCLIP L12BoWold %
Main run, test set split
Colorno304++ 0.433 2
 yes178++ 0.051 51
Spatialno160++ 0.062 9
 yes82++ 0.072 26
Retrained on leftover-free data
Appendix
Table 13: Without leftover old labels, CLIP’s color EICS exceeds BoW, while spatial EICS stays at a fifth of it. Top: test triplets of the main run (CLIP ViT-B/32, L12 [EOS] ) split by whether the old label still occurs in the perturbed caption (point estimates; old % : share predicted as the old label). Bottom: probe and BoW retrained and scored on the leftover-free data only, with 95% bootstrap CIs.
 ColorSpatial
LayerLinearMLPLinearMLP
L00−- 0.009−- 0.021−- 0.001++ 0.020
L02++ 0.039++ 0.099++ 0.045++ 0.214
L04++ 0.056++ 0.120++ 0.111++ 0.177
L06++ 0.025++ 0.074++ 0.107++ 0.165
L08++ 0.035++ 0.049++ 0.070++ 0.111
Appendix
Table 14: An MLP probe raises early-layer spatial EICS, but at no layer does it reach the linear BoW baseline. EICS by layer for a linear and an MLP probe (CLIP ViT-B/32, main protocol, seed 42, [EOS] ). Bottom row: BoW with each probe.
InterventionΔ\Delta mean [95% CI]Img ≻\succ orig
Color flip++ 0.0040 [+0.003,+0.005][+0.003,+0.005]61.6%
Size flip++ 0.0057 [+0.001,+0.010][+0.001,+0.010]55.9%
Spatial flip−- 0.0014 [−0.002,−0.001][-0.002,-0.001]34.0%
Appendix
Table 15: Adding the image helps for color and size but hurts for spatial relations: after a spatial flip, the image prefers the wrong caption 66% of the time. Image–text alignment delta (CIs are tt -intervals). “Img ≻\succ orig” is the fraction of triplets where the image prefers the original caption.
TransitionFlip rateBaselineCorrected95% CI
spatial →\to part-whole0.0180.932−- 0.315[ −- 0.341, −- 0.289]
spatial →\to action0.0870.932−- 0.246[ −- 0.276, −- 0.216]
part-whole →\to spatial0.8080.675++ 0.133[ ++ 0.094, ++ 0.172]
action →\to spatial0.4380.278++ 0.160[ ++ 0.112, ++ 0.208]
Appendix
Table 16: Swapping another relation into a spatial one is detectable, but swapping a spatial relation out is not: spatial predicates dominate the geometry. Cross-category EICS at L12 ( n=4,200n=4{,}200 ). Flip rate : fraction of perturbed captions on which the probe predicts the new (perturbed) label; EICS subtracts chance ( 1/∣C∣=0.3331/|C|=0.333 ). Baseline : probe’s spatial prediction rate on unperturbed source-class captions. Corrected : for edits into the spatial class, flip rate minus this baseline; for edits out of it, flip rate minus chance (1/3).
StyleConceptnnCLIP L12BoWCLIP/BoW
TemplateColor482++ 0.292*++ 0.359*81%
 Spatial242++ 0.066*++ 0.359*18%
VG regionsColor448++ 0.890*++ 0.917*97%
 Spatial184++ 0.291*++ 0.938*31%
LLM paraphraseColor300++ 0.370*++ 0.427*87%
 Spatial110++ 0.065*++ 0.438*15%
Appendix
Table 17: Under every caption format, CLIP keeps nearly all of the color signal but loses much of the spatial signal; one sentence per caption narrows the spatial loss without closing it. EICS at L12 ( [EOS] ) under the main protocol; nn = test triplets. BoW reaches its ceiling ( 1−1/∣C∣1-1/|C| ) on VG regions and single spatial sentences because each names exactly one label of its kind. * = 95% bootstrap CI above zero.
ModelWidthColor EICS [CI]Spatial EICS [CI]
ViT-B/32512++ 0.292 [+0.251, +0.336]*++ 0.066 [+0.024, +0.107]*
ViT-L/14768++ 0.303 [+0.259, +0.346]*++ 0.053 [+0.016, +0.090]*
ViT-L/14@336768++ 0.298 [+0.257, +0.344]*++ 0.045 [+0.008, +0.086]*
BoW—++ 0.359 [+0.315, +0.400]*++ 0.359 [+0.297, +0.421]*
Appendix
Table 18: A wider text encoder does not raise spatial EICS, so the failure is not a capacity limit. EICS at L12 [EOS] under the main protocol (482 color, 242 spatial test triplets; the BoW baseline does not depend on the model). * = 95% CI above zero.

Explore similar work

CardsList

  1. Same Answer, Different Representations: Hidden instability in VLMs

    Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM EvaluationVLM Robustness

    Sapienza University of Rome · University of Edinburgh · University of Oxford +5

  2. Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

    May 28, 2026Cheolhong Min, Jaeyun Jung, Daeun Lee +5VLM EvaluationSpatial Reasoning Benchmarks

    Seoul National University · The Ohio State University · NVIDIA

  3. Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?

    Aug 24, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Visual Spatial ReasoningVision-Language Grounding

    Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · University Of Liverpool