GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

Seulgi Kim; Zhixiong Zhang; Xinwei Zhang; Jie Ling; Ronn Shaw

cs.CVOct 6, 2026

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

Authors: Seulgi Kim, Zhixiong Zhang, Xinwei Zhang, Jie Ling, Ronn Shaw

Organizations: Georgia Institute of Technology · Work done as an intern at Amazon. · Amazon

Abstract

While recent vision-language models (VLMs) have shown outstanding performance across diverse applications, they tend to under-use visual information and over-rely on textual context. In this work, we propose \textsc{GeoPID}, a training-free framework that analyzes multimodal information within VLMs from a geometric perspective. \textsc{GeoPID} decomposes information into Redundant, Modality-Unique, and Synergistic components through the geometric relationships between visual and textual representation subspaces. Through an extensive analysis across 22 VLMs and 14 benchmarks, we confirm that correct predictions exhibit stronger vision-unique components when questions strongly require visual grounding. Building on this geometric analysis, we introduce a targeted intervention technique that selectively amplifies visual representations along the vision-unique subspace during inference. As a result, visual grounding capabilities were enhanced without any additional model parameter updates, achieving an average relative accuracy gain of 7.63%.

Figures & tables

Figure 1: In this figure, we take a visual and textual input and project them into their own representation space (a-b), where vectors represent features that can be extracted from each input modality. (c): The fused representation space reveals geometric relationships and contrasting modality-specific content: text-unique features ( orange ) praise the room’s tidiness, while vision-unique features ( blue ) depict the disorder. The green direction denotes the complementary geometry in which synergy is evaluated. (d): Our approach selectively amplifies along the vision-unique subspace while preserving its orthogonal complement.

Figure 2: (a) Layer-wise geometric measures for the shared, text-unique, vision-unique, and synergy-evaluation subspaces. Models highlighted in red and blue exhibit contrasting text-unique and vision-unique profiles in later layers. (b) Models with more pronounced vision-unique components ( blue ) also tend to show higher $U_{V}$ and $S$ . The x-axis denotes normalized layer depth. Appendix E.5 extends the information measurements in (b) across $14$ benchmarks.

Model	POPE	MME	CVB	Hall	Reef	NatB	RWQA	MMB	SEED	AI2D	AOK	SQA	Rap4o	GenAI
Qwen2.5-VL-7B ( Bai et al., 2025b )	+1.0	+0.0	+2.9	+3.6	+0.5	+2.7	+0.7	+0.2	+1.0	+1.0	+0.2	+2.5	+0.8	+4.7
Qwen3-VL-8B ( Bai et al., 2025a )	+0.5	+3.1	+0.0	+0.6	+0.2	+0.0	+0.0	+0.0	-0.5	+1.0	+0.2	+0.2	+5.9	+4.3
InternVL3-8B ( Zhu et al., 2025 )	+0.2	-0.8	+1.6	+4.2	+0.0	+1.3	+1.3	-0.5	+0.7	+0.5	-0.2	-1.0	+2.3	+11.0
InternVL3-2B ( Zhu et al., 2025 )	+0.2	+1.8	+2.3	+0.6	+0.2	+3.8	+4.7	+1.1	+0.7	-0.8	+0.2	-0.5	+1.1	-5.7
LLaVA-OV-7B ( Li et al., 2024c )	+1.4	+0.8	-0.8	+3.0	+1.2	+0.5	-1.3	+0.0	+0.7	-0.3	+0.9	+0.7	+27.1	+2.7
Idefics2-8B ( Laurençon et al., 2024 )	+0.0	+0.5	+1.8	-1.2	+2.6	+1.1	+1.3	+0.2	+0.0	+1.6	+1.1	-0.5	-1.1	-8.7

Table 2: Accuracy gain (

\%

) of the

G_{U_{V}}

intervention at each cell’s selected

(\ell,\alpha)

, 308 models

\times

benchmarks. Benchmark abbreviation follows Appendix C.2 .

Table 5

Figure 4: Geometry-based intervention effects across three regimes: strong late-layer text-unique

G_{U_{t}}

dominance with high shared geometry

G_{R}

(left), strong text-unique dominance with low shared geometry (middle), and weak text-unique dominance (right). Top: layer-wise profiles of shared geometry

G_{R}

and the difference

G_{U_{t}}-G_{U_{v}}

. Bottom: mean accuracy gains from the intervention in Eq. 11 at different

\ell^{*}

and

\alpha

. The strong text dominance regimes favor early interventions with high amplification strengths, whereas the weak text dominance regime shows degradation under strong early interventions.

Model	POPE	MME	CVB	Hall	Reef	NatB	RWQA	MMB	SEED	AI2D	AOK	SQA	Rap4o	GenAI	Mean
Qwen2.5-VL-7B ( Bai et al., 2025b )	-0.2	+0.0	-0.5	+0.6	+0.2	+2.7	-0.7	+0.5	+0.7	+1.3	+0.0	+2.5	+0.8	+4.7	+1.05
Qwen3-VL-8B ( Bai et al., 2025a )	+0.0	+0.0	+0.0	+0.6	+0.0	+0.0	+0.7	+0.0	-1.7	+0.8	+0.0	+0.2	+5.1	+0.7	+0.49
InternVL3-8B ( Zhu et al., 2025 )	+0.5	+0.3	+0.5	+0.6	+0.0	+0.0	+1.3	-0.2	+0.5	+0.3	+0.0	+0.0	-1.1	+0.0	+0.15
InternVL3-2B ( Zhu et al., 2025 )	+0.0	-0.3	+0.3	+0.6	+0.2	+1.6	+1.3	+0.0	+0.5	-0.8	-0.2	+0.0	+1.1	+2.0	+0.59
LLaVA-OV-7B ( Li et al., 2024c )	+0.7	+0.5	-0.8	+1.2	+0.7	+0.5	-2.7	+0.5	+1.0	-0.3	+0.5	+0.7	+27.4	-6.7	+1.46
Idefics2-8B ( Laurençon et al., 2024 )	+0.5	+0.0	+1.6	+0.0	-0.2	+0.8	+0.7	-0.5	+0.7	+0.3	+0.5	-0.5	-1.7	-1.0	+0.17

Table 5: Accuracy gain (

\%

) of the

G_{U_{v}}

intervention at the

(\ell^{*},\alpha)

chosen by a fixed geometry-based rule described in Section 3.2 , across 22 models

\times

14 datasets. The thresholds were selected using the same 100-question calibration subset and then fixed. Benchmark abbreviation is in Appendix C.2 .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Size	Family / parent	Model type and emphasis
General-purpose instruction-tuned VLMs
Qwen2.5-VL-7B ( Bai et al., 2025b )	7B	Qwen2.5-VL	Image and document understanding with dynamic-resolution visual processing.
Qwen3-VL-8B ( Bai et al., 2025a )	8B	Qwen3-VL	General multimodal instruction following; multi-level visual feature integration.
InternVL3-8B ( Zhu et al., 2025 )	8B	InternVL3	Native multimodal pre-training with broad visual and linguistic capabilities.
InternVL3-2B ( Zhu et al., 2025 )	2B	InternVL3	Smaller-scale member of the same model family.
LLaVA-OV-7B ( Li et al., 2024c )	7B	LLaVA-OneVision	Unified single-image, multi-image, and video understanding.

Appendix

Table 6: The

22

evaluated VLMs. Family denotes the multimodal model family or parent VLM.

Benchmark	Code	Task and visual content	Evaluation format
Broad multimodal understanding
MME ( Fu et al., 2026 )	MME	Perception and cognition, including recognition and reasoning.	Binary answer selection
MMBench ( Liu et al., 2024b )	MMB	Multiple visual perception and reasoning abilities.	Multiple-choice selection
SEED-Bench ( Li et al., 2023a )	SEE	Visual comprehension across several evaluation dimensions.	Multiple-choice selection
Visual perception and hallucination
POPE ( Li et al., 2023c )	POP	Presence or absence of objects in natural images.	Binary answer selection

Appendix

Table 7: The

14

evaluated benchmarks. The final column describes the answer format used in this study. Dataset codes follow the complete appendix tables.

Category	Accuracy drop (pp)	Difference from T (pp)
T	3.9	0.0
$\mathbf{T^{-}}$	19.3	15.4
V	35.4	31.5
J	28.9	25.0

Appendix

Table 8: Accuracy reduction after replacing images with blank inputs, aggregated over model-item pairs. Larger reductions indicate greater sensitivity to visual input removal.

Benchmark	Items	T	$\mathbf{T^{-}}$	V	J
POPE ( Li et al., 2023c )	$700$	$0$	$0$	$700$	$0$
CV-Bench ( Tong et al., 2024 )	$700$	$0$	$0$	$700$	$0$
RealWorldQA ( xAI, 2024 )	$425$	$0$	$0$	$425$	$0$
GenAI-Bench ( Li et al., 2024a )	$700$	$0$	$0$	$0$	$700$
Rapidata-4o ( Rapidata, 2026 )	$700$	$0$	$0$	$0$	$700$
HallusionBench ( Guan et al., 2024 )	$447$	$123$	$32$	$199$	$93$

Appendix

Table 9: Evidence-requirement categories in the evaluated item pool.

$r_{u}$	Mean $U_{v}$	$\rho$ vs $r_{u}=32$
$16$	$0.01455$	$1.000$
$32$	$0.01890$	$1.000$
$64$	$0.02495$	$1.000$
$128$	$0.03115$	$1.000$
$256$	$0.03965$	$1.000$
$512$	$0.04785$	$1.000$

Appendix

Table 10: Sensitivity of

U_{v}

to the vision-unique rank cap

r_{u}

across 6 models and 3 benchmarks. Mean

U_{v}

is the equally weighted average of the two previously reported group means.

\rho

is the Spearman correlation of the model ordering by

U_{v}

with the default setting

r_{u}=32

. The magnitude increases with the cap, while the model ordering remains unchanged.

Model	$r_{u}=4$	$r_{u}=8$	$r_{u}=16$	$r_{u}=32$
ViGoRL-7B ( Sarch et al., 2026 )	$+13.3$ ( $-3.5$ )	$+21.1$ ( $+4.2$ )	$+19.1$ ( $+2.3$ )	$+16.9$
ViGoRL-3B ( Sarch et al., 2026 )	$+2.6$ ( $-0.9$ )	$+5.7$ ( $+2.2$ )	$+3.9$ ( $+0.4$ )	$+3.5$
ViGoRL-MCTS-3B ( Sarch et al., 2026 )	$+2.4$ ( $+0.0$ )	$+3.6$ ( $+1.2$ )	$+3.4$ ( $+1.0$ )	$+2.4$
OpenVLThinker-7B ( Deng et al., 2025 )	$-2.4$ ( $+0.1$ )	$-2.9$ ( $-0.4$ )	$-3.5$ ( $-1.0$ )	$-2.5$
Vision-R1-7B ( Huang et al., 2026 )	$-3.2$ ( $-0.2$ )	$-3.1$ ( $-0.0$ )	$-2.7$ ( $+0.3$ )	$-3.0$

Appendix

Table 11: Accuracy gain by changing the rank of

G_{U_{V}}

during the intervention stage. Each entry is the mean accuracy gain (pp) over the six configurations the rank sweep and the main grid share (

\ell/L\in\{0.5,0.6,0.7\}\times\alpha\in\{2,4\}

), with the change from the

r_{u}{=}32

default in parentheses.

Figure 5: Illustrative two-dimensional projections of visual and textual representations across decoder depth on MMBench ( Liu et al., 2024b ) . Top: Qwen2.5-VL-7B ( Bai et al., 2025b ) at layers $3$ , $14$ , and $26$ under the displayed $28$ -layer indexing. Bottom: ViGoRL-3B ( Sarch et al., 2026 ) at layers $4$ , $18$ , and $33$ under the displayed $36$ -layer indexing. Cyan and orange points denote visual and textual representations, respectively. Arrows mark modality means. Dashed lines indicate answer-readout axes.

Model / Benchmark	Matched	Same category	Different category	Blank
InternVL3-8B ( Zhu et al., 2025 ) / MMBench ( Liu et al., 2024b )	0.4150	0.0323	0.0274	0.0149
Qwen2.5-VL-7B ( Bai et al., 2025b ) / POPE ( Li et al., 2023c )	0.1882	0.0106	0.0055	0.0010
ViGoRL-3B ( Sarch et al., 2026 ) / MMBench ( Liu et al., 2024b )	0.0098	0.0033	0.0031	0.0019

Appendix

Table 12: Answer information measured within

G_{S}

under various image substitutions. Three representative examples are shown. ‘Same category’ and ‘Different category’ indicate whether the original image was replaced with another image from the same question category or from a completely different category.

Benchmark	$I_{2}$	$R$	$U_{v}$	$U_{t}$	$S$
MMBench ( Liu et al., 2024b )	0.0483	0.0063	0.0595	0.0566	0.3001
A-OKVQA ( Schwenk et al., 2022 )	0.0501	0.0067	0.0435	0.0408	0.2973
SEED-Bench ( Li et al., 2023a )	0.0340	0.0052	0.0367	0.0308	0.1940
POPE ( Li et al., 2023c )	0.0459	0.0153	0.0386	0.0385	0.1860
Reefknot ( Zheng et al., 2025 )	0.0341	0.0034	0.0275	0.0192	0.1844
MME ( Fu et al., 2026 )	0.0349	0.0111	0.0358	0.0355	0.1738

Appendix

Table 13: Information measurements across benchmarks.

Figure 6: Layer-wise information profiles by the tasks: perception (

7

benchmarks), knowledge and reasoning (

5

), general (

3

), and image-pair preference (

2

). Image-pair preference tasks show a later rise in

U_{t}

than others.

Benchmark	$S_{\mathrm{Q}}$	$S_{\mathrm{V1}}$	$\Delta S$	$I_{2,\mathrm{Q}}$	$I_{2,\mathrm{V1}}$
POPE Li et al. (2023c)	0.183	0.036	0.147	0.064	0.005
MME ( Fu et al., 2026 )	0.201	0.061	0.140	0.050	0.009
MMBench ( Liu et al., 2024b )	0.309	0.044	0.265	0.047	0.006
HallusionBench Guan et al. (2024)	0.109	0.016	0.093	0.015	0.003
SEED-Bench ( Li et al., 2023a )	0.166	0.045	0.121	0.029	0.006
AI2D ( Kembhavi et al., 2016 )	0.200	0.051	0.149	0.043	0.010

Appendix

Table 14: Per-benchmark information comparison for Qwen2.5-VL-7B ( Bai et al., 2025b ) (Q) and Vision-R1-7B ( Huang et al., 2026 ) (V1).

\Delta S=S_{\mathrm{Q}}-S_{\mathrm{V1}}

is calculated from the rounded entries.

Benchmark / outcome	$R$	$U_{v}$	$U_{t}$	$S$	$I_{2}$	$R-S$
POPE ( Li et al., 2023c )	+0.14	+0.59	+0.40	+0.46	+0.50	-0.58
MME ( Fu et al., 2026 )	+0.77	+0.31	+0.84	+0.57	+0.75	-0.56
MMBench ( Liu et al., 2024b )	+0.53	+0.25	+0.57	+0.65	+0.63	-0.64
HallusionBench ( Guan et al., 2024 )	+0.50	+0.37	+0.83	+0.80	+0.63	-0.80
Rapidata-4o ( Rapidata, 2026 )	+0.07	+0.26	+0.25	+0.33	+0.30	-0.36
GenAI-Bench ( Li et al., 2024a )	+0.81	+0.57	+0.90	+0.83	+0.89	-0.77

Appendix

Table 15: Spearman correlations between information measurements and image-removal accuracy drop, with pooled rows for removal drop and base accuracy. The pooled results are reported for

308

cells.

Aggregation	Included models	Count	Spearman $\rho$
Model-level	All	22	0.79
Model-level	Excluding ViGoRL ( Sarch et al., 2026 )	20	0.68
Cell-level	All	308	0.73
Cell-level	Excluding ViGoRL ( Sarch et al., 2026 )	340	0.67

Appendix

Table 16: Sensitivity of the

S

: image-removal association to excluding the ViGoRL ( Sarch et al., 2026 ) family.

Benchmark inclusion	Mean standardized $G_{U_{v}}$ (correct V/J $-$ correct T)	$95\%$ CI	$p$	Benchmarks with higher $G_{U_{v}}$ for correct V/J than correct T
$\geq 10$ correct T	$+0.71$	$[+0.46,+0.99]$	$0.0002$	$8/8$
All with both types	$+0.45$	$[+0.005,+0.82]$	$0.0002$	$9/10$

Appendix

Table 17: Robustness of higher

G_{U_{v}}

for correct V / J than for correct T to benchmark inclusion criteria. The second column reports the mean standardized

G_{U_{v}}(\text{correct V/J})-G_{U_{v}}(\text{correct T})

; the third gives its benchmark-bootstrap

95\%

confidence interval. Permutation tests compare these two sets of correct items by shuffling type labels within benchmarks. The final column counts benchmarks where correct V / J have higher

G_{U_{v}}

than correct T .

$S$	$\Delta G_{R}$	$\Delta(G_{R}$ , $G_{U_{t}}$ , $G_{U_{v}})$	$\Delta(G_{R}$ , $G_{U_{t}}$ , $G_{U_{v}}$ , $S)$	$\Delta(G_{R}$ , $G_{U_{t}}$ , $G_{U_{v}})$ Boosted	full geometry, Boosted
$-0.54$ (Spearman)	$0.54$ (Pearson)	$0.56$ (Pearson)	$0.61$ (Pearson)	$0.66$ (Pearson)	$0.80$ (Pearson)

Appendix

Table 18: Correlation between intervention gain of Table 2 and the base model’s geometry across 308 model

\times

benchmark combinations.

\Delta G_{R}

measures the drop in shared geometry (the peak value at depths

\geq 0.5

minus the output value).

\Delta(G_{R}

G_{U_{t}}

G_{U_{v}})

indicates the late-layer rise in text-unique geometry minus the drops in both vision-unique and shared geometries. Boosted refers to out-of-fold predictions from gradient-boosted trees ( Friedman, 2001 ) (10-fold cross-validation), from the three features

G_{R}

G_{U_{t}}

G_{U_{v}}

or from the full layer-wise geometric trajectories.

Figure 7: Qualitative impact of the vision-unique ( $G_{U_{v}}$ ) intervention across 14 benchmarks. The scatter plots illustrate the relationship between overall accuracy (x-axis) and the answered-blind rate (y-axis) before and after the intervention. Models identified as having strong text dominance, initially clustered in the red dashed boxes with high answered-blind rates, demonstrate a distinct shift towards the blue dashed boxes following the intervention. This collective shift indicates a significant reduction in blind text reliance and a concurrent improvement in visual grounding and overall performance.

Figure 8: Qualitative examples of open-ended generation before and after the $G_{U_{v}}$ intervention. Baseline models often fall back on language priors (e.g., blindly guessing “corn” or “chihuahua”), whereas the intervened models accurately ground their generated text in the specific visual evidence (e.g., “banana”, “german shepherd”).

$\mathbf{T}$	$\mathbf{T^{-}}$	$\mathbf{V}$	$\mathbf{J}$	All
$+3.20$	$+2.90$	$+3.15$	$+4.40$	$+3.40$

Appendix

Table 19: Accuracy gain by question category (

\%

Question category	True image	Deranged image	Blank image
$T$ : text decides	$+22.6$	$+17.7$	$+1.3$
$A$ : text prior	$+15.3$	$+7.4$	$0.0$
$V$ : image reading	$+13.1$	$+1.0$	$-0.2$
$J$ : joint evidence	$+13.9$	$-3.3$	$+0.3$

Appendix

Table 20: Image-condition control on the

100

data for the models exhibiting strong text dominance in Figure 4 . We report the accuracy gains in

\%

Statistic	Share
Most frequent predicted option before intervention	$72\%$
Most frequent predicted option after intervention	$54\%$
Most frequent ground-truth option	$39\%$

Appendix

Table 21: Concentration of predictions on the most frequently selected answer option before and after intervention. The ground-truth majority share is provided for comparison.

Statistic	Value
Eligible model–item pairs	$678$
Correct after intervention	$74.0\%$
Expected correctness under random switching	$34.1\%$

Appendix

Table 22: Conditional correction among initially incorrect predictions that change after intervention. Only model-item pairs with at least three answer options are included.

Figure 9: Decision-state clouds. Top: Examples of weak text-dominant model-benchmark pairs, where the text-only ( orange ) and text+image ( blue ) decision states naturally maintain a distinct angular separation. Bottom: Examples of strong text-dominant pairs. Initially, their text+image states exhibited small angular differences from the text-only states, indicating heavy text reliance. However, our GeoPID vision-unique amplification effectively pushes the intervened states ( green ) away, establishing a clear separation. Crosses mark cloud centroids and arrows mark the before–after displacement. Displayed angular summaries are approximately $10^{\circ}\!\to 31^{\circ}$ for ViGoRL-MCTS-3B ( Sarch et al., 2026 ) /SEED-Bench ( Li et al., 2023a ) , $29^{\circ}\!\to 48^{\circ}$ for Idefics2-8B ( Laurençon et al., 2024 ) /POPE ( Li et al., 2023c ) , and $21^{\circ}\!\to 39^{\circ}$ for SmolVLM ( Marafioti et al., 2025 ) /MMBench ( Liu et al., 2024b ) . Reported angles are averaged over individual examples.

Amplified target	Rank	Gain	Correction	Damage
Vision-unique $G_{U_{v}}$	$32$	$+0.47$	$3.9$	$1.4$
Shared $G_{R}$	—	$-1.67$	$4.1$	$4.9$
Text-unique $G_{U_{t}}$	$7$	$-0.47$	$4.5$	$3.2$
Synergy-evaluation space $G_{S}$	$51$	$-0.27$	$7.5$	$4.6$
Random hidden-space basis	$32$	$+0.07$	$0.7$	$0.3$
Random text-complement basis	$32$	$+0.07$	$0.6$	$0.2$

Appendix

Table 23: Alternative amplification targets on

15

model-benchmark pairs at normalized

\ell^{*}=0.6

and

\alpha=1

. Gain is in percentage points. Correction and damage are reported percentages.

Decision	Rule
Edit target	Vision-unique basis $G_{U_{v}}$ on visual tokens.
Open-branch strength	$\alpha=8$ ; or $\alpha=12$ when blank-image agreement $\geq 0.9$ .
Closed-branch strength	$\alpha=1$ , a weak edit rather than the unedited baseline.

Appendix

Table 24: Full rule for alternative decision for which models receive stronger intervention.

Figure 10: Effects of intervention depth and amplification strength ( $\alpha$ ) on accuracy for (A) gated ( $n=51$ ) and (B) non-gated ( $n=340$ ) cells, with (C) the corresponding performance trends across varying strengths at the selected depths.

Intervention selection rule	Mean gain (pp)
Blank-image agreement with geometry-based depth selection	$+2.24$
Blank-image agreement with $\alpha=12$ for agreement $\geq 0.9$	$+2.28$
GeoPID geometry-based rule (Figure 4 )	$\mathbf{+4.62}$

Appendix

Table 25: Mean accuracy gains (

\%

) across

308

model-benchmark pairs under different intervention selection rules. The primary result corresponds to Table 5 .

Rank cap	Depth	$\alpha=1$	$\alpha=2$	$\alpha=4$
$16$	$0.50$	$0.00$	$+2.33$	$+21.67$
$16$	$0.60$	$+4.33$	$+19.33$	$+29.67$
$16$	$0.70$	$+1.00$	$+5.00$	$+21.00$
$64$	$0.60$	$+3.67$	$+10.33$	$+21.67$

Appendix

Table 26: Mean accuracy gains (

\%

) for ViGoRL-7B ( Sarch et al., 2026 ) across POPE ( Li et al., 2023c ) , SEED-Bench ( Li et al., 2023a ) , and MMBench ( Liu et al., 2024b ) under different rank caps, normalized intervention depths, and strengths.

$K$	Mismatch standard deviation	Reported SE of $S$	Median $\|S_{K}-S_{1}\|$
$3$	$0.0017$	$0.0010$	$0.0029$
$5$	$0.0023$	$0.0010$	$0.0022$
$10$	$0.0031$	$0.0010$	$0.0038$

Appendix

Table 27: Sensitivity to the number of mismatched-image draws

K

in Eq. 10 . Standard errors summarize mismatch-related variability across

24

model

\times

benchmark pairs.

Ablation	Setting	Rank correlation
Kernel	Cosine reference	$1.00$
	RBF	$0.78$
	Standardized linear	$0.75$
Rényi order	Shannon limit	$0.95$
	$1.5$	$1.00$
	$2$ (reference)	$1.00$

Appendix

Table 28: Ablation study using

22

models and

5

datasets. Values are Spearman correlations between model rankings based on total decision-state information.

Explore similar work

GeoWorld-VLM: Geometry from World Models for Vision-Language Models
May 15, 2026Renjie Gu, Kaichen Zhou, Yan Luo +1Visual Spatial Reasoning VLM Adaptation
Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University
Geometric Encoding for Spatial Reasoning in Vision-Language Models
Sep 28, 2026Antonio Jun, Haoshui Yu, Zhengyi Lu +2Visual Spatial Reasoning VLM Reasoning
School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1Vision-Language Models Object Hallucination in VLMs
Indian Institute of Technology Dhanbad · National University of Singapore

May 15, 2026, cs.CVJ/K move · Enter open · S save

GeoWorld-VLM: Geometry from World Models for Vision-Language Models

Renjie Gu, Kaichen Zhou, Yan Luo, Mengyu Wang

Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University

Visual Spatial Reasoning VLM Adaptation VLM Distillation Visually Grounded Reasoning Vision-Language Alignment Knowledge Distillation

Open paper AskPDF arXiv

Modern Vision-Language Models (VLMs) achieve strong semantic recognition, yet remain brittle on elementary spatial relations such as left of, on, behind, and between. One cause of this failure arises before language reasoning begins: the visual pathway may compress or discard critical 3D structural cues during feature extraction, so the language model receives image representations that are already insufficient for reliable spatial judgment. We introduce GeoWorld-VLM, a VLM-side distillation framework that transfers geometric structure from frozen camera-conditioned video world models into VLMs. GeoWorld-VLM fine-tunes only the image encoder and multimodal projector, aligning post-projector image features with intermediate world-model representations while leaving the main backbone frozen. Given images, a prompt, and a sampled camera trajectory, the world-model teacher converts static visual input into a synthetic multi-view spatial signal. Training combines spatial answer supervision, teacher-student feature alignment, and a preservation anchor to the original VLM. Since the language model remains frozen, GeoWorld-VLM preserves the original model's linguistic capabilities while attributing spatial improvements to the enhanced visual pathway. To evaluate the effectiveness and generality of the proposed method, we apply GeoWorld-VLM to two distinct VLM architectures and observe consistent improvements across both backbones. GeoWorld-VLM improves performance by approximately 4 percent on both the What'sUp and VSR benchmarks, suggesting that world-model-guided visual alignment generalizes across model structures and spatial reasoning datasets.

Abstract

Explore similar work

GeoWorld-VLM: Geometry from World Models for Vision-Language Models

Geometric Encoding for Spatial Reasoning in Vision-Language Models

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models