Probing Spatial State Representations in Visual Foundation Models

Baiqiao Yin; Zhongang Cai; Ruisi Wang; Qiwei Liang; Lei Yang; Ziwei Liu

cs.CVOct 29, 2026

Probing Spatial State Representations in Visual Foundation Models

Authors: Baiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, Ziwei Liu

Organizations: New York University · Nanyang Technological University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong

Abstract

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.

Figures & tables

Figure 1: Spatial state representations from streaming egocentric observations. A moving observer builds an internal spatial representation from partial views of an environment. L1 ( spatial structure ) captures scene geometry and object identity across views. L2 ( object persistence ) retains the observer-relative locations of previously seen objects after they leave the field of view. L3 ( predictive spatial update ) anticipates what the observer would see after an action or a sequence of actions. L4 ( actionability ) uses the represented state and a specified goal to select an action that makes progress toward that goal.

Panel of Figure 2 — Figure 2: Functional relationships among the four spatial capabilities. The levels are ordered by what each demands beyond the current observation. L1 grounds the spatial state in observed scene structure. L2 extends the demand from representing what is visible to retaining information after objects leave the field of view, and L3 requires anticipating how the observer-relative state changes under ego-motion. L4 connects the represented state to goal-directed action. The dashed return arrow illustrates potential closed-loop use: actions change the observer’s viewpoint and hence the subsequent observations, while the environment itself remains static.

Figure 3: Probing four spatial capabilities. Probe inputs and targets at each level: prefix features alone (L1), plus an object query (L2), ego-motions and an isolated future feature (L3), or a goal frame and an action (L4).

Family	Model
Image encoder	DINOv2-Large
Video encoder	V-JEPA2 ViT-L
	VideoMAE-Large
3D encoder	Fast3R ViT-L
Video generator	Wan2.1-T2V-1.3B
	CogVideoX-5B-I2V

Table 1: Ten representations evaluated.

	Metric geometry		Instance identity	View consistency	Object persistence	Direct prediction	Path integration
Model (layer)	Depth loss $\downarrow$	AUC@30 $\uparrow$	Instance loss $\downarrow$	Bal. acc. $\uparrow$	Ang. err. ( ∘ ) $\downarrow$	Global R@1 $\uparrow$	Global R@1 $\uparrow$
DINOv2-L (0)	0.1422	0.0412	1.3828	0.9557	65.38	0.2373	0.0900
Fast3R (24)	0.1386	0.3870	1.6072	0.9808	32.79	0.0133	0.0190
VideoMAE-L (23)	0.2968	0.2026	4.1571	0.9118	36.45	0.0531	0.0685
V-JEPA2-L (23)	0.2708	0.1854	2.9211	0.9256	45.31	0.0625	0.1588
Qwen2.5-VL ( $-1$ )	0.2244	0.0269	2.9476	0.9385	68.59	0.1712	0.0903

Table 2: Main results for all ten representations. Prefix aggregation follows Eq. 3 at each designated layer. The first header row names the diagnostic task and the second row its metric; metric geometry contributes two metrics. Bold , colored values mark the best result in each column and every result whose paired 95% scene-bootstrap interval of the difference from it contains zero; marked cells are excluded.

\dagger

: MiniMax-H3 prediction is evaluated on 171 of the 276 validation windows, and its direct predictor uses the next-observation target only. Table 8 gives the intervals of every cell and Table 6 the shared-window comparison.

Panel of Figure 4 — Figure 4: Spatial readout depends on both layer and streaming prefix. (a) Four multi-layer backbones: cells give the best measured block, colored by normalized depth. Metric geometry averages the percentile ranks of depth and camera-pose performance. (b) Paired comparisons for eight models at their designated layers: direction-corrected, standardized p24 $-$ p8 differences following Eq. 4 ; $+$ favors the longer prefix and $-$ the shorter. Direct prediction pools horizons 1, 2, and 4 for these models. Full layer curves and joint grids are given in Section A.1 .

Model	Motion gain (pp)	History benefit ( ∘ )
V-JEPA2	18.84 [13.04, 25.36]	10.26 [3.51, 16.80]
VideoMAE	10.51 [6.16, 15.22]	2.08 [ $-$ 6.10, 9.97]
Wan	2.90 [ $-$ 1.45, 7.25]	10.01 [3.42, 16.69]
CogVideoX	4.35 [0.35, 7.97]	$-$ 3.41 [ $-$ 9.39, 2.39]

Table 3: Motion and history controls. Motion gain is the correct-minus-wrong-motion R@1 in the matched-candidate protocol; history benefit is the reduction in localization error from full history relative to the current frame alone. Brackets give paired 95% scene-bootstrap intervals.

Panel of Figure 5 — Figure 5: Testing readable structure and goal-specific utility. (a) V-JEPA2 normalized-depth MAE at $h=2$ and $h=4$ with the same frozen reader, except for the retrained control; dashed lines mark the training-layout priors, and error bars show the SD over three seeds. (b) Positive-progress differences in the continuous-output setting under input restriction (full minus goal-only, full minus current-only) and fixed-weight goal shuffling (correct minus shuffled goal); error bars show paired 95% scene-bootstrap intervals.

Representation	Action only	Direct	Shuffled goal	Predict- then-score	Endpoint ref.
DINOv2-L	$67.8{\pm}0.4$	$76.2{\pm}0.8$	$55.5{\pm}0.8$	$76.2{\pm}1.2$	$85.5$
Fast3R	$67.8{\pm}0.4$	$79.9{\pm}1.3$	$62.4{\pm}0.5$	$78.1{\pm}0.3$	$78.0$
V-JEPA2	$67.8{\pm}0.4$	$78.1{\pm}0.7$	$62.0{\pm}1.1$	$76.7{\pm}0.8$	$78.0$
Wan	$67.8{\pm}0.4$	$75.9{\pm}0.9$	$63.7{\pm}0.5$	$63.6{\pm}1.1$	$68.3$

Table 4: Goal-conditioned action prediction: candidate-selection setting. Accuracy (%), mean

\pm

SD over three training seeds; chance is 50%. Shuffled goal evaluates the trained direct scorer with mismatched goals at fixed weights. The endpoint reference observes realized outcomes and is diagnostic only.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Capability profiles across measured layers. Prefix means are direction-corrected and converted to percentile ranks across the available model–layer cells within each task; metric geometry averages the depth and camera-pose ranks. The horizontal axis shows normalized backbone depth, and isolated output representations occupy a separate band on the right. Hollow markers denote the MiniMax-H3 points, which are evaluated on the 171-window subset (with

h=1

only for direct prediction).

Figure 7: Layer–prefix structure of instance identity and object persistence. The four panels show instance identity (top) and object persistence (bottom) for VideoMAE (left) and V-JEPA2 (right) across extraction layers and prefix lengths. Colors are normalized within each panel and therefore describe within-model structure rather than absolute cross-model accuracy; raw values accompany the heatmaps. No probe was trained on block 0 of either model, so it is not shown.

Figure 8: Interventions separate information use from decodability. (a) Change the action supplied to a fixed predictor

F

, or copy the last state. (b) Remove or reorder visual feature tokens while retaining the object query; five illustrative slots are shown, with temporal identifiers and the final frame fixed during order tests. (c) Compare sequential, composed, and swapped updates against the same real endpoint.

Model	Current frame only	Remove target-visible	No visual tokens	Shuffle order	Reverse order
V-JEPA2	10.26 [3.51, 16.80]	9.22 [3.95, 14.59]	9.66 [2.74, 16.85]	0.04 [ $-$ 2.54, 2.98]	2.15 [0.22, 4.56]
VideoMAE	2.08 [ $-$ 6.10, 9.97]	5.83 [ $-$ 2.09, 13.57]	6.06 [ $-$ 2.36, 14.57]	0.02 [ $-$ 1.95, 1.87]	1.09 [ $-$ 1.25, 3.07]
Wan	10.01 [3.42, 16.69]	6.59 [0.58, 12.91]	13.55 [7.31, 19.61]	0.95 [ $-$ 1.85, 3.94]	1.85 [ $-$ 1.00, 4.60]
CogVideoX	$-$ 3.41 [ $-$ 9.39, 2.39]	0.54 [ $-$ 4.33, 5.53]	$-$ 0.83 [ $-$ 6.93, 5.05]	0.00 [0.00, 0.00]	$-$ 0.13 [ $-$ 0.32, 0.00]

Appendix

Table 5: History interventions for object persistence. Increase in localization error ( ∘ ) relative to the full history at p12 (165 samples from 65 scenes); brackets give paired 95% scene-bootstrap intervals, and positive values mean that the intervention hurts. Every condition keeps the historical object query ( Figure 8 b). The current-frame-only column repeats Table 3 .

Model (layer)	Depth loss $\downarrow$	Camera AUC@30 $\uparrow$	Instance loss $\downarrow$	View bal. acc. $\uparrow$	Persistence error ( ∘ ) $\downarrow$	Direct R@1 $\uparrow$	Path R@1 $\uparrow$
DINOv2-L (0)	0.1437	0.0421	1.4325	0.9586	66.03	0.2875	0.1243
Fast3R (24)	0.1371	0.3806	1.6732	0.9817	32.21	0.0200	0.0234
VideoMAE-L (23)	0.3034	0.2012	4.2136	0.9177	34.98	0.0682	0.0838
V-JEPA2-L (23)	0.2802	0.1845	3.1068	0.9225	44.38	0.0775	0.1930
Qwen2.5-VL ( $-1$ )	0.2270	0.0281	3.0099	0.9384	70.28	0.2149	0.1189
Wan (20)	0.3720	0.0653	5.1248	0.8126	60.94	0.0331	0.0521

Appendix

Table 6: Spatial readout on the 171 shared validation windows per prefix. Designated layers and prefix aggregation match Table 2 .

\dagger

: MiniMax-H3 direct prediction uses only

h=1

; the other direct predictors use

h\in\{1,2,4\}

Model (layer)	$h=1$ R@1 $\uparrow$
DINOv2-L (0)	0.6126
Fast3R (24)	0.0585
VideoMAE-L (23)	0.1930
V-JEPA2-L (23)	0.2281
Qwen2.5-VL ( $-1$ )	0.5322
Wan (20)	0.1096

Appendix

Table 7: Direct prediction with a common next-observation target. Mean global R@1 over four prefixes at the designated layer, using the same 171 windows and 171

h=1

candidates per prefix for every model.

\dagger

: next-observation-only training; the other models retain their mixed-horizon training.

	Metric geometry		Instance identity	View consistency	Object persistence	Direct prediction	Path integration
Model (layer)	Depth loss $\downarrow$	AUC@30 $\uparrow$	Instance loss $\downarrow$	Bal. acc. $\uparrow$	Ang. err. ( ∘ ) $\downarrow$	Global R@1 $\uparrow$	Global R@1 $\uparrow$
DINOv2-L (0)	0.1422 [0.1330, 0.1516]	0.0412 [0.0370, 0.0454]	1.3828 [1.2442, 1.5335]	0.9557 [0.9490, 0.9620]	65.38 [61.56, 69.42]	0.2373 [0.2162, 0.2588]	0.0900 [0.0764, 0.1039]
Fast3R (24)	0.1386 [0.1254, 0.1527]	0.3870 [0.3558, 0.4170]	1.6072 [1.4446, 1.7828]	0.9808 [0.9760, 0.9847]	32.79 [29.83, 35.98]	0.0133 [0.0103, 0.0169]	0.0190 [0.0148, 0.0242]
VideoMAE-L (23)	0.2968 [0.2780, 0.3172]	0.2026 [0.1858, 0.2192]	4.1571 [3.7571, 4.5411]	0.9118 [0.9032, 0.9192]	36.45 [33.41, 39.67]	0.0531 [0.0435, 0.0649]	0.0685 [0.0577, 0.0803]
V-JEPA2-L (23)	0.2708 [0.2529, 0.2888]	0.1854 [0.1696, 0.2026]	2.9211 [2.6350, 3.2244]	0.9256 [0.9183, 0.9323]	45.31 [41.38, 49.32]	0.0625 [0.0537, 0.0716]	0.1588 [0.1449, 0.1739]
Qwen2.5-VL ( $-1$ )	0.2244 [0.2104, 0.2387]	0.0269 [0.0244, 0.0295]	2.9476 [2.6158, 3.3016]	0.9385 [0.9310, 0.9460]	68.59 [64.48, 72.11]	0.1712 [0.1564, 0.1875]	0.0903 [0.0776, 0.1033]

Appendix

Table 8: Main results with 95% scene-bootstrap intervals. Same cells as Table 2 : designated layer, mean over the evaluated prefixes ( Eq. 3 ). Intervals resample the 69 validation scenes (1,000 draws, shared by all cells, so differences are paired). Bold : the best complete result in each column and every result whose paired difference from it has an interval containing zero.

\dagger

: MiniMax-H3 prediction uses 171 of the 276 validation windows and, for direct prediction, the next-observation target; not ranked.

	Global R@1 $\uparrow$			Action gain $\uparrow$
Model (layer)	Full	No action	Shuffled action	Gain vs. no action	Gain vs. shuffled
DINOv2-L (0)	0.2373 [0.2162, 0.2588]	0.0223 [0.0175, 0.0275]	0.0691 [0.0595, 0.0794]	0.2150 [0.1929, 0.2373]	0.1682 [0.1476, 0.1911]
Fast3R (24)	0.0133 [0.0103, 0.0169]	0.0018 [0.0003, 0.0033]	0.0088 [0.0060, 0.0121]	0.0115 [0.0082, 0.0151]	0.0045 [0.0015, 0.0075]
VideoMAE-L (23)	0.0531 [0.0435, 0.0649]	0.0021 [0.0003, 0.0039]	0.0133 [0.0094, 0.0175]	0.0510 [0.0411, 0.0631]	0.0399 [0.0299, 0.0513]
V-JEPA2-L (23)	0.0625 [0.0537, 0.0716]	0.0036 [0.0015, 0.0057]	0.0223 [0.0166, 0.0287]	0.0589 [0.0498, 0.0679]	0.0402 [0.0323, 0.0483]
Qwen2.5-VL ( $-1$ )	0.1712 [0.1564, 0.1875]	0.0471 [0.0396, 0.0556]	0.0945 [0.0839, 0.1054]	0.1241 [0.1093, 0.1395]	0.0767 [0.0643, 0.0912]

Appendix

Table 9: Direct prediction against matched action controls. Global R@1 at the designated layer, mean over prefixes, with 95% scene-bootstrap intervals. The same frozen probe is evaluated with the true action, without the action input, and with a shuffled action; gains are paired per query. Gray: the interval of the gain contains zero. Chance is

1/828

(

1/171

for MiniMax-H3 † , next-observation target only).

Model (layer)	Depth loss	AUC@30	Instance loss	Bal. acc.	Ang. err. ( ∘ )	Direct R@1	Path R@1
DINOv2-L (0)	0.0134 [0.0084, 0.0188]	-0.0374 [-0.0455, -0.0290]	-0.2078 [-0.3108, -0.1162]	0.0009 [-0.0093, 0.0125]	-15.54 [-25.48, -6.25]	-0.0205 [-0.0568, 0.0133]	-0.0543 [-0.0845, -0.0229]
Fast3R (24)	0.0203 [0.0131, 0.0281]	0.0657 [0.0279, 0.1030]	-0.4441 [-0.5860, -0.3110]	-0.0021 [-0.0083, 0.0030]	-12.39 [-18.40, -6.44]	-0.0036 [-0.0145, 0.0072]	-0.0085 [-0.0217, 0.0049]
VideoMAE-L (23)	-0.1275 [-0.1441, -0.1100]	-0.1513 [-0.1712, -0.1321]	-8.0407 [-9.0544, -6.8937]	-0.1358 [-0.1532, -0.1188]	-25.25 [-32.90, -17.39]	-0.0664 [-0.0894, -0.0447]	-0.0954 [-0.1232, -0.0700]
V-JEPA2-L (23)	-0.0096 [-0.0214, 0.0013]	-0.0544 [-0.0746, -0.0326]	-1.3906 [-1.8088, -0.9595]	-0.0104 [-0.0242, 0.0028]	-31.76 [-40.04, -23.32]	-0.0048 [-0.0266, 0.0169]	-0.0036 [-0.0362, 0.0254]
Qwen2.5-VL ( $-1$ )	0.0196 [0.0131, 0.0263]	-0.0243 [-0.0296, -0.0191]	-1.1563 [-1.5449, -0.7944]	0.0000 [-0.0127, 0.0125]	-12.33 [-22.91, -3.11]	-0.0085 [-0.0386, 0.0229]	-0.0495 [-0.0797, -0.0241]
Wan (20)	-0.0490 [-0.0636, -0.0345]	-0.0442 [-0.0598, -0.0294]	-4.1312 [-5.0772, -3.2237]	-0.0241 [-0.0481, 0.0010]	-18.42 [-28.93, -8.37]	-0.0048 [-0.0205, 0.0085]	-0.0072 [-0.0266, 0.0121]

Appendix

Table 10: Longer versus shorter prefix with 95% scene-bootstrap intervals. Designated layer, prefix 24 minus prefix 8, oriented so that a positive value means the longer prefix is better (losses and errors are negated). Intervals are paired over the same resampled scenes. Gray: the interval contains zero.

Model	Depth loss $\downarrow$	Instance loss $\downarrow$	View acc. $\uparrow$	Localiz. err. ( ∘ ) $\downarrow$
Wan (20)	-0.043 [-0.050, -0.036]	1.157 [0.647, 1.660]	0.049 [0.040, 0.058]	-18.8 [-22.8, -15.1]
CogVideoX (20)	-0.018 [-0.024, -0.012]	-0.537 [-1.015, -0.092]	0.002 [-0.004, 0.009]	-6.0 [-10.5, -1.7]
MiniMax-H3 (20)	0.006 [0.002, 0.010]	0.229 [-0.160, 0.613]	0.003 [-0.002, 0.008]	-4.1 [-8.2, -0.3]

Appendix

Table 11: Generation coordinate differences with 95% scene-bootstrap intervals. Block 20, mean over prefixes

\{8,12,16,24\}

; each cell is the lower-coordinate score minus the higher-coordinate score (

499-999

for Wan and CogVideoX,

500-1000

for MiniMax-H3), paired over the same resampled scenes. Gray: the interval contains zero.

	Depth loss $\downarrow$		Instance loss $\downarrow$		View acc. $\uparrow$		Localiz. err. ( ∘ ) $\downarrow$
Model	lower	higher	lower	higher	lower	higher	lower	higher
Wan (20)	0.387	0.430	6.810	5.653	0.823	0.774	59.2	78.0
CogVideoX (20)	0.449	0.467	6.541	7.078	0.790	0.787	69.4	75.3
MiniMax-H3 (20)	0.459	0.454	6.249	6.020	0.792	0.789	63.3	67.4

Appendix

Table 12: Spatial readout at two generation coordinates. Depth, instance-identity loss, view consistency, and object persistence at the two scheduler coordinates (block 20, mean over prefixes

\{8,12,16,24\}

). Lower/higher coordinates are

499/999

for Wan and CogVideoX and

500/1000

for MiniMax-H3. Bold marks the better coordinate where the paired 95% interval of the difference excludes zero ( Table 11 ). At block 20, instance identity favors the higher coordinate for Wan and the lower coordinate for CogVideoX, while the MiniMax-H3 difference lies within its interval; object persistence favors the lower coordinate for all three models. These separate-coordinate measurements complement the main table, whose generative-model entries use the default coordinates.

Figure 9: Spatial readout across generation coordinates. Top: each model at the early (dark) and the late (light) scheduler index at block 20, averaged over prefixes; arrows give the better direction, and the values are listed in Table 12 . Bottom: the better coordinate for every layer–prefix pair, E (early) or L (late), grouped by block.

Metric	Wan (15)	V-JEPA2 (11)	Difference [95% CI]	Holm $p$
Gain vs. copy $\uparrow$	0.2051	0.2323	$-0.0273\ [-0.0814,0.0209]$	0.568
Gain vs. no-action $\uparrow$	0.0639	0.0452	$0.0187\ [-0.0050,0.0414]$	0.384
Composition distance $\downarrow$	0.0377	0.0232	$0.0145\ [0.0107,0.0188]$	$<0.004$
Order margin $\uparrow$	$-0.0001$	$-0.0007$	$0.0006\ [-0.0005,0.0019]$	0.568

Appendix

Table 13: One-step-to-two-step structural generalization. Predeclared representative layers; differences are Wan block 15 minus V-JEPA2 block 11 with paired 95% intervals. The four comparisons form one Holm family. Composition distance and order margin are normalized.

Model (layer)	Matching accuracy (%) $\uparrow$	Cosine margin $\uparrow$
DINOv2-L (0)	61.96	0.0292
Fast3R (24)	44.57	$-0.0005$
VideoMAE-L (23)	54.26	0.0004
V-JEPA2-L (23)	54.59	0.0060
Qwen2.5-VL ( $-1$ )	47.80	$-0.0026$
Wan (20)	42.09	$-0.0010$

Appendix

Table 14: Motion-dependent endpoint matching. Mean over four prefixes at the designated layer. All listed heads were trained on offsets 1, 2, and 4. Matching is within three-candidate sets (chance 33.33%); cosine margins need not be positive even when the mean matching accuracy is above chance.

Input / readout condition	$h=2$	$h=4$
Real future / original reader	$0.1824{\pm}0.0003$	$0.1827{\pm}0.0003$
Native prediction / frozen reader	$0.2459{\pm}0.0058$	$0.2427{\pm}0.0054$
Native prediction / retrained reader	$0.2233{\pm}0.0008$	$0.2271{\pm}0.0005$
Copied context / frozen reader	$0.2890{\pm}0.0044$	$0.2974{\pm}0.0041$
Training layout prior	$0.2406$	$0.2377$

Appendix

Table 15: Spatial readout through V-JEPA2’s native predictor. Scene-macro normalized-depth MAE

\downarrow

on 69 validation scenes, spatial branch; mean

\pm

SD over three training seeds. The same reader, trained on real native target states, is used for real, predicted, and copied inputs, except in the explicitly retrained condition. The layout prior uses training labels only.

Model	Direction ( ∘ ) $\downarrow$	Progress (%) $\uparrow$	Goal reduction $\uparrow$	FULL $-$ CURRENT-ONLY goal reduction
DINOv2-L	$65.3{\pm}1.3$	$53.8{\pm}1.1$	$0.008{\pm}0.006$	$-0.008\ [-0.013,-0.004]$
Fast3R	$64.7{\pm}1.3$	$59.8{\pm}3.4$	$0.025{\pm}0.007$	$-0.005\ [-0.012,+0.002]$
V-JEPA2	$66.2{\pm}1.8$	$53.8{\pm}2.4$	$0.007{\pm}0.007$	$-0.009\ [-0.015,-0.002]$
Wan	$70.0{\pm}1.5$	$50.2{\pm}3.5$	$-0.001{\pm}0.009$	$-0.011\ [-0.017,-0.004]$

Appendix

Table 16: Goal-conditioned action prediction: continuous-output setting. FULL uses the relational head; values are mean

\pm

SD over seeds 41–45. Progress is the percentage of trajectories with positive progress, and goal reduction uses the pose-distance metric defined in the text. The final column is the FULL minus CURRENT-ONLY goal reduction with a paired 95% scene-bootstrap interval (5,000 replicates); positive favors FULL. Seed repeats are averaged within each sample before scene resampling.

Model	Probed layers	Primary layer	Extraction coordinate
DINOv2-Large	final hidden state	0	–
V-JEPA2 ViT-L	$\{5,11,17,23\}$	23	–
VideoMAE-Large	$\{5,11,17,23\}$	23	–
Fast3R ViT-L	$\{24\}$	24	–
Wan2.1-T2V-1.3B	$\{0,5,10,15,20,25,29\}$	20	scheduler step 749
CogVideoX-5B-I2V	$\{0,5,10,20,30,35,40\}$	20	scheduler step 749

Appendix

Table 17: Feature extraction details. Layer indices denote the representations used for layer-wise probing. For generative models, features are extracted at the specified denoising or flow coordinate. The table lists the settings of the main sweep; the separate coordinate study additionally uses blocks 15 and 25.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 253 papers

Jun 8, 2026

AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models

Multimodal Foundation Models (MFMs) have made substantial progress, yet remain fragile in spatial reasoning over the physical world. A key bottleneck lies in their inability to transform local egocentric observations into a global allocentric spatial representation. To address this, we propose AlloSpatial, an agentic framework for allocentric spatial cognition in foundation models. AlloSpatial introduces World2Mind, a plug-and-play cognitive mapping sandbox that converts egocentric observations into structured allocentric priors, including Allocentric-Spatial Trees and route maps that support querying object topology, geometric relations, passability, and trajectories. To utilize these priors reliably under noisy reconstruction and ambiguous visual evidence, AlloSpatial introduces a Spatial Reasoning Harness for tool-use judgment, modality-decoupled cue collection, and geometry-semantic arbitration. We further internalize this process in Qwen3-VL through cold-start reinforcement learning with a harness-gated trajectory-level reward. Experiments on VSI-Bench and MindCube show that AlloSpatial improves proprietary models by 5%-18% in a training-free setting, while ASTs alone support strong spatial reasoning even when visual inputs are removed. The trained AlloSpatial agents further outperform larger general-purpose models and competitive spatial baselines, suggesting that structured allocentric representations, active tool use, and verifiable reasoning offer a promising route toward spatially capable foundation models.

Shouwei Ruan, Bin Wang, Zhenyu Wu +5

Institute of Artificial Intelligence, Beihang University · Huawei Noah’s Ark Lab · University of Science and Technology Beijing

VLM Reasoning Multimodal Foundation Models Spatial Reasoning

Ask PDF

May 31, 2026

Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

Humans can reproduce the viewpoint specified by a target image through active head and body motion, yet spatial intelligence in foundation models has largely been studied as passive understanding of pre-collected observations. We introduce Target Viewpoint Reproduction (TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- and TVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turn visual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVR post-training framework covering expert-trajectory SFT, rationale-supervised CoT-SFT, offline Single-turn GRPO, and on-policy Multi-turn GRPO from live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establish TVRBench as a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.

Liyang Li, Muzhi Zhu, Zhiyue Zhao +5

Zhejiang University

3D Foundation Models Visual Spatial Reasoning Vision-Language-Action Models RL for VLMs

Ask PDF

Oct 27, 2026

Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning

Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.

Zengqi Zhao, Zun Wang, Shoubin Yu +2

UNC Chapel Hill

Visual Spatial Reasoning Spatial Reasoning Spatial Reasoning Benchmarks

Ask PDF

Abstract

Visual Spatial Reasoning

Explore similar work

AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models

Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning