From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

Feiran Wang; Xiaoqi Wang; Ziwei Li; Wenbin He; Yan Yan; Liu Ren

cs.CVOct 2, 2026

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

Authors: Feiran Wang, Xiaoqi Wang, Ziwei Li, Wenbin He, Yan Yan, Liu Ren

Organizations: University of Illinois at Chicago · Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI)

Abstract

Predicting future spatial states supports collision avoidance and timely decision-making in dynamic environments. However, existing vision-language models (VLMs) and benchmarks for spatial reasoning primarily focus on observed scenes, leaving predictive spatial reasoning beyond the observed interval underexplored. To this end, we introduce SpatialMind, a metric-scale VLM for spatial reasoning and future prediction. Its metric depth adapter anchors spatial reasoning to real-world scale, while its progressive state chain establishes current spatial states and observed dynamics as the foundation for future prediction. Given a video prefix, SpatialMind predicts distances, motion directions, and spatial relations in both observed and unseen future frames. For training and evaluation, we build a scalable data engine that grounds entity descriptions in metric geometry to generate question-answer pairs and state supervision. Using this engine, we construct the SpatialMind-30K dataset and the SpatialMind-2K benchmark, both covering driving and everyday egocentric scenes. The benchmark spans eight tasks across three levels: current-state understanding, observed-dynamics understanding, and future prediction. Experiments show that SpatialMind substantially outperforms both general and spatially specialized models on our benchmark while achieving competitive zero-shot performance on VSI-Bench, OSI-Bench, and VLM4D.

Figures & tables

Figure 1: SpatialMind enables future spatial prediction by reasoning progressively from observed evidence, which can help identify collision risks and support timely action to avoid hazards. By grounding prediction in current geometry and observed dynamics, SpatialMind yields more accurate estimates than a proprietary VLM. (a) The 3D scene illustrates the spatial context for anticipating future interactions between the ego vehicle and the white sedan. (b) Benchmark results show consistent improvements over the compared models across all eight spatial reasoning tasks.

Figure 2: SpatialMind Data Engine. We assign each target a distinctive natural-language description through a Select–Describe–Verify process, then combine metric geometry, semantic propagation, and temporal alignment to construct spatial metadata, which supports automatic QA generation across three levels and eight spatial reasoning tasks, with the data distribution shown on the right.

Figure 3: Model Architecture. Given an observed RGB video and a spatial question, SpatialMind refines metric depth with multiview geometry through global and local corrections. We separately encode metric geometry and camera motion, align their features temporally, and integrate them into visual tokens through cross-attention. The VLM then generates a progressive state chain that grounds the target, establishes its current spatial state, and extends to observed dynamics and future prediction according to the question level before producing the final answer.

Proprietary Models (API)
		L1: Current state		L2: Observed dynamics			L3: Future prediction
Model	Overall	Motion	Spatial	Ego	Object	Relative	Direc.	Dist.	Relat.
GPT-5.6 Luna	22.60	20.28	14.34	12.59	25.87	44.76	27.27	14.04	25.61
Claude Sonnet 5	27.80	16.43	23.08	9.79	26.57	49.30	30.07	23.51	44.21
Gemini 3.8 Flash	30.19	21.62	18.66	19.39	49.65	52.85	20.28	41.34	22.91
GPT-6-Astra (medium)	39.00	38.46	13.99	19.58	51.75	73.78	39.86	29.12	52.28

Table 1: Results on SpatialMind-Benchmark (%). Tasks are grouped by reasoning level. Among evaluations within each group, bold marks the best result and underline marks the second best.

VSI-Bench
Model	Avg.	Meas.	S-T
GPT-4o	34.0	29.1	28.5
Gemini-1.5-Pro	45.4	46.2	34.6
InternVL2-8B	37.5	40.7	46.4
LLaVA-NeXT-Video-72B	40.9	38.5	48.6
LLaVA-OneVision-72B	40.2	39.7	44.6

Table 2: Zero-shot evaluation on external benchmarks. VSI-Bench reports measurement (Meas.) and spatiotemporal (S-T) reasoning; OSI-Bench reports static and dynamic reasoning; VLM4D reports the real and synthetic splits. Our method achieves competitive results across three benchmarks.

Table 6

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task family	Ego	Driving	Ego	Driving
	Training: SpatialMind-30K		Test: SpatialMind-2K
L1: Motion state	2,889	2,125	143	143
L1: Spatial state	1,968	2,312	143	143
L2: Ego-motion distance	3,758	7,073	143	143
L2: Target-motion distance	0	1,682	0	143
L2: Relative motion	308	1,734	143	143

Appendix

Table 5: Question distribution by source domain and task family. Training statistics describe SpatialMind-30K, and test statistics describe SpatialMind-2K. A zero denotes an absent source-domain and task-family combination.

Figure 4: Persistence of annotation-derived current states in future distance and relation questions. A distance passes if its unchanged current value is within the future item’s tolerance; a relation passes if the current and future categories match. Counts are 143 per task in the egocentric domain and 142 per task in driving. “GT persistence” uses the annotated current state and is not a model receiving only RGB.

Figure 5: Numerical-error composition on the 1,286 numerical questions in SpatialMind-2K. StateChain denotes our progressive state-chain reasoning setting. Wedges group absolute error divided by the item’s primary tolerance into

[0,1]

(1,2]

(2,4]

, and

(4,\infty)

; unparseable responses remain in the denominator as a separate category. Ring centers show the within-tolerance percentages, with the corresponding counts below.

Figure 6: Level 2 relative-motion example, from the question and observed video to structured supervision. The target pickup recedes from the observer, giving option E. The question header and panels A–D show the observed clip, depth and semantics, geometric calculation, and progressive state-chain target.

Figure 7: Reference-answer computation for Figure 6 . This is the expanded record behind the compressed calculation in panel C: grounding and temporal scope, per-frame measurements, normalization and decision rules, cross-checks, answer mapping, and numerical verification. The chain documents how the displayed measurements determine the answer and its supervision.

Figure 8: Level 1 motion-state example of egocentric scene: averaging 24 camera-speed samples over $0.6$ – $5.2$ s gives $1.3218$ m/s, with $1.32$ m/s as the training answer. The displayed state chain is a supervision target.

Figure 9: Level 1 spatial-state example of egocentric scene: the median range of 135 valid samples on the frying pan’s visible surface is $0.645$ m. This question uses a visible-surface statistic rather than an object-centre or nearest-point distance; outlines are reader annotations, and the state chain is a training target.

Figure 10: Level 2 target-motion example: summing five ground-trajectory segments over $5.2$ – $7.2$ s gives the cyclist a route length of $8.589$ m, with $8.6$ m as the training answer. The route uses six source positions; boxes are reader annotations, and the chain is a supervision target.

Figure 11: Level 3 target-future-distance example: frames 1–20 are supplied to query the distance from the backpack wearer’s centre to the optical camera origin at frame 40. The displayed reference is $16.75$ m and the serialized training answer is $16.8$ m. Dashed future references and chain targets illustrate supervision; neither future frames nor future-state text is supplied at inference.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 244 papers

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Jinyue Su, Dingming Li +5Spatial Reasoning Spatial Reasoning Benchmarks
Zhejiang University
World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
Apr 29, 2026Wanyue Zhang, Wenxiang Wu, Wang Xu +6Visual Spatial Reasoning VLM Adaptation
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Harbin Institute of Technology +2
EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning
Oct 3, 2026Yutong Li, Molin Wang, Xiaotong Li +3Video Reasoning VLM Evaluation
Fudan University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China

Oct 8, 2026J/K move · Enter open · S save

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Hongxing Li, Jinyue Su, Dingming Li, Wenqi Zhang+4

Zhejiang University

Spatial Reasoning Spatial Reasoning Benchmarks Visual Spatial Reasoning Spatial Reasoning in Language Models VLM Reasoning 3D Spatial Reasoning

Open paper AskPDF arXiv

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.

Abstract

Visual Spatial Reasoning

Explore similar work

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning

EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning